Data Engineering
Building high-throughput, fault-tolerant, and cost-effective data pipelines and streaming backbones. Here, I write about my experiences scaling big data processing workloads, designing event-driven microservices, and writing clean pipeline orchestration systems.
Designing Dynamic, Scalable Airflow DAGs at Enterprise Scale
How to avoid writing duplicate pipeline code by auto-generating Airflow DAGs from YAML configurations and database metadata.
Event-Driven Data Architectures: Production Kafka Patterns & Schema Evolution
Mastering real-time streaming architectures with Apache Kafka, from partition design to managing schemas using Avro and Confluent Schema Registry.
Staff-Level Guide to Scaling Apache Spark: Tackling Data Skew and OOMs
Deep dive into production strategies for optimizing Spark jobs, managing memory allocation, and solving skew issues in massive datasets.