Skip to content
Back to projects
Diagram: SQL Server, CSV and Kafka feed Spark, orchestrated by Airflow, which writes Bronze, Silver and Gold Delta Lake layers on MinIO
Data EngineeringApplied academic project2026

Spark, Kafka and Airflow data platform

Processes batch and real-time data with Spark, Kafka and Airflow and organises it into layers of increasing quality (Medallion architecture) on Delta Lake.

  • Apache Spark4.0.1
  • Delta Lake4.0.0
  • Apache Kafka4.1.1
  • Apache Airflow3.0.6
  • MinIOS3 local
  • SQL Server2022
  • DockerCompose
  • LicenseMIT
Technical documentationArchitectureRepository

Problem

Maintain a living metadata catalogue by combining structured, semi-structured and real-time event sources.

Architecture

SQL Server, CSV and Kafka feed Spark jobs that publish Bronze, Silver and Gold layers on Delta Lake and MinIO. Airflow orchestrates the batch flows.

Data flow

Sources → batch or streaming ingestion → Spark → Medallion Delta Lake → verifiable Gold previews.

Results

3

Ingestion patterns

Incremental SQL batch, CSV batch and Kafka streaming

View source

3

Data layers

Bronze, Silver and Gold in every pipeline

View source

Contribution and authorship

  • Use-case and Medallion architecture design
  • Implementation of Spark jobs and Airflow DAGs
  • Reproducible integration of Kafka, SQL Server, MinIO and Delta Lake
Alonso Marcos Muñoz · Author and developer

Evidence

Executive architecture
Executive architecture
Technical architecture
Technical architecture

Limitations

  • • Reproducible local environment with demo data, not an enterprise production platform
  • • Included credentials are local development values only

Stack

PythonApache SparkApache AirflowApache KafkaDelta LakeMinIOSQL ServerDocker Compose

https://github.com/alonsomarcosm99/spark-kafka-airflow-data-platform