Why Airflow May Be the Wrong Tool for Medallion Data Pipelines

A data engineer with six years of experience argues that Apache Airflow is poorly suited for medallion architecture pipelines, citing dependency conflicts, lack of atomicity, and excessive orchestrator maintenance overhead. For AWS-based pipelines, the author recommends AWS Step Functions for its server-free state machine model and native support for long-running Spark jobs via callback patterns, while noting its execution history limits. Teams using Delta Lake on Databricks are advised to adopt Databricks Workflows instead, as it integrates directly with cluster lifecycle management and Delta commits, avoiding orphaned clusters. A key architectural warning is raised against monolithic pipelines that chain Bronze, Silver, and Gold layers serially, as a single failure forces costly full reruns. The overall message is that no single orchestrator fits all use cases, and engineers should choose tools based on the specific failure modes their infrastructure is likely to encounter.