At some point, every data team hits the same roadblock. Each night, the script that used to run smoothly starts to go wrong at 2 a.m. No one knows what step has gone wrong, and reruns are double-counting the numbers that the finance team has already published. Apache Airflow is great because it’s been rebuilt from the ground up to suit the way work teams actually do things today.

This cookbook provides you hundreds of solutions that are independent and will take you from the first installation to a platform that supports ETL, ELT, MLOps, AIOps and business operations all at the same time. Each recipe starts with a real problem, has short and easy-to-read code, and ends by showing how the fix works with real terminal output. We can learn to author DAGs with the Task SDK, schedule pipelines on data rather than on the clock, bind extractions so reruns repair instead of duplicates, and run any task inside its own container. We will be practising to write custom operators, packaging them for other teams, extending Airflow through plugins and secrets backends, orchestrating model training and promotion, provisioning infrastructure that tears itself down, and diagnosing stalls from their symptoms.

To me, this book is best suited for every software engineer, backend developer and every such platform teams who keep on building, running and troubleshooting the workflows every day.

Key Learnings

Convert legacy operator DAGs into TaskFlow functions that infer dependencies from ordinary calls

Bound every extraction to its data interval so reruns repair rather than duplicate

Schedule pipelines on asset updates instead of guessing when upstream work finishes

Fan tasks out dynamically over lists discovered at runtime using expand and partial

Run any task inside a Kubernetes pod with its own image and resources

Write custom operators, hooks and sensors, then package them as installable providers

Extend Airflow through plugins, macros, secrets backends and custom XCom storage

Orchestrate model training, registry logging and promotion gates without a serving endpoint

Provision ephemeral infrastructure with setup and teardown pairs that always release resources

Diagnose stuck queues, starved pools and zombie tasks from their distinct symptoms

Table of Content

Getting Airflow 3 Running

Authoring DAGs with Task SDK

Scheduling, Assets and Event-Driven Pipelines

Building ETL Pipelines

ELT and Warehouse Orchestration

Extracting Insights from Batch Processes

Containers and Kubernetes

Custom Operators, Hooks and Sensors

Plugin Interface and Extending Airflow

Managing ML Pipelines

AIOps, Infrastructure and Business Operations

Testing, Monitoring and Troubleshooting

Target Readers

You don’t need to be an Airflow pro or an expert to get started with this book. All you need is to be curious, have a Linux machine and be ready to run the commands instead of just reading them. If you’re committed to creating a strong, flexible and expandable workflow, you’re on the right track.