Apache Airflow From Basics To Mastery

Learn Apache Airflow with practical examples and build efficient, scalable data pipelines

What you'll learn

How to write complex pipelines with just a few lines of code

The look and feel of the Airflow tool and its ecosystem

Step-by-step guidance on moving from simple to complex system components

How to avoid mistakes and understand why they happen

You will learn through practical examples based on common real-world usage

Requirements

You should be familiar with the basics of Python

Description

Apache Airflow From Basics to Mastery is your step-by-step guide to advancing your workflow automation skills. Whether you’re just getting started or looking to deepen your understanding, this course will help you write efficient, scalable, and dynamic pipelines with minimal code. You'll explore the core features of Apache Airflow, learning how to move from simple to complex system components while avoiding common mistakes and understanding why they happen.Throughout the course, you'll gain hands-on experience with real-world examples, ensuring you can confidently apply what you learn in practical scenarios. You’ll work with Docker containers, create your first DAG, and explore key features like branching, trigger rules, and decorators. The course covers essential operators, hooks, and sensors, including AWS S3, Postgres, SQL, and more.As you progress, you’ll dive into scheduling, dataset management, and Airflow’s powerful integration capabilities, such as working with DBT, securing passwords with a Fernet key, and managing access control. Additionally, you’ll learn best practices for versioning your code with Git, ensuring your workflows remain organized and maintainable.This course is designed for anyone looking to boost their Airflow skills or use it regularly for workflow automation. Whether you’re a data engineer, developer, or DevOps professional, you’ll walk away with a strong foundation in Apache Airflow and the confidence to build robust data pipelines.

Overview

Section 1: Introduction

Lecture 1 Introduction

Section 2: Development environment

Lecture 2 How to create development environment

Section 3: Airflow ecosystem

Lecture 3 Airflow ecosystem

Section 4: How to Create Your First DAG

Lecture 4 How to Create Your First DAG

Lecture 5 Branching and trigger rules

Section 5: Decorators

Lecture 6 @dag and @task decorators

Lecture 7 Dynamic tasks and task group

Lecture 8 BashOperator

Section 6: Hooks

Lecture 9 AWS S3 Hook

Lecture 10 Postgres Hook

Section 7: Sensors

Lecture 11 S3 Sensor

Lecture 12 SQL Sensor

Section 8: Scheduling

Lecture 13 Scheduling and datasets

Section 9: Additional information

Lecture 14 Integration with DBT

Lecture 15 Versioning code in GIT

Lecture 16 Storing passwords with Fernet key

Lecture 17 Roles and Access Control

This course is for anyone looking to boost their Airflow skills or use it regularly for workflow automation