Apache Airflow - The Complete Hands-On Zero To Champ

Lab - author, schedule and monitor data pipelines through practical examples using Apache Airflow and Practice Exams

What you'll learn

Develop plugins to extend the capabilities of Apache Airflow.

Master in core functionalities such as DAGs, Operators, Tasks, Workflows, etc

The difference between Sequential, Local and Celery Executors, how do they work and how can you use them.

Install and configure Apache Airflow

Understand and apply advanced concepts of Apache Airflow such as XCOMs, Branching and SubDAGs.

Incorporate Apache Airflow within a Big Data ecosystem.

Engage in critical thinking, devise solutions, and implement them using Airflow to tackle real data processing challenges.

Requirements

A very basic knowledge of Python will be an add-on.

Description

Apache Airflow is a powerful open-source platform designed for programmatically authoring, scheduling, and monitoring workflows. If you find yourself dealing with multiple ETL(s) to manage, Airflow becomes an essential tool in your arsenal.This comprehensive course will equip you with everything you need to begin harnessing the capabilities of Apache Airflow. Through a combination of theory and practical videos, you will embark on a journey starting with the fundamentals, understanding what Airflow is and how it operates. From there, we'll delve into more advanced topics, such as creating plugins and building dynamic pipelines to tackle real-world challenges effectively.What's included in the course ?Complete Apache Airflow concepts explained from Scratch to ADVANCE implementation.Each and every Airflow concept is explained with HANDS-ON examples.Includes each and every, even thin detail of Airflow.After completing this course, you can start working on any Airflow project with full confidence.Who this course is for:Those with a natural curiosity for data engineering.Individuals seeking to grasp both basic and advanced concepts of Apache Airflow.Enthusiasts who prefer a hands-on learning experience.Add-OnsQuestions and Queries will be answered very quickly.The course provides attached Airflow codes and datasets used in lectures for your convenience.I am going to update it frequently, every time adding new components of Airflow.

Overview

Section 1: Introduction

Lecture 1 What is Airflow? What is Workflow?

Lecture 2 Why Airflow?

Section 2: Install Apache Airflow on Windows

Lecture 3 Enable WSL

Lecture 4 Notes: Install WSL

Lecture 5 Install WSL

Lecture 6 Notes: Environment Setup

Lecture 7 Environment Setup

Lecture 8 Install Editor

Lecture 9 Notes: Install Airflow

Lecture 10 Install Airflow

Section 3: Kick start with Airflow Lab

Lecture 11 Basic Workflow with Dummy Operators in Apache Airflow

Lecture 12 Basic Workflow using BashOperator in Apache Airflow

Lecture 13 Basic Workflow using PythonOperator vs BashOperator

Section 4: Catch-up & Backfill

Lecture 14 Airflow DAG with Catchup Enabled

Lecture 15 Airflow DAG with Catchup Disabled

Lecture 16 Backfilling in Apache Airflow

Section 5: Scheduling a DAG

Lecture 17 Scheduling a DAG to Run Daily at 11:45 PM in Airflow.

Individuals with a curiosity for data engineering.,Individuals seeking to acquire knowledge of both fundamental and advanced concepts about Apache Airflow.