Build End - End Azure Data Engineering Project

Build Your Azure Data Engineering Skills with Hands-On Experience in an End-to-End Project - Develop Real-World Solution

What you'll learn

In this course, we will teach you how to build an end-to-end data engineering project on Azure.

You'll learn to gather data from various sources, store it in Azure Data Lake Storage, process it with Azure Data Factory, and apply analytics to the processed

This course is suitable for anyone with knowledge of SQL and Python. Whether you're a software engineer, data analyst, or IT professional

You'll gain valuable insights into building scalable and reliable data pipelines on Azure

Requirements

Basic Knowledge of Azure, Databricks

Description

In this course, we will teach you how to build an end-to-end data engineering project on Azure. You'll learn to gather data from various sources, store it in Azure Data Lake Storage, process it with Azure Data Factory, and apply analytics to the processed data using Azure Databricks. Throughout the course, you'll build a project from scratch, following the best practices and design patterns used by professional data engineers. By the end of the course, you'll have a deep understanding of the Azure data engineering stack and be able to apply your skills to real-world projects. This course is suitable for anyone with knowledge of SQL and Python. Whether you're a software engineer, data analyst, or IT professional, you'll gain valuable insights into building scalable and reliable data pipelines on Azure. Start your journey to becoming an Azure data engineering professional with this comprehensive course. Enroll in this course now and clear your Azure Interviews with 100% guaranteed. I have used below skills1. ADF2. Databricks3. ADLS Gen24. Blob Storage5. Service Principle6. IAM7. Azure Synapse8. Rest API9. Python10. SQL11. Github12. Azure DevopsBuild Your Azure Data Engineering Skills with Hands-On Experience in an End-to-End Project - Develop Real-World Solutions and Earn In-Demand Skills Today!

Overview

Section 1: Let's understand Azure Resources and How to create them

Lecture 1 Architecture

Lecture 2 Introduction of ADF

Lecture 3 Introduction of Blob Storage

Lecture 4 Introduction of AzureDataLake Gen2

Lecture 5 Introduction of Key Vault

Lecture 6 What is SPN and How to register SPN?

Lecture 7 Introduction of SQL

Lecture 8 Introduction of Azure Databricks

Lecture 9 Introduction of Azure Logic App

Lecture 10 Introduction of Azure Automation Account

Lecture 11 What is API? and How to Register API_Key ?

Section 2: Data Preparation and Uploading into the respective sources

Lecture 12 Import files,data into Azure Blob, DataLake Gen2, SQL Database

Section 3: Ingest Data from multiple source

Lecture 13 Create LinkedServices and Grant Access to Resource

Lecture 14 Create Datasets and Ingest CSV files

Lecture 15 Copy SQL tables from Server to DataLake Gen2 using parameterization

Lecture 16 Unzip files and Ingest them into Data Lake Gen2 with single pipeline

Lecture 17 Fetch Data from API using Rest API and Web Activity to fetch credentials

Lecture 18 Setup Repository and commit ADF changes to Github Repository

Lecture 19 Updates on ADF Pipeline

Lecture 20 Create Mount Point of Azure Blob Storage using Databricks

Lecture 21 Create Mount Point of Azure DataLake GEN2 and Dumping PDF files using Databricks

Lecture 22 Copy PDF files from Azure Blob Storage using parameterization to Azure Datalake

Section 4: Cleaning Process and Data Quality Check

Lecture 23 Clean the Data using AUTOLOADER in Databricks - Part 1

Lecture 24 Clean the Data using AUTOLOADER in Databricks - Part 2

Lecture 25 Clean the Data using AUTOLOADER in Databricks - Part 3

Lecture 26 Clean the Data using Batch processing in Databricks

Lecture 27 Data Quality checks on Cleansed Layer using Delta

Lecture 28 Commit Databricks Notebook using Repo feature in GitHub

Section 5: Create Dimesions and Fact tables

Lecture 29 Create Dimesions and Fact tables

Section 6: Publish the data into SQL DataBase using ADF

Lecture 30 Publish final tables into SQL Databases using Azure Databricks Delta Lake

Section 7: Setup complete ADF Pipelines

Lecture 31 Create Master Pipeline to run the complete project

Section 8: Send Data to End user using Logic App

Lecture 32 Multiple Ways to Send Data files to End Users

Lecture 33 Optimize way to send Data Files to End Users

Section 9: Run Automation Account script using Runbook

Lecture 34 Switch to SQL DW and Start/Stop the SQL DW using Automation Account

Section 10: Data Quality Layer between Mart and Publish

Lecture 35 Build Data Checks Utility to check Mart logics in Databricks

Section 11: CI-CD using Azure Devops and Github Actions

Lecture 36 Azure Data Factory Deployment

Lecture 37 Azure Blob Storage and GEN2 files Deployment

Lecture 38 Azure Databricks Deployment using Github Action

Lecture 39 Add Trigger and Global Parameters and Fix CI-CD for Azure Data Factory

Section 12: Handle corner cases in ADF

Lecture 40 How to use Filter Activity to handle Failure Runs in ADF

Section 13: Build Dashboard and Report using PowerBI

Lecture 41 Import Data and Create Report

Students who are looking to change the job