Azure Data Factory +Synapse Analytics End To End Etl Project

Complete end to end Project on OTT platform using Azure Data Factory and Azure Synapse Analytics from Scratch with Labs

What you'll learn

Create incremental pipeline based on last modified date

Calling Azure Synapse notebook from Azure Data Factory

Configure CICD from Scratch in Azure Data Factory

Build end to end project using Azure Data Engineering Services

Create a Self Hosted Integration Runtime and ingest data from On-premise Environment

How to transform data with Azure Synapse Analytics

Create and configure an Azure Data Lake

Project Oriented Learning of Azure Data Engineering services

Create an Azure Key vault service

Using Secrets of Key vault in Linked services

Create and configure an Azure SQL Database

Writing PySpark transformation logic in Synapse Notebook

Use of Triggers in Azure Data Factory

Create an incremental data load pipeline to ingest file in daily basis

Build an Automated pipeline in Azure Data Factory

Setup CICD environment from Scratch

Create incremental pipeline based on file name and pick today's file

Perform Data Analysis Using PySpark Code using Synapse Notebook

Create and configure an Azure Data Factory

Create and configure an Azure Synapse Analytics

Orchestrate all pipelines in Azure Data Factory

Loading Data into Azure SQL Database

Ingest Data from On-premise to Azure Datalake in Azure Cloud

Send an Automatic Alert email when a pipeline is failed

Transform only Today's file in Azure Synapse Analytics

Report transformed data in Power BI

Complete hands-on project using Azure Data Engineering Services

Provides insights on services that needed to clear DP-203

Requirements

Basics of Azure Data Factory

An Azure account with Subscription to perform Labs

No Synapse Analytics Experience needed, You will learn everything you needed.

Description

Are you looking to build an end-to-end ETL project using Azure data engineering services? Look no further than this comprehensive course on Azure Data Factory and Azure Synapse Analytics. This course is designed to guide you through the process of creating all the necessary services from scratch, before building an ETL project from start to finish.Throughout this course, you'll gain practical, hands-on experience with Azure Data Factory and Azure Synapse Analytics, learning how to use these powerful data engineering tools to create a highly effective ETL solution. You'll explore the many features and capabilities of these platforms, as well as their integration with other Azure services like 1. Azure SQL Database2. Azure Synapse Analytics 3. Azure Key Vault 4. Azure Data Factory for Orchestration,5. Azure Storage solutions (Azure Datalake Gen2)6. Microsoft Power BI7. Azure Logic AppsIn addition to the core ETL project, this course also includes an additional section on CICD (Continuous Integration and Continuous Deployment) on Azure Data Factory, helping you to fully automate your data engineering workflow.Whether you're a beginner or an experienced data engineer, this course is designed to help you gain a comprehensive understanding of Azure Data Factory and Azure Synapse Analytics. By the end of the course, you'll be able to confidently create and manage your own ETL projects on the Azure cloud.Enrol now and take your data engineering skills to the next level!

Overview

Section 1: Introduction

Lecture 1 Welcome to the course

Lecture 2 Main Focus and Pre-requisites

Lecture 3 Services used in this project

Lecture 4 Project Overview

Lecture 5 Project Architecture

Lecture 6 Additional section on CICD setup for Azure Data Factory

Lecture 7 Course Structure

Lecture 8 Understanding OTT dataset

Lecture 9 Resources

Section 2: Environment Setup

Lecture 10 Environment setup - Intro

Lecture 11 Creating a budget for our Project

Lecture 12 Creating a resource group

Lecture 13 Creating an Azure Data Factory

Lecture 14 Creating an Azure Datalake Storage Gen2

Lecture 15 Creating an Azure Synapse Analytics Workspace

Lecture 16 Suggestion on Saving costs for Azure SQL Database

Lecture 17 Creating an Azure SQL Database

Lecture 18 Installing Power BI Desktop

Section 3: Data Ingestion

Lecture 19 Data Ingestion - Intro

Lecture 20 Data Ingestion - Integration Runtimes

Lecture 21 Data Ingestion - What is Self Hosted Integration Runtime

Lecture 22 Overview of On-premise data source and Datalake

Lecture 23 Downloading and installing Self Hosted IR in On-premise Environment

Lecture 24 Creating and adding Secrets to Azure Key vault

Lecture 25 Creating Linked Service for Azure Key vault - Demo

Lecture 26 Creating Linked Service and Dataset for On-premise File Storage

Lecture 27 Creating Linked Service and Dataset for Azure Datalake

Lecture 28 Creating Copy Activity to copy all files from On-premise to Azure Datalake

Lecture 29 Incremental data loading using Last Modified Date of File

Lecture 30 Incremental Load based on File Name - Demo

Lecture 31 Incremental Data loading based on Filename - Practical

Section 4: Transformation

Lecture 32 Transformation - Intro

Lecture 33 Azure Synapse Analytics - Introduction

Lecture 34 Assigning Role for Synapse in Azure Datalake - Demo

Lecture 35 Assigning role and creating linked service in Azure Synapse Analytics- Practical

Lecture 36 Reading CSV files from ADLS from Synapse Notebook - Practical

Lecture 37 Stop spark session manually to save cost

Lecture 38 Identify and delete duplicate rows - Demo

Lecture 39 Identify and remove duplicate rows - Practical

Lecture 40 Identify and Remove or Replace NULL values - Demo

Lecture 41 Identify and Remove or Replace NULL values - Practical

Lecture 42 New column based on IMDB Rating - Demo

Lecture 43 New column based on IMDB Rating - Practical

Lecture 44 New column based on Runtime in Hours - Demo

Lecture 45 New column based on Runtime in Hours - Practical

Lecture 46 Practise Activity for creating a new column

Lecture 47 Solution PySpark code for Practise Activity

Lecture 48 Changing data types from String to Date Type- Demo

Lecture 49 Changing String to Date Data Type - Practical

Lecture 50 Writing transformed data to ADLS - Demo

Lecture 51 Writing transformed data to Datalake - Practical

Lecture 52 End of Writing Transformation Code

Lecture 53 Calling Synapse Notebook Activity from Azure Data Factory- Demo

Lecture 54 Calling Synapse notebook from Azure Data Factory - Practical

Lecture 55 Transformation - Conclusion

Section 5: Data Loading

Lecture 56 Data Loading - Section Intro

Lecture 57 Installing and Accessing Azure SQL Database from SSMS

Lecture 58 Loading Data to SQL Database - Demo

Lecture 59 Copying data to SQL Database - Practical

Lecture 60 Fix error in Linked service while creating for SQL Database

Lecture 61 Conclusion

Section 6: Enchancements

Lecture 62 Enhancements -Section Intro

Lecture 63 Enhancement for Copy data from On-premise to ADLS- Demo

Lecture 64 Enhancement For Copy data from On-premise - Practical

Lecture 65 Enhancement for synapse notebook - Demo

Lecture 66 Enhancement for synapse notebook to transform only todays file

Section 7: Orchestration

Lecture 67 Orchestration - Section Intro

Lecture 68 Orchestrating the pipelines- Demo

Lecture 69 Orchestrating all the pipelines and make it an automated pipeline - Practical

Lecture 70 Send an automatic alert email notification when pipeline failed in ADF - Demo

Lecture 71 Send an alert email notification when pipeline is failed - Practical

Lecture 72 Executing all pipelines for loading data into SQL

Section 8: Reporting

Lecture 73 Reporting Section- Intro

Lecture 74 Reporting Data in Power BI - Practical

Section 9: Additional Section - Configuring Continuous Integration Continuous deployment

Lecture 75 CICD - Introduction

Lecture 76 What is Continuous Integration

Lecture 77 What is Continuous Deployement

Lecture 78 CICD- Part - 1

Lecture 79 CICD - Part - 2

Lecture 80 CICD - Part - 3

Lecture 81 CICD - Part - 4

Section 10: Conclusion

Lecture 82 Conclusion for the course

Azure Data Engineers curious to do Hands-on practical Labs,Data Engineers who want real time project experience using Azure Data Engineering Services,Students aspiring to learn real time project on Azure Data Engineering