Azure Data Engineering Preparation With Real World Projects

2 Real World Projects Using Azure Data engineering Tool Like Azure Data Factory, Functions App, Azure Data Bricks, AZSQL

What you'll learn
2 End to End Azure Data Engineering Projects
Learn Azure Data Factory
Learn Azure Databricks
How to transform data in the Azure Databricks using the pyspark
How to mount a storage account in Databricks Azure?
10 Real time use cases of Azure Data engineering services
15 Common Azure Data Engineering Interview questions and answers
Requirements
Basics of Cloud computing
Internet Connections
Mobile Phone or Laptop Or Desktop
Description
Hello, Welcome to this New Azure Data Engineering Course. In this course We will demonstrate two real world scenario based projects using Azure Data Engineering Tool. This will give a real time experience to some of the common scenarios that you may face in your day to day life as azure Data engineer. Is Azure good for data engineer?Microsoft Azure is one of the best tool for data engineers as they can build different applications  and deploy it any azure regions to utilizing their existing capabilities. No matter where their data is stored, azure data engineering services help you to access them, transform them and export them into any storage system that you want use. In this projects we will demonstrate some of the feature that Azure data factory supports. Project One: Read Data From A Online Storge System and process them using Azure Data Factory. This project covers one real-time industry use cases and project work to give you hands-on experiences on using Azure data engineering tools and services like Azure Data factory, Azure functions and Azure SQL .Project Two: Read multiple files using ADF. Validate your file structure using Azure SQL Data base table. Process valid files using Azure Data Factory. Process multiple files using same Azure Data Factory Pipelines Data Flow (Dynamic Data flows) . Make your data flow dynamic, so that single data flow can process multiple files with different structures and columns names and data types. Just wanted add one point: This course cover some of the advanced azure data factory features and functions. It will be really helpful, if you already know some of the basics of azure data factory.  But still we will cover all the topics, that are necessary for completing this project in details (End to End).

Overview

Section 1: Projects 1

Lecture 1 Introduction To Project 1

Lecture 2 Introductions To Part 1 Of This Project

Lecture 3 Introductions To Part 1 Of This Project

Lecture 4 Save Raw Data In GitHub

Lecture 5 Create Azure Data Lake Storage Gen 2 Account (ADLS)

Lecture 6 How To Create Azure Data Factory Account

Lecture 7 How To Create Containers in ADLS?

Lecture 8 How To Create Linked Services ADF?

Lecture 9 How To Create Data Set In ADF?

Lecture 10 How To Create A Pipeline In ADF and Configure Copy Activity

Lecture 11 Create New Data Set and Copy Second files

Lecture 12 How To Reuse Data Set With The Help Of Parameter

Lecture 13 Copy 16 Files Using Single Copy Activity

Section 2: Project 1 :Second Part

Lecture 14 Azure Functions: Intro

Lecture 15 How to Test & Validate Blob Trigger Functions In Azure Functions App

Lecture 16 How To Add Logical Testing Code In Azure Functions, For Validations

Lecture 17 How To Add Output Binding in Azure Functions

Lecture 18 End To End Testing HTTP to Azure Storage Using ADF And Validate Functions App

Lecture 19 Azure Function App: Fix File Name Issues

Section 3: Project 1 : Final Part

Lecture 20 Final Part Of This Project

Lecture 21 How To Create Azure SQL DB ?

Lecture 22 How to Connect To Azure SQL Using SSMS & From Azure Portal

Lecture 23 How To Create Linked Service To Access Azure SQL

Lecture 24 How To Create Data Set To Access Azure SQL DB?

Lecture 25 How To Copy Data Into Azure SQL

Lecture 26 How To Copy Full Data Into Azure SQL

Lecture 27 How To Fix Common Issues

Section 4: Project 2: Mastering ADF Dynamic Pipelines

Lecture 28 Introductions To Project Requirements

Lecture 29 Understand Data and Data Transformations Requirements

Lecture 30 Design Target Table For First Data Set

Lecture 31 Create Data Set: Azure Data Lake and Azure SQL Data set

Lecture 32 Create Data Flow And Add Multiple Source ( ADLS File & Azure SQL Table)

Lecture 33 Make Our Data Flow Using Parameters

Lecture 34 How To Derive New Columns From Existing Columns And Parameters.

Lecture 35 How To Use Exist To Validate Source And Target Data

Lecture 36 Calculate New Surrogate Key And Max Surrogate Key

Lecture 37 Join Max Surrogate Key With New (Or Updated) Data Set

Lecture 38 Derive Additional Columns: Active Status ,and Current Dates

Lecture 39 Select Relevant Column Using Select Activiti-Role Based Mapping

Lecture 40 Process Updated Data Using New Branch Activity

Lecture 41 Select Proper Columns Using Role Based Mappin(Different Expression)

Lecture 42 Define Insert Set Data And Update Set Data

Lecture 43 Merge Insert And Update Data Sets

Lecture 44 Add Sink And Execute Our Pipeline

Lecture 45 Unit Testing: Validate Pipeline Execution Step 1

Lecture 46 Unit Testing : Validate Pipeline Executions Step 2

Section 5: Project 2: Part B

Lecture 47 Introduction To New Data Set

Lecture 48 Make Our Data Set Dynamic Using Parameters

Lecture 49 Make Our Pipeline Dynamic

Lecture 50 Execute Our Pipeline With New Data Set

Section 6: Project 2:Part C

Lecture 51 Introductions To Final Requirements

Lecture 52 Defining Table To Store Structure Of The Table

Lecture 53 How Define A Dynamic Stored Procedure To Read File Structure

Lecture 54 How To Validate File Structure of Two files

Lecture 55 How To Store Structure Details In SQL Table

Lecture 56 Validate Structure Using Azure SQL table and Stored Procedures

Lecture 57 Execute New Pipeline After validations

Lecture 58 Test All the Scenarios (Same Structure , different Structures ) End To END Unit

Section 7: Project 3

Lecture 59 Create Azure Data Lake Gen 2 And Azure Databricks

Lecture 60 Register an application with Azure AD and create a service principal

Lecture 61 Assign Roles To The Application To Provide The Service Principal Permissions

Lecture 62 Add application secret to the Azure Key Vault

Lecture 63 Create a Secret Scope in Azure Databricks

Lecture 64 Create Containers ( bronze/ Raw, silver / Processed , and gold/Final)

Lecture 65 Create Your First Cluster in Databricks

Lecture 66 Create A Notebook

Lecture 67 Mount Azure Data Lake without Key Vault

Lecture 68 Read CSV file from Data Lake

Lecture 69 Mount Data lake using Azure Key Vault

Section 8: Bonus

Lecture 70 Bonus

If you are looking for a Real World Data engineering uses cases, then this course is for you,Any student who is planning to learn azure data bricks,Any student who is planning to learn azure data factory,Students looking for a career in Azure Data Engineering,For all the database developer who wants to learn azure data engineering,For business analyst and data analyst who wants to learn azure data engineering