2 Real World Azure Data Engineer Project End to End

Project Building from scratch including designing, architecting, implementing solution and overall testing.

What you'll learn
You will learn how to Architect, Design and build a real-world enterprise level data platform solution including multiple services.
You will learn design solution using ADF, Azure Function, Databricks, pyspark, Azure Data lake storage Gen 2 (ADLS), Azure SQL Server
You will learn how to build a real-world data pipeline in Azure Data Factory (ADF). This course has been taught using 2 real world use case scenarios.
You will learn how to transform data using Databricks Notebook Activity in Azure Data Factory (ADF) and load into Azure Data Lake Storage Gen2
You will learn how to build production ready pipelines and good practices and naming standards
You will learn how to integrate Databricks with ADF and send the response back from Databricks to ADF
You will learn how to develop the triggered based Azure Function to validate files.
You will learn how to create Azure Key vault and use it to store secret credentials and SAS token
You will learn how to connect the Azure SQL Database and Databricks cluster using the Key Vault
You will learn how to mount he Azure Storage Account in the Databricks to access the files and preform transformation on it.
You will learn how to transform the data in the Azure Databricks using the pyspark.

Requirements
Basic understanding about cloud computing will be useful, but not necessary.
Experience in Azure is not required, I will take you through everything necessary to learn this course and build the project
Description

This course will help you in preparing and mastering your Azure Data engineering Concepts.

It is not like any random project like covid, or twitter analysis. These project is real world projects on which I personally worked and developed it for big clients.

Highlights of the Course

Designed to keep only précised information no beating around the bush. (To save your time).

Real time implementation, no dummy use case.

Can be added as part of your resume.

It will help you to showcase your experience in interviews and discussion.

Involve complex architecture solution which is aligned with industry best practices.

Single projects involve various component integration like ADF, ADLS, Databricks, Azure SQL DB, Key Vault.

Solves the problem of real time experience for new Data engineers.

This course has been developed in mind to keep all the best practices followed in the Industry as an data engineering project and solution.

Technologies involved

Azure Data Lake Storage Gen 2

Azure Data Factory

Data Factory Pipeline

Azure Functions

Azure Key Vault

Azure SQL DB

SSMS

AWS S3 Bucket

Connect ADF to Databricks

Connect Databricks to SQL Server

Connect Databricks to ADLS

Connect S3 to Azure Cloud

Triggers

SAS token

Create Secrets scope in Databricks

Store secretes in Key Vault and access them

What you will learn after this course

How to think, design and develop the solution in the data engineering world.

How to create the architecture diagram for data engineering projects.

How to Create Azure Data Factory Account

How to Create Azure Data Lake Storage Gen 2 account.

How to Create Azure Databricks Workspace.

How to create S3 storage account.

How to create Azure Function.

How to implement logic in the Databricks notebook using pyspark.

How to connect ADF to Databricks.

How to chain the multiple pieces together in project.

How to create Azure SQL Server.

How to load the data from file to Azure SQL server.

How to connect Databricks notebook with Azure SQL Server.

How to Store secrets in the Azure Key Vault.

Who this course is for
Aspiring Data engineer who are searching for project to add in resume
Someone who is looking for Real World uses cases to implement as Data engineering Solution
University students looking for a career in Data Engineering
IT developers working on other disciplines trying to move to Data Engineering
Data Engineers/ Data Warehouse Developers currently working on on-premises technologies, or other cloud platforms such as AWS or GCP who want to learn Azure Technologies
Data Architects looking to gain an understanding about Azure Data Engineering stack
Data Scientists who want extend their knowledge into data engineering