Microsoft Dp-203 Certified: Azure Data Engineer Associate

Preparing for Azure Data Engineer Certification: DP-203 Data Engineering on Microsoft Azure Exam

What you'll learn

This course is ideal for students aspiring to achieve the "Microsoft Certified: Azure Data Engineer Associate" Dp-203 certification.

It includes comprehensive content aligned to pass DP-203 exam.

Students will gain hands-on experience in implementing and managing data engineering workloads using Microsoft Azure.

The course covers key Azure services, including Azure Synapse Analytics, ADF, Azure Data Lake Storage Gen2, Azure Stream Analytics, and Azure Databricks.

Past Papers and Practice: Access to 500 exam questions to solidify knowledge and prepare for certification.

Requirements

Prerequisites:

Familiarity with any database system and basic proficiency in SQL is good to have.

A foundational knowledge of data formats is expected. e.g csv etc

Description

This course is ideal for students aspiring to achieve the "Microsoft Certified: Azure Data Engineer Associate" certification.It includes comprehensive content aligned with the DP-203 exam.The course objectives focus on the following areas:Design and implement data storage (15–20%)Develop data processing (40–45%)Secure, monitor, and optimize data storage and data processing (30–35%)This Course structure organizes the course into a logical progression while providing a clear breakdown of the covered topics. Here’s is the structured outline of the course sections:1. Introduction and SetupOverview of the course and initial setup.2. Design and Implement Data StorageAzure Data Lake: Understanding and implementing data storage with Azure Data Lake.Azure SQL Server: Designing storage solutions using Azure SQL Server.Cosmos DB: Exploring storage capabilities with Cosmos DB.Azure Synapse Analytics: Building and managing storage in Azure Synapse Analytics.3. Develop Data ProcessingAzure Synapse Spark Pool: Leveraging Spark pools in Azure Synapse for data processing.Azure Data Factory: Developing ETL pipelines and data flows in Azure Data Factory.Azure Databricks: Implementing data processing workflows with Azure Databricks.Azure Event Hubs: Streaming and processing real-time data using Azure Event Hubs.Azure Stream Analytics: Real-time data stream processing with SQL-based queries.4. Secure Your DataAzure Data Lake Security: Implementing security best practices for Azure Data Lake.Azure Synapse Analytics Security: Securing data in Azure Synapse Analytics.Azure Data Factory and Databricks Security: Ensuring secure data workflows in Azure Data Factory and Databricks.4. Monitor and OptimizeAzure Data Lake Storage: Monitoring and optimizing storage performance.Azure Data Factory: Ensuring efficient operations with monitoring tools.Azure Synapse Analytics: Performance tuning and monitoring analytics workloads.Azure Stream Analytics and Cosmos DB: Streamlining data streams and database operations.Data Governance with Microsoft Purview: Managing and governing data using Microsoft Purview.5. Exam PreparationPast Papers and Practice: Access to 500 exam questions to solidify knowledge and prepare for certification.

Overview

Section 1: Segment1-Introduction and Setup

Lecture 1 1.1.Overview-Cloud Computing

Lecture 2 1.2.Concepts-Introduction to Cloud Computing.mp4

Lecture 3 1.3.Overview-Microsoft Azure.mp4

Lecture 4 1.4.Overview-How we are going to approach the Exam.mp4

Lecture 5 1.5.HandsOn-Create Azure Account.mp4

Lecture 6 1.6.HandsOn-Azure Portal Overview.mp4

Lecture 7 1.7.Overview-IMPORTANT-How to Complete the Course

Lecture 8 Complete Course Files

Section 2: Segment2-Design and implement data storage-DataLake

Lecture 9 2.0.Overview-Data Engineering Basics Concepts

Lecture 10 2.1.Overview of Types of azure Storage Account

Lecture 11 2.2.HandsOn-Create Azure Storage Account

Lecture 12 2.3.Overview of DataLake Gen2 Storage Account

Lecture 13 2.4.HandsOn-Create DataLake Gen2 Storage Account

Lecture 14 2.5.Different Format Types + HandsOn Uploading Files to Azure Storage

Lecture 15 Congrats on Completing a Section

Section 3: Segment3-Design and implement data storage-Azure SQL Server

Lecture 16 3.1.1.Overview-Azure SQL Database

Lecture 17 3.1.2.Overview Azure SQL Pricing Tier and Configuration Types

Lecture 18 3.2.Overview SQL Server Creation on Azure

Lecture 19 3.3.HandsOn SQL Server Creation

Lecture 20 3.4.HandsOn-connecting with SQL Server Using SSMS

Lecture 21 3.5.Installing Azure Data Studio

Lecture 22 3.6.1.HandsON - T-SQL Select Statement Basics

Lecture 23 3.6.2.HandsOn T-SQL Multiple Conditioning using Where Clause

Lecture 24 3.6.3.HandsOn T-SQL Order By Clause

Lecture 25 3.6.4.HandsOn T-SQL Aggregate Functions

Lecture 26 3.6.5.HandsOn T-SQL Group By Clause

Lecture 27 3.6.6.HandsOn T-SQL Partition By Statement

Lecture 28 3.6.7.HandsOn T-SQL LEAD and LAG Statement

Lecture 29 3.6.8.HandsOn T-SQL Common Table Expression using WITH Clause

Lecture 30 3.6.9.HandsOn T-SQL CASE Statement

Lecture 31 Congrats on Completing a Section

Section 4: Segment4-Design and implement data storage-CosmosDB

Lecture 32 4.1.1.Overview-Cosmos DB

Lecture 33 4.1.2.Concepts-CosmosDB use Cases

Lecture 34 4.1.3.Concepts-CosmosDB as a Solution

Lecture 35 4.2.1.Overview-Cosmos DB Pricing

Lecture 36 4.2.2.Concepts-Cosmos DB Throughput Models

Lecture 37 4.3.1.Overview-Cosmos DB APIs

Lecture 38 4.3.2.Concepts-Data Modelling and their APIs

Lecture 39 4.3.3.Concepts-APIs Use-cases for Cosmos DB

Lecture 40 4.4.1.HnadsOn-Cosmos DB Account Creation

Lecture 41 4.5.1.Concepts-Databases, containers, and items in Cosmos DB

Lecture 42 4.5.2.HandsOn-Create Database inside Cosmos DB account

Lecture 43 4.5.3.HandsOn-Create a Container inside Cosmos DB

Lecture 44 4.5.4.HandsOn-Create Items inside Cosmos DB

Lecture 45 4.6.1.Concepts-Physical Partitions in Cosmos DB

Lecture 46 4.6.2.Concepts-Logical Partitions in Cosmos DB

Lecture 47 4.7.1.Concepts-Consistency Levels in Cosmos DB

Lecture 48 4.7.2.HandsOn-Set Consistency for Cosmos DB

Lecture 49 4.8.1.HandsOn-Scaling a Cosmos DB across the Globe

Lecture 50 Congrats on Completing a Section

Section 5: Segment5-Design and implement data storage-Azure Synapse Analytics

Lecture 51 5.1.Overview-Azure Synapse Analytics basic Concepts

Lecture 52 5.2.Overview-Azure Synapse Components

Lecture 53 5.3.Overview-Azure Synapse Compute Power

Lecture 54 5.5.Concept-SQL Database vs SQL Data Warehouse

Lecture 55 5.6.Overview-Serverless SQL Pools

Lecture 56 5.7.Overview-Synapse SQL Underlying Architecture

Lecture 57 5.9.HandsOn-Create Synapse Workspace

Lecture 58 5.10.1.HandsOn-Query Csv from Data Lake

Lecture 59 5.10.2.HandsOn-Setting Role Assignment inside Data lake for Reading Data

Lecture 60 5.11.Overview-Dadicated SQL Pool

Lecture 61 5.12.HandsOn-Creating and Pausing Dedicated SQL Pool

Lecture 62 5.13.0.Overview-Data Loading Methods

Lecture 63 5.13.1.HandsOn-External Table Serverless SQL CSV format-part1

Lecture 64 5.13.2.HandsOn- External Table Serverless SQL csv format-part2

Lecture 65 5.13.3.HandsOn- External Table Dedicated SQL csv format-part3

Lecture 66 5.13.4.HandsOn-External table parquet serverless&dedicated

Lecture 67 5.13.5.HandOn-Copying Data using Polybase

Lecture 68 5.13.6.HandsOn-Copying Data using Copy into Statement

Lecture 69 5.13.7.HandsOn-Copying Data with Auto Schema Detection using Copy into Statement

Lecture 70 5.13.8.HandsOn-Copying Data using Pipelines

Lecture 71 5.14.Concepts-Transverse Files and Folders for Hadoop and Native file system

Lecture 72 5.15.1.Overview-OLAP and OLTP systems

Lecture 73 5.15.2.Overview-Fact and Dimension Tables

Lecture 74 5.15.3.Overview-Star and Snow Flake Schema Design

Lecture 75 5.15.4.0.HandsOn-Transfer Data from SQL to Synapse SQL dedicated pool

Lecture 76 5.15.4.1.HandsOn-Transfer Data from SQL to Synapse SQL dedicated pool

Lecture 77 5.16.1.Concepts-Table or Data Distribution Methods

Lecture 78 5.16.2.Concepts-Distribution Selection

Lecture 79 5.16.3.Concepts-Hash and Replicated Distribution

Lecture 80 5.16.4.HandsOn-Creating Table Distribution in Dedicated SQL Pool

Lecture 81 5.16.5.HandsON-Query Performance and Monitor Tab inside Synapse

Lecture 82 5.16.6.Concepts-Slowly Changing Dimension Types

Lecture 83 5.16.7.0.HandsOn-Business Key for Creation for Dimension tables

Lecture 84 5.16.7.1.HandsON-Business Key for Creation for Dimension tables

Lecture 85 5.17.1.Overview-Use of Index on tables

Lecture 86 5.17.2.Concepts-Partition and Partition Switching

Lecture 87 5.18.0.HandsOn-Create a Restore Point and Restoring a Dedicated SQL pool

Lecture 88 5.19.0.Overview-Database Templates for Synapse Analytics

Lecture 89 Congrats on Completing a Section

Section 6: Segment6-Develop data processing-Azure Synapse Spark Pool

Lecture 90 6.1.0.Overview-History of Big Data Processing

Lecture 91 6.1.1.Overview-Benifits of Apache Spark

Lecture 92 6.1.2.Overview-Spark Underlying Architecture

Lecture 93 6.2.1.Concepts-Spark Pools inside Azure Synapse

Lecture 94 6.2.2.HandsOn-Creating Spark Pool in Synapse

Lecture 95 6.3.1.HandsOn-Creating Dataframe Using Pyhton

Lecture 96 6.3.2.HandsON-Read Data From Data Lake Using Python

Lecture 97 6.3.3.HandOn-Working with Loading function using pyspark

Lecture 98 6.3.4.HandsOn-Writing Data to Dedicated SQL Using Sparkpools

Lecture 99 Congrats on Completing a Section

Section 7: Segment7-Develop data processing-Azure Data Factory

Lecture 100 7.1.Overview-Azure Data Factory

Lecture 101 7.2.Overview-ETL Extract Transform and Load

Lecture 102 7.3.HandsOn-Create Azure Data Factory on Azure

Lecture 103 7.4.0.HnadsOn-Copy data activity from DataLake to Blob Storage

Lecture 104 7.4.1.HandsOn-Copy Data Activity for parquet to json conversion

Lecture 105 7.4.2.HandsOn-Create Pipeline csv to json

Lecture 106 7.5.0.Overview-Pipelines and Mapping Data Flows in ADF

Lecture 107 7.5.1.HandOn-Mapping Data Flow- SQL to Synapse SQL Fact Table

Lecture 108 7.5.2.HandOn-Mapping Data Flow- SQL to Synapse SQL Dimension Table

Lecture 109 7.5.3.HandOn-Mapping Data Flow- SQL to Synapse SQL Store Procedure Activity

Lecture 110 7.5.4.HandOn-Mapping Data Flow- SQL to Synapse SQL Derived Column Activity

Lecture 111 7.5.5.HandOn-Mapping Data Flow- SQL to Synapse SQL Business Key Activity

Lecture 112 7.5.6.Overview-Mapping Data Flow- Data Debug Feature

Lecture 113 7.5.7.Overview-Mapping Data Flow- Sink Types Available

Lecture 114 7.5.8.HandOn-Mapping Data Flow Json Data(Array and Object)

Lecture 115 7.6.0.Concepts-Integration Runtimes

Lecture 116 7.6.1.HandOn-Installing Self hosted IR

Lecture 117 7.6.2.HandOn-Self Hosted IR Copy Data from PC to DataLake

Lecture 118 7.7.0.Pipeline fail senerio

Lecture 119 7.8.0.Overview-Trigger Types in ADF

Lecture 120 7.9.0.Code Representation of ADF components inside GIT

Lecture 121 7.9.1.What is GIT How to version Control ADF

Lecture 122 7.9.2.HandsOn-Configuring GIT with ADF

Lecture 123 7.9.3.HandOn-How to merge changes in Business and create Pull request in ADF

Lecture 124 Congrats on Completing a Section

Section 8: Segment8-Develop data processing-Azure DataBricks

Lecture 125 8.1.0.Overview-Introduction to Databricks

Lecture 126 8.1.1.Overview-Benefits of DataBricks

Lecture 127 8.1.2.Concepts-Azure Databricks

Lecture 128 8.2.0.HandsOn-Create Databricks Workspace

Lecture 129 8.2.1.Concepts-Compute Cluster Types inside Azure Databricsk

Lecture 130 8.2.2.HandsOn-Create Compute inside DataBricks Workspace

Lecture 131 8.2.3.HandsOn-Read data inside DBFS

Lecture 132 8.2.4.HandsOn-Read Data from ADLS Account

Lecture 133 8.2.5.HandsOn-Visulizations inside Notebook in DataBricks

Lecture 134 8.3.0.HandsOn-Processing Json Data

Lecture 135 8.3.1.HandsOn-Saving to Databricks as Table

Lecture 136 8.3.2.HandsOn-Saving Table inside DataLake

Lecture 137 8.4.0.HandsOn-Reading Stream inside Databricks

Lecture 138 8.4.1.HnadsOn-Copying Stream inside Databricks Table

Lecture 139 8.5.0.HandsOn-Writing Data to Synapse SQL

Lecture 140 8.5.1.HandsOn-Reading data From Synapse Dedicated SQL pool

Lecture 141 8.6.0.HandsOn-History of Delta Tables

Lecture 142 8.7.0.HandsOn-Scheduling a Job

Lecture 143 8.8.0.HandsON-Creation of EventHub namespace resource

Lecture 144 8.8.1.HandsON-Creation of EventHub

Lecture 145 8.8.2.HandsON-Installing Required Libraries

Lecture 146 8.8.3.HandsON-Starting EventHub stream inside DataBricks

Lecture 147 8.8.4.HandsON-Transforming Event Stream inside DataBricks

Lecture 148 8.9.0.HandsON-Setup to Run Databricks Notebook from ADF

Lecture 149 8.9.1.HandsON-Running a Databricks notebook from ADF Pipeline

Lecture 150 8.10.HandsOn-Delete Unused Resources

Lecture 151 Congrats on Completing a Section

Section 9: Segment9-Develop data processing-Azure Event Hubs

Lecture 152 9.1.Processing Batch Data in Azure

Lecture 153 9.2.Processing Stream Data in Azure

Lecture 154 9.2.1.Overview-Azure Event Hubs

Lecture 155 9.2.2.Concepts-Partitions in Event Hubs

Lecture 156 9.3.1.HandsOn-Create EventHub NameSpace

Lecture 157 9.3.2.HandsOn-Create EventHub inside EventHub namespace

Lecture 158 9.3.3.HandsOn-Ingesting Data to EventHub

Lecture 159 9.3.4.HandsOn-Capture Data Feature in Event Hub

Section 10: Segment10-Develop data processing-Stream Analytics

Lecture 160 10.1.1.Overview-Azure Stream Analytics

Lecture 161 10.1.2.Overview-Stream Analytics Costing

Lecture 162 10.1.3.HandsOn-Create Stream Analytics Workspace

Lecture 163 10.2.1.HandsOn-Define Input in Stream Analytics

Lecture 164 10.2.2.1.HandsOn-Create Synapse SQL table

Lecture 165 10.2.2.2.HandsOn-Create a Table inside Synapse SQL

Lecture 166 10.2.2.3.HandsOn-Set the Staging location inside Stream Analytics

Lecture 167 10.2.2.4.HandsOn-Define an Output as Synapse SQL pool in Stream Analytics

Lecture 168 10.2.2.5.HandsOn-Create a Query and Start Streaming Job

Lecture 169 10.2.3.HandsOn-Fetch Data from ADSL to synapse SQL using Stream Analytics

Lecture 170 10.2.4.1.HandsOn-Sending Logs to EventHub

Lecture 171 10.2.4.2.HandsOn-Define an Input inside Stream Analytics

Lecture 172 10.2.4.3.HandsOn-Definie an output for Stream Analytics

Lecture 173 10.2.4.4.HandsOn-Formulate a Query based on your DataStructure

Lecture 174 10.2.5.HandsOn-Monitoring Errors for Stream Analytics

Lecture 175 10.2.6.1.Overview-Builtin Window functions in Stream Analytics

Lecture 176 10.2.6.2.Overview-Tumbling and Hopping Window in Stream Analytics

Lecture 177 10.2.6.3.Overview-Sliding,Session and Snapshot Window functions

Lecture 178 10.2.6.4.HandsOn-Setup for Tumbling Window in Stream Analytics

Lecture 179 10.2.7.HandsOn-Using Reference Data as Input in Stream Analytics

Lecture 180 Congrats on Completing Section 9 and 10

Section 11: Segment11-Secure-Azure Data Lake Security Aspects

Lecture 181 11.1.0.HandsOn-Installing Azure Data Explorer Client tool

Lecture 182 11.1.1.Overview-Security Aspects of Azure Data Lake Gen2

Lecture 183 11.1.2.HandsOn-Connecting Storage Account to Data Explorer using Account KEY

Lecture 184 11.1.3.HandsOn-Connecting Storage Account Through SAS token

Lecture 185 11.2.0.Overview-Authentication and Authorization Process

Lecture 186 11.2.1.Overview-Microsoft Entra ID

Lecture 187 11.2.2.HandsON-Create Microsoft Entra ID User

Lecture 188 11.3.1.Overview-RBAC Roles

Lecture 189 11.3.2.HandsOn-Assigning RBAC to a User

Lecture 190 11.3.3.HandsOn-Login Using New Entra ID USER and Assigning RBAC roles

Lecture 191 11.4.1.Overview-Access Control List (ACLs) Permission Types

Lecture 192 11.4.2.HandsOn-Implement ACLs inside Storage Accounts

Lecture 193 11.5.0.Concepts-Securing your Storage Using Service Endpoint

Lecture 194 11.5.1.HandsOn-Creating a Virtual Machine for Virtual Network

Lecture 195 11.5.2.HandsOn-Running a Virtual Machine for Service EndPoint

Lecture 196 11.5.3.HandsOn-Setup to Access Storage Using Service Endpoint

Section 12: Segment12-Azure Synapse Analytics Security Aspects

Lecture 197 12.1.0.Overview-Azure Synapse Security Encryption Types

Lecture 198 12.1.1.Overview-Transparent Data Encryption for Synapse Dedicated SQL Pools

Lecture 199 12.1.2.HandsOn-Creating a Key Vault

Lecture 200 12.1.3.HandsOn-Configuring a Key Vault and Creating a Key

Lecture 201 12.1.4.HandsOn-Enabling a Double Encryption for Synapse Workspace

Lecture 202 12.2.1.Overview-Microsoft Entra ID for Synapse Analytics

Lecture 203 12.2.2.HandsOn-Setting Entra ID user as Admin User for Synapse Analytics

Lecture 204 12.2.3.HandsOn-Given Microsoft Entra ID user permission dedicated Sql pool

Lecture 205 12.3.1.Overview-Managed Identities on Azure

Lecture 206 12.3.2.HandsOn-Creating External Table using Managed Identities Synapse

Lecture 207 12.3.3.HandsOn-Data Discovery and Classification Synapse SQL pool

Lecture 208 12.4.1.HandsOn-Column Level Security Synapse SQL

Lecture 209 12.4.2.HandsOn-Dynamic Data Masking inside Synapse SQL

Lecture 210 12.4.3.HandsOn-Row Level Security Synapse SQL

Section 13: Segment13-DataFactory and Databricks Security Aspects

Lecture 211 13.1.1.HandsOn-Encrypting Azure Data Factory

Lecture 212 13.2.1.Overview-Encryption using secrete scope in DataBricks

Lecture 213 13.2.2.HandsOn-App Registration inside Microsoft EntraID and Generating a Secret

Lecture 214 13.2.3.HandsOn-Save the secrete inside a Key Vault

Lecture 215 13.2.4.HandsOn-Accessing Vault Using Access Policies for Databricks

Lecture 216 13.2.5.HandsOn-Creating Secrete Scope inside Databricks

Lecture 217 13.2.6.HandsOn-Implementing Scoped Credentials inside Databricks Notebook

Lecture 218 13.2.7.HandsOn-Running a Code

Lecture 219 Congrats on Completing Security Sections

Section 14: Segment14-Monitor and Optimize-Data Lake Storage

Lecture 220 14.1.1.Concepts-Data Lake Storage Practices

Lecture 221 14.1.2.Concepts-Data Lake Access Tiers

Lecture 222 14.1.3.HandsOn-Selecting Default Tier for Blobs

Lecture 223 14.1.4.HandsOn-Changing Tier at file level for blobs

Lecture 224 14.1.5.HandsOn-Life Cycle Management Rules for blobs

Section 15: Segment15-Monitor and Optimize-Azure Data Factory

Lecture 225 15.1.1.Overview-Azure Data Factory IR and Debugging

Lecture 226 15.1.2.HandsOn-Anotations in ADF

Lecture 227 15.1.3.HandsOn-Monitor Service in Azure

Lecture 228 15.1.4.HandsOn-Creating Alert for Data Factory

Lecture 229 15.1.5.HandsOn-Monitor Data Factory Logs through Log Analytics Workspace

Section 16: Segment16-Monitor and Optimize-Synapse Analytics

Lecture 230 16.1.1.Overview-DMV Commands

Lecture 231 16.1.2.0.Overview-Work Load Management in Synapse

Lecture 232 16.1.2.1.HandsOn-WorkLoad Management in dedicated SQL pool

Lecture 233 16.1.3.0.Concepts-Result Set Caching in Synapse SQL

Lecture 234 16.1.3.1.HandsOn-Turning On Result Set Caching

Lecture 235 16.1.4.HandsOn-Checking Data Skewness

Lecture 236 16.1.5.HandsOn-Monitoring of Synapse Workspace

Lecture 237 16.1.6.HandsOn-Connecting Synapse with Log Analytics

Lecture 238 16.1.7.HandsOn-Deleting Diagnostic Settings

Section 17: Segment17-Monitor and Optimize-Stream Analytics, Cosmos DB

Lecture 239 17.1.1.Overview-Streaming Units and Monitoring for Stream Analytics

Lecture 240 17.1.2.Overview-Event Partitioning and Parallel Processing in Stream Analytics

Lecture 241 17.1.3.Concepts-time handling in Azure Stream Analytics

Lecture 242 17.1.4.HandsOn-Monitor Stream Analytics Job

Lecture 243 17.2.1.HandsOn-Monitor Cosmos DB through Matrices

Lecture 244 17.2.2.HandsOn-Setup Log Analytics with Cosmos DB

Section 18: Segment18-Monitor and Optimize-Data Governance with Microsoft Purview

Lecture 245 18.1.1.Concepts-Microsoft Purview Data Governance and Management

Lecture 246 18.1.2.HandsOn-Creating Microsoft Purview Account

Lecture 247 18.1.3.HandsOn-Scan Data Lake Assets with Purview

Lecture 248 18.1.4.HandsOn-Scanning Synapse SQL pool using Purview

Lecture 249 18.1.5.HandsOn-Browsing inside Microsoft Purview

Lecture 250 18.1.6.HandsOn-Connecting DataFactory with Purview

Lecture 251 18.1.7.HandsOn-Removing Assets

Lecture 252 Congrats on Completing Monitor and Optimize Sections

Section 19: Segment19-Practice Papers-500 Questions

Lecture 253 Overview-Exam on Microsoft for Dp-203

If you want to boost your career in the field of data engineering.,Professional who will like to transform yourself into a skilled data engineer, this course is for you.,Mastering DP-203 empowers you to build analytical solutions using Microsoft Azure's data platform technologies, paving the way for a successful career in data engineering.,This course is tailored for data professionals, and BI and Data Analytics specialists aiming to deepen their knowledge of data engineering.