Data Engineering On Microsoft Azure

DP 203 - Azure Data Engineer Associate

What you'll learn

Implement a partition strategy

Design and implement the data exploration layer

Ingest and transform data

Develop a batch processing solution

Develop a stream processing solution

Manage batches and pipelines

Implement data security

Monitor data storage and data processing

Optimize and troubleshoot data storage and data processing

Requirements

Foundational Knowledge of Azure

Description

In this course, you will learn how to implement and manage data engineering workloads on Microsoft Azure, using Azure services such as Azure Synapse Analytics, Azure Data Lake Storage Gen2, Azure Stream Analytics, Azure Databricks, and others. The course focuses on common data engineering tasks such as orchestrating data transfer and transformation pipelines, working with data files in a data lake, creating and loading relational data warehouses, capturing and aggregating streams of real-time data, and tracking data assets and lineage. You can become a data professional, a data architect, or a business intelligence professional by learning about data engineering and building analytical solutions using data platform technologies that exist on Microsoft Azure. This course will give you a flavor of end-to-end processing of big data in Azure.As a candidate for this certification, you should have subject matter expertise in integrating, transforming, and consolidating data from various structured, unstructured, and streaming data systems into a suitable schema for building analytics solutions.As an Azure data engineer, you help stakeholders understand the data through exploration, and build and maintain secure and compliant data processing pipelines by using different tools and techniques. You use various Azure data services and frameworks to store and produce cleansed and enhanced datasets for analysis.

Overview

Section 1: Microsoft Azure Data Engineering Introduction

Lecture 1 Introduction

Section 2: Get started with data engineering on Azure

Lecture 2 Introduction to data engineering on Azure

Lecture 3 Introduction to Azure Data Lake Storage Gen2

Lecture 4 Introduction to Azure Synapse Analytics

Lecture 5 Lab - Explore Azure Synapse Analytics

Section 3: Build data analytics solutions using Azure Synapse Analytics serverless SQL pool

Lecture 6 Use a serverless SQL pool to query files in a data lake

Lecture 7 Use a serverless SQL pool to transform data

Lecture 8 Lab - Transform files using a serverless SQL pool

Lecture 9 Create a lake database

Section 4: Perform data engineering with Azure Synapse Apache Spark Pools

Lecture 10 Analyze data with Apache Spark in Azure Synapse Analytics

Lecture 11 Transform data with Apache Spark in Azure Synapse Analytics

Lecture 12 Lab - Transform data using Spark in Synapse Analytics

Lecture 13 Use Delta Lake in Azure Synapse Analytics

Lecture 14 Lab - Use Delta Lake with Spark in Azure Synapse Analytics

Section 5: Work with data warehouses using Azure Synapse Analytics

Lecture 15 Analyze data in a relational data warehouse

Lecture 16 Load data into a relational data warehouse

Lecture 17 Lab - Load Data into a Relational Data Warehouse

Section 6: Transfer and transform data with Azure Synapse Analytics Pipelines

Lecture 18 Build a data pipeline in Azure Synapse Analytics

Lecture 19 Lab - Build a data pipeline in Azure Synapse Analytics

Lecture 20 Use Spark Notebooks in an Azure Synapse Pipeline

Lecture 21 Lab - Use an Apache Spark notebook in a pipeline

Section 7: Hybrid transactional and analytical processing Solutions using Synapse Analytics

Lecture 22 Plan hybrid transactional and analytical processing

Lecture 23 Implement Azure Synapse Link with Azure Cosmos DB

Lecture 24 Lab - Use Azure Synapse Link for Azure Cosmos DB

Lecture 25 Implement Azure Synapse Link for SQL

Section 8: Implement a data streaming solution with Azure Stream Analytics

Lecture 26 Get started with Azure Stream Analytics

Lecture 27 Streaming data using Azure Stream Analytics and Azure Synapse Analytics

Lecture 28 Lab - Ingest data with Azure Stream Analytics and Azure Synapse Analytics

Lecture 29 Visualize real-time data with Azure Stream Analytics and Power BI

Section 9: Govern data across an enterprise

Lecture 30 Introduction to Microsoft Purview

Lecture 31 Integrate Microsoft Purview and Azure Synapse Analytics

Lecture 32 Lab - Use Microsoft Purview with Azure Synapse Analytics

Section 10: Data engineering with Azure Databricks

Lecture 33 Explore Azure Databricks

Lecture 34 Use Apache Spark in Azure Databricks

Lecture 35 Lab - Use Spark in Azure Databricks

Lecture 36 Run Azure Databricks notebooks in Azure Data Factory

Section 11: Conclusion

Lecture 37 Redemption of Badges

Data Engineers: Professionals who focus on preparing "big data" for analytical or operational uses. These individuals are responsible for designing, building, and maintaining the architecture (such as databases and large-scale processing systems) for data ingestion, processing, and analytics.,Data Architects: These are professionals who design the blueprint for managing data across the organization. They work on designing data solutions that utilize Azure services effectively to meet both the technical and business requirements.,Data Professionals: This broad category includes anyone working with data in a technical capacity and looking to leverage Azure's data services for their data solutions. This could include database administrators, data analysts, and software developers with a focus on data.,IT Professionals: IT professionals who are not necessarily data specialists but are looking to expand their skills into the data engineering space can benefit from this course. This includes system administrators, software developers, and IT managers who need to understand how data solutions are designed and implemented on Azure.,Students and Recent Graduates: Those who are studying in fields related to computer science, data science, information technology, or similar areas and are looking to enter the workforce with a strong set of skills in cloud-based data solutions.