End To End Azure Data Engineering Course

Data Engineering

What you'll learn

Migrate on Premises SQL Server database to Azure using Data Factory

Use Databricks to transform data through the Medallion Architecture (Bronze, Silver, Gold)

Build a Data Warehouse using Synapse Analytics

Use PowerBI to Create Actionable Insights

Test End to End Pipeline ingesting new data

Requirements

Some knowledge of SQL Server would be helpful, however this course aims to show you how to install, setup and configure all the necessary tools required for this course.

Description

This comprehensive course guides you step-by-step through an entire end-to-end Azure data engineering project, providing you with hands-on experience in designing, building, and optimizing data pipelines and reporting solutions in the cloud. By the end of this course, you’ll have developed practical, real-world skills in Azure, empowering you to confidently build and manage data workflows, from ingestion to insightful visualization. Here’s what you’ll explore:1. Azure Data Factory - Learn how to use this versatile tool to orchestrate data flows and automate data movement. You’ll set up pipelines to securely transfer data from an on-premises SQL Server database to the Azure cloud.2. Azure Data Lake Storage Gen2 - Dive into cloud storage and data organization, leveraging the scalable, high-performance capabilities of Data Lake Storage to store and manage raw data. You’ll organize data layers following the Medallion architecture for efficient access and transformation.3. Azure Databricks - With Databricks, you’ll process and transform your data using Apache Spark, preparing it for analysis through cleansing, enrichment, and aggregation. In this course, you’ll see how to transition data across the Bronze (raw), Silver (cleansed), and Gold (curated) layers, applying transformations in Python for efficient, scalable data processing.4. Azure Synapse Analytics - Explore the powerful capabilities of Synapse Analytics to build a logical data warehouse. You’ll create tables, define relationships, and structure data, setting the stage for efficient querying and reporting.5. Azure Key Vault - Learn how to use Key Vault to securely store and manage secrets, such as database connection strings and API keys, ensuring secure access throughout your pipelines.6. Identity Management - Discover best practices for managing identities and permissions across Azure resources, including roles, policies, and access controls that provide secure, streamlined access to your data.7. Power BI - Once your data is fully prepared, Power BI enables you to analyze and visualize data for actionable insights. You’ll build an interactive Power BI report, creating charts, dashboards, and visuals that bring your data to life and enable stakeholders to make informed decisions.Project Outline:1. Data Migration to Azure    Begin by migrating an on-premises SQL Server database to the cloud. You’ll configure Azure Data Factory to securely connect to your on-premises environment and schedule data extractions, establishing a robust data pipeline.2. Data Storage and Medallion Architecture    You’ll store the data in Azure Data Lake Gen2, organizing it according to the Medallion architecture. This three-layer design (Bronze, Silver, Gold) will help you manage data through different stages, from raw ingestion to cleaned and curated datasets ready for analysis.3. Data Transformation with Azure Databricks    Use Azure Databricks to transform data as it moves through each layer. Using Python, you’ll apply data-cleaning techniques, remove duplicates, standardize formats, and aggregate data to prepare for downstream analysis in Synapse Analytics and Power BI.4. Building a Data Warehouse with Synapse Analytics    With your curated data in the Gold layer, you’ll load it into Azure Synapse Analytics to design a logical data warehouse. This layer of structured, high-performance storage enables efficient querying and sets the foundation for your Power BI reports.5. Data Security with Azure Key Vault and Identity Management   Incorporate Azure Key Vault to manage sensitive information securely, and apply identity management principles to enforce access control and protect your data assets.6. Data Visualization with Power BI     Finally, you’ll create a Power BI report using the structured data from Synapse Analytics. You’ll develop visuals and dashboards that reveal insights, making your data useful and accessible to stakeholders through interactive, shareable reports.Learning Outcomes:By completing this course, you will gain experience with an entire Azure data engineering pipeline and acquire skills in:- Configuring and managing data pipelines with Azure Data Factory.- Storing and organizing data in Azure Data Lake Gen2 using Medallion architecture.- Performing data transformation using Azure Databricks with Python.- Creating a logical data warehouse in Azure Synapse Analytics.- Implementing security and identity management practices in Azure.- Designing and developing Power BI reports for data visualization and insights.This course equips you with the knowledge and hands-on skills to build efficient, scalable data pipelines on Azure and leverage cloud-based analytics for data-driven decision-making in real-world projects.Perquisites :Accounts:To begin this course, it would be helpful to have an Azure account setup.  ExperienceSome SQL Server knowledge would be helpful but it is not a requirement as this course aims to take you through step by step including coding.Software.The following software is required to complete this course, but don't worry if you don't have it installed, this course will guide you through that.SQL Server 2022SQL Sever Management StudioMicrosoft Open JDKPower BI

Overview

Section 1: Azure End to End Data Engineering

Lecture 1 Tools Installation, Setup and First Pipeline

Lecture 2 Data Ingestion and Transformation

Lecture 3 Pipeline Bug Fixes

Lecture 4 End to End Pipeline Testing and Power BI Build

Lecture 5 Summary

Data Engineers