Essential Sql: Azure Data Factory And Data Engineering

Data Engineering using ADF | Azure Data Engineer | Data Mapping Flows | SQL Stored Procedure | Data Engineering

What you'll learn

Learn how to construct ELT solutions using Azure Data Factory and SQL Server.

Implement various ELT solutions using Mapping Data Flows, Copy Activities, and Stored Procedures.

Develop the skills to build and troubleshoot Azure Data Factory pipelines with confidence.

Enhance pipeline reliability by mastering the use of parameters to optimize your data workflows.

Learn efficient methods for storing and structuring data using Azure Storage services.

Understand the distinctive roles played by Azure Data Factory, Data Storage, SQL Server, and Data Bricks in the creation of an ELT solution.

Requirements

Familiarity with SQL and basic programming skills can be beneficial.

Proficiency in programming will be valuable when working with scripting and data transformation tasks

Experience in troubleshooting and debugging technical issues will aid in building and troubleshooting Azure Data Factory pipelines.

Description

Use Azure data factory to automate data engineering tasks. This is a great course to introduce yourself to Azure Data Factory’s capabilities. We’ll look at the data factory from a data engineer perspective.Through examples we’ll work side by side to create an Extract Transform and Load process to ingest movie rating data.We are going to explore three different ways to transform your data.The first is by using Azure Data Mapping flows. These are great no-code ways to do ETL.We’ll then look at how you can do the same transformations using Python. This way, if you love Python, you know a solid way to use Azure data factory to work with your data.Lastly, we’ll look at how you can create pipelines to use your knowledge of SQL and stored procedures.What you’ll like about this course is that once you learn one way to transform the data, you can use that knowledge to learn about the other methods. So, if you’re a SQL expert but soft on python, you can learn the SQL way first, then go back and try it using Python. In the end you come out learning and appreciating alternative methods to ingesting, transforming, and storing data.

Overview

Section 1: Introduction

Lecture 1 Welcome to the Course

Lecture 2 Who Should Take Azure Data Factory?

Lecture 3 Important! What to Expect from the Course.

Lecture 4 How to Take the Course

Section 2: Set Up Azure

Lecture 5 Setup Azure Account

Lecture 6 Setup Azure Resource Group

Lecture 7 Setup Azure Storage Account

Lecture 8 Create Microsoft Data Factory

Lecture 9 Test Data Factory to Storage Link

Lecture 10 Setup Azure Database

Lecture 11 Set up Azure Data Studio

Lecture 12 Test Azure Database

Lecture 13 Setup Microsoft DevOps

Section 3: Case Study - Using Data Mapping Flows to Refine Data

Lecture 14 Medallion Architecture and Bronze Layer Introduction

Lecture 15 Bronze Layer - Setup Data Lake Folders to support the Medallion Architecture

Lecture 16 Bronze Layer - Prep data using Data Mapping Flows.

Lecture 17 Bronze Layer - Execute Mapping Data Flow using ADF Pipeline.

Lecture 18 Silver Layer - Introduction

Lecture 19 Silver Layer - Building Mapping Flow Part 1 - Data Types

Lecture 20 Silver Layer - Building Mapping Flow Part 2 - Filter and Deduplication

Lecture 21 Silver Layer - Building Mapping Flow Part 3 - Parsing Addresses

Lecture 22 Silver Layer - Building Mapping Flow Part 4 - Touchups and Final Check

Lecture 23 Gold Layer - Introduction

Lecture 24 Gold Layer - Build Dimensions and Fact table using Mapping Flow

Lecture 25 Gold Layer - Automate Dimensions and Fact table Builds using ADF

Lecture 26 Gold Layer - Refactor ADF Pipeline and then See Results in PowerBI

Section 4: Case Study - SQL and the Azure Data Factory Copy Activity to Refine Data

Lecture 27 Medallion Architecture and Bronze Layer Introduction

Lecture 28 Bronze Layer - Using ADF Copy Activity

Lecture 29 Silver Layer - Introduction

Lecture 30 Silver Layer - Land Data into SQL

Lecture 31 Silver Layer - Create Address Split Query

Lecture 32 Silver Layer - Create Stored Procedure Transform Loaded Data

Lecture 33 Silver Layer - Execute Stored Proc from Data Factory Pipeline

Lecture 34 Gold Layer - Introduction

Lecture 35 Gold Layer - Dimension Building

Lecture 36 Gold Layer - Build the Sales Order Fact

Lecture 37 Gold Layer - Move Table to Gold (Production)

Lecture 38 Gold Layer - Review Start Schema in PowerBI

Section 5: ADF Pipeline and Activities Reference

Lecture 39 Azure Data Factory - Introduction

Lecture 40 Azure Data Factory - Variables and Parameters

Lecture 41 Azure Data Factory - Parameters and Return Values

Lecture 42 Azure Data Factory - If Activity

Lecture 43 Azure Data Factory - Switch Activity

Lecture 44 Azure Data Factory - For Each Activity

Lecture 45 Azure Data Factory - Get Meta Data and IF

Lecture 46 Azure Data Factory - Filter Activity

Section 6: Resources

Lecture 47 Product Database

Aspiring Data Engineers: Individuals who are new to data engineering and want to learn how to use Azure Data Factory to build data solutions.,Business Analysts: Analysts who work with data to make business decisions. Understanding data engineering can help them better utilize and secure data for decision-making.,Business Intelligence Developers: BI developers seeking to automate data pipelines for their reporting and analytics needs using Azure Data Factory.,Cloud Architects: Professionals responsible for designing and implementing cloud solutions. They may want to understand how Azure Data Factory fits into their architecture.,Data Analysts: Analysts who want to expand their skills by learning how to move and transform data using Azure Data Factory for better analysis.,Data Architects: Professionals responsible for designing data architectures and systems. They need to ensure that the data solutions they design are secure, scalable, and well-managed.,Data Scientists: Data scientists who want to work with clean and well-organized data, making Azure Data Factory knowledge beneficial for data preparation.,Database Administrators: DBAs looking to integrate Azure Data Factory into their data management processes, especially for data movement and ETL (Extract, Transform, Load) tasks.,Software Developers: Developers who want to gain expertise in building data solutions and working with databases. They might be interested in integrating data management and security practices into their software development workflows.,System Administrators: IT administrators responsible for managing Azure accounts and resources may need to understand Azure Data Factory for resource allocation and monitoring.