Master Azure Data Factory And Azure Sql

Build a Foundation for Azure Data Engineering Career

What you'll learn

Azure Fundamentals For Data Engineering viz. Azure Active Directory, Key Vault, Storage, Data Lake

Programming with Azure SQLDB Using T-SQL

Azure Data Factory Pipelines, Datasets, Triggers, Global Variables, Linked Services

Azure Active Directory Users, Groups, AD Roles, RBAC Roles

Implement AAD Custom Roles Similar To Real World Projects

Dynamic Pipeline Execution

Configure AAD Users & Groups Access in Azure SQLDB

Build Metadata Driven ETL Orchestration Framework Using Azure SQLDB And Data Factory

Requirements

No Azure background is required. You will learn everything you need to know.

No Programming background is required. T-SQL Programming is covered including Stored Procs, Functions, Cursors, SQL Dynamic Execution, Transactions.

Basics of Azure SQLDB is covered.

Azure Data Factory setup is covered, no prior knowledge is necessary.

Azure Data Factory is covered from scratch.

An end to end ETL Metadata driven Orchestration framework is designed & developed so that learners can get insights in ETL Design & Approach.

Description

We have launched an in-depth technical course for learning Azure Data Factory (ADF) & Azure SQL. There are several other online courses that cover Azure Data Factory. So what is different in this course and why you should consider enrolling in this course?Here are the key differentiators.Unlike other courses, we recognize that ADF cannot be learnt in a silo. One must know Azure Fundamentals, a few key Azure Services and in many instances, knowledge of Azure SQLDB & T-SQL is also necessary. At a minimum, one must know, Azure Active Directory, Azure Storage & Azure Key Vault. Knowledge of these 3 Azure Fundamental Services is required by all Azure Data professionals.The focus of this course is to provide a well rounded & structured step by step training in all the areas that we have mentioned above. The learnings & knowledge gained in this course can be considered as a foundational step in starting your journey towards becoming an Azure Data Engineer. After covering the technical knowledge areas, we go on to develop a metadata driven ETL Orchestration Framework. The aim is to help you build architectural & design skills so that you can grow in your career as an Azure Data Architect or Azure Solution Architect.  We welcome you to this course & hope that you gain useful skills to solve complex problems. Our best wishes on your learning journey!

Overview

Section 1: Azure Fundamentals

Lecture 1 Introduction To The Course & Important Note On Volume Control

Lecture 2 Evolution of Data Processing Applications

Lecture 3 Azure Fundamental Concepts

Lecture 4 Sign-up For Azure Free Account

Lecture 5 Upgrade to a Pay-As-You-Go Account

Lecture 6 Azure Portal Overview

Lecture 7 Setting A Budget

Lecture 8 Introduction To Azure Active Directory & Users

Lecture 9 Azure Active Directory Roles (AD Roles, RBAC Roles & Custom Roles)

Lecture 10 Azure Key Vault

Lecture 11 Lab - Azure Active Directory & Key Vault (Part 1)

Lecture 12 Lab - Azure Active Directory & Key Vault (Part 2)

Lecture 13 Azure Storage Accounts

Lecture 14 Lab - Azure Storage Accounts (Part 1)

Lecture 15 Lab - Azure Storage Accounts (Part 2)

Lecture 16 Azure Data Lake

Section 2: Azure SQL Database & T-SQL

Lecture 17 Introduction To Azure SQL Offerings (Part 1)

Lecture 18 Introduction To Azure SQL Offerings (Part 2)

Lecture 19 Lab - Provision Azure SQL Database

Lecture 20 Lab - Install SQL Server Management Studio

Lecture 21 Azure SQL Database Schema/Tables/Logins/Users/Roles/Permissions

Lecture 22 Lab - Azure SQL Database Logins/Users/Permissions

Lecture 23 Lab Script - Azure SQL Database Logins/Users/Permissions

Lecture 24 Azure SQL Database Tables

Lecture 25 Lab - Azure SQL Database Tables

Lecture 26 Lab Script - Azure SQL Database Tables

Lecture 27 Azure SQL Database Indexes With Lab

Lecture 28 Lab Script - Azure SQL Database Indexes

Lecture 29 Azure SQL Database Table Constraints With Lab

Lecture 30 Lab Script - Azure SQL Database Table Constraints

Lecture 31 Azure SQL Database SELECT With Lab

Lecture 32 Lab Script - Azure SQL Database SELECT

Lecture 33 Azure SQL Database Table Joins With Lab

Lecture 34 Lab Script - Azure SQL Database Table Joins

Lecture 35 Azure SQL Database Views With Lab

Lecture 36 Lab Script - Azure SQL Database Views

Lecture 37 Introduction To Azure T-SQL With Lab (Part 1)

Lecture 38 Introduction To Azure T-SQL With Lab (Part 2)

Lecture 39 Lab Script - Introduction To Azure T-SQL

Lecture 40 Azure SQL Cursors With Lab

Lecture 41 Lab Script - Azure SQL Cursors

Lecture 42 Azure SQL Transactions With Lab

Lecture 43 Lab Script - Azure SQL Transactions

Lecture 44 Azure SQL Database Stored Procedures With Lab

Lecture 45 Lab Script - Azure SQL Database Stored Procedures

Lecture 46 Azure SQL Database User Defined Functions With Lab

Lecture 47 Lab Script - Azure SQL Database User Defined Functions

Section 3: Azure Data Factory

Lecture 48 ADF Overview

Lecture 49 ADF Provisioning

Lecture 50 Linked Services

Lecture 51 First Pipeline - Lookup / Set Variable / Datasets

Lecture 52 Foreach Activity - Processing Items In A Loop

Lecture 53 Using Stored Procedures With Lookup Activity

Lecture 54 Read File/Folder Properties Using Get Metadata Activity

Lecture 55 Validation Activity Vs Get Metadata Activity

Lecture 56 Conditional Execution Using IF Activity

Lecture 57 Design & Build First Metadata Driven ETL Framework (Part 1)

Lecture 58 Design & Build First Metadata Driven ETL Framework (Part 2)

Lecture 59 Using Wait Activity As A Timer

Lecture 60 Using Fail Activity To Raise Exceptions

Lecture 61 Using Append Activity With Array Variables

Lecture 62 Using Filter Activity To Selectively Process Files

Lecture 63 Using Delete Activity To Cleanup Files After Processing

Lecture 64 Copy A Single JSON File

Lecture 65 Copy A Single TEXT File

Lecture 66 Copy A Single PARQUET File

Lecture 67 Copy All Files In A Folder

Lecture 68 Binary File Copy & Implementing File Move

Lecture 69 File To Table Copy Using Insert & Upsert Techniques

Lecture 70 File To Table Copy Using Stored Procedure

Lecture 71 File To Table Copy - Large File Issue

Lecture 72 Table To File Copy

Lecture 73 Table To Table Copy

Lecture 74 Master-Child Pattern Using Execute Pipeline Activity

Lecture 75 Implement WHEN-THEN Logic Using Switch Activity

Lecture 76 Iterative Execution Using Until Activity

Lecture 77 Consume REST API Using Web Activity As Well As Copy Activity

Lecture 78 Persist REST API Consumed Data To a File In Azure Storage

Lecture 79 Provision Databricks & Use Databricks Notebook Activity

Lecture 80 Using Self Hosted Integration Runtime To Ingest On-Prem Data Into Azure

Lecture 81 Parameterized Linked Service & ADF Global Parameters

Lecture 82 Automated Execution Of Pipeline Using Scheduled Triggers

Lecture 83 Event-Based Execution Of Pipeline Using Event Triggers

Lecture 84 ADF Limitation - No Iterative Activity Within Foreach Activity

Lecture 85 ADF Limitation - No Iterative Activity Within IF Activity

Lecture 86 ADF Limitation - No Iterative Activity Within SWITCH Activity

Lecture 87 Sequential Vs Parallel Batch In A Foreach Activity

Lecture 88 ADF Limitation - Dynamic Execution Of Pipeline Not Supported In Execute Pipeline

Lecture 89 ADF Limitation - Record Number & Data Size Restrictions In Lookup Activity

Lecture 90 ADF Limitation - Dynamic Variable Name In Set Variable Activity Not Supported

Lecture 91 ADF Overall Limitations

Section 4: Project - Metadata Driven ETL Framework

Lecture 92 Set Up Azure Active Directory Users/Groups & Key Vault

Lecture 93 Set Up Azure Storage

Lecture 94 Set Up Azure SQL Database

Lecture 95 Set Up Additional Groups & Users

Lecture 96 ETL Framework Metadata Tables

Lecture 97 Lab Script - ETL Framework Metadata Tables

Lecture 98 Set Up Azure Data Factory

Lecture 99 Modular & Reusable Design

Lecture 100 Generic Pipeline to Extract From SQL Database (Part 1)

Lecture 101 Generic Pipeline to Extract From SQL Database (Part 2)

Lecture 102 Generic Pipeline to Extract From SQL Database (Part 3)

Lecture 103 Generic Pipeline to Extract From SQL Database (Part 4)

Lecture 104 Lab Script - Metadata Table Configurations

Lecture 105 Lab Script - sp_task_parameters

Lecture 106 Lab Script - sp_job_task_log

Lecture 107 Lab Script - sp_update_job_control

Lecture 108 Lab Script - sp_get_last_run_date

Lecture 109 Lab Script - sp_update_last_run_date

Lecture 110 Generic Pipeline to Load Extracted Data Into Staging Table

Lecture 111 Lab Script - Metadata Table Configurations

Lecture 112 Lab Script - Database Configurations For Bulk Loads

Lecture 113 Lab Script - Customer_Target_Stg Table

Lecture 114 Lab Script - sp_load_stg

Lecture 115 Generic Pipeline to Load Data From Staging Table Into Target Table

Lecture 116 Lab Script - Metadata Table Configurations

Lecture 117 Lab Script - Customer_Target Table

Lecture 118 Lab Script - sp_load_customer

Lecture 119 Job Orchestration Pipeline (Part 1)

Lecture 120 Job Orchestration Pipeline (Part 2)

Lecture 121 Lab Script - Environment Parameters

Lecture 122 Lab Script - sp_get_tasks

Lecture 123 Lab Script - sp_init_job_control

Lecture 124 Lab Script - sp_job_status_update

Lecture 125 Lab Script - sp_task_status

Developers who want to learn Azure Data Factory and Azure SQLDB with no prior knowledge of Azure.,Azure Data Engineers who want to advance their knowledge and learn to design & build professional Azure ETL pipelines.,Azure Data Factory developers who want to learn Azure SQLDB and use it extensively in Data Factory projects