2023 Databricks Etl Data Engineer From Beginner To Expert

Hot Open Banking ETL Project on The Current Market Inside Out

What you'll learn

Get the similar expertise with the front-line data engineers who earn $200, 000+ yearly

Get the capability of resloving complex ETL issues like stablity or performance issues, as an expert

Enjoy more free time saved by the ready-to-use automation frameworks and tools

Harden your irreplaceability with advanced ETL capablity and test strategy

Get real open banking ETL project expereince from scracth

Maximize your opportunities of switching to data engineer

Requirements

Basic level of Programming languages, like Python and SQL

Basic level of SQL and NoSQL databases

Basic concepts for Cloud, Data Warehouse, testing

Description

Are you thinking of getting higher income for a long time with a short training period? Then, you come to the right place. As you know AI and big data are the driving forces behind Industry 4.0. Data engineer is high-paid role. As the shortage of data engineers continues and the demand for skilled data engineer is soaring. Switching to data engineer is one of the best options for your career, which can get higher pay, long-term contracts and less pressure. By joining this course, you will know how the front-line data engineers execute ETL with real open banking project from the scratch, get the ready-to-use, practical and powerful ETL framework and the ISTQB testing strategy.  So, it will be highly possible to pass your job interview because you really have the similar capabilities with data engineers who earn $200, 000+ yearly and be able to resolve complex ETL issues on your own after this course.If you are already a data engineer, this course will sharpen your ETL skills. This course can free yourself via saving huge amount of computing time and cost during ETL process. More importantly, you will get the capabilities of resolving complex issues in terms of performance, stability and reliability. You will enjoy more free time and leave the hard work to the ETL framework and scripts.With 6 sections and about 5 hours of high-quality content, you will learn the following topics in depth:Section #1: Values We Can Bring to YouYou will understand what valuable capabilities you can take away after this course and extra free live resources you can keep learning.Section #2: Azure Databricks and Cosmos DB SetupYou are able to set up Azure Databricks, Cosmos DB and connect them on your own. Section #3:  ETL Project InitializationGet real commercial open banking project experience from scratch including grabbing business requirement, Coverting it to technical requirement, designing the solution and implementing it.Section #4: ETL and Testing Business Goal and Test StrategyIdentify business goals we are going to achieve and adopt right test strategies and plan static testing and shift-left testing approach.Section #5: Data Extraction and TransformationImplement data extraction and transformation functionalities and create a smart mechanism to detect if data is modified with lower cost in terms of time and efficiency.Section #6:  Stable and Speedy Data IngestionResolve complex ETL issues in terms of performance, stability and reliability, etc. as an expert and get the powerful framework and scripts and customize them and apply to your own projects. Last but not least:Enjoy this course and don't forget to check out my other courses if you want to learn how to test the project with ISTQB recognized automation testing framework:2023 Databricks ETL Testing Services Complete Guide

Overview

Section 1: Values We Can Bring To You

Lecture 1 What You Can Get From This Course

Lecture 2 ETL And Testing Process

Lecture 3 Tool Set For ETL

Lecture 4 More Resources For Learning Tools

Lecture 5 Prerequisite For ETL

Section 2: Azure databricks and Cosmos DB setup

Lecture 6 Azure Databricks Setup

Lecture 7 Azure Cosmos DB Setup

Lecture 8 Connect Cosmos DB From Databricks

Lecture 9 Insert Data Into Cosmos DB

Section 3: ETL Project Initialization

Lecture 10 Go Through Business Requirement

Lecture 11 Analyze Legacy Data Tables

Lecture 12 Identify Technical Requirement

Lecture 13 High Level Design

Lecture 14 Load Sample Data Into Delta Tables

Section 4: ETL and Testing Business Goal and Test Strategy

Lecture 15 Defined Test Strategy

Lecture 16 Static Testing

Lecture 17 Plan Shift Left Automation Test Approach

Section 5: Data Extraction and Transformation

Lecture 0 Extract Data From Source System

Lecture 18 Add Prefix For Table Dataframe

Lecture 19 Transform Data Part 1

Lecture 20 Transform Data Part 2

Lecture 21 Add Validation Hash

Lecture 22 Sort Records

Lecture 23 Refactor Code

Section 6: Resolve complex issues as an expert during data ingestion

Lecture 24 Get New Records To Be Loaded

Lecture 25 Get Modified Records

Lecture 26 Get Records To Be Deleted

Lecture 27 Resolve Stability Issue

Lecture 28 Resolve Performance Issue

Lecture 29 Understand Table Paths

Lecture 30 Delete Sample Data

Lecture 31 Ingest Data To Cosmos

Lecture 32 Dynamically Update Cosmos Clone Table

Lecture 33 Enhance Reliablity

Graduates who want to get a job as data engineer or data tester,Data engineers who wan to sharpen their skills,Test Analysts or developers who want to switch to data engineering