Transform Data Into Insights With Dagster And Deepnote

Data Engineering for Empowered Business Decisions: ETL, Exploration & Visualization

What you'll learn

Turn messy, real-world data into actionable insights.

Gain familiarity with tools such as Deepnote, Dagster, and Metabase.

Use Deepnote as a data engineering development environment.

Generate realistic development data for analysis and visualization.

Learn data exploration and preprocessing techniques using Python and SQL.

Clean and normalize data from various sources, such as relational databases, JSON, .xls files and more.

Set up Dagster to orchestrate your data pipeline.

Integrate the processing logic into a scalable ETL pipeline with Dagster.

Deploy your pipeline to Dagster Cloud (serverless)

Optimize processing through techniques such as parallelization or streamed processing.

Create powerful data visualizations using Metabase.

Requirements

Basic Python Knowledge

Description

Do you struggle with making data-driven decisions for your business due to scattered, inconsistent, and inaccessible data? This course is the solution! Learn to build a streamlined and efficient ETL pipeline that will allow you to turn data into actionable insights.This course teaches you how to build a system that collects data from multiple sources, normalizes it, and stores it in a consistent and accessible format. You will learn how to extract data, explore and preprocess it, and ultimately visualize it to support better decision-making and optimize business processes.Forget about big data and cluster management headaches, this course is designed to get you up and running quickly with a real-time ETL pipeline. With infrastructure costs under $50 a month, you can start seeing immediate results and return on investment for your clients or company.In the first part of the course, I will walk you through the architecture and introduce you to the tools we will be using:Deepnote, as a setup-free development environmentDagster, as the pipeline orchestratorMetabase, as a low-code data visualization platformWhile the course will introduce you to the relevant features of Deepnote and Metabase, it is mostly focused on Dagster.In the next part, we will get started by generating dummy sales data of a hypothetical company using Deepnote. The code will be provided for this. Once we have the data, the course will dive into data exploration and preprocessing techniques using Python and SQL in Deepnote, including cleaning and normalizing data from various sources such as relational and JSON data, Excel sheets, and more. We will implement the processing logic in Deepnote, then commit it to a Git repository that will be shared with Dagster.In the following section, we will wrap the business logic with Dagster operations and jobs, then deploy them to Dagster Cloud (self-hosted option also available), which will allow you to manage everything from a single, unified view. In this section, you will also learn a few tricks to speed up and optimize processing, such as parallelization or streamed processing.In the final section of this course, you'll bring your preprocessed data to life with Metabase. With a few simple clicks, even non-technical individuals will be able to create stunning, powerful visualizations that unlock the full potential of your data.By the end of this course, you'll have a comprehensive understanding of the tools used and how they work together, empowering you to provide tangible benefits to your clients or company from day one, measured in thousands or tens of thousands of dollars.The choice is yours - will you seize this opportunity to deliver massive benefits to your company or clients, and claim your fair share of the rewards?

Overview

Section 1: Introduction

Lecture 1 Welcome to the World of Data Engineering

Lecture 2 The Power of Clean, Organized Data

Lecture 3 The Skills and Tools Needed to be a Successful Data Engineer

Lecture 4 An ETL pipeline for Small and Medium-Sized Businesses

Section 2: Exploring the Tools of the ETL pipeline: Deepnote, Dagster, and Metabase

Lecture 5 DeepNote

Lecture 6 Dagster

Lecture 7 Metabase

Lecture 8 Other tools

Section 3: Designing the ETL Pipeline: From Data Sources to Dashboards

Lecture 9 Building the Solution Architecture

Section 4: Setting Up Your Development Environment and Generating Dummy Data

Lecture 10 Creating a PostgreSQL Database on Google Cloud

Lecture 11 Generating Synthetic Data of a Hypothetical Client

Lecture 12 Explanation of the data generation process (optional)

Lecture 13 Verifying the Generated Data

Section 5: Getting Started with Deepnote: An Introduction to Python and SQL for Data Explor

Lecture 14 Extracting and Viewing Data in Deepnote

Lecture 15 Digging Deeper: Identifying Data Issues

Lecture 16 Digging Deeper: Coming Up with a Strategy

Lecture 17 Creating a Database Table for Storing Normalized Data

Section 6: Data Preprocessing in Deepnote: Cleaning and Normalizing Data

Lecture 18 Preprocessing Relational Data: POS Transactions

Lecture 19 Preprocessing Relation Data: Crypto Transactions

Lecture 20 Preprocessing JSON Data

Lecture 21 Preprocessing Excel Sheets: Loading Files from Google Drive

Lecture 22 Preprocessing Excel Sheets: Market Transactions

Lecture 23 Refactoring Business Logic: Challenge

Lecture 24 Refactoring Business Logic: Solution

Lecture 25 Unit Testing

Section 7: Setting up the ETL pipeline with Dagster

Lecture 26 Overview of Dagster Concepts

Lecture 27 Set up Local Dagster Development

Lecture 28 Extracting Data

Lecture 29 Transforming and Loading Data

Lecture 30 Partitioned Processing

Lecture 31 Job Configuration

Lecture 32 Streamed Data Processing

Lecture 33 Processing Files

Lecture 34 Creating Dagster Schedules

Lecture 35 Creating Dagster Sensors

Lecture 36 Deploying to Dagster Cloud

Section 8: Visualizing Data in Metabase

Lecture 37 Creating Visualizations from the Processed Data

Section 9: Bonus Content

Lecture 38 Bonus Lecture

Developers seeking to build scalable and efficient ETL pipelines.,Entrepreneurs looking to leverage data for business growth.,Data analysts and scientists who want to streamline their data processing workflow.,Business professionals looking to improve their data-driven decision-making abilities.,Students and recent graduates interested in a career in data engineering.,Data managers tasked with organizing and making data accessible for analysis.,Project managers looking to implement data-driven solutions for clients or company.,Individuals interested in learning cutting-edge tools and techniques in data engineering.