Data Engineering With Informatica, Snowflake And Streamlit

From source, through IICS, to Snowflake tables, dashboards and Streamlit apps in a single stack

What you'll learn

Setup Informatica Cloud, Snowflake and AWS Accounts

Design IICS Taskflows and assets

Ingest data from S3 to Snowflake with CDI

Build Snowflake Dashboards

Build Streamlit apps inside of Snowflake

Requirements

Basic knowledge on Informatica Cloud Data Integration (CDI)

Proficient knowledge on SQL and basic knowledge on Snowflake database

Basic knowledge on data modeling and engineering

Basic Python knowledge

Description

Unlock the power of data integration and transformation with this comprehensive course designed to elevate your skills in handling data and leveraging Snowflake tables through Informatica Intelligent Cloud Services (IICS).Throughout this course, you'll embark on a transformative journey, starting from the basics of extracting, transforming, and loading data (ETL) sourced from CSV files into Snowflake tables using IICS Data Integration. Delve deep into understanding the intricacies of data movement and manipulation, gaining proficiency in streamlining processes for seamless integration.As the course progresses, you'll transition to exploring Snowflake's Dashboard capability, unlocking new avenues for data validation and analysis. Dive into creating visually appealing and insightful dashboards that provide a comprehensive overview of your data landscape, empowering you to make informed decisions with confidence.But the learning doesn't stop there. We'll take it a step further by introducing you to Streamlit, a powerful tool for building interactive data applications with Python. Discover how to harness the capabilities of Streamlit to develop custom data apps that not only enhance the ETL process but also offer dynamic functionalities that can potentially replace certain steps altogether.By the end of this course, you'll emerge equipped with a robust skill set in data integration, validation, and application development, ready to tackle real-world data challenges head-on. Whether you're a seasoned data professional or just starting your journey in the world of data engineering, this course provides invaluable insights and practical knowledge to propel your career forward.

Overview

Section 1: Introduction

Lecture 1 Introduction to the course

Section 2: Snowflake Setup

Lecture 2 Creating a Snowflake account

Lecture 3 Loging into Snowflake

Lecture 4 Creating Databases and Schemas

Lecture 5 Creating course tables - 1

Lecture 6 Creating course tables - 2

Lecture 7 Creating course tables - 3

Lecture 8 Creating course tables - 4

Section 3: IICS Setup

Lecture 9 Creating an IICS Data Integration trial account

Lecture 10 Installing the local secure agent

Lecture 11 Creating a flat file connection

Lecture 12 Creating a Snowflake connection

Lecture 13 Setting up a parameter file

Section 4: SOS - Changing objects permission/ownership on Snowflake

Lecture 14 Updating objects ownership on Snowflake

Section 5: IICS Data Ingestion

Lecture 15 Source files overview

Lecture 16 STG Product Mapping

Lecture 17 Issue on using mapping task instead of mass ingestion for loading files

Lecture 18 STG Touchpoint Mapping

Lecture 19 PSA Calendar

Lecture 20 STG Media Spend

Lecture 21 Data Review

Section 6: IICS Delete + Insert

Lecture 22 PSA Product Mapping

Lecture 23 PSA Product Mapping

Lecture 24 PSA Media Spend

Section 7: DWH Logic

Lecture 25 DWH mapping - 1

Lecture 26 DWH mapping - 2

Lecture 27 DWH mapping - 3

Lecture 28 DWH mapping - 4

Lecture 29 DWH mapping - 5

Lecture 30 DWH mapping - 6

Lecture 31 DWH mapping - 7

Section 8: Snowflake Dashboard

Lecture 32 Snowflake Dashboard Design - 1

Lecture 33 Snowflake Dashboard Design - 2

Lecture 34 Snowflake Dashboard Design - 3

Lecture 35 Snowflake Dashboard Design - 4

Lecture 36 Snowflake Dashboard Design - 5

Lecture 37 Snowflake Dashboard Design - 6

Lecture 38 Snowflake Dashboard Design - 7

Section 9: Streamlit - Master Product App

Lecture 39 Product Master app design - 1

Lecture 40 Product Master app design - 2

Lecture 41 Product Master app design - 3

Lecture 42 Product Master app design - 4

Lecture 43 Product Master app design - 5

Lecture 44 Product Master app design - 6

Lecture 45 Product Master app design - 7

Lecture 46 Product Master app design - 8

Lecture 47 Product Master app design - 9

Lecture 48 Product Master app design - 10

Section 10: Streamlit - Mapping Apps

Lecture 49 Product mapping app design - 1

Lecture 50 Product mapping app design - 2

Lecture 51 Product mapping app design - 3

Lecture 52 Product mapping app design - 4

Lecture 53 Touchpoint mapping app design - 1

Lecture 54 Touchpoint mapping app design - 2

Lecture 55 Touchpoint mapping app design - 3

Section 11: Course Review

Lecture 56 Historical load

Lecture 57 Course Review

Data Engineers looking to get proficient on IICS using cloud platforms for data load/ingestion,Data Engineers looking to get into building Streamlit apps in Snowflake