Snowflake Cortex Masterclass 2024 Hands-On!

by World-Class Snowflake Expert, former Data Superhero and SnowPro Certification SME

What you'll learn

Everything about Snowflake Cortex, the new AI & ML platform from Snowflake

How to implement end-to-end ML pipelines using both Snowpark and Snowpark ML

How to develop ML experiments with Snowflake using notebooks and code snippets

How to use the ML-powered classes and functions from Snowflake Cortex

How to call the new LLM functions from Snowflake Cortex

How to use Snowflake Copilot and other super-new LLM UI features in Snowsight

How to integrate Snowflake with ChatGPT using the OpenAI REST API

How to use Snowpark over in-memory Pandas DataFrames

Requirements

Basic knowledge of Snowflake

Basic knowledge of programming in SQL

Basic knowledge of programming in Python

Basic knowledge of Data Science and Machine Learning

Basic knowledge of Deep Learning and Transformers

Basic knowledge of LLMs (like ChatGPT) and their use cases

Optional knowledge of Streamlit

Optional knowledge of Time Series

Description

What is Snowflake CortexSnowflake got heavily involved into AI and ML only in the past two years. I know, because I've been around since the beginning. In Jan 2021 I was selected by them as a "Snowflake Data Superhero". And in my last two years alone I passed many DS and ML certification exams (see below).In mid-2023 they came up with Snowpark ML. Then with some built-in ML-powered functions, about regression and classification. The new Model Registry from Snowpark ML - added in Jan 2024 - allows now end-to-end ML on the platform.You'll see side-by-side data science experiments I will present to you "the old way", on Snowflake. Many of them with integrations with ChatGPT (about which I talked in detail in another course of mine). To compare them now with the new LLM functions, as Snowflake decided to host their own Large Language Models.Most of these (and other features yet to come) are now presented under the Snowflake Cortex umbrella.What you will learnHigh-level picture of the new Snowflake Cortex AI & ML platform.Detailed views on each of the Snowflake Cortex areas.How ML experiments were done on Snowflake before Cortex.How ML experiments can be implemented today with the Snowpark ML APIs, part of Snowflake Cortex.End-to-end Machine Learning with Snowpark ML and its Model Registry.How to use the new regression and classification ML-powered classes and functions, in Snowflake Cortex.How to call the new LLM functions from Snowflake Cortex, and compare them with ChatGPT.What to expect from Snowflake Copilot and other incoming Snowflake features in Cortex.What was the roadmap and what are the future plans of Snowflake for Snowflake Cortex or in the AI & ML areas.What this course is NOT aboutI will not teach you data science and machine learning here from ground up. You are expected to have some basic knowledge about ML, DL, LLMs…I will not teach you about many other areas in Snowflake. You are expected to have basic knowledge of Snowflake and data warehouses in general.I will not teach programming in Python or SQL. It's a hands-on course and you are expected to have some basic knowledge in this area. However, I may come-up with some small Streamlit apps, but I'll keep everything simple and easy to understand.This will not cover EVERYTHING you can do as data science in Snowflake. While you will see experiments "the old way" with scikit-learn, or integrations with ChatGPT, these are not part of Cortex! We have to limit mostly to Snowflake Cortex, as that's a huge platform.While Snowpark Container Services are also very new and they also target mostly ML and DL experiments (especially the new containers with GPUs), there will not be enough time to go deeper in detail. And this is also a very specialized and more difficult platform to understand. I may come up later on with a more advanced course on SPCS, but for now this course does not require such an advanced level of preparation.Who I amThe only world-class expert from Canada selected for the Snowflake Data Superhero program in 2021.SnowPro Certification SME (Subject Matter Expert) - many SnowPro exam questions have been created by me.Passed four SnowPro certification exams to date (with no retakes): Core, Architect, Data Engineer, Data Analyst.Specialized in Snowflake for the past few years: I worked for Snowflake Partner companies. I served dozens of clients in this capacity or as an independent consultant. Today I share my knowledge with highly specialize courses on Snowflake.A few of my latest Data Science and Machine Learning certificationsAWS Certified in Machine LearningMicrosoft Azure Data Scientist AssociateMicrosoft Azure AI Engineer AssociateMicrosoft Azure AI FundamentalsTensorFlow Developer CertificateAlteryx Machine Learning Fundamentals CertifiedDataiku ML Practitioner CertifiedDataiku MLOps Practitioner CertifiedNeo4j Graph Data Science CertifiedTigerGraph Graph Algorithms for Machine LearningThis course truly offers a complete coverage of the new Snowflake Cortex, and my intention is to update it frequently. Enroll today, and keep this course forever!

Overview

Section 1: Introduction to Snowflake Cortex

Lecture 1 Course Structure and Content

Lecture 2 Welcome to This Course

Lecture 3 All About This Course (FAQ Post)

Lecture 4 Roadmap to Snowflake Cortex

Lecture 5 Quick Tips: SQL Query Without Typing SQL

Lecture 6 Related Features and Technologies

Lecture 7 Overview of Snowflake Cortex

Lecture 8 Quick Tips: TRANSLATE LLM Function

Lecture 9 Quick Checkpoint: About …Quick Checkpoints

Section 2: ML Pipelines on Datasets (outside Snowflake)

Lecture 10 About this Section

Lecture 11 Quick Tips: Correlation Heatmap

Lecture 12 Introduction: Machine Learning Basics

Lecture 13 Introduction: ML Pipeline Phases

Lecture 14 Introduction: ML Pipeline Architectures

Lecture 15 Quick Checkpoint: What if You Already Know All This?

Lecture 16 Data Collection: Time Series Generation

Lecture 17 Data Collection: Make Regression/Classification

Lecture 18 Data Collection: Realistic Fake Data Generation

Lecture 19 Data Collection: Data Access

Lecture 20 Data Collection: Data Split

Lecture 21 Data Collection: Overview

Lecture 22 Quick Tips: Fake but Realistic Data Generation

Lecture 23 Data Exploration: Overview

Lecture 24 Data Exploration: Correlation Matrix Heatmap

Lecture 25 Data Exploration: Pandas Profiling

Lecture 26 Quick Checkpoint: About Pandas Profiling

Lecture 27 Data Wrangling: Overview

Lecture 28 Data Wrangling: Feature Engineering with Pandas DataFrame

Lecture 29 Data Wrangling: Data Preprocessing with Transformers

Lecture 30 Data Wrangling: Data Preprocessing with Pipeline

Lecture 31 Quick Checkpoint: About Basic ML on Datasets

Lecture 32 Quick Tips: SUMMARIZE LLM Function

Lecture 33 Model Training: Overview

Lecture 34 Model Training: Regression

Lecture 35 Model Training: Classification

Lecture 36 Model Validation: Manual Hyperparameter Optimization

Lecture 37 Model Validation: Manual Cross-Validation

Lecture 38 Model Validation: GridSearchCV for Regression

Lecture 39 Model Validation: RandomizedSearchCV for Classification

Lecture 40 Quick Checkpoint: About Model Validation

Lecture 41 Model Evaluation: Performance Metrics for Regression

Lecture 42 Model Evaluation: Performance Metrics for Classification

Lecture 43 Model Serving: Save/Load the Trained Model File

Lecture 44 Quick Tips: Signup for a Free Snowflake Trial Account

Section 3: ML Pipelines using Snowpark (before Cortex)

Lecture 45 About this Section

Lecture 46 Quick Tips: Uploading Files in Snowflake

Lecture 47 Introduction: Snowpark Components

Lecture 48 Introduction: Procedures and Functions from SQL

Lecture 49 Introduction: Snowpark for Python

Lecture 50 Introduction: Procedures and Functions from Python

Lecture 51 Introduction: Vectorized User-Defined Functions

Lecture 52 Introduction: Runtimes and Package Versions

Lecture 53 Introduction: Snowpark for ML Pipelines

Lecture 54 Data Collection: Populating with SQL Statements

Lecture 55 Data Collection: Synthetic Data Generation

Lecture 56 Data Collection: Faker Library in Python Worksheet

Lecture 57 Quick Tips: Easiest Way to Connect to Snowflake

Lecture 58 Data Collection: Uploading with SQL Scripts

Lecture 59 Data Collection: Uploading with Python Code

Lecture 60 Data Collection: Uploading from External Stages

Lecture 61 Data Collection: Uploading Other Datasets

Lecture 62 Data Collection: Sample Data Extraction

Lecture 63 Data Collection: Data Split

Lecture 64 Quick Checkpoint: About Ingesting Data in Snowflake

Lecture 65 Quick Tips: Correlation Heatmap in Snowflake

Lecture 66 Data Exploration: Snowsight Charts and Dashboards

Lecture 67 Data Exploration: Snowflake Partner Notebooks

Lecture 68 Data Exploration: Snowflake Notebooks

Lecture 69 Data Exploration: Overview

Lecture 70 Quick Tips: Data Profiling in Snowflake

Lecture 71 Quick Checkpoint: Pandas vs Snowpark Data Frames

Lecture 72 Feature Engineering: Pandas vs Snowpark DataFrames

Lecture 73 Feature Engineering: Using Pandas DataFrames

Lecture 74 Feature Engineering: Using Snowpark DataFrames

Lecture 75 Feature Engineering: Scalability Check with Python Worksheets

Lecture 76 Feature Engineering: Overview

Lecture 77 Quick Checkpoint: About the Python Worksheets

Lecture 78 Quick Tips: DataFrame Queries

Lecture 79 Data Preprocessing: When You Cannot Avoid Pandas

Lecture 80 Model Training: Sentiment Analysis in Local Mode

Lecture 81 Model Training: Sentiment Analysis with Stored Procedure

Lecture 82 Model Training: Overview

Lecture 83 Model Training: Sentiment Analysis with Imported Modules

Lecture 84 Model Training: House Predictions with Stored Procedure

Lecture 85 Model Serving: Overview

Lecture 86 Model Serving: Sentiment Predictions with UDFs

Lecture 87 Model Serving: Sentiment Predictions with SQL

Lecture 88 Model Serving: House Predictions with Vectorized UDF

Lecture 89 Model Serving: Introduction to Cachetools

Lecture 90 Model Serving: UDFs vs Vectorized UDFs

Section 4: ML Pipelines with Snowpark ML (in Cortex)

Lecture 91 About this Section

Lecture 92 Introduction: Snowpark ML APIs

Lecture 93 Data Collection: FileSystem

Lecture 94 Data Collection: FileSet and Framework Connectors

Lecture 95 Data Collection: SnowflakeFile

Lecture 96 Data Collection: Overview

Lecture 97 Distributed Preprocessing: Sklearn vs Snowpark ML

Lecture 98 Distributed Preprocessing: Snowpark vs Snowpark ML

Lecture 99 Distributed Preprocessing: Notebook Experiments

Lecture 100 Distributed Preprocessing: Overview

Lecture 101 Quick Tips: Python Worksheets

Lecture 102 Model Training: Sklearn vs Snowpark ML

Lecture 103 Model Training: Snowpark vs Snowpark ML

Lecture 104 Model Training: Notebook Experiment

Lecture 105 Model Training: Overview

Lecture 106 Quick Tips: Estimator Pattern in Snowpark ML Modeling

Lecture 107 Quick Checkpoint: About the Roadmap to Snowpark ML

Lecture 108 Distributed HPO: Sklearn vs Snowpark ML

Lecture 109 Distributed HPO: Snowpark vs Snowpark ML

Lecture 110 Distributed HPO: Notebook Experiment

Lecture 111 Distributed HPO: Overview

Lecture 112 Distributed Metrics: Sklearn vs Snowpark ML

Lecture 113 Distributed Metrics: Snowpark vs Snowpark ML

Lecture 114 Distributed Metrics: Notebook Experiment

Lecture 115 Distributed Metrics: Overview

Lecture 116 Snowflake MLOps: Overview

Lecture 117 Snowflake MLOps: Logging a Model

Lecture 118 Snowflake MLOps: The Model Registry

Lecture 119 Snowflake MLOps: Model Predictions from Registered Models

Lecture 120 Snowflake MLOps: Model Types and Providers

Lecture 121 Quick Tips: Prediction Functions from Model Registry

Lecture 122 Cost of Snowpark ML

Lecture 123 Quick Tips: Warehouse Auto-Suspend Value

Lecture 124 Quick Checkpoint: About Auto-Suspend in Warehouses

Section 5: ML Functions (in Cortex)

Lecture 125 About this Section

Lecture 126 Quick Tips: Simple Classification through Wizard

Lecture 127 Introduction: ML Classes

Lecture 128 Introduction: ML Class Methods

Lecture 129 Introduction: Snowflake SQL Classes

Lecture 130 Introduction: Snowflake SQL Class Instances

Lecture 131 Quick Checkpoint: About the ML-Powered Functions

Lecture 132 Classification: Binary Classifier

Lecture 133 Classification: Multiclass Classifier

Lecture 134 Classification: Bank Classifier

Lecture 135 Classification: Overview

Lecture 136 Quick Tips: Confusion Heatmap for Classification ML Class

Lecture 137 Forecasting: Time Series Data

Lecture 138 Forecasting: Prepare Sales Data

Lecture 139 Forecasting: Train Model and Predict Sales

Lecture 140 Forecasting: Train Model and Predict Temperatures

Lecture 141 Forecasting: Overview

Lecture 142 Anomaly Detection: Overview

Lecture 143 Anomaly Detection: Detect Outliers in Sales

Lecture 144 Anomaly Detection: Automation with Tasks and Alerts

Lecture 145 Anomaly Detection: Detect Outliers in Temperatures

Lecture 146 Quick Tips: Marking Outliers for Anomaly Detection

Lecture 147 Quick Checkpoint: About Forecasting and Anomaly Detection

Lecture 148 Gradient Boosting: Algorithm

Lecture 149 Gradient Boosting: Classifier & Regressor

Lecture 150 Contribution Explorer: Overview

Lecture 151 Contribution Explorer: What Led to a Change in Sales

Lecture 152 Contribution Explorer: What Makes a Customer Take to a Loan

Lecture 153 Contribution Explorer: How to Survive on Titanic

Lecture 154 Quick Checkpoint: TOP_INSIGHTS is NOT a Time Series Function!

Lecture 155 Access Rights: Introduction to Roles

Lecture 156 Access Rights: Classification

Lecture 157 Access Rights: Forecasting and Anomaly Detection

Lecture 158 Quick Checkpoint: About Access Rights to ML Classes and Functions

Lecture 159 Cost of ML Functions

Section 6: LLM Functions and Extensions (in Cortex)

Lecture 160 About this Section

Lecture 161 Quick Tips: SENTIMENT LLM Function

Lecture 162 Introduction to LLM Functions: Overview

Lecture 163 Introduction to LLM Functions: Quick Demo

Lecture 164 Introduction to Data Science: Important Milestones

Lecture 165 Introduction to Data Science: Deep Learning Review

Lecture 166 Introduction to Data Science: Generative AI Review

Lecture 167 Quick Checkpoint: About Deep Learning in Snowflake

Lecture 168 ChatGPT Integrations: Local Applications

Lecture 169 ChatGPT Integrations: Snowflake Applications

Lecture 170 ChatGPT Integrations: Overview

Lecture 171 COMPLETE LLM Functions

Lecture 172 EXTRACT_ANSWER LLM Function

Lecture 173 SENTIMENT LLM Function

Lecture 174 SUMMARIZE LLM Functions

Lecture 175 TRANSLATE LLM Function

Lecture 176 Quick Checkpoint: About the Specialized LLM Functions

Lecture 177 Applications with Cortex LLM Functions

Lecture 178 Access Rights to LLM Functions

Lecture 179 Cost of LLM Functions

Lecture 180 Quick Tips: Mistral-Large Cost

Lecture 181 Quick Checkpoint: About Mistral Large

Lecture 182 LLM Extensions in Snowsight

Lecture 183 Universal Search: Overview

Lecture 184 Snowflake Copilot: Quick Demo

Lecture 185 Snowflake Copilot: Overview

Lecture 186 Snowflake Copilot: SQL Query Generation with LangChain and ChatGPT

Lecture 187 Quick Checkpoint: Is Snowflake Copilot Reliable Enough?

Lecture 188 Document AI: Overview

Lecture 189 Document AI: Private Data Access with LlamaIndex and ChatGPT

Lecture 190 Quick Checkpoint: About ChatGPT Integrations

Section 7: Wrapping Up

Lecture 191 Setup Instructions: GitHub Project and VSCode

Lecture 192 Setup Instructions: Free Snowflake Trial Account

Lecture 193 Setup Instructions: ChatGPT/OpenAI Account

Lecture 194 Congratulations, You Made It!

Lecture 195 Bonus Lecture

Data Scientists who want to learn about all AI & ML opportunities in Snowflake,Data Analysis looking how to use the new ML-based and LLM functions,Data and Software Engineers looking to expand into AI & ML on Snowflake,Project Managers looking for a 360 degree view of the new Snowflake Cortex platform,Data Architects willing to understand fast how Snowflake Cortex is built,Anyone else looking for a high-level (but detailed) picture of Snowflake Cortex,Anyone looking to understand the code and query pushdown model of Snowflake,Anyone looking how to save money on Snowflake using new built-in ML and LLM functions,Anyone looking for other Snowflake features yet to come in the AI & ML area