Pyspark Crash Course - Learn Spark, Quickly!

Accelerate Your Data Skills: Dive into PySpark!

What you'll learn

Learn to load data into PySpark dataframes

Learn to wrangle your data to clean, handle nulls & handle duplicates

Learn to create calculated fields, aggregate your data & extract insights

Learn to implement advanced PySpark techniques such as window functions and user-defined functions (UDFs)

Requirements

Some basic Python knowledge is desirable but not entirely necessary

Description

Ready to dive into the fascinating world of Apache Spark (PySpark)? This course is your ticket to unraveling the mysteries of Spark, starting from the ground up and zooming all the way into some seriously cool stuff like window functions and user-defined functions (UDFs).What You'll Discover:Playing with Data: Get your hands dirty with Spark SQL and learn how to wield DataFrames like a pro, mastering the art of manipulating, filtering, and crunching data.Next-Level Tricks: Ever heard of window functions or UDFs? We'll guide you through these advanced concepts, empowering you to perform super-smart analytics and craft custom functions for your data.Why This Course Rocks: We're all about making it count! Instead of dragging things out, we're here to give you the essential skills pronto. We believe that having the core knowledge means you can jump right into action. Who's Welcome Here:Data wizards (and those aspiring to be one)Tech enthusiasts hungry for big data actionAnyone itching to explore Spark and take their data skills up a notchHow We Roll:Short and Sweet: Bite-sized modules for quick learning bursts.Hands-On Fun: Dive into real-world examples and projects for that practical edge.What's in Store for You: Once you've completed this ride, you'll be armed with a solid Spark foundation. You'll confidently handle data, wield window functions like a champ, and even create your own custom UDFs. Get ready to tackle real-world data puzzles and unearth meaningful insights from big datasets.Ap

Overview

Section 1: Introduction

Lecture 1 Course Intro

Lecture 2 Exploring The Data

Lecture 3 Our Development Environment

Section 2: Basics

Lecture 4 Ingesting Our Data

Lecture 5 Inspecting Our Dataframe

Lecture 6 Creating a custom schema

Lecture 7 Handling Null Values

Lecture 8 Running SQL on our dataframes

Lecture 9 Group by & Aggregation

Lecture 10 Creating Calculated Fields

Lecture 11 Handling Duplicates

Lecture 12 Writing to Files

Section 3: Case Statements (When)

Lecture 13 Case Statements: Section 1

Lecture 14 Case Statements: Section 2

Lecture 15 Case Statements: Section 3

Lecture 16 Case Statements: Section 4

Lecture 17 Case Statement: Challenge

Lecture 18 Case Statement: Solution

Section 4: Window Functions

Lecture 19 Rank Window Function

Lecture 20 Row Number Window Function

Lecture 21 Lead / Lag Window Function

Lecture 22 Sum Window Function

Lecture 23 Window Function Challenge

Lecture 24 Window Function Challenge Solution

Section 5: Filtering Dataframes

Lecture 25 Filtering Dataframes: 1

Lecture 26 Filtering Dataframes: 2

Lecture 27 Filtering Dataframes: 3

Lecture 28 Filtering Dataframes: 4

Section 6: UDFs (User Defined Functions)

Lecture 29 UDFs: 1

Lecture 30 UDFs: 2

Lecture 31 UDF: Challenge

Lecture 32 UDF: Challenge Solution

Section 7: Working With Datetimes

Lecture 33 Working with Datetimes

Lecture 34 Datetime: Challenge

Lecture 35 Datetime: Solution

Section 8: Wrapping Up

Lecture 36 Congratulations!

Lecture 37 Next Steps

Anyone with a desire to learn Apache Spark - to enhance their careers or break into the field of data engineering