Aws Managed Streaming For Apache Kafka (Msk) Crash Course

Learn to provision, operate, integrate and develop with Managed Kafka on AWS

What you'll learn

AWS Managed Streaming Kafka (MSK)

MSK Kafka Connect

Glue Schema Registry

Deployment Recipies

Spring Kafka Java Clients

Python Kafka Clients

Requirements

General AWS Knowledge

Python Programming

Java Programming (optional, only needed for Java examples)

Apache Kafka Fundamentals

Description

This is the AWS Managed Streaming Apache Kafka (MSK) Crash Course where students learn the essential concepts and operational characteristics of this wildly popular AWS service. The MSK Crash Course is designed specifically for mid to senior level engineers and architects who are already proficient with general AWS Cloud concepts (things like IAM, VPCs, SSM Parameter Store, Secrets Manager, EC2 and Lambda) as well as Apache Kafka fundamentals (things like Brokers, Topics, Partitions, Scalability and Reliability, Kafka Connect, Message Schemas). The course goal is to equip developers and architects with the knowledge needed to confidently implement MSK into existing projects or dive in as a productive and trusted contributor to existing projects utilizing AWS MSK.Information in this course is presented in a manner that balances enough theory to understand the capabilities of the technologies covered immediately followed by practical hands on demos of the theoretical content. The hands on demos heavily rely on easy to understand Apache Kafka client code samples and high quality Infrastructure as Code (IaC) based deployment recipes based on AWS Cloud Development Kit (CDK). This approach gives students the fundamental knowledge and confidence to dive deeper into the examples and further tweak or extend them to fit a variety of use cases.

Overview

Section 1: Introduction

Lecture 1 Course Introduction

Lecture 2 Course Purpose

Lecture 3 Course Format

Lecture 4 Course Intended Audience

Lecture 5 Course Outline

Lecture 6 Course Author

Section 2: Overview of AWS Cloud Development Kit (CDK)

Lecture 7 Section Introduction

Lecture 8 Defining Infrastructure as Code

Lecture 9 Deployment Recipe: Example Network and Bastion Host (Part 2)

Lecture 10 Cloud Development Kit (CDK)

Lecture 11 CDK CLI and Constructs Library

Lecture 12 CDK Construct Types

Lecture 13 AWS IaC Abstractions Hierarchy

Lecture 14 Hello API Demo App Overview and Installing AWS CDK

Lecture 15 Deploy Hello API Demo App with CDK

Lecture 16 Recap of Deployed Hello API CDK App

Lecture 17 Cleanup Hello API CDK App

Section 3: MSK Overview & Lab Setup

Lecture 18 Section Introduction

Lecture 19 High Level Apache Kafka Cluster Architecture (1K Foot View)

Lecture 20 Generic MSK Network Architecture

Lecture 21 Deployment Models & Operational Characteristics

Lecture 22 MSK Broker Configuration and Documentation Overview

Lecture 23 MSK Integrations

Lecture 24 Discussion of Labs Network & Ideal Production Network Architecture

Lecture 25 Deploy Recipe: Example Network and Bastion Host Part 1

Lecture 26 Deploy Recipe: Example Network and Bastion Host Part 2

Section 4: IAM Authentication Model

Lecture 27 Section Introduction

Lecture 28 IAM Authentication Cluster Deployment Architecture

Lecture 29 Deploy Recipe: Provisioned MSK Cluster with IAM Auth Part 1

Lecture 30 Deploy Recipe: Provisioned MSK Cluster with IAM Auth Part 2

Lecture 31 Kafka CLI: IAM Auth & Topic Creation from Bastion Host

Lecture 32 IAM Authentication Example Client Flow

Lecture 33 Deploy Recipe: Spring Kafka on ECS Fargate with IAM Auth and SSM Params (Part 1)

Lecture 34 Deploy Recipe: Spring Kafka on ECS Fargate with IAM Auth and SSM Params (Part 2)

Section 5: SASL/SCRAM Authentication Model

Lecture 35 Section Introduction

Lecture 36 SASL/SCRAM Authentication Cluster Deployment Architecture

Lecture 37 Deploy Recipe: Provisioned MSK Cluster with SASL/SCRAM Auth (Part 1)

Lecture 38 Deploy Recipe: Provisioned MSK Cluster with SASL/SCRAM Auth (Part 2)

Lecture 39 Kafka CLI: SASL/SCRAM Auth & Topic Creation from Bastion Host

Lecture 40 Python Demo: SASL/SCRAM from Bastion Host

Lecture 41 Deploy Recipe: Spring Kafka on ECS Fargate with SASL/SCRAM Auth (Part 1)

Lecture 42 Deploy Recipe: Spring Kafka on ECS Fargate with SASL/SCRAM Auth (Part 2)

Lecture 43 Deploy Recipe: Python on ECS Fargate with SASL/SCRAM Auth (Part 1)

Lecture 44 Deploy Recipe: Python on ECS Fargate with SASL/SCRAM Auth (Part 2)

Section 6: MSK Connect - Managed Apache Kafka Connect

Lecture 45 Section Introduction

Lecture 46 MSK Connect Overview

Lecture 47 MSK Connect Key Points

Lecture 48 Aurora PostgreSQL Change Data Capture with Debezium PostgreSQL Connector

Lecture 49 Deploy Recipe: Debezium PostgreSQL CDC on Aurora

Section 7: MSK Operations

Lecture 50 Section Introduction

Lecture 51 Broker Storage Scaling

Lecture 52 Deploy Recipe: MSK Broker Storage Auto Scaling

Lecture 53 Horizontal Scaling: Adding Cluster Brokers

Lecture 54 Deploy Recipe: Adding Brokers to MSK Cluster

Lecture 55 Kafka CLI: Rebalancing Broker Partitions

Lecture 56 MSK Metrics Overview

Lecture 57 Monitoring Consumer Lag

Section 8: Glue Schema Registry

Lecture 58 Section Introduction

Lecture 59 Glue Schema Registry Overview

Lecture 60 Client Request Flow & Integration Architecture

Lecture 61 Deploy Recipe: Glue Schema Registry with Java & Spring Kafka on ECS Fargate

Lecture 62 Deploy Recipe: Glue Schema Registry with Python on ECS Fargate

Section 9: MSK Replicator

Lecture 63 Section Introduction

Lecture 64 MSK Replicator Overview

Cloud Engineers and Architects looking to quickly get up to speed and productive with AWS MSK