Reactive Kafka From Scratch

Getting Started With Event Driven Microservices

What you'll learn

Learn Kafka From Scratch

Spring Kafka - Event Driven Microservices

Batch & Parallel Processing

Kafka Transaction

Writing Integration Tests

Best Practices

Requirements

Java Reactive Programming

Spring

Comfortable with Indian Accent

Description

**  Reactive Spring Kafka - Getting Started With Event Driven Microservices **This course is particularly for any senior or staff level engineer who wants to learn Reactive Kafka From Scratch. By the end of the course, you will become very comfortable with Kafka.Here is what we will do in this course:KafkaSetting up Kafka - latest KRaft mode using custom docker image Bootstrap ServerVarious Kafka Topic commands and OptionsConsole ProducerConsole consumerVarious Kafka propertiesHow apps publish and consume messagesHow serialization and Deserialization works in KafkaHow Kafka Log Retention worksOffset and How to reset Offset to consume messages from specific timeMessage FormatHow to scale Applications using Kafka Consumer GroupMessage OrderingKafka With Reactive ProgrammingCreating a simple Java Kafka producer and consumer applications using reactive programmingVarious producer and consumer configurationAuto offset resetHow to acknowledge the messagesEmitting millions events to check the performanceSeeking offsetPartition assignment strategyCooperative sticky assignorRange assignorKafka ClusterHow cluster worksSetting up Kafka cluster using docker composeKafka cluster - Fault tolerance demo. How producer and consumer applications work without any issues when Kafka brokers go up and down.Best PracticesWhat is producer acknowledge and why we should do that?Min in sync replicasCreating idempotent producerCreating idempotent consumer to take care of duplicate messagesCompression - to reduce the size of messages and improve the throughputDiscussing Common QuestionsHow many topics should I create?How many partitions should I create?How many replicas should I have?Batch / Parallel ProcessingConsuming messages in parallel and proper order to make use of all the CPUs.Error HandlingRetryDead Letter TopicPoison Pill MessagesKafka TransactionHow to produce and consume messages in transactional mode (like Database Transaction)Integration TestsDeveloping integration tests using Embedded Kafka Kafka SecurityPlain CredentialsSSLAssignmentLets test our learning by developing a couple of Microservices communicating via Kafka Topic to analyze the user behaviors and provide metrics to Business team on Product Trending.

Overview

Section 1: Introduction

Lecture 1 Introduction

Lecture 2 Need For Event Driven Architecture / Kafka

Lecture 3 Resource Download

Section 2: Kafka Setup

Lecture 4 Introduction

Lecture 5 Custom Docker Image

Lecture 6 Docker Compose Setup

Section 3: Kafka Crash Course

Lecture 7 Introduction

Lecture 8 Cluster

Lecture 9 Bootstrap Server

Lecture 10 Topic Command

Lecture 11 Exploring Data Directory

Lecture 12 Documenting Commands

Lecture 13 Console Producer

Lecture 14 Console Consumer

Lecture 15 Producer & Consumer Side By Side

Lecture 16 [Properties] linger.ms vs batch.size

Lecture 17 Consumer Push vs Pull

Lecture 18 Serialization vs Deserialization

Lecture 19 [Properties] Log Retention

Lecture 20 Offset

Lecture 21 Message Timestamp

Lecture 22 2 Consumers Demo

Lecture 23 Consumer Group

Lecture 24 Consumer Group - Demo

Lecture 25 Need For Message Ordering

Lecture 26 Partition & Key

Lecture 27 Who is calculating the Partition?

Lecture 28 Topic With 2 Partitions

Lecture 29 Consumer Group Demo with Partitions - Part 1

Lecture 30 Consumer Group Demo with Partitions - Part 2

Lecture 31 Consumer Group Scaling - Scenarios

Lecture 32 Consumer Group - LAG

Lecture 33 Multiple Consumer Group - LAG

Lecture 34 Resetting Offset

Lecture 35 Resetting Offset - Demo

Lecture 36 Message Format

Lecture 37 Summary

Section 4: Reactor Kafka

Lecture 38 Maven Dependencies

Lecture 39 Project Setup

Lecture 40 Consumer Config

Lecture 41 Creating Kafka Receiver

Lecture 42 Kafka Receiver Demo

Lecture 43 Auto Offset Reset

Lecture 44 Session Timeout Config

Lecture 45 Acknowledging Message

Lecture 46 Enabling Auto Commit

Lecture 47 Will Messages Be Redelivered?

Lecture 48 Out Of Order Commit

Lecture 49 Consume From Multiple Topics

Lecture 50 Kafka Producer Config

Lecture 51 Creating Sender Record

Lecture 52 Kafka Producer Demo

Lecture 53 Sender Close

Lecture 54 Receiver Close

Lecture 55 Emitting 1 Million Events

Lecture 56 Max In Flight

Lecture 57 Message Headers

Lecture 58 Consumer Group Demo - Part 1

Lecture 59 Consumer Group Demo - Part 2

Lecture 60 Cooperative Sticky Assignor

Lecture 61 Range Assignor

Lecture 62 Seeking Offset

Lecture 63 Summary

Section 5: Kafka Cluster

Lecture 64 Introduction

Lecture 65 Quick Note On Listeners

Lecture 66 Cluster Properties

Lecture 67 Cluster Demo

Lecture 68 Kafka Fault Tolerance Demo

Lecture 69 Summary

Section 6: Best Practices

Lecture 70 Producer acks

Lecture 71 min.insync.replica

Lecture 72 Idempotent Producer

Lecture 73 Producing Duplicate Messages

Lecture 74 Idempotent Consumer

Lecture 75 Compression

Lecture 76 How Many Topics Should I Create?

Lecture 77 How Many Partitions Should I Create?

Lecture 78 How Many Replication Factor Should I Have?

Lecture 79 Summary

Section 7: Batch / Parallel Processing

Lecture 80 Introduction

Lecture 81 Batch - Receiver Auto Ack / concatMap

Lecture 82 Parallel - flatMap

Lecture 83 Schedulers with FlatMap

Lecture 84 Message Ordering Issue With FlatMap

Lecture 85 GroupedFlux - Parallel and Message Ordering - Part 1

Lecture 86 GroupedFlux - Parallel and Message Ordering - Part 2

Lecture 87 Summary

Section 8: Error Handling

Lecture 88 Introduction

Lecture 89 Simple Retry

Lecture 90 Separating Receiver & Processor - Theory

Lecture 91 Separating Receiver & Processor - Implementation - Part 1

Lecture 92 Separating Receiver & Processor - Implementation - Part 2

Lecture 93 Separating Receiver & Processor - Implementation - Part 3

Lecture 94 Retry Depending On Error

Lecture 95 Dead Letter Topic - Introduction

Lecture 96 Reactive Dead Letter Topic Producer - Part 1

Lecture 97 Reactive Dead Letter Topic Producer - Part 2

Lecture 98 Order Event Processor

Lecture 99 Code Cleanup

Lecture 100 Dead Letter Topic Demo

Lecture 101 Poison Pill Messages

Lecture 102 Poison Pill Messages - Fix

Lecture 103 Error Handling At Producer Side

Lecture 104 Summary

Section 9: Kafka Transaction

Lecture 105 Introduction

Lecture 106 Demo App Walkthrough

Lecture 107 Transfer Event Consumer

Lecture 108 Transfer Event Processor - Part 1

Lecture 109 Transfer Event Processor - Part 2

Lecture 110 Transfer Config

Lecture 111 Adding Delay

Lecture 112 Kafka Commands

Lecture 113 Kafka Transaction Demo

Lecture 114 How Kafka Transaction Works

Lecture 115 read_committed vs read_uncommitted

Lecture 116 Send Transactionally

Lecture 117 Producer Fenced Exception

Lecture 118 Delivery Semantics

Section 10: Reactive Kafka With Spring

Lecture 119 Introduction

Lecture 120 Simple Spring Kafka Consumer

Lecture 121 Order Event DTO

Lecture 122 Spring Kafka Producer Config

Lecture 123 Order Event Decoding Issue

Lecture 124 Importance Of Type ID

Lecture 125 Order Event Demo

Lecture 126 Printing Type ID Header

Lecture 127 Decoding As Custom Object

Lecture 128 Summary

Section 11: Integration Testing

Lecture 129 Introduction

Lecture 130 Embedded Kafka Test

Lecture 131 Embedded Kafka Demo

Lecture 132 Embedded Kafka Random Port

Lecture 133 Project Restructure

Lecture 134 Abstract Integration Test - Utilities - Part 1

Lecture 135 Abstract Integration Test - Utilities - Part 2

Lecture 136 OrderEventProducer - Test

Lecture 137 OrderEventProducer - Demo

Lecture 138 @DirtiesContext

Lecture 139 Creating Sender Util

Lecture 140 OrderEventConsumer - Demo

Section 12: *** ASSIGNMENT ***

Lecture 141 Introduction

Lecture 142 R2DBC - Persistable Interface

Lecture 143 ** ASSIGNMENT **

Lecture 144 [ASSIGNMENT SOLUTION] - Product Service Setup

Lecture 145 [ASSIGNMENT SOLUTION] - Product Service - Implementation - Part 1

Lecture 146 [ASSIGNMENT SOLUTION] - Product Service - Implementation - Part 2

Lecture 147 [ASSIGNMENT SOLUTION] - Product Service - Integration Test - Part 1

Lecture 148 [ASSIGNMENT SOLUTION] - Product Service - Integration Test - Part 2

Lecture 149 [ASSIGNMENT SOLUTION] - Analytics Service - Setup

Lecture 150 [ASSIGNMENT SOLUTION] - Analytics Service - Implementation - Part 1

Lecture 151 [ASSIGNMENT SOLUTION] - Analytics Service - Implementation - Part 2

Lecture 152 [ASSIGNMENT SOLUTION] - Analytics Service - Implementation - Part 3

Lecture 153 [ASSIGNMENT SOLUTION] - Analytics Service - Integration Test - Part 1

Lecture 154 [ASSIGNMENT SOLUTION] - Analytics Service - Integration Test - Part 2

Lecture 155 Final Demo

Lecture 156 Final Demo With Companion Flux

Section 13: Kafka Security

Lecture 157 Introduction

Lecture 158 App With Kafka Credentials

Lecture 159 SSL Introduction

Lecture 160 Kafka SSL

Lecture 161 Kafka SSL Demo

Section 14: Thank you

Lecture 162 What's Next

Any senior / staff level engineer