Apache Kafka - Kafka Connect (Everything You Need To Know)

Master Kafka Connect with hands-on projects: source & sink connectors, REST API, transformations, and error handling.

What you'll learn

Understand the architecture and core concepts of Kafka Connect, including how connectors, workers, and tasks operate

Set up and configure source and sink connectors to integrate Kafka with various data systems.

Use the Kafka Connect REST API to create, manage, and monitor connectors effectively.

Apply Single Message Transformations (SMTs) to modify data in-flight.

Implement robust error handling strategies, including configuring dead letter queues (DLQs) for failed records.

Requirements

Basic understanding of Apache Kafka concepts (producers, consumers, topics, partitions).

Familiarity with Docker and Docker Compose for running containers.

Description

Apache Kafka is one of the most popular platforms for real-time data streaming, and Kafka Connect is the key to integrating Kafka with external systems—without writing a single line of producer or consumer code. In this course, you’ll gain practical, hands-on experience setting up and managing Kafka Connect, learning exactly how to move data into and out of Kafka efficiently and reliably.We’ll start by exploring the fundamentals—how Kafka Connect works, the difference between source and sink connectors, and the two modes of operation: standalone and distributed. Then we’ll jump into real-world projects, deploying connectors that stream data from databases, files, and APIs into Kafka, and export Kafka topics to systems like Elasticsearch and S3.Along the way, you’ll learn how to:Use the Kafka Connect REST API to create, update, and manage connectors programmatically.Apply Single Message Transformations (SMTs) to filter, modify, and enrich data “in-flight.”Configure error handling with dead letter queues to build fault-tolerant pipelines.Work with plugins and manage your connector environment with Docker for consistency across any machine.By the end of this course, you’ll have the confidence and skills to design, deploy, and manage production-grade Kafka Connect pipelines. Whether you’re a developer, data engineer, or DevOps professional, this course will give you the tools you need to integrate Kafka into any data ecosystem.

Overview

Section 1: Introduction

Lecture 1 Intro

Section 2: Kafka Connect - Theory

Lecture 2 What is kafka connect

Lecture 3 How Kafka Connect works

Section 3: Environment Setup

Lecture 4 Setup

Section 4: Rest API

Lecture 5 Rest API - Theory

Lecture 6 Rest API - Hands on

Section 5: Standalone Mode

Lecture 7 Standalone mode - Hands on

Section 6: Source Connector

Lecture 8 File Source Connect - Hands on

Section 7: Sink Connector

Lecture 9 File Sink Connector - Hands on

Lecture 10 JDBC Sink Connector - Hands on

Lecture 11 Elasticsearch Sink Connector - Hands on

Section 8: Transformations

Lecture 12 Transformations - Theory

Lecture 13 Transformations - Hands on

Section 9: Error Handling

Lecture 14 Error Handling - Theory

Lecture 15 Error Handling - Hands on

This course is for software engineers, data engineers, DevOps professionals, and architects who want to integrate Apache Kafka with external systems using Kafka Connect.