Python Kafka Mastery: Real-Time Streaming & Analytics

Build End-to-End Real-Time Streaming Pipelines with Kafka, Flink, Elasticsearch & Kibana in Python (last versions)

What you'll learn

Master data ingestion: Efficiently process diverse data streams.

Unleash Kafka's power: Understand core concepts for optimal streaming.

Learn major CLIs: kafka-topics, kafka-console-producer, kafka-console-consumer…

Python + Kafka: Build practical skills for producers, consumers, Topics, Partitions, Brokers, and more.

Create your Producers and Consumers in Python to interact with Kafka

Real-world Twitter data: Ingest and scale data seamlessly into HDFS.

Streamline with Kafka and Flink: Craft end-to-end pipelines for real-time analytics.

Real-time Analytics and Visualization: Master the art of analyzing streaming data and visualizing insights using Elasticsearch and Kibana in your Kafka pipeline

Requirements

Some understanding of Python Programming

Good to have knowledge about Linux command line

Desire to Master Big Data Streaming

Good to have knowledge on big data processing with Flink or Spark

Description

Unleashing the Power of Apache Kafka and Flink: Cutting-Edge Hands-on Experience with real life case studiesThis is the only updated Big Data Streaming Course using Kafka with Flink in python ! (Course newly recorded with Kafka 3.+, Flink 1.14.4, ES 7.17.7) Discover the unrivaled potential of Apache Kafka and the hidden gem of data processing, Flink, in our dynamic course. While Flink may be lesser-known than Spark, it's a powerful tool that surpasses its counterparts in certain aspects.We'll dive deep into Kafka's core concepts, equipping you with the knowledge to build robust streaming pipelines. But we won't stop there – we'll showcase Flink's prowess as we explore real-time data processing and analytics.Rest assured, all hands-on exercises are meticulously crafted using the latest versions of Kafka and Flink. Forget about outdated code or compatibility issues – we ensure you're working with cutting-edge tools, ready to conquer the real world.Although Flink may have a smaller community compared to Spark, this presents a unique opportunity for you to become an early adopter and join the pioneering minds pushing the boundaries of streaming analytics.We'll guide you step-by-step as you build a complete streaming pipeline that captures live Twitter data, processes it in real-time, and unlocks valuable insights. With our carefully crafted exercises, you'll gain practical experience in ingesting, transforming, and analyzing Twitter data using the latest versions of Kafka and Flink.Imagine harnessing the pulse of social media to gain actionable insights, all in real-time. From sentiment analysis to trending topics, you'll explore the limitless possibilities of Twitter data analytics.So, step into the future of stream data processing with Kafka and Flink. Enroll now to gain an edge in the industry, with hands-on expertise on the latest versions of these powerful tools. Don't miss out on this transformative learning experience – the world of real-time data awaits!

Overview

Section 1: Introduction

Lecture 1 Introduction

Section 2: Introduction to Data Ingestion

Lecture 2 Introduction & problematic of Data Ingestion

Lecture 3 what is Data Ingestion

Lecture 4 Data Ingestion Tools

Section 3: Introduction to Apache Kafka

Lecture 5 Introduction to Kafka - Publish Subscribe Architecture

Lecture 6 Advantages of Apache Kafka

Section 4: Kafka fundamentals

Lecture 7 Overview of Kafka components

Lecture 8 kafka topic

Lecture 9 Kafka Partitions

Lecture 10 Kafka Topic Replication

Section 5: Kafka CLI hands-on

Lecture 11 Kafka CLI Hands-on | Introduction

Lecture 12 Start 3 nodes Kafka Cluster

Lecture 13 Create Kafka topic with CLI

Lecture 14 Delete Kafka Topic with CLI

Lecture 15 Kafka Console CLI: Producer and Consumer

Lecture 16 Linux commands used for 3 nodes Kafka Cluster

Section 6: Kafka Hands-on in python

Lecture 17 Kafka Producer and Consumer Hands-on in python | Introduction

Lecture 18 Simple Kafka Producer in Python - Part 1 (coding)

Lecture 19 Kafka Producer in Python - Part 2 - ( testing)

Lecture 20 Kafka Consumer in python -Part 1 - (coding)

Lecture 21 Kafka Consumer in python -Part 2 - (running)

Lecture 22 Complete Kafka (Producer + Consumer) python code resources

Section 7: Real World Big Data Ingestion Python Project: Streaming Twitter Data with Kafka

Lecture 23 The architecture Design of the hands-on project

Lecture 24 Real time Data Source: Twitter API from Developer Platform

Lecture 25 Extracting Twitter Data Stream from API in python

Lecture 26 Create a Tweets Data Kafka Producer

Lecture 27 Create a Tweets Data Kafka Consumer

Lecture 28 Kafka Consumer : Store Data in Hadoop HDFS

Lecture 29 Complete Python Code resources (Twitter Producer & Consumer + HDFS Consumer)

Section 8: Real Time Streaming pipeline Handson : Kafka, Flink, ElasticSearch and Kibana

Lecture 30 Real Time Streaming Architecture Design: Kafka , Flink, Elasticsearch, & Kibana

Lecture 31 Requirement for this project (updated versions 3.+.+)

Lecture 32 Apache Flink Introduction | Apache Spark -VS- Flink | PyFlink

Lecture 33 Configure Flink to consume data from a Kafka topic as a data source

Lecture 34 Configure Flink to write the processed data to a Elasticsearch sink

Lecture 35 Real Time Tweets Word Count with pyFlink and Kafka

Lecture 36 Complete Python Code : Streaming pipeline

Section 9: Real World Project Exercice-Solution

Developers who want to learn the Data Ingestion, Apache Kafka , Streaming with Apache Flink,Big Data Architects who want to understand how Apache Kafka fits into their solution architecture,Those desiring to build robust streaming pipelines for real-time analytics,Big Data enthusiasts and software developers looking to expand their skill set,Professionals aiming to stay ahead in the rapidly evolving Big Data landscape,Data analysts and professionals in the field of real-time data processing who want to leverage Kafka and Flink for advanced analytics.,Beginners seeking to enter the world of Kafka and Flink streaming and gain practical hands on skills.,Aspiring data engineers and software developers eager to master Kafka, Flink, Elasticsearch, and Kibana for end-to-end real-time analytics pipelines.,Professionals aiming to stay ahead in the dynamic data landscape by acquiring comprehensive skills in real-time data processing and visualization.