Apache Kafka - Spark - Cassandra Real Time Streaming Project

Real Time Data Streaming Project, that involved building the architecture from scratch and deployment.

What you'll learn

Professionals looking for an end to end kafka, spark and cassandra streaming pipeline.

Anyone who wants to understand how to use Apache kafka in their current architecture.

Engineers who are looking to design a Kafka Solution Pipeline.

A complete case study from start to execution.

Requirements

Baisc understanding of Kafka Architecure, Spark and Cassandra or SQL

Description

The course is designed to provide a comprehensive understanding of real-time big data processing using Kafka, Spark, and Cassandra. In today's world, data is produced at an unprecedented rate, and the ability to process and analyze this data in real-time is critical for making informed decisions. This course focuses on the fundamental concepts and architecture of Kafka, Spark, and Cassandra, and how they work together to create a robust big data processing pipeline.Students will learn how to set up Kafka clusters and work with Kafka producers and consumers. Students will also learn about Kafka Streams, a client library for building real-time streaming applications that process data directly within Kafka.Throughout the course, students will gain hands-on experience through practical exercises and projects that simulate real-world scenarios. By the end of the course, students will have a understanding of how to use Kafka, Spark, and Cassandra to build real-time big data processing systems.Course Objectives:Understand the fundamental concepts of real-time big data processingLearn the architecture setup of Kafka, Spark, and CassandraUnderstand how Kafka, Spark, and Cassandra work together to create a real-time big data processing pipelineGain hands-on experience with Kafka, Spark, and Cassandra through practical exercises and projectsLearn how to build a real-time big data processing pipeline from scratchThis course is intended for software engineers, data engineers, and data analysts who have a basic understanding of programming concepts and are familiar with SQL.

Overview

Section 1: Infrastructure Setup for the Pipeline

Lecture 1 Introduction: Project_Overview

Lecture 2 Documents

Lecture 3 Ubuntu_Machine_Creation

Lecture 4 Putty_Installation

Lecture 5 Java_Installation

Lecture 6 Kafka_Setup

Lecture 7 Hadoop_Installation

Lecture 8 SPARK SCALA Installation

Lecture 9 Cassandra_Installation

Section 2: Pipeline Setup For KAKFA SPARK Real Time Streaming

Lecture 10 JSON Data Overview and Cassandra Table Creation

Lecture 11 KAFKA Producer & Consumer Configuration

Lecture 12 Code Overview(Spark Scala)

Section 3: FINAL Pipeline Execution End to End(KAFKA + SPARK + CASSANDRA)

Lecture 13 Pipeline Execution

Lecture 14 Destroy Droplet (*Important)

Engineers looking for a CASE STUDY on Real Time Data Streaming involving KAKFA and SPARK