Su	Mo	Tu	We	Th	Fr	Sa
1	2	3	4	5	6	7
8	9	10	11	12	13	14
15	16	17	18	19	20	21
22	23	24	25	26	27	28
29	30	31	1	2	3	4

Apache Spark for Java Developers

Posted By: ParRus

Date: 16 Nov 2020 19:00:37

Get processing Big Data using RDDs, DataFrames, SparkSQL and Machine Learning - and real time streaming with Kafka!

What you'll learn
Use functional style Java to define complex data processing jobs
Learn the differences between the RDD and DataFrame APIs
Use an SQL style syntax to produce reports against Big Data sets
Use Machine Learning Algorithms with Big Data and SparkML
Connect Spark to Apache Kafka to process Streams of Big Data
See how Structured Streaming can be used to build pipelines with Kafka

Requirements
Some basic Java programming experience is required. A crash course on Java 8 lambdas is included
You will need a personal computer with an internet connection.
The software needed for this course is completely freely and I'll walk you through the steps on how to get it installed on your computer

Description
Get started with the amazing Apache Spark parallel computing framework - this course is designed especially for Java Developers.

If you're new to Data Science and want to find out about how massive datasets are processed in parallel, then the Java API for spark is a great way to get started, fast.

All of the fundamentals you need to understand the main operations you can perform in Spark Core, SparkSQL and DataFrames are covered in detail, with easy to follow examples. You'll be able to follow along with all of the examples, and run them on your own local development computer.

Included with the course is a module covering SparkML, an exciting addition to Spark that allows you to apply Machine Learning models to your Big Data! No mathematical experience is necessary!

And finally, there's a full 3 hour module covering Spark Streaming, where you will get hands-on experience of integrating Spark with Apache Kafka to handle real-time big data streams. We use both the DStream and the Structured Streaming APIs.

Optionally, if you have an AWS account, you'll see how to deploy your work to a live EMR (Elastic Map Reduce) hardware cluster. If you're not familiar with AWS you can skip this video, but it's still worthwhile to watch rather than following along with the coding.

You'll be going deep into the internals of Spark and you'll find out how it optimizes your execution plans. We'll be comparing the performance of RDDs vs SparkSQL, and you'll learn about the major performance pitfalls which could save a lot of money for live projects.

Throughout the course, you'll be getting some great practice with Java 8 Lambdas - a great way to learn functional-style Java if you're new to it.

NOTE: Java 8 is required for the course. Spark does not currently support Java9+ (we will update when this changes) and Java 8 is required for the lambda syntax.

Who this course is for:
Anyone who already knows Java and would like to explore Apache Spark
Anyone new to Data Science who want a fast way to get started, without learning Python, Scala or R!

also You can find my other helpful Programming-posts
(if old file-links don't show activity, try copy-paste them to the address bar)

MediaInfo:

General
Complete name : 8. Reading from Disk\1. Reading from Disk.mp4
Format : MPEG-4
Format profile : Base Media
Codec ID : isom (isom/iso2/avc1/mp41)
File size : 138 MiB
Duration : 13 min 19 s
Overall bit rate : 1 448 kb/s
Writing application : Lavf58.12.100

Video
ID : 1
Format : AVC
Format/Info : Advanced Video Codec
Format profile : Main@L3.1
Format settings : CABAC / 4 Ref Frames
Format settings, CABAC : Yes
Format settings, RefFrames : 4 frames
Codec ID : avc1
Codec ID/Info : Advanced Video Coding
Duration : 13 min 19 s
Bit rate : 1 312 kb/s
Nominal bit rate : 3 000 kb/s
Width : 1 280 pixels
Height : 720 pixels
Display aspect ratio : 16:9
Frame rate mode : Constant
Frame rate : 30.000 FPS
Color space : YUV
Chroma subsampling : 4:2:0
Bit depth : 8 bits
Scan type : Progressive
Bits/(Pixel*Frame) : 0.047
Stream size : 125 MiB (91%)
Writing library : x264 core 148
Encoding settings : cabac=1 / ref=3 / deblock=1:-1:-1 / analyse=0x1:0x111 / me=umh / subme=6 / psy=1 / psy_rd=1.00:0.15 / mixed_ref=1 / me_range=16 / chroma_me=1 / trellis=1 / 8x8dct=0 / cqm=0 / deadzone=21,11 / fast_pskip=1 / chroma_qp_offset=-3 / threads=22 / lookahead_threads=3 / sliced_threads=0 / nr=0 / decimate=1 / interlaced=0 / bluray_compat=0 / constrained_intra=0 / bframes=3 / b_pyramid=2 / b_adapt=1 / b_bias=0 / direct=1 / weightb=1 / open_gop=0 / weightp=2 / keyint=60 / keyint_min=6 / scenecut=0 / intra_refresh=0 / rc_lookahead=60 / rc=cbr / mbtree=1 / bitrate=3000 / ratetol=1.0 / qcomp=0.60 / qpmin=0 / qpmax=69 / qpstep=4 / vbv_maxrate=3000 / vbv_bufsize=6000 / nal_hrd=none / filler=0 / ip_ratio=1.40 / aq=1:1.00

Audio
ID : 2
Format : AAC
Format/Info : Advanced Audio Codec
Format profile : LC
Codec ID : mp4a-40-2
Duration : 13 min 19 s
Bit rate mode : Constant
Bit rate : 128 kb/s
Channel(s) : 2 channels
Channel positions : Front: L R
Sampling rate : 44.1 kHz
Frame rate : 43.066 FPS (1024 SPF)
Compression mode : Lossy
Stream size : 12.2 MiB (9%)
Default : Yes
Alternate group : 1

Screenshots