Elasticsearch 8 and the Elastic Stack: In Depth and Hands On

Complete Elastic search tutorial - search, analyze, and visualize big data with Elasticsearch, Kibana, Logstash, & Beats

What you'll learn
Install and configure Elasticsearch 7 on a cluster
Create search indices and mappings
Search full-text and structured data in several different ways
Import data into Elasticsearch using various techniques
Integrate Elasticsearch with other systems, such as Spark, Kafka, relational databases, S3, and more
Aggregate structured data using buckets and metrics
Use Logstash and the "ELK stack" to import streaming log data into Elasticsearch
Use Filebeats and the Elastic Stack to import streaming data at scale
Analyze and visualize data in Elasticsearch using Kibana
Manage operations on production Elasticsearch clusters
Use cloud-based solutions including Amazon's Elasticsearch Service and Elastic Cloud
Requirements
You need access to a Windows, Mac, or Ubuntu PC with 20GB of free disk space
You should have some familiarity with web services and REST
Some familiarity with Linux will be helpful
Exposure to JSON-formatted data will help
Description
Elasticsearch and  the Elastic Stack are important tools for managing massive data. You need to know the problems it solves and how it works to design the best systems, and be the most valuable engineer you can be.Elasticsearch 8 is a powerful tool for analyzing big data sets in a matter of milliseconds! It’s increasingly popular technology for powering search and analytics on big websites, and a valuable skill to have in today's job market. This course covers it all, from installation to operations. Learn how to use Elasticsearch 8 and implement it in your work within the next few days.We've teamed up with Coralogix to co-produce the most comprehensive Elastic Stack course we've seen— with over 100 lectures including 15 hours of video.We'll show you how to set up search indices on an Elasticsearch 8 cluster (if you need Elasticsearch 6 or 7 - we have other courses on that), and query that data in many different ways. Fuzzy searches, partial matches, search-as-you-type, pagination, sorting - you name it. And it's not just theory, every lesson has hands-on examples where you'll practice each skill using a virtual machine running Elasticsearch on your own PC.We'll explore what's new in Elasticsearch 8 and illustrate all the new syntax requirements of Elasticsearch commands, now that things deprecated through the Elasticsearch 7 have been removed. Almost every hands-on activity has been re-recorded to ensure compatibility with Elasticsearch 8.We cover, in depth, the often-overlooked problem of importing data into an Elasticsearch index. Whether it's via raw RESTful queries, scripts using Elasticsearch API's, or integration with other "big data" systems like Spark and Kafka - you'll see many ways to get Elasticsearch started from large, existing data sets at scale. We'll also stream data into Elasticsearch using Logstash and Filebeat - commonly referred to as the "ELK Stack" (Elasticsearch / Logstash / Kibana) or the "Elastic Stack".Elasticsearch isn't just for search anymore - it has powerful aggregation capabilities for structured data, which allows you to glean new insights from your indexed data. We'll bucket and analyze data using Elasticsearch, and visualize it using the Elastic Stack's web UI, Kibana and Kibana Lens.You'll learn how to manage operations on your Elastic Stack, monitoring your cluster's health, and how to perform operational tasks like scaling up your cluster, and doing rolling restarts. We'll also spin up Elasticsearch clusters in the cloud using Amazon Opensearch Service and the Elastic Cloud.Elasticsearch is positioning itself to be a much faster alternative to Hadoop, Spark, and Flink for many common data analysis requirements.  It's an important tool to understand, and it's easy to use! Dive in with me and I'll show you what it's all about.

Overview

Section 1: Installing and Understanding Elasticsearch

Lecture 1 Udemy 101: Getting the Most From This Course

Lecture 2 Section 1 Intro

Lecture 3 Installing Elasticsearch [Step by Step]

Lecture 4 Elasticsearch Overview

Lecture 5 Intro to HTTP and RESTful API's

Lecture 6 Elasticsearch Basics: Logical Concepts

Lecture 7 Term Frequency / Inverse Document Frequency (TF/IDF)

Lecture 8 Using Elasticsearch

Lecture 9 What's New in Elasticsearch 8

Lecture 10 How Elasticsearch Scales

Lecture 11 Quiz: Elasticsearch Concepts and Architecture

Lecture 12 Section 1 Wrapup

Section 2: Mapping and Indexing Data

Lecture 13 Section 2 Intro

Lecture 14 Connecting to your Cluster

Lecture 15 Note: alternate download location for the MovieLens data set

Lecture 16 Introducing the MovieLens Data Set

Lecture 17 Analyzers

Lecture 18 A note on entering CURL commands.

Lecture 19 Import a Single Movie via JSON / REST

Lecture 20 Insert Many Movies at Once with the Bulk API

Lecture 21 Updating Data in Elasticsearch

Lecture 22 Deleting Data in Elasticsearch

Lecture 23 [Exercise] Insert, Update and Delete a Movie

Lecture 24 Dealing with Concurrency

Lecture 25 Using Analyzers and Tokenizers

Lecture 26 Data Modeling and Parent/Child Relationships, Part 1

Lecture 27 Data Modeling and Parent/Child Relationships, Part 2

Lecture 28 Flattened Datatype

Lecture 29 Dealing with Mapping Exceptions

Lecture 30 Section 2 Wrapup

Section 3: Searching with Elasticsearch

Lecture 31 Section 3 Intro

Lecture 32 "Query Lite" interface

Lecture 33 JSON Search In-Depth

Lecture 34 Phrase Matching

Lecture 35 [Exercise] Querying in Different Ways

Lecture 36 Pagination

Lecture 37 Sorting

Lecture 38 More with Filters

Lecture 39 [Exercise] Using Filters

Lecture 40 Fuzzy Queries

Lecture 41 Partial Matching

Lecture 42 Query-time Search As You Type

Lecture 43 N-Grams, Part 1

Lecture 44 N-Grams, Part 2

Lecture 45 "Search as you Type" Field Type

Lecture 46 Section 3 Wrapup

Section 4: Importing Data into your Index - Big or Small

Lecture 47 Section 4 Intro

Lecture 48 Importing Data with a Script

Lecture 49 Importing with Client Libraries

Lecture 50 [Exercise] Importing with a Script

Lecture 51 Introducing Logstash

Lecture 52 Installing Logstash

Lecture 53 Running Logstash

Lecture 54 Logstash and MySQL, Part 1

Lecture 55 Logstash and MySQL, Part 2

Lecture 56 Importing CSV Data with Logstash

Lecture 57 Importing JSON Data with Logstash

Lecture 58 Logstash and S3

Lecture 59 Parsing and Filtering Logstash with Grok

Lecture 60 Logstash Grok Examples for Common Log Formats

Lecture 61 Logstash Input Plugins, Part 1: Heartbeat

Lecture 62 Logstash Input Plugins, Part 2: Generator Input and Dead Letter Queue

Lecture 63 Logstash Input Plugins, Part 3: HTTP Poller

Lecture 64 Logstash Input Plugins, Part 4: Twitter

Lecture 65 Syslog with Logstash Deep Dive

Lecture 66 Elasticsearch and Kafka, Part 1

Lecture 67 Elasticsearch and Kafka, Part 2

Lecture 68 Elasticsearch and Apache Spark, Part 1

Lecture 69 Elasticsearch and Apache Spark, Part 2

Lecture 70 [Exercise] Importing Data with Spark

Lecture 71 Section 4 Wrapup

Section 5: Aggregation

Lecture 72 Section 5 Intro

Lecture 73 Aggregations, Buckets, and Metrics

Lecture 74 Histograms

Lecture 75 Time Series

Lecture 76 [Exercise] Generating Histogram Data

Lecture 77 Nested Aggregations, Part 1

Lecture 78 Nested Aggregations, Part 2

Lecture 79 Section 5 Wrapup

Section 6: Using Kibana

Lecture 80 Section 6 Intro

Lecture 81 Installing Kibana

Lecture 82 Playing with Kibana

Lecture 83 [Exercise] Exploring Data with Kibana

Lecture 84 Kibana Lens

Lecture 85 Kibana Management

Lecture 86 Elasticsearch SQL

Lecture 87 Using Kibana Canvas

Lecture 88 Elasticsearch and Apache Hadoop

Lecture 89 Section 6 Wrapup

Section 7: Analyzing Log Data with the Elastic Stack

Lecture 90 Section 7 Intro

Lecture 91 Data Frame Transforms

Lecture 92 FileBeat and the Elastic Stack Architecture

Lecture 93 X-Pack Security

Lecture 94 Installing FileBeat

Lecture 95 Analyzing Logs with Kibana Dashboards

Lecture 96 [Exercise] Log analysis with Kibana

Lecture 97 Section 7 Wrapup

Section 8: Elasticsearch Operations

Lecture 98 Section 8 Intro

Lecture 99 Choosing the Right Number of Shards

Lecture 100 Adding Indices as a Scaling Strategy

Lecture 101 Index Alias Rotation

Lecture 102 Index Lifecycle Management

Lecture 103 Choosing your Cluster's Hardware

Lecture 104 Heap Sizing

Lecture 105 Monitoring

Lecture 106 Troubleshooting Common Issues

Lecture 107 Failover in Action, Part 1

Lecture 108 Index Design Changes (Grouping, Splitting, and Shrinking Indices)

Lecture 109 Snapshots

Lecture 110 Snapshot Lifecycle Management

Lecture 111 Rolling Restarts

Lecture 112 Uptime Monitoring with Heartbeat

Lecture 113 Section 8 Wrapup

Section 9: Elasticsearch in the Cloud

Lecture 114 Section 9 Intro

Lecture 115 Amazon Opensearch Service, Part 1

Lecture 116 Amazon Opensearch Service, Part 2

Lecture 117 The Elastic Cloud

Lecture 118 Section 9 Wrapup

Section 10: You Made It!

Lecture 119 Wrapping Up

Lecture 120 Bonus Lecture: More Courses to Explore!

Any technologist tasked with fast, scalable searching and analysis of big data sets.