Web Scraping Tutorial For Beginners With Scrapy & Python

Learn and Master Web Scraping using Scrapy Framework and this Step-By-Step & In-Depth Guide

What you'll learn
Define the Steps Involved in Web Scraping and Creating Web Crawlers
Install and Setup Scrapy in Windows, Mac OS, Ubuntu (Linux) & Anaconda Environments
Send Request to a URL to Scrape Websites Using Scrapy Spider
Get the HTML Response From URL and Parse it for Web Scraping
Select Desired Data From Websites Using Scrapy Selector, CSS Selectors & XPath
Scrapy Crawl Spiders to Get Data From Websites And Extract it to JSON, CSV, XLSX ( Excel ) and XML Files
Use Scrapy Shell Commands to Test & Verify CSS Selectors or XPath
Export and Save Scraped Data to Online Databases Like MonogoDB Using Scrapy Item Pipelines
Define Scrapy Items to Organize Scraped Data And Load Items Using Scrapy Itemloaders with Input & Output Processors
Scrape Data From Multiple Web Pages Using Scrapy Pagination And Extract Data From HTML Tables
Login Into Websites Using Scrapy FormRequest With CSRF Tokens
Scrape Dynamic/JavaScript Rendered Websites Using Scrapy-Playwright And Interact With Web Elements, Take Screenshot of Websites or Save as PDF
Identify API Calls From a Website and Scrape Data From API Using Scrapy Request
Requirements
Python Programming
HTML Basics (+point)
Description
Web scraping is the process of scraping websites and extracting desired data from the same, and in this course, you'll learn and master web scraping using python and scrapy with a step-by-step and in-depth guide.A Step-By-Step GuideAssuming that you don't know anything about web scraping, scrapy python web scraping,  or even web scraping meaning -  we will start from the complete basics. In the first section, you'll learn about the web scraping process step-by-step (with infographics - no code), how to scrape data from websites and how to use scrapy for the same (i.e. scrapy meaning).After getting the basics clear and having an idea of how web scraping works, we will start web scraping using python & scrapy framework! Again, we'll move step-by-step and perform each step learned in the basics with bite-sized lessons. We'll take it slow so that it's easier for you to understand each and every step involved in scraping and extracting data from websites.Web Scraping & Scrapy EssentialsHaving built an actual web scraper, you'll get an idea of how web scraping works firsthand. Now it's crucial to cover the essential concepts of web scraping and scrapy, which we will do next.CSS Selectors to select web elementsXPath to select web elementsScrapy Shell to test & verify selectorsItems to organize extracted dataLoad Items with ItemLoaders with input & output ProcessorsExport data to JSON, CSV, XLSX (Excel) & XML file formatsSave extracted data to online databases like MongoDB using ItemPipelinesMaster Web Scraping In-DepthLearning how to scrape websites and the essentials already makes you a complete web scraper but, we'll take this even further and learn the advanced web scraping techniques to become an expert!Follow links in a webpage to another pageCrawl multiple pages and extract data i.e. PaginationScrape data using Regular Expressions (RegEx)Extract Data From HTML TablesLogin Into Websites Using Scrapy FormRequestBypass CSRF protected Login formsScrape Dynamic or JavaScript Rendered Websites using Scrapy PlaywrightInteract with web elements like fill forms, click buttons, etc.Handle Infinite Scroll websitesWait For Elements when contents/data takes time to loadTake Screenshot of websitesSave websites as PDFIdentify API calls from websites and scrape data from APIsUse middleware in a scrapy projectConfigure settings in a scrapy projectUse and Rotate User-Agents & ProxiesWeb scraping Best PracticesReal-World ProjectsAfter master web scraping, we need projects to get started! That's why you'll perform three projects as well:Champions League Table [ ESPN ]Product Tracker [ Amazon ]Scraper Application [ GUI ]Join us in this in-depth course where you'll learn about web scraping from scratch and master the process of extracting data from websites step-by-step. Check out the preview lessons to get started and learn how web scraping works! See you there~

Overview

Section 1: Introduction

Lecture 1 What is Web Scraping?

Lecture 2 How Web Scraping Works?

Lecture 3 Web Scraping With Scrapy

Section 2: Scrapy Installation

Lecture 4 Scrapy Installation for Windows

Lecture 5 Scrapy Installation for Ubuntu (Linux)

Lecture 6 Scrapy Installation for Mac

Lecture 7 Scrapy Installation for Anaconda

Lecture 8 Creating Scrapy Project

Lecture 9 Project Walkthrough

Section 3: Scrapy Spider

Lecture 10 Creating Spider

Lecture 11 Sending Request

Lecture 12 Getting the Response

Lecture 13 Scrapy CSS Selector

Lecture 14 Selecting All The Data

Lecture 15 Extracting Data

Lecture 16 Spider Overview

Section 4: CSS Selectors

Lecture 17 CSS Selectors v/s XPath : How to Select Web Elements?

Lecture 18 Tagname, Class and Id Selectors

Lecture 19 Attribute Selectors

Lecture 20 CSS Selectors Cheat Sheet

Section 5: XPath

Lecture 21 XPath Expressions

Lecture 22 XPath Attribute Selectors

Lecture 23 XPath text( ) Function

Lecture 24 XPath Cheat Sheet

Section 6: Scrapy Shell

Lecture 25 What is the Scrapy Shell and How to Use it?

Lecture 26 fetch( ) Response

Lecture 27 Shell Configuration

Section 7: Scrapy Items

Lecture 28 Structuring Data Into Scrapy Item

Lecture 29 Using Item in Spiders

Lecture 30 Define Input and Output Processors For Item Fields

Lecture 31 Loading Items with Scrapy ItemLoaders

Lecture 32 Items, Processors & ItemLoaders Overview

Section 8: Exporting Data

Lecture 33 Output Extracted Data In JSON, CSV & XML Format

Lecture 34 Overwrite Previous Output

Lecture 35 Appending Data to Previous Output

Section 9: Scrapy Item Pipeline

Lecture 36 How to use Scrapy Item Pipelines?

Lecture 37 Saving Data Locally to Excel ( XLSX ) Files

Lecture 38 Enable Item Pipelines in Settings

Lecture 39 MongoDB (Account) Setup

Lecture 40 Saving Data To MonogoDB

Section 10: Pagination

Lecture 41 Extracting Links From href Attributes

Lecture 42 Send Request to the Next Page

Lecture 43 start_requests( ) method

Section 11: Following Links

Lecture 44 How to Follow Links?

Lecture 45 How to Select Data Using Regular Expressions With Scrapy

Lecture 46 Setting Up Custom Callback Function

Lecture 47 Parse Product Details Page

Section 12: Scraping Tables

Lecture 48 HTML Tables

Lecture 49 Selecting Tables Data

Lecture 50 Extract Data From HTML Tables

Section 13: Logging Into Websites

Lecture 51 Data Hidden With Logging Forms

Lecture 52 Inspecting HTML Forms and Website Activity With Dev Tools

Lecture 53 Logging Into Websites With FormRequest

Lecture 54 CSRF Protected Login Forms

Lecture 55 Extract CSRF Values From Forms

Section 14: Scraping JavaScript Rendered Websites

Lecture 56 What are JavaScript Rendered/Dynamic Websites?

Lecture 57 scrapy-playwright Installation

Lecture 58 Setting Up Playwright in Scrapy Project

Lecture 59 Using Playwright To Render Websites

Lecture 60 Scraping Data From Dynamic Websites

Section 15: Scrapy Playwright

Lecture 61 Playwright Overview

Lecture 62 Playwright Page Object

Lecture 63 Logging In With Playwright

Lecture 64 Dynamic Websites With Loading Screens

Lecture 65 Wait For Selector/Elements Using Page Couroutines

Lecture 66 Dynamic Websites With Infinite Scroll

Lecture 67 Taking Screenshot of Websites

Lecture 68 Rendering Websites To PDF

Section 16: API Endpoints

Lecture 69 Identifying API Calls

Lecture 70 Requesting Data From API

Lecture 71 Extracting Data From API

Section 17: Settings

Lecture 72 Scrapy Project Settings

Lecture 73 Robots Text

Lecture 74 Middleware

Lecture 75 Autothrottle Extension

Section 18: User Agents & Proxies

Lecture 76 What are User Agents?

Lecture 77 User Agents With Scrapy

Lecture 78 What are Proxies?

Lecture 79 Proxies With Scrapy

Section 19: Tips & Tricks

Lecture 80 Spider Arguments

Lecture 81 Standalone Spiders

Lecture 82 Scrapy Shell With bpython

Lecture 83 Scrapy get vs extract method

Lecture 84 Logging

Section 20: Project #1: Champions League Table From ESPN.com

Lecture 85 Overview

Lecture 86 Website Visual Inspection

Lecture 87 Finding The Selectors

Lecture 88 Building The Spider: Extract Teams Data

Lecture 89 Building The Spider: Extract Teams Details

Section 21: Project #2: Amazon Product Rank

Lecture 90 Overview

Lecture 91 Scraper Visualization

Lecture 92 Finding The Selectors

Lecture 93 Building The Spider

Section 22: Project #3: Extending Scraper With GUI

Lecture 94 Scraper Application

Lecture 95 Building The GUI (Application Interface)

Lecture 96 Running the Spider From the Application

Beginner Python Developers Who Wants to Master Web Scraping,Freelancer Web Scrapers Looking To Polish Their Skills