Complete Python Web Scraping: Real Projects & Modern Tools

Learn BeautifulSoup, Selenium, Scrapy and Scrapy-Playwright. 4 Project-like Exercises + 4 Real Life Projects

What you'll learn

Scraping websites using Python

Python's most popular and effective web scraping libraries

Scrapy-Playwright

Web Automation with Selenium

Scrapy

Requests and Beautiful Soup

Using the right method according to the structure of the website

Reading and Analyzing HTML code

Saving scraped data

Downloading bulk images

Requirements

Just a PC with internet connection.

No programming experience is needed. I will teach you everything you need to know.

Fundamental Python knowledge is nice to have but not a must.

Description

In today's data-driven world, web scraping is a powerful tool that enables you to gather data from websites efficiently.I designed this course to be the most complete web scraping course on Udemy. It is practical and exercise-based, ensuring you learn by doing through exercises and real-life projects.We'll start with the basics on bookstoscrape and quotestoscrape (which are designed to be scraped) to help you grasp the fundamentals of web scraping. After learning the basics, we’ll dive deep into web scraping on real websites.If you're new to Python, don't worry, we've got an extra section covering Python fundamentals to get you ready for this course.What You'll Learn:Requests and BeautifulSoup:      Parse and extract data from HTML using eBay as an example.Selenium:      Automate browser interactions with real projects from IMDb.Scrapy:      Build scalable web scrapers with real-life examples from Flying Tiger and Yelp.Scrapy-Playwright:      Learn how to scrape dynamic websites with Scrapy by integrating Playwright.Why This Course?Hands-on Learning: The course is packed with exercises and real-life projects to help you apply what you learn immediately.Practical Approach: I will focus on teaching you practical skills that you can use in your own projects.Support for Beginners: An extra section on Python fundamentals ensures that even those new to programming can follow along and succeed.Join me in this journey to unlock the full potential of web scraping. With practical exercises and real-world examples, you'll be well-equipped to gather data from the web effectively. Let's get started!

Overview

Section 1: INTRODUCTION

Lecture 1 Course Introduction

Lecture 2 Section Introduction

Lecture 3 Course Material

Lecture 4 Checking If the Website is Static or Dynamic

Lecture 5 Deciding on the Method to be Used

Lecture 6 Reading and Analyzing HTML Code

Lecture 7 Installing Python and Pycharm (WINDOWS)

Lecture 8 Installing Python and Pycharm (LINUX)

Lecture 9 Installing Python and Pycharm (MACOS)

Lecture 10 Installing Jupyter Notebook on Windows, Linux and MacOS (OPTIONAL)

Lecture 11 About Real Life Examples…

Lecture 12 Course Syllabus

Section 2: BEAUTIFUL SOUP and REQUESTS 1 - BASICS

Lecture 13 Section Introduction

Lecture 14 Installing Libraries and Making the First Request

Lecture 15 Selectors of Beuatiful Soup

Lecture 16 Scraping 1 Book's Data

Lecture 17 Constructing the Outer Loop and Dealing with Pagination

Lecture 18 Constructing the Inner Loop and Scraping Every Book

Lecture 19 Saving the Data and Summary of the Project

Lecture 20 Downloading Images with Requests

Lecture 21 Using Proxies with Requests

Lecture 22 About Proxy Authentication

Section 3: BEAUTIFUL SOUP and REQUESTS 2 - EXERCISE 1 (QUOTES)

Lecture 23 Section Introduction

Lecture 24 Exercise 1

Lecture 25 Exercise 1 - Solution

Section 4: BEAUTIFUL SOUP and REQUESTS 3 - REAL LIFE EXAMPLE 1 (EBAY)

Lecture 26 Section Introduction

Lecture 27 Scraping 1 Laptop's Data, Usage of Full Headers and Cookies

Lecture 28 Pagination

Lecture 29 Saving the Data

Section 5: SELENIUM 1 - BASICS

Lecture 30 Section Introduction

Lecture 31 Installing Selenium and Chromedriver

Lecture 32 Creating Driver and Opening Browser

Lecture 33 CSS Selectors

Lecture 34 XPATH

Lecture 35 Login with Selenium

Lecture 36 Scraping the First Page

Lecture 37 Browsing Through Pages

Lecture 38 Infinite Scroll

Lecture 39 Waits

Lecture 40 Actions

Section 6: SELENIUM 2 - EXERCISE 2 (AUTHORS)

Lecture 41 Section Introduction

Lecture 42 Exercise 2

Lecture 43 Exercise 2 - Solutions

Section 7: SELENIUM 3 - REAL LIFE EXAMPLE 2 (IMDB)

Lecture 44 Section Introduction

Lecture 45 Creating Driver and Opening Browser

Lecture 46 Important Note

Lecture 47 Adjustment to the Next Video (Automation)

Lecture 48 Reaching the Page with Automation

Lecture 49 Scrolling to Load All Items

Lecture 50 Scraping the Data

Section 8: SELENIUM 4 - EXERCISE 3 (IMDB DIRECTORS)

Lecture 51 Section Introduction

Lecture 52 Important Note

Lecture 53 Exercise 3

Lecture 54 Important Note

Lecture 55 Exercise 3 - Solutions

Section 9: SCRAPY 1 - BASICS

Lecture 56 Section Introduction

Lecture 57 Installing Scrapy, Creating Scrapy Project and Spider

Lecture 58 Scrapy Shell and Scrapy's Selectors

Lecture 59 Constructing Parse Method

Lecture 60 Items

Lecture 61 Constructing Parse Books Method

Lecture 62 Pipelines

Lecture 63 Dropping Items with Pipelines

Lecture 64 Saving Data to Excel with Pipelines

Lecture 65 Note about SQLite

Lecture 66 Saving Data to SQLite Database with Pipelines

Lecture 67 Middlewares

Lecture 68 Crawler

Section 10: SCRAPY 2 - EXERCISE 4 (QUOTES)

Lecture 69 Section Introduction

Lecture 70 Exercise 4

Lecture 71 Exercise 4 - Soluitons

Section 11: SCRAPY 3 - REAL LIFE EXAMPLE 3 (FLYING TIGER)

Lecture 72 Section Introduction

Lecture 73 Creating Project and Spider

Lecture 74 Scraping the Data

Lecture 75 Adjustment on Previous Video - Log Files and Default Image URL

Lecture 76 Json Writer Pipeline

Lecture 77 Downloading Images

Section 12: SCRAPY 4 - REAL LIFE EXAMPLE 4 (YELP)

Lecture 78 Section Introduction

Lecture 79 Addition to the Next Video

Lecture 80 Real Life Crawler

Section 13: SCRAPY - PLAYWRIGHT

Lecture 81 Section Introduction

Lecture 82 Installing Virtual Box and Ubuntu for Windows

Lecture 83 Installing Scrapy - Playwright

Lecture 84 Adding Playwright Features to Scrapy Project

Lecture 85 Scraping Dynamic Data

Lecture 86 Playwright Page

Lecture 87 Clicking with Playwright

Lecture 88 Wait for Selector and Wait for Timeout

Section 14: EXTRA - QUICK PYTHON FUNDAMENTALS COURSE

Lecture 89 Section Introduction

Lecture 90 Python Basics: Variables, Data Types and Operators

Lecture 91 Data Structures

Lecture 92 Control Flow Statements (if, for, while)

Lecture 93 Functions

Lecture 94 Errors, Modules-Libraries and File Operations

Lecture 95 Strings

Lecture 96 Mini Project - ToDo List

Lecture 97 OOP - Introduction to Object Oriented Programming

Section 15: CONGRATULATIONS

Lecture 98 End of the Course

Total beginners who want to learn Web Scraping,Developers who intend to start Data Science from the essence of it