Web Scraping With Python: Beautifulsoup, Requests & Selenium

Web Scraping and Crawling with Python: Beautiful Soup, Requests & Selenium

What you'll learn
Python Refresher: Review of Data Structures, Conditionals, File Handling
How Websites are Hosted on Servers; Basic Calls to Server (GET, POST Methods)
Web Scraping with Python Beautiful Soup and Requests
Using Selenium to handle JavaScript and AJAX
Diverse Web Scraping Exercises
Source codes (*.py files) for all Exercises can be downloaded
Q&A board to send your questions and get them answered quickly
Requirements
Some prior programming experience in Python (e.g. Data Structures and OOP) will help. The course includes a full Python refresher section.
Complete beginners may wish to take a beginner Python course first, and then transition to this course afterwards.
This course adopts a step-by-step approach and requires you to open a Python editor, download available *.py code files, and start applying the provided examples and exercises.
Python 3: Codes of this course are tested on Python 3. It is up to you to adapt them if you want to run them in Python 2.
Description
Web Scraping (also termed Screen Scraping, Web Data Extraction, Web Harvesting, etc.) is a technique for extracting large amounts of data from websites  and save the the extracted data to a local file or to a database.
In this course, you will learn how to perform web scraping using Python 3 and the Beautiful Soup, a free open-source library written in Python for parsing HTML.
We will use lxml, which is an extensive library for parsing XML and HTML documents very quickly; it can even handle messed up tags. We will also be using the Requests module instead of the already built-in urllib2 module due to improvements in speed and readability.
Finally, we will use Selenium alongside Beautiful Soup to crawl AJAX & JavaScript driven pages.
The course cover the following topics: accessing web pages programmatically; scraping web pages to extract the required data using Beautiful Soup to parse web pages; interacting with web pages to do different things with them programmatically; and using Selenium for web scraping and when we need it.
By the end of this course, you will be able to understand how websites and servers function, diverse data extraction techniques, and methods of handling and organizing data.
This Web Scraping course covers the following topics:
Review of data structures (Lists, Dictionaries, Tuples, File Handling)How websites are hosted on serversCalls to the server (GET, POST methods)Review of HTML and CSSRequests Module and BeautifulSoup Module overviewParsing HTML using BeautifulSoupFiltering elements using BeautifulSoup and navigating the Parse TreeJavaScript and AJAX overviewSelenium and the need for itSelecting elements using Selenium CSS selectors XPath selectors Navigating pages using Selenium Practical Projects



Overview

Section 1: Web Scraping Course Overview

Lecture 1 Web Scraping Course Overview

Section 2: Python Refresher: Data Structures (Optional)

Lecture 2 Lists

Lecture 3 Dictionaries

Lecture 4 Tuples

Lecture 5 List Comprehensions - Part 1

Lecture 6 List Comprehensions - Part 2

Lecture 7 Inline - if else and List Comprehensions

Lecture 8 Installing xlrd and XlsxWriter​ to Read/Write to Excel Files

Lecture 9 Wrting to Excel Files

Lecture 10 Reading from Excel Files

Lecture 11 Python Editor & Other Software

Lecture 12 Exercise #1: YOU: Web Scraping Expert

Section 3: How Servers Work

Lecture 13 How Websites are Hosted

Lecture 14 HTML Revision

Section 4: BeautifulSoup Warm-up Exercise

Lecture 15 BeautifulSoup Solved Exercise

Section 5: Installing Required Python Packages

Lecture 16 Installing Required Python Packages

Section 6: Introduction to Requests Python Library

Lecture 17 Requests Get Method

Lecture 18 User Agent

Lecture 19 Installing fake_useragent Package

Section 7: Introduction to Beautiful Soup Python Library

Lecture 20 Web Scraping with Beautiful Soup - Overview

Lecture 21 Web Scraping with Beautiful Soup - Overview P.2

Lecture 22 Accessing Tags

Lecture 23 Navigable Strings

Section 8: Navigating with Beautiful Soup - Going Down

Lecture 24 Navigating through Tag Names

Lecture 25 Contents and Children Methods

Lecture 26 Descendants Method

Section 9: Navigating with Beautiful Soup - Going Up

Lecture 27 Parent Method

Lecture 28 Parents Method

Section 10: Navigating with Beautiful Soup - Going Sideways

Lecture 29 next_sibling

Lecture 30 previous_sibling

Lecture 31 next_siblings & previous_siblings

Section 11: Regular Expressions with Python

Lecture 32 Metacharacters Overview

Lecture 33 Compile Function and Character Class

Lecture 34 Special Sequences

Lecture 35 * Repeating Things

Lecture 36 + Repeating Things

Lecture 37 ? and {m,n} Repeating Things

Lecture 38 Metacharacters part2

Section 12: Searching the Parse Tree Using Beautiful Soup

Lecture 39 Introduction to Searching with BeautifulSoup

Lecture 40 find_all Function

Lecture 41 find_all More Parameters

Lecture 42 find Function

Section 13: Project 1: Scraping CustomerReports Website

Lecture 43 Web Scraping CustomerReports - part 1

Lecture 44 Web Scraping CustomerReports - part 2

Section 14: Project 2: Web Scraping CodingBat Website with Beautiful Soup

Lecture 45 Project 2 Description

Lecture 46 Web Scraping CodingBat - part 1

Lecture 47 Web Scraping CodingBat - part 2

Lecture 48 Web Scraping CodingBat - part 3

Section 15: Using Selenium to Handle AJAX & JavaScript Driven Web Pages

Lecture 49 JavaScript, AJAX and Selenium intro

Lecture 50 Installing Selenium

Lecture 51 Installing ChromeDriver

Lecture 52 Introduction to Selenium

Lecture 53 Searching Elements and Inputting Data

Lecture 54 Clicking Elements

Lecture 55 XPath Introduction

Lecture 56 XPath Examples

Section 16: Project 3: Web Scraping Your Instagram Account

Lecture 57 Project 3 Description

Lecture 58 Logging in to Instagram

Lecture 59 Settings Tab!

Lecture 60 Opening Target Profile (NEW)

Lecture 61 Scrolling Down v.1 (NEW)

Lecture 62 Scrolling Down v.2 (NEW)

Lecture 63 Exception Handling (NEW)

Lecture 64 Making Folders (NEW)

Lecture 65 Downloading Images v.1 (NEW)

Lecture 66 Downloading Images v.2 (NEW)

Lecture 67 Downloading Captions (NEW)

Lecture 68 Writing Captions to Excel File (NEW)

Lecture 69 Instagram Final Code - Updated: 2018-06-22

Section 17: Web Scraping Best Practices

Lecture 70 Web Scraping Best Practices

Section 18: Bonus: Data Extraction with APIs

Lecture 71 Data Extraction with APIs (Free Tutorial)

Section 19: Bonus: Scrapy: Powerful Web Scraping and Crawling Framework in Python

Lecture 72 Coupon for "Scrapy: Powerful Web Scraping & Crawling with Python" Course

Those who want to learn how to use Python for web scraping and data extraction.