Web Scraping with Beautiful Soup
Turn a real web page into clean Python data
Big Idea: Many sites show useful information as ordinary HTML but never offer a JSON API for it. Web scraping is the practice of downloading that HTML and pulling the pieces you want out of it in code. You already know how to fetch JSON with
requests; here you fetch HTML instead and hand it to Beautiful Soup, a library that turns the tangled markup into a tree you can search and read.
By the end you will be able to:
- fetch a page with
requestsand build a soup from it, - find elements with
find,find_alland CSS selectors, - extract text and attributes like
href, - scrape a multi-row table into a list of dicts,
- scrape responsibly: robots.txt, rate limits and a real User-Agent.
1. The two-step pipeline: fetch, then parse
Scraping is always two jobs. First you fetch the raw HTML text over the network -> that is the requests library you met in "Calling Web APIs with requests". Second you parse that text into a searchable structure -> that is Beautiful Soup. Keep the two steps clearly separated in your head; they fail for different reasons.
Install both libraries into your virtual environment first:
python -m pip install requests beautifulsoup4Then the smallest complete scraper looks like this:
import requests
from bs4 import BeautifulSoup
url = "https://example.com/books"
response = requests.get(url, timeout=10)
response.raise_for_status() # stop on 404 / 500 errors
soup = BeautifulSoup(response.text, "html.parser")
print(soup.title.text) # Book StoreNotice response.raise_for_status(): if the server answered 404 or 500, you want to stop now, not feed an error page into the parser. Passing timeout=10 means a hung server cannot freeze your program forever.