The essential points from this guide -- each one is explained in detail below.
BeautifulSoup parses HTML and XML using simple methods like find() and find_all().
Install it with pip install beautifulsoup4 lxml for the fastest, most reliable parser backend.
BeautifulSoup only reads static HTML -- JavaScript-rendered pages need Selenium or Puppeteer first.
Scrapy suits large crawl pipelines; BeautifulSoup fits smaller, one-off parsing jobs.
Proxies keep a BeautifulSoup script from getting rate-limited or blocked at scale.
Installing BeautifulSoup takes one pip command. Run pip install beautifulsoup4 in your terminal to add the library to your Python environment. You also need a parser, since BeautifulSoup does not read HTML by itself -- it hands that job to a parser backend.
Python ships with a built-in parser called html.parser, but lxml parses faster and handles broken or malformed HTML more reliably. Most guides recommend installing both packages together:
pip install beautifulsoup4 lxmlAfter installing, importing BeautifulSoup is a one-line statement. Tell it to use lxml as the parser when you build your soup object:
from bs4 import BeautifulSoup
import requests
response = requests.get('https://example.com')
soup = BeautifulSoup(response.text, 'lxml')If installing beautifulsoup on your system fails, confirm pip points to the same Python version you plan to run your script with. On systems running both Python 2 and 3, use pip3 install beautifulsoup4 lxml instead. Virtual environments (venv or conda) avoid this version conflict and are standard practice for any real scraping project.
An ImportError after installing usually means the package landed in a different Python environment than the one running your script. Check which interpreter pip is tied to with pip show beautifulsoup4, and compare it against the output of python --version. If you manage several projects, keep a separate virtual environment per project so one project's dependency versions never collide with another's.
Once you have a soup object, finding data means calling find() or find_all() with the tag name, class, or attribute you want to match. find() returns the first matching element; find_all() returns a list of every match on the page.
Here is a beautifulsoup example that pulls every product title from a page:
titles = soup.find_all('h2', class_='product-title')
for title in titles:
print(title.get_text(strip=True))You can also search by attribute instead of class. This beautifulsoup tutorial pattern locates a single element and reads its text:
price_tag = soup.find('span', attrs={'data-testid': 'price'})
price = price_tag.get_text(strip=True) if price_tag else NoneCSS selectors work too, through the select() method, which is often faster to write if you already know CSS. soup.select('div.card > span.price') returns every span.price element nested directly inside a div.card. Combining find_all() with a loop, plus select() for one-off lookups, covers most scraping jobs without extra libraries.
Always check for a missing element before reading it. find() returns None instead of raising an error when nothing matches, so calling .get_text() on that result directly crashes the script. Wrap each lookup in a simple if-check, or use the pattern price_tag.get_text(strip=True) if price_tag else None shown above, so one missing tag on one page does not stop the whole scrape. For navigating the tree itself -- moving from a matched tag to its parent, next sibling, or child elements -- BeautifulSoup exposes .parent, .next_sibling, and .children as plain attributes, no extra method calls needed.
BeautifulSoup and Scrapy solve different problems, even though both parse HTML in Python. BeautifulSoup is a parsing library: you fetch the page yourself with requests or httpx, then hand the HTML to BeautifulSoup to extract data. Scrapy is a full crawling framework: it manages the request queue, follows links, handles retries, and runs multiple spiders in parallel out of the box.
The scrapy vs beautifulsoup choice usually comes down to project size. A script that scrapes a handful of pages once a day works fine with requests plus BeautifulSoup -- it is quick to write and easy to debug. A project that needs to crawl thousands of pages, respect crawl delays, and export data on a schedule benefits from Scrapy's built-in scheduler, pipelines, and middleware system.
Many teams use both. They write a quick BeautifulSoup script to validate a scraping approach on a handful of pages, then port the working selectors into a Scrapy spider once the project needs to scale.
Scrapy also has a steeper learning curve. It introduces its own project structure, item pipelines, and settings file, which is overhead you do not want for a script that runs once and pulls data from twenty pages. BeautifulSoup has almost no learning curve beyond basic Python, which is why most tutorials teach it first before moving on to full frameworks.
BeautifulSoup only reads the HTML it is given -- it cannot execute JavaScript. If a page loads its content after the initial HTML request, through an API call or a client-side framework, BeautifulSoup sees an empty shell where that content should be.
Scraping python selenium setups solve this by opening a real or headless browser, letting the JavaScript run, and then reading the fully rendered page. Selenium drives Chrome or Firefox directly and works well on sites with heavy client-side rendering:
from selenium import webdriver
from bs4 import BeautifulSoup
driver = webdriver.Chrome()
driver.get('https://example.com')
soup = BeautifulSoup(driver.page_source, 'lxml')Puppeteer scraping follows the same idea in Node.js, driving headless Chrome to render the page before you extract data. Playwright is a newer alternative that supports Chrome, Firefox, and Safari's engine from one API. The pattern stays the same: render first with a browser automation tool, then parse the rendered HTML with BeautifulSoup or an equivalent parser.
One common mistake is calling driver.page_source too early, before the JavaScript finishes loading the data you want. Add an explicit wait -- Selenium's WebDriverWait combined with expected_conditions -- so the script pauses until a specific element appears, rather than guessing with a fixed time.sleep(). Running the browser in headless mode (webdriver.Chrome(options=headless_options)) is standard for scraping since you do not need to see the window, and it uses less memory when running many scrapes in parallel.
Scraping a website with BeautifulSoup at any real volume means sending many requests from the same IP address, which triggers rate limits or outright blocks. Routing those requests through a proxy spreads them across many IP addresses instead.
Passing a proxy to requests before handing the response to BeautifulSoup takes one extra dictionary:
import requests
from bs4 import BeautifulSoup
proxy_url = 'http://USER:PASS@gw.knoxproxy.com:7000'
proxies = {'http': proxy_url, 'https': proxy_url}
response = requests.get('https://example.com', proxies=proxies, timeout=30)
soup = BeautifulSoup(response.text, 'lxml')
titles = soup.find_all('h2', class_='product-title')This is the full beautiful soup for web scraping pattern used in production: fetch through a proxy, parse with BeautifulSoup, extract with find_all(). For a scraper that runs on a schedule or hits the same site repeatedly, rotating residential proxies (from $2.10/GB) keep each request looking like a different visitor. Datacenter proxies (from $0.60/GB) work well on sites with lighter bot detection and cost less per GB. Choose based on how aggressively your target blocks repeat visitors, not just on price.
For a script that sends more than a handful of requests, set the proxies dictionary on a requests.Session() instead of passing it to every call individually. The session reuses the same proxy configuration and connection pool across the whole loop, which is both faster and easier to maintain than repeating the proxies argument on every request. Wrap each request in a try/except block too -- a single proxy timeout or connection reset should not crash a scrape that is otherwise working through hundreds of pages.
Ready to put this into practice? See web scraping proxies
KnoxProxy Research Team · Technical Content
Network engineers and proxy infrastructure specialists with 10+ years in anti-bot systems, web scraping, and IP routing.
90.4M+ ethically sourced residential IPs across 195 countries. Instant activation, 14-day money-back guarantee.