The essential points from this guide -- each one is explained in detail below.
requests plus BeautifulSoup covers most python web scraping jobs that do not need a full browser.
A missing or default User-Agent header is the most common reason a scraper gets blocked.
Proxies spread requests across many IPs, which matters once headers and delays alone stop working.
Store scraped data as CSV, JSON, or SQLite depending on how big the job is and who reads it next.
Wrap every request in error handling that tells transient failures apart from permanent ones.
Install Python 3.9 or newer before writing any scraper, since older versions miss features some scraping libraries expect. Check your version with python3 --version, then create a virtual environment so this project's packages stay separate from every other Python project on your machine.
python3 -m venv venv
source venv/bin/activate # Windows: venv\Scripts\activate
pip install requests beautifulsoup4 lxmlrequests handles the HTTP side of python web scraping, sending the actual request and returning the page. BeautifulSoup parses the returned HTML, and lxml is the fast parser backend BeautifulSoup uses under the hood. This three-package stack covers the large majority of python internet scraping jobs that pull data from ordinary, non-JavaScript pages.
For pages that load content after the initial request through JavaScript, this stack is not enough on its own. Playwright or Selenium render the page first, then hand the finished HTML to BeautifulSoup the same way. Start with requests and BeautifulSoup regardless, since most scraping targets do not need a full browser, and adding one only when a page actually requires it keeps your script faster and easier to run at scale.
A minimal python program to scrape website data needs three parts: a request, a parse step, and an extraction step. Here is a complete first script that pulls every product title from a page:
import requests
from bs4 import BeautifulSoup
url = 'https://example.com/products'
headers = {'User-Agent': 'Mozilla/5.0 (compatible; research-bot/1.0)'}
response = requests.get(url, headers=headers, timeout=30)
soup = BeautifulSoup(response.text, 'lxml')
titles = soup.find_all('h2', class_='product-title')
for title in titles:
print(title.get_text(strip=True))The requests.get() call downloads the raw HTML. BeautifulSoup then turns that HTML into a tree you can search with find_all(), matching on tag name and class the same way you would target an element in CSS. get_text(strip=True) pulls out just the visible text and trims extra whitespace around it.
Always set a timeout on the request. Without one, a slow or unresponsive server can hang your script indefinitely instead of failing and moving on. Check response.status_code before parsing too, since a 200 means success while a 403 or 429 usually means the site blocked the request, which the next section covers. Request, then parse, then extract: that is the core of nearly every way to scrape a website with Python, no matter how much the target site changes underneath it.
The single biggest reason a scraper gets blocked is a missing or default User-Agent header. The requests library sends 'python-requests/x.x' by default, and many sites block that string immediately since no real browser sends it. Set a realistic browser User-Agent on every request, as shown in the previous section, before troubleshooting anything else.
Rate limiting is the second most common block. Sending dozens of requests per second from one IP looks nothing like normal browsing. Add a short pause between requests with time.sleep(1) or a similar delay, and check the target's robots.txt file first to see which paths it asks automated tools to avoid.
Some sites run dedicated anti-bot services that check for more than headers and speed, including browser fingerprints and TLS handshake details that a plain requests call cannot reproduce. Older documentation sometimes calls this general problem python screen scraping, a term that predates the web and now covers pulling structured data out of any rendered output, not just web pages.
Status codes tell you which category you are dealing with. A 429 means you are being rate-limited and should slow down. A 403 often means the request was flagged outright. Repeated blocks from a single IP, even with correct headers and delays in place, usually mean the target needs requests spread across many different IP addresses instead of just one, which is what proxies solve directly.
Proxies solve the part of anti-bot protection that headers and delays cannot: spreading requests across many IP addresses instead of sending everything from one. Add a proxy to the same requests script by passing a proxies dictionary:
import requests
from bs4 import BeautifulSoup
proxy_url = 'http://USER:PASS@gw.knoxproxy.com:7000'
proxies = {'http': proxy_url, 'https': proxy_url}
response = requests.get('https://example.com/products', proxies=proxies, timeout=30)
soup = BeautifulSoup(response.text, 'lxml')Every request through a backconnect gateway like this one gets a different exit IP automatically, so you do not manage a proxy list yourself. For a script running on a schedule or hitting the same site repeatedly, rotating residential proxies (from $2.10/GB) keep each request looking like a different visitor, which matters most on sites with real anti-bot protection. Datacenter proxies (from $0.60/GB) cost less per GB and work fine on lighter targets that do not police IP type closely.
For a scraper that sends more than a handful of requests, put the proxies dictionary on a requests.Session() instead of passing it to every call. The session reuses the same configuration and connection pool across the whole loop, which is both faster to run and simpler to maintain. Check current KnoxProxy pricing before committing to a plan, since the right proxy type depends on how aggressively your specific target blocks repeat visitors.
A script that only prints results is fine for testing, but a real python site scraper needs to save what it finds. CSV works well for flat, table-shaped data and opens directly in a spreadsheet:
import csv
with open('products.csv', 'w', newline='', encoding='utf-8') as f:
writer = csv.writer(f)
writer.writerow(['title', 'price'])
for title, price in results:
writer.writerow([title, price])JSON fits better when your data has nested fields, like a product with a list of reviews attached. Python's built-in json module handles this with json.dump(results, f, indent=2). For a job that runs on a schedule and needs to check for duplicates before adding new rows, SQLite through Python's built-in sqlite3 module gives you a real database with no extra service to install or manage.
Whichever format you pick, write to a temporary file first and rename it once the scrape finishes successfully. This way a script that crashes halfway through never overwrites yesterday's complete, working data file with a half-finished one.
A scraper that stops at the first failed request is not ready for real use. Wrap each request in a try/except block, and tell transient failures, timeouts, connection resets, 429 and 503 responses, apart from permanent ones like a 404 that will never succeed on retry:
import time
import requests
def fetch(url, retries=3):
for attempt in range(retries):
try:
response = requests.get(url, headers=headers, proxies=proxies, timeout=30)
if response.status_code == 200:
return response
if response.status_code in (429, 503):
time.sleep(2 ** attempt)
continue
return None # permanent failure, do not retry
except requests.exceptions.RequestException:
time.sleep(2 ** attempt)
return NoneThe 2 ** attempt pattern doubles the wait between each retry, which gives a temporarily overloaded server or proxy time to recover instead of hammering it again immediately. Log every URL that ultimately fails to a separate file so you can review and re-run just those pages later, rather than rerunning the entire scrape from the start. For any python web scraper running on a schedule, this kind of handling is the difference between a job that silently loses data for weeks and one that fails loudly enough to get noticed and fixed.
Ready to put this into practice? See web scraping proxies
KnoxProxy Research Team · Technical Content
Network engineers and proxy infrastructure specialists with 10+ years in anti-bot systems, web scraping, and IP routing.
90.4M+ ethically sourced residential IPs across 195 countries. Instant activation, 14-day money-back guarantee.