BeautifulSoup parses HTML pulled through KnoxProxy proxies, letting a Beautiful Soup Python scraper rotate residential, datacenter, or mobile IPs.
Install both packages from PyPI in one command. BeautifulSoup parses HTML; requests fetches the page through your proxy first.
pip install beautifulsoup4 requestsUse your KnoxProxy dashboard username and password to build the proxy URL that requests will use for every fetch.
PROXY_USER = "your_username"
PROXY_PASS = "your_password"
proxy_url = f"http://{PROXY_USER}:{PROXY_PASS}@gw.knoxproxy.com:7000"Pass the proxy URL to requests, then hand the response text to BeautifulSoup for parsing. This two-step pattern is the core of every Beautiful Soup Python scraper.
import requests
from bs4 import BeautifulSoup
proxies = {"http": proxy_url, "https": proxy_url}
response = requests.get("https://example.com", proxies=proxies, timeout=30)
soup = BeautifulSoup(response.text, "html.parser")
print(soup.title.text)Target a specific country by adding a country flag to your username before the fetch runs.
# Country targeting via username flag
proxy_url = f"http://{PROXY_USER}-country-us:{PROXY_PASS}@gw.knoxproxy.com:7000"Each fetch gets a fresh IP by default. Add a session ID to your username for sticky sessions across a multi-page crawl.
# Sticky session (same IP for up to 30 min)
proxy_url = f"http://{PROXY_USER}-session-abc123:{PROXY_PASS}@gw.knoxproxy.com:7000"Extract data with standard BeautifulSoup selectors once the page has been fetched through the proxy.
titles = soup.find_all("h2")
for t in titles:
print(t.get_text(strip=True))"""KnoxProxy + BeautifulSoup -- complete working example."""
import requests
from bs4 import BeautifulSoup
PROXY_USER = "your_username"
PROXY_PASS = "your_password"
PROXY_HOST = "gw.knoxproxy.com"
PROXY_PORT = 7000
proxy_url = f"http://{PROXY_USER}:{PROXY_PASS}@{PROXY_HOST}:{PROXY_PORT}"
proxies = {"http": proxy_url, "https": proxy_url}
# Confirm the exit IP before scraping
ip_check = requests.get("https://httpbin.org/ip", proxies=proxies, timeout=30)
print(f"Exit IP: {ip_check.json()['origin']}")
# Fetch a page through the proxy, then parse it with BeautifulSoup
response = requests.get("https://example.com", proxies=proxies, timeout=30)
soup = BeautifulSoup(response.text, "html.parser")
print(f"Page title: {soup.title.text if soup.title else 'none'}")
# Sticky session for a multi-page crawl (same IP across every request)
session_proxy = f"http://{PROXY_USER}-session-mysession1:{PROXY_PASS}@{PROXY_HOST}:{PROXY_PORT}"
session = requests.Session()
session.proxies = {"http": session_proxy, "https": session_proxy}
for page in range(1, 4):
r = session.get(f"https://example.com/page/{page}", timeout=30)
page_soup = BeautifulSoup(r.text, "html.parser")
links = [a.get("href") for a in page_soup.find_all("a", href=True)]
print(f"Page {page}: found {len(links)} links")Default: a fresh IP on every requests.get() call, since BeautifulSoup itself has no network layer to configure. For sticky sessions across a multi-page crawl, append `-session-{id}` to your username. Sessions hold the same IP for up to 30 minutes on residential, 60 minutes on mobile.
| Problem | Fix |
|---|---|
| ModuleNotFoundError: No module named 'bs4' | Run pip install beautifulsoup4 in the same environment or virtualenv you use to run the script, then confirm with pip show beautifulsoup4. |
| 407 Proxy Authentication Required | Check credentials in your KnoxProxy dashboard and confirm the proxy_url string matches exactly, including special characters. |
| AttributeError: 'NoneType' object has no attribute 'text' | Print response.status_code and the first 200 characters of response.text before parsing, to confirm the page fetched correctly. |
| Parsed page looks empty or incomplete | Fetch the page with Selenium or Playwright through the same KnoxProxy connection, then pass the rendered HTML to BeautifulSoup. |
| ConnectTimeout after 30s | Increase the timeout to 60 seconds and add retry logic around the requests.get() call. |
USER-country-de-city-berlin-session-profile07Order matters -- geo flags before the session flag. The session name is free text; use the profile ID so the mapping is self-documenting. Password stays as issued; no flags belong there. HTTP on :7000, SOCKS5 on :7001, same credentials.
BeautifulSoup parses HTML and XML documents into a searchable tree, so a Python script can pull out titles, links, and text by tag or CSS selector. It does not fetch pages on its own -- pair it with requests to download HTML through a proxy first, then parse the response.
Install BeautifulSoup with pip install beautifulsoup4, not pip install beautifulsoup, which installs an outdated Python 2 package from 2012. Add requests in the same line with pip install beautifulsoup4 requests, since BeautifulSoup needs a separate library to fetch pages before it can parse them.
No. BeautifulSoup only parses HTML and has no network code, so it cannot connect to a proxy on its own. Configure the proxy in requests, urllib, or another HTTP client, fetch the page through KnoxProxy, then pass the response text to BeautifulSoup for parsing.
Both refer to the same library. The PyPI package name is beautifulsoup4 and the import line is from bs4 import BeautifulSoup. The misspelling beutiful soup shows up often in search, but the correct spelling in code and documentation is BeautifulSoup, written as one word.
BeautifulSoup is a parsing library paired with requests for smaller scripts, while Scrapy is a full scraping framework with built-in scheduling, retries, and pipelines. Pick BeautifulSoup for quick, targeted extraction jobs, and Scrapy for large, ongoing crawls that span many pages and domains.
No. BeautifulSoup only reads the raw HTML a server returns, so content added by JavaScript after the page loads will not appear. Fetch JavaScript-heavy pages with Selenium or Playwright through your KnoxProxy connection first, then pass the rendered HTML to BeautifulSoup for parsing.
html.parser ships with Python and needs no extra install, making it the simplest default choice for most scripts. For faster parsing or messier HTML, install lxml with pip install lxml and pass lxml as the parser name. Both options work the same way with proxied requests.
Rotating residential proxies -- 3 minutes setup, instant activation, 14-day money-back guarantee.