Python Requests is the top Python HTTP client. Install BeautifulSoup with pip to parse pages, then route both through a KnoxProxy proxy in two lines.
Install the requests library from PyPI.
pip install requestsUse your KnoxProxy dashboard username and password in the proxy URL.
PROXY_USER = "your_username"
PROXY_PASS = "your_password"
proxy_url = f"http://{PROXY_USER}:{PROXY_PASS}@gw.knoxproxy.com:7000"Pass the proxy URL as a dict to the proxies parameter.
import requests
proxies = {
"http": proxy_url,
"https": proxy_url,
}
response = requests.get("https://httpbin.org/ip", proxies=proxies)
print(response.json())Target a specific country by adding it to your username string.
# Country targeting via username flag
proxy_url = f"http://{PROXY_USER}-country-us:{PROXY_PASS}@gw.knoxproxy.com:7000"
# Or via header
response = requests.get("https://httpbin.org/ip",
proxies=proxies,
headers={"X-KnoxProxy-Country": "us"})Each request gets a fresh IP by default. Use session IDs for sticky sessions.
# Sticky session (same IP for up to 30 min)
proxy_url = f"http://{PROXY_USER}-session-abc123:{PROXY_PASS}@gw.knoxproxy.com:7000"Verify the proxy is working by checking your exit IP.
response = requests.get("https://httpbin.org/ip", proxies=proxies, timeout=30)
print(f"Exit IP: {response.json()['origin']}")
print(f"Status: {response.status_code}")BeautifulSoup performs data parsing on the HTML your requests proxy setup already fetched.
pip install beautifulsoup4 lxml
from bs4 import BeautifulSoup
response = requests.get("https://example.com", proxies=proxies, timeout=30)
soup = BeautifulSoup(response.text, "lxml")
titles = soup.find_all("h2")
for title in titles:
print(title.get_text())"""KnoxProxy + Python requests -- complete working example."""
import requests
PROXY_USER = "your_username"
PROXY_PASS = "your_password"
PROXY_HOST = "gw.knoxproxy.com"
PROXY_PORT = 7000
proxy_url = f"http://{PROXY_USER}:{PROXY_PASS}@{PROXY_HOST}:{PROXY_PORT}"
proxies = {"http": proxy_url, "https": proxy_url}
# Basic request with rotation (new IP each time)
response = requests.get(
"https://httpbin.org/ip",
proxies=proxies,
timeout=30,
)
print(f"Exit IP: {response.json()['origin']}")
# Country-targeted request
us_proxy = f"http://{PROXY_USER}-country-us:{PROXY_PASS}@{PROXY_HOST}:{PROXY_PORT}"
response = requests.get(
"https://httpbin.org/ip",
proxies={"http": us_proxy, "https": us_proxy},
timeout=30,
)
print(f"US IP: {response.json()['origin']}")
# Sticky session (same IP for multiple requests)
session_proxy = f"http://{PROXY_USER}-session-mysession1:{PROXY_PASS}@{PROXY_HOST}:{PROXY_PORT}"
session = requests.Session()
session.proxies = {"http": session_proxy, "https": session_proxy}
for i in range(3):
r = session.get("https://httpbin.org/ip", timeout=30)
print(f"Request {i+1}: {r.json()['origin']}")Default: new IP per request. For sticky sessions, append `-session-{id}` to your username. Sessions last up to 30 minutes for residential, 60 minutes for mobile.
| Problem | Fix |
|---|---|
| ProxyError: Cannot connect to proxy | Verify gateway is gw.knoxproxy.com:7000. Check your firewall allows outbound TCP on port 7000. |
| 407 Proxy Authentication Required | Check credentials in your KnoxProxy dashboard. Make sure username and password are URL-encoded if they contain special characters. |
| ConnectTimeout after 30s | Increase timeout to 60s. If persistent, try a different proxy type or contact support. |
| SSLError: certificate verify failed | Update certifi: pip install --upgrade certifi. Do not use verify=False in production. |
| bs4.FeatureNotFound: Couldn't find a tree builder with the features you requested: lxml | Run pip install lxml, then pass BeautifulSoup(response.text, 'lxml') again. Python's built-in html.parser also works if you skip lxml entirely. |
USER-country-de-city-berlin-session-profile07Order matters -- geo flags before the session flag. The session name is free text; use the profile ID so the mapping is self-documenting. Password stays as issued; no flags belong there. HTTP on :7000, SOCKS5 on :7001, same credentials.
Yes, Python requests supports SOCKS5 with KnoxProxy. Install the socks extra with pip install requests[socks], then set your proxy URL to socks5://USER:PASS@gw.knoxproxy.com:7001 instead of the HTTP port. SOCKS5 works on every KnoxProxy plan and proxy type, and each request through the gateway still rotates to a fresh IP by default.
Each new request through the KnoxProxy gateway rotates automatically to a fresh IP address, so a plain requests.get() call gives you a new exit IP every time. To keep the same IP across a login flow or multi-step session, append -session-{id} to your username, which holds that IP for up to 30 minutes before rotating again.
The requests library itself is synchronous and has no built-in async mode. For real concurrency, switch to aiohttp or httpx, both of which support Python asyncio and work with KnoxProxy using the same proxy URL format: http://USER:PASS@gw.knoxproxy.com:7000. If you need to keep using requests, run it inside threads with concurrent.futures.ThreadPoolExecutor for parallel calls instead.
KnoxProxy places no cap on concurrent connections on any plan, so the maximum concurrency with Python requests is set by your machine and the target site, not the proxy gateway. Since requests is synchronous, use concurrent.futures.ThreadPoolExecutor to run many requests in parallel threads, and watch the target site rate limits rather than KnoxProxy limits.
Install BeautifulSoup with pip install beautifulsoup4 in the same virtual environment as requests. The pip beautifulsoup package name is beautifulsoup4, not beautifulsoup -- installing beautifulsoup this way also pulls in soupsieve automatically. Add pip install lxml too, since BeautifulSoup uses it as a faster parser than Python's built-in html.parser.
BeautifulSoup find() and find_all() search parsed HTML for tags, attributes, or CSS classes and return matching elements. Use soup.find('div', class_='price') for the first match, or find_all() for every match on the page. Beautiful soup find methods work the same whether the page came from requests or lxml.etree parsing.
Use lxml for production scraping since it parses HTML faster than Python's built-in html.parser. Install it with pip install lxml, then pass BeautifulSoup(html, 'lxml') instead of BeautifulSoup(html, 'html.parser'). Python lxml also exposes lxml.etree directly for XPath queries, which BeautifulSoup does not support on its own.
Yes. Set a Python proxy on the requests call before passing the response text to BeautifulSoup, since BeautifulSoup only parses HTML and has no network or proxy support of its own. A typical requests python proxy setup fetches the page through KnoxProxy, then hands response.text to BeautifulSoup for parsing.
Rotating residential proxies -- 2 minutes setup, instant activation, 14-day money-back guarantee.