The most powerful Python web scraping framework. Configure KnoxProxy as the default proxy middleware for all Scrapy spiders.
Install from PyPI.
pip install scrapyAdd proxy middleware and credentials.
# settings.py
DOWNLOADER_MIDDLEWARES = {
'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware': 110,
}
HTTP_PROXY = 'http://USER:PASS@gw.knoxproxy.com:7000'
HTTPS_PROXY = 'http://USER:PASS@gw.knoxproxy.com:7000'Set proxy in spider request meta.
yield scrapy.Request(url,
meta={'proxy': 'http://USER:PASS@gw.knoxproxy.com:7000'})Add country to proxy username.
meta={'proxy': 'http://USER-country-us:PASS@gw.knoxproxy.com:7000'}Custom middleware for per-request rotation.
class KnoxProxyMiddleware:
def process_request(self, request, spider):
request.meta['proxy'] = 'http://USER:PASS@gw.knoxproxy.com:7000'Run a test spider.
scrapy fetch --nolog "https://httpbin.org/ip""""KnoxProxy + Scrapy -- spider with proxy rotation."""
import scrapy
class KnoxSpider(scrapy.Spider):
name = "knox_demo"
start_urls = ["https://httpbin.org/ip"]
custom_settings = {
"DOWNLOADER_MIDDLEWARES": {
"scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware": 110,
},
"CONCURRENT_REQUESTS": 32,
"DOWNLOAD_TIMEOUT": 30,
}
def start_requests(self):
for url in self.start_urls:
yield scrapy.Request(
url,
meta={"proxy": "http://USER:PASS@gw.knoxproxy.com:7000"},
callback=self.parse,
)
def parse(self, response):
yield {"ip": response.json()["origin"], "url": response.url}Gateway rotates IPs per request by default. Each Scrapy Request through the middleware gets a fresh exit IP. Use -session-{id} in the proxy username for crawl continuity.
| Problem | Fix |
|---|---|
| TunnelError: Could not open CONNECT tunnel | Verify credentials. Ensure proxy URL uses http:// not https:// for the gateway. |
| 403 on target site | Switch to residential proxies. Add realistic User-Agent and headers. |
| TimeoutError on many requests | Reduce CONCURRENT_REQUESTS. Add DOWNLOAD_DELAY: 0.5. |
USER-country-de-city-berlin-session-profile07Order matters -- geo flags before the session flag. The session name is free text; use the profile ID so the mapping is self-documenting. Password stays as issued; no flags belong there. HTTP on :7000, SOCKS5 on :7001, same credentials.
Not natively. Scrapy runs on the Twisted networking library, which has no built-in SOCKS5 support, so a plain HttpProxyMiddleware setup will not work with a socks5:// URL. Install the scrapy-socks package to add SOCKS5 support, or simply stick with the HTTP/HTTPS proxy protocol on KnoxProxy port 7000, which every spider already supports without extra dependencies.
Set proxy URLs on a per-spider basis using the custom_settings dictionary in each Scrapy spider class, which overrides the project-wide HTTP_PROXY and HTTPS_PROXY settings for just that spider. For finer control down to individual requests, set meta={"proxy": "..."} on each scrapy.Request instead, which lets one spider mix multiple KnoxProxy configurations across its crawl.
Yes. Write a custom downloader middleware that assigns a different KnoxProxy proxy URL, such as residential, datacenter, or mobile, based on the target domain or the current retry count of the request. This lets one Scrapy project escalate from cheaper datacenter proxies to residential or mobile proxies automatically when a site starts blocking requests.
KnoxProxy residential IP rotation already cuts down how often Scrapy spiders trigger CAPTCHAs, since rotating residential IPs look far less suspicious to target sites than a static datacenter address. For sites that still serve CAPTCHAs often, add retry middleware that detects the CAPTCHA response and switches the proxy to a mobile IP on the retry attempt.
Rotating residential proxies -- 5 minutes setup, instant activation, 14-day money-back guarantee.