The most powerful Python web scraping framework. Configure KnoxProxy as the default proxy middleware for all Scrapy spiders.
Install from PyPI.
pip install scrapyAdd proxy middleware and credentials.
# settings.py
DOWNLOADER_MIDDLEWARES = {
'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware': 110,
}
HTTP_PROXY = 'http://USER:PASS@gw.knoxproxy.com:7000'
HTTPS_PROXY = 'http://USER:PASS@gw.knoxproxy.com:7000'Set proxy in spider request meta.
yield scrapy.Request(url,
meta={'proxy': 'http://USER:PASS@gw.knoxproxy.com:7000'})Add country to proxy username.
meta={'proxy': 'http://USER-country-us:PASS@gw.knoxproxy.com:7000'}Custom middleware for per-request rotation.
class KnoxProxyMiddleware:
def process_request(self, request, spider):
request.meta['proxy'] = 'http://USER:PASS@gw.knoxproxy.com:7000'Run a test spider.
scrapy fetch --nolog "https://httpbin.org/ip""""KnoxProxy + Scrapy -- spider with proxy rotation."""
import scrapy
class KnoxSpider(scrapy.Spider):
name = "knox_demo"
start_urls = ["https://httpbin.org/ip"]
custom_settings = {
"DOWNLOADER_MIDDLEWARES": {
"scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware": 110,
},
"CONCURRENT_REQUESTS": 32,
"DOWNLOAD_TIMEOUT": 30,
}
def start_requests(self):
for url in self.start_urls:
yield scrapy.Request(
url,
meta={"proxy": "http://USER:PASS@gw.knoxproxy.com:7000"},
callback=self.parse,
)
def parse(self, response):
yield {"ip": response.json()["origin"], "url": response.url}Gateway rotates IPs per request by default. Each Scrapy Request through the middleware gets a fresh exit IP. Use -session-{id} in the proxy username for crawl continuity.
| Problem | Fix |
|---|---|
| TunnelError: Could not open CONNECT tunnel | Verify credentials. Ensure proxy URL uses http:// not https:// for the gateway. |
| 403 on target site | Switch to residential proxies. Add realistic User-Agent and headers. |
| TimeoutError on many requests | Reduce CONCURRENT_REQUESTS. Add DOWNLOAD_DELAY: 0.5. |
USER-country-de-city-berlin-session-profile07Order matters -- geo flags before the session flag. The session name is free text; use the profile ID so the mapping is self-documenting. Password stays as issued; no flags belong there. HTTP on :7000, SOCKS5 on :7001, same credentials.
Scrapy uses Twisted under the hood, which does not natively support SOCKS5. Use the scrapy-socks package or stick with HTTP/HTTPS proxy protocol via KnoxProxy port 7000.
Use custom_settings in each spider class to define different proxy URLs, or set the proxy in the meta dict of each Request.
Yes. Create a custom middleware that assigns different proxy URLs (residential, datacenter, mobile) based on the target domain or retry count.
KnoxProxy residential rotation reduces CAPTCHA encounters. For persistent CAPTCHAs, add retry middleware that switches to mobile proxies on the retry.
Free trial on rotating residential -- 5 minutes setup, no credit card.