Web scraping API with anti-bot bypass and browser rendering. Integrate KnoxProxy directly as a cost-effective alternative for proxy rotation while optionally using Scrapfly for JavaScript rendering only.
Install your preferred HTTP library.
pip install requests httpx # Python
npm install axios node-fetch # JavaScriptConfigure KnoxProxy as your proxy layer instead of Scrapfly proxy API.
# Python
KNOX_PROXY = "http://your_username:your_password@gw.knoxproxy.com:7000"
# JavaScript
const KNOX_PROXY = "http://your_username:your_password@gw.knoxproxy.com:7000"Add country code to username for location-specific scraping.
KNOX_PROXY = "http://your_username-country-de:your_password@gw.knoxproxy.com:7000"For multi-page scrapes, use sticky sessions to maintain IP consistency.
KNOX_PROXY = "http://your_username-country-de-session-myid:your_password@gw.knoxproxy.com:7000"For high-throughput workloads, use async HTTP clients with KnoxProxy.
import httpx
async with httpx.AsyncClient(proxy=KNOX_PROXY) as client:
resp = await client.get("https://example.com")Run test scrapes and compare success rates and costs with Scrapfly. KnoxProxy residential IPs deliver comparable results at lower bandwidth-based pricing.
import httpx
import asyncio
from typing import Optional
# Direct KnoxProxy integration as Scrapfly alternative
# Scrapfly: per-API-credit pricing
# KnoxProxy: per-GB bandwidth pricing (lower cost at scale)
KNOX_PROXY = "http://your_username-country-us:your_password@gw.knoxproxy.com:7000"
HEADERS = {
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36",
"Accept": "text/html,application/xhtml+xml",
"Accept-Language": "en-US,en;q=0.9",
}
async def scrape_urls(urls: list[str], country: str = "us") -> list[Optional[str]]:
"""Scrape multiple URLs concurrently through KnoxProxy."""
proxy = f"http://your_username-country-{country}:your_password@gw.knoxproxy.com:7000"
async with httpx.AsyncClient(proxy=proxy, headers=HEADERS, timeout=30) as client:
tasks = [client.get(url) for url in urls]
responses = await asyncio.gather(*tasks, return_exceptions=True)
return [
r.text if isinstance(r, httpx.Response) and r.status_code == 200 else None
for r in responses
]
urls = ["https://example.com/page1", "https://example.com/page2"]
results = asyncio.run(scrape_urls(urls))
print(f"Fetched {sum(1 for r in results if r)} / {len(urls)} pages")KnoxProxy rotates IPs per connection automatically. For async scraping with many concurrent requests, each connection gets a unique residential IP. Use -session-{id} for multi-request continuity.
| Problem | Fix |
|---|---|
| Async requests timing out | Limit concurrency with asyncio.Semaphore(10). Increase timeout to 60 seconds for slow-loading pages. |
| SSL certificate error through proxy | Ensure you are using the http:// scheme in your proxy URL even for HTTPS targets. The proxy handles CONNECT tunneling. |
| Inconsistent geo-location results | Use -country-{cc} in the username to lock all connections to one country. |
USER-country-de-city-berlin-session-profile07Order matters -- geo flags before the session flag. The session name is free text; use the profile ID so the mapping is self-documenting. Password stays as issued; no flags belong there. HTTP on :7000, SOCKS5 on :7001, same credentials.
Use KnoxProxy directly for cost-effective proxy rotation. Use Scrapfly when you need their cloud browser rendering or anti-bot SDK. You can combine both: KnoxProxy for the proxy, Scrapfly for rendering.
Each async connection gets its own rotated IP from KnoxProxy. Use httpx.AsyncClient with the proxy parameter for high-throughput concurrent scraping.
KnoxProxy has no connection limit. Limit concurrency based on what the target site can handle to avoid blocks.
The Scrapfly SDK routes through their infrastructure. For KnoxProxy, use standard HTTP clients (requests, httpx, axios) with proxy configuration.
Free trial on rotating residential -- 5 minutes setup, no credit card.