AI-powered web crawler for LLM data extraction. Route Crawl4AI through KnoxProxy for geo-targeted crawling with residential IPs.
Install from PyPI.
pip install crawl4ai
crawl4ai-setupPass proxy to AsyncWebCrawler.
from crawl4ai import AsyncWebCrawler
async with AsyncWebCrawler(
proxy="http://USER:PASS@gw.knoxproxy.com:7000"
) as crawler:
result = await crawler.arun(url="https://example.com")
print(result.markdown)Country in proxy URL.
proxy="http://USER-country-de:PASS@gw.knoxproxy.com:7000"Extract structured data with LLM.
result = await crawler.arun(
url="https://example.com",
extraction_strategy=LLMExtractionStrategy(
provider="openai/gpt-4o-mini",
instruction="Extract product names and prices"
)
)Run a test crawl.
print(result.success, len(result.markdown))Crawl multiple URLs.
results = await crawler.arun_many(urls=url_list)"""KnoxProxy + Crawl4AI -- geo-targeted AI-powered crawling."""
import asyncio
from crawl4ai import AsyncWebCrawler
PROXY = "http://USER:PASS@gw.knoxproxy.com:7000"
async def main():
async with AsyncWebCrawler(proxy=PROXY) as crawler:
result = await crawler.arun(url="https://example.com")
print(f"Success: {result.success}")
print(f"Markdown length: {len(result.markdown)}")
print(result.markdown[:500])
asyncio.run(main())Each arun() call gets a fresh IP. Use -session-{id} for multi-page crawls that need session continuity.
| Problem | Fix |
|---|---|
| Crawl failed: proxy connection error | Verify URL format and credentials. |
| Empty markdown output | Crawl4AI uses Playwright. Ensure browser is installed: crawl4ai-setup. |
USER-country-de-city-berlin-session-profile07Order matters -- geo flags before the session flag. The session name is free text; use the profile ID so the mapping is self-documenting. Password stays as issued; no flags belong there. HTTP on :7000, SOCKS5 on :7001, same credentials.
Crawl4AI renders pages with Playwright before extracting content for LLM pipelines, and repeated requests from one server IP get flagged and blocked quickly. Routing through KnoxProxy residential IPs makes each crawl look like an organic visitor from a real household connection, which keeps the crawler running long enough to gather full markdown output instead of empty pages.
Yes. Append -country-{cc} to the proxy username, for example -country-de for Germany, and AsyncWebCrawler routes the request through a KnoxProxy exit IP in that country. This returns the same localized content, pricing, and language variants that a real visitor browsing from that country would see, without any other code changes.
Yes. Calling arun_many() with a list of URLs runs several crawls concurrently, and each one picks up a fresh KnoxProxy exit IP automatically without extra configuration. This spreads requests across many residential IPs at once, which lowers the chance of any single IP getting rate-limited or blocked mid-batch.
The LLMExtractionStrategy in Crawl4AI works with OpenAI, Anthropic, and local models served through Ollama for structured data extraction from crawled pages. The KnoxProxy proxy setting only routes the web crawling traffic that fetches pages; it has no effect on the separate API calls your chosen LLM provider makes to process that content.
Rotating residential proxies -- 5 minutes setup, instant activation, 14-day money-back guarantee.