AI-powered web crawler for LLM data extraction. Route Crawl4AI through KnoxProxy for geo-targeted crawling with residential IPs.
Install from PyPI.
pip install crawl4ai
crawl4ai-setupPass proxy to AsyncWebCrawler.
from crawl4ai import AsyncWebCrawler
async with AsyncWebCrawler(
proxy="http://USER:PASS@gw.knoxproxy.com:7000"
) as crawler:
result = await crawler.arun(url="https://example.com")
print(result.markdown)Country in proxy URL.
proxy="http://USER-country-de:PASS@gw.knoxproxy.com:7000"Extract structured data with LLM.
result = await crawler.arun(
url="https://example.com",
extraction_strategy=LLMExtractionStrategy(
provider="openai/gpt-4o-mini",
instruction="Extract product names and prices"
)
)Run a test crawl.
print(result.success, len(result.markdown))Crawl multiple URLs.
results = await crawler.arun_many(urls=url_list)"""KnoxProxy + Crawl4AI -- geo-targeted AI-powered crawling."""
import asyncio
from crawl4ai import AsyncWebCrawler
PROXY = "http://USER:PASS@gw.knoxproxy.com:7000"
async def main():
async with AsyncWebCrawler(proxy=PROXY) as crawler:
result = await crawler.arun(url="https://example.com")
print(f"Success: {result.success}")
print(f"Markdown length: {len(result.markdown)}")
print(result.markdown[:500])
asyncio.run(main())Each arun() call gets a fresh IP. Use -session-{id} for multi-page crawls that need session continuity.
| Problem | Fix |
|---|---|
| Crawl failed: proxy connection error | Verify URL format and credentials. |
| Empty markdown output | Crawl4AI uses Playwright. Ensure browser is installed: crawl4ai-setup. |
USER-country-de-city-berlin-session-profile07Order matters -- geo flags before the session flag. The session name is free text; use the profile ID so the mapping is self-documenting. Password stays as issued; no flags belong there. HTTP on :7000, SOCKS5 on :7001, same credentials.
Crawl4AI uses Playwright for rendering. Without proxies, your server IP gets blocked. KnoxProxy residential IPs make each crawl look like an organic visitor.
Yes. Use -country-{cc} in the proxy username to access localized content from any country.
Yes. arun_many() runs concurrent crawls. Each gets a fresh KnoxProxy IP automatically.
OpenAI, Anthropic, and local models via Ollama. The proxy only affects web crawling, not the LLM API calls.
Free trial on rotating residential -- 5 minutes setup, no credit card.