AI-powered web scraping API that converts websites to clean markdown for LLM consumption. Route self-hosted Firecrawl requests through KnoxProxy as an upstream proxy to bypass IP blocks and access geo-restricted content at scale.
Deploy Firecrawl self-hosted via Docker or sign up for the cloud API.
git clone https://github.com/mendableai/firecrawl
cd firecrawl && docker compose up -dSet KnoxProxy as the upstream proxy in Firecrawl environment variables.
PROXY_SERVER=http://USER:PASS@gw.knoxproxy.com:7000Install the Python or JavaScript SDK for your application.
pip install firecrawl-py
# or: npm install @mendable/firecrawl-jsCreate a Firecrawl client pointing to your self-hosted instance or cloud API.
Modify the proxy username for country-specific crawling.
PROXY_SERVER=http://USER-country-uk:PASS@gw.knoxproxy.com:7000Use the scrape or crawl endpoints to convert web pages to markdown through KnoxProxy.
import os
from firecrawl import FirecrawlApp
# Option 1: Self-hosted with KnoxProxy as upstream proxy
# Set in docker-compose.yml or .env:
# PROXY_SERVER=http://USER:PASS@gw.knoxproxy.com:7000
# Option 2: Use Firecrawl API with proxied requests via SDK
app = FirecrawlApp(
api_key="fc-YOUR_API_KEY",
# For self-hosted:
# api_url="http://localhost:3002",
)
# Scrape a single page to markdown
result = app.scrape_url(
"https://example.com/pricing",
params={
"formats": ["markdown", "html"],
"waitFor": 3000,
},
)
print(result["markdown"][:500])
# Crawl an entire site
crawl_result = app.crawl_url(
"https://example.com",
params={
"limit": 50,
"scrapeOptions": {
"formats": ["markdown"],
},
},
poll_interval=5,
)
for page in crawl_result.get("data", []):
print(f"Crawled: {page['metadata']['url']}")
print(f"Content: {page['markdown'][:200]}\n")
# ---
# Self-hosted docker-compose.yml proxy config:
# services:
# firecrawl:
# environment:
# - PROXY_SERVER=http://USER:PASS@gw.knoxproxy.com:7000
# # Country targeting:
# # - PROXY_SERVER=http://USER-country-us:PASS@gw.knoxproxy.com:7000Each page request through Firecrawl gets a fresh IP via KnoxProxy. For sticky sessions during a site crawl, use USER-session-{crawl_id} in the PROXY_SERVER username.
| Problem | Fix |
|---|---|
| Firecrawl returns empty markdown | Firecrawl uses Playwright internally. Ensure the proxy is set at the environment level so Playwright browser traffic is also proxied. |
| CrawlError: 407 Proxy Authentication Required | Verify the PROXY_SERVER environment variable format: http://USER:PASS@gw.knoxproxy.com:7000 |
| CrawlError: Connection timeout after 30s | Try a residential proxy by adding -type-residential to the username. Increase waitFor in scrape options. |
USER-country-de-city-berlin-session-profile07Order matters -- geo flags before the session flag. The session name is free text; use the profile ID so the mapping is self-documenting. Password stays as issued; no flags belong there. HTTP on :7000, SOCKS5 on :7001, same credentials.
Firecrawl cloud manages its own proxy infrastructure internally, so there is no PROXY_SERVER setting you can override from outside the service. KnoxProxy is most useful with a self-hosted Firecrawl deployment, where you set PROXY_SERVER directly in the Docker environment and control every outbound crawl request yourself.
Yes, change the PROXY_SERVER environment variable to a different USER-country-{cc} username before each crawl job, for example switching from uk to us between runs. Alternatively, run multiple self-hosted Firecrawl instances at once, each with its own docker-compose.yml proxy config, so different countries crawl in parallel.
Firecrawl automatically converts scraped pages into clean markdown formatted for LLM consumption using its built-in Playwright renderer, while a raw Scrapy spider only returns HTML that you must parse yourself with selectors. Firecrawl fits AI and RAG pipelines better, while Scrapy suits classic structured data extraction and large-scale crawling tasks.
Yes, Firecrawl uses Playwright internally for browser rendering, and Playwright supports SOCKS5 proxies natively without extra configuration. Set the PROXY_SERVER environment variable to socks5://USER:PASS@gw.knoxproxy.com:7001 instead of the default HTTP port 7000, and the same country and session username patterns still apply on top of it.
Rotating residential proxies -- 5 minutes setup, instant activation, 14-day money-back guarantee.