Training corpora need petabyte-scale coverage without concentrating traffic on a handful of IPs that get rate-limited or banned. Datacenter proxies handle the bulk of open-web volume at $0.02/IP; residential covers the protected sources that block bulk collection outright.
Use datacenter proxies for the bulk of corpus collection -- open-web sources, government data, and permissive archives need throughput, not stealth. Route protected or paywalled sources through rotating residential, and reserve mobile for app-exclusive content. Blend the three to hit petabyte scale without one target's block list stalling the whole crawl.
| Expected success | 99%+ on open-web sources, 95%+ on protected |
| Throughput | Unlimited concurrency across all plans |
| Strategy | Hybrid routing -- match proxy type to source protection level |
| Cost fit | Blended $0.15-$2.10/GB depending on source mix |
import requests
dc_proxy = "http://USER:PASS@dc.knoxproxy.com:8000"res_proxy = "http://USER:PASS@gw.knoxproxy.com:7000"
def fetch(url, protected=False): proxy = res_proxy if protected else dc_proxy r = requests.get(url, proxies={"https": proxy}, timeout=15) return r.text if r.status_code == 200 else None
sources = load_source_manifest() # each entry flags protected=True/Falsefor src in sources: text = fetch(src["url"], protected=src["protected"]) if text: write_to_corpus(src["domain"], text)Corpus collection must stay within our public-data-only acceptable-use policy. Collect only publicly accessible pages, respect robots.txt and rate limits, and exclude sources that explicitly prohibit automated collection in their terms.
A crawl that stalls on the sources with the strongest anti-bot defenses produces a corpus biased toward whatever is easiest to reach. Datacenter proxies clear the majority of the open web that does not block automated traffic; residential rotation opens the protected long tail that would otherwise be entirely absent from the dataset.
The only reliable way to see what a real user sees is to become one.
Scheduler, proxy fetch, parser, store -- the proxy is one line in the fetch step. Everything else is pipeline you already run.
Tag each source as permissive or protected before the first fetch. Route permissive sources to datacenter and protected ones to residential -- guessing wrong wastes both cost and time.
When a datacenter-routed source drops below 90% success, move it to residential rather than retrying. Persistent failures usually mean the target upgraded its detection, not a transient block.
Near-duplicate pages -- pagination, tracking parameters, mirrors -- inflate storage and skew training weight. Hash normalized content before writing to the corpus, not after.
Failed fetches are never billed, so your effective cost tracks the success rate you actually observe.
Datacenter for the bulk of open, permissive sources -- it is the cheapest way to hit petabyte scale at $0.02 per IP. Route protected and paywalled sources through rotating residential, since datacenter IPs get blocked on anti-bot-hardened domains, and reserve mobile proxies for any app-exclusive content the corpus still needs.
Blended cost typically lands between $0.15 and $2.10 per GB, depending on how much of the corpus comes from protected sources versus open datacenter-reachable ones. Classifying each source correctly before crawling is the biggest lever on total cost, since routing a permissive source to residential wastes budget for no gain.
Yes, within our public-data-only acceptable-use policy. Collect only publicly accessible pages, respect robots.txt and each source's rate limits, and exclude any source whose terms explicitly prohibit automated collection, whether the source is permissive and datacenter-routed or protected and reached through residential rotation.
Route protected and paywalled sources through residential rotation instead of dropping them from the crawl entirely. A datacenter-only crawl systematically excludes exactly the sources with the strongest defenses, which biases the resulting corpus toward whatever content was easiest to reach rather than what is actually representative.
Rotating residential with city targeting included -- instant activation, 14-day money-back guarantee.