The essential points from this guide -- each one is explained in detail below.
Classify errors into retryable (502/503/timeout), auth failures (407), and blocks (403/429) -- each needs a different response.
Exponential backoff with jitter prevents retry storms that compound the problem.
Rotate to a new proxy IP on 403/429 responses -- retrying with the same IP wastes time.
Circuit breakers prevent a failing proxy from slowing down your entire scraper.
Log error details (status code, proxy used, target URL) for post-hoc analysis and provider debugging.
Understanding what each error code means helps you choose the right recovery strategy. A 407 Proxy Authentication Required means your proxy credentials are wrong or expired -- retrying with the same credentials will never work. A 403 Forbidden usually means the target site blocked your IP, not a proxy infrastructure issue -- rotate to a new IP. A 429 Too Many Requests means you are hitting rate limits and need to slow down, not just rotate.
502 Bad Gateway and 503 Service Unavailable typically indicate temporary proxy infrastructure issues -- the proxy server could not reach the target or is overloaded. These are retryable after a short wait. Connection timeouts mean the proxy could not establish a connection, possibly due to the target blocking the proxy IP at the network level or the proxy being genuinely down.
A 200 response with CAPTCHA content is a soft block -- the target site suspects automation but has not blocked you outright. Detect this by checking response body length or looking for CAPTCHA markers in the HTML.
Exponential backoff increases the wait time between retries to give the proxy infrastructure and target site time to recover. Adding random jitter prevents multiple scrapers from retrying in lockstep, which would create a thundering herd problem.
import random
import time
import requests
def fetch_with_retry(url, proxy, max_retries=3):
retryable_codes = {500, 502, 503, 504}
rotate_codes = {403, 429}
for attempt in range(max_retries):
try:
response = requests.get(
url,
proxies={'https': proxy},
timeout=30,
)
if response.status_code == 200:
return response
if response.status_code in rotate_codes:
# Target blocked this IP -- rotating on next attempt via gateway
wait = (2 ** attempt) + random.uniform(0, 1)
time.sleep(wait)
continue
if response.status_code in retryable_codes:
wait = (2 ** attempt) + random.uniform(0, 1)
time.sleep(wait)
continue
if response.status_code == 407:
raise Exception('Proxy authentication failed -- check credentials')
return response # Non-retryable status
except requests.exceptions.Timeout:
wait = (2 ** attempt) + random.uniform(0, 1)
time.sleep(wait)
except requests.exceptions.ConnectionError:
wait = (2 ** attempt) + random.uniform(0, 1)
time.sleep(wait)
raise Exception(f'Failed after {max_retries} retries: {url}')With a backconnect gateway, each retry automatically gets a new exit IP. With a static proxy list, explicitly rotate to the next proxy in your pool on each retry attempt.
A circuit breaker prevents a failing proxy from dragging down your entire scraper. When a proxy accumulates too many consecutive failures, the circuit breaker "opens" and removes it from the active pool for a cooldown period. After the cooldown, it enters a "half-open" state where a single test request determines whether it returns to service.
from enum import Enum
from time import time
class CircuitState(Enum):
CLOSED = 'closed' # Normal operation
OPEN = 'open' # Failing, not used
HALF_OPEN = 'half_open' # Testing recovery
class ProxyCircuitBreaker:
def __init__(self, failure_threshold=5, cooldown_seconds=300):
self.failure_threshold = failure_threshold
self.cooldown_seconds = cooldown_seconds
self.state = CircuitState.CLOSED
self.failure_count = 0
self.opened_at = 0.0
def record_success(self):
self.failure_count = 0
self.state = CircuitState.CLOSED
def record_failure(self):
self.failure_count += 1
if self.failure_count >= self.failure_threshold:
self.state = CircuitState.OPEN
self.opened_at = time()
def is_available(self) -> bool:
if self.state == CircuitState.CLOSED:
return True
if self.state == CircuitState.OPEN:
if time() - self.opened_at > self.cooldown_seconds:
self.state = CircuitState.HALF_OPEN
return True
if self.state == CircuitState.HALF_OPEN:
return True
return FalseAttach a circuit breaker to each proxy in your pool. Before selecting a proxy, check is_available(). This pattern is less relevant for backconnect gateways where the provider manages the pool, but valuable for self-managed proxy lists.
Structured error logging is essential for diagnosing proxy issues in production. Log the proxy used, target URL, status code, response time, and error type for every failed request. This data helps you identify patterns like specific target domains that block more aggressively or times of day when success rates drop.
import logging
import json
from datetime import datetime
logger = logging.getLogger('proxy_errors')
def log_proxy_error(proxy, url, status_code, latency, error_type):
entry = {
'timestamp': datetime.utcnow().isoformat(),
'proxy': proxy.split('@')[1] if '@' in proxy else proxy, # Strip credentials
'target_url': url,
'status_code': status_code,
'latency_ms': round(latency * 1000),
'error_type': error_type,
}
logger.error(json.dumps(entry))Never log proxy credentials. Strip the username:password portion from proxy URLs before writing to logs or error reporting systems. Aggregate error logs periodically to detect trends -- a spike in 429 errors means you need to slow down, while a spike in 502 errors suggests a provider infrastructure issue worth reporting to their support team.
Ready to put this into practice? Test your proxy
KnoxProxy Research Team · Technical Content
Network engineers and proxy infrastructure specialists with 10+ years in anti-bot systems, web scraping, and IP routing.
90.4M+ ethically sourced residential IPs across 195 countries. Start free -- no credit card required.