A tested walkthrough for Indeed Job Listings -- the right proxy type, 91% tested success rate, working code, and what to avoid to stay compliant.
An Indeed scraper pulls job title, company, salary range, location, and description snippet from search-result pages using residential rotating proxies at a moderate pace. Indeed job scraper scripts that rely on datacenter IPs or rapid pagination trigger a Cloudflare managed challenge, while KnoxProxy residential IPs held a 91% success rate on Indeed web scraping runs across search-result pages.
| Anti-bot system | Cloudflare |
| Challenge types | Cloudflare managed challenge (JavaScript and Turnstile), Per-IP rate limiting on search results, 429 on burst traffic, CAPTCHA on flagged sessions |
| Rate limit behavior | Search result pages tolerate a moderate pace of requests from residential IPs; datacenter IPs and rapid pagination are common triggers for a Cloudflare managed challenge. |
| Tested success rate | 91% |
"""
KnoxProxy scraper for Indeed job search results.
Extracts job title, company, location, salary snippet, and description
snippet across paginated search-result pages, then writes the results
to a CSV file for later analysis.
"""
import csv
import random
import time
import requests
from bs4 import BeautifulSoup
PROXY_HOST = "proxy.knoxproxy.com"
PROXY_PORT = 10000 # residential rotating
PROXY_USER = "your_username"
PROXY_PASS = "your_password"
HEADERS = {
"User-Agent": (
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
"(KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36"
),
"Accept-Language": "en-US,en;q=0.9",
}
def proxy_dict() -> dict:
proxy_url = f"http://{PROXY_USER}:{PROXY_PASS}@{PROXY_HOST}:{PROXY_PORT}"
return {"http": proxy_url, "https": proxy_url}
def search_indeed_page(query: str, location: str, start: int, max_retries: int = 3) -> list[dict]:
url = "https://www.indeed.com/jobs"
params = {"q": query, "l": location, "start": start}
for attempt in range(1, max_retries + 1):
try:
resp = requests.get(url, headers=HEADERS, params=params, proxies=proxy_dict(), timeout=20)
if resp.status_code == 200:
return parse_indeed_results(resp.text)
print(f"Attempt {attempt}: Indeed returned status {resp.status_code}")
except requests.exceptions.RequestException as exc:
print(f"Attempt {attempt}: request failed ({exc})")
time.sleep(random.uniform(4, 6))
raise RuntimeError(f"Failed to search Indeed for '{query}' at start={start} after {max_retries} attempts")
def parse_indeed_results(html: str) -> list[dict]:
soup = BeautifulSoup(html, "html.parser")
jobs = []
for card in soup.select(".job_seen_beacon"):
title_el = card.select_one(".jobTitle span")
company_el = card.select_one(".companyName")
salary_el = card.select_one(".salary-snippet-container")
summary_el = card.select_one(".job-snippet")
if not title_el:
continue
jobs.append({
"title": title_el.get_text(strip=True),
"company": company_el.get_text(strip=True) if company_el else None,
"salary": salary_el.get_text(strip=True) if salary_el else None,
"description": summary_el.get_text(" ", strip=True) if summary_el else None,
})
return jobs
def search_indeed(query: str, location: str, max_pages: int = 5) -> list[dict]:
"""Paginate an Indeed job search 10 results at a time until a page
returns no job cards or max_pages is reached."""
all_jobs = []
for page in range(max_pages):
start = page * 10
jobs = search_indeed_page(query, location, start)
if not jobs:
break
all_jobs.extend(jobs)
time.sleep(random.uniform(4, 6))
return all_jobs
def save_jobs_to_csv(jobs: list[dict], filename: str = "indeed_jobs.csv") -> None:
if not jobs:
return
with open(filename, "w", newline="", encoding="utf-8") as f:
writer = csv.DictWriter(f, fieldnames=["title", "company", "salary", "description"])
writer.writeheader()
writer.writerows(jobs)
if __name__ == "__main__":
results = search_indeed("data engineer", "Austin, TX", max_pages=3)
save_jobs_to_csv(results)
print(f"Saved {len(results)} jobs to indeed_jobs.csv")
[
{
"title": "Senior Data Engineer",
"company": "Northline Analytics",
"salary": "$130,000 - $160,000 a year",
"description": "Design and maintain ETL pipelines feeding analytics dashboards used across the data team..."
}
]Install requests and BeautifulSoup for parsing job search-result pages.
pip install requests beautifulsoup4Route requests through the residential rotating gateway, matched to the target country's jobs board.
PROXY_HOST = "proxy.knoxproxy.com"
PROXY_PORT = 10000
PROXY_USER = "your_username"
PROXY_PASS = "your_password"Request the jobs endpoint with q (query) and l (location) parameters.
Extract title, company, salary snippet, and description snippet from each job card in the results.
Increment the start parameter by 10 for each additional page (0, 10, 20...) and stop once a page returns no job cards, matching how Indeed's own pagination works.
Write each job record to a CSV file with title, company, salary, and description fields as you scrape, instead of holding everything in memory until the run finishes.
Indeed Job Listings runs cloudflare. The tested setup is residential rotating proxies, targeting match the country jobs board (indeed.com, indeed.co.uk, etc.), with this rotation: Rotate IP every 5-10 requests. That combination held a 91% success rate on the Jul 5, 2026 test run.
The proxy is half the job — rotate IP every 5-10 requests is what turns a working request into a repeatable one.
Rotate IP every 5-10 requests.
Match the country jobs board (indeed.com, indeed.co.uk, etc.).
Indeed Job Listings leans on cloudflare managed challenge (JavaScript and Turnstile). Search result pages tolerate a moderate pace of requests from residential IPs; datacenter IPs and rapid pagination are common triggers for a Cloudflare managed challenge.
Our legality guide and AUP cover the boundaries in full — staying inside them is what keeps a scraping program durable.
Only when the employer provides it or Indeed generates an estimated range; many listings omit a salary snippet entirely, so the field should be treated as optional. The scraper above already returns None for a missing salary-snippet-container instead of raising an error, so a missing value never breaks a scraping run.
Residential rotating proxies work best, rotated every 5-10 requests, since Cloudflare's managed challenge triggers faster on datacenter IPs and rapid pagination. Pacing requests to roughly one every 4-6 seconds on top of that rotation held KnoxProxy's proxies to a 91% success rate on Indeed search-result pages during testing.
Add an l (location) query parameter to the search URL, and match the proxy country to the Indeed country domain being queried for consistent results. The search_indeed_page() function above already accepts a location argument, so scraping a new city is just a matter of passing a different value.
Indeed updates its search-result card structure periodically, so selectors like job_seen_beacon should be re-verified against a live page before a production run. The parser above already skips any card missing a title element, so a markup change quietly drops results instead of crashing the whole scraper outright.
The Python code above is a tested Indeed scraper for search-result pages. Run it as-is or extend it to cover more cities; proxy rotation and pagination handling are already built in, the setup that held a 91% success rate on Indeed search-result pages during the most recent test.
Increment the start query parameter by 10 for each page (0, 10, 20...) and stop once a page returns no job cards. This mirrors how Indeed's own pagination links work and avoids requesting pages that no longer exist. The search_indeed() function above already wraps this loop with a max_pages safety limit.
Write each job record to a CSV or JSON file with title, company, salary, and description fields as you scrape, rather than holding everything in memory. This lets you reopen and analyze the dataset without re-running the scraper. The save_jobs_to_csv() helper above already does exactly this after each search run.
residential proxies -- 91% tested success, instant activation, 14-day money-back guarantee.