A tested walkthrough for Google Search Results -- the right proxy type, 85% tested success rate, working code, and what to avoid to stay compliant.
Scraping Google search results means sending automated queries to google.com/search and parsing the rendered HTML for rankings, titles, and snippets. Google backs those results with reCAPTCHA challenges and aggressive per-IP rate limiting, so a plain requests script without rotation gets blocked fast. KnoxProxy residential rotating proxies held an 85% success rate at roughly 8-10 queries per minute per IP during the last test.
| Anti-bot system | reCAPTCHA + rate limiting |
| Challenge types | reCAPTCHA interstitial ("detected unusual traffic"), Per-IP and per-subnet rate limiting, JavaScript-rendering requirements for some SERP features, Query-pattern anomaly detection |
| Rate limit behavior | Google tolerates only a handful of rapid queries per IP before serving a "detected unusual traffic" reCAPTCHA page; datacenter ranges are flagged far faster than residential IPs, and any single IP running more than roughly 8-10 queries per minute is at high risk of a block. |
| Tested success rate | 85% |
"""
KnoxProxy scraper for Google search results.
Extracts ranked position, title, display URL, and snippet from a
rendered Google SERP. Selectors change often -- re-verify against a
live page before production use.
"""
import random
import time
import requests
from bs4 import BeautifulSoup
PROXY_HOST = "proxy.knoxproxy.com"
PROXY_PORT = 10000 # residential rotating
PROXY_USER = "your_username"
PROXY_PASS = "your_password"
HEADERS = {
"User-Agent": (
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
"(KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36"
),
"Accept-Language": "en-US,en;q=0.9",
}
def proxy_dict(country: str = "us") -> dict:
user = f"{PROXY_USER}-country-{country}"
proxy_url = f"http://{user}:{PROXY_PASS}@{PROXY_HOST}:{PROXY_PORT}"
return {"http": proxy_url, "https": proxy_url}
def search_google(query: str, country: str = "us", language: str = "en", max_retries: int = 3) -> list[dict]:
url = "https://www.google.com/search"
params = {"q": query, "gl": country, "hl": language, "num": 10}
proxies = proxy_dict(country=country)
for attempt in range(1, max_retries + 1):
try:
resp = requests.get(url, headers=HEADERS, params=params, proxies=proxies, timeout=20)
if resp.status_code == 200 and "unusual traffic" not in resp.text.lower():
return parse_google_serp(resp.text)
print(f"Attempt {attempt}: Google returned status {resp.status_code} or a traffic warning")
except requests.exceptions.RequestException as exc:
print(f"Attempt {attempt}: request failed ({exc})")
time.sleep(random.uniform(5, 9))
raise RuntimeError(f"Failed to search Google for '{query}' after {max_retries} attempts")
def parse_google_serp(html: str) -> list[dict]:
soup = BeautifulSoup(html, "html.parser")
results = []
for position, block in enumerate(soup.select("div.g"), start=1):
title_el = block.select_one("h3")
link_el = block.select_one("a")
snippet_el = block.select_one(".VwiC3b")
if not title_el or not link_el:
continue
results.append({
"position": position,
"title": title_el.get_text(strip=True),
"url": link_el.get("href"),
"snippet": snippet_el.get_text(strip=True) if snippet_el else None,
})
return results
if __name__ == "__main__":
for result in search_google("best residential proxy providers", country="us")[:10]:
print(result)
"""
Minimal SERP scraping API wrapping the KnoxProxy Google scraper.
Exposes one endpoint so other services can request fresh search
results on demand instead of running the script by hand each time.
Run with: uvicorn serp_api:app --reload
Example: GET /serp?q=best+residential+proxy+providers&country=us
"""
from fastapi import FastAPI, Query
from google_serp_scraper import search_google
app = FastAPI(title="SERP Scraping API")
@app.get("/serp")
def get_serp(
q: str = Query(..., description="Search query"),
country: str = "us",
language: str = "en",
):
results = search_google(q, country=country, language=language)
return {"query": q, "country": country, "results": results}
"""
Stores parsed Google SERP results in SQLite for rank tracking over
time. Each run appends a new snapshot instead of overwriting the
last one, so position changes over time stay queryable.
"""
import sqlite3
from datetime import datetime, timezone
from google_serp_scraper import search_google
DB_PATH = "serp_results.db"
def init_db() -> None:
conn = sqlite3.connect(DB_PATH)
conn.execute(
"""
CREATE TABLE IF NOT EXISTS serp_results (
id INTEGER PRIMARY KEY AUTOINCREMENT,
query TEXT,
country TEXT,
position INTEGER,
title TEXT,
url TEXT,
snippet TEXT,
fetched_at TEXT
)
"""
)
conn.commit()
conn.close()
def store_results(query: str, country: str = "us") -> None:
results = search_google(query, country=country)
fetched_at = datetime.now(timezone.utc).isoformat()
conn = sqlite3.connect(DB_PATH)
conn.executemany(
"""
INSERT INTO serp_results
(query, country, position, title, url, snippet, fetched_at)
VALUES
(:query, :country, :position, :title, :url, :snippet, :fetched_at)
""",
[
{**r, "query": query, "country": country, "fetched_at": fetched_at}
for r in results
],
)
conn.commit()
conn.close()
if __name__ == "__main__":
init_db()
store_results("best residential proxy providers", country="us")
[
{
"position": 1,
"title": "Best Residential Proxy Providers in 2026",
"url": "https://example.com/best-residential-proxies",
"snippet": "A comparison of pool size, pricing, and success rates across providers."
}
]Install requests and BeautifulSoup for querying and parsing rendered search-result pages.
pip install requests beautifulsoup4Route queries through the residential rotating gateway, matched to the target country.
PROXY_HOST = "proxy.knoxproxy.com"
PROXY_PORT = 10000
PROXY_USER = "your_username"
PROXY_PASS = "your_password"Set the gl (country) and hl (language) parameters to match the market being researched.
Extract ranked results with BeautifulSoup, and re-check selectors against a live page periodically since Google's SERP markup changes often.
Turn search_google() into a small FastAPI endpoint so other services can request fresh SERP data on demand instead of running the script by hand for every query.
Write each parsed result to a lightweight SQLite table with the query, position, and timestamp so rank changes are queryable later instead of living only in console output.
Google Search Results runs reCAPTCHA + rate limiting. The tested setup is residential rotating proxies, targeting match the country and language edition of Google being queried (google.com vs google.co.uk, gl and hl parameters), with this rotation: Rotate IP on every query. That combination held a 85% success rate on the Jul 7, 2026 test run.
The proxy is half the job — rotate IP on every query is what turns a working request into a repeatable one.
Rotate IP on every query.
Match the country and language edition of Google being queried (google.com vs google.co.uk, gl and hl parameters).
Google Search Results leans on reCAPTCHA interstitial ("detected unusual traffic"). Google tolerates only a handful of rapid queries per IP before serving a "detected unusual traffic" reCAPTCHA page; datacenter ranges are flagged far faster than residential IPs, and any single IP running more than roughly 8-10 queries per minute is at high risk of a block.
Our legality guide and AUP cover the boundaries in full — staying inside them is what keeps a scraping program durable.
Google's robots.txt disallows most /search paths and its Terms of Service prohibit automated querying, so scraping rendered public results sits in a compliance gray area. Google's official Custom Search JSON API, or a licensed SERP-data provider, is the more defensible path for production use, even though many rank-tracking tools still scrape at a conservative pace.
Residential rotating proxies work best since reCAPTCHA triggers on datacenter ranges almost immediately, while well-paced residential IPs sustain a modest query rate before being challenged. Rotating the IP on every query and staying near 8-10 queries per minute held an 85% success rate against Google's rate limiting in the last test.
Set the gl (geolocation) and hl (host language) query parameters to match the target market, and route the request through a proxy IP in that same country for consistent results. The scraper above already builds this proxy-country pairing automatically, so switching markets is just a one-line change to the country argument.
Google changes its SERP HTML structure frequently, sometimes multiple times a year, so CSS selectors like div.g need to be re-verified against a live results page before any production run. The parser above also skips any block missing a title or link element, so a markup change quietly drops results instead of crashing outright.
The Python code above is a tested Google SERP scraper you can run as-is or extend for more result types. Proxy rotation and the country/language targeting are already built in, along with retry logic on traffic warnings, the setup that held an 85% success rate in testing.
Send a GET request to google.com/search with requests, route it through a rotating residential proxy, and parse the HTML with BeautifulSoup. The tested google_serp_scraper.py script above covers all three steps, including retries on a traffic-warning response, plus gl and hl parameters for targeting a specific country and language.
Yes -- wrap the search_google() function in a small FastAPI endpoint that accepts a query and country, then calls the scraper and returns JSON. The serp_api.py sample above does exactly that in under 20 lines, ready to run with uvicorn and query with a simple GET request.
Free scrapers without proxy rotation usually break within a few dozen queries once Google's rate limiting kicks in. A rotating residential proxy is what keeps a scraper running at real volume; the code itself can stay simple and free to write.
Google Flights loads fares through dynamic, JavaScript-driven calls instead of static HTML, so the requests-plus-BeautifulSoup approach used for search results does not work directly. A headless browser like Playwright, paired with residential proxies matched to the origin country, fits that rendering model better.
Baidu is the other major search engine with a dedicated KnoxProxy guide, using China-region residential proxies instead of the global rotation used for Google. Google itself relies on reCAPTCHA-triggered rate limits near 8-10 queries per minute, and those same principles -- rotation, geo matching, and selector upkeep -- carry over to most search engines.
residential proxies -- 85% tested success, instant activation, 14-day money-back guarantee.