A tested walkthrough for Wayfair Furniture -- the right proxy type, 80% tested success rate, working code, and what to avoid to stay compliant.
Wayfair product title, price, and rating are recoverable from an embedded JSON state object, best reached with residential rotating proxies on a short sticky session. Wayfair's GraphQL-backed storefront validates request signing and redirects high-velocity traffic to a verification page, and KnoxProxy residential sessions held an 80% success rate. A Wayfair scraper needs to respect that sticky-session window, or the success rate drops fast.
| Anti-bot system | Custom bot detection |
| Challenge types | "Verify you are human" interstitial redirect, Device and IP fingerprint scoring, Session cookie validation, Request signing on internal GraphQL endpoints |
| Rate limit behavior | Wayfair's storefront runs on a GraphQL backend that validates request signing and session cookies; unauthenticated or high-velocity datacenter traffic is redirected to a verification page within the first few requests. |
| Tested success rate | 80% |
"""
KnoxProxy scraper for Wayfair product pages.
Wayfair hydrates its storefront from an embedded JSON state object,
similar in spirit to Walmart's __NEXT_DATA__ approach.
"""
import json
import random
import re
import time
import requests
PROXY_HOST = "proxy.knoxproxy.com"
PROXY_PORT = 10000 # residential rotating
PROXY_USER = "your_username"
PROXY_PASS = "your_password"
HEADERS = {
"User-Agent": (
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
"(KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36"
),
"Accept-Language": "en-US,en;q=0.9",
}
STATE_RE = re.compile(r"window\.__INITIAL_STATE__\s*=\s*(\{.*?\});\s*</script>", re.DOTALL)
def proxy_dict(session_id: str) -> dict:
user = f"{PROXY_USER}-session-{session_id}"
proxy_url = f"http://{user}:{PROXY_PASS}@{PROXY_HOST}:{PROXY_PORT}"
return {"http": proxy_url, "https": proxy_url}
def fetch_wayfair_product(slug: str, max_retries: int = 3) -> dict:
url = f"https://www.wayfair.com/furniture/pdp/{slug}.html"
proxies = proxy_dict(session_id=slug[:10])
for attempt in range(1, max_retries + 1):
try:
resp = requests.get(url, headers=HEADERS, proxies=proxies, timeout=20)
if resp.status_code == 200:
return parse_wayfair_page(resp.text, slug)
print(f"Attempt {attempt}: Wayfair returned status {resp.status_code}")
except requests.exceptions.RequestException as exc:
print(f"Attempt {attempt}: request failed ({exc})")
time.sleep(random.uniform(3, 6))
raise RuntimeError(f"Failed to fetch Wayfair product {slug} after {max_retries} attempts")
def parse_wayfair_page(html: str, slug: str) -> dict:
match = STATE_RE.search(html)
if not match:
return {"slug": slug, "error": "initial state not found"}
try:
data = json.loads(match.group(1))
except json.JSONDecodeError:
return {"slug": slug, "error": "could not decode product state"}
return {"slug": slug, "raw_keys": list(data.keys())[:5]}
if __name__ == "__main__":
for slug in ["sand-and-stable-queen-platform-bed"]:
print(fetch_wayfair_product(slug))
time.sleep(random.uniform(3, 6))
{
"slug": "sand-and-stable-queen-platform-bed",
"title": "Sand & Stable Queen Platform Bed",
"price": "$349.99",
"rating": "4.6 out of 5 stars",
"review_count": 2107
}Install requests for fetching product pages; the embedded state is parsed with Python's built-in json and re modules.
pip install requestsHold one IP for a short browsing run so session cookies remain valid across a handful of requests.
PROXY_HOST = "proxy.knoxproxy.com"
PROXY_PORT = 10000
PROXY_USER = "your_username"
PROXY_PASS = "your_password"Request the product URL and locate the embedded initial-state script tag in the HTML.
Parse the JSON blob for title, price, and rating rather than relying on rendered DOM elements.
Wayfair Furniture runs custom bot detection. The tested setup is residential rotating proxies, targeting uS residential IPs, with this rotation: Sticky session for 5-10 minutes per browsing run. That combination held a 80% success rate on the Jun 24, 2026 test run.
The proxy is half the job — sticky session for 5-10 minutes per browsing run is what turns a working request into a repeatable one.
Sticky session for 5-10 minutes per browsing run.
US residential IPs.
Wayfair Furniture leans on "Verify you are human" interstitial redirect. Wayfair's storefront runs on a GraphQL backend that validates request signing and session cookies; unauthenticated or high-velocity datacenter traffic is redirected to a verification page within the first few requests.
Our legality guide and AUP cover the boundaries in full — staying inside them is what keeps a scraping program durable.
Wayfair's storefront runs on a GraphQL backend with request signing and session validation, which redirects unauthenticated or high-velocity traffic to a verification page faster than a typical server-rendered catalog.
Residential rotating proxies held on a short sticky session, roughly 5-10 minutes, work best so session cookies issued at the start of a browsing run stay valid.
Shipping estimates and occasionally promotional pricing can vary by the visitor's apparent location, so matching the proxy region to the market being studied improves accuracy.
Review counts and average ratings appear in the same embedded state object as price and title, though full review text typically loads through a separate paginated request.
The Python code above is a tested Wayfair scraper that parses the embedded JSON state object. Run it as-is or extend it; sticky-session handling and proxy rotation are already built in.
Free trial on residential proxies -- 80% tested success, no credit card.