A tested walkthrough for Walmart Products -- the right proxy type, 85% tested success rate, working code, and what to avoid to stay compliant.
Walmart product pages expose title, price, and availability through an embedded JSON state object, best reached with residential rotating proxies on sticky sessions. Walmart runs PerimeterX (HUMAN Security) bot mitigation, which blocklists datacenter IPs almost immediately, and KnoxProxy residential sessions held an 85% success rate on product pages. A Walmart scraper built this way holds up for both single product checks and larger catalog sweeps.
| Anti-bot system | PerimeterX (HUMAN Security) bot mitigation |
| Challenge types | px-captcha interstitial, JavaScript challenge requiring headless-browser execution, Cookie-based session fingerprinting (_px3 cookie), Access-denied block pages for flagged IP/device combinations |
| Rate limit behavior | PerimeterX scores every session in real time; datacenter IPs and requests missing a matching browser fingerprint are blocked almost immediately, while well-paced residential sessions with realistic headers sustain longer runs before a challenge appears. |
| Tested success rate | 85% |
"""
KnoxProxy scraper for Walmart product pages.
Walmart's storefront is a Next.js app, so product data lives in an
embedded __NEXT_DATA__ JSON blob rather than plain HTML attributes.
"""
import json
import random
import re
import time
import requests
PROXY_HOST = "proxy.knoxproxy.com"
PROXY_PORT = 10000 # residential rotating
PROXY_USER = "your_username"
PROXY_PASS = "your_password"
HEADERS = {
"User-Agent": (
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
"(KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36"
),
"Accept-Language": "en-US,en;q=0.9",
}
NEXT_DATA_RE = re.compile(
r'<script id="__NEXT_DATA__" type="application/json">(.*?)</script>', re.DOTALL
)
def proxy_dict(session_id: str) -> dict:
user = f"{PROXY_USER}-session-{session_id}"
proxy_url = f"http://{user}:{PROXY_PASS}@{PROXY_HOST}:{PROXY_PORT}"
return {"http": proxy_url, "https": proxy_url}
def fetch_walmart_product(item_id: str, max_retries: int = 3) -> dict:
url = f"https://www.walmart.com/ip/{item_id}"
proxies = proxy_dict(session_id=item_id)
for attempt in range(1, max_retries + 1):
try:
resp = requests.get(url, headers=HEADERS, proxies=proxies, timeout=20)
if resp.status_code == 200:
return parse_walmart_next_data(resp.text, item_id)
print(f"Attempt {attempt}: Walmart returned status {resp.status_code}")
except requests.exceptions.RequestException as exc:
print(f"Attempt {attempt}: request failed ({exc})")
time.sleep(random.uniform(2, 5))
raise RuntimeError(f"Failed to fetch Walmart item {item_id} after {max_retries} attempts")
def parse_walmart_next_data(html: str, item_id: str) -> dict:
match = NEXT_DATA_RE.search(html)
if not match:
return {"item_id": item_id, "error": "product JSON not found"}
data = json.loads(match.group(1))
product = (
data.get("props", {})
.get("pageProps", {})
.get("initialData", {})
.get("data", {})
.get("product", {})
)
return {
"item_id": item_id,
"name": product.get("name"),
"price": product.get("priceInfo", {}).get("currentPrice", {}).get("priceString"),
"availability": product.get("availabilityStatus"),
"brand": product.get("brand"),
}
if __name__ == "__main__":
item_ids = ["414989451", "5085077954"]
for item_id in item_ids:
print(fetch_walmart_product(item_id))
time.sleep(random.uniform(3, 6))
{
"item_id": "414989451",
"name": "Great Value Purified Drinking Water, 16.9 fl oz, 40 Count",
"price": "$5.98",
"availability": "IN_STOCK",
"brand": "Great Value"
}Install requests for fetching pages and the built-in json module for parsing Walmart's embedded state object.
pip install requestsUse a session ID in the proxy username so the same product page loads through one consistent IP.
PROXY_HOST = "proxy.knoxproxy.com"
PROXY_PORT = 10000
PROXY_USER = "your_username"
PROXY_PASS = "your_password"Request the product detail page and locate the __NEXT_DATA__ script tag Walmart's Next.js frontend embeds.
Parse the embedded JSON rather than the rendered HTML, since Walmart hydrates most product data client-side.
Walmart Products runs perimeterX (HUMAN Security) bot mitigation. The tested setup is residential rotating proxies, targeting uS residential IPs, with this rotation: Sticky session per product page for 5-10 minutes, rotate IP between products. That combination held a 85% success rate on the Jul 2, 2026 test run.
The proxy is half the job — sticky session per product page for 5-10 minutes, rotate IP between products is what turns a working request into a repeatable one.
Sticky session per product page for 5-10 minutes, rotate IP between products.
US residential IPs.
Walmart Products leans on px-captcha interstitial. PerimeterX scores every session in real time; datacenter IPs and requests missing a matching browser fingerprint are blocked almost immediately, while well-paced residential sessions with realistic headers sustain longer runs before a challenge appears.
Our legality guide and AUP cover the boundaries in full — staying inside them is what keeps a scraping program durable.
Walmart uses PerimeterX (HUMAN Security) to score every session in real time and blocks datacenter IPs and non-browser traffic patterns almost immediately. It also tracks sessions with a _px3 fingerprint cookie and can serve a px-captcha interstitial. Residential IPs with realistic headers and steady pacing get considerably further before a challenge appears.
Walmart's storefront is built on Next.js, which hydrates the page client-side from a __NEXT_DATA__ script tag rather than rendering price and stock into static markup. Parsing that JSON directly and pulling name, price, availability, and brand fields from it is far more reliable than trying to select elements out of the rendered HTML.
A sticky residential session of 5-10 minutes per product page, then rotating to a new IP for the next product, balances PerimeterX's session scoring against the need to move through a catalog efficiently. Pace requests to roughly one page every 4-6 seconds per IP, since rotating mid-page instead of between products re-triggers the JavaScript challenge.
Store-specific pricing requires passing a store ID or ZIP code parameter recognized by Walmart's fulfillment logic, since the standard product page returns national online pricing by default. The scraper above does not pass a store parameter, so add one to the request URL if per-location price comparison is the goal.
The Python code above is a tested Walmart scraper that parses the __NEXT_DATA__ JSON directly instead of scraping rendered HTML. Run it as-is or extend the field list; proxy rotation, sticky sessions, and retry logic are already built in, and this exact setup held an 85% success rate on product pages during the last test cycle.
residential proxies -- 85% tested success, instant activation, 14-day money-back guarantee.