A tested walkthrough for Walmart Products -- the right proxy type, 85% tested success rate, working code, and what to avoid to stay compliant.
Walmart product pages expose title, price, and availability through an embedded JSON state object, best reached with residential rotating proxies on sticky sessions. Walmart runs PerimeterX (HUMAN Security) bot mitigation, which blocklists datacenter IPs almost immediately, and KnoxProxy residential sessions held an 85% success rate on product pages. A Walmart scraper built this way holds up for both single product checks and larger catalog sweeps.
| Anti-bot system | PerimeterX (HUMAN Security) bot mitigation |
| Challenge types | px-captcha interstitial, JavaScript challenge requiring headless-browser execution, Cookie-based session fingerprinting (_px3 cookie), Access-denied block pages for flagged IP/device combinations |
| Rate limit behavior | PerimeterX scores every session in real time; datacenter IPs and requests missing a matching browser fingerprint are blocked almost immediately, while well-paced residential sessions with realistic headers sustain longer runs before a challenge appears. |
| Tested success rate | 85% |
"""
KnoxProxy scraper for Walmart product pages.
Walmart's storefront is a Next.js app, so product data lives in an
embedded __NEXT_DATA__ JSON blob rather than plain HTML attributes.
"""
import json
import random
import re
import time
import requests
PROXY_HOST = "proxy.knoxproxy.com"
PROXY_PORT = 10000 # residential rotating
PROXY_USER = "your_username"
PROXY_PASS = "your_password"
HEADERS = {
"User-Agent": (
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
"(KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36"
),
"Accept-Language": "en-US,en;q=0.9",
}
NEXT_DATA_RE = re.compile(
r'<script id="__NEXT_DATA__" type="application/json">(.*?)</script>', re.DOTALL
)
def proxy_dict(session_id: str) -> dict:
user = f"{PROXY_USER}-session-{session_id}"
proxy_url = f"http://{user}:{PROXY_PASS}@{PROXY_HOST}:{PROXY_PORT}"
return {"http": proxy_url, "https": proxy_url}
def fetch_walmart_product(item_id: str, max_retries: int = 3) -> dict:
url = f"https://www.walmart.com/ip/{item_id}"
proxies = proxy_dict(session_id=item_id)
for attempt in range(1, max_retries + 1):
try:
resp = requests.get(url, headers=HEADERS, proxies=proxies, timeout=20)
if resp.status_code == 200:
return parse_walmart_next_data(resp.text, item_id)
print(f"Attempt {attempt}: Walmart returned status {resp.status_code}")
except requests.exceptions.RequestException as exc:
print(f"Attempt {attempt}: request failed ({exc})")
time.sleep(random.uniform(2, 5))
raise RuntimeError(f"Failed to fetch Walmart item {item_id} after {max_retries} attempts")
def parse_walmart_next_data(html: str, item_id: str) -> dict:
match = NEXT_DATA_RE.search(html)
if not match:
return {"item_id": item_id, "error": "product JSON not found"}
data = json.loads(match.group(1))
product = (
data.get("props", {})
.get("pageProps", {})
.get("initialData", {})
.get("data", {})
.get("product", {})
)
return {
"item_id": item_id,
"name": product.get("name"),
"price": product.get("priceInfo", {}).get("currentPrice", {}).get("priceString"),
"availability": product.get("availabilityStatus"),
"brand": product.get("brand"),
}
if __name__ == "__main__":
item_ids = ["414989451", "5085077954"]
for item_id in item_ids:
print(fetch_walmart_product(item_id))
time.sleep(random.uniform(3, 6))
{
"item_id": "414989451",
"name": "Great Value Purified Drinking Water, 16.9 fl oz, 40 Count",
"price": "$5.98",
"availability": "IN_STOCK",
"brand": "Great Value"
}Install requests for fetching pages and the built-in json module for parsing Walmart's embedded state object.
pip install requestsUse a session ID in the proxy username so the same product page loads through one consistent IP.
PROXY_HOST = "proxy.knoxproxy.com"
PROXY_PORT = 10000
PROXY_USER = "your_username"
PROXY_PASS = "your_password"Request the product detail page and locate the __NEXT_DATA__ script tag Walmart's Next.js frontend embeds.
Parse the embedded JSON rather than the rendered HTML, since Walmart hydrates most product data client-side.
Walmart Products runs perimeterX (HUMAN Security) bot mitigation. The tested setup is residential rotating proxies, targeting uS residential IPs, with this rotation: Sticky session per product page for 5-10 minutes, rotate IP between products. That combination held a 85% success rate on the Jul 2, 2026 test run.
The proxy is half the job — sticky session per product page for 5-10 minutes, rotate IP between products is what turns a working request into a repeatable one.
Sticky session per product page for 5-10 minutes, rotate IP between products.
US residential IPs.
Walmart Products leans on px-captcha interstitial. PerimeterX scores every session in real time; datacenter IPs and requests missing a matching browser fingerprint are blocked almost immediately, while well-paced residential sessions with realistic headers sustain longer runs before a challenge appears.
Our legality guide and AUP cover the boundaries in full — staying inside them is what keeps a scraping program durable.
Walmart uses PerimeterX (HUMAN Security) to score sessions and blocks datacenter IPs and non-browser traffic patterns quickly. Residential IPs with realistic headers and pacing get considerably further before a challenge appears.
Walmart's storefront is built on Next.js, which hydrates the page client-side from a __NEXT_DATA__ script tag. Parsing that JSON directly is more reliable than trying to select rendered HTML elements.
A sticky residential session of 5-10 minutes per product page, then rotating to a new IP for the next product, balances PerimeterX's session scoring against the need to move through a catalog efficiently.
Store-specific pricing requires passing a store ID or ZIP code parameter recognized by Walmart's fulfillment logic; national online pricing is available without one on the standard product page.
The Python code above is a tested Walmart scraper that parses the __NEXT_DATA__ JSON directly. Run it as-is or extend the field list; proxy rotation and retry logic are already built in.
Free trial on residential proxies -- 85% tested success, no credit card.