A production price monitoring system has five stages: scheduling (when to check), fetching (getting the page through proxies), parsing (extracting the price), storage (persisting the data), and alerting (reacting to changes). Each stage has distinct failure modes and scaling characteristics.
The architecture matters because price monitoring pipelines run continuously. A poorly designed pipeline wastes proxy bandwidth on unnecessary checks, misses price changes due to parsing failures, or generates false alerts from transient errors. Getting the architecture right saves money and produces better data.
Not every product needs checking at the same frequency. Electronics prices change multiple times per day. Furniture prices change weekly. Books almost never change. An intelligent scheduler adjusts cadence based on observed volatility.
Start with daily checks for all products. After a week of data, calculate the change frequency per product category. Products that changed zero times in a week move to weekly checks. Products that changed daily move to hourly. This simple rule cuts total requests by 40-60% with no loss of price change coverage.
Separate scheduling, fetching, parsing, storage, and alerting so each stage can fail and recover independently.
The fetcher sends requests through KnoxProxy residential proxies with country-level targeting. Each market gets its own country header, so the retailer returns localized prices. Per-request rotation ensures each check arrives from a fresh IP.
Build retry logic around HTTP status codes, not content. A 403 means the request was blocked -- retry with a different IP. A 200 with unexpected content (CAPTCHA page instead of product page) requires content validation before retry. A 429 means rate limiting -- back off before retrying. Log every response status for monitoring.
Product pages change layout frequently. A parser that relies on CSS selectors breaks every time the target redesigns. A better approach: extract structured data first. Check for JSON-LD product schema, Open Graph price tags, and data attributes before falling back to HTML parsing.
Maintain a validation basket -- a set of products with known prices that you verify manually. Run the parser against this basket after every code change and on a daily schedule. If accuracy drops below 98%, the parser needs updating before production data is affected.
Price data is a time series. Each row is a product ID, timestamp, market, and price. Use a time-series database or a partitioned table in PostgreSQL. Partition by date so old data can be archived without affecting query performance on recent data.
Store raw and normalized prices separately. The raw price is exactly what the page showed (including currency symbol and format). The normalized price converts to a common currency for comparison. Keep the raw value -- it is your audit trail.
The alerting system compares each new price against the previous value and a rolling average. A price change alert fires when the delta exceeds a threshold (e.g., more than 5% change). A trend alert fires when the rolling average shifts direction.
Suppress alerts during known sale events (Black Friday, Prime Day) unless the price drops below the historical floor. Alert fatigue kills monitoring pipelines -- if your team ignores 90% of alerts, the threshold is too sensitive.
import requests
import json
from datetime import datetime
PROXY = "http://USER:PASS@gw.knoxproxy.com:7000"
def fetch_price(url, country):
r = requests.get(url, proxies={"https": PROXY},
headers={"x-kx-country": country})
r.raise_for_status()
return r.text
def parse_price(html):
# Prefer JSON-LD over CSS selectors
if '"@type":"Product"' in html:
ld = json.loads(extract_jsonld(html))
return float(ld["offers"]["price"])
return parse_html_fallback(html)
def monitor(products, markets):
for p in products:
for cc in markets:
html = fetch_price(p["url"], cc)
price = parse_price(html)
store(p["id"], cc, price, datetime.utcnow())
check_alert(p["id"], cc, price)Build five stages, not one script. Separate scheduling, fetching, parsing, storage, and alerting so each can fail and recover independently. Start with daily cadence and let observed volatility drive frequency -- this alone cuts proxy costs by 40-60%.
Start free, run a thousand requests, and log the exit IPs. The difference is measurable in the first batch.