Working, tested guides for 30+ popular websites -- each one covers web page scraping the way the live target really behaves: the anti-bot system in play, the right proxy type, working Python code, and a measured success rate. New to the topic? Start with what web scraping is.
Retail and marketplace sites run some of the toughest anti-bot systems on the web, since scraped pricing and inventory data carries real competitive value. These guides cover product titles, prices, stock status, and ratings across major marketplaces and retailers -- Amazon, eBay, Walmart, and more -- with the residential or datacenter proxy setup each target actually needs. Every ecommerce scraper here handles pagination, variant selection, and rate limiting the way the live site behaves today, not a simplified demo version.
Amazon product titles, prices, ratings, and availability are scrapable with residential rotating proxies routed through US IPs.
residentialGuide →eBay ListingsAn eBay scraper can pull listing titles, prices, shipping cost, and seller ratings using datacenter proxies for casual lookups or residential rotating proxies for continuous tracking.
residentialGuide →Walmart ProductsWalmart product pages expose title, price, and availability through an embedded JSON state object, best reached with residential rotating proxies on sticky sessions.
residentialGuide →AliExpress ProductsAliExpress product title, price range, orders count, and store rating live in an embedded window.
residentialGuide →Shopee MarketplaceShopee's internal item API returns clean JSON for price, stock, and rating data when reached through residential rotating proxies matched to the regional domain.
residentialGuide →Etsy HandmadeAn Etsy scraper can collect listing titles, prices, and shop names using residential rotating proxies at a light, human-paced request rate.
residentialGuide →Target ProductsTarget's RedSky product API returns structured price, availability, and rating data directly and pairs well with residential rotating proxies.
residentialGuide →Best Buy ElectronicsA Best Buy scraper can reach SKU pages with residential rotating proxies, though the official Products API offers a more durable path at higher volume.
residentialGuide →Wayfair FurnitureWayfair product title, price, and rating are recoverable from an embedded JSON state object, best reached with residential rotating proxies on a short sticky session.
residentialGuide →Nike SNKRS ReleasesNike's public product feed API exposes upcoming SNKRS release data and is best monitored with a mix of residential and mobile rotating proxies.
residentialGuide →Alibaba B2BAn Alibaba scraper can collect supplier listing titles, price ranges, and minimum order quantities with residential rotating proxies at a light request pace.
residentialGuide →StockX Sneaker MarketStockX last-sale price, bid/ask spread, and size-chart data are reachable through its public GraphQL market-data API using residential rotating proxies.
residentialGuide →Taobao MarketplaceA Taobao scraper can reach search-result titles and prices without login using China-region residential rotating proxies; full detail requires an authenticated session outside this guide's scope.
residentialGuide →Public profile data, post metadata, and engagement counts are reachable on every platform below without logging in or reaching past a login wall. These guides show how social media scraping works in practice for Instagram, YouTube, Twitter/X, Facebook, Pinterest, and Reddit -- what the public endpoint actually returns, which anti-bot challenge to expect, and the residential rotation pattern that keeps a script running past a few hundred requests. Each guide sticks to data the platform already renders publicly.
Instagram scraping without logging in reaches profile bio, follower count, and post count through the web profile info endpoint, using residential rotating proxies.
residentialGuide →LinkedIn Public ProfilesA LinkedIn scraper can reach public profile and company page data in limited amounts via residential rotating proxies before a login wall appears.
residentialGuide →TikTok VideosTikTok video stats such as play count, like count, and share count are recoverable from the embedded SIGI_STATE JSON blob on video and profile pages using residential rotating proxies.
residentialGuide →Twitter/X PostsA Twitter/X scraper can reach individual post text, like count, and retweet count without login through the public syndication endpoint, using residential rotating proxies.
residentialGuide →Facebook Public PagesA Facebook scraper can reach Page name, about text, and public post content via the mobile-optimized site using residential rotating proxies.
residentialGuide →Pinterest PinsPinterest pin image URLs, descriptions, and stats are recoverable from the embedded PWS_DATA JSON blob using datacenter rotating proxies for most workloads.
residentialGuide →YouTube Video DataA YouTube scraper can reach video title, view count, like count, and description through the embedded ytInitialData JSON blob or the official Data API v3, both workable with residential rotating proxies.
residentialGuide →Reddit Posts & CommentsReddit posts and comments are directly available as JSON through the public .
residentialGuide →Search results change by IP, language, and location, so one scrape from one server says little about what a real searcher in a specific market actually sees. These search engine scraping guides cover Google and Baidu results collection for rank tracking, SERP feature audits -- featured snippets, People Also Ask, local packs -- and competitive research, with the geo-targeted proxy rotation needed to match results to a market instead of wherever the scraping server happens to sit.
A Google SERP scraper can pull search-result rankings, titles, and snippets with residential rotating proxies matched to the query's country and language edition.
residentialGuide →Baidu Search ResultsA Baidu scraper can collect search-result titles, links, and snippets with China-region residential rotating proxies at a light query pace.
residentialGuide →Hotels, review platforms, real estate portals, job boards, and classifieds hold data that changes by city and by the day -- room rates, review counts, listing prices, and job postings. These guides cover Booking.com, TripAdvisor, Yelp, Zillow, Indeed, Glassdoor, and Craigslist, each with the anti-bot profile that platform runs today and the proxy targeting that returns local, market-specific results instead of a generic average. All of it comes from public pages -- no login, no paywall, no review text pulled from behind an account wall.
A Booking.
residentialGuide →TripAdvisor ReviewsA TripAdvisor scraper can pull review text, ratings, and titles from listing pages with residential rotating proxies at a moderate pace.
residentialGuide →Yelp Business ReviewsA Yelp scraper can pull business name, rating, review count, and address from business pages with residential rotating proxies at a light pace.
residentialGuide →Zillow Property ListingsZillow property price, address, Zestimate, and bed/bath count are recoverable from an embedded JSON blob on listing pages using residential rotating proxies.
residentialGuide →Indeed Job ListingsAn Indeed scraper can pull job title, company, salary range, and location from search-result pages with residential rotating proxies at a moderate pace.
residentialGuide →Glassdoor Company ReviewsA Glassdoor scraper can reach company name, overall rating, and review count anonymously through structured data on the company overview page, using residential rotating proxies.
residentialGuide →Craigslist ListingsA Craigslist scraper can pull listing titles, prices, and neighborhoods from city-specific search pages with datacenter rotating proxies at a slow, steady pace.
residentialGuide →Search for how to scrape a website and three kinds of results turn up: managed APIs that hand back parsed data, proxy services you pair with a scraper you write, and tutorials for building that scraper yourself. All three solve real problems, and the right choice depends on what you are building.
A scraping API takes a URL and returns clean data -- HTML, text, or structured JSON -- without you managing requests, retries, or IP rotation. This is the fastest path to a working pipeline when the target site is not the main product you are building. The tradeoff: less control over exactly what gets extracted.
This is what every guide below uses: your own Requests or Scrapy code, routed through rotating residential or datacenter proxies depending on how well defended the target is. More setup than a managed API, full control over parsing, and no per-record fees once it is running. See proxy fit by workload.
Requests and BeautifulSoup handle most static pages; Selenium or Playwright pick up anything that needs a real browser. Configure the proxy once, then reuse the same setup across targets. Every guide on this page includes working code -- copy it, or use it as a reference for your own scraper.
Every guide on this page covers public data only -- the pages, endpoints, and fields a site already shows to any visitor, with no login-wall or paywall circumvention. That scope is not just an ethics stance: in the US, the hiQ Labs v. LinkedIn (2022) ruling held that scraping publicly accessible data does not violate the Computer Fraud and Abuse Act, and later rulings extended the same reasoning to public social data. Respecting robots.txt on the target site, keeping request rates reasonable, and staying out of anything behind a login are the practical rules every guide here follows. For the full legal picture, including what changes once personal data or a paywall is involved, see is web scraping legal.
30 guides across 4 categories -- e-commerce, social media, search engines, and travel, reviews & listings -- each with the target's anti-bot profile, the right proxy type, and tested Python code.
Collecting publicly available data is broadly lawful in most jurisdictions, though it depends on the site, the data, and the method. Every guide here covers public data only, respects robots.txt, and never circumvents logins or paywalls.
It depends on the target. Permissive sites work with datacenter proxies; protected e-commerce and search sites need residential; the hardest anti-bot systems call for mobile IPs. Each guide names the recommended type up front.
Yes. Each guide includes a runnable Python sample and sample output, tested against the live target, plus a "last tested" date and success rate from the most recent run.
A scraper is a script or tool that requests a web page, reads the HTML or JSON it gets back, and pulls out specific fields -- prices, titles, ratings -- into structured data. Every guide on this page is a working example of web page scraping for one specific site.
Web scraping reads a page's underlying HTML or API response directly. Screen scraping reads what's rendered on screen instead -- useful for older systems or apps with no accessible markup, but slower and more fragile than parsing HTML or JSON directly. Every guide here uses the faster, HTML-based approach.
Pick a target, copy the tested setup, and run it on a free trial. Switch proxy type with one flag when a target gets harder.