The essential points from this guide -- each one is explained in detail below.
Check a site's RSS or Atom feed first -- it often already has headline, byline, and summary data without any HTML parsing.
A news article page usually follows a fixed pattern: one h1 headline, a byline element, and body text split across p tags inside an article container.
Metered paywalls hide already-loaded HTML with CSS or JavaScript; only collect what a logged-out visitor legitimately sees, never bypass a login or subscription gate.
Large publishers commonly sit behind Cloudflare, and rotating residential proxies lower block rates because each request exits from a different real residential IP.
Read robots.txt and the site's terms before scraping at volume, and rate-limit requests to a level a small newsroom's server can handle.
Most news publishers still run an RSS or Atom feed that lists every new story with a headline, byline, publish date, and short summary in a fixed XML format. Checking that feed first saves you from parsing HTML at all, because feeds already give you the exact fields most projects need. Look for a feed at predictable paths like /feed, /rss.xml, or /atom.xml, or check the page's link rel="alternate" tag in the raw HTML. In Python, feedparser turns a feed URL into a list of dictionaries with title, link, published, and summary keys in a few lines of code:
import feedparser
feed = feedparser.parse('https://example-news.com/feed')
for entry in feed.entries:
print(entry.title, entry.link, entry.published)Feeds usually stop short of the full article body and drop anything sitting behind a paywall, so treat them as the fast path for headline and metadata collection, not a full substitute for scraping news articles at the content level. When a feed does not exist or is missing fields you need, move on to parsing the actual page.
A typical news article page follows a consistent pattern: one h1 tag for the headline, a byline near the top wrapped in a class like "byline" or an a rel="author" tag, and the body text sitting inside an article tag or a div such as "article-body" or "story-content", split across p tags. Learning how to scrape news articles comes down to finding those three selectors once per site template and reusing them across every story on that domain. Start with your browser's inspector, click the headline, and read the class name attached to it, then repeat for the byline and body container. Once you have those, BeautifulSoup pulls all three fields in a handful of lines:
import requests
from bs4 import BeautifulSoup
r = requests.get(url, headers={'User-Agent': 'Mozilla/5.0'})
soup = BeautifulSoup(r.text, 'html.parser')
headline = soup.select_one('h1').get_text(strip=True)
byline = soup.select_one('.byline, [rel="author"]').get_text(strip=True)
paragraphs = [p.get_text(strip=True) for p in soup.select('article p, .article-body p')]
body_text = '\n'.join(paragraphs)For sites where the layout changes often, the newspaper3k library auto-detects headline, author, and body text without hand-written selectors, trading some accuracy for less maintenance. Read what web scraping is if the request/parse/store cycle above is new to you.
News sites use two paywall types: a hard paywall that blocks the page entirely before you log in, and a metered paywall that lets a visitor read a set number of free articles before blocking further requests. A metered wall usually still sends the full article in the initial HTML response and hides it with CSS or a JavaScript overlay after a few page views, which is why a script with no browser session or cookies can sometimes read further than a person reliably would. Check the page's JSON-LD block for an isAccessibleForFree field, since many publishers set it explicitly to tell search engines and scrapers whether a story sits behind the wall. The responsible approach only collects what a logged-out visitor legitimately sees: the headline, byline, and the free preview paragraphs or meta description, never content unlocked by bypassing a login, cookie check, or subscription gate. If a project needs the full text of paywalled stories at scale, look for a licensed data feed or partner API from the publisher instead of routing around their access controls, since circumventing a paywall's technical protection can trigger both a breach-of-terms claim and, in the US, exposure under anti-circumvention rules.
Most news archives expose a category or date page with a "next" link or numbered pagination at the bottom, and the fastest way to scrape news articles across a date range is to follow that chain link by link instead of guessing at URL patterns. Start from the archive's first page, collect every article link on it, follow the "next" href, and repeat until the page stops returning one. Keep a set of already-seen URLs so a story that appears on two overlapping archive pages only gets scraped once:
seen = set()
url = 'https://example-news.com/archive/2026/07'
while url:
r = requests.get(url, headers={'User-Agent': 'Mozilla/5.0'})
soup = BeautifulSoup(r.text, 'html.parser')
for a in soup.select('.archive-item a'):
link = a['href']
if link not in seen:
seen.add(link)
next_link = soup.select_one('a.next-page')
url = next_link['href'] if next_link else None
time.sleep(2)Add a short delay between requests so the archive crawl does not read as a burst of automated traffic hitting one page every few milliseconds.
Large news publishers commonly sit behind Cloudflare or a similar CDN, and that CDN watches for the request patterns a scraper produces: many requests from one IP, no cookies, and a User-Agent that does not match a real browser. When it flags that pattern, it serves a challenge page instead of the article, which is the single biggest reason web scraping news articles projects stall on major outlets rather than smaller local sites. Rotating residential proxies lower the block rate because each request exits from a different real residential IP address instead of a shared datacenter range that Cloudflare already associates with bot traffic. Rotating proxies handle the IP-cycling automatically through one gateway endpoint, and residential proxies specifically are the standard choice when a target's anti-bot system is aggressive. KnoxProxy prices residential access at $2.10 per GB with coverage across 195+ countries, which matters if a project needs to read a publisher's regional or country-specific edition rather than only its default one. Check current plans on the pricing page before committing a scraper to a specific proxy type.
Before scraping at any real volume, open the site's robots.txt file at the domain root and read which paths are disallowed for your user agent, since a publisher's own crawl rules are the clearest signal of what they consider acceptable automated access. Robots.txt is a convention rather than a legal wall, so also check the site's terms of service page, because some publishers explicitly prohibit automated collection even on paths robots.txt leaves open. Identify your scraper with a real, descriptive User-Agent string rather than spoofing a browser's, and rate-limit requests to a level a small newsroom's server can handle without noticing, generally one request every few seconds rather than dozens per second. A similar rate-limiting approach applies to scraping Google, where respecting the target's stated limits keeps a project running instead of triggering a full IP ban. Following these rules protects both the scraper's access and the smaller publishers who cannot absorb a sudden traffic spike.
Ready to put this into practice? See web scraping proxies
KnoxProxy Research Team · Technical Content
Network engineers and proxy infrastructure specialists with 10+ years in anti-bot systems, web scraping, and IP routing.
90.4M+ ethically sourced residential IPs across 195 countries. Instant activation, 14-day money-back guarantee.