Web Scraping with Node.js: A Complete Guide

Cheerio parses HTML on the server using the same selector syntax as jQuery, without loading a browser or running any JavaScript. It is one of the most common tools for scraping with node js on static pages that do not depend on client-side rendering. Install it with npm install cheerio alongside an HTTP client such as node-fetch or Axios to fetch the raw page first. See what web scraping is for a broader definition if this is your first scraping project.

A typical cheerio scraper fetches a URL, loads the response body into Cheerio, and selects elements with CSS-style selectors, the same approach used in cheerio web scraping tutorials across the ecosystem:

import fetch from 'node-fetch';
import * as cheerio from 'cheerio';

const res = await fetch('https://example.com/products');
const html = await res.text();
const $ = cheerio.load(html);

const titles = [];
$('.product-title').each((i, el) => {
  titles.push($(el).text().trim());
});

console.log(titles);

This pattern covers most web scraping cheerio use cases: product listings, article titles, prices, and links. Cheerio has no rendering engine, so it only sees the HTML returned by the server, not content added later by client-side JavaScript. For sites that build their content with a framework like React or Vue, a cheerio scraper returns an empty or near-empty page. That is the signal to switch to Puppeteer, which is covered next.

Axios and node-fetch are the two HTTP clients most Node.js scrapers use to fetch page HTML before handing it to Cheerio. Both send a GET request and return the response body as text, but they differ in API shape and built-in features.

Axios includes automatic JSON parsing, request and response interceptors, and built-in timeout handling:

import axios from 'axios';

const response = await axios.get('https://example.com/products', {
  timeout: 10000,
  headers: { 'User-Agent': 'Mozilla/5.0' },
});
const html = response.data;

node-fetch mirrors the browser fetch API and requires an extra step to read the response body as text:

import fetch from 'node-fetch';

const res = await fetch('https://example.com/products', {
  headers: { 'User-Agent': 'Mozilla/5.0' },
});
const html = await res.text();

Either client works well for node scraping inside a node web scraper. Axios is the more common default in production scrapers because of its interceptor system, which makes it simple to attach a proxy agent and retry logic to every request in one place, rather than repeating configuration on each call. Whichever client is chosen, set a realistic User-Agent header, since many sites block requests from clients using default library user agents. Either pattern is enough to node scrape website targets that return their content directly in the initial HTML response.

A Node.js scraper needs a proxy to avoid IP bans once it sends more than a handful of requests to the same site. Pass a proxy agent to Axios or node-fetch instead of rewriting each request's connection logic by hand.

Create an HttpsProxyAgent once and attach it to every Axios request:

import axios from 'axios';
import { HttpsProxyAgent } from 'https-proxy-agent';

const agent = new HttpsProxyAgent('http://USER:PASS@gw.knoxproxy.com:7000');

const response = await axios.get('https://example.com/products', {
  httpsAgent: agent,
  timeout: 10000,
});

node-fetch uses the same agent object:

import fetch from 'node-fetch';
import { HttpsProxyAgent } from 'https-proxy-agent';

const agent = new HttpsProxyAgent('http://USER:PASS@gw.knoxproxy.com:7000');
const res = await fetch('https://example.com/products', { agent });
const html = await res.text();

Puppeteer needs a different approach since Chrome's --proxy-server flag does not accept credentials directly; see the dedicated Puppeteer proxy guide for the proxy-chain package pattern that handles authentication.

This is the standard javascript web scraping setup for any nodejs scraper running at volume: KnoxProxy's backconnect gateway rotates the exit IP on each request, so a target site sees varied traffic instead of one address hammering it thousands of times. KnoxProxy residential proxies start at $2.10/GB, and rotating proxies handle the IP-switching automatically without any extra code in the scraper. See web scraping proxy plans and current pricing before choosing a package for a production scraper.