DEFGlossary · Web Scraping

What is User Agent?

Definition

A user agent is an HTTP header string that identifies the client software making a request, including the browser name, version, operating system, and rendering engine. Servers use this information to tailor responses.

How User Agent works

Every HTTP request includes a User-Agent header. A typical Chrome user agent looks like: "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36". Servers parse this string to determine the client type. Web scrapers set custom user agents to match real browsers and avoid detection.

Your scraperissues the request

→

Proxy poolassigns an exit IP

→

Target pagereturns the response

The proxy layer sits between your code and the page it reads.

Building around User Agent

Handle it deliberately in production scrapers -- most breakage traces back to skipping this step.

Handle it deliberately when

Building a scraper meant to run repeatedly, not once
Debugging incomplete or inconsistent results
Writing a pipeline someone else will maintain
Scaling past a handful of manual requests

A simple approach is fine when

Pulling data once, by hand, in a browser tab
The site offers the same data through an official API
The job is small enough to check by eye
A scraping framework already handles it for you

Putting User Agent into practice

proxy credentials

USER-country-de-session-task01

Add this string to your scraper's proxy credentials and every request in the job shares one exit IP, which keeps user agent-related behavior consistent across the run. Change "task01" per worker to isolate parallel scrapes.

Build it once, reuse it

Isolate the logic for this step so every scraper in the project shares one tested implementation.

Re-verify on a schedule

Sites change layouts and behavior over time -- recheck this part of the scraper on a schedule, not just at launch.

Pair it with a real proxy

This works best over residential or ISP IPs, so the target sees ordinary browsing rather than clustered datacenter traffic.

Log failures, do not just retry

Capture what actually failed so a broken selector or a new status code surfaces instead of getting masked by automatic retries.

Practical example

A scraping framework rotates through a list of 50 real browser user agent strings, selecting one randomly for each session to avoid pattern-based detection.

Why it matters

Using the default user agent of your HTTP library (like python-requests/2.31.0) immediately flags your traffic as automated. Setting a realistic browser user agent is the most basic anti-detection step.

How often should I rotate user agents?

Rotate user agents per session, not per request. Real browsers keep the same user agent for the entire browsing session. Changing it on every request is a detectable pattern in itself.

Where can I find current real browser user agent strings?

Check your own browser at whatismybrowser.com or use databases like useragentstring.com. Update your user agent list every few months since browser versions change frequently.

RELRelated terms

HTTP Headers Browser Fingerprinting Web Scraping Headless Browser Session Management Full glossary — 157 terms

In practice

Ready to put this into practice? User Agent Checker

See it in action

Start a free trial and test with real targets -- no credit card, no sales call.

Start for free Browse glossary