A tested walkthrough for YouTube Video Data -- the right proxy type, 93% tested success rate, working code, and what to avoid to stay compliant.
A YouTube scraper reaches video title, view count, like count, and description through the embedded ytInitialData JSON blob or the official Data API v3, both workable with residential rotating proxies. The same approach doubles as a YouTube channel scraper, pulling an entire creator's video list instead of one watch page at a time. YouTube runs anti-automation checks and consent screens rather than a hard WAF, and KnoxProxy residential IPs held a 93% success rate on watch-page requests.
| Anti-bot system | Anti-automation checks + consent screens |
| Challenge types | EU/UK cookie-consent interstitial before content loads, "Sign in to confirm you're not a bot" prompt on repeated automated access, Per-IP rate limiting on watch, channel, and search pages, Per-endpoint rate limiting on the InnerTube API that powers comments and continuations, Periodic signature-cipher changes on stream URLs |
| Rate limit behavior | Metadata scraping such as title, views, likes, and description via the public watch page, channel page, or the official Data API v3 is tolerant at moderate volume; heavy automated traffic from one IP eventually triggers the "confirm you're not a bot" interstitial, and comment continuations tighten that threshold further. |
| Tested success rate | 93% |
"""
KnoxProxy scraper for public YouTube video metadata.
Extracts title, view count, like count, and description from the
ytInitialData JSON blob embedded in a public watch page.
"""
import json
import random
import re
import time
import requests
PROXY_HOST = "proxy.knoxproxy.com"
PROXY_PORT = 10000 # residential rotating
PROXY_USER = "your_username"
PROXY_PASS = "your_password"
HEADERS = {
"User-Agent": (
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
"(KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36"
),
"Accept-Language": "en-US,en;q=0.9",
}
INITIAL_DATA_RE = re.compile(r"var ytInitialData = (\{.*?\});</script>", re.DOTALL)
def proxy_dict(country: str = "us") -> dict:
user = f"{PROXY_USER}-country-{country}"
proxy_url = f"http://{user}:{PROXY_PASS}@{PROXY_HOST}:{PROXY_PORT}"
return {"http": proxy_url, "https": proxy_url}
def fetch_youtube_video(video_id: str, max_retries: int = 3) -> dict:
url = f"https://www.youtube.com/watch?v={video_id}"
cookies = {"CONSENT": "YES+1"} # skip the EU/UK consent interstitial
for attempt in range(1, max_retries + 1):
try:
resp = requests.get(
url, headers=HEADERS, cookies=cookies, proxies=proxy_dict(), timeout=20
)
if resp.status_code == 200:
return parse_youtube_video(resp.text, video_id)
print(f"Attempt {attempt}: YouTube returned status {resp.status_code}")
except requests.exceptions.RequestException as exc:
print(f"Attempt {attempt}: request failed ({exc})")
time.sleep(random.uniform(2, 5))
raise RuntimeError(f"Failed to fetch YouTube video {video_id} after {max_retries} attempts")
def parse_youtube_video(html: str, video_id: str) -> dict:
match = INITIAL_DATA_RE.search(html)
if not match:
return {"video_id": video_id, "error": "ytInitialData not found"}
data = json.loads(match.group(1))
results = data.get("contents", {}).get("twoColumnWatchNextResults", {})
primary = results.get("results", {}).get("results", {}).get("contents", [{}])[0]
video_info = primary.get("videoPrimaryInfoRenderer", {})
return {
"video_id": video_id,
"title": video_info.get("title", {}).get("runs", [{}])[0].get("text"),
"view_count": video_info.get("viewCount", {})
.get("videoViewCountRenderer", {})
.get("viewCount", {})
.get("simpleText"),
}
if __name__ == "__main__":
for video_id in ["dQw4w9WgXcQ"]:
print(fetch_youtube_video(video_id))
time.sleep(random.uniform(2, 5))
"""
KnoxProxy scraper for a YouTube channel's video list.
Fetches the channel's /videos tab and parses the ytInitialData grid,
so the same technique works as a channel-wide YouTube video scraper
instead of pulling one watch page at a time.
"""
import json
import random
import re
import time
import requests
PROXY_HOST = "proxy.knoxproxy.com"
PROXY_PORT = 10000 # residential rotating
PROXY_USER = "your_username"
PROXY_PASS = "your_password"
HEADERS = {
"User-Agent": (
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
"(KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36"
),
"Accept-Language": "en-US,en;q=0.9",
}
INITIAL_DATA_RE = re.compile(r"var ytInitialData = (\{.*?\});</script>", re.DOTALL)
def proxy_dict(session_id: str) -> dict:
user = f"{PROXY_USER}-session-{session_id}"
proxy_url = f"http://{user}:{PROXY_PASS}@{PROXY_HOST}:{PROXY_PORT}"
return {"http": proxy_url, "https": proxy_url}
def fetch_channel_videos(channel_handle: str, max_retries: int = 3) -> list[dict]:
url = f"https://www.youtube.com/@{channel_handle}/videos"
cookies = {"CONSENT": "YES+1"}
proxies = proxy_dict(session_id=channel_handle)
for attempt in range(1, max_retries + 1):
try:
resp = requests.get(url, headers=HEADERS, cookies=cookies, proxies=proxies, timeout=20)
if resp.status_code == 200:
return parse_channel_grid(resp.text)
print(f"Attempt {attempt}: channel page returned status {resp.status_code}")
except requests.exceptions.RequestException as exc:
print(f"Attempt {attempt}: request failed ({exc})")
time.sleep(random.uniform(2, 5))
raise RuntimeError(f"Failed to fetch channel '{channel_handle}' after {max_retries} attempts")
def parse_channel_grid(html: str) -> list[dict]:
match = INITIAL_DATA_RE.search(html)
if not match:
return []
data = json.loads(match.group(1))
tabs = (
data.get("contents", {})
.get("twoColumnBrowseResultsRenderer", {})
.get("tabs", [])
)
videos = []
for tab in tabs:
renderer = tab.get("tabRenderer", {})
if renderer.get("title") != "Videos":
continue
items = (
renderer.get("content", {})
.get("richGridRenderer", {})
.get("contents", [])
)
for item in items:
video = item.get("richItemRenderer", {}).get("content", {}).get("videoRenderer")
if not video:
continue
videos.append({
"video_id": video.get("videoId"),
"title": video.get("title", {}).get("runs", [{}])[0].get("text"),
"view_count_text": video.get("viewCountText", {}).get("simpleText"),
})
return videos
if __name__ == "__main__":
for video in fetch_channel_videos("examplechannel")[:10]:
print(video)
"""
KnoxProxy scraper for YouTube comment threads via the continuation
endpoint. Run after fetching a watch page and pulling its
continuation token out of ytInitialData.
"""
import random
import time
import requests
PROXY_HOST = "proxy.knoxproxy.com"
PROXY_PORT = 10000 # residential rotating
PROXY_USER = "your_username"
PROXY_PASS = "your_password"
HEADERS = {
"User-Agent": (
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
"(KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36"
),
"Content-Type": "application/json",
}
def proxy_dict() -> dict:
proxy_url = f"http://{PROXY_USER}:{PROXY_PASS}@{PROXY_HOST}:{PROXY_PORT}"
return {"http": proxy_url, "https": proxy_url}
def fetch_comment_page(continuation_token: str, api_key: str, max_retries: int = 3) -> dict:
url = f"https://www.youtube.com/youtubei/v1/next?key={api_key}"
payload = {
"context": {"client": {"clientName": "WEB", "clientVersion": "2.20260701.00.00"}},
"continuation": continuation_token,
}
for attempt in range(1, max_retries + 1):
try:
resp = requests.post(url, headers=HEADERS, json=payload, proxies=proxy_dict(), timeout=20)
if resp.status_code == 200:
return resp.json()
print(f"Attempt {attempt}: comment request returned status {resp.status_code}")
except requests.exceptions.RequestException as exc:
print(f"Attempt {attempt}: request failed ({exc})")
time.sleep(random.uniform(2, 4))
raise RuntimeError(f"Failed to fetch comment page after {max_retries} attempts")
if __name__ == "__main__":
# continuation_token and api_key come from parsing a watch page's
# ytInitialData; pass them in once extracted from that response.
pass
"""
Local storage helper for scraped YouTube video records (proxy scraper companion).
Keeps a SQLite table keyed by video_id so repeat scraper runs update
existing rows for analysis instead of writing duplicates.
"""
import sqlite3
def init_db(path: str = "youtube_data.db") -> sqlite3.Connection:
conn = sqlite3.connect(path)
conn.execute("""
CREATE TABLE IF NOT EXISTS videos (
video_id TEXT PRIMARY KEY,
title TEXT,
view_count_text TEXT,
last_seen TEXT
)
""")
return conn
def upsert_video(conn: sqlite3.Connection, video: dict, scraped_at: str) -> None:
conn.execute(
"""
INSERT INTO videos (video_id, title, view_count_text, last_seen)
VALUES (:video_id, :title, :view_count_text, :last_seen)
ON CONFLICT(video_id) DO UPDATE SET
title = excluded.title,
view_count_text = excluded.view_count_text,
last_seen = excluded.last_seen
""",
{**video, "last_seen": scraped_at},
)
conn.commit()
if __name__ == "__main__":
conn = init_db()
sample = {"video_id": "dQw4w9WgXcQ", "title": "Example Video", "view_count_text": "1,482,309 views"}
upsert_video(conn, sample, scraped_at="2026-07-22")
conn.close()
{
"video_id": "dQw4w9WgXcQ",
"title": "Example Channel - Official Video",
"view_count": "1,482,309 views"
}Install requests; parsing uses the built-in json and re modules for the embedded state.
pip install requestsRoute requests through the residential rotating gateway, and set a region matching your target market.
PROXY_HOST = "proxy.knoxproxy.com"
PROXY_PORT = 10000
PROXY_USER = "your_username"
PROXY_PASS = "your_password"Request a channel's /videos URL and locate the ytInitialData script tag the same way as a watch page, since channel listings embed the video grid in the same JSON format.
Walk the richItemRenderer entries inside ytInitialData to pull each video's ID, title, and view count text, turning the channel page into a full YouTube video scraper in one pass.
Request the individual video URL to scrape YouTube videos one at a time, locating the ytInitialData script tag for title, view count, and description.
YouTube loads comments through a separate continuation request rather than the initial page load. Pull the continuation token out of ytInitialData and POST it to the InnerTube endpoint for comment text and author names.
For sustained volume, the official API with a free key is more durable than HTML parsing and is not subject to the same bot-detection triggers.
Write each parsed video or comment record to a local SQLite table keyed by video ID, so repeat runs update existing rows instead of duplicating them.
YouTube Video Data runs anti-automation checks + consent screens. The tested setup is residential rotating proxies, targeting match region for region-specific trending lists and consent handling, with this rotation: Rotate IP every 10-20 requests for single-video lookups; rotate every request when pulling a full channel's video list or paging through comments. That combination held a 93% success rate on the Jul 2, 2026 test run.
The proxy is half the job — rotate IP every 10-20 requests for single-video lookups; rotate every request when pulling a full channel's video list or paging through comments is what turns a working request into a repeatable one.
Rotate IP every 10-20 requests for single-video lookups; rotate every request when pulling a full channel's video list or paging through comments.
Match region for region-specific trending lists and consent handling.
YouTube Video Data leans on eU/UK cookie-consent interstitial before content loads. Metadata scraping such as title, views, likes, and description via the public watch page, channel page, or the official Data API v3 is tolerant at moderate volume; heavy automated traffic from one IP eventually triggers the "confirm you're not a bot" interstitial, and comment continuations tighten that threshold further.
Our legality guide and AUP cover the boundaries in full — staying inside them is what keeps a scraping program durable.
For any sustained volume, yes. The official Data API v3 is free within its quota, returns structured data directly, and skips the bot-detection interstitials that HTML scraping runs into. It also rate-limits by API key rather than by IP, so it scales better past a few thousand lookups per day without extra proxy rotation.
Residential rotating proxies work best for HTML-based scraping, since sustained traffic from one IP eventually triggers the "confirm you're not a bot" prompt, especially on comment continuations. Rotate every 10-20 requests for single-video lookups, switch to rotating on every request for full channel lists or comment paging, and match the proxy region for accurate trending lists and consent handling.
Setting a CONSENT cookie on the request, as shown in youtube_scraper.py above, skips the EU/UK consent interstitial that would otherwise interrupt the page load before the video content renders. This matters most when requests route through European or UK exit nodes, since that's when YouTube serves the interstitial instead of the watch page directly.
Yes, but comments load through a separate continuation request rather than the initial ytInitialData blob, since YouTube loads comments after the initial page load rather than embedding them upfront. Extract the continuation token from the watch page's ytInitialData first, then POST it to the InnerTube endpoint with youtube_comments.py above to get comment text and author names.
Request the channel's /videos URL the same way as a watch page, then parse the richItemRenderer entries inside ytInitialData for each video's ID, title, and view count, as shown in youtube_channel_scraper.py above. Since a channel page can list many videos in one response, rotate the proxy IP on every request instead of every 10-20 to stay closer to normal browsing.
Rotate IPs on every request instead of every 10-20, and split the channel queue across separate KnoxProxy sessions so no single IP accumulates more requests per hour than one browsing session would. Once the daily lookup count climbs past a few thousand, switch that volume to the official Data API v3, which is rate-limited by key instead of by IP.
Write each parsed video record to a local SQLite table keyed by video_id, as shown in youtube_storage.py above. Repeat runs then update existing rows with fresh view counts and titles instead of creating duplicate entries, which keeps a growing channel or watchlist scrape from bloating the database over weeks of repeated runs.
The code above is a tested starting point covering single videos, full channels, comments, and storage, and KnoxProxy residential IPs held a 93% success rate on watch-page requests during testing. Run it as-is or extend the fields; proxy rotation and retry logic are already built into every sample, so you don't need to add error handling from scratch.
residential proxies -- 93% tested success, instant activation, 14-day money-back guarantee.