Data framework for LLM applications with advanced RAG capabilities. Route LlamaIndex web readers through KnoxProxy to index web content from any geography without IP restrictions for retrieval-augmented generation pipelines.
Install LlamaIndex core and web reader packages.
pip install llama-index llama-index-readers-web requestsSet up proxy configuration for the web reader using environment variables or inline config.
Use SimpleWebPageReader or BeautifulSoupWebReader with a custom requests session routed through KnoxProxy.
Load web pages through the proxy and build a VectorStoreIndex for querying.
Modify the proxy username to load content as seen from a specific country.
proxy_url = "http://USER-country-de:PASS@gw.knoxproxy.com:7000"Use the index with a query engine to ask questions about the proxied web content.
import requests
from llama_index.core import VectorStoreIndex, Document
from llama_index.readers.web import SimpleWebPageReader, BeautifulSoupWebReader
PROXY_URL = "http://USER:PASS@gw.knoxproxy.com:7000"
# Option 1: Use BeautifulSoupWebReader with custom session
session = requests.Session()
session.proxies = {
"http": PROXY_URL,
"https": PROXY_URL,
}
# Custom proxied web reader
class ProxiedWebReader:
"""Web reader that routes all requests through KnoxProxy."""
def __init__(self, proxy_url: str, country: str = ""):
self.session = requests.Session()
username = "USER"
if country:
username = f"USER-country-{country}"
self.session.proxies = {
"http": f"http://{username}:PASS@gw.knoxproxy.com:7000",
"https": f"http://{username}:PASS@gw.knoxproxy.com:7000",
}
def load_data(self, urls: list[str]) -> list[Document]:
documents = []
for url in urls:
response = self.session.get(url, timeout=30)
response.raise_for_status()
doc = Document(
text=response.text,
metadata={"source": url},
)
documents.append(doc)
return documents
# Load pages through KnoxProxy
reader = ProxiedWebReader(proxy_url=PROXY_URL, country="us")
documents = reader.load_data([
"https://en.wikipedia.org/wiki/ESim",
"https://en.wikipedia.org/wiki/Roaming",
])
# Build index and query
index = VectorStoreIndex.from_documents(documents)
query_engine = index.as_query_engine()
response = query_engine.query("What is an eSIM and how does it work?")
print(response)Each load_data() call creates new connections with fresh IPs. For sticky sessions during a multi-URL load, use USER-session-{index_build_id} as the proxy username.
| Problem | Fix |
|---|---|
| requests.exceptions.ConnectionError: Proxy connection refused | Test proxy connectivity: curl -x http://USER:PASS@gw.knoxproxy.com:7000 https://httpbin.org/ip |
| Empty documents returned | Use a headless browser reader or crawl4ai with KnoxProxy instead for JS-rendered pages. |
| ImportError: llama_index.readers.web not found | Install it: pip install llama-index-readers-web |
USER-country-de-city-berlin-session-profile07Order matters -- geo flags before the session flag. The session name is free text; use the profile ID so the mapping is self-documenting. Password stays as issued; no flags belong there. HTTP on :7000, SOCKS5 on :7001, same credentials.
Yes, any LlamaHub reader built on requests or httpx under the hood can be proxied, either by setting global HTTPS_PROXY environment variables or by passing a custom session or client with KnoxProxy configured, the same pattern used in the ProxiedWebReader example, to readers like SimpleWebPageReader and BeautifulSoupWebReader.
Yes, use httpx.AsyncClient(proxy=proxy_url) inside an async custom reader class instead of the synchronous requests.Session shown in ProxiedWebReader, so multiple documents load in parallel through KnoxProxy rather than one request at a time. This speeds up large document sets before you build the VectorStoreIndex for querying.
Only if you set global HTTPS_PROXY environment variables, which route every outbound request including embedding and LLM API calls through KnoxProxy. The session-based approach used in the ProxiedWebReader example proxies only the web reader document fetches, leaving OpenAI or other LLM API calls direct and unaffected.
Set the country parameter in ProxiedWebReader, for example "jp" for Japan, which appends -country-{country} to the KnoxProxy username used inside the custom session. Content then loads exactly as if requested from an IP in that region, which is useful for indexing geo-restricted or geo-personalized pages into your VectorStoreIndex.
Rotating residential proxies -- 5 minutes setup, instant activation, 14-day money-back guarantee.