Data framework for LLM applications with advanced RAG capabilities. Route LlamaIndex web readers through KnoxProxy to index web content from any geography without IP restrictions for retrieval-augmented generation pipelines.
Install LlamaIndex core and web reader packages.
pip install llama-index llama-index-readers-web requestsSet up proxy configuration for the web reader using environment variables or inline config.
Use SimpleWebPageReader or BeautifulSoupWebReader with a custom requests session routed through KnoxProxy.
Load web pages through the proxy and build a VectorStoreIndex for querying.
Modify the proxy username to load content as seen from a specific country.
proxy_url = "http://USER-country-de:PASS@gw.knoxproxy.com:7000"Use the index with a query engine to ask questions about the proxied web content.
import requests
from llama_index.core import VectorStoreIndex, Document
from llama_index.readers.web import SimpleWebPageReader, BeautifulSoupWebReader
PROXY_URL = "http://USER:PASS@gw.knoxproxy.com:7000"
# Option 1: Use BeautifulSoupWebReader with custom session
session = requests.Session()
session.proxies = {
"http": PROXY_URL,
"https": PROXY_URL,
}
# Custom proxied web reader
class ProxiedWebReader:
"""Web reader that routes all requests through KnoxProxy."""
def __init__(self, proxy_url: str, country: str = ""):
self.session = requests.Session()
username = "USER"
if country:
username = f"USER-country-{country}"
self.session.proxies = {
"http": f"http://{username}:PASS@gw.knoxproxy.com:7000",
"https": f"http://{username}:PASS@gw.knoxproxy.com:7000",
}
def load_data(self, urls: list[str]) -> list[Document]:
documents = []
for url in urls:
response = self.session.get(url, timeout=30)
response.raise_for_status()
doc = Document(
text=response.text,
metadata={"source": url},
)
documents.append(doc)
return documents
# Load pages through KnoxProxy
reader = ProxiedWebReader(proxy_url=PROXY_URL, country="us")
documents = reader.load_data([
"https://en.wikipedia.org/wiki/ESim",
"https://en.wikipedia.org/wiki/Roaming",
])
# Build index and query
index = VectorStoreIndex.from_documents(documents)
query_engine = index.as_query_engine()
response = query_engine.query("What is an eSIM and how does it work?")
print(response)Each load_data() call creates new connections with fresh IPs. For sticky sessions during a multi-URL load, use USER-session-{index_build_id} as the proxy username.
| Problem | Fix |
|---|---|
| requests.exceptions.ConnectionError: Proxy connection refused | Test proxy connectivity: curl -x http://USER:PASS@gw.knoxproxy.com:7000 https://httpbin.org/ip |
| Empty documents returned | Use a headless browser reader or crawl4ai with KnoxProxy instead for JS-rendered pages. |
| ImportError: llama_index.readers.web not found | Install it: pip install llama-index-readers-web |
USER-country-de-city-berlin-session-profile07Order matters -- geo flags before the session flag. The session name is free text; use the profile ID so the mapping is self-documenting. Password stays as issued; no flags belong there. HTTP on :7000, SOCKS5 on :7001, same credentials.
Any LlamaHub reader that uses requests or httpx under the hood can be proxied via environment variables or a custom session.
Yes. Use httpx.AsyncClient(proxy=proxy_url) in an async custom reader for parallel document loading.
Only if you use global HTTPS_PROXY env vars. The session-based approach only proxies the web reader, not LLM API calls.
Set the country parameter in ProxiedWebReader to get IPs from that country (e.g., "jp" for Japan). Content will be served as if requested from that region.
Free trial on rotating residential -- 5 minutes setup, no credit card.