Popular LLM application framework for building chains and agents. Route LangChain web retrieval requests through KnoxProxy to load web content for RAG pipelines, research agents, and knowledge-grounded LLM applications without IP blocks.
Install LangChain with community loaders and HTTP libraries.
pip install langchain langchain-community requests beautifulsoup4Configure HTTP proxy environment variables so LangChain web loaders route through KnoxProxy.
export HTTP_PROXY="http://USER:PASS@gw.knoxproxy.com:7000"
export HTTPS_PROXY="http://USER:PASS@gw.knoxproxy.com:7000"LangChain WebBaseLoader respects the requests session. Pass a custom session with proxy settings.
Modify the proxy username for country-specific content retrieval.
export HTTPS_PROXY="http://USER-country-us:PASS@gw.knoxproxy.com:7000"Combine the proxied loader with embeddings and a vector store for a full RAG pipeline.
Run the chain and verify that web content is loaded through the proxy by checking the source IP in logs.
import requests
from langchain_community.document_loaders import WebBaseLoader
from langchain.text_splitter import RecursiveCharacterTextSplitter
PROXY_URL = "http://USER:PASS@gw.knoxproxy.com:7000"
# Create a requests session with KnoxProxy
session = requests.Session()
session.proxies = {
"http": PROXY_URL,
"https": PROXY_URL,
}
session.headers.update({
"User-Agent": "Mozilla/5.0 (compatible; ResearchBot/1.0)",
})
# Load web pages through KnoxProxy
loader = WebBaseLoader(
web_paths=[
"https://en.wikipedia.org/wiki/Large_language_model",
"https://en.wikipedia.org/wiki/Retrieval-augmented_generation",
],
requests_per_second=2,
session=session,
)
documents = loader.load()
# Split for RAG pipeline
splitter = RecursiveCharacterTextSplitter(
chunk_size=1000,
chunk_overlap=200,
)
chunks = splitter.split_documents(documents)
print(f"Loaded {len(documents)} pages, split into {len(chunks)} chunks")
# Use with any LangChain chain:
# vectorstore = FAISS.from_documents(chunks, embeddings)
# qa_chain = RetrievalQA.from_chain_type(llm, retriever=vectorstore.as_retriever())Each WebBaseLoader.load() call creates new connections with fresh IPs. For sticky sessions across a multi-page load, use USER-session-{chain_run_id} as the proxy username.
| Problem | Fix |
|---|---|
| requests.exceptions.ProxyError: Cannot connect to proxy | Verify the proxy URL format: http://USER:PASS@gw.knoxproxy.com:7000. Test with curl first. |
| WebBaseLoader returns empty documents | Add a realistic User-Agent header to the session. Try a residential proxy by adding -type-residential to the username. |
| SSL: CERTIFICATE_VERIFY_FAILED | KnoxProxy does not perform SSL interception. This is likely a network issue. Set session.verify = True and check your system CA bundle. |
USER-country-de-city-berlin-session-profile07Order matters -- geo flags before the session flag. The session name is free text; use the profile ID so the mapping is self-documenting. Password stays as issued; no flags belong there. HTTP on :7000, SOCKS5 on :7001, same credentials.
Only if you set global HTTP_PROXY and HTTPS_PROXY environment variables, which route every outbound request including LLM provider API calls through KnoxProxy. The session-based approach shown in the setup, where you pass a custom requests session to WebBaseLoader, proxies only web-loading traffic and leaves LLM API calls direct.
Yes, set the proxy on the aiohttp or httpx client configuration that AsyncHtmlLoader uses internally to fetch pages asynchronously. Pass a custom session created with aiohttp.ClientSession(proxy=proxy_url), the same KnoxProxy URL used elsewhere in your pipeline, so multiple pages load through the proxy concurrently instead of one at a time.
Yes, add cookies or authentication headers to the same requests session you already configured with the KnoxProxy proxies dict, since login state lives in the session object, not the proxy. KnoxProxy passes all headers through transparently as a forward proxy, so authenticated requests reach WebBaseLoader targets exactly as sent.
Set the proxy username to USER-country-{cc}, for example USER-country-fr for France or USER-country-jp for Japan, to get an exit IP located in that country. WebBaseLoader then loads pages exactly as a visitor from that region would see them, which matters for geo-restricted or geo-personalized web content in a RAG pipeline.
Rotating residential proxies -- 5 minutes setup, instant activation, 14-day money-back guarantee.