Autonomous AI research agent that generates comprehensive reports from web sources. Route GPT Researcher web requests through KnoxProxy to access global sources without IP blocks and retrieve geo-specific research data.
Install the gpt-researcher package and its dependencies.
pip install gpt-researcherConfigure your OpenAI or other LLM provider API key.
export OPENAI_API_KEY="sk-..."Set HTTP proxy environment variables so all web requests route through KnoxProxy.
export HTTP_PROXY="http://USER:PASS@gw.knoxproxy.com:7000"
export HTTPS_PROXY="http://USER:PASS@gw.knoxproxy.com:7000"Modify the proxy username in the env var for country-specific research.
export HTTPS_PROXY="http://USER-country-us:PASS@gw.knoxproxy.com:7000"Write a Python script that initializes GPTResearcher with a query and runs the research pipeline.
Execute the research script and review the generated report. Verify sources were accessed through the proxy.
python research_task.pyimport asyncio
import os
from gpt_researcher import GPTResearcher
# Configure KnoxProxy for all web requests
os.environ["HTTP_PROXY"] = "http://USER:PASS@gw.knoxproxy.com:7000"
os.environ["HTTPS_PROXY"] = "http://USER:PASS@gw.knoxproxy.com:7000"
# For geo-targeted research (e.g., research from UK perspective):
# os.environ["HTTPS_PROXY"] = "http://USER-country-gb:PASS@gw.knoxproxy.com:7000"
# Optional: bypass proxy for LLM API calls
os.environ["NO_PROXY"] = "api.openai.com,api.anthropic.com"
async def main():
# Initialize researcher with a query
researcher = GPTResearcher(
query="What are the latest trends in eSIM adoption worldwide?",
report_type="research_report",
)
# Conduct research (web searches + scraping through KnoxProxy)
await researcher.conduct_research()
# Generate the final report
report = await researcher.write_report()
print(report)
# Access individual sources
sources = researcher.get_source_urls()
print(f"\nResearch used {len(sources)} sources:")
for url in sources:
print(f" - {url}")
if __name__ == "__main__":
asyncio.run(main())Each web request during research gets a fresh IP. GPT Researcher makes many requests per run, and IP rotation helps avoid rate limits. For consistent identity during a research session, use USER-session-{research_id} as the proxy username.
| Problem | Fix |
|---|---|
| aiohttp.ClientProxyConnectionError: Cannot connect to proxy | Verify the proxy URL format. Test with: curl -x $HTTPS_PROXY https://httpbin.org/ip |
| Research returns no sources | Try residential proxies with USER-type-residential. Set NO_PROXY for search API endpoints if using direct API keys. |
| OpenAI API calls timing out | Add api.openai.com (or your LLM provider) to NO_PROXY environment variable. |
USER-country-de-city-berlin-session-profile07Order matters -- geo flags before the session flag. The session name is free text; use the profile ID so the mapping is self-documenting. Password stays as issued; no flags belong there. HTTP on :7000, SOCKS5 on :7001, same credentials.
The impact is minimal. KnoxProxy adds roughly 50-100ms of latency to each request, which is small next to the time GPT Researcher spends waiting on LLM report generation. Research quality actually improves overall, since fewer scraping requests get blocked or rate-limited compared to running without a rotating proxy.
Yes, GPT Researcher supports Ollama and other local LLMs for the report-writing step, independent of the proxy setup. KnoxProxy, configured via HTTP_PROXY and HTTPS_PROXY, only affects the web search and scraping requests gathering source material, so local model inference on your own machine is never routed through it.
Set the HTTPS_PROXY environment variable with a country code in the username, for example USER-country-de for Germany or USER-country-us for the United States. Search results and scraped source content then reflect that geography, since every outbound request GPT Researcher makes exits from an IP located in that country.
Yes, all sub-agents spawned in GPT Researcher multi-agent mode inherit the same HTTP_PROXY and HTTPS_PROXY environment variables set on the parent process. Each individual agent request still gets its own rotated IP through KnoxProxy, which helps the higher request volume of multi-agent research avoid rate limits and IP blocks.
Rotating residential proxies -- 5 minutes setup, instant activation, 14-day money-back guarantee.