The essential points from this guide -- each one is explained in detail below.
The hiQ v. LinkedIn ruling confirmed that scraping publicly accessible data does not violate the CFAA.
Terms of Service alone do not create criminal liability for scraping -- they are a contractual matter.
Scraping personal data requires compliance with GDPR (EU) and CCPA (California) privacy regulations.
Circumventing technical access controls (login walls, CAPTCHAs) carries higher legal risk than scraping public pages.
Respecting robots.txt is a best practice but not a legal requirement in most jurisdictions.
The legal landscape for web scraping has been shaped by several landmark cases in the 2020s. In hiQ Labs v. LinkedIn (2022), the U.S. Supreme Court let stand the Ninth Circuit ruling that scraping publicly available LinkedIn profile data did not violate the Computer Fraud and Abuse Act (CFAA). The court reasoned that "without authorization" in the CFAA refers to accessing systems protected by authentication, not publicly available web pages.
In Meta Platforms v. Bright Data (2024), a California court ruled that scraping publicly accessible Facebook and Instagram data was lawful. The court found that Bright Data did not violate the CFAA because it accessed only public pages, and that Meta's Terms of Service could not convert public access into unauthorized access under federal law.
X Corp. v. Bright Data (2024) produced a similar outcome. The court dismissed X's CFAA claims regarding the scraping of public tweets, reinforcing the principle that publicly available data on the open web can be collected. These rulings collectively establish a strong legal foundation for scraping public data in the United States.
The CFAA (18 U.S.C. 1030) criminalizes accessing a computer "without authorization" or "exceeding authorized access." The central legal question for scraping is whether accessing a public website counts as "without authorization" when the site operator would prefer you not access it.
Post-hiQ, the consensus is clear: if a page is publicly accessible (no login required, no technical barrier), accessing it does not violate the CFAA regardless of the site's Terms of Service. The CFAA is a computer trespass statute, not a breach-of-contract statute. TOS violations may give rise to civil contract claims, but they do not create criminal CFAA liability.
However, circumventing authentication (creating fake accounts to access logged-in content) or bypassing technical access controls (defeating CAPTCHAs programmatically, spoofing session tokens) enters riskier territory. The Van Buren v. United States (2021) Supreme Court decision narrowed the CFAA's scope, but did not address circumvention of technical barriers specifically. The safest legal position is to scrape only pages accessible without authentication.
In the European Union, web scraping legality is governed by the Database Directive (96/9/EC), the GDPR, and national laws. The Database Directive protects databases that required substantial investment to create. Scraping a substantial portion of such a database may infringe the sui generis right, even if the individual data points are factual.
The GDPR adds complexity when scraped data includes personal information (names, email addresses, profile photos, location data). Any processing of personal data requires a lawful basis under Article 6. "Legitimate interest" (Article 6(1)(f)) is the most commonly cited basis for scraping-related processing, but it requires a balancing test against the data subjects' rights and interests.
Practically, this means EU scraping projects must implement data minimization (collect only what you need), conduct a Legitimate Interest Assessment (LIA), and maintain clear records of processing activities. If you scrape personal data of EU residents, you must also comply with data subject access requests (DSARs) and deletion requests.
To scrape legally and ethically, follow these guidelines. First, only scrape publicly accessible data -- pages that any visitor can see without logging in. Avoid creating fake accounts to access protected content. Second, respect rate limits and do not overload target servers. Excessive request volume that degrades site performance could support a trespass to chattels claim.
Third, do not collect personal data unless you have a clear lawful basis and a plan for compliance with applicable privacy laws. If you collect personal data, implement data minimization -- extract only the specific data fields you need and discard the rest. Fourth, review the target site's robots.txt file. While not legally binding in most jurisdictions, following robots.txt demonstrates good faith.
Fifth, document your scraping practices. Maintain records of what you scrape, why you scrape it, your legal basis, and your data retention policy. This documentation is essential if you ever need to defend your practices. Sixth, consult legal counsel for high-risk targets -- government databases, financial data, health-related data, or data about minors carry additional regulatory requirements.
Ready to put this into practice? See web scraping proxies
KnoxProxy Research Team · Technical Content
Network engineers and proxy infrastructure specialists with 10+ years in anti-bot systems, web scraping, and IP routing.
90.4M+ ethically sourced residential IPs across 195 countries. Start free -- no credit card required.