Is web scraping legal? A plain-language guide

First, a note

The short answer

In many places, collecting publicly available, non-personal information from websites, such as product prices, listings or published articles’ metadata, is lawful and common. Courts in the United States have recently been reluctant to treat scraping of public pages as hacking.

Risk rises sharply in four situations: when you collect from behind a login, when you collect personal data, when you ignore clear restrictions in a site’s terms, and when you republish copyrighted content.

What recent court cases say

hiQ Labs v. LinkedIn (United States, 2017 to 2022). The Ninth Circuit twice indicated that scraping publicly accessible profiles likely did not violate the Computer Fraud and Abuse Act’s “without authorization” provision. However, LinkedIn later won on breach of contract: hiQ had accepted LinkedIn’s user agreement and had used fake accounts. The case ended in a settlement in December 2022, with hiQ agreeing to stop scraping LinkedIn.

Meta Platforms v. Bright Data (United States, January 2024). A federal court in California ruled that Bright Data did not breach Meta’s terms by scraping public data while logged out, because the terms govern people using the services through their accounts. Someone collecting public pages without logging in was treated more like an ordinary visitor.

EDPB draft guidelines (European Union, July 2026). The European Data Protection Board published draft guidelines, open for consultation until 30 October 2026, that focus on scraping for AI training. Their central message applies more widely: personal data that is publicly visible is still personal data under GDPR, and collecting it needs a lawful basis and strict minimization.

Terms of service and logins

A website’s terms are a contract, and contracts usually bind people who agree to them, for example by creating an account. That is why collecting while logged in carries much more risk than collecting public pages as a visitor.

• Don’t create accounts or fake profiles to collect data.
• Don’t bypass paywalls, logins or technical access controls.
• Read the terms anyway. A clear prohibition is a signal to look for another source or ask for permission.

Robots.txt and polite collection

Robots.txt is a file where websites state which areas automated tools should avoid. It isn’t a law, but respecting it is good practice and good evidence of acting in good faith. So is collecting politely:

• Limit request rates so you never burden a site
• Collect only the pages and fields you need
• Identify a contact point if a site owner wants to reach you
• Stop collecting from a source if you’re asked to

Personal data needs extra care

Names, email addresses, phone numbers, social profiles and anything else that identifies a person counts as personal data under GDPR and similar laws, even when it’s publicly visible. Collecting it requires a lawful basis (often legitimate interest, with a documented balancing test), a clear purpose, minimal fields and respect for people’s rights.

For most business use cases, such as prices, products, listings, company information and published coverage, personal data simply isn’t needed. Leaving it out is the easiest way to reduce risk.

Copyright and database rights

Facts, such as a price or a stock level, are generally not protected by copyright. Creative works are: article text, photos, reviews and descriptions. Collecting facts for analysis is very different from republishing someone else’s content.

In the EU, databases can also have their own protection, which can limit extracting substantial parts of a protected database. Collect what you need for your analysis, and link to sources instead of copying their content.

A practical checklist before you start

Use this before any new collection project. It won’t replace legal advice, but it covers the issues that cause most problems:

• The data is visible to anyone without logging in or paying
• No accounts, fake profiles or access workarounds are involved
• The site’s terms and robots.txt have been reviewed, and concerns recorded
• Personal data is excluded, or a lawful basis and minimization are documented
• Only facts are collected; creative content is linked, not copied
• Collection is rate-limited and can be stopped on request
• The purpose of the data is written down and matches how it will be used

How we approach it

We collect publicly available information, don’t bypass logins or paywalls, don’t collect personal data unless there is a lawful basis agreed in writing, review each source’s terms and robots.txt during scoping, and tell you about any concerns before collection starts. Read our standards.

Frequently asked questions