Is Web Scraping Legal? A Practical Guide for Businesses

If you are asking is web scraping legal for business, the short answer is: collecting publicly available data is often lawful, but "often" is doing a lot of work in that sentence. The risk depends on what you collect, how you collect it, which terms you have agreed to, and what you do with the data afterwards.

I build scrapers and data pipelines for a living (see my web scraping and data extraction service), so this is the practical version I walk clients through before a project starts. I am an engineer, not a lawyer, and nothing here is legal advice. Treat it as a map of the questions to ask your own counsel.

The short answer: it depends on five things

Scraping is not one legal question. It is five, and each can sink a project on its own:

1. Is the data public, or behind a login? Public pages carry far less risk than anything you can only reach by signing in.
2. Is it personal data? Names, emails, phone numbers and profiles trigger privacy law almost everywhere.
3. What do the site's terms say, and did you agree to them? Accepting terms by creating an account is very different from visiting a public page.
4. Is the content protected by copyright or database rights? Facts are treated differently from creative text, images and structured databases.
5. How hard are you hitting the server? Scraping that degrades a site can create its own problems, however public the data is.

If you can answer all five, you can usually tell whether a project is low, medium or high risk.

Public data versus data behind a login

In the United States, the main anti-hacking law people cite is the Computer Fraud and Abuse Act (CFAA). In Van Buren v. United States (2021), the Supreme Court read its "exceeds authorized access" language narrowly, and later decisions, including the Ninth Circuit's hiQ v. LinkedIn, leaned toward the view that scraping pages anyone can open is unlikely to count as "unauthorized access".

That does not make public scraping a free-for-all. The hiQ dispute ended in a settlement, and courts have kept looking at other claims such as breach of contract and interference with the site's systems.

My working rule for clients is simple: public pages, no login, no circumventing technical blocks is the lowest-risk category. The moment a scraper logs in, uses someone else's credentials, or defeats an access control, the risk jumps, because you are now acting under an account and its terms.

Terms of service and contracts

Terms of service are where many scraping disputes are really fought. A site's terms may forbid automated access, and a contract claim does not need a hacking law at all.

Recent case law shows how fact-specific this is. In Meta v. Bright Data, a federal court in California found in early 2024 that Meta's terms governed a user's "use" of its products, and that scraping public pages while logged out was not that kind of use. Legal commentary on the decision is clear that this does not establish that all logged-off scraping is allowed. It turned on Meta's specific wording.

The practical lesson:

• Never scrape while logged in to a service whose terms forbid it. By logging in you agreed to those terms.
• Read the terms of every source before you build, and keep a dated copy.
• If a source matters to the business, ask for permission or an API. It is often cheaper than a dispute.

Personal data: GDPR, CCPA and similar laws

This is the area where I see businesses underestimate risk the most. Data being public does not make it free to use. Under the EU's GDPR, personal data stays personal data even when it is published on a website. You still need a lawful basis, you must tell people how their data is used, and you must respect their rights. California's CCPA and several other US state laws add their own rules.

Practical ways to reduce the exposure:

• Don't collect personal data you don't need. If you only need prices, don't store seller names.
• Separate personal fields from the rest of the dataset, or drop them at ingestion.
• Document why you collect each field and how long you keep it.
• Be extra careful with sensitive categories and with scraping social profiles to build contact lists.

If your use case is lead generation from personal profiles, speak to a privacy lawyer before the first request goes out.

Copyright and database rights

Facts, such as a price, a date or a record number, are generally not protected by copyright in the US. Creative expression is: article text, photos, reviews and descriptions. In the EU and UK, a separate database right can protect a substantial investment in collecting a database.

So scraping product prices to analyse the market is a very different thing from copying a competitor's product descriptions and photos onto your own site, or from feeding a publisher's full articles into a product you sell.

A simple filter: extract and use facts, don't republish creative content.

robots.txt and rate limiting

robots.txt is the convention sites use to say which paths automated tools should avoid. The format is documented in RFC 9309. On its own it is a voluntary standard, not a law, but ignoring it looks bad if a dispute starts, and it is a good signal of what the site owner expects.

I treat it as a rule of the project:

• Respect robots.txt and crawl-delay hints unless there is a clear reason not to, agreed with the client.
• Identify your scraper honestly where that is practical.
• Rate-limit so you never slow the site down for real users.
• Back off automatically when you get errors or 429 responses.

Hammering a server is the fastest way to turn a grey area into a complaint.

Real-world example: public municipal data

On the Violerts platform, which consolidates fragmented NYC municipal property data into one compliance tool, the work involved multi-agency data ingestion and asynchronous scraping. Public government sources are about as low-risk as scraping gets, because the data is published for public use. Even so, the engineering question was never just "can we fetch it?" It was how to collect the data reliably and politely, and how to keep the pipeline healthy when sources change, which I cover in scraped data change detection.

You can read the full Violerts case study for how that system is structured. The same discipline applies in my write-up on building a resilient scraper for government data sources.

A practical checklist before you scrape

Before any scraping project starts, I ask the client to answer these questions:

1. What exact fields do we need, and why?
2. Is the source public with no login required?
3. Do the terms of service forbid automated access? Have we kept a copy?
4. Does the dataset include personal data? If yes, what is the lawful basis?
5. Is there an official API, data feed or licence we could use instead?
6. Are we respecting robots.txt and limiting request rates?
7. How will we store, secure and delete the data?
8. Who reviews the legal position, and when?

If question 5 has a yes, use the API. It is more stable, and it removes most of the legal grey area.

When to get a lawyer involved

Bring in a lawyer when any of these are true:

• You plan to scrape behind a login or through an account
• The dataset includes personal data about EU, UK or California residents
• You will resell or republish the data as your product
• The source is a direct competitor or has already sent a cease-and-desist
• Your business depends on one source staying available

For everything else, a clear written policy and a well-built scraper that behaves politely will take you most of the way.

Key takeaways

• Scraping public, non-personal data without logging in is usually the lowest-risk category, but "usually" is not "always".
• Terms of service, privacy law and copyright are where businesses get into trouble, not the act of fetching a page.
• Respect robots.txt, rate-limit, and never defeat access controls.
• Collect only what you need, and drop personal data you don't.
• Prefer an official API or licence when one exists.

If you want a scraping or data-extraction project scoped sensibly, with the compliance questions answered up front, book a call and we can go through your use case.

Originally published at aqibjavaid.site.