How much does web scraping cost?

The short answer

The cost of web scraping depends far more on what you collect and how often than on how many rows you end up with. A one-time pull from a few simple websites is a small project. Daily collection from dozens of complex, heavily protected sites is a continuous operation with real maintenance.

The chart and tables below explain almost every quote you will see.

The five cost drivers

1. Number of sources. Each website needs its own extraction logic and its own maintenance. Ten sources cost far more than one source with ten times the records.
2. Source difficulty. Pages that load content with JavaScript, use bot protection, require location settings or change layout often take more work to build and keep running.
3. Refresh frequency. Hourly or daily collection multiplies infrastructure use and the number of runs that need checking.
4. Volume. Millions of pages need more computing and bandwidth than thousands, though volume matters less than most buyers expect.
5. Quality requirements. Matching products across stores, deduplication, validation and custom formats all add effort, and are usually where the value is.

Common pricing models

For recurring work, the most important line in any quote is maintenance. Ask directly: “When a website changes, is fixing the scraper included?”

The costs people forget

If you’re comparing against building it yourself, include the costs that don’t show up on an invoice:

• Engineer time to maintain scrapers after launch, usually the largest cost
• Proxy, browser and server infrastructure
• Cleaning, matching and validating the output before anyone can use it
• Monitoring runs and investigating when numbers look wrong
• The cost of decisions made on bad or stale data

Three example scenarios

Prices depend on the details, but the shape of the work is predictable. Here is how effort compares across three typical projects.

Notice that volume barely appears. A daily feed of 2,000 products from twenty difficult sites usually costs far more than a one-time pull of 200,000 rows from one simple site.

Why the cheapest quote can cost more

Low quotes usually leave something out: product matching, validation, delivery monitoring or fixing scrapers when websites change. Each of those then lands on your team, often after the data has already been used in a decision.

When comparing quotes, line them up against the same scope, then ask each provider the same questions:

• Is fixing scrapers after website changes included, and how quickly?
• How are partial or failed runs reported?
• Which quality checks run on every delivery?
• Is there a free sample from my real sources?
• What happens to the price if I add a source or a field?

How to lower your cost

• Collect only the fields the decision needs. Each optional field adds work.
• Match frequency to the decision. Weekly may be enough for a weekly meeting.
• Start with fewer sources. Prove the value on the top three sites, then expand.
• Accept a standard format. Custom delivery setups add build time.
• Write a clear brief. Ambiguity is the most expensive part of any project.

Our guide to scoping a web scraping project includes a template that covers all of this.

How to get an accurate quote

Send a provider four things and you’ll get a quote you can rely on:

• The websites, with two or three example URLs each
• The fields you need, and which are required
• How often you need the data, and in what format
• Roughly how many products, listings or pages are involved

Frequently asked questions