A 200 OK response is not success: what 3 days of benchmarking scraping APIs taught me about hidden costs

I got burned by a "99.9% success rate" claim last year. A scraping vendor's dashboard said everything was fine; my pipeline was quietly filling up with garbage. So when I finally had time, I did the thing I should have done first: I measured. This post is what three days of benchmarking two popular scraping APIs revealed — including a failure-cost gap of 397× on identical failures.

The setup

Two commercial scraping APIs (ScraperAPI and Scrape.do), same targets, same network, same validator. Three targets:

1. A static product catalog page (books.toscrape.com)
2. A static product detail page
3. A real JD.com product page — JavaScript-heavy, the kind of page people actually pay these APIs for

Six requests per target per provider per day, for three consecutive days. And the rule that matters most:

Success = HTTP 200 AND the product content is actually present. A 200 that returns a block page, an interstitial, or an empty shell counts as a failure.

That single definition changes everything, and it's the one most benchmarks skip.

Part 1: The static pages — boring, in a good way

Both providers returned 18/18 validated successes on both static targets, all three days. Median latencies between 1.1s and 2.5s. If you're scraping simple pages, either service works — pick by price. This is not where the story is.

Part 2: The real product page — 0 for 54

The JD.com item page is where it fell apart. Both providers, all three days, zero validated successes — 0/18 each. And here's the detail that took me by surprise: my direct-connection baseline (no API, no JS rendering) also got HTTP 200s. The page answers. The product DOM just never arrives in the rendered output.

That reframes the problem. This wasn't "the API failed to get past the shield." Something at the rendering layer never delivered the content. Which leads to the most useful lesson of the whole exercise:

"We can pass Cloudflare" and "we can return this page's content" are two different claims. Vendors market the first one. Your business needs the second one. Sometimes they coincide; on a real e-commerce product page, in my test, they didn't.

Part 3: The $2.38 vs $0.006 bill for the same zero

Now the part that should be in every procurement checklist. ScraperAPI bills JS-rendering at 25 credits (standard requests are 1). Scrape.do bills rendering at the same flat per-request price. Both failed all 18 render attempts, every day. The declared cost of those failures over three days:

Same targets, same zero successes — a 397× cost gap created entirely by billing structure, not capability. Scale that to a production workload running 50k render attempts a month against a hard target and the difference is $6,600/month vs $16/month. (Caveat, because honesty: these are declared list prices from the vendors' public pricing pages, not invoice-verified. Read your plan's failure-billing terms before committing volume — some plans refund failed renders, some don't, and few make it obvious.)

The checklist I now send to any scraping vendor

1. What exactly is a "request"? Per-request, per-credit, per-GB — normalize everything to $/1k successful before comparing plans.
2. Do failed requests bill? Get it in writing. This is the biggest hidden cost in the industry, and my 3-day test put a 397× multiplier on it.
3. What are the JS-render and residential-IP multipliers? 5–25× is common. Model your real traffic mix before you sign.
4. What's the success rate on my target? Not your demo page. A paid pilot on the actual site, with a content validator, costs a day and answers the only question that matters.
5. What's the exit cost? Data formats, delivery options, whether historical re-pulls are included.

Where this came from

These tests are part of a small benchmark project I run — topscraper.online — where I also analyze 30-day run outcomes for ~3,500 scraping actors from a public marketplace (224M runs/month of production telemetry; the dedicated bypass cohort spans 0.00–3.12% measured failure rates). Full tables, per-target breakdowns, and methodology for everything above are in the 3-day report. Affiliate links exist on the site and are marked; rankings are computed from measurements only — the first provider I tested loses the cost comparison, because test order followed registration order, not commission size.

The repo-harvested numbers refresh weekly. If your target site matters to your business, measure before you commit — and count a 200 without content as what it is: a failure you paid for.

What's the worst hidden cost you've hit with scraping infrastructure? I'm collecting failure-billing stories for a follow-up.