Stop Using Slow APIs: The Right Way to Bulk Check HTTP Statuses (50+ URLs in Seconds)
Last week, I found myself staring at a dead-links report for an old client project. Over 300 pages. I needed to know which URLs were returning 404s, which ones were silently redirecting, and which ones had just given up
Last week, I found myself staring at a dead-links report for an old client project. Over 300 pages. I needed to know which URLs were returning 404s, which ones were silently redirecting, and which ones had just given up entirely.
My first instinct was to write a quick loop in Node.js using fetch.
I hit two massive walls immediately:
- The Rate Limit / IP Ban wall: After about 15 requests, the target servers started dropping my connection.
- The Timeout wall: My script hung for 5 minutes because three random URLs just never responded.
If you've ever tried to bulk check HTTP statuses at scale, you know the pain. So, let me save you a few hours of frustration.
Here is the breakdown of how this should work, and a free tool I started using to bypass the headache entirely.
The Naive Approach (And Why It Fails)
Hereβs how most of us start writing a bulk HTTP checker:
javascript
// DON'T DO THIS FOR BULK CHECKS
async function checkUrls(urls) {
for (const url of urls) {
try {
const res = await fetch(url, { method: 'HEAD' });
console.log(url, res.status);
} catch (e) {
console.log(url, 'Error');
}
}
}
This is a disaster for a few reasons:
HEAD isn't always supported: Some servers block HEAD requests and only return a valid 200 for GET.
No Concurrency Control: If you use Promise.all(), youβll fire 50 requests at once and your IP will get blocked by Cloudflare instantly.
No Timeouts: A single hanging URL will block your entire script.
The "Developer" Way to Fix It
To do this properly in a script, you need to implement:
Connection Pooling / Semaphores (to limit concurrency to ~5 requests at a time).
Custom AbortControllers (to kill requests after 5 seconds).
User-Agent spoofing (so it looks like a real browser).
Itβs a fun technical exercise, but when youβre on a deadline and just need the data, writing this whole infrastructure is a waste of time.
The Alternative: Automate It Without the Code
When I realized I was spending more time debugging my scraper than actually auditing the site, I switched to using a bulk HTTP checker tool I found online.
Iβve been using Bulk HTTP Checker on Site Tools Box.
Itβs a lightweight, backend-optimized tool that handles all the heavy lifting for you. You can paste up to 50 URLs, hit run, and it handles the concurrency, redirects, and timeouts instantly.
It returns a clean list of:
Status Codes (200, 301, 404, 500)
Redirect chains (where it actually goes)
Response times
And because itβs built for developers, it doesn't try to sell you a $29/month subscription for a simple check.
One Last Pro-Tip for SEO / Auditing
If you have a sitemap and want to find out which specific pages on your site are broken, you don't even need to copy-paste manually. Just use the Dead Link Checker on Site Tools Box. It scans a page, crawls the outgoing links, and highlights any broken ones in red. I used it to find 3 dead links in my footer that had been silently 404-ing for months.
Don't waste your weekends writing semaphores for a script you'll only use once. Optimize your workflow, and get back to writing actual code.
Have you ever accidentally DoS'd a server with a Promise.all() loop? Let me know in the comments. We've all been there.
Originally published by Dev.to WebDev. Aggregated on AIWithGhost for educational purposes β full credit and traffic to the original publisher.