Dev.to WebDev 🛠 Dev 👁 0 📖 3 min read

The sites that always look broken to a link checker

Some destinations refuse automated requests as a matter of policy. Link to them and they will show up in a check, every month, forever, and the links are fine. Here is what the common ones answer, so you can recognise th

Some destinations refuse automated requests as a matter of policy. Link to them and they will show up in a check, every month, forever, and the links are fine. Here is what the common ones answer, so you can recognise them on sight instead of opening each one.

The usual suspects

Destination What a checker typically gets What it means
LinkedIn 999 Not a real HTTP code at all. It is a private "no" for automated traffic. It appears in no specification, and it means only "you are not a browser".
Amazon 503, sometimes 403 Bot mitigation dressed as a server problem. A 503 normally means "temporarily unavailable"; here it usually means "not to you".
X / Twitter 403, or a page about unsupported browsers Content is assembled in the browser and unauthenticated automated access is refused.
Instagram, Facebook 403, or a login wall with status 200 The 200 case is the nastier one: the check "succeeds" while fetching a login page rather than the post.
Medium 403, or a paywall interstitial Automated access is refused; a person gets the article, possibly metered.
DOI links (doi.org) A redirect, then a publisher refusing The DOI resolves fine. It is the academic publisher at the end of the chain that turns away robots. The citation is not broken.
archive.org Timeouts, 429, sometimes 503 Rate limiting and genuine slowness under load. Ironic, given that half of these links exist to replace dead ones.
Anything behind a bot firewall A challenge page, enable JavaScript and cookies to continue The checker was handed a challenge it cannot solve. The status varies; the content is never the page you linked to.
Government and bank portals 403, or a geo or consent wall Frequently region-locked or consent-gated. Often loads fine for you and never for a server abroad.
Shortened links (bit.ly and friends) Whatever the destination answers, plus a hop Worth checking for a different reason: if the shortening service ever closes, every one of these dies at once.

These behaviours change. A site that answers politely today can add a firewall next month, and the reverse happens too. Treat the table as a way to recognise a pattern, not as a permanent fact about any company.

How to tell this apart from a link that really died

Open it in a browser. If it loads for you and the checker says 403 or 999, that is the whole answer: the server told you yes and the robot no.

Check a single link here → It shows the status, the redirects and what the server said. Free, nothing to install.

The only answers that genuinely mean the page is gone are 404 and 410, and even those deserve a second attempt by a different method. Some servers answer 404 to a HEAD request and 200 to a GET.

What to do with them

  1. Leave the links alone. They work. Removing a working link to silence a warning is the one outcome worth actively avoiding.
  2. Put these domains on a skip list, once. If a domain refuses every robot every month, that is a standing fact about the domain, not a monthly task for you.
  3. Do keep an eye on the ones that matter. A skip list means you will not be told if such a link ever does die. For a handful of critical links, that trade may not be worth it. Check those by hand once a year instead.

The one case that is not a false alarm

A login wall answering 200. The check succeeds, the report says fine, and the link takes your readers to a sign-in page rather than the thing you cited. No status code catches this. If you link to social posts or gated articles, it is worth opening a few by hand, logged out, once in a while.

Related

The plugin we make

Onsite Link Verdict puts every one of the cases above in an Unverifiable tab, with the actual response in plain language, rather than in the broken pile. It calls a link broken only on proof. It runs on your own server, and it is free on the WordPress.org directory.

Originally published on onsitetools.dev.

📰 Read the original article on Dev.to WebDev

Originally published by Dev.to WebDev. Aggregated on AIWithGhost for educational purposes — full credit and traffic to the original publisher.