More of the Traffic Reaching What You Build Is Not From People
Look at the requests reaching any public website today and a large share of them are not from people. Search engine crawlers, uptime monitors, price comparison scrapers, credential stuffing tools, and now crawlers collec
Look at the requests reaching any public website today and a large share of them are not from people. Search engine crawlers, uptime monitors, price comparison scrapers, credential stuffing tools, and now crawlers collecting text for artificial intelligence models. One widely cited industry report, from the security company Imperva, estimated that in 2024 automated traffic made up just over half of all web traffic, the first time in a decade it had overtaken human traffic.
The new crawlers have changed the conversation. Some site owners found that crawlers for AI companies were fetching thousands of pages for every visitor they ever sent back. In 2025 Cloudflare, which sits in front of a large part of the web, began blocking known AI crawlers by default for new sites, and started testing ways to charge them for access. Expect more of this. Deciding which machines may read your content, and on what terms, is becoming a normal product decision.
What does that mean for you as a developer?
First, your numbers may be lying to you. Page views, sign up funnels and performance dashboards can all be distorted by automated traffic. Before you celebrate a spike or worry about a drop, check how much of it came from real browsers.
Second, design every public endpoint as if a machine will call it a million times. Search pages that run expensive queries, login forms, password reset and anything that sends an email or text message are favourite targets. Rate limits, sensible pagination and caching are no longer extras.
Third, learn the basic tools. Read your access logs and look at user agents and request patterns. Understand what robots.txt does, and what it does not do. It is a polite request, honoured by well behaved crawlers and ignored by the rest.
And remember that some of these visitors are welcome. Search engines bring customers, and accessibility tools and assistants may increasingly act on behalf of real people. The skill is not blocking every machine. It is knowing which ones you are serving, and deciding on purpose.
– Asael Shinder
Originally published by Dev.to Security. Aggregated on AIWithGhost for educational purposes — full credit and traffic to the original publisher.