258 requests in 22 hours: what a crawl budget looks like

Log analysis from a 22-hour window on anclum.com, November 7, 2026. Every figure has a command next to it at the end.

The number is 258.

That is how many requests Google's published IP ranges sent our site in a window of 21 hours, 47 minutes and 56 seconds. Not 258 thousand. Two hundred and fifty-eight. For a site with 12,362 English pages and 24,667 HTML pages in total.

If you have never looked at your own logs this way, the number probably sounds wrong. It is not. It is what a small site's crawl budget looks like before anything else goes right.

Pages and requests are not the same axis

A generated directory makes it easy to confuse inventory with attention. We can produce pages faster than anything will ever read them. The question is not "how many pages do we have" but "how many did anything come back for."

In that window:

requests from Google IP ranges   258
distinct URLs requested          233
English pages on disk             12,362

Google touched 233 distinct URLs while 12,362 English pages sat there. Even if every one of those 233 had been a unique English content page — and they were not — a full pass would take about 53 days at the same rate. That is an order-of-magnitude estimate, not a schedule, but it is the right order.

This is why "we published 500 new pages" is not a traffic plan. Publishing adds to the denominator. It does nothing to the numerator.

Where the 258 went

Every percentage below is a share of those 258 requests.

Those three rows are disjoint and add up to 258. Separately, 38 of the 258 responses were 410 Gone (14.7%) — almost all of them from the deleted /ny/ tree. The point of showing both views is that the dead tree consumed a sixth of the budget on its own.

A quarter of the crawl went to pagination. These are the /p/62/, /p/15/ style pages that slice a list into chunks. They are not in our sitemap and never have been. But they are linked from every list page, so a crawler that follows links will keep finding them. They are navigation, not content, and they were the single largest line item in the budget.

A sixth went to pages we deleted. We retired an old product tree at /ny/ and returned 410 Gone for it. Google kept checking: 40 requests, and 38 410 responses overall. This is the system working. A 410 is how you tell a crawler that a URL used to exist and is intentionally gone. It decays on its own.

The instinct to "clean this up" by blocking /ny/ in robots.txt would be wrong. A blocked URL cannot be seen to be gone; it just stays in the index longer. The correct move is to keep returning 410 and let the crawler finish its bookkeeping.

What actually increases the budget

The tempting conclusion is: fix the split, and you get more crawling. That is not how it works.

Reducing pagination waste does not create requests. It changes what those requests are spent on. The size of the budget is set mostly outside your site: how many other pages link to you, how often those pages change, how authoritative the whole domain looks. Internal tidiness moves the allocation. External signals move the ceiling.

That distinction matters because it changes what you do next:

1. Mark pagination noindex, follow. Keep the follow so the links still pass through. This does not add requests; it stops wasting them on pages that can never rank.
2. Leave 410s alone. They are the correct signal, and they expire. Blocking them is the one change that reliably makes things worse.
3. Spend the remaining attention on pages that can actually rank. If 59% of the crawl reached content, the question becomes whether that content was worth the trip.
4. Do not confuse a bigger sitemap with more crawling. We have watched our sitemap grow by 58 URLs and our crawl rate not move at all. The sitemap is a list of candidates, not a schedule.

The last one is the hardest to internalize, because a sitemap is a number you control. Crawl rate is not. It is easy to optimize the number you control and call it progress.

How to measure your own

You need three things.

One: Google's published IP ranges. The User-Agent string is self-reported, and a large fraction of traffic that claims to be Googlebot is not. Match on IP. In the same window, 6,121 requests claimed to be Googlebot; only 90 both claimed it and came from Google's ranges. Counting by User-Agent would have produced a number 24 times too large.

Two: a status-code breakdown of only those requests. Volume alone tells you nothing about whether the budget is being spent on pages that exist. Our split was 200 OK 77.5%, 410 14.7%, 301 5.0%, 404 2.3%, 302 0.4%. Without it, "258 requests" sounds fine; with it, you can see the pagination and the dead tree.

Three: distinct URLs, not just request count. 258 requests and 233 distinct URLs means 25 repeats: Google was re-checking a small set rather than exploring a wide one.

A shell pipeline is enough:

zgrep -h '' /var/log/nginx/access.log* \
  | awk '{print $1, $7, $9}' \
  | sort | uniq -c | sort -rn | head -50

The hard part is not the pipeline. It is deciding, before you look, which number would change your mind. If the answer is "none of them," you are not measuring; you are collecting.

The uncomfortable summary

We have 24,667 pages. Google asked for 233 of them in a day. A quarter of that attention went to pagination, a sixth to a tree we already deleted, and the rest to content.

None of that is a penalty. It is the normal state of a site that has not yet earned more attention. The fixes available to us are modest: stop spending the budget on navigation, keep the 410s honest, and make the pages that do get crawled worth the trip.

The number that would change the trajectory is not on this page. It is the number of other sites that link to us. Internal work can make the crawl more efficient. It cannot make it bigger.

Numbers and where they came from