Crawl Budget for Small Sites: When It Matters and When It Doesn't

Every few months someone on a dev team asks me, "Should we be worried about crawl budget?" The site in question usually has 200 pages and runs on a $10 VPS. The honest answer is almost always "no, not directly." But that answer hides something useful: the bugs that would burn crawl budget on a big site still exist on small ones, and they usually show up as indexing problems you're already confused about.

This post is the developer/ops version. Less theory, more logs, curl, and config files.

The 30-second definition

Googlebot decides how many URLs to fetch from your host based on two things:

1. Capacity: how fast and reliably your server responds. Slow responses or 5xx errors make it back off.
2. Demand: how valuable and fresh it thinks your URLs are.

If your site has a few hundred real URLs and responds in under a second, Google can crawl everything many times over. Budget isn't your bottleneck. The question to ask instead is: how many URLs can Googlebot discover, compared to how many I actually want indexed? If that ratio is 1:1, relax. If it's 1:50, keep reading.

Step 1: Look at what Googlebot actually requests

Search Console's Crawl Stats report is fine, but your access logs are better. Here's a quick pass on an Nginx combined log:

# Top paths Googlebot hit in the last log file
grep "Googlebot" /var/log/nginx/access.log \
  | awk '{print $7}' \
  | sed 's/?.*//' \
  | sort | uniq -c | sort -rn | head -20

And the same thing with query strings kept, which is where the surprises live:

grep "Googlebot" /var/log/nginx/access.log \
  | awk '{print $7}' | grep '?' \
  | sort | uniq -c | sort -rn | head -20

A real (anonymized) sample from a 150-page client site:

66.249.66.12 - - [14/Oct/2026:03:12:44 +0000] "GET /shop?sort=price&color=blue&page=7 HTTP/1.1" 200 18342 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; ...)"
66.249.66.12 - - [14/Oct/2026:03:12:46 +0000] "GET /shop?color=blue&sort=price&page=7 HTTP/1.1" 200 18342 "-" "..."
66.249.66.15 - - [14/Oct/2026:03:13:02 +0000] "GET /events/calendar/2031/04 HTTP/1.1" 200 4120 "-" "..."
66.249.66.15 - - [14/Oct/2026:03:13:05 +0000] "GET /blog/tag/news/page/2/page/2/ HTTP/1.1" 200 9877 "-" "..."

Four lines, four separate problems: parameter order duplication, faceted navigation, an infinite calendar, and a relative-link loop. That site had 150 pages and Google had discovered roughly 11,000 URLs.

Also verify it's really Google before trusting the numbers. Scrapers spoof the user agent constantly:

host 66.249.66.12
# 12.66.249.66.in-addr.arpa domain name pointer crawl-66-249-66-12.googlebot.com.
host crawl-66-249-66-12.googlebot.com
# should resolve back to 66.249.66.12

Step 2: Parameter explosion

Facets, sort orders, tracking tags, and session IDs multiply. Three filters with five values each plus four sort options is already hundreds of combinations per category, and parameter order doubles it.

Fixes, roughly in order of preference:

• Don't link to it. If filters are applied with JavaScript or form submissions instead of crawlable <a href> links, most of the explosion never gets discovered.
• Normalize. Sort parameters into a fixed order server-side and 301 to the canonical version.
• Strip tracking params (utm_*, fbclid, gclid) from internal links entirely. They belong on inbound campaign links only.
• Canonical tags pointing back to the clean URL. Helpful, but remember Google still has to crawl a page to see its canonical.

Step 3: robots.txt mistakes I keep seeing

robots.txt is a crawl control, not an index control. That distinction causes most of the mistakes.

# Mistake 1: blocking CSS/JS, so Google can't render the page
User-agent: *
Disallow: /wp-includes/
Disallow: /assets/

# Mistake 2: blocking a URL you also tagged noindex.
# Google can't crawl it, so it never sees the noindex.
Disallow: /thank-you/

# Mistake 3: the staging file that shipped to production
User-agent: *
Disallow: /

A saner version for a site with a faceted shop:

User-agent: *
Disallow: /*?*sort=
Disallow: /*?*sessionid=
Disallow: /search
Sitemap: https://example.com/sitemap.xml

Always check what's actually served, not what's in your repo:

curl -s https://example.com/robots.txt

Step 4: Sitemap bloat

Your sitemap should be a list of URLs you want indexed, returning 200, self-canonical, and not noindexed. Plugins love to dump in tag archives, author pages, attachment pages, and paginated archives.

A quick status audit:

curl -s https://example.com/sitemap.xml \
  | grep -oP '(?<=<loc>)[^<]+' \
  | while read url; do
      code=$(curl -o /dev/null -s -w "%{http_code}" "$url")
      [ "$code" != "200" ] && echo "$code $url"
    done

Anything printed is a URL you're telling Google is important that isn't. Redirects, 404s, and 410s in a sitemap teach Google to trust it less. For sitemap indexes, run the same loop on each child sitemap.

Step 5: The soft-404 connection

This is where small sites actually feel pain. A soft 404 is a page that returns 200 OK but looks empty: "No products found," empty tag pages, expired listings showing a generic message.

curl -s -o /dev/null -w "%{http_code}\n" https://example.com/shop?color=purple
# 200   <- but the page says "0 results"

Google keeps recrawling these, flags them in the Page indexing report, and you end up with a long "Crawled - currently not indexed" list that makes it look like Google is ignoring your real content. Return a proper 404 or 410 for genuinely gone content, noindex empty filter states, and stop linking to zero-result combinations in the first place.

Step 6: Server health is part of the budget

If your logs show Googlebot getting 5xx errors or 3-second response times, it slows down. Check timing quickly:

curl -o /dev/null -s -w "TTFB: %{time_starttransfer}s  total: %{time_total}s\n" https://example.com/

Consistent TTFB above ~1s on simple pages is worth fixing for users anyway; crawl rate is a bonus.

So when does it matter?

It doesn't matter much if:

• You have under a few thousand URLs and discovered URLs roughly match real pages.
• New pages get indexed within days.
• Your server is fast and stable.

It matters if:

• Logs show Googlebot spending most of its hits on parameters, calendars, or loops.
• Crawl Stats shows lots of requests but important pages are stale in the index.
• Page indexing is full of soft 404s, duplicates, and "Discovered - currently not indexed."

A small housekeeping tip

When sharing links in campaigns, keep tracking parameters off your internal navigation. If you need clean, trackable links for social or email, a shortener like this free URL shortener keeps the long UTM strings out of your site's link graph.

Wrap-up

Crawl budget is mostly a big-site concept. But the bugs behind crawl waste, parameter explosion, sloppy robots.txt, bloated sitemaps, and soft 404s, are universal. Fix them because they make your site cleaner and your indexing reports readable, not because you're afraid Google will run out of patience with your 200 pages.

More technical SEO notes on my blog.

Written by Riichi Mirza