My AI directory has 92 categories. Ten of them hold 37% of the listings.

I ran one query against my AI tools directory last night, and the result explained why some of its pages get crawled while others sit ignored.

Those ten categories hold 2,576 of 7,006 category assignments — 36.8% of the whole taxonomy. The directory has 4,323 tools across 92 categories.

That last number is not the problem. The tail is.

14 categories have ten tools or fewer

• New AI Tools: 0
• AIxploria Selection: 0
• Programming: 0
• AI Presentations: 2
• AI Music: 4
• AI Education: 4
• AI 3D: 4
• AI Research: 4
• AI SEO: 5
• AI Translation: 7
• AI Chatbots: 8
• AI Image Generators: 8
• AI Social Media: 8
• AI Customer Support: 10

Three of those are empty, and they are also the most tempting pages to publish, because the names sound valuable: "New AI Tools" and "Programming" are exactly what a searcher types. So I checked what the site actually returns for them:

$ BASE=https://aiverse.mindutil.com
$ curl -o /dev/null -w "%{http_code}" "$BASE/category/last-ai-en"
404
$ curl -s "$BASE/category/last-ai-en" | grep robots
<meta name="robots" content="noindex, nofollow"/>

And the sitemap filter is one line:

where: { tools: { some: {} } }

So 89 of 92 categories ship in the sitemap. The three empty ones do not.

Why a 404 beats an empty page

An empty category that returns 200 with "0 tools found" is a page a crawler can index. Do that twenty times and you have told a search engine your site has twenty useless URLs before it has read a single tool page. The 404 does the opposite: it says this hub does not exist yet.

None of this is free. A user who bookmarked the page gets a dead end, and the internal link from the categories index has to be removed or the link graph starts leaking 404s. The trade is simple: a little maintenance for the absence of thin pages. On a directory where the median tool page is still fighting for its first indexing, that trade is worth taking.

Category is not a tag

4,323 tools produce 7,006 category assignments, an average of 1.62 per tool. Exactly 2,100 tools (48.6%) sit in two or more categories. That overlap is real — a video editor can be "Video Editing", "Productivity", and "Business" without stretching any of those labels.

Overlap is also what keeps category pages from being template clones: each one is a different basket of results. Directory sites get flagged as doorway pages when the only unique word on 92 pages is the category name. The fix is not fewer categories. It is making each category page answer something the tool pages cannot: which five of these should I try first, and why.

The 405-tool graveyard

RIP AI is the largest bucket in the whole directory: 405 tools, 9.4% of everything, 236 of them marked free. These are discontinued or dead products. Publishing that number is useful — a directory that never deletes anything is lying by omission — but RIP AI is not a category. It is a status.

Mixing the two is how you end up with a tool that is both "RIP AI" and "Video Generators", an empty "Programming" bucket, and all three looking equally important in a sitemap.

Four rules I am adopting

1. A category ships only if it has at least one tool. Zero-tool hub gets a 404 and noindex, never a 200 with an empty state.
2. A category with fewer than 10 tools does not get its own indexable template until it reaches 10. Until then it is a filter on the parent listing.
3. Statuses — dead, free, new, verified — stay filters, not categories.
4. Every category page needs a hand-written shortlist paragraph. If I cannot write one, the category is not ready to be indexed.

Only the first rule is already live. The other three are the work for the next few weeks.

The part that surprised me

The top ten are not the categories I would have guessed. I expected "AI Chatbots" to lead; it has 8 tools. Video Generators has 294. The taxonomy drifted toward whatever the crawler found in bulk rather than toward what people actually search for. That is a data problem wearing a design problem's clothes, and it is the next thing on the list.

If you want to see the current state of the catalog, including the free and paid filters, it is here: https://aiverse.mindutil.com/free-ai-tools?utm_source=devto&utm_medium=social&utm_campaign=topic