Dev.to AI 🤖 Ai 👁 0 📖 5 min read

I counted my 4,323 listings as companies. The biggest one is GitHub Pages.

I counted my 4,323 listings as companies. The biggest one is GitHub Pages. I run an AI tools directory. The homepage says "4,300+ tools" and the number is real: my database has exactly 4,323 rows in the Tool table. Wha

I counted my 4,323 listings as companies. The biggest one is GitHub Pages.

I run an AI tools directory. The homepage says "4,300+ tools" and the number is real: my database has exactly 4,323 rows in the Tool table. What I had never done is ask what one row is. Yesterday I found that the rank badges on those pages had drifted, so today I went one level down and measured the unit itself.

The answer: 4,323 listings, 3,317 companies, and the single largest "vendor" in my directory is a hosting platform I don't own.

What I actually store per listing

One row, one name, one slug, one websiteUrl, one pricingType. The "Visit Website" button on a tool page points at /go/{slug}, which records a click and then 302s to whatever is in websiteUrl:

GET /go/descript                   -> 302 https://get.descript.com/sx9ahgvct8j9
GET /go/underlord-by-descript-ai-video -> 302 https://get.descript.com/sx9ahgvct8j9

Those are two different listings, two different pages, two different slugs. The second one is a feature of the first one. That is the only byte-for-byte websiteUrl collision in the whole table - 4,323 rows, 4,322 distinct URLs - so I initially read that as good news. It isn't. Uniqueness at the string level just means the duplicates are one level up: they point at the same company, just at different pages of it.

The method, stated so you can repeat it wrong the same way

For every row I took the hostname of websiteUrl, dropped a leading www., and then collapsed to the registrable domain with the crudest rule that works: the last two labels, with a short exception list for co.uk / com.cn style suffixes. No real public-suffix library, no brand knowledge, no verification that two domains belong to the same legal entity. So everything below is a lower bound on how clustered my catalog is, not a clean company count - deepmind.google and blog.google count as two domains here even though they are one company.

Nothing is unparseable: 4,323 URLs, 0 failures.

4,323 rows is not 4,323 destinations

measure value
rows 4,323
distinct hostnames 3,576
distinct registrable domains 3,317
rows that share a domain with at least one sibling row 1,200 (27.7%)
domains involved in that sharing 194

So better than one row in four sits behind a domain that already appears elsewhere on my site. The largest clusters are not the things you'd expect from a "curated list of AI companies":

github.io      103     huggingface.co   78     sjv.io         28
github.com      81     openai.com       50     pxf.io         24

Merge github.com and github.io and GitHub is my single biggest vendor at 184 listings - more than Google (64 rows on google.com, plus 24 on deepmind.google and 19 on blog.google) and more than OpenAI (50). Not because GitHub has that many products in my catalog, but because a project page on someone's GitHub Pages and a repository on GitHub's own domain both get treated by my crawler as "the tool's website".

337 listings have no company domain at all

If I filter rows whose host is a platform or an affiliate network (github.io, github.com, huggingface.co, sjv.io, pxf.io, prf.hn, partnerlinks.io), I get 337 rows - 7.8% of the catalog - where the address I present as "official website" is not the vendor's address. Live checks, done today:

/go/adobe-firefly      -> 302 https://prf.hn/l/EJXxwOp
/go/capcut             -> 302 https://capcutaffiliateprogram.pxf.io/9LPy7W
/go/yue2-ai-model-music-> 302 https://map-yue2.github.io/
/go/trellis-2-microsoft-> 302 https://microsoft.github.io/TRELLIS.2

Adobe Podcast, Adobe Firefly, Looka, n8n, Klaviyo: their entry in my "official website" column is a link belonging to an affiliate network, inherited from the directory I crawl. My own affiliate store is empty - I have not signed up for a single program - so when a reader clicks those, some third party gets attributed for the visit and I get a click in my analytics and nothing else. A stricter rule (hostname equal to or under one of those networks, plus four rows that sit on the vendor's own domain carrying a fp_ref= / ref=aixploria= query param) catches 79 rows; 75 of them on a network's domain.

And 303 of those 337 rows are labelled free, which is the filter most of my readers use. That is the part I'd fix first.

The prose is one template with the name swapped in

I checked the text too. Zero rows have byte-identical description values - but if I replace the tool's own name with a placeholder and the "ranks #N in the X category" clause with another, 4,323 descriptions collapse into 4,205 shapes. The largest shape is shared by 18 rows, all sitting in a category literally named This*DoesNotExist (21 tools). Their pages open with the same sentence, differing only in the product name.

The one sentence I'd delete before any of that: 2,780 rows (64.3%) end their description with "Common tags for this tool include Latest AI." "Latest AI" is not a tag - it is the name of the source site's chronological feed, and it is attached in my database to 2,860 tools, making it the most common tag I have. 1,446 rows print no tag sentence at all, and 824 have zero tag rows. So two thirds of my catalog tells the reader, in a field labelled "tags", that the item appeared recently in somebody else's list.

What this changes about how I can describe the site

My content pipeline today produced 5 more written pages out of the 130 still templated in my top-150-by-traffic queue, which is the number I actually report. But the honest framing for the catalog itself is now: roughly 3,300 destinations, 4,300 listings, about 340 of them pointing at infrastructure rather than a company, and one page that is genuinely a duplicate of another page. The free filter shows 2,198 tools; behind those 2,198 rows there are 1,538 registrable domains, and 101 of them are GitHub Pages sites.

I have not deduplicated anything, and I am not going to pretend a merge is free: "Underlord by Descript" might be a listing a reader searches for on purpose. The fix I'd defend is a canonical vendor key stored next to the URL, so "same company" is a fact I can query instead of something a reader has to notice. Until then I won't write "4,323 tools" and mean 4,323 products - I mean 4,323 rows, and the free tools list is the page where that difference shows up most.

📰 Read the original article on Dev.to AI

Originally published by Dev.to AI. Aggregated on AIWithGhost for educational purposes — full credit and traffic to the original publisher.