38.5 million domains on a 2 GB droplet: the honest story of building an open search engine

Disclosure: I used AI assistance to help draft this post. The story, the mistakes, and every number below are mine.

In 2025, I started crawling the web.

Not metaphorically. I pointed a crawler at the open internet from a 2 GB VPS and decided to build a search engine — a free, open-source one that doesn't log your searches, doesn't need an account, and that anyone can self-host. The first crawl records date to April 2025. Then things went quiet for a while. Then this June, cl0q took off: 8.5 million domains in a single month.

Today, cl0q knows 38.5 million domains and has indexed 24.8 million pages. The goal is 100 million.

This is the honest version of how that went.

The first million nearly killed the project

The naive approach to crawling is: start with a list of sites, follow every link, repeat. What nobody tells you is that the web has traps. Point a crawler at the open web and it finds Blogspot — about a million subdomains hanging off a single domain — and happily spends a week indexing them while the rest of the internet waits.

Getting to the first million was the hardest part, because I didn't know the right crawl path. Crawl too narrow and you re-walk the same neighborhood forever. Crawl too wide and you fall down a subdomain hole and your index fills up with a million pages from the same blog host.

The fix was treating the known web as a rotating frontier instead of a finished list: keep re-sampling domains you already know and mine their homepages for fresh outbound links. And Certificate Transparency logs turned out to be a goldmine — at peak, they were surfacing around 1,700 new domains a minute. Discovery never ends; it just keeps moving.

Everything broke, mostly because of disk

For most of this project, cl0q ran on that 2 GB VPS. A web crawl is, at its core, a fight against disk space. Every page is a row, every favicon is bytes, every DNS record is overhead, and the crawl never stops growing.

Lots broke. Things crashed, crawls stalled, I ran out of room at the worst moments. These days the storage lives on a remote host connected over Tailscale, which helped enormously — and I'm still fighting storage issues today. If there's one thing I'd tell anyone starting a crawl project: budget ten times the disk you think you need, then double it.

The database has one core, and everything fights over it

Here's something the docs won't tell you: search and the crawler share a single database, a single core, and the same table. When a burst of background jobs stampedes the database, search times out. When the crawler picks its next targets, it eats a big share of DB time doing it.

There's no elegant fix yet — just triage, scheduling, and the occasional 2 a.m. realization that the thing slowing down search is my own crawler. The one part that never touches the database: the stats and drill-down pages, which are served from precomputed files. If the site feels fast while the crawler is hammering away, that's why.

What actually worked

A few decisions paid off:

• The pipeline is four small stages — discovery, page crawl, enrichment, serving — each incremental and restartable. Nothing depends on a full rebuild. (That's the simplified version. The real version involves the single-core database above.)
• Refusing to index garbage. When a site serves an anti-bot challenge page, lazy crawlers index the text "Checking your browser…" as if it were content. cl0q detects interstitials and throws them out. A search engine is only as good as what it refuses to index.
• Favicons turned out to be a superpower. Every icon gets hashed with SHA-256: 16.3 million sites in the index have a favicon, collapsing down to about 7 million distinct icons. Sites sharing an icon are usually related — same operator, same framework, same phishing kit. What started as a storage deduplication trick became one of the most useful OSINT pivots in the index.
• Privacy as architecture, not policy. Signed out, cl0q sets no cookies and stores nothing about your searches — there's no per-request log to leak, subpoena, or sell. Accounts are optional and passkey-based, and search history exists only if you turn it on. The code is open source, though I'll be honest: the public mirror is a couple months behind right now, and refreshing it is on my list.

Where things stand

The numbers, measured from the actual database rather than the dashboard: cl0q knows 38.5 million domains — 97% have been attempted, 92% resolve in DNS. It's indexed 24.8 million pages, 24.4 million of them live, across 1,027 TLDs. About 250,000 to 300,000 pages get crawled every day, and re-crawls start October 9 so pages stay fresh instead of going stale.

The road to 100 million

The plan is simple: hit 100 million domains, then open up the API properly. There's a free beta API today and paid tiers are coming — and once the project makes money, the next frontier is depth. Right now cl0q fetches homepages; with real revenue behind it, the crawl goes deeper into sites instead of just wider across them.

If you want to help, you can run a crawler and contribute pages — that's by invitation today, so get in touch — or just try a search at cl0q.com.

A year and a half of crawling, 38.5 million domains, one very tired VPS. On to 100.