robots.txt for a downloader: what I allow search engines to crawl and what I block
When I built ramrom.io, a small video downloader that runs on plain PHP hosting, I spent most of my time on the tool itself: pasting a link, resolving it, handing the visitor a file. The robots.txt file was an afterthoug
When I built ramrom.io, a small video downloader that runs on plain PHP hosting, I spent most of my time on the tool itself: pasting a link, resolving it, handing the visitor a file. The robots.txt file was an afterthought, a one-line file I copied from somewhere and forgot about. That was a mistake, and it took me a while to understand why it matters for a site like this.
A downloader site is a strange thing for a search engine to crawl. Most of the site is a handful of real pages: the homepage, a page for each platform, a few guides, a blog. But behind those pages sit the endpoints that do the actual work: the resolve endpoint that talks to the provider APIs, and the download proxy that streams the file back. Those are not pages. There is nothing for a search engine to index there, and nothing a visitor would ever want to find in search results.
The problem is that a crawler does not know that. If it finds a link to an API endpoint, or guesses a path, it will happily request it. On a shared hosting plan that means a PHP process spins up, a provider API call gets spent, and the crawl budget that should have gone to my real pages goes to endpoints that return JSON errors. I was spending quota on bots.
So my robots.txt now does two simple jobs. First, it tells crawlers what to stay away from. The API directory is disallowed outright, and so are the resolve and proxy scripts. Nothing in there is a page, and nothing in there should ever show up in search results. The second job is pointing crawlers at what I do want indexed: the sitemap. One line, the full URL of sitemap.xml, so Google and Bing know where the real pages are without guessing.
What I allow is everything else. The tool pages, the guides, the blog, the about page — all open. I do not block the download pages themselves, because those are the pages people actually search for. A visitor who searches for a TikTok downloader should land on the tool, not on a guide about the tool. Blocking the tool pages would be cutting off the whole point of the site.
There are a couple of things I deliberately left out. I do not block images, CSS or JavaScript. Google renders pages before it ranks them, and if it cannot fetch the stylesheet it sees a broken page. I also do not try to block every bot by name. There are dozens of crawlers out there, and a list that names ten of them gives a false sense of control while the eleventh walks straight in. One rule for all user agents, plus the sitemap line, is enough for a site this size.
One thing that surprised me: robots.txt is a request, not a lock. A polite crawler obeys it. A rude one ignores it. That is why the API endpoints have their own protection too — the resolve script checks that the request came from one of my own pages, and the config file with the API key sits behind an .htaccess rule that returns a 403 to any direct request. Robots.txt keeps the honest bots out of the machinery. The server rules keep everyone else out.
I also learned to check the file actually works after every deploy. A robots.txt that returns a 404 because I forgot to upload it is worse than a bad one, because crawlers then assume everything is allowed. A quick curl of the live URL after each upload takes five seconds, and it has caught a missing file more than once.
For a small site on shared hosting, robots.txt is not SEO magic. It will not rank you. What it does is much duller and much more useful: it keeps crawlers on the pages that matter and out of the endpoints that cost money. On a site where every API call is metered, that is worth the ten minutes it takes to write it properly.
Originally published by Dev.to WebDev. Aggregated on AIWithGhost for educational purposes — full credit and traffic to the original publisher.