Our two hosts have opposite robots.txt policies, and the one that forbids everything still names a sitemap
Munchable runs on two hosts. munchable.app is a Next.js site with 516 URLs in its sitemap: a guide per gut condition, a page per ingredient question, recipes, legal pages. app.munchable.app is the product itself, the sam
Munchable runs on two hosts. munchable.app is a Next.js site with 516 URLs in its sitemap: a guide per gut condition, a page per ingredient question, recipes, legal pages. app.munchable.app is the product itself, the same Expo app that ships to phones, served on the web through react-native-web.
Both are mine, both are on Vercel, and their robots.txt files say opposite things. You can read both in a terminal in ten seconds, which is the honest way to check a post like this one.
$ curl -s https://munchable.app/robots.txt
User-Agent: *
Allow: /
Disallow: /api/
Disallow: /auth/
Host: https://munchable.app
Sitemap: https://munchable.app/sitemap.xml
$ curl -s https://app.munchable.app/robots.txt
# app.munchable.app is the Munchable web app, not a content site. Every path
# rewrites to the same single-page shell, so anything a crawler indexed here
# would be a duplicate of every other path and would compete with the marketing
# site at munchable.app for the brand's own queries.
User-agent: *
Disallow: /
Sitemap: https://munchable.app/sitemap.xml
One allows everything except two machine prefixes. The other forbids everything and then tells the crawler where the content it may not have actually lives.
The marketing site allows the crawl on purpose
The interesting line of the generated robots file is not a rule, it is the comment above the rules:
// Only the machine paths. The signed-in flows (/premium/*, /continue,
// /get-started) carry `noindex` instead, deliberately: a Disallowed URL
// is never fetched, so its noindex is never read, and a page linked from
// the homepage would sit in the index as a bare URL forever. Allowing the
// crawl is what lets the directive actually drop them.
This is the part of robots.txt people get wrong, including me for years. Disallow is not "do not index". It is "do not fetch". A URL that is linked from a page you do allow can be indexed on the strength of the link alone, with no title and no description, and the noindex you carefully put in the head is never read because the crawler was told not to open the door.
So the sign-up page is crawlable and tells the crawler to drop it:
$ curl -s https://munchable.app/get-started | grep -o '<meta name="robots"[^>]*>'
<meta name="robots" content="noindex, nofollow, nocache"/>
$ curl -s https://munchable.app/ | grep -o '<meta name="robots"[^>]*>'
<meta name="robots" content="index, follow"/>
Allow plus noindex removes a page. Disallow plus noindex freezes it in place.
The file is generated by app/robots.ts rather than checked in as a static asset, for one reason that has paid off twice: the same function that builds it knows whether this deployment is production.
if (!IS_INDEXABLE) {
return { rules: [{ userAgent: '*', disallow: '/' }] };
}
A preview deploy is functionally identical to production and would compete with it, so it disallows everything, and the sitemap route for the same deployment returns an empty array. Two files that would otherwise be free to contradict each other are reading one boolean.
The app host has no 404 to offer
The other host is a different kind of problem, and the difference is one line of vercel.json:
"rewrites": [{ "source": "/(.*)", "destination": "/index.html" }]
Every path on that domain serves the same single-page shell. There is no 404, and I mean that literally rather than as a figure of speech:
$ curl -s -o /dev/null -w "%{http_code}\n" https://app.munchable.app/definitely-not-a-page-9f3
200
$ curl -s -o /dev/null -w "%{http_code}\n" https://munchable.app/definitely-not-a-page-9f3
404
A crawler on the marketing site is walking a finite set of 516 URLs that each say something different. A crawler on the app host is walking an unbounded set of URLs that all say the same thing. Those are not the same job and they do not want the same policy.
Every response from that host also carries a header, applied to /(.*):
x-robots-tag: noindex, nofollow
Which is where this gets uncomfortable, because that is exactly the combination the marketing site's own comment argues against. Disallow means the header is never fetched. Belt and braces, where the braces cannot be seen.
Why both, and why it is still right here
I went looking for my own mistake and talked myself back out of it, so here is the reasoning rather than the conclusion.
The header is not useless. It covers every way a URL gets fetched without robots.txt being consulted first: an internal tool, an agent that ignores the file, a preview service generating a card, a crawler that respects headers more faithfully than directives. The Disallow covers crawl volume, which on this host is the actual risk, because the URL space has no end and each one costs a 200 and a shell.
So the two hosts are not inconsistent. They are the two correct answers to a question I had not realised was conditional:
-
Finite URL space: allow the crawl, use
noindexon the pages you want gone. The directive gets read, the page leaves the index, and the cost is bounded by the number of pages you have. -
Unbounded URL space: disallow, and accept that your
noindexis decoration. The thing you are protecting is not the index, it is the crawl.
The second case has a cost worth naming. If someone links to app.munchable.app/whatever, that URL can sit in an index as a bare string for a long time, and I have given up the mechanism that would have removed it. The mitigation is that nothing links there. Not as a policy, as a measurement:
$ curl -s https://munchable.app/ | grep -o 'href="https://app.munchable.app[^"]*"' | wc -l
0
The constant holding that host appears in exactly one file on the marketing site, and the one thing that uses it is a window.location.assign after sign-in. There is no anchor tag pointing at the app host anywhere a crawler can see, and the app's own navigation is client-side routing, which produces no hrefs to follow either.
The sitemap line in the file that forbids everything
Which leaves the oddest line in either file: a disallow-all robots.txt that ends with
Sitemap: https://munchable.app/sitemap.xml
Sitemap is not scoped by the Disallow rules; it is an independent directive, so this is valid rather than contradictory. It is also, for most crawlers, pointless: a sitemap on another host is normally only honoured when both hosts are verified by the same owner, which they are here, but that is a fact in a search console rather than in a file.
I keep it anyway, and the argument is not technical. A crawler that reads that file has just been told there is nothing for it on this domain. One extra line tells it where there is something. The cost is 48 bytes and the upside is that the next automated thing to look at the wrong host finds the right one.
The surrounding infrastructure follows the same split. The marketing host publishes an IndexNow key file at its root and runs an hourly job that fingerprints every URL it serves and submits only the ones that changed, so a new answer page reaches Bing within the hour. The app host announces nothing to anybody, ever, which is the correct amount of announcing for software.
If you want to check any of this, both files are public and the whole post is four curls. The content side worth actually reading is munchable.app/conditions and the question pages indexed at munchable.app/answers, which between them are most of those 516 URLs.
Originally published by Dev.to WebDev. Aggregated on AIWithGhost for educational purposes β full credit and traffic to the original publisher.