Dev.to WebDev πŸ›  Dev πŸ‘ 0 πŸ“– 4 min read

Our sitemap hit 25 MB in 80 languages. Moving hreflang to the page head took it to 1 MB

We run Saluvo, a health information site and app that is written in 80+ languages. The marketing pages, lab test explainers, symptom guides and blog posts each exist in many of those languages, which adds up to about 3,9

We run Saluvo, a health information site and app that is written in 80+ languages. The marketing pages, lab test explainers, symptom guides and blog posts each exist in many of those languages, which adds up to about 3,900 URLs.

A few weeks in, our sitemap.xml was 25 MB. Google's limit for a single sitemap is 50 MB uncompressed (and 50,000 URLs), so we were halfway to the ceiling with only a few thousand pages. Today the same sitemap lists 3,920 URLs in about 1 MB, and hreflang still works. Here's what happened and what we changed.

How a sitemap grows quadratically

Next.js (App Router) lets you return the sitemap as data from app/sitemap.ts, including language alternates for each URL. We used it the obvious way:

// app/sitemap.ts (before)
export default function sitemap(): MetadataRoute.Sitemap {
  return PAGES.flatMap((path) =>
    LOCALES.map((locale) => ({
      url: `${SITE}/${locale}${path}`,
      alternates: {
        languages: {
          ...Object.fromEntries(LOCALES.map((l) => [l, `${SITE}/${l}${path}`])),
          "x-default": `${SITE}/en${path}`,
        },
      },
    })),
  );
}

Every URL carries an <xhtml:link rel="alternate" hreflang="…"> line for every language version of itself. With 80 languages that's up to 80 extra lines per URL, and up to 80 URLs per page. So each page can cost 80 Γ— 80 = 6,400 lines, and the file grows with the square of the number of languages.

At roughly 100 bytes per alternate line, 3,900 URLs Γ— ~80 alternates is in the 25-30 MB range, which is exactly what we saw. Add ten more languages and you're past 50 MB without adding a single new article.

hreflang only needs to be declared once

Google accepts hreflang in one of three places: the HTML <head>, HTTP headers, or the sitemap. You don't need all of them. Our pages already declared their alternates in the head through Next.js metadata:

// app/[locale]/about/page.tsx
export async function generateMetadata({ params }) {
  const { locale } = await params;
  return {
    alternates: {
      canonical: `/${locale}/about`,
      languages: {
        ...Object.fromEntries(LOCALES.map((l) => [l, `/${l}/about`])),
        "x-default": "/en/about",
      },
    },
  };
}

So the sitemap copy was pure duplication. We deleted the alternates block from sitemap.ts and left a comment so nobody adds it back:

/**
 * Language alternates (hreflang) live in each page's <head>, not here: listing ~80 alternates on
 * every one of ~4,000 URLs made this file 25 MB and would pass the 50 MB sitemap limit.
 */

The sitemap went from 25 MB to about 1 MB. Each <url> is now just loc, lastmod, changefreq, priority and, where it helps, images and a video.

Only list languages a page is really written in

The second rule mattered as much as the size. Our lab test and symptom guides aren't translated into every language the site supports. The tempting shortcut is to serve English when a translation is missing and list every language anyway, but then the sitemap and the hreflang advertise, say, a Hungarian URL that is really an English page.

So each content type knows which languages it's actually written in, and both the sitemap and the head only list those:

...TESTS.flatMap((t) =>
  testLocales(t).map((locale) => ({          // only the languages this guide exists in
    url: `${SITE}/${locale}/tests/${t.slug}`,
    lastModified: t.updated,
    images: [`${SITE}/art/home/${t.art}.webp`],
  })),
),

And a request for a language that doesn't have that guide yet gets a temporary redirect, not a permanent one. A permanent redirect gets cached; when the translation lands later, browsers and crawlers would keep going to the English page.

Images and video: keep them, they're cheap

We kept the image and video sitemap extensions, because they're a few lines per URL rather than eighty. Feature pages list their hero and showcase images and their demo video:

videos: [{
  title: f.landing.video.title,
  description: f.landing.video.description,
  content_loc: `${SITE}${f.landing.video.src}`,
  thumbnail_loc: `${SITE}${f.landing.video.poster}`,
}],

Google requires a title, a description, a thumbnail and either content_loc or player_loc for each video; without them it can't use the entry. After the change, Search Console read the sitemap as Success: 3,920 pages and 400 videos discovered.

What I'd check on any multilingual site

  1. Download your sitemap and look at its size. If it's more than a few MB for a few thousand URLs, look for per-URL alternates.
  2. Pick one place for hreflang. The head is the easiest to keep in sync with what the page actually renders.
  3. Never advertise a language you fall back from. If /hu/page shows English, it shouldn't be in the sitemap or in any hreflang set.
  4. Use temporary redirects for missing translations so they don't stick once the translation exists.
  5. Check the domain without www. Registrar forwarding often redirects only the home page and returns 404 for every deeper path. Test a deep URL, not just /.

If you're curious what the multilingual pages look like, the lab test guides are a good example: same template, each one only in the languages it has been written and checked in.

πŸ“° Read the original article on Dev.to WebDev

Originally published by Dev.to WebDev. Aggregated on AIWithGhost for educational purposes β€” full credit and traffic to the original publisher.