Dev.to WebDev 🛠 Dev 👁 0 📖 5 min read

How to Make a Small Website Crawlable: Sitemap, robots.txt and Internal Links

A practical checklist for developers who built a site and now wonder why Google is not showing it. In this tutorial, we’ll be seeing how the XML sitemap, robots.txt, and your internal linking structure help decide that

A practical checklist for developers who built a site and now wonder why Google is not showing it.

In this tutorial, we’ll be seeing how the XML sitemap, robots.txt, and your internal linking structure help decide that for your website. Weeks pass, and when you are trying to find your site through search, it does not show up. Do not rush to re-write your content. First, do a simpler check – can search engines find and access your pages at all?

In this tutorial, we will see how your XML sitemap, robots.txt, and the internal linking of your small website decide this for you. There is no need for any plugin or premium tool; just an easy-to-use text editor is all that is needed.

Crawling, indexing and ranking are different steps

There is a sequence of steps involved in the process of working of search engines. The first step is that the search engine finds the URL; the second step involves crawling the website. It then decides whether to include the URL in its indexes or not, and finally it ranks the site. Crawlability problems will hinder this process either at step one or step two, and hence no matter how good the content may be, it won’t serve any purpose for the site.

1. The XML sitemap: a list of pages you want found

A sitemap is a file, usually at /sitemap.xml, that lists the URLs you want search engines to know about. It is a hint, not a command, but for new or small sites it speeds up discovery a lot. Google explains how it works in its sitemap overview, and the format itself is defined at sitemaps.org.

A minimal example:

<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <url>
    <loc>https://example.com/</loc>
    <lastmod>2026-11-20</lastmod>
  </url>
  <url>
    <loc>https://example.com/mens-jackets/</loc>
    <lastmod>2026-11-18</lastmod>
  </url>
</urlset>

Rules that make your life easier:

  • Only list pages you want indexed: 200 status codes with no redirection or blocking.
  • Don’t forget your protocol and domain: https://example.com/page and not just /page.
  • Be honest about lastmod – update it only when it really changes.
  • A single sitemap file can contain up to 50,000 URLs or 50MB of uncompressed data; small sites will never reach such numbers.

Submit it using Search Console, as well as refer to it in your robots.txt (shown below).

2. robots.txt: tell crawlers where not to waste time

The robots.txt file lives at the root of your domain (https://example.com/robots.txt). It tells crawlers which paths they should not request. Google's robots.txt introduction is a good reference.

User-agent: *
Disallow: /cart/
Disallow: /checkout/
Disallow: /search

Sitemap: https://example.com/sitemap.xml

Three common mistakes:

  1. Robots.txt is not for privacy protection. The excluded URL may be included in search results due to links from other pages pointing to it. For exclusion from indexation, apply noindex meta tag or password protection, but allow crawlers to access the page to recognize the tag.

  2. Do not block CSS and JavaScript necessary for rendering the page. They are used by crawlers to analyze your page structure.

  3. Look for Disallow: / directive. Quite often everything is blocked on the staging website, and the developers forget about it when launching. One single line can exclude your website.

3. Internal links: the roads between your pages

Crawlers move through your site by following links. If a page has no link pointing to it, it is an orphan page, and it may only be found through the sitemap, if at all.

The secret of good internal linking lies in your habits:

  • Each significant page must be no further than a couple of clicks away from the homepage.
  • Don't create non-clickable links with a JavaScript function – Google explicitly recommends that links created in this way may go unnoticed.
  • Anchor text should be meaningful. "Men's shearling jackets" provides much more information than just "click here".
  • Connect similar pages, e.g. a category page to its products, a product page to its category, a guide to the relevant pages.
<!-- Crawlable -->
<a href="/mens-jackets/">Men's leather jackets</a>

<!-- Easy to miss -->
<span onclick="location.href='/mens-jackets/'">Men's leather jackets</span>

A real-world example: a small online store

Small online stores show all three of these problems in one place. Take a leather jacket shop like celebsjacket.com, a store I work on from the SEO side. A typical structure looks like this:

  • HOME PAGE
  • Pages of categories (men’s jackets, women’s jackets, shearling, bombers)
  • Page for each individual jacket
  • Utility pages, such as cart, checkout and search

The application of three tools looks as follows:

  • Sitemap: include the homepage, category and product pages. Do not include cart, checkout and internal search results.
  • robots.txt: disallow the paths of cart, checkout and search so that crawlers concentrate on pages that can really rank.
  • Internal links: all products must be linked from the page of corresponding category via a normal <a href> link, and categories must be linked from the main navigation. Products which can be accessed only after filtering or using a JS widget are orphan pages.

This model suits practically any small store, blog and portfolio. Replace “jackets” by “articles” or “projects” and you will still have the same checklist.

How to check your own site in 10 minutes

  1. Open https://yourdomain.com/robots.txt in a browser and read it line by line.
  2. Open https://yourdomain.com/sitemap.xml and confirm it loads and lists the right URLs. From a terminal you can also run curl -I https://yourdomain.com/sitemap.xml and look for a 200 status.
  3. In Google Search Console, submit the sitemap and run URL Inspection on an important page to see whether it is indexed and when it was last crawled.
  4. Search site:yourdomain.com in Google for a rough idea of how many pages are indexed.
  5. Click through your own navigation. If a page needs a search box or a script to reach it, add a normal link.

Common mistakes at a glance

  • Redirects and blocked URLs on a sitemap
  • Important pages without any internal links pointing to them
  • Navigation based completely on click events in JavaScript code
  • Blocking essential CSS and JS files

Wrap-up

Crawlability is the boring aspect of SEO, but it is the backbone – a page that cannot be crawled cannot be ranked. Correctly configured sitemaps, robot.txt and proper internal links usually solve issues for small websites.

📰 Read the original article on Dev.to WebDev

Originally published by Dev.to WebDev. Aggregated on AIWithGhost for educational purposes — full credit and traffic to the original publisher.