Dev.to AI 🤖 Ai 👁 0 📖 8 min read

A copy of the MCP registry that remembers what each server's tools/list returned

The official MCP registry answers one question well: what did a publisher declare? A name, a version, a remote URL or a package, a description. It does not say whether that URL answers without an account, how many tools

A copy of the MCP registry that remembers what each server's tools/list returned

The official MCP registry answers one question well: what did a publisher declare? A name, a version, a remote URL or a package, a description. It does not say whether that URL answers without an account, how many tools it offers, or whether the tool list changed last Tuesday. Nothing in the registry is wrong about that; it was never meant to record it.

That gap matters more now that registries are becoming policy. GitHub Copilot lets an organisation or enterprise admin enter an "MCP Registry URL" and choose between "Allow all", where the registry's servers are recommendations, and "Registry only", where developers can run only what the registry lists (GitHub docs; public preview). The admin who builds that list sees whatever the registry holds, which is the declaration.

HISTOR, the log we run, has been reading the tool lists of the servers in that registry since October: every remote without credentials on a schedule, and every npm and PyPI server package once per published version. Since 21 November it also serves a copy of the registry in the registry's own v0.1 API, with what it saw attached to each server. The base URL is https://histor.modelmarket.dev.

What an entry looks like

Every route of the official API is there: GET /v0.1/servers with cursor, limit, search, updated_since and version, plus /v0.1/servers/{name}/versions and /v0.1/servers/{name}/versions/{version} (or latest). The server object and the registry's own _meta block come back exactly as the official registry sent them. HISTOR's part sits next to them under its own key, dev.modelmarket.histor/observed, which is how the registry's guidance for aggregators asks a subregistry to add data.

Here is our own scanner, WARDEN, read both ways (curl and jq; any recent version):

curl -s 'https://registry.modelcontextprotocol.io/v0.1/servers/io.github.alexar76%2Fwarden/versions/latest' \
  | jq '._meta | keys'
curl -s 'https://histor.modelmarket.dev/v0.1/servers/io.github.alexar76%2Fwarden/versions/latest' \
  | jq '._meta["dev.modelmarket.histor/observed"].targets[] | {endpoint, lastStatus, toolCount, unchangedSince, changes}'
[
  "io.modelcontextprotocol.registry/official"
]
{
  "endpoint": "npm:@aimarket/warden",
  "lastStatus": "ok",
  "toolCount": 9,
  "unchangedSince": "2026-11-18T20:23:31Z",
  "changes": 5
}

Output as of 21 November 2026. One target per remote URL or package the entry declares. Besides the fields above, each target carries the digest of the tool set HISTOR last recorded, the digest of the tool annotations and other extras, when it first pinned the tool list, when it last read it, and a link to the server's page in the log, where every recorded version of the tool list can be diffed.

A diagram. Two sources feed one registry entry. The official MCP registry (what the publisher declared: name, version, remotes, packages) is copied hourly with updated_since and walked in full weekly. The HISTOR crawler (tools/list of each remote without credentials, and of each npm or PyPI package version, in a signed log) is joined by server name. HISTOR's copy stores server and the official _meta as read and adds _meta dev.modelmarket.histor/observed on the latest version: per remote or package, lastStatus, toolCount, toolSetDigest, unchangedSince and changes. The copy answers GET /v0.1/servers and /v0.1/servers/{name}/versions/{version or latest}. Three readers: a Copilot policy's MCP Registry URL (from GitHub docs, public preview, not tested by us), a reviewer using curl, jq or check_list.py, and your own HISTOR copy with an allow list whose upstream is histor.modelmarket.dev.

Most of what it can say, and the part it can't

A record of tool lists is only as good as the share of servers it could read. We walked the whole copy on 21 November and put every server in one bucket, first match wins:

A horizontal bar chart of 69,634 servers, the latest version of each in HISTOR's copy of the registry. Tool list recorded: 41,970 (60.3 %). Asks for credentials, HTTP 401 or 403: 9,612 (13.8 %). robots.txt disallows the crawler or could not be read: 5,258 (7.6 %). Failed (package did not install or exited, 404 or 410, protocol error, private address, timeout): 7,238 (10.4 %). Not attempted (package type, transport or templated URL HISTOR does not read): 4,376 (6.3 %). No observation block yet: 1,180 (1.7 %). Of the servers with a recorded tool list, 6,709 have it only from an earlier read, 18,368 changed it at least once and 6,545 in the 7 days before the walk.

What the observed block says Servers Share
tool list recorded 41,970 60.3 %
asks for credentials (401 / 403) 9,612 13.8 %
robots.txt disallows the crawler, or could not be read 5,258 7.6 %
failed: package did not install or exited, 404/410, protocol error, private address, timeout 7,238 10.4 %
not attempted: package type, transport or templated URL 4,376 6.3 %
no observation block yet 1,180 1.7 %

Source: 69,634 entries read from https://histor.modelmarket.dev/v0.1/servers?version=latest between 17:02 and 17:11 UTC on 21 November 2026, counted with count.py; the per-bucket rules are in its docstring and the result is counts-2026-11-21.json. These are HISTOR's own reads of public definitions, not a test of what the servers do.

For 41,970 servers the copy can tell you how many tools the server offered when it was last read and whether that list has moved: 18,368 of them changed it at least once since HISTOR first recorded it, 6,545 in the last week. For 6,709 of the 41,970 the list comes from an earlier read; the latest one failed, or the site's robots.txt now disallows the crawler. lastStatus and lastObserved say which, so a stale list does not pass for a fresh one. Finding any of this out otherwise means connecting to each server yourself.

The second bar is the honest limit. HISTOR reads without credentials, so a server that asks for an account answers 401 and gets a status instead of a tool list. That includes the servers people are most likely to approve first. GitHub's remote server answers 401, and its container image is a package type HISTOR does not run. Notion's remote answers 401 too. For those, the copy says "we could not read this, and here is why", which is still more than the registry says, but it is not a tool list. The robots bar is a choice on our side: HISTOR honours robots.txt and does not change its user agent to get past it.

Check a list before you approve it

If you already keep a list of server names, a short script reads the copy for each one. check_list.py needs Python 3.9 or newer and nothing outside the standard library:

"""For each server name in a file, print what HISTOR recorded about its latest registry version.

    python3 check_list.py names.txt
Python 3.9+, standard library only. Reads https://histor.modelmarket.dev/v0.1/servers/{name}/versions/latest.
"""
import json
import sys
import urllib.error
import urllib.parse
import urllib.request

BASE = "https://histor.modelmarket.dev/v0.1/servers/"
KEY = "dev.modelmarket.histor/observed"

for line in open(sys.argv[1]):
    name = line.split("#", 1)[0].strip()
    if not name:
        continue
    url = BASE + urllib.parse.quote(name, safe="") + "/versions/latest"
    try:
        with urllib.request.urlopen(url, timeout=30) as r:
            entry = json.load(r)
    except urllib.error.HTTPError as e:
        print(f"{name}: {e.code} (not in the registry copy)")
        continue
    print(f"{name} {entry['server']['version']}")
    observed = entry["_meta"].get(KEY)
    if not observed:
        print("  no observation yet")
        continue
    for t in observed["targets"]:
        if t["toolCount"] is not None:
            print(f"  {t['endpoint']}: {t['toolCount']} tools, changed {t['changes']}x, "
                  f"unchanged since {t['unchangedSince']}, read {t['lastObserved']}")
        else:
            print(f"  {t['endpoint']}: not read ({t['notAttempted'] or t['lastStatus']})")
printf 'io.github.alexar76/warden\nio.github.github/github-mcp-server\ncom.notion/mcp\nio.example/not-there\n' > names.txt
python3 check_list.py names.txt
io.github.alexar76/warden 0.35.0
  npm:@aimarket/warden: 9 tools, changed 5x, unchanged since 2026-11-18T20:23:31Z, read 2026-11-20T20:28:37Z
io.github.github/github-mcp-server 2.2.0
  https://api.githubcopilot.com/mcp/: not read (http-401)
  oci:ghcr.io/github/github-mcp-server:2.2.0: not read (package-type-not-observed)
com.notion/mcp 1.0.1
  https://mcp.notion.com/mcp: not read (http-401)
  https://mcp.notion.com/sse: not read (transport-not-observed)
io.example/not-there: 404 (not in the registry copy)

Run on 21 November 2026. The last name is made up, to show what an unknown name returns.

Your own copy, with only your servers

An organisation's registry is usually a short list, not 69,000 servers. HISTOR can run as exactly that: no crawl of its own, an allow list of names or namespace/* lines, and an upstream that can be the official registry or another subregistry. With histor.modelmarket.dev as the upstream, the observations come along. The published aimarket-histor 0.10.0 on PyPI predates the subregistry, so install from the public source at a fixed commit (Python 3.11 or newer):

python3 -m venv .venv
.venv/bin/pip install "git+https://github.com/alexar76/histor@fb35e7a0681a3ec009d61ccf49ed1858ab6dbc63"
cat > allow.txt <<'EOF'
# servers our developers may use
io.github.alexar76/warden
io.github.github/github-mcp-server
com.notion/*
EOF
export HISTOR_SUBREGISTRY=1 \
  HISTOR_SUBREGISTRY_UPSTREAM=https://histor.modelmarket.dev \
  HISTOR_SUBREGISTRY_ALLOW="$PWD/allow.txt" \
  HISTOR_CRAWL_INTERVAL_S=0 HISTOR_REMOTE_INTERVAL_S=0 \
  HISTOR_DATA_DIR="$PWD/data" HISTOR_PORT=9491
.venv/bin/python -m histor.subregistry sync --full
.venv/bin/python -m histor serve &
sleep 5
curl -s http://127.0.0.1:9491/v0.1/servers \
  | jq -r '.servers[] | [.server.name, .server.version, (._meta["dev.modelmarket.histor/observed"].targets | map(.lastStatus // .notAttempted) | join(","))] | @tsv'
{"at": "…", "mode": "allow-list", "upstream": "https://histor.modelmarket.dev", "pages": 3, "read": 3, "stored": 3, "skipped": 0, "filtered": 0, "gone": 0}
com.notion/mcp  1.0.1   http-401,transport-not-observed
io.github.alexar76/warden   0.35.0  ok
io.github.github/github-mcp-server  2.2.0   http-401,package-type-not-observed

The sync's at field, the local clock's time, is left out above. With an allow list the sync asks the upstream for each listed name instead of walking everything, so the first copy took about four seconds here. The two crawl intervals at 0 mean this instance reads no servers itself; it only keeps the list and serves it. The defaults are SQLite and a development profile. For anything other people rely on, HISTOR wants PostgreSQL and HISTOR_PROFILE=prod, which refuses to start without an https upstream, an https public base and an operator token.

How the copy is kept

The sync follows the routine in the registry's aggregator guidance. The first run walks version=latest; after that it asks for updated_since a checkpoint, with ten minutes of overlap, and moves the checkpoint only once every page has been applied, so an interrupted sync repeats rather than skips. Entries are upserted by name and version. Once a week it walks everything again and marks servers the upstream no longer lists. That step has a guard: the registry is in preview and warns about data resets, so a full walk that returns fewer than half the servers the copy holds marks nothing as gone.

On production the first full copy took about 19.5 minutes (10:28:06 to 10:47:41 UTC): 695 pages, 69,484 entries read, 69,483 stored, one skipped. We then picked 30 entries at random and compared server and the official _meta with the official registry's versions/latest: all 30 were identical. Hourly syncs have run since; the status route, /api/v1/subregistry, shows the last one.

The code is one module, histor/subregistry.py, about 600 lines, with the routes, the sync, the allow list and the shaping of _meta.

What it does not do

  • We have not connected it to a Copilot organisation. The routes, the URL form (a base URL; Copilot appends /v0.1 itself, per the docs) and the CORS headers (Access-Control-Allow-Origin: *, methods GET, OPTIONS) are what the documentation asks for, and the responses match the official registry's, but that is not the same as having watched Copilot read it. If you try it, we would like to hear what happened.
  • Only the latest version of a server carries an observation. Older versions in the copy come back as the registry sent them.
  • An observation is what HISTOR's crawler received without credentials. A server can offer different tools to a logged-in user.
  • It is one public instance with no uptime promise, rate-limited to 600 requests a minute per address, with responses cacheable for five minutes. If your hosts depend on it, run your own copy with it as the upstream, as above.
  • It does not filter or score anything. Every server the registry lists is in the copy, whatever its observation says. Deciding what to allow stays with you.

We run HISTOR and build WARDEN. AI assistance was used for the code, the count, the figures and this article.

📰 Read the original article on Dev.to AI

Originally published by Dev.to AI. Aggregated on AIWithGhost for educational purposes — full credit and traffic to the original publisher.