Your Model Alias Is Not a Version: A Drift Canary for Free Model Access

The claim that keeps showing up

"Nothing changed on my side." I see this in every eval regression thread.

The model string in the config is identical. The prompt file is untouched. So the blame lands on randomness, right?

Not quite. A model name is a pointer. It is not a checksum.

Here are five things developers repeat. Each one sounds reasonable. Each one can cost you a week of debugging.

1. "I pinned model-x in my config, so my results are pinned."
2. "Temperature 0 means the same input returns the same output."
3. "If the answers still look good, the model is the same."
4. "One great demo proves the endpoint is healthy."
5. "My free server can run these checks unattended, forever."

Let's take them apart, one at a time.

Q1: Does a pinned model name pin the weights?

Short answer: no. Check your provider's docs for two separate concepts.

• Aliases. A moving label that points at the current snapshot. Providers rotate these on purpose.
• Dated snapshots. A name with a date suffix. Usually frozen, but still inside a deprecation window.

Ask yourself which one you actually put in your config. Most projects pin an alias and call it a version.

What you should log on every run:

• the model string you requested
• the model string the response echoed, if it echoes one
• UTC timestamp
• the endpoint base URL

That last one matters. The same alias can route to different backends over time.

The corrected mental model

An alias is a DNS record. A dated snapshot is an IP address. Both can move, but one moves far more often.

Verify the current mapping in your provider's documentation. Do not trust a blog post, including this one.

Q2: Is temperature 0 reproducible?

Here is the question nobody asks: does temperature=0 disable sampling? No. It turns one knob.

Other sources of variance stay in play:

• batched inference groups your request differently each call
• floating-point accumulation order is not fixed
• mixture-of-experts routing and speculative decoding add jitter
• one alias can be served by several backend pools

You can test this in a minute. Loop the same prompt twenty times and count distinct outputs.

# unexecuted example, plug in your own client
outs = {client.chat(prompt, temperature=0) for _ in range(20)}
print(len(outs), "distinct outputs")

If that number is bigger than one, your exact-match assertions are already broken.

Corrected model: a model version is a distribution, not a binary. You compare distributions, not single strings.

Q3: Can I just eyeball the drift?

No. Fluency is the worst possible detector.

When a snapshot changes, the output usually reads better, not worse. It gets shorter, or friendlier, or more structured. Nothing looks wrong. That is exactly the problem.

Cheap things you can measure instead:

• Self-consistency: how often K samples of one prompt agree.
• Output length distribution: stable models keep a stable shape.
• Character n-gram overlap: a crude but effective fingerprint against a stored baseline.
• Error and refusal rate: tracked separately, never mixed into drift.

None of these prove the weights are identical. Together they catch the swaps that break you.

The artifact: a two-command drift canary

Label: proposed and unexecuted in your environment. Standard library only, OpenAI-compatible endpoint.

Start with a prompt file. Keep it small and boring on purpose.

{"id":"p01","prompt":"Reply with exactly: OK"}
{"id":"p02","prompt":"List the three primary colors. Lowercase, comma separated."}
{"id":"p03","prompt":"What is 17 * 23? Reply with the number only."}
{"id":"p04","prompt":"How many days are in February 2024? Number only."}

Twelve prompts is a sane starting set. Constrained tasks expose formatting changes fast.

Now the recorder and comparer:

#!/usr/bin/env python3
"""canary.py - fingerprint an OpenAI-compatible chat endpoint.

usage:
  python canary.py record  runs/baseline.json
  python canary.py record  runs/today.json
  python canary.py compare runs/baseline.json runs/today.json
"""
import json, os, statistics, sys, time, urllib.request
from collections import Counter

BASE = os.environ["LLM_BASE_URL"].rstrip("/")
MODEL = os.environ["LLM_MODEL"]
KEY = os.environ.get("LLM_API_KEY", "")
K = int(os.environ.get("CANARY_SAMPLES", "5"))
PROMPTS = os.environ.get("CANARY_PROMPTS", "prompts.jsonl")

def call(prompt):
    body = json.dumps({
        "model": MODEL,
        "temperature": 0,
        "messages": [{"role": "user", "content": prompt}],
    }).encode()
    req = urllib.request.Request(
        BASE + "/chat/completions", data=body,
        headers={"Content-Type": "application/json",
                 "Authorization": "Bearer " + KEY})
    with urllib.request.urlopen(req, timeout=60) as r:
        d = json.load(r)
    return d["choices"][0]["message"]["content"] or ""

def grams(texts, k=3):
    out = set()
    for t in texts:
        t = " ".join(t.lower().split())
        out |= {t[i:i + k] for i in range(max(0, len(t) - k + 1))}
    return out

def record():
    rows = []
    for line in open(PROMPTS):
        item = json.loads(line)
        outs = []
        for _ in range(K):
            try:
                outs.append(call(item["prompt"]))
            except Exception as e:
                outs.append("__ERR__" + type(e).__name__)
            time.sleep(0.3)
        ok = [o for o in outs if not o.startswith("__ERR__")]
        top = Counter(ok).most_common(1)
        rows.append({
            "id": item["id"],
            "n_ok": len(ok),
            "self_consistency": round(top[0][1] / len(ok), 3) if ok else 0.0,
            "mean_len": round(statistics.mean([len(o) for o in ok])) if ok else 0,
            "grams": sorted(grams(ok))[:600],
        })
    return rows

def compare(base_path, cur_path):
    base = {r["id"]: r for r in json.load(open(base_path))["rows"]}
    cur = {r["id"]: r for r in json.load(open(cur_path))["rows"]}
    drift = []
    for pid, b in base.items():
        c = cur.get(pid)
        if not c:
            drift.append((pid, "missing", 0.0))
            continue
        gb, gc = set(b["grams"]), set(c["grams"])
        j = len(gb & gc) / max(1, len(gb | gc))
        dl = abs(c["mean_len"] - b["mean_len"]) / max(1, b["mean_len"])
        if j < 0.90 or dl > 0.15:
            drift.append((pid, "jaccard=%.2f dlen=%.0f%%" % (j, dl * 100), j))
    for row in sorted(drift, key=lambda x: x[2]):
        print(*row)
    print("drifted prompts:", len(drift), "/", len(base))

if __name__ == "__main__":
    if sys.argv[1] == "record":
        rows = record()
        json.dump({"model": MODEL, "ts": int(time.time()), "rows": rows},
                  open(sys.argv[2], "w"))
    else:
        compare(sys.argv[2], sys.argv[3])

Run record once against a known-good week. That file is your baseline. Then run it again after any unexplained metric shift.

Adjust the thresholds to your own noise floor. 0.90 and 15% are starting guesses, not laws.

Scheduling it on a free server

A canary is a tiny workload. A handful of short requests per cycle is enough.

That makes a free server a legitimate host for the scheduler. MonkeyCode offers free model access and a free server option, which covers both halves of this loop.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Size your cadence to your own quota. I cannot tell you your limits, and running 12 prompts x 5 samples every 30 minutes adds up fast. Hourly with 3 samples is a calmer default.

# /etc/cron.d/canary
17 * * * * runner cd /opt/canary && flock -n .lock ./run.sh >> canary.log 2>&1

#!/usr/bin/env bash
set -euo pipefail
cd /opt/canary
STAMP=$(date -u +%Y%m%dT%H%M)
python3 canary.py record "runs/$STAMP.json"
python3 canary.py compare runs/baseline.json "runs/$STAMP.json" > "reports/$STAMP.txt" || true
git add -A && git commit -qm "canary $STAMP" && git push -q

The free-server caveats matter here:

• Idle sleep breaks an hourly promise. A gap is not evidence of stability.
• Ephemeral disk loses runs on restart. Push results off-box every cycle.
• Egress and quota limits truncate runs mid-way. Log n_ok so you notice.
• No SLA means your monitor's uptime is also a variable, not a constant.

Reading the output: a decision table

The last row is the important one. A clean canary run proves very little. It just means you did not catch a change this hour.

Where this workflow breaks

• It cannot prove weights are identical. It only observes behaviour.
• Twelve prompts is a smoke test, not a regression suite.
• Thresholds are guesses until you measure your own variance.
• Errors masquerade as drift if you never separate the two.
• Baselines decay silently. You must re-take them after planned upgrades.

Who should skip this

• Prototypes where a model swap costs you nothing.
• Teams with a contract that guarantees changelogs and notice windows.
• Anyone who cannot store baselines outside the free box.
• Anyone who needs a reproducibility guarantee rather than a detector.

The corrected mental model

1. A model name is a pointer, and pointers move.
2. Temperature 0 narrows a distribution. It never collapses it.
3. Compare distributions, never single outputs.
4. Errors and drift are different signals. Label them separately.
5. Every baseline has an expiry date. Re-take it deliberately.

So here is my question for you: what twelve prompts would you put in your set?

Pick the tasks where a silent change would hurt most. Write them down today. Then let the canary tell you when the pointer moved.