Learn Weak Oracles by Building a Tiny Spec Checker

Thursday night in Halifax. Rain on the window, a retrieval lab due at 9 a.m., and a notes bot that had just dumped a confident paragraph about “vector space understanding.” My test was green. I almost closed the laptop.

Then I read the paragraph again. Had I tested the assignment, or had I tested whether a model could emit fifty words that looked like a lab note?

Here is the fixture that should have failed, and the printout I wanted before I trusted the green check:

NAIVE: PASS
SPEC : FAIL
  - missing required term: cosine similarity
  - forbidden claim: understands the document
  - top-k out of bounds: 64

Single learning question: if a generated answer can pass a length-and-keyword oracle while violating the rubric, what tiny checker would have caught it on my machine, before I pasted it into a PDF?

This is a case study of one evening project. Background, goal, a runnable checker, the results on two canned “model” outputs, and what I will not pretend this is.

Background

I am an AI student. I use generators the way I used to use Stack Overflow: as a draft, not as a witness. The trouble is the draft now arrives already wearing a lab-report costume. A weak oracle — a test that only asks “did something show up?” — will clap for the costume.

That is not a hot take about the industry. It is a homework problem. The campus Slack this month has been full of people arguing whether “vibe coding” is engineering. I do not have a manifesto. I have a due date. If the only test I wrote was len(text) > 50 and "vector" in text, I was not evaluating retrieval. I was evaluating vibes.

The assignment itself was ordinary. Explain how a toy retriever ranks lecture chunks. Mention cosine similarity. State a top-k in a sane range. Do not claim the model “understands” the document. That last line is the one generators love, and the one my naive test never looked for.

I still needed a place to sample candidate paragraphs when my laptop was in class. Disclosure: This article was prepared as part of MonkeyCode's product outreach. I used MonkeyCode’s free model access and the free server option for that generation half, then brought the text home. The checker below does not talk to any vendor. If you delete every product name in this piece, the lesson is unchanged: the oracle lives next to the rubric, not next to the autocomplete.

Goal

I wanted a contract I could read in one screen. Required terms. Forbidden claims. A numeric bound on top-k. Two fixtures: one that a lazy test would bless, one that a picky rubric should bless. No framework. No embeddings library. Python 3.11, standard library only.

If the bad fixture passed the naive oracle and failed the spec, the project succeeded. If both fixtures passed the spec, I had written another costume.

Implementation

Save this as tiny_spec_checker.py. I ran it with python3 tiny_spec_checker.py on Python 3.11. Nothing else to install.

#!/usr/bin/env python3
"""tiny_spec_checker.py — catch answers that pass a weak oracle."""
from __future__ import annotations

import re
import sys
from dataclasses import dataclass


@dataclass(frozen=True)
class Spec:
    required_terms: tuple[str, ...]
    forbidden_claims: tuple[str, ...]
    k_min: int = 1
    k_max: int = 20


@dataclass(frozen=True)
class Verdict:
    name: str
    naive_pass: bool
    spec_pass: bool
    reasons: tuple[str, ...]


K_RE = re.compile(r"\btop[\s-]*k\s*=\s*(\d+)\b", re.I)

RUBRIC = Spec(
    required_terms=("cosine similarity",),
    forbidden_claims=("understands the document", "truly understands"),
)

# The costume: long, mentions vectors, sounds finished.
BAD_NOTE = (
    "Our vector retriever truly understands the document collection "
    "and returns the most relevant lecture chunks with top-k = 64 "
    "so the student can skim the semester in one sitting."
)

# The boring passing note. No poetry. That is the point.
GOOD_NOTE = (
    "Rank lecture chunks with cosine similarity on bag-of-words vectors. "
    "Retrieve top-k = 5 neighbors, then quote the chunk ids in the answer. "
    "The score is geometry, not comprehension."
)


def naive_oracle(text: str) -> bool:
    return len(text) > 50 and "vector" in text.lower()


def check_spec(text: str, spec: Spec) -> tuple[bool, tuple[str, ...]]:
    reasons: list[str] = []
    low = text.lower()
    for term in spec.required_terms:
        if term.lower() not in low:
            reasons.append(f"missing required term: {term}")
    for claim in spec.forbidden_claims:
        if claim.lower() in low:
            reasons.append(f"forbidden claim: {claim}")
    match = K_RE.search(text)
    if match is None:
        reasons.append("missing top-k assignment")
    else:
        k = int(match.group(1))
        if not (spec.k_min <= k <= spec.k_max):
            reasons.append(f"top-k out of bounds: {k}")
    return (not reasons, tuple(reasons))


def judge(name: str, text: str, spec: Spec) -> Verdict:
    spec_pass, reasons = check_spec(text, spec)
    return Verdict(name, naive_oracle(text), spec_pass, reasons)


def render(v: Verdict) -> str:
    lines = [
        f"FIXTURE: {v.name}",
        f"NAIVE: {'PASS' if v.naive_pass else 'FAIL'}",
        f"SPEC : {'PASS' if v.spec_pass else 'FAIL'}",
    ]
    for reason in v.reasons:
        lines.append(f"  - {reason}")
    return "\n".join(lines)


def main() -> int:
    bad = judge("bad_note", BAD_NOTE, RUBRIC)
    good = judge("good_note", GOOD_NOTE, RUBRIC)
    print(render(bad))
    print()
    print(render(good))
    if not (bad.naive_pass and not bad.spec_pass and good.spec_pass):
        print("\nunexpected verdicts; the checker or fixtures drifted", file=sys.stderr)
        return 1
    return 0


if __name__ == "__main__":
    raise SystemExit(main())

Read the naive oracle once more. Fifty characters and the substring vector. That is a hallway motion sensor, not a lock. The spec is the lock: a phrase the rubric named, a phrase the rubric banned, and a top-k that would not silently turn a toy retriever into a dump of the whole corpus.

Why regex for top-k? Because students, and models, write top-k = 5, top k = 5, and top-k=5. I did not try to parse English. I tried to refuse a missing number. If your generator writes “we retrieve a handful of neighbors,” this checker should fail closed. That is a feature. Ask it out loud: do you want a lab note that never commits to k?

Results

Expected output, character for character on the fixtures above:

FIXTURE: bad_note
NAIVE: PASS
SPEC : FAIL
  - missing required term: cosine similarity
  - forbidden claim: understands the document
  - top-k out of bounds: 64

FIXTURE: good_note
NAIVE: PASS
SPEC : PASS

Both notes pass the naive oracle. Only one survives the rubric. That split is the whole artifact. The bad note is the error input. Feed it to a length check and you will ship it. Feed it to check_spec and you get three reasons, not a vibe.

I did generate other candidates on the free remote side. Some were worse. One said “cosine” without “similarity” and still felt scholarly in the first glance. The checker does not grade style. It asks whether the contract fired. When a candidate failed, I either edited the sentence or threw the sample away. I did not enlarge the naive oracle until the costume fit. That is the habit I am trying to break.

A common mistake is to “fix” the bad note by adding the word cosine somewhere in a footnote and leaving top-k = 64. The checker will still fail on bounds. Another mistake is to lowercase the forbidden claim in the spec and then miss Understands The Document — that is why check_spec compares on low. A third is to accept top-k = 0. Zero is an integer. Zero is not a retriever.

If you want to watch it reject a missing k, swap BAD_NOTE for a long vector paragraph with no digits. You should see missing top-k assignment. Predict that before you run it. If you cannot predict it, the spec is not in your head yet.

What I actually learned

After this file, I want a student to be able to say four things without a blog post in the way. A weak oracle measures presence, not contract. A rubric can be compiled into a few predicates. Generated text will hunt the predicates you forgot to write. Local fixtures beat a green demo.

I also learned where not to spend money. The expensive part of this lab was not inference. It was deciding what “done” meant. A free model and a free server are useful when the laptop is in a backpack and you want more samples. They are not a substitute for RUBRIC. If the remote half vanished tomorrow, I could still run tiny_spec_checker.py on the two strings in this file.

Limitations, said plainly. This is not an evaluation harness. It does not score faithfulness against retrieved chunks. It does not handle paraphrases of cosine similarity such as “normalized dot product of L2-unit vectors,” which is a real phrase and would fail my required-term check. It does not know Unicode dashes. It is a tripwire for the exact failure I shipped last month: a pretty paragraph that never named the geometry, claimed understanding, and quietly set k to something that would have printed half the corpus.

Who should not use this approach? Anyone submitting a paper with “we evaluated the model.” Anyone who needs inter-annotator agreement. Anyone replacing a human TA. Anyone whose spec is “sound smart.” If you cannot write the forbidden claims in advance, you do not have a spec. You have a mood.

Lessons I am keeping

The case study ends where the file ends. I will keep the split printout — NAIVE versus SPEC — at the top of any lab that involves a generator. I will write the failing fixture first, the way I should have written the failing CSV row in a different week, except this time the bug is semantic and the costume is fluent English.

Extension if you have twenty minutes: add a predicate that cosine similarity must appear before the first top-k, because a note that names k and then waves at geometry is how I used to pad word count. If your extra predicate fails GOOD_NOTE, you overfit the tripwire. If it only fails a new counterexample you wrote, you learned something. Paste the counterexample in a comment. I would rather collect ugly fixtures than collect applause.

If you need extra samples for your own rubric and you do not want that loop on a laptop in the Killam, MonkeyCode’s free model access and free server option are how I fetched candidates for this write-up. Keep the checker on disk. The green check that matters is the one you can explain.