Cite or It Didn't Happen: Line-Pinned Claims for Model-Assisted CI Triage

A red pipeline lands at 02:00. The log is 9,000 lines long.

You paste the tail into a model. Four seconds later it names a confident root cause. You merge the fix, and the pipeline fails again for a different reason.

Sound familiar? This post is a myth-busting FAQ about that exact moment.

Each myth below gets a claim, the evidence, and a corrected mental model. The artifact is a triage script that makes the model pin claims to log lines, then verifies those pins with plain Python.

Myth 1: "The model read the whole log."

Claim: If the model answered, it saw your entire job trace.

Evidence: You cannot observe what the endpoint did with your input. Context limits and silent truncation live outside your log file.

Corrected model: Treat every model read as partial. Force it to prove it reached the tail.

Append a canary line before you send anything.

CANARY = "CANARY_7f3a91b2_MUST_ECHO"

def append_canary(log: str) -> str:
    return log + f"\n999999| {CANARY}\n"

The prompt must require that canary string inside the JSON reply. If the reply omits it, the request was truncated. Refuse the verdict and shrink the input instead.

I point this script at MonkeyCode's free model access, and I run the triage step on the free server option it offers. Both are operator-supplied availability claims, not benchmark numbers. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Myth 2: "Line 412 was cited, so line 412 proves it."

Claim: A cited line number makes the model's conclusion verifiable.

Evidence: A citation is a string. Nothing stops the model from pairing a real line number with an unrelated pattern.

Corrected model: Make the model ship the evidence, not a pointer to evidence. Ask for a regex that must match the cited line.

Number your log lines first, so both sides share one coordinate system.

def number_lines(log: str) -> list[str]:
    return [f"{i + 1:>6}| {line}" for i, line in enumerate(log.splitlines())]

Then require a verdict shaped like this.

{
  "canary": "CANARY_7f3a91b2_MUST_ECHO",
  "claims": [
    {
      "lines": [412, 413],
      "pattern": "denied: requested access to the resource is denied",
      "summary": "registry push rejected for the job token"
    }
  ]
}

Now verification is boring, deterministic work.

import re

def verify(verdict: dict, numbered: list[str]) -> list[str]:
    problems: list[str] = []
    if verdict.get("canary") != CANARY:
        problems.append("canary missing: input was truncated")
    for claim in verdict.get("claims", []):
        pattern = re.compile(claim.get("pattern", r"$^"))
        for n in claim.get("lines", []):
            if not 1 <= n <= len(numbered):
                problems.append(f"line {n}: out of range")
            elif not pattern.search(numbered[n - 1]):
                problems.append(f"line {n}: no match for {pattern.pattern!r}")
    return problems

Exit non-zero when problems is non-empty. The model now fails loudly instead of confidently.

Myth 3: "Redaction is optional because the endpoint is free."

Claim: A free endpoint makes log hygiene less important.

Evidence: Job traces leak tokens, registry URLs, internal hostnames, and sometimes customer identifiers. Price has nothing to do with blast radius.

Corrected model: Redact before the request, not after the incident.

SECRETS = [
    (re.compile(r"glpat-[A-Za-z0-9_\-]{20,}"), "glpat-<redacted>"),
    (re.compile(r"(?i)authorization:\s*bearer\s+\S+"), "Authorization: Bearer <redacted>"),
    (re.compile(r"\b[A-Za-z0-9+/]{40,}={0,2}\b"), "<redacted-blob>"),
]

def redact(text: str) -> str:
    for pattern, replacement in SECRETS:
        text = pattern.sub(replacement, text)
    return text

This is best-effort, not a guarantee. Add your own patterns for hostnames and project paths.

Myth 4: "A free server turns CI into a free service."

Claim: If the model host is free, the whole pipeline is free.

Evidence: Model hosting and pipeline compute are different systems with different trust boundaries.

Corrected model: Keep untrusted code execution on your runner. Use the remote box for reading text, not running jobs.

Ask one question before you move any step: does this step execute code I did not write? If yes, it stays on the runner.

Myth 5: "The verdict replaces the failing test."

Claim: A good model summary means the failure is understood.

Evidence: A summary that survives the citation check can still explain the wrong layer. Green triage is not green CI.

Corrected model: Treat triage output as a hypothesis with a citation, never as a gate.

That table is the whole point. Triage narrows the search space. It does not close the incident.

A minimal runbook

1. Pull the job trace with a read-only API token.
2. Redact tokens, bearer headers, and long opaque blobs.
3. Append the canary line and number every line.
4. Request JSON containing the canary plus line-pinned claims.
5. Run verify(). Drop the reply if any pin fails.
6. Post the surviving claims as a merge request comment.

Step 6 is optional. Steps 2 through 5 are not.

Limitations and who should skip this

The citation check proves a pattern matched a line. It never proves the interpretation is correct.

Model output is non-deterministic, so two runs can disagree. Never alert on a single verdict.

Do not use this where logs are regulated or residency-bound. Do not use it for security findings, secret rotations, or anything needing a compliance trail. Skip it entirely if your team requires a deterministic gate, because this is a hypothesis generator, not a gate.

Redaction regexes also drift. Re-check them whenever your CI images change.

Closing thought

The useful question is not "did the model sound smart?" Ask instead: "can I verify the claim without trusting the model?"

Canary plus line-pinned regex is one small way to answer yes. If you wire it up, I want to know which myth broke first for you.