Pin the Crash in a Characterization Test Before You Patch Upstream

A first-time contributor cloned a popular command-line tool and reproduced a parsing crash within minutes of setup. The stack trace pointed at a single helper, so the contributor inserted a defensive return and watched the process exit cleanly. The pull request still stalled because maintainers could not replay the old failure against the new tree. They needed a test that encoded the crash, not a narrative that the path now succeeded.

This article describes a characterization-first workflow for open-source bug patches inside libraries and command-line tools. The method records buggy behavior as an executable oracle before any production source file is changed. Free coding models may review that oracle for weak assertions after the crash is pinned locally. The procedure below is proposed sample material rather than a measured result from any one repository.

Happy-path diffs still bounce in review

Open-source maintainers read stranger diffs under time pressure and with incomplete local context around the crash. A patch that only proves a new success path leaves the original failure as folklore inside the issue thread. Characterization tests freeze observed behavior so later production edits cannot drift without a failing automated suite.

Michael Feathers described characterization tests for untested code that teams must change under real maintenance risk. The same idea helps first-time contributors who lack a mental model of every upstream caller. An oracle that fails on unpatched main and passes after the fix gives reviewers a replay button.

Common holes appear in first patches for crashing CLIs and libraries:

• The reproduction lives in a screenshot or comment instead of tests/.
• The new test asserts the desired API, so it cannot fail on main.
• The fix changes return values that other callers still depend on.
• Continuous integration stays green because the suite never sent the crashing input.

A useful oracle fails on the unpatched tree and passes on the patched commit for one named input. Maintainers can then replay the crash without reconstructing the original issue from screenshots and comment threads. Contributors should refuse to open the pull request until that replay exists in the test tree. Folklore in the issue tracker is not an acceptable substitute for that recorded executable failure.

Capture the crash before production edits

The contributor should stop after reproduction and write a test that documents current behavior, including the crash. That test may expect an exception, a nonzero process exit, or a known-bad stdout fixture on disk. Production files stay untouched until the oracle lives on a local branch that still matches upstream main. Only then does a later source change have a failing baseline that reviewers can move with confidence.

The following Python example is a proposed template for a CLI parser that dies on an empty flag value. It is unexecuted sample code and not a harvested result from any named upstream project. Reviewers should adapt file paths and exception names to the real tree before running any command. Inline comments mark the file as characterization work rather than as a finished contract test.

# tests/test_characterize_empty_flag.py
# Proposed characterization oracle. Unexecuted example.

import subprocess
import sys
from pathlib import Path

CLI = Path(__file__).resolve().parents[1] / "src" / "cli.py"


def run_cli(args):
    return subprocess.run(
        [sys.executable, str(CLI), *args],
        capture_output=True,
        text=True,
        check=False,
    )


def test_empty_flag_currently_crashes():
    """Pin the bug: empty --out should not be a silent success on main."""
    result = run_cli(["--out", "", "sample.txt"])
    # Characterization: record what main actually does today.
    assert result.returncode != 0
    assert "Traceback" in result.stderr or "ValueError" in result.stderr

On the unpatched tree this characterization test should pass because it still asserts the original crash. After the production fix, the same file must be rewritten to assert a controlled error message and a stable exit code. Both assertion blocks belong in the pull request notes so reviewers can inspect the behavioral flip. A short shell helper then makes that flip visible without rereading the entire test suite.

# proposed_oracle_flip.sh
# Unexecuted sample: prove the oracle on main, then after the assertion rewrite.
set -euo pipefail

git switch --detach main
python -m pytest tests/test_characterize_empty_flag.py -q
# Expect PASS on main because the crash still exists.

git switch --detach HEAD@{1}  # patched tree; adjust the ref as needed
python -m pytest tests/test_characterize_empty_flag.py -q
# After rewriting assertions, expect PASS for the intended contract.

Label the two assertion blocks in the pull request body as oracle-before and oracle-after for later readers. Maintainers can replay the failure story without guessing which exception used to appear on standard error. Contributors should attach the helper output as a gist or a PR comment rather than a screenshot. Those labels keep the discussion tied to files instead of vague adjectives like fixed and broken.

A three-ref behavior grid

A single passing test on the feature branch remains a weak signal for busy upstream maintainers. The same crashing input should run against the last release tag, current main, and the patched commit. The grid shows whether the crash is a regression, a long-standing hole, or an environmental false alarm. Contributors should read that comparison table before they touch production parsers or shared helpers.

The sample script assumes a clean throwaway clone because detached HEAD checkouts will disrupt a dirty working tree. Copy the repository into a scratch directory before looping across refs. Record fingerprints of stderr instead of full traces that include local paths. The script below is unexecuted sample material and will need real tag names.

#!/usr/bin/env bash
# proposed_behavior_grid.sh — unexecuted sample
set -euo pipefail
INPUT_ARGS=(--out "" sample.txt)
REFS=("v1.4.0" "main" "HEAD")

echo "ref,exit,stderr_fingerprint"
for ref in "${REFS[@]}"; do
  if ! git switch --detach "$ref" >/dev/null 2>&1; then
    printf '%s,unbuildable,\n' "$ref"
    continue
  fi
  set +e
  python src/cli.py "${INPUT_ARGS[@]}" >out.txt 2>err.txt
  code=$?
  set -e
  fp="$(head -n 3 err.txt | tr '\n' ' ' | cut -c1-80)"
  printf '%s,%s,%s\n' "$ref" "$code" "$fp"
done

Read the grid before writing the production patch and keep the CSV in the request body. If the last release already crashes, the hole is old and the test belongs in the suite even without a fix. If only main crashes, a bisect remains a separate investigation and should not mix into this oracle work. Unbuildable refs should be recorded as unbuildable in the CSV rather than as invented exit codes.

Let a free model review the oracle, not the patch

Human reviewers still miss vacuous assertions, copied fixtures, and tests that never call the changed function. A coding model can critique the oracle file before the contributor edits any product source code. The prompt should ask for weak assertions, missing inputs, and accidental coupling to log wording. It should refuse any request to invent a production patch from the issue title alone.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode offers free model access and a free server option for this kind of oracle review. Contributors can paste the oracle test, the public issue excerpt, and the three-ref CSV into that workspace. The model should list holes in the characterization rather than generate a speculative production fix. No model name, quota, or hardware claim is made here beyond that operator-supplied free access option. Contributors who already pin crashes locally can use that free server option when they want a second reader on the oracle file.

A proposed review prompt, labeled as unexecuted sample text, follows in the fenced block below. The instructions ban product-code edits and ask only for defects inside the characterization oracle. Contributors should treat every bullet from the model as a hypothesis to prove against main. A mutated assertion that still passes on main means the check never encoded the crash.

You are reviewing a characterization test, not writing a patch.
Issue excerpt:
<paste the frozen acceptance lines>

Oracle file:
<paste tests/test_characterize_empty_flag.py>

Behavior grid CSV:
<paste ref,exit,stderr_fingerprint>

List:
1. Assertions that would pass even if the crash disappeared.
2. Inputs from the issue that the oracle never sends.
3. Log phrases that will drift across versions.
4. A minimal assertion rewrite for the post-fix contract.
Do not suggest product-code edits.

Treat the chat transcript as commentary rather than as evidence attached to the pull request. If the model flags a weak assertion, mutate that assertion on main and watch whether the suite still passes. That mutation check is the load-bearing step in this characterization workflow for reviewers. Maintainers should ignore a model "LGTM" that is not backed by the grid and the oracle flip.

Decision table for the oracle loop

The following table is a proposed decision aid, not a measured policy from a maintainer team. It keeps free-model review inside a narrow job that starts after the crash is pinned locally. Models help find vacuous asserts and missing inputs copied poorly from the public issue. They are a poor source of the original crash record because they were not present when the process died.

Limitations

Characterization tests freeze whatever the machine did today, including accidental environment details that will not travel. A traceback assertion that mentions a local virtualenv path will fail on CI and teach nothing about the parser. Golden files that capture full help text will churn on unrelated copy edits across later releases. Contributors should fingerprint a stable error substring rather than the entire stderr buffer from the process.

The three-ref grid assumes tags are buildable with the same toolchain as HEAD on the contributor laptop. Old releases may need different language versions, native extensions, or lockfiles that this sample script ignores. When a ref cannot run, the CSV should say unbuildable instead of fabricating a numeric exit. That honest CSV cell still helps maintainers decide whether the crash is actually new.

Model review cannot certify correctness of a patch or the absence of a regression in callers. A free server is the wrong place for secrets, unreleased customer data, or embargoed security issues. Prompts should contain only public issue text plus the contributor's own characterization oracle file. This workflow also adds latency that tiny documentation and typo fixes do not deserve.

Maintainers who already requested a specific unit test in the issue should follow that written request. This template is a fallback for sparse suites, not a replacement for a named test in the ticket. Extra ceremony on a one-character patch wastes reviewer attention and delays an otherwise obvious merge. Skip the three-ref grid when the issue already pins a file, a function, and a crashing input.

Who should not use this approach

Security patches that disclose an exploit path should not be developed as public characterization tests on a shared server. Contributors without a failing local repro should not ask a model to imagine the crash from the issue title. Teams that lack permission to push extra test files should not treat this article as license to expand scope. Those constraints matter more than any convenience that comes from free model access during review.

The method is aimed at functional bugs in libraries and CLIs where subprocess or unit tests can pin a crash. It is a poor fit for graphical timing issues, distributed race conditions, and hardware-dependent failures in drivers. Those classes need specialized harnesses that this sample workflow does not attempt to provide. Using an oracle template in those domains creates false confidence and noisy continuous integration.

Contributors who already maintain a strong, input-driven suite can skip the characterization rename step entirely. They can add a normal failing test on main and then implement the production change against that test. The oracle pattern exists for sparse suites that cannot yet name the bug in existing tests. Extra renaming on a mature suite is ceremony without new information for the reviewing maintainers.

A maintainer can replay a crash that lives in tests/ without trusting a chat log or a screenshot from the contributor. The original contributor still owes a production fix that flips the oracle for a documented reason. Optional model commentary on that oracle is useful only after the failure is executable in tests. Recorded failures outlast the issue thread that described them only in prose and screenshots.