Design full test-space scenarios at once and delegate precised evidence-based exploratory testing to your AI

You Can’t List Your Unknown Bugs, but You Can Model What Varies.

Combinatorics-Framework-JVM turns a model of settings, action histories and optional events into planned test cases, and keeps the experiment inspectable from design through evidence.

⚡️ Straight to the point for most impatient/busy-reader who respects human's work (let your AI read 'AI_DELEGATION.md' after dowloading the release, and it will answer you live): Combinatorics-Framework-JVM

“What kinds of bugs does my system under test have?”

It sounds like the obvious first question for any testing tool. It’s also the wrong one. If you could answer it, you’d already have the tests.

The defects that hurt are the ones nobody knew to name: a guest, a coupon and a repeated payment request, each fine alone and wrong together. AI coding assistants sharpen the problem. An assistant can implement the feature, generate the unit tests and explain why the code should work. It can’t tell you which interactions those tests exercised.

So don’t start from bugs. Start from three questions you can answer:

• What varies? Settings, inputs, the order of actions, events that may or may not happen.
• What must stay true? An invariant, such as “the same payment request creates at most one charge.”
• How much am I willing to run? A bounded budget of cases you can actually inspect.

Bugs aren’t the input to that exercise. They’re the output.

Investigation, not diagnosis

Combinatorics-Framework-JVM is built for exactly that kind of investigational testing: a Java workbench with a desktop app (guif), a command line and an embedding API. You describe the variations, it generates test candidates, runs them through your configured runners and helps you examine the results. State a model, run a bounded experiment, read the evidence, refine the model, repeat.

It won’t discover unknown defects for you. It turns a test idea into a visible model, a bounded run and results another engineer can inspect.

Model the context, and the history

Take a small checkout:

That’s 24 static combinations. Run them all. Now add history: submit, time out, retry, a price change at different moments. The input values barely changed. The history did, and no parameter table shows it.

So the Framework models three kinds of variation separately:

• Static context: account type, payment method, feature flags. Covering selection can reduce this part to a declared interaction strength.
• Action sequences: what happens first, next and again, including order and repetition.
• Sudden or optional events: a timeout, a duplicate request, a price change, including when it strikes and the case where it never does.

Sequences and sudden events are then crossed with every selected context, not squeezed into the same matrix. That stops you from flattening everything into interchangeable parameters and assuming you tested the histories that matter. For a workflow bug, the timeline deserves more attention than a big parameter matrix.

Constraints need the same care. An impossible combination can be excluded. A valid input the application should reject (cash on delivery over the API, say) deserves a negative test, not a deletion. The modeler decides which question a test answers. The generator can’t.

A model you can read in one minute

Two modes, three actions. Every candidate gets one mode and two different actions in a chosen order (AB, BA, AC, CA, BC, CB): 2 × 6 = 12 candidates. Save this as first-scenario.json:

{
  "spec_version": "1",
  "title": "Two modes and ordered pairs of actions",
  "slots": [
    {
      "sheet": "MODE",
      "alias": "choose_one",
      "raw": true,
      "values": ["safe", "fast"]
    },
    {
      "sheet": "OPS",
      "alias": "permute(2)",
      "flags": ["FW_Concatenator="],
      "raw": true,
      "values": ["A", "B", "C"]
    }
  ]
}

And these settings as first-scenario.properties:

cf.coverage.strength=0
cf.coverage.budget=0
cf.plan.autoPromoteSingleVerb=true

Then, with JDK 25 and fw-app.jar in your working folder:

java -Djava.awt.headless=true -jar fw-app.jar validate --scenario first-scenario.json --config first-scenario.properties
java -Djava.awt.headless=true -jar fw-app.jar plan --scenario first-scenario.json --config first-scenario.properties
java -Djava.awt.headless=true -jar fw-app.jar candidates --scenario first-scenario.json --config first-scenario.properties --limit 12

Validate, inspect the plan, read the candidates, all before anything runs. These are plain-text candidates for learning the model, not programs to execute.

Prefer spreadsheets? The classic workbook style (data sheets plus an FW_Seq control sheet) expresses the same model, and the wizard in guif guides a first scenario.

When order is the bug

A basic retry test submits a request twice with the same key and confirms that one order exists. Now change the history: the provider accepts the first request, its response times out, a confirmation arrives while the client is retrying, and the client tries again. Is there still exactly one charge and one order?

A small timeline model might include these histories:

The invariant: reusing a request key must never create a second charge or a second order. Check durable business state (charges, orders, how they relate), not just the final HTTP status.

One honest boundary: the Framework generates ordered choices, permutations, repetitions and optional events. It doesn’t control a thread scheduler or a remote payment provider. Your adapter has to make those events controllable: hold and release a callback, simulate a timeout, reset state. Once it does, you vary the histories without hand-copying a test method for every combination.

A planted defect, caught

(quick self-test)
You don’t have to take this on faith. The project’s manual ships a self-contained checkout demo whose stand-in system has a deliberately planted lost-order defect. validate, plan and candidates report 72 generated cases, 24 removed by one constraint and 48 executable. The full run ends with 44 PASS and 4 DOMAIN_FAIL, with no harness failures. The repo’s validation report records this on JDK 25 and a 0.1.0-SNAPSHOT build.

A larger showcase in the manual runs 4,128 generated Cucumber scenarios against an in-JVM checkout engine and 714 browser scenarios (Selenium, headless Firefox) against a dummy web shop, both with planted defects hiding in action order and event timing. A second run of each design gave the same verdict for every case.

Two things to notice:

• A run that finds defects still exits with code 0. A failure is a result, not a crash, so read the verdicts. Keep product failures (DOMAIN_FAIL) apart from harness outcomes such as BROKEN, TIMEOUT and INFRA_FAIL.
• Every defect here was planted. That shows the harness can see a fault, not that it finds ones nobody knows about. It’s why the manual says to calibrate the oracle with a known-failing case before trusting a model. The unknown bugs are what the experiment is for.

What comes back

Failures arrive as cases, not red lights: the candidate, its verdict, its measurements and the context to reproduce it. Checkpoints and exports keep the evidence, and triage groups failures into findings with representative witnesses. The documented “minimal witness” is the failing case with the least source in its group: a good place to start, not a proven mathematical minimum. Values associated with elevated failure rates are leads, not causes.

When nothing fails, you still get something: “within this model, at this strength, no violation observed.” That isn’t proof of absence. It’s a claim another engineer can inspect, challenge and extend.

What you bring, and what it doesn’t promise

The Framework generates and organizes. You make it real. The model and the invariant are yours: it doesn’t discover what must stay true (a far easier question than “what is broken?”, but not a free one). You also bring:

• An adapter that connects candidates to your system and controls its state and dependencies.
• An oracle: assertions that encode the invariant. A generated case without a meaningful check is just activity.
• A controlled environment with a reset strategy, plus the judgment about which failures matter and what went untested.

And be honest about the limits:

• Pairwise isn’t magic. It covers pairs of modeled values, not every three-way defect, action order or timing failure, and the selection is greedy, so it never claims a minimum suite. With fewer than three varying factors, every pair is already the whole space and nothing is reduced. Its value is that you can state what you selected and inspect it.
• One verdict per case. The first failing check names the verdict and hides later ones, so report every check as a metric too.
• No OS sandbox for Java. Java cases run inside the Framework’s own JVM with your permissions, so treat every run as code execution on your machine: use a dedicated workspace and keep production out until the adapter, assertions and side effects are clear. Python cases run as separate processes and can be sandboxed with bwrap or Docker.

The same machinery, for AI experiments

Prompts × retrieval strategies × thresholds is a combinatorial space too. Three retrieval strategies, two prompts and two thresholds make 12 configurations. Evaluate each on 50 questions, twice: 1,200 evaluations, not 12. Mind that arithmetic when a provider bills per request.

Multi-goal analysis, including a Pareto front, keeps cost and quality trade-offs visible instead of collapsing them into one early score. It describes what you measured, not the best system that could exist. Bring an evaluation method you trust, because there is no universal judge of LLM answers, and test the evaluator with a deliberately wrong answer. Generation is deterministic; a remote model isn’t, so record versions, settings and rate limits.

Hand your assistant an experiment, not a vibe

The model and the recorded evidence are files, so a second engineer, or a later assistant session, can pick them up without reconstructing a chat. Give your assistant evidence, not a symptom:

Here is the failing candidate (submit → timeout → retry → confirmation) and the two charge IDs it returned. The invariant is one request key, at most one charge. Propose a mechanism that fits this evidence, a focused change, and which existing cases would distinguish your explanation from the alternatives.

That asks for a testable claim instead of a guessed fix. Then rerun the same candidate and its surrounding histories against the change: if a repair fixes one sequence and breaks another, the regression stays visible.

The repo is built for this hand-off. The manual, AI_DELEGATION.md, is written for an AI operator acting for a human supervisor, and the desktop’s AI_delegation button copies its path for you to paste. Copying a path isn’t consent. The manual’s first rule is no agreement, no operation: the assistant needs your explicit yes, then must ask before it finalizes the design, executes any case (a smoke run comes first), builds or installs anything, or touches the network, files outside the agreed workspace or your system under test. It must report what it did and didn’t check, and never present a plausible explanation as an observed result. You stay the director, and the license makes that a condition.

Try it small

• Pick a behavior you already understand, ideally a defect you’ve already fixed.
• Write three or four factors, one troublesome sequence and one invariant.
• Stay small enough to preview every candidate, and change one thing at a time.
• Inject a fault into a local implementation. Does the scenario catch it? Fix it and rerun.
• Save the scenario, run ID, checkpoint and evidence.

If you can express the behavior, understand the generated cases and recognize a useful failure, it fits. The manual itself rates a few situations as poor fits: one or two parameters, purely continuous spaces, no reliable oracle, and state you can’t reset. Use unit or property tests, fuzzing or an optimizer there, and keep the plain unit test whenever it already pins a defect down. Reach for a scenario model when the variations themselves are getting hard to track.

Requirements, status and license

• First of all: AS IS..., no warranty
• Requirements: JDK 25, a full JDK with the compiler rather than a bare JRE, because generated cases are compiled in memory. The core workflow needs no Python, no relational database and no containers. Python 3.11+ and the Python add-on are optional, for Python candidates.
• Status: first public release (Oct 2026), one maintainer. It’s early-stage, so validate it on your own workload before it carries anything critical.

The case for it is specific. Interacting settings, action sequences and timed events get one shared model, the model becomes inspectable candidates, and the results stay available for review and retesting. For the bugs that live between individually working features, and for AI-assisted work that needs evidence rather than confidence, that’s worth a small trial.

Don’t ask it where your bugs are. Tell it what varies and what must stay true, then read what comes back.

P.S. You are a Scenario Test-space Architect/Designer (puzzle-items of any kind) and a supervisor of AI (who operates on "Combinatorics Framework" as a precise tool for all of these)

🧩🧩🧩🧩🧩

GitHub project and downloads