ReasonKit v0.2: What I changed after my benchmark showed no quality gain

Reasoning systems tend to grow by accumulation.

You start with a useful system prompt.

Then you add another policy.
Another role.
Another specialist.
Another protocol.
Another fallback.

Eventually every task carries a large instruction bundle whether it needs it or not.

I ran into exactly that problem while building ReasonKit.

v0.1 gave me a result I didn't want

ReasonKit is a model-agnostic reasoning and agent orchestration prompt-pack.

The basic idea is to structure AI work around a small loop:

classify → evidence → bounded specialists when needed → execute → verify → stop

When I evaluated v0.1, I expected the orchestration layer to produce a measurable quality advantage.

It didn't.

Across the tested tasks, ReasonKit v0.1 did not show a measurable quality improvement while consistently adding context overhead.

The adaptive specialist layer also wasn't activated in the evaluated runs.

That was actually useful evidence.

Instead of adding more agents or more prompting, I froze v0.1 and changed the goal for v0.2.

The v0.2 question

The new question became:

Can the orchestration layer become smaller and more selective without reducing task quality?

So v0.2 focused mainly on two things:

• selective context loading
• observable specialist routing

The default path should be cheap.

A task should not need to load every protocol.

A task should not need a specialist simply because specialists exist.

And "no specialist required" should be a valid result.

Tiny kernel + lazy context

ReasonKit v0.2 starts with a compact kernel.

Instead of injecting the full reasoning bundle, the task is classified first and the host loads the relevant protocol.

For example, a debugging task can load the debugging protocol without loading architecture, research, design, or other unrelated modules.

The full bundle is not the default fallback.

Modules are also hash-verified and fail closed if the expected content does not match.

The intended flow remains:

classify
→ evidence
→ bounded specialists only when needed
→ execute
→ verify
→ stop

Specialists are optional

Another change was separating:

1. authorization to use a specialist
2. evidence that a specialist actually started

That distinction sounds small, but it matters for observability.

A system saying "a specialist may be useful" is not the same thing as actually launching another agent.

ReasonKit v0.2 records those separately.

And zero specialists is a first-class outcome.

What happened in the final held-out test?

I froze the v0.2 candidate before creating the final evaluation task.

Then I ran four conditions:

Every condition solved the task successfully.

So this benchmark does not demonstrate a ReasonKit quality advantage.

That part is important.

But the ReasonKit run did behave differently.

It classified the task as debugging at L2 complexity.

It loaded only the debugging protocol.

It did not load the full bundle.

It did not authorize or start a specialist.

And compared with the Luna + Reliable Engineering condition, it used about 8.1% less provider input context in that run.

That is one held-out task and one successful final run per condition.

It is not statistically significant and should not be generalized to every coding task.

Also, Astra vanilla used the least measured input and output overall.

Why I think the result is still interesting

The interesting result isn't:

"ReasonKit made the model smarter."

The benchmark doesn't support that.

The more useful result is:

ReasonKit was able to selectively route reasoning context without sacrificing the task result in this evaluation.

That is much closer to what I want the project to become.

Not an increasingly large agent framework.

A thin orchestration layer that knows when not to add more context or agents.

How to try it

ReasonKit v0.2 is currently a prompt-pack/orchestration surface, not an npm package, hosted service, or provider runtime.

Clone the frozen release:

bash
git clone --branch v0.2.0 --depth 1 https://github.com/sabahattink/reasonkit.git
cd reasonkit