I Pre-Registered a Hypothesis. 600 API Calls Later, the Data Killed It.

A stranger on DEV.to said "run this experiment." I ran it at n=600. Here's what happened — including the part where he caught me reporting post-hoc findings as if they were planned.

The Backstory

Mike Czerwinski read my article about AI agents following rules. He proposed a specific experiment: test prose-format rules under a mechanical gate. Can you get both 100% compliance AND deep reasoning by writing rules as narrative instead of commands?

I had pilot data suggesting yes. Mike pointed out the pilot ceiling was probably noise. He was right. I designed a full 2×2 factorial experiment and ran it at n=600.

But I did something else first. Something I'd never done before.

The Pre-Registration (Solo-Researcher Edition)

I don't have an OSF registry. I don't have an advisor. What I have is a Git repository and a Python script.

Before running a single API call, I wrote the hypothesis into the experiment script header:

"""Pre-registered hypothesis: Format effect on reasoning depth
is LARGER under GateGuard-OFF."""

The script defined all 5 rules, all 4 conditions, the deterministic regex scoring, and the analysis plan — committed BEFORE execution. git log shows the timestamp. That's my pre-registration: a timestamped, immutable snapshot of what I predicted before I saw the data.

It's not peer-reviewed. It's not a third-party registry. But it's honest, and it creates a paper trail that can't be rewritten after the fact.

The Experiment

Design: 2×2 factorial — Format (code / prose) × Gate (ON / OFF)
Rules: 5 governance rules (delivery gate, health check, self-review, fact-check, self-model regeneration), each in both code and prose format
Trials: 30 per condition per rule = 600 API calls
Model: DeepSeek V4 Pro, temperature=0
Scoring: Deterministic regex — no LLM judge

The Pre-Registered Hypothesis: KILLED

My pre-registered prediction was that format would matter MORE when the gate was off — that code format and mechanical enforcement overlap, so removing the gate would reveal format's true effect.

The data said no:

• Gate ON: d(code−prose) = −0.277
• Gate OFF: d(code−prose) = −0.250

The format effect on reasoning is nearly identical regardless of gate status. The hypothesis was wrong.

This is the point of pre-registration. If I hadn't written down the prediction beforehand, I could have looked at these numbers and said "I predicted this." Post-hoc rationalization is cheap. A timestamped Git commit isn't.

What the Data Said Instead (Three Things)

1. The ceiling was noise. My pilot found code_OFF reasoning = 4.42. At n=30, it's 2.67 — below ALL gate conditions. Mike's skepticism was correct. Small-n pilot ceilings are not findings.

2. Gates improve reasoning, not suppress it. The gate added +0.32 reasoning in prose and +0.15 in code. Everyone's intuition is that enforcement constrains thinking. The data shows the opposite: mechanical structure improves reasoning depth in both formats. The gate acts as cognitive scaffolding, not a straitjacket.

3. Prose + Gate is the best configuration for reasoning depth. Prose format consistently outperforms code format on reasoning (~0.25 SD advantage), regardless of gate status. Combined with the gate's structural boost, prose+gate produces the deepest reasoning (3.23 vs 2.82 for code+gate). Cohen's d = 0.605.

The practical takeaway: if you care about compliance, use code format + gate (99.3%). If you care about reasoning depth, use prose format + gate (3.23/5).

The Part Where I Got Caught

I also reported a per-rule breakdown: prose helps meta-cognitive rules (self_review: +1.08 over code) but hurts precision-dependent rules (fact_check: −0.21). I suggested "hybrid deployment" — prose for some rules, code for others.

Mike replied with a methodological question: "Was the per-rule breakdown pre-registered?"

It wasn't.

The data was always going to be collected — rule_id is in every trial, all 5 rules were defined before execution. But the specific pattern I reported was discovered after seeing the results, not predicted beforehand. Pre-registered per-rule predictions would be strong evidence. Post-hoc pattern finding is exactly the kind of result that regresses on the next run.

I updated the README to flag this. The hybrid deployment recommendation now rests on the pre-registered overall effect (d=0.605), with per-rule heterogeneity marked as exploratory. Mike also pointed out that the 8% regex detection gap in prose+gate cases means the fact_check measurement might be an artifact, not a real decline.

This exchange — someone catching a methodological gap in your work, and you fixing it publicly — is what peer review is supposed to be. DEV.to comments aren't peer review. But they're also not nothing.

What Pre-Registration Buys (Even Without an Advisor)

I'm an undergraduate with one laptop and no lab. Pre-registration for me looks different than for a funded research group. But the core mechanism is the same:

1. Write down what you predict. Script header, design doc, Git commit message — any timestamped, immutable record.
2. Define your scoring before you see data. My regex patterns were committed before execution. No tuning after seeing results.
3. Report the null result. The pre-registered hypothesis was wrong. That makes the finding MORE credible, not less — because I can't have p-hacked my way to NOT_CONFIRMED.
4. Separate pre-registered from exploratory. When someone asks "was that planned?", have an honest answer.

The point isn't to predict correctly. It's to make "wrong" useful. A null result with a timestamped prediction is evidence. A null result with post-hoc explanation is a story.

Honest Limitations

Single model (DeepSeek V4 Pro). Single rater (deterministic regex — consistent but limited; the 8% detection gap means some effects are measurement artifacts). No holdout sample. Per-rule breakdown is exploratory. Pre-registration was via Git commit, not a public registry — I could theoretically amend the commit (though the GitHub timestamp trail would show it).

I'm working on a pre-registered per-rule replication with second-rater scoring for the fact_check rule specifically. If you have suggestions for a lightweight pre-registration workflow for solo researchers, I'm listening.

👋 林宇浩 — Building verification infrastructure for AI agents. One laptop, 50+ sessions, 1,200+ API calls. github.com/YuhaoLin2005/hermes-workspace