Rules in markdown vs rules in CI: what I measured in my own repositories

Disclosure: I'm the author of rebar, the open-source tool this post is about. It is alpha, and I'm its only contributor so far.

I keep a written standard for the sites I build with AI coding agents. The complaint that started this project was mine: a lot of it got ignored. Values were hardcoded, parts of the stack were forgotten, the AI was listed as a co-author, the component library was skipped.

The obvious fix is to write the rules more clearly. Before doing that, I measured.

What I measured

On 2026-08-25 I went through 161 commits across six of my own repositories:

Six repositories, all mine, is a small sample. I'm not claiming a general result. The last row is what changed my approach. That repository had an AGENTS.md with a "Hard rules" section, plus SECURITY.md, GOVERNANCE.md, CONTRIBUTING.md, and a check script that chained format, lint, types, tests and build. Nothing ever ran that check.

Every rule was written down. None of them was enforced. The short version, as the README puts it: a rule written in markdown has close to zero compliance, and a rule in CI has 100%. The 100% is not a discovery. A required check that fails blocks the merge by construction. The useful part is the consequence: if a rule can move down a level, from prose into something that blocks, it should.

That needs one condition, and most of what I built came from learning it the hard way: the enforcement has to be more reliable than the rule it replaces.

Five times the enforcement was the problem

1. A score that counted nothing as a pass. The first version of the checker treated "this rule does not apply here" as a pass. An empty folder with an empty .git/ scored 8 of 14 and tied with the tool's own repository. Now every rule ends in one of three states: passed, failed, or not applicable, with the reason printed. Not applicable leaves the denominator.

rebar-check · prumo
  ✓ editorconfig       has .editorconfig
  ✗ dependabot         automated dependency updates  no dependabot, no renovate
  ✓ ci                 has CI
  ✓ ci-gates           CI reaches the verification the repository declares
  – typecheck          has a typecheck script  no TypeScript here
  11 of 13  ·  1 not applicable

(An excerpt of the sample in the README, measured on 2026-08-30 against another project of mine; only five of its rules are shown.)

2. Tests that could not tell passed from not applicable. Every rule has pass and fail cases. Each case builds a small repository in a temp directory with its own git init. At first the cases only read the exit code, but "passed" and "not applicable" both exit 0, so 13 of the 20 rules at the time could not be proved that way. When I applied 70 mutations to the checker, 30 survived with the suite fully green. The cases now declare the expected state and read it from --json.

3. A heuristic with zero true positives. A rule for literal colour values, measured on a real repository, gave 7 hits and zero true positives. Five of the seven were comments documenting the rule itself. So heuristics only warn. A heuristic that blocks teaches people to switch the whole output off.

4. A green gate over a broken product. During a rename, the generator started reading a workflow file from a folder where it had a different name. rebar new crashed with ENOENT, and npm run verify stayed green for six commits, because no step actually generated a project. There is now a step that does, and it is itself checked by mutation: putting the original defect back makes two of its five tests fail.

5. My own README. It got numbers wrong six times ("20 checks" when there were 19, "8 steps" when there were 12). The fix was not more care. Every number that belongs to the tree is now derived from the source and written between invisible markers, and a gate step fails if the document drifts.

The only layer an agent cannot edit

Hooks live in .git/hooks or a core.hooksPath setting. Workflows live in .github/workflows. Both are files, and an agent with write access can delete or bypass them. One attack I reproduced against my own gate: git update-index --skip-worktree on the config file, then rewrite it on disk. git status and git diff both came back empty, and the gate printed APPROVED. The gate now reads git ls-files -v, the one command that shows that flag.

The layer that holds is on the server. rebar's main has a GitHub ruleset with a required check and bypass_actors: [], so not even the owner goes around it. I tested it with a planted pull request that deleted .editorconfig. Both CI jobs failed and the merge was blocked. A direct push to main is refused too.

Agents read what reviewers don't see

The second ruler, rebar-security, looks at versioned files for known prompt-injection signatures. It checks invisible Unicode and terminal control bytes, agent settings that run commands or widen approval, and MCP server launches, each fingerprinted so an allowlisted command cannot quietly gain an environment variable. It also flags agent CLIs started with approval turned off, and AI steps in GitHub workflows that text from an outside account can reach. A finding names a position, never the payload, because the output is read by agents too.

A pass means "no known signature", never "free of injection". The signature list is public, and the attacker moves second.

Telling the agent before it writes

Projects generated by rebar new point their agent at an MCP server that serves rebar's rules, the gate steps, and the measured reason behind each rule. The rules artifact is generated from the rulers' source, and the gate regenerates it in memory and fails if the file on disk differs. Changing a rule and forgetting the MCP is not possible without a red build.

What it does not do

• It does not check Broken Access Control, and it does not run your application.
• Some rules reflect my own stack: a hardcoded Brazilian phone number check, shadcn's components.json, an allowlist for co-author trailers. On your repository they may be not applicable, or not useful.
• The generator has one preset, a static Next.js site.
• The MCP server needs a checkout and npm ci --prefix mcp. npx alone does not start it.
• It is alpha. As of the project's state file it gates one repository for real: its own.

Try it, and tell me where it is wrong

It needs Node 22 or newer, has zero runtime dependencies, and never writes to the repository it audits:

npx github:Navesz/rebar .                       # format and gate rules
npx -p github:Navesz/rebar rebar-security .     # security and prompt-injection signatures

The most useful contribution right now is a false positive. Run it on a repository of yours. When a verdict is wrong, open an issue with the exact output and why it is wrong. One reported false positive is worth more than a new rule.