Two small models ship in my Markdown editor. Three more didn't.

TL;DR — Marksage is a local-first Markdown editor whose assistant suggests fixes as you write. The AI in it can't write a word. Each model picks one answer from a fixed list, like which language a code block is in. It does that from a 385 KB file, in about 0.1 ms, inside your browser. Because every answer is right or wrong against a label, every model can be measured before it ships. That decided what's in the product: two models ship, and three were trained and left out. One of the three was better overall than the model in use. It still didn't ship, because it missed the one thing it was built to fix.

On October 30, I ran a new code-language model once against a test set it had never seen. It won. It got 391 suggestions right at 93.8% precision, against 380 at 93.1% for the model already in the app.

It didn't ship.

The bar was written down before the run. This model existed to fix the near-twin languages: TypeScript and JavaScript, C++ and C, C# and Java. The model in use mixed those up. On exactly those languages, the new one lost: 103 right at 88.0%, against 106 at 89.8%. Better on average, worse where it was aimed. A model that wins the scoreboard and misses its own goal isn't an upgrade.

That decision is the whole article in miniature. Marksage is a Markdown editor I built. It has no backend, no accounts and no analytics: your documents stay in your browser. Its assistant, Smart Assist, suggests fixes as you write. That means a language for an untagged code block, a list item for -item, a table for a pasted CSV. Tab accepts, Esc ignores, undo works. Behind some of those suggestions are small models trained in the repository. This is how they're built, how they're measured, and why most of what I trained isn't in the app.

A model that picks, never writes

Most AI features today generate text. Smart Assist never does. Every question it asks has its answers listed up front, and the engine has to pick from the list. The whole contract is three methods:

export interface DecisionEngine {
  readonly kind: 'heuristic' | 'model'
  ready(): Promise<void>
  /** Pick one of `options` (key → human description). `q` is a stable question id. */
  choice<T extends string>(state: string, q: string, options: Record<T, string>): Promise<Decision<T>>
  /** Place `state` on an ordered scale. `value` is the argmax index. */
  score(state: string, q: string, scale: string[]): Promise<{ value: number; probs: number[] }>
  /** Probability that `statement` is true about `state`. */
  noul(state: string, statement: string): Promise<number>
  dispose?(): void
}

choice picks an option. score places text on a scale. noul gives the probability that a statement is true. The engine decides, and my own code makes the edit. The rule that follows: if you can't list the possible answers up front, it's not a Smart Assist job.

This kind of model is called a decision model. The term reached a wide audience in September 2026, when TypeSafe AI released Jev, a hosted model built to answer typed questions instead of writing text. Three days later, Convai Innovations released Laya, an open-weight model that answers the same three question types and runs on your own machine. Laya was my first plan. But its smallest checkpoint has 322 million parameters. The model I trained for "which language is this code?" has 262,333 numbers in all. That's over a thousand times smaller, in a 385 KB file.

A closed answer space buys three things:

• Nothing to parse. The answer never comes back in an unexpected shape, so there is no output to validate. Checking generative output at runtime is a real problem, and a separate article.
• It's cheap. A decision is a few thousand multiplications: about 0.1 ms in plain JavaScript. No ML runtime, no GPU, no worker.
• It's measurable. The answer is right or wrong against a label. Generative output has many acceptable answers, which makes it much harder to score.

The third one shaped the whole project.

Rules first

Every question goes through the same order, and stops at the first step that's good enough:

1. Heuristic rules. Patterns written by hand that the code checks directly. A line that starts with def and ends with : points to Python. They're instant, every decision can be explained, and they're rarely wrong.
2. A small model trained for that question. Only where the rules leave too much unanswered, and only where real labeled examples exist.
3. A general decision model. For questions with no training data at all. No feature needs one yet, and Laya hasn't been tried in a browser.

For code-block languages, step 1 wasn't enough. The rules are precise, but they stay quiet on about three blocks in four. Writing a rule for every pattern of 20 languages never ends. That gap is what the first model was trained to fill.

Rules and models share one confidence gate. A suggestion shows only if the top answer is at least 80% and at least 20 points ahead of the second:

export function passesGate(probs: number[], threshold = 0.8, minMargin = 0.2): boolean {
  const sorted = [...probs].sort((a, b) => b - a)
  const top = sorted[0] ?? 0
  const second = sorted[1] ?? 0
  return top >= threshold && top - second >= minMargin
}

A missing suggestion costs nothing. A wrong one costs trust. So the number I care about most is precision when shown: of the suggestions you actually see, how many are right.

385 KB, trained in the repo

The labels were already there. Documentation is full of fenced code blocks whose authors typed the language after the opening backticks. Those tags are free labels. For the current model, about 270,000 blocks were collected from 309 public repositories. Training used 63,751 of them, at most 6,000 per language, so the common languages don't drown out the rest.

The split is by repository, never by block. Blocks from one project share names and style, so a model could pass a test by remembering the project. Whole repositories go to training, to dev (where choices are made) or to test. A leak guard also drops any training block that matches a test block.

The model is fastText-style, and small on purpose:

1. The snippet becomes a set of clues: tokens, pairs of neighboring tokens, how lines start and end, the indentation, three-letter pieces of longer words.
2. Each clue is hashed into one of 32,768 slots. There's no dictionary, and an unseen word still lands somewhere.
3. Each slot holds 8 learned numbers. The snippet's clues are averaged into one list of 8.
4. One linear layer scores the 21 answers (20 languages and "not code"), and softmax turns the scores into probabilities.

The numbers are stored as 8-bit integers with one scale per slot: 394,280 bytes. Training runs 8 epochs in plain TypeScript, in about 15 seconds. The same code builds the clues in training and in the browser, so the model sees in the app exactly what it saw in training.

Raw confidences from a model like this are too high: when it says 99%, it's right much less often than that. So version 2 fits its temperature on repositories it never trained on. Five extra models each hold out a fifth of the training repositories, which gives about 7,600 out-of-repository predictions. The temperature goes up until 95% of the suggestions it would show are right. On the dev set, the effect looks like this:

Fewer suggestions, and more of them right.

A second model of the same size reads document sections. The Outline gives each section a type (setup, reference, security…) from its heading. When the heading says nothing, like "Before you begin", the model reads the text under it. Its labels come from unambiguous headings, and the heading is then hidden from it. On 134 repositories it never saw, it types 13.5% of sections confidently and agrees with their heading 88.9% of the time.

Bars written before the run

The process is the same for every model. Choices are made on the dev set. The held-out set runs once, at the end, against a bar written down before the run. Once a held-out set has been seen, it's used up: the next version needs a fresh one. The code model has been judged on three of them.

The results are mixed, and I'd rather show that than round it off. The first model's bar was the rules' precision, with far more suggestions. It met part of it: 133 right suggestions against the rules' 73 (1.8×), but about 94% precise against the rules' 97%, and just short of the coverage target. So it shipped as an opt-in mode, not the default. Version 2 was still about a point under the rules on its fresh set: 91.8% against 92.9%.

Here is how the opt-in mode, Rules + AI, compares with today's rules on the three sets:

That's 1.7–2.2× the right suggestions, at about the rules' precision: a little below on two sets, a little above on the third. Real, and modest. The mode stays opt-in, and it downloads the models only after a notice.

What didn't ship

Three more models went through the same process and came out the other way. They're documented next to the ones that shipped, with the numbers.

Structure hints. The question: is this near-miss line, like -item or **Title**, really a list item or a heading? The model learned from 66,076 lines. It was judged on 227 hand-labeled lines from the issues and pull requests of 43 repositories it never saw.

It met the bar by one line in 227. That's not worth a 385 KB download.

Paste types. The question: what kind of text is this paste (CSV, a list, JSON, code…)? It trained on 16,977 pastes, and was judged on 53 real pastes from 19 sources outside its data. The rules showed 41 answers at 97.6%. The model showed 44 at 90.9%. It took kubectl output for code.

Code model v3. This is the one from the top. Mean pooling averages a snippet's clues, so a few decisive ones can drown: : number pushes hard toward TypeScript, but it's 5 clues out of 70. Version 3 kept each dimension's strongest clue next to the average (mean + max pooling). It was better overall, and worse on the near-twins. On TypeScript and JavaScript alone, it got 22 right and 5 wrong, against v2's 33 right and 4 wrong.

The cause was in the labels, not the model. Only 42% of the code tagged TypeScript in training has any TypeScript-only syntax. The rest is valid JavaScript, and its tag says which project it came from. No way of pooling clues can tell those apart.

The section model also has a type it never learned: "explanation". Two training rounds got 0 right on the dev set, because explanatory prose uses the same words as any topic. That type is read from headings only.

Not shipping these was my call, and so was the rule that came out of them. A model is used only where, on a fresh held-out set, it gets clearly more answers right than the rules at about the same precision. The three modes say exactly that: Off, Rules and Rules + AI.

The v3 run found one more problem. Rebuilding the training data showed that v2 couldn't be reproduced exactly from its recorded sources: 77% dev accuracy against the shipped file's 81.8%. So the shipped file became the fixed baseline that every new model is compared with. Every code model shipped from now on carries a fingerprint of the data it learned from.

Marksage reads its own docs

If Marksage can tell that a README has a vague link or a 90-word sentence, its own README should pass. So npm run check:docs runs every Smart Assist and Document-tab feature on the README and the docs pages, in English and Portuguese, in both modes. The rule is in the script's header:

/**
 * Two kinds of document:
 * - The ones we ship to be read (README, the docs pages, EN + PT-BR) must be clean: no suggestion and no "Worth a
 *   look" flag, unless it's listed in ALLOWED with the reason Pablo approved. Fix the document, never the rules:
 *   a flag that's wrong is a rules finding (ROADMAP), not something to tune against these files.
 * - The sample (Pitwall, lib/sampleDocument.ts) is the inverse: every slip left in on purpose must trigger its
 *   feature, in EN and PT-BR, nothing else may show — and once every suggestion is accepted, only the two slips
 *   that have no automatic fix (the vague link and the long sentence) may remain.
 */

The first run found no suggestions and 40 long-sentence flags. I fixed the documents. The run also found real bugs in the rules, which went to the roadmap instead of being tuned away:

• Portuguese never read as personal. The tone check looked for "você" with \b in a JavaScript regex. Without the u flag, \b only knows ASCII letters, so there's no word boundary after "ê". Every Portuguese text came out "impersonal".
• A fix that broke the document. Accepting every suggestion in the sample turned **Options** under ## Configuration into ## Options. That left Configuration an empty section. Now the new heading goes one level under the heading above it.
• Short sentences glued together. The sentence splitter only splits before a capital letter, a digit, a quote or a parenthesis. A sentence that starts with inline code stays glued to the one before it.

The sample is a made-up README for a lap-time CLI, with slips planted on purpose. It's the inverse test: every slip must fire, in both languages, and nothing else may show.

All of it runs in CI on every push: type-check, lint, unit tests with a 50% line-coverage floor, the build, a first-load size budget, the docs check and a version check. The Smart Assist evaluation prints 1,215 "✗" lines today, and CI fails if that count moves either way. A change isn't wrong by itself, but it has to be noticed and explained in the commit that makes it. The end-to-end tests run in Chromium, WebKit (desktop and iPhone) and Firefox.

What it doesn't do yet

The repository keeps its own list of gaps. These are the ones that matter if you try it:

• The section model reads Portuguese poorly. In the sample, the privacy section is typed Security at 0.75 in English, and "other" at 0.30 in Portuguese. It learned mostly from English docs.
• One-liners get overconfident answers. x = 1 comes out as Python at 97%, though it's also valid Ruby and JavaScript. A "too short to judge" rule is the obvious next step.
• The held-out sets are used up. Any model change now needs fresh test sets first.
• No test runs Rules + AI in a browser. The evaluation checks both models' answers in Node. Nothing yet downloads them in a page and uses them while typing.
• Editor basics are missing. The editor has no syntax coloring, Tab doesn't indent lists, and Export downloads a copy instead of saving back to the file you opened.
• No real users yet. With no analytics by design, the only feedback loop is people telling me. This is the first round.

What's next

I built Marksage with AI assistance (Claude) for implementation and review. The architecture and the measurement bars are mine.

Next on the list: TypeScript/JavaScript labels that say what the code is, not which project it came from. After that, connecting other AI models with your own key: an agent, with the model and the key you choose, to help with anything inside Marksage. It will be opt-in and off by default. The built-in features will still never use a generative model.

You can try Marksage on a sample document, read the story with every model's numbers, or read the code, the training pipeline and the model files (AGPL-3.0).

Every number here comes from the repository's docs and model cards. If you've shipped a small model next to rules, I'd like to know where yours drew the line.