Making a Small Decision Model: Part 1, The Basics

You run a service calling an LLM hundreds of thousands of times a day, but not a single call is actually creating anything. You’re just trying to get a decision out of it: is this email spam or not?

In all of those cases, what you’re actually trying to figure out is a closed decision: "Is this email phishing?", "Which queue owns this bug?", or "Should we halt this agent run?" Yet you pay full generation prices: chunky prompts, strict JSON schemas, 30-step token decoding loops, fragile regexes, and zero honest signal about whether the model was confident or just guessing.

That is the exact problem Jev (TypeSafe AI's proprietary decision model) was built to solve. Gev is our attempt at building one using Kev as inspiration, porting that recipe to Google's T5Gemma 2 270M backbone. In neural networks, parameters are the adjustable weights the model learns during training. While mainstream chat models pack 8 billion to 70+ billion parameters and need big GPU clusters, Gev has just 270 million (roughly 30 to 100 times smaller), so it runs fast on a single GPU.

Throughout this work I ran into a few tricky hurdles, but we eventually took Gev to score 11.13 on the Jev Decision Index.

So why use a decision model?

When a standard generative model answers "Is this ticket urgent?", it runs an autoregressive decode loop: generating tokens one by one across sequential steps, feeding each word back into the model:

{"reasoning": "Customer mentions data loss.", "urgent": true}

You run thirty sequential rounds just to extract a single bit of boolean information. That hidden tax quietly eats your day.

So in this case how is a decision model like Jev or Gev (in our case) different?

Borrowing from Daniel Kahneman's Thinking, Fast and Slow, standard chat models act like "System 2": slow, step-by-step deliberation, writing out dozens of words even when answering a simple yes-or-no flag.

A decision model acts like "System 1": fast, automatic pattern recognition, like hitting the brakes when a pedestrian steps into the street. A generative model is like asking someone to write a full incident memo just to confirm a server is down. A decision model is like handing them a clipboard with pre-printed checkboxes to tick in one second.

A decision model skips text generation entirely. It runs in a single forward pass (or prefill pass). In machine learning, the prefill pass is where the model ingests the entire prompt at once, computing representations for all input tokens simultaneously in parallel. Instead of decoding text, it points directly at the best option from your list ("billing", "database", or "frontend") and returns a confidence percentage for each choice in 45 ms on a single GPU.

At a high level, Gev maps every question into three typed primitives:

• noul: Binary yes/no decisions. Outputs a calibrated probability P(yes) between 0% and 100%. A probability distribution is a complete set of percentage scores across all possible outcomes summing to 100%. Saying a model is calibrated means its confidence numbers match real-world frequency: when Gev reports 90% confidence across 100 tickets, it gets roughly 90 of them right. In production, this allows you to write clean threshold rules (if decision.confidence > 0.90: auto_route() else: flag_human_review()).
• choice: Selection over K categorical options. Accepts optional natural-language descriptions and outputs a complete probability distribution across all K choices summing to 100%.
• score: Selection over an ordered scale. Evaluates ratings like 1 to 5 stars or low/medium/critical severity, outputting the distribution across levels plus the weighted expected value.

Every question collapses to the exact same prompt structure:

<q> Is this ticket a billing issue? no <opt> yes <opt> <decide>

Here is what those markers do:

• <q>: Unscored start label. Tells the model "the question begins here".
• <opt>: Option boundary marker. Sits at the end of each candidate option.
• <decide>: Terminal scoring marker. Sits at the end of the entire prompt after reading the question and all candidate choices.

As the model reads the prompt, it turns each word into a compact list of numbers called a hidden state, which acts as a numerical summary of what that token means in context.

At the top of the network sits a lightweight comparison layer called a bilinear pointer head. We call it a "pointer" head because it points directly at one of your existing options instead of inventing new words, and "bilinear" because it scores how well two summaries match up: specifically, comparing the final question summary at <decide> against each option's summary at <opt>:

score_i = match(decide_summary, option_i_summary)

Those raw match scores are called logits (unnormalized numbers where a higher score means a stronger fit). Finally, Gev runs those scores through a standard softmax step, which turns raw numbers into clean percentages that add up to 100% in a single pass.

Why does an encoder-decoder backbone fit multi-question workflows?

Most open LLMs are decoder-only, where tokens only look backward in one continuous string. Asking three independent questions about one document forces you to pack everything into one string, build custom attention masks, or re-run the document three separate times.

T5Gemma 2 is an encoder-decoder, giving us clean question isolation for free.

Wait, what are encoders, decoders, and cross-attention?

Before diving into the multi-question benefits, it helps to demystify the alphabet soup of transformers. At a high level, this is how they each work:

• The Encoder reads context. Think of the encoder as a reader reading a whole document in one sitting before taking any notes. It processes all input tokens simultaneously in parallel using bidirectional attention: every word looks both backward and forward at every other word to build a rich, interconnected map of the source text.
• The Decoder produces outputs. Think of the decoder as a writer filling out an answer sheet line by line. In standard autoregressive models, it generates tokens sequentially while looking backward at what it just wrote.
• Cross-attention connects the two. Cross-attention is straight-up the bridge between reader and writer. While the decoder evaluates candidate choices, its attention layers query the encoder's completed context map, pulling out relevant facts from the source document without altering the underlying document representation.

Splitting the workload this way provides three major engineering benefits in production:

• Clean question isolation. If a support ticket has three questions attached (such as urgency, sentiment, and queue routing), all three run as parallel rows in the decoder batch. They physically cannot leak into each other because they live in separate batch rows.
• Caching efficiency. You encode the heavy background document once, cache its output hidden states in memory, and evaluate multiple independent questions against it with tiny, fast decoder passes.
• Vocabulary memory savings. Standard chat models keep a huge dictionary table at the very end of the network so they can pick the next word out of 260,000+ possibilities. Because Gev only points at your options and never generates words, we drop that dictionary table completely, saving about 28% of GPU memory.

So, where to next?

Looking back at the architecture, swapping out a multi-step text generator for a single-pass pointer model turns out to be a remarkably clean fit for the kinds of routing, triage, and policy checks that quietly dominate production LLM traffic. So now that we have a base understanding and an architecture to work with, how do we train Gev? That’s what we’re going to explore in the next post.