Jev’s Typed Outputs Replace LLM Output Parsing for Closed-Set Decisions
Picture a support-ticket router inside an enterprise agent. A ticket comes in, the agent asks a language model which of eight queues it belongs to, and the model answers warmly: "This looks like a billing issue, so I'd s
Picture a support-ticket router inside an enterprise agent. A ticket comes in, the agent asks a language model which of eight queues it belongs to, and the model answers warmly: "This looks like a billing issue, so I'd send it to Billing." The code downstream was waiting for the single word billing. It got a sentence instead. A regex catches that most days. On the days it doesn't, the ticket sits in an error queue until a human notices.
Nobody would design it this way on a whiteboard, yet a lot of production AI works like this: we pay a model to write a paragraph, then pay engineers to squeeze it back into a value.
This piece covers what that pattern costs, what changes when a model returns typed decisions directly, and where the idea stops helping. Jev, from TypeSafe AI, is the working example. The short version: typed outputs remove format failures, not reasoning failures.
Which of your AI decisions are actually closed-set?
A closed-set decision is one where you can list every acceptable answer before the request is made: route this ticket to one of eight queues, is this a refund request, approve, escalate or block this tool call.
Three traits make a decision a good candidate for a different approach:
- The outputs are known in advance. If you can write the enum, the decision is closed.
- It happens often. A decision made ten times a day isn't worth redesigning. One made ten million times a month is.
- A wrong answer can be caught or undone. Misrouting a ticket is annoying but recoverable. Wiring money is not.
Open-ended work, such as drafting a customer reply or planning a multi-file change, still needs a generative model.
What the generate-then-parse path really costs
When a general-purpose model makes a closed-set decision, the work runs through a prompt, token-by-token decoding, parsing and validation, and a retry whenever the reply is malformed. Each stage has a price.
Here is a deliberately simple, hypothetical illustration. An agent platform makes three million routing and gating decisions a month, and two percent of replies fail validation and need a retry. That is sixty thousand extra model calls a month that produced nothing useful, before counting the added delay and the engineering hours spent maintaining parsers and fallbacks. Swap in your own numbers.
It's worth separating four cost lines when you build your own version of this sum:
- Tokens: prompt scaffolding plus generated output, per decision.
- Latency: time per decision, and the tail (p95 and p99) when retries kick in.
- Failure handling: retry volume, validation compute, and the cost of decisions that fail permanently.
- People: the time spent building, monitoring and debugging the parsing layer.
The fourth line is the one finance teams rarely see, and often the largest.
Why structured-output mode only fixes half of it
The obvious response is that modern APIs already let you force a reply into a JSON schema. That's true, and it helps. Constraining the output removes much of the brittle string parsing that early LLM applications relied on.
But look at what it does and doesn't change. Schema-constrained generation influences what the model is permitted to output. The underlying process is still an autoregressive model producing a sequence of tokens, one after another, even when the application only needs one value out of a few. You've made the reply easier to parse. You haven't changed how the decision is reached or how long it takes.
So structured output is best understood as a reliability improvement on the format of the answer. Useful, and often enough for low-volume work. For high-volume, latency-sensitive decisions, it leaves most of the inference cost where it was.
What typed outputs change, using Jev as the example
A different approach is to build a model whose job is the decision, not the sentence. TypeSafe AI released Jev on September 15, 2026, and describes it as its first "System One" model. It takes unstructured state, such as an email, a log line, or a JSON record, together with a set of questions whose possible answers are declared up front. It returns typed values with probabilities attached, which software can use directly.
There are three kinds of question:
- Choice: pick one option from a named set, such as which team owns this ticket.
- Score: place the input on a numeric scale, such as how urgent it is.
- Yes/no: answer a binary question, such as whether this is a refund request.
Because the answer space is defined by you, the model can't return a label that isn't in the set. The invented category, the stray emoji and the misspelled enum are ruled out by construction rather than caught afterwards. Multiple questions about the same state can also be evaluated together, instead of one slow generation per question.
The parse path loops back whenever a reply fails validation. The typed path returns a value from your set with a probability, and that probability decides what happens next.
It's a familiar idea in new clothes. Classic classifiers also returned a probability per permitted class with no text in between. What's different is the pairing: the typed, bounded interface of a classifier, combined with the broader language understanding of a pretrained model, so you don't have to collect a labeled dataset for every new question. That is the vendor's positioning, and it deserves testing rather than trust.
On the numbers: TypeSafe reports end-to-end latency between 70 and 500 milliseconds, and a price of $0.042 per million input tokens with no separate charge for output tokens. Those are the vendor's figures, not independent measurements. Treat them as a hypothesis to check against your own traffic, your own inputs and your own accuracy bar.
One caution from the open-weight side. A reproduction effort called Laya reports that its base checkpoints score close to chance on its typed-decisions benchmark until they are fine-tuned. The lesson isn't that the idea fails. It's that the quality of a decision model depends heavily on how it was trained, and you should expect to evaluate it as you would any model.
See it in code: the parse path versus the written route
The difference shows up fastest in code: the parse path needs a defensive loop, while the typed path needs one call and a threshold.
The parse path. Here is a typical closed-set router built on a general model. Count the places it can go wrong.
import json
LABELS = {'billing', 'technical', 'account', 'refund', 'other'}
def route_with_llm(ticket, llm, max_retries=2):
prompt = (
'Classify this support ticket into exactly one of: '
+ ', '.join(sorted(LABELS))
+ '. Reply with a JSON object that has one key, label. Ticket: '
+ ticket
)
for attempt in range(max_retries + 1):
reply = llm.generate(prompt) # tokens in, tokens out, every time
try:
label = json.loads(reply)['label'].strip().lower()
except (json.JSONDecodeError, KeyError, AttributeError):
continue # malformed reply: pay for another call
if label in LABELS:
return label # valid and inside the set
return 'other' # silent fallback hides the failure
That is five failure points in about fifteen lines: a reply that isn't JSON, a missing key, a label outside the set, a retry that costs another full generation, and a silent fallback that hides the problem. In production, each one needs logging and alerting.
The typed path. The same job with a typed decision model is one call, with the questions declared up front. The sketch below is pseudocode based on the call shape shown in public Jev tutorials. The real class and parameter names are in TypeSafe's SDK docs.
# Pseudocode: names are illustrative, check TypeSafe's SDK docs for the real types
jev = TypeSafeClient() # reads TYPESAFE_API_KEY from the environment
result = jev.system_one(
state=ticket, # raw text or a JSON record
model='jev-1.13.0', # use the current version from the docs
questions={
'queue': choice(['billing', 'technical', 'account', 'refund', 'other']),
'urgency': score(low=1, high=5),
'is_refund': yes_no(),
},
)
result['queue'].value # 'billing'
result['queue'].probability # e.g. 0.97
Look at what is missing: no prompt scaffolding, no JSON parsing, no check against the label set, no retry loop. The answer can't fall outside the options you declared, and each one arrives with a probability. Public descriptions of Jev also say that several questions about the same state are answered in one pass, instead of one slow sequential generation per question.
Under the hood. The core idea is old: restrict the answer to a known set, and read the model's probabilities instead of its prose. You can approximate it with an open-weight model on vLLM by constraining decoding to your labels and asking for log-probabilities.
from openai import OpenAI
import math
client = OpenAI(base_url='http://localhost:8000/v1', api_key='unused')
resp = client.chat.completions.create(
model='your-open-weight-model',
messages=[{'role': 'user', 'content': 'Queue for this ticket: ' + ticket}],
extra_body={'guided_choice': ['billing', 'technical', 'account', 'refund', 'other']},
logprobs=True,
top_logprobs=5,
max_tokens=8,
)
label = resp.choices[0].message.content
first_token = resp.choices[0].logprobs.content[0]
confidence_proxy = math.exp(first_token.logprob) # a proxy, not calibrated confidence
Two honest caveats. Parameter names for constrained decoding differ between vLLM versions, so check the docs for yours. And raw token probabilities from a general model are not calibrated confidence. The vendor positions Jev as trained for calibration. On the do-it-yourself route, you would have to calibrate on a held-out set yourself.
Confidence as a routing signal
The quiet benefit of typed decisions is the probability that comes with them. A text reply that says "Billing" carries no information about how sure the model was. A typed reply that says Billing at 0.97 versus Billing at 0.52 lets your system behave differently in each case.
That opens a simple routing pattern:
- High confidence: act automatically.
- Middle band: escalate to a larger generative model for a second opinion.
- Low confidence or high impact: send to a person.
Pydantic AI's Jev integration is built around the same split: the typed model answers the typed fields, and anything that needs written language is passed to a language model. It's the cascade idea applied to decisions, where the cheap path handles the easy majority and the expensive path is reserved for what actually needs it.
In code, that cascade is a few lines. The thresholds below are placeholders: set yours from shadow-mode data, not instinct.
AUTO_APPROVE = 0.90 # placeholders: tune on your own labeled sample
ESCALATE_BELOW = 0.60
def decide(answer):
if answer.probability >= AUTO_APPROVE:
return 'act', answer.value # cheap path, no generation
if answer.probability >= ESCALATE_BELOW:
return 'ask_llm', answer.value # second opinion from a larger model
return 'ask_human', answer.value # low confidence: a person decides
The operating-cost question then becomes concrete. What fraction of your volume can be settled on the cheap path at an accuracy you accept? If it's seventy percent, you've removed most of the expensive generations. If it's ten percent, the architecture isn't worth the added complexity. You find out by measuring, not by reading a vendor page.
Where typed outputs fall short
This is the section a risk committee will read first, so it's worth being direct.
Typed output prevents format mistakes, not reasoning mistakes. A decision returned in perfect type can still be wrong. TypeSafe itself notes that typed output does not guarantee a correct decision, especially when live customer data looks different from what the model was trained on. A confident wrong answer is still wrong, and it arrives faster.
Calibration can drift. Probabilities are only useful if 0.9 really means right about nine times in ten. When your input mix changes, that relationship can quietly break. Monitor it the way you monitor any model in production.
Irreversible actions need more than one authority. For deletion, payments, access revocation, or anything that can't be undone, a fast decision model should never be the sole gate. It can be one input among several, alongside policy rules and human approval.
Untrusted input can push on the verdict. If the state you feed in includes text written by outsiders, such as emails, tickets or web content, assume someone may try to influence the decision. Design the gate so that a manipulated verdict still can't cause serious damage.
Governance needs a paper trail. Log the input, the typed output, the probability and the action taken. Typed results make this easier than free text, which is one of the less glamorous but more valuable advantages.
None of this is an argument against the approach. It's an argument for deploying it where the decision is closed, frequent and recoverable, which is where the economics also work best.
A layered architecture you can reuse
It helps to stop thinking of an agent as one big model and see it as layers with different jobs.
- Deliberation layer. A capable generative model plans, explains and writes. This is where open-ended reasoning belongs.
- Decision layer. A typed decision model routes, filters, scores and gates. It handles the frequent, bounded calls.
- Escalation layer. A larger model or a human takes the low-confidence and high-impact cases.
- Control layer. Ordinary code, policy rules and audit logging wrap all of it.
The rule of thumb that falls out of this is short enough to put on a wall: generate when the output is open, type when the output is closed. Most mature systems will use several techniques at once, including structured output, tool calling, classifiers, and typed decision models, each where it fits.
How to pilot this in 30 days
You don't need a platform decision to learn whether this helps. A small pilot answers the question with your own data.
Week 1: choose the workflow. Pick one high-volume, closed-set, recoverable decision, such as ticket routing or tool-call approval. Write down the allowed outputs and the current accuracy, p95 latency and cost per thousand decisions.
Week 2: build a labeled sample. Collect a few hundred real examples with answers your team agrees on, including awkward edge cases. This set will judge every option fairly.
Week 3: run in shadow mode. Send the same traffic to the typed path alongside your current LLM path. Don't let it act yet. Compare accuracy, tail latency, cost, and the rate of format failures.
Week 4: decide with a threshold you set in advance. Agree up front what result counts as a go: for example, matching current accuracy on the confident band while cutting cost per thousand decisions by a margin worth the integration effort. Then route a small share of live traffic, keep the fallback in place, and watch calibration.
If the numbers don't clear your bar, you've spent a month and learned something real. If they do, you have a defensible case with your own evidence behind it instead of a vendor's.
The takeaway
Many agent decisions are closed-set, frequent, and recoverable, and for those, generating prose and parsing it back is an expensive detour. Typed outputs take the detour out: the model returns a value from your list, with a probability, in a form your code can act on. That removes a whole class of format failures and, if the vendor's figures hold up for your workload, a meaningful slice of latency and inference cost.
What it doesn't remove is the possibility of being wrong. Keep the governance, keep humans on the irreversible calls, and measure on your own data.
So here is the exercise to take back to your team this quarter: inventory your agent workloads and tag every model call as open-ended or closed-set.
How much of your inference bill goes to generating text that your code immediately turns back into a label?
Originally published by Dev.to AI. Aggregated on AIWithGhost for educational purposes — full credit and traffic to the original publisher.