What app code should assume about an LLM call

Strip away the chat window and the model does one thing. Given the text so far, it predicts the next token, a word or a piece of a word. Software samples a token, appends it, and runs the model again. A few hundred words means that loop ran once per token. Temperature changes the sampling probabilities, and the prompt supplies context and instructions the model may not reliably follow. Your checks have to catch a bad sample.

What is the model doing on every API call?

The model assigns a probability to every token in its vocabulary. The API usually samples from those probabilities, which is why the same prompt can return two different strings.

Dan Klein, a computer science professor at UC Berkeley and co-founder and CTO of Scaled Cognition, traces that job to speech recognition. The acoustic model could match sounds. A language model scored which of several sound-alike transcriptions a person would actually say. On Chain of Thought episode 54 he described the failure you still see in products:

the system will do something plausible, but not actually correct. So it looks like it has the right shape.

Dan Klein, co-founder and CTO of Scaled Cognition, on Chain of Thought episode 54

He argues the same model now holds long-distance knowledge, topics, syntax, and real-world plausibility. The architecture almost every current LLM uses is the transformer, introduced by Google researchers in the 2017 paper "Attention Is All You Need". Attention lets each token draw on the earlier tokens in the context. Standard attention scales with the square of the sequence length, so a very long prompt costs more compute and memory than a short one.

Do not read the loop as "no planning." Anthropic's interpretability team traced Claude writing rhyming poetry and found it picks the rhyme for the end of a line before writing the words that lead there. Generation still emits one token at a time. The model can plan ahead inside that constraint.

For application code, treat the reply as a plausible sample. Nothing in next-token prediction tests it against the facts, which is one reason models hallucinate. I would validate every field I parse. I would also keep prompts short when the task allows it. On API pricing, every token read and written is billed, so long conversations and multi-step agents get expensive, and long contexts get expensive again because attention grows with the square of the length.

What did pre-training put in the weights?

During pre-training, the model reads text from the web, books, code, and other sources and learns to predict the next token at every position. Nobody labels that data. The text itself supplies the answers. Training nudges billions of parameters so the guesses get less wrong. Over trillions of tokens it picks up grammar, facts, styles, and reasoning patterns. Meta's paper on its open Llama 3 models describes the largest as a transformer with 405 billion parameters. A frontier run takes weeks to months on thousands of specialized chips, so few organizations pre-train from scratch.

What comes out is a base model. It is good at continuing text. It does not reliably answer questions or follow instructions, because that stage never taught it to. Some hallucinations start here, because rare facts are hard to learn.

The limit I would code against is the knowledge cutoff. Training does not cover events after that date, or your private data, unless the model is trained further on them or you put that information in the context. Retrieval-augmented generation is the common way to supply it in context. I would treat a date, a price, a policy, or a private record as unknown until my code places a current source in the prompt. A fluent sentence about last week can still be a continuation of older text.

What does post-training change before you call the API?

Post-training is what turns that base model into an assistant. It usually starts with supervised fine-tuning on examples of good conversations, then preference training, where the model learns from human or AI rankings of its answers (RLHF or DPO). When the rankings come from people, the method is reinforcement learning from human feedback. Newer reasoning models add reinforcement learning on problems with checkable answers, like math and code.

OpenAI's InstructGPT paper showed how large this stage can be. On the prompts the paper tested, human evaluators preferred answers from a 1.3-billion-parameter model trained this way over answers from the 175-billion-parameter GPT-3, despite the trained model having 100 times fewer parameters.

Adapting a model this way is usually much cheaper than pre-training one, though a large reinforcement learning run can be expensive too. Companies now post-train open models themselves. Joel Hron, Thomson Reuters' chief technology officer, describes starting from an open-source model, realigning it to the company's standards, then continuing pre-training and post-training it on the company's legal and tax expertise. On episode 69 he stated the bet:

And as the tide of open source rose, our ability to post-train our unique knowledge on top of that would rise as well with that tide.

Joel Hron, chief technology officer at Thomson Reuters, on Chain of Thought episode 69

Intercom did a smaller version of the same move, post-training a 14-billion-parameter open model for one high-volume job.

Instruction following and answer style come from this stage, not from the raw next-token loop. Klein argues that a prompt has no precise semantics and no guarantee: natural language is ambiguous, and nothing forces the model to do what you asked. I would pin the model id, and I would retest format, refusals, and tone whenever that id changes. The parser is the check.

Why does temperature change the string you get back?

At each step the model produces a logit for every vocabulary token, and a softmax turns those scores into probabilities. Temperature divides the logits by a number t before the softmax. Below 1, the distribution skews toward the most likely tokens and the tail shrinks. Above 1, it flattens, and unlikely tokens get picked more often. Near 0, sampling approaches always taking the top token.

Holtzman and coauthors, in "The Curious Case of Neural Text Degeneration", document the tradeoff. Lowering temperature improves quality at the cost of diversity. Always picking the most likely continuation tends to produce bland, repetitive text. Sampling from the full distribution drifts into incoherence. The authors proposed nucleus (top-p) sampling as another way to cut that tail.

This sketch only shows the division before the softmax. It is not a provider SDK, and it is not how any API samples.

# Sketch only. Not a vendor sampler.
import math

def probabilities(logits, temperature):
    t = max(temperature, 1e-6)
    scaled = [logit / t for logit in logits]
    peak = max(scaled)
    weights = [math.exp(value - peak) for value in scaled]
    total = sum(weights)
    return [weight / total for weight in weights]

I would set temperature near 0 for extraction, labels, and any field the next line of code parses. I would raise it only when I want several different drafts and I can drop a bad one. Low temperature buys consistency. It can also make open-ended writing bland and repetitive, so I would not use it as a universal default.

Checklist

• Treat the completion as a sampled continuation, and validate every field you parse.
• Treat facts after the knowledge cutoff, and any private data, as absent until you put a current source in the context.
• Pin the model id and retest format, refusals, and tone when it changes. That behavior comes from post-training.
• Keep temperature near 0 when software will parse the text. Raise it only if you can discard a weak sample.
• Budget tokens. Long prompts are billed per token, and standard attention scales with the square of the sequence length.

The longer explainer, with the episode clips, is on Chain of Thought. It draws on this episode.

Subscribe to the Chain of Thought newsletter for new episodes and write-ups like this one.

Drafted with AI assistance from the episode transcripts.