Jarvis Nelson Osorio on Mechanistic Interpretability: What Founders Should Ask About the Inside of a Model

Canonical: Jarvis Nelson Osorio on Mechanistic Interpretability: What Founders Should Ask About the Inside of a Model — also at jarvisosorio.com. Originally published by Jarvis Nelson Osorio (Jarvis Osorio).

When a model agrees with a user a little too quickly, or refuses a request for reasons nobody can reconstruct, the problem is not only "alignment" in the abstract. It is operational. Someone on your team has to explain the behavior to a customer, a teacher, a regulator, or a child. I am Jarvis Nelson Osorio — also Jarvis Osorio — a technology entrepreneur and innovator, AI systems architect, DAO institutional designer, author and artist based in Chile. I have become convinced that mechanistic interpretability is no longer a research hobby. It is a founder skill.

What the phrase actually means

"Interpretability" is often used as a polite synonym for a heatmap or a paragraph of marketing. Mechanistic interpretability aims at something stricter: can we identify the internal parts of a model — circuits, features, attention patterns — that cause a particular output? The research community is still early. That is not an excuse to ignore it. Founders do not need to invent a new science. They need a habit of asking whether a system can be inspected when it matters.

A black box can still be useful. A black box that spends money, talks to children, or writes policy for a company is a different object. In The Compliant Mind I write about yes-machines: systems optimized to agree. Interpretability is one way to notice when agreement is a performance rather than a reasoned answer.

Three questions that belong in a vendor call

You do not have to run a research lab to use this lens. In a serious conversation with a model provider or an internal machine-learning team, I would ask:

1. When this system fails, can we see more than the final sentence? Logs of prompts are necessary and not sufficient. Can we recover which tools were called, which retrieved documents were used, and — where the vendor allows it — which internal signals fired?
2. How would we detect confident nonsense? Multi-agent setups can agree and still be wrong. Interpretability will not magically fix that, but a culture of inspection makes it harder for a smooth paragraph to close the case.
3. Which behaviors are off-limits even if we cannot fully "see" inside? If a model can initiate a payment, change production data, or speak to a minor, the absence of interpretability should tighten permissions, not loosen them.

Those questions are institutional, not theatrical. They belong in the same family as the audit trails I argue for in Breaking the Centralized Order: write the rules, separate proposer from approver, keep a record someone else can read.

Security is the practical door

A lot of enterprise interest in interpretability arrives through security. If you cannot tell whether a model is being steered by a hostile prompt, or whether it is leaking a memorized secret, "trust the output" is not a control. Teams already inspect network traffic and application logs. Model internals will not look the same, but the instinct should: serious systems leave an inspectable trail.

I do not claim that today's tools give us a complete map of a frontier model's mind. I do claim that teams who treat the model as an oracle will be last to notice when the oracle starts improvising.

What to do this quarter

A practical program for a small company can start without a research budget:

• Inventory every workflow where an AI system can take an action, not just write a draft.
• For each one, write down the evidence you would want after a bad day: inputs, retrieved sources, tool calls, human approvals, and the output as the user saw it.
• Prefer vendors who offer tracing, evaluation hooks, and a way to export those records.
• Pair inspection with limits. An interpretable system that can still move money with no second person is not "safe enough."
• Teach the team a simple language: "We don't know why" is an acceptable interim answer; "the model said so" is not.

Art sits in the background of this work for me. Explosive visual representation is a way of making pressure visible. Interpretability is the same instinct applied to cognition we did not hand-write.

The work is hopeful. We are early enough that habits still travel. A founder who asks better questions this year will hire better, buy better, and sleep better when the first confusing incident arrives — as it will.

Where to read more

I publish essays, books and profiles at jarvisosorio.com and on the hub Jarvis Nelson Osorio — articles, books, videos and profiles. For the human side of systems that prefer agreement, start with The Compliant Mind.

The inside of a model will never be as tidy as a flowchart. That is not a reason to stop looking. It is a reason to look with humility — and to design products that still work when the looking is incomplete.