Build a multi-agent orchestrator from scratch in Python before reaching for LangGraph

Most teams do not have a LangGraph problem. They have a design problem that they first notice while they are inside LangGraph.

The demo looks fine: a researcher hands work to a writer, the writer hands work to a reviewer, and a report appears. Then a tool times out, a human rejects an approval two days later, or a specialist’s scratchpad leaks into the next agent, and suddenly you are debugging control flow through someone else’s abstractions. CrewAI’s role-and-task model fails in a different way — it is a very fast afternoon, and a very slow incident.

I am not arguing that you should avoid those frameworks. I use them, and I teach them. I am arguing that you should build a small orchestrator yourself first, so that LangGraph or CrewAI becomes an adapter around a design you already understand.

What you are actually building

A multi-agent system is not a group chat with job titles. It is a piece of software that has to do at least five things well:

1. Run a perceive-reason-act loop against a model, without hiding the messages and tool calls.
2. Call tools that have types, permissions, and idempotency keys.
3. Dispatch specialists with isolated context, then merge only the results you trust.
4. Pause for a human, resume after a crash, and refuse a replay that would double-submit.
5. Leave a trace you can audit, and a way to shut the whole thing down.

If your framework does those things, good. If you cannot point to the module that does each one, you do not own the system yet.

Start with one agent and a typed tool

The smallest useful core is boring, which is the point.

• A Message type and an Agent that takes messages and returns messages.
• An LLMClient you can swap. For tests, use a scripted or echo backend. For a real model, point the same client at a local OpenAI-compatible server (Ollama is enough).
• Tools as ordinary functions with a schema, not as free-form prompt text. The model may request create_ticket(title=..., project=...). Your runtime is what actually calls it.

Two properties are worth enforcing on day one.

Types. If the argument is not the shape you declared, the tool does not run. A surprising amount of “the agent went crazy” is a string where an identifier should have been.

Idempotency. Every tool that has a side effect takes a key. A retry with the same key returns the original result. Without that, every eval, every crash recovery, and every “please try again” is a potential duplicate email, ticket, or payment.

ReAct and Reflection are then just loops over this core: reason, maybe call a tool, observe, repeat; or produce an answer, critique it, revise. You do not need a framework to learn those patterns. You need a test that the loop terminates and that a tool error becomes an observation rather than an uncaught exception.

Isolation is the reason to have more than one agent

A second agent is worth it when you need isolation, not when you need more personality.

Give the specialist a brief, not a transcript: the goal, the constraints, the tools it may use, and the artefacts it may read. It should not see the coordinator’s failed tool calls, and it should not see another client’s memory. When it finishes, it returns a structured result — findings, citations, open questions — which the coordinator merges on purpose.

This is also how retries become possible. If the researcher failed halfway through, you resume that node from a checkpoint and you do not force the writer to re-read a pile of obsolete errors.

Teams and swarms are variations on the same idea. A team has named roles and a merge policy. A swarm needs a tighter decision rule, because “let everyone talk” is how context and cost explode. In all cases the coordinator owns the state. Peer-to-peer chatter is optional and usually overrated.

Put the control plane outside the model

A prompt that says “ask before you refund” is not a control. The tool gateway should refuse a risky call unless it carries a valid approval, and that approval should be bound to a hash of the tool name and arguments. If the agent changes the amount after a human has signed off, the call fails.

I like a default-deny policy, an approval record a person can explain later, a tamper-evident audit log that you own (not only the framework’s checkpoint table), and a kill switch that stops dispatch even if a model is mid-loop. Traces should describe model calls, tool calls, and policy decisions in something close to the OpenTelemetry generative-AI conventions, so you are not trapped in one vendor’s dashboard.

This is also why I want an offline evaluation harness. Freeze real tasks as fixtures. Run them first against the scripted backend so the tests stay deterministic and do not need API keys. Then run the same fixtures against a real model if you want quality scores. Include the ugly cases: process killed mid-tool, approval rejected, approval arriving two days later, tool result that tries to talk the agent into a forbidden call.

Then, and only then, reach for LangGraph

Once the skeleton exists, re-expressing it on a framework is a straightforward exercise.

• In LangGraph, your coordinator state becomes the graph state, each specialist is a node or a subgraph, and interrupt() is the approval pause. Use a durable checkpointer; test crash recovery yourself, including the durability mode.
• In CrewAI, roles and tasks map cleanly to a first demo, but for production-shaped control you want Flows and an explicit state, not implicit message history. Check that @human_feedback behaves in a real worker, not only at the console.
• AutoGen’s group-chat model is useful as history, but the project has been in maintenance and new work should be tested on Microsoft Agent Framework. Keep an adapter if you have old code; do not start a new system on a frozen runtime.

The test of whether you did this right is exit cost. Your tools, policy, audit log, and business logic should still make sense if you delete the framework dependency. If they do not, the framework owns you.

A concrete capstone

I wanted a single product that forces all of the above into the same codebase: a Research-to-Report application that plans a research brief, dispatches specialist agents in parallel, and produces a cited report with a live dashboard. Planning, isolation, merge, traces, and the governance plane all have to exist or the app is just a chatbot with extra steps.

The teaching implementation is called OrchestraKit. It is Python, it grows over 15 chapters, and it runs offline by default. When you want a real model, Gemma 4 via Ollama drops into the same client. I also re-express the same design on LangGraph, CrewAI, and AutoGen/AG2 so you can see the adapters side by side.

If you want to read the code first:

https://github.com/zahurul-islam/Agentic-AI-Orchestration

The longer, chapter-by-chapter treatment is in my book, Agentic AI Orchestration: Building Multi-Agent Systems from Scratch. The repo is enough to try the argument: clone it, run the tests with the network unplugged, then replace the echo backend with a local model and see what still holds.