Three Agent Harness Pitfalls I Hit: Why Context Gets Slower as It Grows
While building my own agent harness, I ran into a problem that took longer to diagnose than to fix: the longer a session ran, the slower it got. My first assumption was model behavior. The actual cause was unaddressed ga
While building my own agent harness, I ran into a problem that took longer to diagnose than to fix: the longer a session ran, the slower it got. My first assumption was model behavior. The actual cause was unaddressed gaps in the harness layer itself. This article documents three of those gaps with the code changes that closed them, for anyone else running LLM applications in production.
Pitfall One: Context Bloat — Everything Gets Resent Every Time
The initial design was the simplest possible one. On every model call, push the full conversation history.
Short tasks tolerated this. After a few dozen rounds, the arithmetic stopped working. Context length grew without bound, latency climbed with it, and token cost roughly doubled per turn at the tail end of a session. The same prompt that responded in under a second at round five took several seconds at round forty.
There are three distinct problems bundled inside that symptom, and they need three separate fixes.
Layer the message history by type. System prompts, user input, tool results, and assistant replies have different lifetimes and different volatility. Managing them as one undifferentiated array means every compression or truncation operation applies uniformly, which is wrong. Tool results in particular are large, disposable, and rarely needed again in full.
Summarize beyond a round threshold. After N rounds, the older portion of the conversation should be compressed into a summary rather than retained verbatim. The original text of a tool call from twenty rounds ago is almost never load-bearing. Its conclusion is.
Truncate tool output by default. Long text returns — file contents, search results, log dumps — should retain only the first N tokens. If the model needs more, it can ask for more. Paying for a full 50,000-token file read on every subsequent turn because it happened once is pure waste.
Here is the implementation.
def trim_context(messages, max_tokens=8000):
while estimate_tokens(messages) > max_tokens:
if len(messages) > 4:
# compress the middle, keep the head and tail
summary = summarize(messages[1:-2])
messages = [messages[0],
{"role": "user", "content": summary}] + messages[-2:]
else:
break
return messages
The structural invariant matters more than the specific numbers. The first message — the system prompt — is never touched. The last two messages are never touched, because they carry the immediate task state. Everything between them is eligible for compression. The loop exits when the token estimate falls under the ceiling, or when there is nothing left to compress.
After this change, context length stabilized and latency dropped with it. The key insight is that the ceiling is enforced by the harness rather than negotiated with the model.
| Symptom | Root cause | Fix |
|---|---|---|
| Latency grows with round count | Full history resent each call | Round-threshold summarization of middle messages |
| Token cost doubles late in session | Unbounded context accumulation | Hard token ceiling in trim_context
|
| Single huge tool output dominates | File reads, search dumps retained wholesale | First-N-token truncation, refetch on demand |
| System prompt degraded by compression | Uniform treatment of all messages | Type-layered management; head and tail protected |
Pitfall Two: The Tool Retry Dead Loop
The second issue appeared during a longer agent run. The agent was stuck on a single tool call, retrying it more than ten times in sequence.
The mechanism was not subtle. The tool returned an error. The harness had no retry limit. The model saw the error message, reasoned that it should try again, and issued an identical call. The tool failed identically. The cycle repeated until the session was killed manually.
Two layers of protection resolved it.
Cap retries per tool invocation. Set a maximum of three attempts. Beyond that, return an explicit failure message to the model so it switches strategy instead of repeating the same call.
Classify errors before retrying. Network timeouts are retryable — the same call may succeed on the next attempt. Parameter errors are not. A malformed argument will fail identically every time it is sent. Non-retryable errors should be returned immediately with an instruction to use a different approach.
def call_tool_with_retry(name, args, max_retries=3):
for i in range(max_retries):
result = call_tool(name, args)
if not result.is_error:
return result
if is_non_retryable(result.error):
return result
return ToolError(f"{name} failed {max_retries} times consecutively; try another approach")
The distinction between retryable and non-retryable is the part that actually saves tokens. A blind retry loop burns the full cost of the failed call plus the reasoning that produced it, multiplied by the retry count. Three attempts on a timeout is a reasonable cost. Three attempts on a schema violation is three times the price of a guaranteed failure.
One additional habit is worth adopting at the same time: include the failure reason in the message handed back to the model. A bare "error" gives the model nothing to reason about. "Argument start_date must be ISO format" gives it an immediate corrective action.
Pitfall Three: Failure Recovery — No Idea Where It Stopped
The third problem is specific to long-running tasks. If the model crashes or the request times out, the next session starts with no knowledge of where the previous one stopped.
The naive response is to add a checkpoint. The actual implementation requires three pieces, not one.
Write a snapshot after each round. The snapshot has to capture the current message list, the tool call record, and intermediate state. Message history alone is insufficient — if the agent had already written two files and was on the third, resuming with only the conversation transcript means re-executing the first two.
Check on startup and offer a choice. If a checkpoint exists, present the user with an explicit decision: start over, or resume from the last breakpoint. Automatic resumption is the wrong default. Sometimes the right move is to discard the run and start clean, especially if the failure was caused by a bad instruction.
Refeed the checkpoint context on resume. When resuming, the model needs to be told what it already completed. Without that, it will redo work, or worse, contradict a decision it made before the break.
def save_checkpoint(messages, state):
json.dump({"messages": messages, "state": state},
open("checkpoint.json", "w"))
def load_checkpoint():
if os.path.exists("checkpoint.json"):
return json.load(open("checkpoint.json"))
return None
The state object is where most implementations cut corners. Recording the message list is straightforward. Recording which tools have executed, which artifacts exist, and which subtasks are complete is the part that makes resumption actually work. A checkpoint without state is a transcript, not a recovery point.
| Checkpoint component | Purpose if omitted |
|---|---|
| Message list | Model loses conversational context and contradicts prior decisions |
| Tool call record | Completed calls are re-executed, duplicating side effects |
| Intermediate artifacts | Files written before the break are orphaned or overwritten |
| Subtask completion flags | Agent repeats finished work and never reaches remaining tasks |
Engineering Tradeoffs: Self-Built vs Mature Frameworks
Three harness approaches are in common use.
- Fully self-built — starting from the OpenAI SDK and assembling everything manually. Maximum flexibility, maximum pitfall count. This is what the preceding three sections describe.
- Codex-style frameworks — tool invocation, context management, and retry logic already integrated. Faster to a working agent, less control over edge behavior.
- Harness-style frameworks — heavier emphasis on state management and multi-agent coordination. Better for orchestration workloads, less suited to single-agent prototypes.
| Approach | Strength | Cost |
|---|---|---|
| Fully self-built | Every step is inspectable; behavior is fully under your control | Every pitfall is yours to discover and fix |
| Codex-style | Retry, context, and tooling already handled | Must adopt its abstractions; some requirements are unsupported |
| Harness-style | State management and multi-agent coordination built in | Overhead for simple single-agent workloads |
The recommendation is straightforward. If the goal is learning, build the harness yourself and walk through these pitfalls — the understanding gained is durable. If the goal is a production environment, use a mature framework and avoid rebuilding the same failure modes from scratch.
What the Harness Actually Abstracts
A harness is not an assembly of the OpenAI SDK. Wiring SDK calls together produces a script, not a harness.
The abstraction has four parts: context management, tool invocation, error recovery, and state persistence.
Context management decides what the model sees on each call, in what order, and at what cost. It owns summarization thresholds, truncation policies, and the token ceiling.
Tool invocation owns schema validation, the retry ceiling, error classification, and the failure messages returned to the model.
Error recovery owns checkpoint cadence, state capture, and resumption semantics.
State persistence owns the durable record: what completed, what remained, and what artifacts exist on disk.
Get those four right, and the model behind them becomes a swappable component. GPT, Claude, or an open-source engine can be substituted without rewriting the workflow, because the workflow lives in the harness layer rather than in the prompt.
That is the layer worth abstracting. A prompt is a one-time instruction. A harness is a machine that survives repeated execution, and repeated execution is where every failure in this article surfaced.
One more consideration belongs at the same layer: the boundary to the model provider itself. Teams running several models behind one harness usually maintain a routing layer so that endpoint configuration, credentials, and per-model policy live in one place instead of being duplicated per client. 4sapi provides unified model access as an API gateway, which keeps that boundary in a single configuration surface while the harness owns context, tools, recovery, and state. Swapping the underlying model then becomes a change to one setting rather than a refactor of four subsystems.
Checklist
The three pitfalls reduce to a short list that is worth verifying before any agent ships.
- A hard token ceiling is enforced by the harness, not hoped for in the prompt.
- Message history is layered by type, with the system prompt and the two most recent messages protected from compression.
- Tool output is truncated at ingest, with full retrieval on explicit request.
- Retry attempts are capped at a specific number per tool call.
- Errors are classified retryable and non-retryable, and non-retryable failures are returned with a corrective instruction.
- Checkpoints capture state, not just transcripts.
- Resumption is a user decision, not an automatic default.
- The model provider is a swappable component behind the harness boundary.
None of these require a sophisticated architecture. All of them require that the harness, and not the model, own the decision.
International access: https://4sapi.com
Domestic access: [https://4sapi.org](https://4sapi.org
Originally published by Dev.to AI. Aggregated on AIWithGhost for educational purposes — full credit and traffic to the original publisher.