Robot-Use Agents: Why the Harness Matters More Than the Model
TL;DR General-purpose LLM agents can now control robots with little or no robot-specific training. The hard engineering work is moving from the model to the harness around it: the tools it can call, the checks that run
TL;DR
General-purpose LLM agents can now control robots with little or no robot-specific training. The hard engineering work is moving from the model to the harness around it: the tools it can call, the checks that run before it acts, and the evals that tell you whether a change helped.
I watched YC's episode Robot-Use Agents: Why General-Purpose Models May Win in Robotics with the founders of Waddle Labs and RoboCurve. Below are the four ideas that stayed with me, then a small harness you can run locally. I have built an MCP based control layer that puts a quadruped and a humanoid behind one tool interface, so this maps closely to my own work.
The idea
The episode builds on MIT professor Phillip Isola's idea of robot-use agents: general-purpose models that control different robots, write policies, and learn new physical tasks with little or no robot-specific training.
Earlier approaches such as RT-2 trained a vision-language model to output robot actions instead of text. The guests argue that today's models are strong enough that this kind of fine-tuning is often unnecessary. They can call tools and write code directly.
Four ideas worth taking away
1. Code as policy
Give the LLM a list of robot functions, such as picking up an object or moving to a pose, and let it write the plan. Google's Code as Policies work (2022) showed this can work with only a few examples in the prompt, because the model has already seen so much code that sequences steps.
2. Compile what works into skills
An LLM thinking at every step is slow. The guests' approach: when a task repeats, turn it into code or a reusable skill. Keep the model for the points of variation, such as recognising an unfamiliar object or deciding what to do after an action fails.
They also said frontier LLM latency is improving about 2x per month. Treat that as their claim, not an established figure, but it explains why they expect real-time control to get closer.
3. Computer-use data may teach spatial reasoning
Dragging a cursor to orbit a CAD model teaches top-down, left-right and near-far. The guests believe this kind of data helps newer models reason about space. One practical tip from the episode: design robot tools so they resemble computer-use tools, and the model does better on physical tasks.
4. Evals decide what ships
One guest shared an experiment where in-context learning stopped improving after roughly 20 to 40 examples. So lessons from a deployment have to leave the context window: first as skills, eventually as weights. It also means you need evals, or you cannot tell whether a harness change helped.
A small harness you can run
The pattern: the model asks, the harness checks, the adapter acts. This uses FastMCP and a simulated robot, so it runs without hardware.
harness.py
from typing import Protocol
from fastmcp import FastMCP
mcp = FastMCP("robot-harness")
class RobotAdapter(Protocol):
"""One adapter per platform. Transport details stay inside."""
def move_to(self, x: float, y: float, z: float) -> None: ...
def stop(self) -> None: ...
class SimAdapter:
"""Stand-in robot so the harness runs without hardware."""
def move_to(self, x: float, y: float, z: float) -> None:
print(f"[sim] moving to ({x}, {y}, {z})")
def stop(self) -> None:
print("[sim] stopped")
# Swap SimAdapter for a real adapter (WebRTC, DDS, ROS, and so on).
robot: RobotAdapter = SimAdapter()
WORKSPACE = {"x": (-0.4, 0.4), "y": (-0.4, 0.4), "z": (0.0, 0.5)} # meters
def in_workspace(x: float, y: float, z: float) -> bool:
return all(
lo <= v <= hi
for v, (lo, hi) in zip((x, y, z), WORKSPACE.values())
)
@mcp.tool()
def move_to(x: float, y: float, z: float) -> str:
"""Move the end effector to a target in meters, in the robot base frame."""
if not in_workspace(x, y, z):
return "rejected: target is outside the allowed workspace"
robot.move_to(x, y, z)
return "ok"
@mcp.tool()
def stop() -> str:
"""Stop all motion immediately."""
robot.stop()
return "stopped"
if __name__ == "__main__":
mcp.run()
try_it.py calls the tools the same way an agent would, through a real MCP client:
import asyncio
from fastmcp import Client
from harness import mcp
async def main() -> None:
async with Client(mcp) as client:
tools = await client.list_tools()
print("tools:", [t.name for t in tools])
ok = await client.call_tool("move_to", {"x": 0.2, "y": 0.1, "z": 0.3})
print("inside workspace:", ok.data)
bad = await client.call_tool("move_to", {"x": 2.0, "y": 0.0, "z": 0.3})
print("outside workspace:", bad.data)
asyncio.run(main())
Run it:
pip install fastmcp
python try_it.py
Output:
tools: ['move_to', 'stop']
[sim] moving to (0.2, 0.1, 0.3)
inside workspace: ok
outside workspace: rejected: target is outside the allowed workspace
Three things to notice:
- The model never sees the transport. It only sees tools. A new robot means a new adapter, not a new agent.
- The safety check lives in code. The workspace limit does not depend on the model behaving well, and no prompt can talk its way past it.
- Every tool returns a clear result. Failures are visible, which is what you need for recovery steps and for evals.
What I would add next
- A task suite with pass or fail checks per tool, so every harness change is measured.
- Logging of every tool call and result, so failures can be replayed.
- A promotion path: when the same sequence of calls succeeds three times in a row, save it as a skill and stop asking the model to rediscover it.
Takeaway
Physical AI engineering is starting to look a lot like agent engineering. Tool design, safety gates, and evals decide whether a good model becomes a useful robot.
Which matters more right now, a better model or a better harness? I would like to hear your view in the comments.
References
Originally published by Dev.to AI. Aggregated on AIWithGhost for educational purposes — full credit and traffic to the original publisher.