LLMs in Robotics: Applications and Opportunities

We're building a natural-language mission planner for mobile manipulation robots. It converts unstructured human commands into structured action sequences with preconditions and failure recovery. This helps robotics teams prototype autonomy stacks without writing brittle rule-based parsers.

What you'll need

• Python 3.10 or newer
• The OpenAI SDK: pip install openai
• An Oxlo.ai API key from https://portal.oxlo.ai

Oxlo.ai offers flat per-request pricing, which keeps costs predictable when you are sending long robot capability manifests and multi-turn planning prompts. See https://oxlo.ai/pricing for details.

Step 1: Configure the Oxlo.ai client

First, initialize the OpenAI SDK pointing at Oxlo.ai. I use Llama 3.3 70B as the general-purpose backbone because it handles structured generation reliably.

from openai import OpenAI

client = OpenAI(base_url="https://api.oxlo.ai/v1", api_key="YOUR_OXLO_API_KEY")

Step 2: Model the robot action space

Before prompting the LLM, we define a strict JSON schema of primitives the robot actually supports. This keeps the model grounded in hardware capabilities.

ROBOT_SCHEMA = {
    "actions": [
        {
            "name": "navigate",
            "args": {"target_zone": "string", "speed": "float"},
            "preconditions": ["battery_above_10", "path_clear"]
        },
        {
            "name": "detect_object",
            "args": {"object_type": "string", "zone": "string"},
            "preconditions": ["camera_online"]
        },
        {
            "name": "pick",
            "args": {"object_id": "string", "max_weight_kg": 5.0},
            "preconditions": ["gripper_empty", "object_in_range"]
        },
        {
            "name": "place",
            "args": {"target_zone": "string"},
            "preconditions": ["holding_object"]
        },
        {
            "name": "charge",
            "args": {"dock_id": "string"},
            "preconditions": ["at_dock"]
        }
    ]
}

Step 3: Craft the system prompt

The system prompt constrains the model to emit only valid JSON plans and includes physical constraints like max payload and battery thresholds.

SYSTEM_PROMPT = """You are a robot mission planner. Convert the user's natural language command into a strict JSON plan.

Rules:
- Use only the actions defined in the provided robot schema.
- Include a "preconditions" checklist for each step.
- Add a "recovery" field to each step describing what to do if that step fails.
- Respect max_payload_kg of 5.0 and minimum battery of 10 percent.
- Output raw JSON only. No markdown fences.

Respond with this structure:
{
  "mission_id": "string",
  "steps": [
    {
      "step_num": 1,
      "action": "navigate",
      "args": {...},
      "preconditions": [...],
      "recovery": "..."
    }
  ],
  "estimated_battery_drain_percent": "number"
}
"""

Step 4: Generate a mission plan

We send the user command and action schema to Oxlo.ai, then parse the structured plan. I use Qwen 3 32B here because its agentic reasoning produces robust multi-step plans.

import json

def generate_plan(user_command: str) -> dict:
    response = client.chat.completions.create(
        model="qwen-3-32b",
        messages=[
            {"role": "system", "content": SYSTEM_PROMPT},
            {"role": "user", "content": f"Robot schema: {json.dumps(ROBOT_SCHEMA)}\nCommand: {user_command}"},
        ],
        temperature=0.2,
    )
    raw = response.choices[0].message.content
    return json.loads(raw)

Step 5: Validate the plan

Generated plans need a hard guardrail before execution. We run a lightweight validator that checks for collision risks and battery feasibility.

def validate_plan(plan: dict) -> bool:
    steps = plan.get("steps", [])
    for i, step in enumerate(steps):
        action = step.get("action")
        args = step.get("args", {})
        
        if action == "pick":
            weight = args.get("max_weight_kg", 0)
            if weight > 5.0:
                raise ValueError(f"Step {i+1}: payload {weight}kg exceeds 5kg limit")
        
        if action == "navigate" and i > 0:
            prev = steps[i-1].get("action")
            if prev == "pick" and "holding_object" not in step.get("preconditions", []):
                raise ValueError(f"Step {i+1}: navigate while holding object needs holding_object precondition")
    
    drain = plan.get("estimated_battery_drain_percent", 0)
    if drain > 90:
        raise ValueError("Estimated battery drain too high for safe mission")
    
    return True

Step 6: Run the interactive loop

This ties everything together. We read a natural language command, generate the plan, validate it, and print the executable sequence.

if __name__ == "__main__":
    command = (
        "Pick up the red crate from zone B and deliver it to zone C. "
        "If battery drops below 15 percent during the mission, abort and go to charge dock alpha."
    )
    
    print("Generating plan via Oxlo.ai...")
    plan = generate_plan(command)
    
    print("Validating plan...")
    if validate_plan(plan):
        print("\n--- Valid Mission Plan ---")
        print(json.dumps(plan, indent=2))
    else:
        print("Plan validation failed.")

Run it

Save the script as robot_planner.py, export your key, and execute:

export OXLO_API_KEY="sk-..."
python robot_planner.py

Example output:

Generating plan via Oxlo.ai...
Validating plan...

--- Valid Mission Plan ---
{
  "mission_id": "red_crate_delivery_001",
  "steps": [
    {
      "step_num": 1,
      "action": "navigate",
      "args": {"target_zone": "zone B", "speed": 0.5},
      "preconditions": ["battery_above_10", "path_clear"],
      "recovery": "Re-plan path or request manual assistance"
    },
    {
      "step_num": 2,
      "action": "detect_object",
      "args": {"object_type": "red crate", "zone": "zone B"},
      "preconditions": ["camera_online"],
      "recovery": "Sweep zone B and retry detection"
    },
    {
      "step_num": 3,
      "action": "pick",
      "args": {"object_id": "red crate", "max_weight_kg": 3.2},
      "preconditions": ["gripper_empty", "object_in_range"],
      "recovery": "Adjust gripper position and retry"
    },
    {
      "step_num": 4,
      "action": "navigate",
      "args": {"target_zone": "zone C", "speed": 0.3},
      "preconditions": ["battery_above_10", "path_clear", "holding_object"],
      "recovery": "Abort and go to charge dock alpha"
    },
    {
      "step_num": 5,
      "action": "place",
      "args": {"target_zone": "zone C"},
      "preconditions": ["holding_object"],
      "recovery": "Reposition and retry placement"
    }
  ],
  "estimated_battery_drain_percent": 35
}

Wrap-up

Robotics workloads often involve long system prompts and iterative agentic planning, which makes token-based billing unpredictable. Oxlo.ai's flat per-request pricing removes that variance, so you can send full capability manifests and multi-turn correction loops without watching costs scale with context length. For next steps, wire this planner into a ROS2 node or a simulator like Isaac Sim to execute the generated steps on a virtual robot. You can also swap in Kimi K2.6 for vision-enabled missions where the model processes camera descriptions alongside the plan.