What If Vibe Coding Weren't a Chat — but a 3D Drive?
When an agent's candidate directions become forks in the road, its execution becomes actual driving, and every choice leaves a trace you can revisit — can we move from "waiting for the AI's answer" to driving a human–AI
When an agent's candidate directions become forks in the road, its execution becomes actual driving, and every choice leaves a trace you can revisit — can we move from "waiting for the AI's answer" to driving a human–AI collaboration?
The cover is an AI-generated concept image. Every visual below is labeled in one of three evidence tiers: ① AI concept art; ② UI demos with locally injected candidates (marked "demo"); ③ **live-run recordings and screenshots against a real backend* (marked "live run"). In this live run, execution uses a small local model served by Ollama on the author's machine; direction forecasting goes through the fast local channel and is honestly labeled "local preview," while the execution chain is genuine streaming model output.*
Prototype view ① — the cockpit before any question is asked. Actually rendered in the browser; no simulated results.
01 | Today's vibe coding feels like directing an invisible car over the radio
You say: "Give this page a cleaner layout." The agent starts writing code. A moment later, it tells you it's done. You ask: why this layout? Was there a bolder version? Where does this change sit in the product's bigger picture? Is the next step interaction, performance, or checking whether users even understand it?
These questions pile up inside a single chat log. A chat window can hold code, explanations, and commands — but it struggles to show you the relationships between directions: which options are parallel alternatives, which are sequential dependencies, and which are attempts already taken and rolled back. Your finger is on the Send button, but you aren't necessarily holding a map.
This project proposes an ambitious inversion: don't just let the AI tell you what it plans to do — let you see the possible routes in space, and drive the agent down one of them yourself.
This is not a chat window glued next to a 3D toy car. It attempts to redefine the object of interaction: from "the next message" to "the next leg of a journey." If the idea holds, vibe coding becomes more than writing code in natural language — it becomes a collaboration you can preview, choose, follow, verify, and replay.
02 | How does one question grow a road?
Imagine the task: "Help me build the home page of a weather app." In a chat box, that's a prompt awaiting a reply. In the cockpit, it's an intersection about to unfold.
Ahead of you, signposts appear: define the information architecture first, build a runnable page first, establish the visual style first, verify the weather data source first. These aren't four decorative buttons; they should be genuinely distinct courses of action. You can read each route's rationale and steps before choosing which one to drive into — or propose your own direction instead of being confined to the machine's menu.
The prototype's core interaction is already built around this metaphor: a forecaster generates candidates and estimates their divergence; small differences render as something closer to a straight road, while meaningful differences fork; the chosen signpost triggers execution; on completion, the result is anchored to the node you drove to; continuing the exploration spawns new directions ahead. The prototype also experiments with side-by-side candidate comparison, keyboard route selection, and custom directions within the same interaction.
Live animation ① — after typing "help me plan the development steps for a weather app home page," a dark path skeleton unfolds first, then lights up into five colored signposts when the forecast returns. A real backend is in the loop; this batch of candidates comes from the fast local channel, honestly labeled "local preview" in the UI.
Live screenshot ① — the full UI at the same moment: 3D forks, candidate cards on the right, and a "local preview" provenance bar on top.
Prototype view ② — the actual WebGL scene, forks, and candidate cards. The candidates ("information architecture first / runnable page / visual style") were injected by a local script as a UI demo; the "local fallback" badge on top is a genuinely displayed source label and does not mean a model produced them.
Prototype view ③ — the actual panel after clicking "compare candidates"; the candidate data is likewise locally injected demo data.
Translating the workflow into driving language is a re-division of labor for the AI product interface:
| The question in agent collaboration | Its expression in the 3D drive | The capability the user gains |
|---|---|---|
| What could happen next? | Signposts and forks ahead | Compare directions before departing |
| How do the options differ? | Road geometry, placement, candidate notes | Understand the choice instead of blindly clicking "continue" |
| Where are we now? | In-drive status and HUD widgets | Follow execution — provided the signals are real |
| What did this round produce? | Arrival sign and result card | Open the artifact instead of just seeing "done" |
| Why did we end up here? | Driven trajectory, timeline, decision chain | Review, compare, and re-fork |
This is the truly disruptive part: the agent stops being an answerer on the other side of a dialog box and becomes an action system you navigate together with.
03 | From prompt engineering to steering-wheel engineering
Classic prompt interaction compresses a huge number of decisions into one sentence. You're expected to settle style, features, boundaries, and validation before you start; anything you forget costs another round of prompt patching. The 3D-driving idea externalizes some of those decisions: you don't have to write the whole route up front — you see a few different roads ahead and judge where to steer this step.
For vibe coding, that means a new collaboration rhythm. Building the weather app: you first drive into "runnable page" and get a first version. On arrival, the road ahead forks into "wire up real data," "tune the mobile layout," and "add severe-weather alerts." If the page turns out pretty but unusable, you revisit an earlier node, review the choice you made, and decide to re-drive along another direction. This isn't the AI deciding your product for you — it's human intent re-entering the system at every meaningful fork.
Space also offers something a chat stream can't naturally provide: memory that isn't just a wall of scrolling text. Roads you've driven become locatable nodes. You can put two results side by side, know which round came from the model and which was only a fast local preview, and tell whether a "re-drive" actually executed a new task or merely replayed an old answer.
Browser capture ① — switching from the fork view to side-by-side comparison. These are genuinely rendered prototype frames; the candidate content is the labeled local demo data.
Browser capture ② — real UI changes while selecting a direction via keyboard; not a task-execution recording.
Live run: pick a road and watch it actually execute
The next two clips come from a full execution deliberately wired to a real backend: keyboard-select a candidate → genuine streaming model generation → arrival → open the result card → drive a second leg along a "future signpost." The execution backend is a small local model; the output is genuinely generated text, and the forecast's provenance is honestly distinguished in the UI.
Live animation ② — real execution of step one, "research and plan a weather app home page": the task bar's streaming text grows token by token, then the arrival sign rises and the result card expands with real model output.
Live animation ③ — the second real drive after selecting the "organize the research" future signpost: the camera advances along the new road segment, a "forecasting next direction" placeholder sign appears ahead, and once it lands, a new arrival sign and result card take over.
But let's be clear: the spatial metaphor is not itself a capability. If "wire up data first" and "polish the page first" differ only in wording but execute identically, the fork carries no decision value. If a user-proposed direction can't actually be executed, the steering wheel is just an ornament. The interaction revolution hinges on real semantic divergence, real actions in execution, and verifiable results.
04 | Two coordinate systems inside the cockpit
The prototype can be understood through two coordinate systems. The first is semantic space: do the candidates truly represent different task directions? The project uses synonym-cluster folding, bucketing, and distance metrics to avoid treating "look up references," "retrieve references," and "search for references" — three phrasings of the same intent — as three separate roads. The second is path space: how are those directions laid out in 3D, selected, driven into, and left behind as trajectories?
These two coordinate systems must translate into each other. If semantic space distinguishes several meaningful directions but the scene merely arranges them as structurally identical boards, the user still can't choose. Conversely, if roads crisscross the scene with no genuinely different tasks or executions behind them, you're just chrome-plating a void with neon.
Both visuals above are explanatory illustrations. The project calls this failure of semantic change and spatial movement to mutually attest a "bidirectional broken link." It is a design and research hypothesis, not a user-psychology mechanism confirmed by human experiments.
So the genuinely interesting future question isn't "render more roads" — it's making the geometry of the roads carry decision information: which directions are close, which diverge sharply, which steps must run in sequence, and which road forks off an older node. Only when space is treated as an information structure rather than stage scenery can the driving metaphor add a channel of cognitive bandwidth that a chat box lacks.
05 | The killer scenario: turning a coding task into a journey you can replay
Now zoom from abstract "agent collaboration" down to a day of development work. You tell the agent: "Add a login page to the existing app, and make it work on mobile." If the interaction is still a single chat thread, you typically read its plan, then read its code, and stitch the causality together yourself across multiple replies.
In the cockpit's ideal flow, you see several routes before departure: implement within the existing component system, run a low-fidelity interaction check first, or map out the auth APIs first. After choosing "implement within the existing component system," what the drive displays should be actual events: which file was modified, which build or test is running, where it failed, and what the outcome was. On arrival, you can open the change list and a preview — not just receive a "completed" badge.
This is a design goal, not a claim that the current prototype already has a full code-editing, testing, and verification toolchain. In fact, an audit of the old execution chain found that intermediate steps wearing names like "retrieval," "tools," and "validation" had no corresponding real work behind them. To use this cockpit for deeper vibe coding, the roads must connect to real development tools and inspectable artifacts; otherwise, the more a demo resembles a real dev workflow, the more misleading it becomes.
Imagine the experience once that lands: you chose to build the page first; along the way, mobile overflow appears and the road surface warns "layout regression failed." You stop the car and see the failure screenshot and the file involved. The agent proposes two comparable roads: "fix the responsive grid" or "reduce card density." You don't need a lengthy new prompt restating all the context — you simply drive toward the direction that better matches the product's intent. This doesn't replace code review; it embeds review and choice into the journey.
06 | The hardest question: how do you prove the car is actually moving?
This is also the project's most public act of self-correction. Early versions iterated on candidate counts, preview fleets, immersion and depth, semantic bucketing. Then a user said: "It feels like nothing substantial is happening." That feedback shouldn't be dismissed as "3D doesn't work" — it pointed the developers at something more fundamental: does every step the cockpit displays correspond to a real change in the backend?
The audit of the old execution chain found intermediate steps updating progress with asyncio.sleep(0.09), then templating copy like "retrieved" and "validated" — while the actual model call happened only at the tail-end output. In the runtime records, the intermediate events ticked at a suspiciously regular ~90–100 ms cadence. For one specific recorded run, the project estimated that fake cadence accounted for roughly 86% of total duration; that figure applies only to that diagnostic run, not as a universal ratio across the product.
This lesson shouldn't steal the spotlight from the article's main idea. What it really shows is: the more convincingly a 3D drive makes people believe "we are making progress," the more strictly you must constrain when a visual movement may be called work progress. For vibe coding, a road segment must not be an animation timeline — it should be a chain of input, operation, result, and evidence. If no retrieval ran, don't light up "retrieval complete." If no tests ran, don't equate reaching the destination with "verification passed."
The project chose an honest subtraction first: v0.9.12 deleted the intermediate chain that did no real work and converged on a single real model request with token-streamed output, attributing exactly one item, output. On re-testing, the tidy fake-cadence fingerprint disappeared. Deleting extra road segments wasn't a failure of the driving idea — it cleared space for the real road segments of the future.
The GIFs are explanatory animations and the image is AI-generated concept art; neither is a runtime recording.
07 | Turning "done" into a fact you can question
A trustworthy agent cockpit must answer more than "where are we." It must answer four questions: Did this step really happen? Can the result be verified? Who did it, and at which step? Did the user perceive it in the UI? The project calls these P1 authenticity, P2 verifiability, P3 traceability, and P4 perceivability. They are working definitions, not a final formula stamped by human experiments.
| What an "arrival" should answer | What it could map to in vibe coding |
|---|---|
| What actually changed? | Files, code, pages, or configuration modified |
| How can the user verify it? | Diffs, live previews, test results, and failure messages |
| Whose result is this? | The user's chosen direction, the agent's execution steps, source labels |
| How is the arrival expressed? | A spatial anchor tied to the actual result with a clear status — not a universal green check |
That's why the prototype gradually gained provenance credentials, a per-round ledger, candidate comparison, a decision chain, and generation cancellation. These mechanisms are less eye-catching than a fleet of speeding neon cars, but they let you answer "what did I choose last round," "did this round truly re-run," and "why did a signpost appear when no model was involved." The steering wheel must not only turn — it must let the driver own the consequences.
Live screenshot ② — the result card after arriving at step two: genuine model text plus execution credentials (elapsed time, tokens, chain=output, source=llm).
Live screenshot ③ — the 2D timeline: both execution rounds are replayable, with "forecast = local preview / execution = llm" labeled separately, never conflated.
Live screenshot ④ — the decision chain lists every step as "asked → chose → got," corresponding to P3 traceability.
One distinction matters especially: "genuinely generated" is not the same as "reliable." A drafted piece of page copy is inspectable on its own merits; but if the model analyzes business data that was never provided, even genuinely emitted concrete numbers don't make the analysis complete. For missing data, model non-participation, or interrupted execution, the cockpit needs the corresponding states — not an unconditional drive into a glowing finish line.
08 | This isn't replacing chat — it's adding a spatial layer for complex collaboration
Not every "make the button blue" needs a three-dimensional highway. For simple direct answers, single-file tweaks, and quick Q&A, a chat box is probably lighter. Where 3D driving has a real chance to help is tasks with multiple meaningful routes, sequential decisions, and results worth comparing repeatedly.
Think of a product going from "idea" to "usable prototype": decide whether to validate the need or write the frontend first; choose low-fidelity or straight-to-interactive; after a build failure, decide to fix or roll back; compare the final experience across versions. The cognitive burden here isn't "I can't understand one reply" — it's "I can't see which decisions happened across the whole process." Putting choices, states, and history into space is what could add value over linear chat.
But "could" still needs measurement. The research has prepared protocols contrasting transparent vs. opaque execution, across task types such as drafting, analysis with attached data, and analysis with missing data; think-aloud sessions and controlled experiments with real participants remain to be run at the stage described in the project docs. So we cannot claim users have proven 3D beats chat, and "cooler" must not be mistaken for "more efficient." The right validation isn't just asking whether people like it — it's whether they find artifacts faster, attribute sources more accurately, and choose next steps more soundly.
09 | Finally, imagine a real "drive"
You open the project — not to a long, blurry chat log, but standing at the intersection where you left off. Behind you are the attempts completed and abandoned; ahead are a few routes with clear costs and goals. You pick one and the agent starts executing; the car's movement is driven by real events, not preset seconds. When a step fails, the road doesn't pretend to be clear. When a result is usable, the destination doesn't merely glow — it opens the real artifact.
This is the vision most worth discussing: turning interaction with agents — especially multi-round vibe coding — from "reading a string of answers" into "driving a workspace with directions, evidence, and memory." Self-audit isn't the theme of this project; it's the floor that keeps it from becoming a pretty toy.
The car can be cool. But every lit-up stretch of road should lead to something that actually happened — and that the user can verify.
Open-source repository
Every screenshot, GIF, and audit conclusion in this article comes from one public repository. You're welcome to check the code and docs directly:
GitHub: https://github.com/showkeyjar/agent-3d-cockpit-research
The repo contains the research documents (docs/), the runnable prototype (prototype/), and this article's visual assets (blog/assets/).
Authorship & evidence notes
This article is based on the project's README.md, docs/SUBDISCIPLINES.md, docs/EVIDENCE.md, docs/AUDIT.md, docs/RUNTIME-AUDIT.md, docs/RESEARCH-PLAN.md, and prototype/DESIGN.md. The vibe-coding vignettes (weather app, login page) are hypothetical scenarios illustrating the interaction concept; they do not imply the current prototype has a complete code-editing and testing loop. The explanations of the bidirectional broken link and users' sense of progress still require validation through human experiments.
The three static illustrations are AI-generated concept art; the 01/02/03-*.gif series are explanatory animations, not screen recordings. The real-* series shows the browser's actual rendering as UI demos (candidates injected by a local script, labeled "local fallback" in the UI). The hw-* series is a live run against a real backend: execution runs on a small local Ollama model (gemma3:1b-it-qat), with the execution chain being genuine streaming model output (the UI credentials show source=llm, tokens, and elapsed time); forecasting goes through the fast local channel and is honestly labeled "local preview." The live-run output is genuinely generated text, but a small model's output quality is limited — it demonstrates interaction and authenticity, not product-grade answer quality. No test-hook simulations were passed off as live runs. Please keep the relative directory structure of this Markdown file and the assets/ folder.
Originally published by Dev.to AI. Aggregated on AIWithGhost for educational purposes — full credit and traffic to the original publisher.
















