Building a multi-pane workspace for coding agents: PTYs, replay, and the backpressure we have not solved

SpaceApp is a self-hosted workspace that runs AI coding CLIs side by side in the browser, each in its own pane. I build it, it's MIT-licensed, and this post is the terminal side of it as it stands in the public release (v1.1.9): how a pane gets a real PTY, how output gets from that PTY to xterm.js and back after a reconnect, what broke on the way, and what we still haven't solved. Every number below comes from the code; file names are in the public repo so you can check them.

Two containers, one socket between them

A Docker install made with npx --yes run-spaceapp@latest install runs a small compose project. Two services matter here:

• spaceapp-core runs the web app, the API and the WebSocket. It listens on 127.0.0.1:4911 by default and mounts no provider keys or login volume.
• spaceapp-cli owns every PTY. It runs the pane host (packages/cli-host) as an unprivileged user (uid 10001).

They talk over a Unix socket on a shared volume (/run/spaceapp-cli/host.sock, mode 0660). PostgreSQL holds the rest, and a browser container is optional.

The split pays off on the first restart: restarting the core doesn't kill your CLIs, because they are processes of another container.

One PTY per pane

Each pane is one node-pty process (packages/cli-host/src/node-pty-spawn.ts):

export const spawnNodePty: CliHostSpawn = (spec) =>
  spawn("/bin/sh", ["-c", '/usr/bin/stty -isig; exec "$@"', "space-cli-host", spec.command, ...spec.args], {
    name: "xterm-256color", cols: spec.cols, rows: spec.rows, cwd: spec.cwd, env: spec.env });

That stty -isig is there because of a bug. Without it, Ctrl-C in a pane became SIGINT and killed the whole CLI process tree. With signals off on the PTY, Ctrl-C reaches the CLI as the byte 0x03, which is the CLI's own cancel key.

The rest of the lifecycle lives in session-registry.ts:

• Sessions are keyed by a CLI session id. Each spawn gets a generation id and a monotonically increasing output sequence number, which is what makes replay possible later.
• kill() in node-pty only sends SIGHUP and swallows errors, and we had process trees that survived it. The registry now escalates: SIGHUP, then SIGTERM, then SIGKILL on the whole process group (the PTY child is a session leader, so kill(-pid, SIGKILL) reaches everything under it).
• Some exit events never arrive. A sweep checks process.kill(pid, 0) and reconciles them.
• A session with no attachment is reaped after 30 minutes by default (swept every 60 s). When you come back, the pane host recreates it and resumes. So, honestly: an agent does not keep running forever after the last browser tab on it closes. Warm rooms, below, change when that happens.

The pane host protocol

The API and the pane host speak length-prefixed JSON over the socket (framing.ts): a 4-byte big-endian length, then UTF-8 JSON, 8 MiB at most per frame. The methods are inspect, attach, input, resize, detach, terminate, reapDetached and ping, and requests on one connection run strictly in order through a promise queue.

Two details turned out to matter.

Replay from a ring buffer. Output is chunked to 8,000 characters and kept per session in an 8 MiB ring buffer. An attach says "everything after sequence N". If those events already left the ring, the host says so instead of pretending:

if (earliestSequence !== undefined && afterSequence < earliestSequence - 1) {
  throw new CliHostError("CLI_HOST_REPLAY_GAP", `CLI host output before sequence ${earliestSequence} is no longer buffered …`);
}

Lanes. Strict ordering per connection means one slow session could delay every other session on that connection. So the API opens 32 connections to the host and pins each session to one with an FNV-1a hash of its id. One session's queue never sits in front of another's.

The API: two paths for the same output

When output arrives from the host, the API does two things with it (apps/api/src/cli-terminal.ts):

1. It sends it straight to the connected browsers.
2. Separately, it batches it for PostgreSQL, every 20 ms or 32 KiB, into a transcript table. Before anything is stored, it goes through pattern redaction (API keys of the common shapes, Bearer and Basic headers, session cookies).

So there are two stores: the host's ring buffer, which is fast and gone if the host restarts, and the transcript, which survives. A reconnect replays the stored transcript and yields to the event loop every 16 frames, so a long replay doesn't freeze the API for everyone else. Redaction applies to what is stored; the live stream your own browser receives is the raw output.

The WebSocket

Each pane's terminal is one WebSocket at /api/panes/:id/cli/terminal:

• It opens with a one-time ticket (32 random bytes, base64url, with a TTL).
• Every frame is a JSON text frame, validated with zod on both sides. Clients send input (up to 1,000,000 characters), resize (2-400 columns, 2-200 rows), interrupt, ping and control messages. The server sends output, status, ready, error, pong and session_replaced.
• Output is split into frames of at most 8 KiB, cut at code-point boundaries so a split never breaks a multi-byte character.
• permessage-deflate is on above 1 KiB, and maxPayload is 16 MiB.

Several tabs can watch the same pane, but only one can type: the holder of a control lease. The lease lasts 30 s. We first renewed it at 30 s, which raced the TTL and dropped keystrokes in a live session (20 September). The heartbeat is now capped at 15 s.

xterm.js in the browser

Each pane is one @xterm/xterm 6 terminal with 2,000 lines of scrollback. Two settings come with stories:

• The terminal starts at the PTY's stored size, not xterm's default, so a replay isn't reflowed through the wrong geometry first.
• convertEol stays off. The PTY's termios already handles line endings, and turning it on corrupted the cursor positioning of full-screen TUIs.

Writes are paced in the browser. Only one terminal.write(chunk, callback) is in flight at a time, and the next chunk goes after xterm's callback. Chunks are at most 1 MiB, with a yield between them.

Hidden and minimized panes don't parse anything. Their output is queued in JavaScript until the pane becomes visible. Past 128 KiB for one pane, or 1 MiB in total, the app records a pressure diagnostic. The queue itself isn't dropped at those sizes, so it is a warning, not a cap.

Then there are warm rooms. Up to 6 recently used rooms stay mounted with live sockets, up to 96 connected panes (24 on mobile). We used to park hidden sockets after 30 s to save resources. Every return to a room then needed a fresh ticket, an xterm rebuild and a full replay. Parking is off now, and leaving a room no longer rebuilds its terminals. A room that falls out of the six disposes its terminals and every addon properly.

One more fix that you only find by running real CLIs: OpenCode's TUI renderer sometimes drops the ESC[ introducer between colour runs, and xterm then prints the raw parameters as text. A small repair parser in the output coordinator puts the introducer back.

What we haven't solved: backpressure

This is the part I'd most like opinions on. There is no end-to-end backpressure:

• The server doesn't check bufferedAmount and doesn't pause the PTY when a browser is slow. It just sends.
• On the host socket, socket.write is called without checking its return value.
• What we do instead: 8 KiB frames, one xterm write in flight, 1 MiB chunks with yields, hidden panes that queue instead of parsing, and a replay that yields every 16 frames.

That holds up for normal agent output. A CLI that dumps a huge log while a browser on a slow link is attached is exactly the case it doesn't cover. Pausing the PTY when a socket's buffer grows would be the honest fix; the open question is how it should behave when several tabs watch the same pane at different speeds.

Isolation, precisely

People ask whether panes are sandboxed. They are not, and the README says so: rooms organize sessions; they are not a security boundary. Precisely:

• Core, CLI, browser and PostgreSQL are separate containers on one compose network. All run with no-new-privileges, and none gets the Docker socket.
• Inside the CLI container, every pane is a process of the same user. Each provider gets its own HOME, TMPDIR and config root with umask 077. That separates configuration, not security: one CLI could read another CLI's files.
• Host folders are mounted only after you add them as workspaces (run-spaceapp workspace add). An opt-in host-root mode on Linux mounts / read-write into the CLI container. It does what it says, so read the docs before you turn it on.

Caps and sizes

• 16 panes per room, enforced by a database check (pane_cap BETWEEN 1 AND 16) and the schema.
• 6 rooms by default, configurable up to 16.
• All panes share one CLI container with a memory and CPU cap: 2 GB / 1.5 CPUs on the small profile, up to 4 GB / 3 CPUs on the large one. The host should have at least 4 CPUs and about 8 GB of RAM. It isn't lightweight, and I'd rather say that up front.

Things that broke, in one list

Each of these is written in a comment next to its fix in the public code:

1. Two OpenCode 2 --standalone servers on one store failed to start. Now there is one shared opencode serve per CLI container, with a lock and a health check, and panes attach to it.
2. OpenCode's update dialog opened over a fresh pane and swallowed the user's first prompt. Auto-update is off inside the container.
3. Ctrl-C killed the whole CLI tree. Fixed with stty -isig.
4. node-pty kill() left process trees alive. Fixed with process-group SIGKILL escalation.
5. Parking hidden sockets forced a rebuild and a replay on every room switch. Parking is off.
6. Renewing the control lease at its TTL dropped keystrokes. The heartbeat is 15 s.
7. A TUI renderer dropped CSI introducers. A repair parser restores them.

Try it, or tell me how you'd do backpressure

npx --yes run-spaceapp@latest install

Node.js 20.11 or newer; Linux, macOS or Windows 11. The code is at https://github.com/oll4com/spaceapp and the paths above are where to start reading.

If you've built a multi-viewer terminal over WebSockets: how do you pause a PTY when one watcher is slow and another isn't?