Building a Video AI Companion: What Happens After You Tap?

Tap a video companion while its next reaction is still downloading. What should happen?

That question reaches into more of the stack than it first appears: the generated footage, the action scheduler, the browser decoder and the cache all affect whether the tap feels answered.

I'm building Cariora, a product with AI companion rooms, including digital dogs and cats. The room combines conversation with prepared video reactions. This article focuses on the video side: how those separate clips become something a person can interact with.

A still from the public pet demo. The playable room uses filmed reactions; this image is its scene reference.

1. Give the clips somewhere to return

A collection of good clips can make a bad interactive sequence. One ends with the cat leaning forward; the next starts with it sitting upright. Both look fine alone. Switching between them makes the cat jump.

For ordinary room reactions, Cariora uses a shared keyframe. The generation request supplies that same image as the starting and ending frame:

room anchor -> reaction -> room anchor
room anchor -> glance   -> room anchor
room anchor -> rest     -> room anchor

This is a constraint on the footage, not a promise that every generated take will obey it. The pipeline checks the delivered frames and rejects unsuitable takes. Within a scene, keeping the video model consistent also helps avoid changes in texture and framing between reactions.

A second posture needs its own anchor and real transitions. Standing up, for example, needs a clip from the seated pose to the standing pose, a standing rest loop, and a way back. Moving a still image upward would leave the actual body motion missing.

The cost is that new postures require more footage. They also add paths the scheduler must understand. That is a real limit of this approach.

2. A quiet frame can still be the wrong cut

One mistake in the earlier cut-point logic was treating low motion as sufficient. A character holding a raised hand is almost motionless. Cutting from that frame to a seated idle still changes the pose abruptly.

For a return to the room anchor, a useful cut needs both low motion and a pose close to that anchor. Other transitions can use reviewed frame matches between specific clips. Larger movements need to finish at their permitted boundary.

Playback adds another failure mode. An earlier demo replaced the current video with a poster between actions. The poster and video had slightly different aspect ratios, so the crop changed during the handoff. It looked like the background shifted, even though the source clips shared an anchor.

The player has to retain the current picture while the next decoder gets ready. play() resolving does not prove that the new frame has reached the screen. Where supported, requestVideoFrameCallback provides a callback tied to a video frame being sent to the compositor. Cariora uses frame observations during decoder handoff, with bounded recovery when preparation fails.

This still leaves visible waits when a movement has no safe early exit. The material determines where a transition is possible.

3. Load a response before loading variety

The earlier opening order loaded idle, several expressive clips, and then the general touch reaction. A visitor could see a moving pet and reasonably assume it was ready to be touched. The clip needed to answer that touch was still waiting in the download queue.

The opening order now puts the general touch response immediately after idle, followed by other useful early responses. Entering a different room has its own order because its entrance footage must arrive in time for the scene transition.

The loader uses limited concurrency and can prioritize a requested clip. When measured transfer speed is low, it reduces concurrency so background downloads do not compete as heavily with the response being awaited.

There are three different facts to track:

A room being visible establishes none of those facts for an arbitrary action.

If the requested clip is still arriving, a tap gets an acknowledgement and a bounded wait. A compatible, already loaded touch clip can answer when the interaction rules allow it. If the wait expires, the room explains that it is still getting ready.

An action with no matching footage is unavailable. Its control and explanation must reflect that state. The same rule applies to character families: a human companion should not inherit a dog's play controls or sound bank because a shared component happened to start with dogs.

4. Warm the exact resource the player will read

Preloading only helps if playback can reuse the result. Cariora routes player media through one same-origin URL helper, including media shown before entering the room.

For a video object, the helper converts an object key into a player route:

object key:   uploads/cariora/ab/<content-hash>.mp4
player URL:  /api/pet-assets/uploads/cariora/ab/<content-hash>/mp4

The trailing extension segment avoids a local middleware issue where paths ending in .mp4 were treated as static files before reaching the handler.

Warm-up uses the same plain GET as the room and consumes the response body. The room loader fetches the clip bytes and gives decoders blob URLs, so decoder switches can reuse the local clip.

The shared URL does not make all media public. The edge worker checks an explicit index of public official-catalogue assets. Those can use shared CDN caching. Other keys go to the application worker for owner authorization and use private browser caching.

Content hashes identify immutable bytes; they do not establish permission to read those bytes. The distinction matters particularly for custom companions made from uploaded photos. The public path also preserves byte-range behavior for browser video requests.

For the underlying HTTP rules, MDN's Cache-Control reference explains the difference between public and private caching.

5. Test the first tap with a cold cache

A useful browser check starts with an empty cache and throttled network. Enter through the real adoption or companion-list flow, wait for the first idle to appear, then tap before the remaining clips finish loading.

Measure the interval to a presented response frame. Also record whether an acknowledgement appeared, whether the right reaction played, and whether a greeting or background action displaced the request.

These checks found bugs that isolated module tests missed. One used a zero-valued initial playback timestamp, which made never-played reactions look as though they were on cooldown immediately after opening the room. Another let arriving greeting footage replace a pending touch request.

Both could produce the same complaint: “I tapped and nothing happened.” Neither was fixed by changing the video prompt.

The current approach has limits. A long movement may need to finish before another can begin. Speech uses prepared motion rather than arbitrary phoneme-accurate animation. Sparse footage limits how many distinct responses a companion can show. Better scheduling helps use the footage already available; adding a missing movement still requires filming and reviewing it.

You can inspect the basic interaction in Cariora's public pet demo, which offers a small set of prepared reactions without signing in. For a similar project, that small loop is a useful first build: one anchor, an idle, one touch response, and a cold-cache test of the first tap.