I built an app that replaces the whole person in a video. Here's what was hard.

Most "face swap" results you've seen paste one face onto the same head. The hair, the body and the clothes stay, and it ends up looking like makeup.

I wanted the opposite: keep the performance from a clip (gestures, timing, camera) and rebuild the person completely, from one photo or from a sentence describing a character that doesn't exist. That became ActLike AI, an iPhone and Android app.

AI-generated example: same gestures and kitchen, a completely different person from one photo.

This post is about the parts that turned out to be harder than the "AI" itself.

The pipeline, roughly

1. Upload a short clip of one person (capped at 20 seconds) and either a photo or a text description.
2. Grab a reference frame from the start of the clip.
3. Build the character image. An image model takes the reference frame plus the photo (or the description) and produces the new person in the clip's exact pose, framing and light.
4. Replace the person in the video. A video model takes the clip and that character image and renders the clip with the new person, keeping the original motion.
5. Store the result and hand the phone a short-lived signed URL.

Steps 3 and 4 are where the quality comes from, but almost every bug I shipped lived somewhere else.

Hard part 1: the person looked pasted on

The first results were technically correct and visually wrong. A photo taken in soft studio light, dropped into a rainy street at night, looked like a sticker.

The fix was to stop feeding the user's photo straight into the video step. Instead, step 3 re-creates the new person inside the first frame of the clip: same pose, same head position, same lighting and shadows, with the face, hair and clothing from the photo. The video model then animates that image. In the app this is the "Scene Match" toggle.

Same photo, Scene Match off vs on: with it, the rain and neon reach her hair and clothes.

Two lessons from tuning it:

• Lock the prompt. Free-form prompts drifted: the background changed, the camera "zoomed", clothing got restyled. A fixed prompt that lists what must not change (background, camera, framing, depth of field) made results consistent.
• Big features transfer, small ones don't. Age, hair, beard and clothing come through reliably. Eye colour, teeth and scars often don't, because the video model follows the performer's own eyes and mouth. I now say this in the app instead of pretending otherwise.

Hard part 2: renders take minutes, and users close apps

A render takes a few minutes. People don't stare at a loading screen for that long; they switch apps, lock the phone, or swipe the app away. My first version kept the job alive only while the screen that started it existed, so a killed app meant a lost video that the user had already paid for.

What fixed it:

• The server owns the job. Each render is a job document with a status (queued → processing → completed/failed). The app only reads it.
• The device keeps a small ledger of every job it started. Not just the latest one. On launch, on returning to the foreground and on a notification tap, the app walks that list, asks the server for each job's status, and files any finished video in History.
• A notification tap carries the job id. Even if the device lost track of a job, the tap is enough to fetch the result and open it.
• Timeouts never delete work. A render that runs long is shown as "still preparing, it'll appear in History", and the ledger picks it up later.

The general rule I took away: any long-running job on mobile needs a recovery path that doesn't depend on a screen being alive.

Hard part 3: money and failures

The app uses one-time credit packs, no subscription: 1 credit per second at 480p, 2 per second at 720p, plus 1 if the character image has to be generated (Scene Match or a described character; they share that one credit).

The rule that saved me the most support email: the server is the only place that decides the price, and a failed render refunds its credits automatically. The client shows an estimate, but the charge, the refund and the reason all live in a server-side ledger. When I briefly let my website calculate costs differently from the server, it was wrong by one credit in exactly the combination I hadn't tested. Same formula, one source of truth.

Hard part 4: making examples honest

For the website I wanted every guide to show a real result, not a mock-up. So every example is made with the app's own pipeline and settings, the performers and photos are AI-generated people (no real faces), and each video is labelled with its settings and credit cost.

AI-generated example: a typed description turned a young maker into an old sea captain.

One unexpected design rule came out of this: the new person has to differ from the performer (gender or age plus hair and clothing). When I used a character too similar to the performer, viewers read it as makeup on the same face, which undersold what the model actually does.

What I'd tell someone building something similar

• Treat generation as a server job with a status, never as a request the phone waits on.
• Lock your prompts and list what must stay the same, not just what should change.
• Put pricing and refunds on the server, and test every combination of options.
• Be explicit about limits (one person per clip, the face must be visible, shaky footage glitches more). Users forgive limits; they don't forgive surprises.

If you want to see the results with their exact settings and costs, they're here: before-and-after examples. The app is on the App Store and Google Play.

Happy to answer questions about any of this in the comments.