Virtual try-on with a one-image edit model: collage, edit in place, crop

A virtual try-on needs two pictures: a photo of the person and a photo of the garment. The image endpoint we run our standard model on accepts exactly one input image. We didn't want to change that endpoint, or the API contract our mobile app depends on, just to add a feature to the website.

The workaround turned out to be simple, and one detail in it made the difference between a feature that works most of the time and one that works every time we tried it.

Step 1: put both photos in one image, in the browser

The client builds a single image: the person on the left, the garment on the right on a plain light background, like a product shot. The pair is 1024 pixels tall, so the person keeps the same resolution as a normal single-photo edit.

The tricky part is the shape. The edit model only produces a fixed set of aspect ratios (1:1, 4:5, 3:4, 2:3, 9:16 and their landscape versions). If the collage doesn't match one of them exactly, the model pads or crops it, and the person ends up shifted or squeezed. So the browser picks the closest supported ratio, comparing in log scale so that 4:5 versus 5:4 is treated symmetrically, and sizes the collage to match it exactly:

const RATIOS: Array<[string, number]> = [
  ['1x1', 1], ['4x5', 4 / 5], ['5x4', 5 / 4], ['3x4', 3 / 4], ['4x3', 4 / 3],
  ['2x3', 2 / 3], ['3x2', 3 / 2], ['9x16', 9 / 16], ['16x9', 16 / 9],
]

function closestRatio(r: number): [string, number] {
  let best = RATIOS[0]
  for (const p of RATIOS) {
    if (Math.abs(Math.log(p[1] / r)) < Math.abs(Math.log(best[1] / r))) best = p
  }
  return best
}

The client also computes, in fractions of the collage, the box where the person sits. We'll need it at the end.

Step 2: ask the model to edit in place, not to return half

Our first instinct was to ask the model for just the result: "dress the person on the left in the garment on the right and return only the person". It sounds reasonable, and it failed often. In one round of six tests, the model zoomed the camera out in three and left the garment floating next to the person in one.

What worked was asking for less. The prompt now tells the model to dress the person on the left and leave the garment on the right exactly as it is, which means returning the whole collage edited in place. Same framing in, same framing out. In the next round, all nine tests kept the person's framing intact.

Models are much better at "change this region" than at "change this region and also recompose the picture". Every extra transformation you ask for is one more thing that can drift.

Step 3: crop on the server, before saving

The model returns the full collage, and nobody wants a picture of themselves next to a floating shirt. So the server keeps only the person, using the box the client sent:

import sharp from 'sharp'

interface Box { x: number; y: number; w: number; h: number } // fractions of the image

export function readBox(value: unknown): Box | null {
  if (typeof value !== 'string') return null
  const [x, y, w, h] = value.split(',').map((p) => Number(p.trim()))
  if (![x, y, w, h].every(Number.isFinite)) return null
  // A sane minimum: the person covers a reasonable part of the image.
  if (x < 0 || y < 0 || w < 0.15 || h < 0.3 || x + w > 1.0001 || y + h > 1.0001) return null
  return {x, y, w, h}
}

export async function cropTo(bytes: Buffer, box: Box): Promise<Buffer> {
  const img = sharp(bytes, {failOn: 'none'})
  const {width = 0, height = 0, format} = await img.metadata()
  if (!width || !height) return bytes
  const left = Math.max(0, Math.round(box.x * width))
  const top = Math.max(0, Math.round(box.y * height))
  const w = Math.min(width - left, Math.round(box.w * width))
  const h = Math.min(height - top, Math.round(box.h * height))
  if (w < 16 || h < 16) return bytes
  const out = img.extract({left, top, width: w, height: h})
  if (format === 'webp') return out.webp({quality: 80}).toBuffer()
  if (format === 'jpeg') return out.jpeg({quality: 90}).toBuffer()
  return out.png().toBuffer()
}

A few choices that matter here:

• The box is in fractions, not pixels. The model may return the collage at a different size than the client sent. As long as the aspect ratio is the one requested, fractions map correctly onto any output size.
• The box is validated, and invalid input means "don't crop". It comes from the client, so the server never trusts it blindly, and a box that is too small or out of bounds is ignored.
• A failed crop never fails the generation. If anything goes wrong, the server logs it and stores the full image. A slightly odd result is better than a charged generation with nothing to show for it.
• The crop is opt-in. Only the try-on variant sends a box. Every other generation, including all the mobile app's, goes through the same code path untouched.

What this bought us

• A two-image feature on a one-image endpoint, with no change to the image service or the app's API contract.
• Stable framing, because the model edits in place.
• The person's photo comes back at the same resolution as a normal edit.

There are limits. The result depends on the garment photo: a shop product shot or a flat lay is the easy case, while a garment worn by someone else in a busy scene gives the model more to untangle. We accept all three, because that's what people actually have on hand.

You can try it in the AI clothes changer on CreArt: upload a photo of yourself and a photo of the garment, and the collage, the edit and the crop all happen as described above.

Have you found other ways to feed multiple images to single-input models? I'd like to compare notes.