Dev.to AI 🤖 Ai Tue, 28 Jul 2026 16:10:09 +0000 👁 0 📖 8 min read

Browser-Based Face Detection for ID Photos + Multi-Provider AI Task Orchestration

The browser has become a surprisingly capable platform for computer vision. In this post, I'll walk through two production patterns — a browser-based ID photo maker with multi-tier face detection, and a provider-agnostic

Part 1: Building a Browser-Based ID Photo Maker

The Face Detection Problem

ID photos have strict requirements: head height ratios, eye-line positions, margin rules. Tools like HivisionIDPhotos, dpar39/ppp, and commercial services like dreamega.ai and photoaid.com all need to solve the same core problem: given an uploaded portrait, detect the face accurately enough to auto-crop to spec.

Server-side tools can afford heavyweight models (HivisionIDPhotos uses ONNX runtime with dedicated face parsing networks). But if you want to run everything client-side — no upload, no server cost, instant preview — the browser environment is unpredictable. Not every user has Chrome's experimental FaceDetector API, and WebGL support varies.

Our solution: a three-tier detection chain that tries the best option first and gracefully falls back.

// Detection chain: MediaPipe (478 landmarks) → Browser FaceDetector → BlazeFace
export async function detectFace(image: HTMLImageElement): Promise<FaceDetectResult> {
  if (typeof window !== "undefined" && !window.isSecureContext) {
    return { supported: false, face: null, reason: "insecure-context" };
  }

  // 1. MediaPipe Face Landmarker – best quality, 478 landmarks
  try {
    const mpResult = await detectWithMediaPipe(image);
    if (mpResult.face || mpResult.reason === "no-face-found") {
      return mpResult;
    }
  } catch { /* Fall through */ }

  // 2. Browser FaceDetector API (Chrome)
  const detector = await getDetector();
  if (detector) {
    try {
      const nativeResult = await detectWithNative(detector, image);
      if (nativeResult.face || nativeResult.reason === "no-face-found") {
        return nativeResult;
      }
    } catch { /* Fall through */ }
  }

  // 3. BlazeFace TensorFlow model
  try {
    const fallbackResult = await detectWithBlazeFace(image);
    if (fallbackResult.supported || fallbackResult.reason === "no-face-found") {
      return fallbackResult;
    }
  } catch { /* Fall through */ }

  return { supported: false, face: null, reason: "api-missing" };
}

Each tier returns a FaceBox with bounding coordinates and optional landmarks (eyes, forehead, chin). The key insight: each detector gives you different landmark richness, so the geometry estimation adapts.

Head Height Estimation: Three Formulas

Accurate head measurement is crucial — a few pixels off and the photo fails compliance. The approach here borrows from dpar39/ppp's CrownChinEstimator and extends it with a MediaPipe-first path:

export function estimateHeadHeight(face: FaceBox): number {
  // Best: MediaPipe forehead-to-chin + hair offset (1.35x multiplier)
  // HivisionIDPhotos avoids this problem by using alpha-channel contour
  // detection after background removal — a neat trick if you have a server.
  if (face.foreheadTop && face.chinBottom) {
    const chinY = Math.max(face.chinBottom.y, face.y + face.height);
    const foreheadToChin = chinY - face.foreheadTop.y;
    return foreheadToChin * 1.35;
  }

  // Good: ppp formula (chinCrownCoeff = 1.7699)
  if (face.leftEye && face.rightEye) {
    const ipd = Math.hypot(
      face.rightEye.x - face.leftEye.x,
      face.rightEye.y - face.leftEye.y
    );
    const eyeMidY = (face.leftEye.y + face.rightEye.y) / 2;
    const chinY = face.y + face.height;
    const frownToChin = chinY - eyeMidY;
    return 1.77 * (ipd + frownToChin);
  }

  // Fallback: anthropometric multiplier (Farkas LG reference)
  return face.height * 1.35;
}

The 1.35x multiplier accounts for hair volume above the forehead — MediaPipe landmarks don't detect hair, so this is intentionally an overestimate to prevent head cropping. The IPD formula comes from anthropometric research (Farkas LG, "Anthropometry of the Head and Face").

Segmentation-Refined Head Top

Landmarks estimate where the head top should be, but segmentation gives you pixel-accurate results. This is where our approach diverges from passport-photo-online and similar services that rely purely on face detection — combining both gives noticeably better results for people with voluminous hair:

export function refineWithSegmentation(
  geom: HeadGeometry,
  segBounds: { topY: number; bottomY: number } | null
): HeadGeometry {
  if (!segBounds) return geom;

  // Only use segmentation topY — bottomY detects body silhouette, not chin
  const refinedTopY = Math.min(segBounds.topY, geom.topY);

  return {
    ...geom,
    topY: refinedTopY,
    headHeight: geom.bottomY - refinedTopY,
  };
}

An important gotcha: segmentation bottomY is the body bottom (shoulders, torso), not the chin. Using it would over-extend the head measurement. We only take topY from segmentation (hair boundary) and keep the chin position from face landmarks.

The Auto-Fit Algorithm

With accurate head geometry, auto-cropping becomes a three-step process:

export function computeAutoFitCropState({ imageWidth, imageHeight, face, spec, canvasWidth, canvasHeight, segBounds }) {
  const rules = getNormalizedHeadRules(spec);
  const head = refineWithSegmentation(estimateHeadGeometry(face), segBounds);

  // 1. SCALE — head matches target height ratio
  const targetH = guide.headBottom - guide.headTop;
  const scale = clampedTarget / Math.max(1, head.headHeight);

  // 2. PLACE — two anchor strategies
  if (rules.hasEyeLineRule && hasEyes) {
    // Strategy A: eyes at target Y (used by ICAO/US passport specs)
    crop = cropFromAnchor(eyeImg, { x: cw/2, y: eyeTargetY }, ...);
  } else {
    // Strategy B: head top at target margin (common in Asian passport specs)
    crop = cropFromAnchor({ x: head.centerX, y: head.topY }, ...);
  }

  // 3. GUARD — prevent head from being cropped
  crop = guard(crop, head, face, ...);
  return crop;
}

The guard pass handles edge cases: crown too close to the top, chin below frame, or both overflowing (in which case it scales down to fit).

Background Removal with Color Decontamination

When replacing backgrounds, edge pixels bleed the original background color. We use @imgly/background-removal (ISNet model) — the same engine behind remove.bg's open-source alternative — then apply color decontamination:

// Decontaminate semi-transparent edge pixels
for (let i = 0; i < rPx.length; i += 4) {
  let a = rPx[i + 3] / 255;

  // Erode very low alpha pixels — eliminates faint halo
  if (a < 0.15) { rPx[i + 3] = 0; continue; }

  // Squeeze alpha: remap [0.15, 1] → [0, 1]
  a = Math.min(1, (a - 0.15) / 0.85);
  rPx[i + 3] = Math.round(a * 255);

  // Color decontamination for edge pixels
  if (a > 0.01 && a < 0.95) {
    const inv = 1 - a;
    rPx[i]     = clamp255((rPx[i]     - inv * bgR) / a);
    rPx[i + 1] = clamp255((rPx[i + 1] - inv * bgG) / a);
    rPx[i + 2] = clamp255((rPx[i + 2] - inv * bgB) / a);
  }
}

The math reverses the alpha compositing formula: if the composited color is C = α·F + (1-α)·B, then the true foreground is F = (C - (1-α)·B) / α. The alpha squeeze ([0.15, 1] → [0, 1]) acts as an erosion filter that kills the soft halo around hair.

Part 2: Task System Architecture for Multi-Provider AI

When you're routing requests across Fal, WaveSpeed, Volcengine, ChatFire, Google, OpenAI, and more, you need a system that:

Splits a single user request into provider-specific subtasks
Handles async (webhook) and sync execution modes
Calculates credits before execution and refunds on failure
Falls back to alternative providers on errors

The Task → SubTask Model

Every user request becomes a Task with one or more SubTasks. Each subtask targets a specific provider:

export async function createOrUpdateTask(params: CreateOrUpdateTaskParams): Promise<TaskExecutionData> {
  const credits = preCalculatedCredits ?? 
    (await calculateTaskCredits({ taskType, metadata, request, systemRequest })).totalCredits;

  // Generate subtasks based on task type
  const subTasksData = taskType === TaskType.Template
    ? await generateTemplateSubTasks({ ... })
    : await generateModelDirectInvocationSubTasks({ ... });

  // Atomic creation: task + subtasks in one transaction
  return await prisma.task.create({
    data: {
      taskType,
      status: TaskStatus.PENDING,
      executionMode: prismaExecutionMode,
      request, systemRequest, metadata,
      credits, actualCredits: credits,
      subTasks: { createMany: { data: subTasksWithMode } },
    },
    select: taskExecutionSelect,
  });
}

The draft pattern is notable — tasks can be created as drafts (for preview/confirmation) then atomically promoted to PENDING with a status guard that prevents double-submission.

Provider Routing with Registry Pattern

Instead of a giant if-else chain for subtask generation, we use a provider registry:

const subTaskGeneratorRegistry: Partial<Record<Provider, SubTaskGenerator>> = {
  [Provider.fal]: generateFalSubTasks,
  [Provider.kie_ai]: generateKieAiSubTasks,
  [Provider.openai_next]: generateOpenAINextSubTasks,
  [Provider.volcengine]: generateVolcengineSubTasks,
  [Provider.chatfire]: generateChatFireSubTasks,
  [Provider.wavespeed]: createWavespeedSubTasks,
  [Provider.meshy]: generateMeshySubTasks,
  [Provider.google]: generateGoogleSubTasks,
  // ... add new providers here
};

const generator = subTaskGeneratorRegistry[provider];
return await generator({ request, systemRequest, metadata, credits });

Each generator knows how to translate a unified request format into provider-specific API calls. Adding a new provider means writing one module and registering it — the same pattern Replicate uses for their model routing, scaled down to a single codebase.

Execution: Async vs Sync vs Streaming

The runner supports three modes. Async fires subtasks and waits for webhooks:

export async function runTaskByProvider(task: TaskExecutionData): Promise<void> {
  for (const subTask of task.subTasks) {
    const provider = await getTaskProvider({ metadata, taskType, request, systemRequest });

    try {
      const res = await runProviderTask(provider, { taskId, subTaskId, ... });
      if (res) await updateSubTaskResponseAndStatus({ subTaskId, response: { ...res, submittedAt: new Date().toISOString() } });
    } catch (error) {
      // Try fallback provider before marking as FAILED
      const fallbackCreated = await handleSubTaskFailureWithFallback({ taskId, subTaskId, error, task });
      if (!fallbackCreated) throw error;
    }
  }
}

The fallback mechanism is key — when a provider fails, the system can transparently create a new subtask targeting an alternative provider, so the user never sees the error.

Streaming mode uses SSE for text generation tasks:

const streamResult = await streamTextFromTemplate({ templateSlug, input, stream: true });
const { readable, writable } = new TransformStream();

(async () => {
  let fullText = "";
  for await (const chunk of streamResult.textStream) {
    fullText += chunk;
    await writer.write(encoder.encode(\`data: \${JSON.stringify({ text: chunk })}\\n\\n\`));
  }
  // Save final result to DB, confirm credit transaction
  await updateSubTaskResponseAndStatus({ subTaskId, status: TaskStatus.COMPLETED, response: { ...parsedResult } });
  if (creditTransactionId) await confirmCreditReverse({ creditTransactionId });
})();

return new Response(readable, { headers: { "Content-Type": "text/event-stream" } });

Credit System: Reserve → Confirm/Cancel

Credits follow a two-phase commit pattern:

Reserve credits when the task is created
Confirm the reservation on success (credits permanently deducted)
Cancel the reservation on failure (credits returned)

export function calculateRefundCredits(task: TaskWithDetailItem): number {
  const refundCredits = task.subTasks.reduce((total, subTask) => {
    if ([TaskStatus.FAILED, TaskStatus.CANCELLED, TaskStatus.ABORTED].includes(subTask.status)) {
      return total + (subTask.credits || 0);
    }
    // PENDING/PROCESSING subtasks refunded if parent task failed
    if ([TaskStatus.PENDING, TaskStatus.PROCESSING].includes(subTask.status) &&
        [TaskStatus.CANCELLED, TaskStatus.FAILED, TaskStatus.PARTIALLY_COMPLETED].includes(task.status)) {
      return total + (subTask.credits || 0);
    }
    return total;
  }, 0);

  // Cap refund to prevent over-refund from race conditions
  return Math.min(refundCredits, task.credits || 0);
}

The Math.min cap is a safety valve — when a fallback subtask is created but the original subtask's credits haven't been zeroed yet, naive summing would over-refund.

Key Takeaways

Progressive enhancement works for ML: The three-tier face detection chain means every user gets the best experience their browser supports, with no manual feature-flag management.
Separate detection from estimation: Face detection gives you coordinates; head geometry estimation interprets them. Keeping these separate lets you swap in better models without touching the layout logic.
The registry pattern scales: When you're integrating 10+ AI providers, a registry of typed generators beats a switch statement. Each provider is isolated and testable.
Two-phase credit commits prevent pain: In async systems with fallbacks, a simple debit-on-start model will either over-charge (failed tasks) or under-charge (race conditions). Reserve/confirm is worth the extra complexity.
Segmentation complements, doesn't replace landmarks: Use segmentation for what it's good at (hair boundaries, background separation) and landmarks for what they're good at (facial feature positions). Mixing their strengths produces better results than either alone.

Hope this helps if you're building similar browser-based CV tools or multi-provider orchestration systems. Happy to answer questions in the comments.

📰 Read the original article on Dev.to AI

Originally published by Dev.to AI. Aggregated on AIWithGhost for educational purposes — full credit and traffic to the original publisher.

← News 🤖 Ai 🏠 Home