Wiring Android's CameraX ML Kit Pipeline to a On-Device Vision-Language Model

---
title: "CameraX + On-Device VLM: Frame Sampling and the Inference Budget That Fits on a Pixel 9"
published: true
description: "Wire CameraX ImageAnalysis to a quantized vision-language model on Android. Frame drop strategy, YUV tensor conversion, NNAPI vs GPU delegate, and memory budgeting on Snapdragon 8 Gen 3."
tags: android, kotlin, mobile, architecture
canonical_url: https://mvpfactory.co/blog/camerax-on-device-vlm-frame-sampling-pixel9
---

## What We Are Building

A live-camera inference pipeline that wires CameraX `ImageAnalysis` to a quantized vision-language model (MobileVLM-class, int4) on a Pixel 9. By the end you will have a coherent pipeline contract — frame drop policy, zero-copy YUV-to-RGB preprocessing, delegate routing, and a memory budget that holds under sustained load.

This is not a getting-started guide. It is the guide for after you benchmarked the model in isolation, declared victory at 300ms per frame, wired it to a live feed, and watched the system collapse within 90 seconds.

---

## Prerequisites

- Android project targeting API 26+
- CameraX `1.3.x` added to your Gradle dependencies
- A quantized VLM loaded via TFLite (MobileVLM or equivalent int4/int8 model)
- Basic familiarity with `ImageAnalysis` use case setup

---

## Step 1 — Understand the Real Problem

The camera does not wait for your model. `ImageAnalysis` delivers frames at the cadence the sensor dictates — 30fps by default. At 300ms per inference, you have a 9× overproduction ratio before you account for KV-cache memory pressure. The fix is not faster inference. It is a coherent pipeline contract.

Let me show you a pattern I use in every project: treat the inference budget as a hard constraint *before* you write a single line of pipeline code.

---

## Step 2 — Frame Drop Policy Under Backpressure

`ImageAnalysis` gives you two backpressure strategies. Never use `STRATEGY_BLOCK_PRODUCER` with a slow model — you will starve the camera HAL. Always use `STRATEGY_KEEP_ONLY_LATEST`, then gate your sample rate explicitly with a token-bucket in the analyzer.

Target 2–4fps for most VLM tasks. Scene understanding does not require 30fps temporal resolution.

---

## Step 3 — Zero-Copy YUV-to-RGB Conversion

Here is the gotcha that will save you hours. CameraX delivers `ImageProxy` frames in `YUV_420_888`. Your VLM expects a normalized RGB tensor. The naive CPU path — convert, write to `ByteBuffer`, pass to TFLite — involves two full-frame copies and a format-conversion pass that burns ~8ms on a Snapdragon 8 Gen 3 for a 640×480 input.

The correct path uses a RenderScript or Vulkan compute shader to perform the conversion directly in GPU memory, then hands the texture pointer to the GPU delegate without a CPU round-trip:

This alone reduces preprocessing latency by roughly 60% on sustained throughput workloads. Keep tensors on GPU from acquisition through inference — the CPU round-trip is the hidden cost.

---

## Step 4 — NNAPI vs GPU Delegate Routing

The docs do not make this decision obvious, but here is the breakdown for Snapdragon 8 Gen 3 at 640×480 int8:

| Delegate | Latency | Power | KV-cache | Cold Start |
|---|---|---|---|---|
| CPU (4 big cores) | ~420ms | High | Full | ~0ms |
| NNAPI (DSP path) | ~180ms | Low | Partial | ~220ms |
| GPU delegate | ~140ms | Medium | Full | ~350ms |
| NNAPI (GPU path) | ~155ms | Medium | Partial | ~300ms |

For sustained live-camera inference, the GPU delegate wins on latency with acceptable thermal behavior. NNAPI DSP is better for battery-sensitive background tasks, but its partial KV-cache support means you will manage context windowing manually for multi-turn queries.

---

## Step 5 — Budget Memory Before You Benchmark Latency

Here is the minimal setup constraint to get this working on a Pixel 9. A quantized 3B-parameter VLM (int4, MobileVLM-class) requires approximately 1.6GB of model weights plus 300–500MB of KV-cache at a 512-token context window. Add ~200MB peak for the GPU delegate's internal working memory. Budget 2.2GB total for the pipeline.

CameraX camera buffers at 640×480 `YUV_420_888` are ~460KB each — negligible. Upsample to 1080p with 4 buffers in-flight and you are at ~12MB camera-side, which still fits. The Pixel 9's 12GB helps, but on 8GB devices this hard-caps you at int4 quantization with a reduced context window.

Profile under load, not idle:

Run that under sustained 30-second camera inference before committing to a model size.

---

## Gotchas

**`STRATEGY_BLOCK_PRODUCER` will silently stall your pipeline.** There is no exception — the camera HAL just falls behind and frames queue. Use `KEEP_ONLY_LATEST` unconditionally with a slow model.

**Benchmarking in isolation is a lie.** A 300ms single-shot inference becomes a 600ms+ sustained-load inference once KV-cache pressure and thermal throttling kick in after 30–60 seconds of live use.

**Cold start latency on the GPU delegate is 350ms.** Initialize it once at startup, not per-frame. Lazy initialization on the first analyzed frame will drop that frame and spike your first-inference latency.

**YUV conversion is not free.** Even at 640×480, skipping the hardware buffer path costs you 8ms per frame and blocks a CPU core. At 4fps sampling that is tolerable. At anything higher, you will see it in your profiler traces.

---

## Conclusion

The pipeline contract is: `KEEP_ONLY_LATEST` + token-bucket sampler at 2–4fps + hardware buffer YUV path + GPU delegate initialized at startup + 2.2GB memory budget verified under load. Miss any one of these and you will hit a wall that looks like a model performance problem but is actually a systems integration problem.

**Further reading:**
- [CameraX ImageAnalysis reference](https://developer.android.com/training/camerax/analyze)
- [TFLite GPU delegate guide](https://www.tensorflow.org/lite/performance/gpu)
- [MobileVLM model card](https://huggingface.co/mtgv/MobileVLM-3B)