Wiring Android's MediaPipe LLM Inference API to a Multi-Turn Conversation Engine
--- title: "Android MediaPipe LLM: Session State, KV-Cache, and the Context Budget That Keeps Latency Flat" published: true description: "Deep dive into MediaPipe LLM session management, KV-cache reuse, and context budge
---
title: "Android MediaPipe LLM: Session State, KV-Cache, and the Context Budget That Keeps Latency Flat"
published: true
description: "Deep dive into MediaPipe LLM session management, KV-cache reuse, and context budgeting to keep P99 latency flat on mid-range Android devices."
tags: android, kotlin, mobile, architecture
canonical_url: https://mvpfactory.co/blog/android-mediapipe-llm-session-state-kvcache
---
## What You Will Build
By the end of this tutorial you will have a multi-turn `ConversationEngine` backed by MediaPipe's `LlmInference` API that holds its session open across turns, enforces a token budget with a sliding window, and handles LoRA adapter switching without destroying your latency profile.
Building a single-turn prompt wrapper is easy. Building one that stays responsive across 20 exchanges on a Pixel 6a — that is what we are here for.
---
## Prerequisites
- Android project targeting API 26+
- MediaPipe Tasks dependency added to your `build.gradle`
- A quantised on-device model (Gemma 2B INT4 is the reference throughout)
- Basic familiarity with Kotlin coroutines
---
## The Core Problem: Session Lifecycle
Here is the gotcha that will save you hours.
Most teams treat `LlmInference` like a stateless HTTP endpoint and re-create it on every turn. That nukes your KV-cache on every call and forces the model to re-process the entire conversation prefix from scratch — every single time.
The `LlmInference` API maintains an internal KV-cache storing key-value attention pairs for all previously processed tokens. MediaPipe does not expose a stateful "add turn" API — you pass the full accumulated prompt on each call. But as long as you hold the session open, the runtime uses prefix cache matching: tokens the model has already seen at the same position are served from cache. Only net-new tokens at the tail require full attention computation.
**You pass full history. The session only computes the delta.**
kotlin
// WRONG: session torn down each turn — full re-inference on every call
fun respondToUser(context: Context, options: LlmInference.Options, input: String): String {
val session = LlmInference.createFromOptions(context, options)
return session.generateResponse(input).also { session.close() }
}
// CORRECT: session held open across turns
class ConversationEngine(context: Context, options: LlmInference.Options) {
private val session: LlmInference = LlmInference.createFromOptions(context, options)
private val history = mutableListOf()
fun respond(userInput: String): String {
history.add("User: $userInput")
val prompt = history.joinToString("\n")
val response = session.generateResponse(prompt)
history.add("Model: $response")
return response
}
fun close() = session.close()
}
---
## Context Window Budget: The Latency Cliff
Context growth is not a linear problem — it is quadratic in attention cost. These are illustrative order-of-magnitude estimates based on profiling patterns for a Gemma 2B INT4 model on a Snapdragon 778G versus a Snapdragon 8 Gen 2:
| Context Tokens | TTFT (Flagship) | TTFT (Mid-range) | P99 Risk |
|---|---|---|---|
| 256 | ~180ms | ~320ms | Low |
| 512 | ~260ms | ~610ms | Moderate |
| 1024 | ~440ms | ~1,400ms | High |
| 2048 | ~850ms | ~3,200ms+ | Budget exceeded |
If you target devices outside the flagship tier, your effective usable context budget is roughly **512–768 tokens** before P99 crosses the 1-second user-perception threshold.
Budget by tokens, not turns. Use `sizeInTokens()`. Here is the minimal setup to get this working:
kotlin
fun buildPromptWithBudget(
session: LlmInference,
history: List,
budget: Int = 600
): String {
val builder = ArrayDeque(history)
fun tokenCount(text: String) = try {
session.sizeInTokens(text)
} catch (_: UnsupportedOperationException) {
text.length / 4
}
var tokenEstimate = tokenCount(builder.joinToString("\n"))
while (tokenEstimate > budget && builder.size > 2) {
tokenEstimate -= tokenCount(builder.removeFirst())
}
return builder.joinToString("\n")
}
Keep your system prompt pinned outside the sliding window and account for its token cost separately. The 4-chars-per-token heuristic is a reasonable fallback for Latin-script prose but breaks down for code-heavy or multilingual content.
---
## LoRA Switching Is a Session Reset
Let me show you a pattern I use in every project that handles LoRA adapters.
Switching adapters always invalidates the KV-cache — there is no partial cache preservation across adapter boundaries in the current MediaPipe contract. Design your conversation flow to group turns by mode before transitioning:
kotlin
sealed class ConversationMode {
object General : ConversationMode()
object CodeAssist : ConversationMode()
}
class AdaptiveSession(private val context: Context, private val baseOptions: LlmInference.Options) {
private var currentMode: ConversationMode? = null
private var session: LlmInference? = null
fun ensureMode(mode: ConversationMode): LlmInference {
if (mode != currentMode) {
session?.close()
session = LlmInference.createFromOptions(context, baseOptions)
currentMode = mode
}
return session!!
}
}
Batch all turns of a given mode together. Scattered adapter switches throughout a session are a P99 latency time bomb.
---
## Gotchas
**KV-cache invalidation triggers to avoid:**
- Calling `close()` and re-creating the session between turns
- Swapping LoRA adapters mid-session without a full reset
- Exceeding the model's `maxTokens` window (the cache is fully evicted)
- Modifying the system prompt between turns
**The docs do not mention this, but** heap pressure from NNAPI delegate allocation is the silent killer of session stability. Use Android Studio's Memory Profiler with the **NNAPI heap** category enabled. If that heap climbs without bounds across turns, you have a session lifecycle leak — not a context window problem.
---
## Conclusion
Three things worth repeating:
1. **Hold your session open.** Session re-creation is the single highest-impact latency regression most teams ship without realising it.
2. **Budget by tokens, not turns.** Use `sizeInTokens()`, set a hard ceiling of 512–768 tokens for mid-range targets, and preserve your system prompt outside the sliding window.
3. **Treat LoRA switches as session resets.** Make them predictable and infrequent.
On-device LLM work involves long debugging sessions — I keep [HealthyDesk](https://play.google.com/store/apps/details?id=com.healthydesk) running in the background to catch myself before hour three without moving. Small thing, real difference.
For further reading: [MediaPipe LLM Inference API docs](https://ai.google.dev/edge/mediapipe/solutions/genai/llm_inference/android).
Originally published by Dev.to WebDev. Aggregated on AIWithGhost for educational purposes — full credit and traffic to the original publisher.