Building an Android AI Agent, Part 15: Session Memory Without Storing Sensitive Data
Building an Android AI Agent, Part 15: Session Memory Without Storing Sensitive Data In Part 14 we handled context misfires — cases where AgentContext builds a confident but wrong model of what the user is doing. This
Building an Android AI Agent, Part 15: Session Memory Without Storing Sensitive Data
In Part 14 we handled context misfires — cases where AgentContext builds a confident but wrong model of what the user is doing.
This part covers a different problem: memory across sessions. Right now, every time the user opens StareBrain, the agent starts fresh. It knows what's on screen. It knows recent app switches. It doesn't know that this user always drafts emails before sending them, or that they use a specific task app to close out their day, or that "send the doc" always means the most recently edited file in Google Drive.
That kind of pattern is useful. It shouldn't require the agent to watch you fail and recover a dozen times before it learns it.
What's worth remembering
The constraint: StareBrain runs on-device and handles sensitive context — open apps, screen content, messages in progress. We don't want to persist any of that.
What's safe to persist is behavioral patterns at the intent level — not what the user did, but how they tend to structure tasks:
data class SessionPattern(
val intentType: String, // e.g. "send_message", "create_event", "open_file"
val preferredPackage: String?, // which app they use for this intent
val typicalStepCount: Int, // how many steps this usually takes
val confirmationHistory: ConfirmationHistory,
val lastSeenAt: Long
)
data class ConfirmationHistory(
val alwaysConfirms: Boolean, // user always confirms this action type
val neverConfirms: Boolean, // user consistently skips confirmation
val confirmCount: Int,
val skipCount: Int
)
No screen content. No message text. No contact names. Just the shape of what the user does and which apps they use to do it.
Writing patterns
Patterns are written at task completion, not during execution:
class SessionMemory(private val prefs: SharedPreferences) {
fun recordCompletedTask(task: AgentTask) {
if (task.status != TaskStatus.COMPLETED) return
val intentType = inferIntentType(task.description)
val existing = loadPattern(intentType)
val updated = SessionPattern(
intentType = intentType,
preferredPackage = task.steps.firstOrNull()?.targetPackage,
typicalStepCount = task.steps.size,
confirmationHistory = updateConfirmationHistory(
existing?.confirmationHistory,
task.steps
),
lastSeenAt = System.currentTimeMillis()
)
savePattern(intentType, updated)
}
private fun inferIntentType(description: String): String {
// Simple keyword bucketing — not LLM-based, runs on device
return when {
description.contains("send", ignoreCase = true) -> "send_message"
description.contains("schedule", ignoreCase = true) ||
description.contains("calendar", ignoreCase = true) -> "create_event"
description.contains("open", ignoreCase = true) -> "open_file"
description.contains("pay", ignoreCase = true) ||
description.contains("transfer", ignoreCase = true) -> "payment"
else -> "general"
}
}
private fun updateConfirmationHistory(
existing: ConfirmationHistory?,
steps: List<AgentStep>
): ConfirmationHistory {
val confirmedSteps = steps.count { it.result?.confirmedBy == ConfirmationSource.LLM_FLAGGED }
val skippedSteps = steps.count {
it.result?.confirmedBy == null && it.status == StepStatus.COMPLETED
}
return ConfirmationHistory(
alwaysConfirms = (existing?.confirmCount ?: 0) + confirmedSteps > 3,
neverConfirms = (existing?.skipCount ?: 0) + skippedSteps > 5,
confirmCount = (existing?.confirmCount ?: 0) + confirmedSteps,
skipCount = (existing?.skipCount ?: 0) + skippedSteps
)
}
}
Reading patterns into context
When AgentContext is built, known patterns are injected:
fun AgentContextBuilder.build(
pendingTask: AgentTask? = null,
lastUserIntent: String? = null,
memory: SessionMemory? = null
): AgentContext {
val intentHint = lastUserIntent?.let {
memory?.loadPattern(inferIntentType(it))
}
return AgentContext(
activePackage = activePackage,
previousPackage = switches.lastOrNull()?.fromPackage,
activeScreenSnapshot = activeSnapshot,
recentAppSwitches = switches.toList(),
pendingTask = pendingTask,
lastUserIntent = lastUserIntent,
confidenceLevel = inferConfidence(),
knownPattern = intentHint // new field
)
}
The knownPattern tells the LLM: "this user has done this kind of task before, they typically use this app, and they usually don't need confirmation for this action type." The LLM can weight its step generation accordingly.
Expiring stale patterns
Patterns older than 30 days are treated as stale and not injected into context. Apps get uninstalled, workflows change, what was true last month may not be true now:
fun SessionMemory.loadPattern(intentType: String): SessionPattern? {
val pattern = loadRaw(intentType) ?: return null
val thirtyDays = 30L * 24 * 60 * 60 * 1000
return if (System.currentTimeMillis() - pattern.lastSeenAt > thirtyDays) {
deletePattern(intentType)
null
} else pattern
}
What this doesn't do
This is not a full memory system. It doesn't remember specific tasks, specific contacts, specific files, or anything the user said. It remembers the shape of behavior, not the content.
A user who regularly sends morning updates to their team doesn't have their team's names stored. The agent learns "this user sends messages in the morning using this app and doesn't need confirmation for it." The specific message, the specific recipient — those come from context at execution time, not from memory.
The goal is a faster, less interrupted experience, not surveillance.
What's next
Part 16: the StareBrain overlay UI — how the agent surfaces its current state, pending confirmations, and task progress without taking over the screen.
Originally published by Dev.to AI. Aggregated on AIWithGhost for educational purposes — full credit and traffic to the original publisher.