Building Kensho: a voice companion for mental wellbeing that actually understands Bangla

Most AI mental-health tools are text chat in English. A lot of people who need support don't want to type at 2 a.m., and many of them don't think in English. I'm from Bangladesh, and I wanted something you could simply talk to, in Bangla or English or any other language you want, the way people actually speak.

That became Kensho: a voice companion for mental wellbeing. You open the site, press call, and talk. It remembers you between sessions, can run standard check-ins like PHQ-9 and GAD-7, and switches to a crisis-focused agent when things get serious.

This post covers the three hardest problems I hit:

1. Getting speech-to-text to understand Bangla at all
2. Giving the agent memory across sessions without that memory growing forever
3. Making "delete this session" actually remove it from what the AI remembers

Kensho is a wellbeing companion, not a therapist and not a replacement for professional care.

The stack

• LiveKit Agents (Python) on LiveKit Cloud: the real-time voice pipeline (WebRTC, turn detection, interruptions, agent handoffs)
• Deepgram Nova-3 for speech-to-text
• Cartesia Sonic for text-to-speech, including native Bangla voices
• An LLM behind an OpenAI-compatible endpoint, configured by environment variables so I can swap models without code changes
• Supabase: Postgres with row-level security, Auth, Edge Functions
• React + Vite + TypeScript frontend on Vercel

There are four agents the caller can be handed between: a therapist-style listener, a psychologist that runs structured screeners, a mindfulness guide, and a crisis agent. LiveKit's handoff model made this clean: each agent is its own class with its own instructions, tools and voice, and they share session state.

Problem 1: no provider could detect Bangla

Deepgram Nova-3 has a multi mode that auto-detects language and handles code-switching. Perfect, I thought. Then I spoke Bangla to it.

multi covers exactly ten languages, and Bangla isn't one of them. The model doesn't fail loudly. It forces the audio onto the nearest language it knows. On real Bangla speech, multi returned Devanagari nonsense at 0.73 confidence. The same audio with language="bn" returned the exact sentence at 0.994.

0.73 is high enough that nothing downstream notices. The LLM just gets garbage and answers it politely. From the caller's side, the agent is "deaf" to Bangla.

So I needed language detection. I tried everything I could find:

What worked: race the candidates

Deepgram's confidence turns out to be comparable across languages when the input audio is held constant. So the detector takes a few seconds of the caller's real speech, transcribes it in several candidate languages in parallel, and compares the scores.

Raw confidence picked the wrong answer on code-switched speech, though. On a Bangla-English clip, en scored 0.999, but it had transcribed only the 19-character English tail and thrown away the rest. bn scored 0.961 and captured all 60 characters. So I weight confidence by coverage:

score = confidence * min(1.0, len(transcript) / best_len)

Two guards sit on top of that. With a wide candidate set, some languages will transliterate anything at high confidence; Tamil scored 0.943 on that same Bangla clip, only 0.018 behind the right answer.

• A confidence floor drops candidates that returned nothing or near-zero confidence.
• A tie margin. Scores within the margin of the leader count as indistinguishable, and ties break by a fixed priority order (Bangla first, since this is a Bangladeshi service), optionally overridden by the language saved in the caller's profile.

I deliberately chose "priority breaks ties" over "refuse to switch when unsure". Refusing looks safe, but it silently leaves a Bangla speaker on multi, which is exactly the bug I was fixing.

The winner is handed to the STT at the next turn boundary. The whole thing runs off the reply's critical path and fails open: any error leaves the session on whatever language it already had. Nothing in the detector is allowed to raise into the voice pipeline. In a live call, a crash is much worse than a wrong guess.

A smaller gotcha: with Sonic and non-English languages, LiveKit's word-aligned transcript sync crashed. I turned off word timestamps for the multilingual voices.

Problem 2: memory that doesn't grow forever

A companion that forgets you every call isn't much of a companion. The first version did the obvious thing. After each call, an LLM rewrote one running summary:

summary_n = LLM(summary_{n-1}, transcript_n)

This has two problems. Each rewrite slowly loses detail, like a photocopy of a photocopy. Worse, you can't remove anything. Once a session has been blended into the summary, there's no way to take it back out.

The redesign splits memory into two layers:

1. One note per session. When a call ends, the agent writes a short clinical-style note about that session alone and stores it in a session_notes table.
2. A stateless aggregate. The "case formulation" the agent reads at the start of the next call is always rebuilt from the notes, never from the previous formulation.

formulation = LLM(note_1, note_2, ..., note_n)

For long-time callers, the notes are map-reduced: chunks of 60 notes are condensed in parallel, then combined into the final formulation. Cost grows slowly, and nothing is chained.

At the start of a call, if a formulation is missing, the agent doesn't make the caller wait for a rebuild. It uses the last few notes right away and kicks off the full rebuild in the background.

Problem 3: "delete" has to mean delete

For a mental-health service, being able to delete a conversation isn't a nice-to-have. And deleting the transcript isn't enough if the AI still "remembers" what you said in it.

With the two-layer memory, deletion becomes tractable. One Postgres function, delete_session_cascade, does it in a single transaction:

• checks that the session belongs to auth.uid()
• locks the caller's profile row
• deletes the session, its messages, its note and its screener results
• wipes every stored formulation version for that caller
• writes an audit row

Then the browser calls a Supabase Edge Function that rebuilds the formulation from the remaining notes. In my testing, that took about 20 seconds.

The race condition

There's a subtle bug waiting here. Suppose a rebuild starts (say, triggered at the end of a call), reads ten notes, and calls the LLM. While it's waiting, the user deletes session 4. The rebuild finishes and writes a formulation that still contains session 4.

The fix is a guarded write. Every rebuild records the exact set of note IDs it read. The write goes through a set_formulation_if_current RPC that locks the profile, compares that set against the notes that exist now, and refuses the write if they differ. The rebuilder then retries with fresh notes, up to three times.

The Edge Function and the Python agent both implement the rebuild, so their prompts must stay identical. The prompts live in Python, a script generates the TypeScript copy, and a test fails if the two drift.

Locking down the database

Early on, the browser could write to tables directly, guarded by row-level security. RLS stopped users touching other people's rows, but it didn't stop a user from forging their own records, for example inserting a fake screener result.

Now the client role has SELECT on its own rows, UPDATE on a whitelist of profile columns, and EXECUTE on the delete function. That's it. Every other write goes through the agent's server-side service role. Default privileges were changed too, so new tables start with no client access instead of inheriting it.

Privacy choices

• Call recording is off. The LiveKit session starts with record=False; the recording toggle is gone from the UI.
• Calls are encrypted in transit. WebRTC uses DTLS-SRTP for audio, and everything else is HTTPS.
• Transcripts and notes are per-account and deletable, session by session, including from the agent's memory.

I'm still working through the rest: vendor agreements, data-retention settings at every provider, tighter logging. Privacy for health data is a process, not a checkbox.

What I'd tell someone building a voice agent

• Test in the languages your users actually speak, with real audio. "Supports 50 languages" on a pricing page and "works for my callers" are different claims.
• Watch for confident garbage. The scariest failure was 0.73 confidence on nonsense. Nothing errored.
• Make background work fail open. In a live call, a wrong guess is recoverable; a crash isn't.
• Design memory for deletion from day one. Chained summaries are easy to build and impossible to un-build.
• Assume races. Anything that reads, thinks for 20 seconds, then writes needs a guard.

Try it

Kensho is live at kenshovoice.tech. I'd love feedback, especially from Bangla speakers and from anyone who has built voice agents for languages that big providers treat as second-class.

If you're in crisis, please contact a local emergency number or crisis line. Kensho is a companion, not an emergency service.