I Built an AI That Forgets on Purpose

Every conversation with an AI starts from zero. It doesn’t remember what you told it yesterday. It doesn’t connect facts across sessions. And it definitely can’t tell the difference between a casual "good morning" and a critical lab result that changes everything.

So I built a memory system where information has to earn its place.

Every message gets scored for importance and emotional weight the moment it arrives. If it proves useful later, its lifespan extends. A nightly job consolidates the most valuable insights into permanent summaries and structured facts. Everything else fades away.

The name Àtúnbí means "reborn" in Yoruba. Raw memory dies. Knowledge sticks around.

I built this for the Qwen Cloud Hackathon. To test it, I tracked two diabetic patients — Ada and Kai — across multiple sessions: doctor visits, medication adjustments, and changing lab results. But this isn’t just for healthcare. Swap patients for codebases, legal cases, research papers, or customer support tickets — the cognitive memory model works exactly the same.

The Architecture: Two Paths, One Brain

The system is split into two complementary execution paths: one optimized for fast, real-time responses, and another for deep, offline consolidation — balancing speed with long-term learning.

At the heart sit five distinct memory tiers, each with its own purpose and lifespan:

1. Working Memory: Raw incoming messages with dynamic, auto-adjusting expiration timestamps.
2. Episodic Memory: Structured conversation summaries generated during the nightly consolidation phase.
3. Semantic Memory: Standalone verified facts. When new information contradicts old, it never deletes — instead, it marks the previous entry as superseded, preserving a full audit trail.
4. Entity Memory: A NetworkX graph tracking relationships between people, places, things, and concepts.
5. Procedural Memory: Learned patterns, preferences, and workflows that crystallize over repeated use.

The Model Router: Why One Giant LLM Isn’t Enough

I use seven specialized Qwen models. The biggest lesson here: routing is everything. The vision model hallucinates numbers. The text model can’t see pixels. Use the wrong tool for the job, and you’ll get burned.

• qwen3-asr-flash: Audio transcription for voice notes and recorded meetings — handled medical terminology far better than I expected.
• qwen3.5-omni-flash: Vision and image/video description — great for high-level context, but never ask it for exact values.
• text-embedding-v4: Generates 1536-dimensional vectors for semantic search, supporting up to 8192 tokens — rock solid.
• qwen-flash: Real-time classifier — scores every incoming message on importance (0–1) and emotional valence (-1 to +1). Fast, lightweight, reliable.
• qwen-plus-latest: The workhorse — entity extraction, structured data parsing from PDFs, and final response generation. Handles JSON and graph construction beautifully.
• qwen-turbo: Binary cross-encoder reranker — for every candidate memory, it answers one question: Is this relevant? Yes or no. It’s not a ranker; it’s a sharp filter.
• qwen-max: Heavy-lifter for deep summarization during the nightly "Dream Phase". Highly accurate, but expensive — so used sparingly.

The Bug That Took Three Days (And What I Learned)

Most of my debugging time went here — three silent failures, each masking the next.

The Demo: A doctor uploads Ada’s blood work PDF and asks: "What changed?" The file clearly shows her HbA1c is 8.2% — her condition is worsening.
The Response: The system confidently replies: "Ada’s HbA1c is 7.1%."
When asked again, it doubled down.

Problem 1: Vision Hallucination

Originally, I rendered PDF pages as PNGs and sent them to the vision model. It would confidently invent plausible-sounding numbers that were completely wrong — 8.2% became 7.1%, eAG 183 became 154.

The Fix: Switch to extracting raw text from PDFs with pymupdf first, then send that text to a text-only model for structured extraction. Text models don’t make up numbers from pixels.

# Before: render PDF → send to vision model
pages = _render_pdf_pages(file_content)
# Model hallucinates exact values

# After: extract raw text → send to text model
pdf_text = _extract_pdf_text(file_content)
# Exact values extracted reliably

Problem 2: The Poisoned Cache

Fixed — or so I thought. The system still said 7.1%.

Why? That first hallucinated answer had been stored as a "fact" in Working Memory. Every new query pulled that cached mistake instead of reprocessing the file. I had to manually clean it out:

DELETE FROM workingmemory WHERE message ILIKE '%HbA1c%7.1%';

Problem 3: Silent Truncation

Fixed again. Still wrong.

Finally I checked the database directly: the model was outputting the correct values — but they were never saved. My ingestion pipeline silently cut every chunk at 500 characters. The lab report table was 800+ characters long; the HbA1c row sat right at position 600, getting sliced off before it hit the disk.

# The culprit — buried in ingest_text()
message=f"[{source_label}]: {para[:500]}", # Silent data loss

# Now: full documents stored whole; chat messages still chunked
if source_type in ["pdf", "document"]:
    message=f"[{source_label}]: {para}"
else:
    message=f"[{source_label}]: {para[:500]}"

The Takeaway: Never trust just the chat response — check the source, check the model output, then check the database.

Deployment: Keep It Boring (In the Best Way)

Everything runs on Alibaba Cloud, kept simple and maintainable:

• Compute: Single Docker container with FastAPI + Nginx, deployed via GitHub Actions
• Database: PostgreSQL + pgvector on ApsaraDB RDS
• Storage: Raw files in Alibaba OSS
• Scheduling: EventBridge triggers a nightly cron → Function Compute runs the Dream Phase, then scales to zero until next run

One docker run command with environment variables wires it all together; credentials live safely in GitHub Secrets. No Kubernetes overhead, no over-engineering — just what works.

Hard-Won Lessons (So You Save Time)

1. Always check your dependencies. pymupdf wasn’t in requirements.txt — the container crashed on deploy, and it took an hour to spot why.
2. Vision models describe; text models transcribe. If you need exact numbers, dates, or measurements, extract raw text first.
3. One size does not fit all. Chat messages and lab reports are fundamentally different — don’t force the same chunk size on everything.
4. Test your background jobs. I polished the real-time pipeline perfectly, but barely tested the nightly cron. Always verify the parts that run when you’re asleep.

Àtúnbí. Reborn.

Built for the Qwen Cloud Hackathon. View the full code on GitHub.

How do you approach long-term memory in LLM apps? What’s the biggest challenge you’ve faced? Let’s talk in the comments!