Our hospital chatbot cheated at its own exams — on purpose

When a hospital rolls out an HIS like InterSystems TrakCare, the knowledge lives in three places: classroom sessions, a Moodle full of training videos, and a pile of assessments everyone must pass. Six months later that knowledge is rotting in recordings nobody replays and PDFs nobody opens.

We wanted the opposite: ask a chat icon, get the right answer — in English or Arabic. So we built one. And to train it, we did something slightly ridiculous.

Try the live demo → — training environment only, no PHI, runs on free open models and may time out under load.

The exam heist

The full story (with the confession) lives on our blog. The engineering core:

1. A robot drives a real, logged-in Chrome tab by speaking Chrome DevTools Protocol, and clicks through the Moodle assessments by itself. All 36 of them — nobody's thumb has time for that by hand.
2. The loophole: each assessment's "Print to PDF" of a perfect attempt contains the correct answers verbatim. Complete the exam once, and the write-up is the answer key.
3. 36 PDFs parsed down to QA_BANK.md: 433 verified question–answer pairs. A self-solving dataset — the act of cheating generated the ground truth.

43 videos in, searchable knowledge out

The other half of the treasure was video. A few network-tab requests turned Moodle's BrightCove embeds into direct MP4s: 38 short how-tos (register a patient, overbook an appointment, move a patient on the floor plan) plus five hour-long live trainer recordings. A faster-whisper script — small model, int8 quantization, CPU-only — burned through all of them, produced timestamped JSON segments, and we distilled those into eleven topic files. My CPU core filed a formal complaint and kept going.

The retrieval engine that refuses to bluff

Most people's "RAG pipeline" means embeddings, a vector store, and a monthly invoice. Ours means BM25 — the old information-retrieval workhorse, reimplemented by hand — because the whole corpus is ~1,100 chunks. It fits in a process, it's instant, and we can give it opinions:

• Source-trust tiers. A Q/A pair stamped "reviewed and correct" from a finished assessment outranks any stray transcript line. Official workbooks get a 1.2× boost, video distillations 1.1×, and the glossary is gospel at 1.15×.
• Arabic. The deploying hospitals are largely Arabic-speaking, and Arabic doesn't stem with English rules. So the bot normalizes (alef variants → ا, ى → ي, ة → ه), strips the definite article (المواعيد → مواعيد), and consults a 100+ entry clinical synonym map (موعد ⇒ appointment, طوارئ ⇒ emergency, صيدلية ⇒ pharmacy, كاشير ⇒ cashier).

Ask in Arabic, retrieve in English, answer in Arabic.

Serving it

A small Node server on an unguessable port behind a Cloudflare tunnel, a soft-touch LLM proxy (temperature 0.3), and four bilingual hint chips with inline stills for the common questions. The whole harness — CDP drivers, answer-key JSON, whisper pipeline, BM25 core, FAQ layer, test battery — is maybe two thousand lines of actual Node. No vector database. No GPU. No SaaS.

• Live demo: https://tk.aidentech.com/
• Full story with screenshots: Chatbot aces TrakCare TOT by cheating at its own exams!
• More from the blog: https://aidentech.com/blog/

If you're rolling out TrakCare — or any HIS — and dread the training phase, talk to us before your next rollout. The 433 question–answer pairs are already written.