Building an AI Interviewer That Teaches Instead of Just Evaluating
Most interview preparation tools focus on answering more questions. But I wanted to explore a different question: What happens when an interview preparation tool focuses on helping users understand their mistakes? That
Most interview preparation tools focus on answering more questions. But I wanted to explore a different question:
What happens when an interview preparation tool focuses on helping users understand their mistakes?
That idea led me to build N.O.V.A., an AI-based technical interview preparation platform.
The learning loop
Its learning workflow follows six stages:
- Ask a technical question appropriate to the candidate's level.
- Evaluate the response for accuracy, clarity, completeness, and reasoning.
- Ask adaptive follow-up questions.
- Explain the underlying concept when the candidate gets stuck.
- Track weaknesses and misconceptions.
- Use that performance history to build a personalized preparation plan.
One feature I'm particularly interested in is the "I don't know" mode.
It's designed to turn uncertainty into a learning opportunity. The system explains the concept and then gives the candidate another opportunity to demonstrate understanding.
The larger goal is to move away from memorizing answers and toward understanding technical concepts.
How it's built
Stack. N.O.V.A. is a Next.js app (App Router, React 19, Tailwind) with SQLite as the database: the built-in node:sqlite locally, and Turso (libSQL embedded replica) in the cloud, so the app can redeploy on a host with an ephemeral disk without losing data. Models are served through NVIDIA's OpenAI-compatible API, and the API key is encrypted at rest and never sent to the browser.
One prompt per job, not one giant prompt. Instead of a single "be an interviewer" prompt, each role is a separate agent with its own prompt file: interviewer, question generator, answer evaluator, teacher, coding interviewer, code reviewer, system-design evaluator, hiring committee, and so on. Each returns strict JSON, which makes outputs testable and lets me swap or tune one agent without breaking the others.
What "I don't know" actually does. When a candidate says they don't know, the attempt is recorded with a score of 0 (so it still counts against that topic), and the teacher agent produces a structured lesson rather than free text:
- a plain-language explanation
- a detailed technical explanation
- a real-world example
- an optional diagram (layers, flow, or sequence)
- a concise interview-ready answer
- common mistakes
- follow-up questions
The candidate can then retry the question, continue, or get a similar one. The lesson is saved to their notes, and the miss goes into a mistake notebook with spaced-repetition reviews (1 → 2 → 4 → 7 → 15 → 30 days).
Mastery and readiness. Every scored answer updates a per-topic mastery value with an exponential moving average, so recent performance counts more than old performance. Weak topics (low mastery) drive the next daily plan, and a readiness score is recalculated after each session.
Designing for latency. Reasoning models were the biggest UX problem: some calls took 20–70 seconds and truncated JSON. What helped:
- A "fast" mode that turns model thinking down where supported (e.g.
reasoning_effort: low, or thinking disabled on Nemotron 3). On NVIDIA's hosted API, Nemotron 3 Super answered in about 3s this way. - Only the interviewer's reply sits between an answer and the next question. Scoring runs in the background, and the next question is generated while the candidate is still answering.
- If the model takes more than 12s on the first question, or returns invalid JSON, the app falls back to a curated, human-checked question bank with keyword-coverage scoring, and the report labels it as such. The app works end to end even with the AI switched off.
Running code safely. Coding and SQL practice run entirely in the browser: Python via Pyodide in a Web Worker with a time limit, and SQL on SQLite compiled to WebAssembly. Reference solutions stay on the server until the candidate solves the problem.
What I'm still figuring out
The hard question isn't generating feedback. It's knowing whether the feedback actually helps. Right now I look at whether mastery on a topic goes up after an "I don't know" lesson plus a later retry, but I'd love better ideas.
I'd appreciate feedback from developers and educators on how to evaluate whether this type of learning system is actually effective.
Project: https://nova-50046556708.development.catalystappsail.in
Originally published by Dev.to AI. Aggregated on AIWithGhost for educational purposes — full credit and traffic to the original publisher.