What's the first thing that breaks when a RAG system leaves the notebook?
I've been studying and building RAG systems for a while, and I've noticed a pattern. Most tutorials stop once the demo works. But production introduces a completely different set of problems. In my experience, the first
I've been studying and building RAG systems for a while, and I've noticed a pattern.
Most tutorials stop once the demo works.
But production introduces a completely different set of problems.
In my experience, the first issues usually aren't model-related.
They're system-related.
Examples:
- Retrieval returns technically relevant but practically useless context
- Costs grow much faster than expected
- Evaluation becomes difficult and inconsistent
- Model updates introduce subtle behavior changes
- Latency becomes a real constraint
The more I worked on RAG systems, the more I realized that many failures happen between components rather than inside the model itself.
That shifted how I think about AI engineering.
Instead of asking:
"Can this work?"
I started asking:
"What happens when it fails?"
I've been organizing everything I've learned into an open-source project called AI Model Atlas, focused on AI system design rather than specific frameworks.
GitHub:
Hao610
/
AI-Model-Atlas
Bilingual open-source AI learning map: RAG, agents, fine-tuning & system design (EN/ZH) | ไธญ่ฑๅ่ฏญๅผๆบ AI ๅญฆไน ่ทฏ็บฟๅพ๏ผRAGใๆบ่ฝไฝใๅพฎ่ฐๅ็ณป็ป่ฎพ่ฎก
๐บ๏ธ AI Model Atlas
Open-Source AI Learning Map โ From Zero to RAG, Agents & Fine-Tuning
๐ Bilingual docs (EN/ZH) ยท 36 curriculum modules ยท 17 deep-dive chapters ยท Runnable RAG sandbox
๐ฏ A learning-focused architecture simulator โ not a production framework, not a live model catalog.
๐ Reading guide
- โ Great for: learning AI concepts, RAG system design, and hands-on experimentation
- โ ๏ธ Model names & API prices may change โ always verify against official docs
- ๐ซ Not intended for: production deployment or real-time benchmarking
๐งญ Start Here
| I want toโฆ | Go to |
|---|---|
| ๐ Learn from scratch (step-by-step) | CURRICULUM.md โ 36 modules, Phase 1โ5 |
| ๐งฌ Understand the math & internals | DEEP_DIVES.md โ 17 chapters |
| ๐ See the system architecture | ARCHITECTURE.md |
| โถ๏ธ Run the demo in 5 min | Quick Start โ |
| ๐บ๏ธ Pick a learning track | Getting Started Guide |
๐ฆ What's Inside
| Content | Count | Description |
|---|---|---|
| Curriculum | 36 modules | Prompt โ RAG |
For people who have deployed RAG systems:
What was the first production issue that surprised you?
Originally published by Dev.to AI. Aggregated on AIWithGhost for educational purposes โ full credit and traffic to the original publisher.