Local AI on CPU Without a GPU: How Far Can It Go? — Building an Auto-Tracking, Benchmarking Loop, and the Conclusions I Reached

Riel's chat runs on a cloud API (Google Gemini). For a while, though, I've been stuck on the hypothetical: "What if I couldn't use this API?" Whether due to model export regulations or policy changes, I wanted to build a **minimal Korean chatbot fallback that runs solely on my home desktop** in case external AI access is cut off. This post is a record of how far I pushed this on CPU-only, without a GPU, and why it's no longer my primary solution.

The Condition: Effectively No GPU

My desktop has a Ryzen 5700X (8 cores, 16 threads), 32GB RAM, and an RX580 4GB for graphics. The problem is, the RX580 (Polaris) doesn't support the latest ROCm, meaning **I can't properly use the GPU for local LLM acceleration**. Plus, 4GB isn't enough to load a 7B model entirely. In the end, **the actual engine was the CPU**. "How far can I go without a GPU, on CPU?" was the real question from the start.

What I Built — A Loop of Measuring, Tracking, and Self-Replacement

I didn't want to just run a model and call it a day. My goal was a system that could **automatically swap out models for better CPU-compatible ones when they become available**, even if I miss it.

1. **Bench Harness** — I created Korean auto-grading questions (math, multi-step reasoning, logical negation, unit conversion, facts, instruction following) to measure accuracy and speed (tok/s). It started with 8 questions, but they **lacked discriminative power** (more on this later). I increased it to 25, then to 50 questions.
2. **Automatic Tracking of New Hugging Face Models** — I detect new releases from manufacturers (EXAONE, Qwen, Google, etc.) on a weekly basis and add them to a candidate queue. Initially, I scraped "trending" models, which brought in a lot of 2023 community fine-tunes. I refined this by using a **watchlist of authors + filters for size and family** to cut down on noise (from 78 candidates down to 24).
3. **Autonomous Benchmarking Loop (auto\_bench)** — My desktop automatically pulls new models from the queue, runs the 50-question benchmark, and uploads the results to a server. The server then **automatically replaces the currently serving model if the new one is definitively better** (either in accuracy margin or speed advantage). This loop actually discovered and swapped in models I hadn't explicitly scouted.
4. **Public Demo (/local-ai)** — Using an outbound polling worker structure that doesn't open any inbound ports on my home network, anyone could experience this CPU model while my desktop was on. ## Actual Measurements — And the Moment the Illusion Broke Final scoreboard (my desktop, CPU) based on 50 questions: | Model | Accuracy | Speed | | -------------- | -------- | -------- | | exaone3.5:7.8b | 90% | 14 tok/s | | exaone3.5:2.4b | 84% | 53 tok/s | | qwen2.5:3b | 84% | 43 tok/s | | gemma2:2b | 76% | 31 tok/s | The biggest takeaway here was **how much the measurement design influences the conclusion**. gemma2:2b initially scored 100% on the first 8 questions, making it seem "the best." But as I increased the question count, its score dropped continuously from 84% (25 questions) to 76% (50 questions). It was an overestimation caused by easy questions. Conversely, the value of exaone 7.8b only became apparent with harder questions (it tied with 2.4b on the initial 8 questions). **Seeing rankings remain stable even with an expanded sample size** finally became a mark of true reliability. Ultimately, the optimal balance for CPU serving was exaone3.5:2.4b (84% accuracy + fastest). ## Why It Stepped Down From Primary Use To be honest, **the engineering was successful, and that's precisely what clearly showed its limitations.** The auto-tracking -> auto-benchmarking -> auto-replacement loop actually works. However: * The capabilities of models runnable on CPU (roughly 2~8B parameters) are about **GPT-3.5 level**. The gap is too large to replace the latest cloud models Riel's chat uses. * The realistic answer to "local AI gets better on its own" isn't that **models inherently become smarter (weights are fixed), but rather that they detect new models and replace them with better ones**. But the ceiling for that replacement is inherently low. * To reach true API-level performance, you need a GPU (which means money). I decided not to make that investment at this stage. So, local CPU AI has been **demoted from primary use to a "fallback and experimental demo" status.** The /local-ai endpoint is still alive, allowing direct experience if my desktop is on, but it no longer handles Riel's actual responses. ## What Remains (Lessons Learned) * **Measurement comes first.** That 100% on 8 questions was an illusion. If your yardstick is flimsy, you'll pick the wrong model. * **Autonomous loops can be built, but they can't overcome hardware ceilings.** Even if you automate detection, benchmarking, and replacement with software, if the best you can afford is GPT-3.5 level, that's where you'll stay. * **Knowing when to fold is also a result.** Confirming with numbers, not just gut feeling, that something "works/doesn't work" and stepping back — and documenting that record — is valuable. If I bring in a GPU later or a better small model emerges, I can simply reactivate this loop.

💬 This is part of *Riel** — a full AI product I'm building solo, in public (failures and all). Read more build logs → · See the product →*