LLMs for 8 GB VRAM: What Fits and What Doesn't
I have been running local models on a 24 GB card for a while, and I keep getting asked the same question by people building their first box: can an 8 GB card actually run a useful LLM? The honest answer is yes, but only
I have been running local models on a 24 GB card for a while, and I keep getting asked the same question by people building their first box: can an 8 GB card actually run a useful LLM? The honest answer is yes, but only if you pick the model to fit the card instead of buying the card for a model it cannot hold. Here is the arithmetic I use, with real file sizes.
Short answer
An 8 GB card runs 7Bβ8B models at 4-bit with a few thousand tokens of context. A 14B model at 4-bit is about 9 GB, so it does not fit without offloading layers to system RAM. If you are buying new for local AI and may want 14B later, 12β16 GB is the safer long-term buy. If you already own an 8 GB card, you are not stuck: start with a 7B/8B model and keep context modest.
The 8 GB budget is not 8 GB of model
Eight gigabytes is the whole pool: weights, the KV cache that holds the conversation, the runtime and driver, and whatever the desktop compositor borrows. A realistic budget for an 8B model at 4-bit looks like this:
| Item | Typical size | Notes |
|---|---|---|
| Model weights (8B, 4-bit) | ~4.9 GB | the GGUF file itself |
| KV cache at 4K context | ~1.5 GB | grows linearly with tokens kept |
| Runtime + driver | ~0.6 GB | CUDA context, buffers |
| Headroom | ~1.0 GB | desktop, transient spikes |
That is roughly 8.0 GB, and it leaves no room for a 32K conversation. At 32K the cache alone can grow past the free space, which is why an 8B model can load fine and then stall or spill to CPU halfway through a long chat. The file size is the entry ticket, not the whole requirement.
What fits, by file size
These are file sizes from the Ollama library, not total VRAM. Add the cache and runtime on top.
| Model | 4-bit file size | Verdict on 8 GB |
|---|---|---|
| Llama 3.2 3B | 2.0 GB | comfortable, long context possible |
| Phi-3.5-mini 3.8B | 2.2 GB | comfortable |
| Mistral 7B | 4.4 GB | fits with room for 4Kβ8K context |
| Qwen2.5 7B | 4.7 GB | fits, 4K context is realistic |
| Llama 3.1 8B | 4.9 GB | fits at short/medium context |
| Gemma 2 9B | 5.4 GB | tight, low context only |
The pattern is simple: below about 5 GB of weights you are comfortable, 5.4 GB is tight, and anything at 9 GB is a different class of card.
What does not fit
Qwen2.5 14B is about 9.0 GB at 4-bit, so it exceeds the pool before any cache is allocated. A 32B model is far beyond that, and 70B is not in the conversation. The failure mode is worth knowing: the model may still load, but the runtime will report a CPU/GPU split and tokens per second collapse to a few per second. That is not "running a 14B model," it is running part of one on the CPU.
How to make a model fit
Change one variable at a time and measure tokens/s and memory after each change. In rough order of usefulness:
- Lower the context. Go to 4K before you try 32K. The KV cache is the easiest thing to cut and usually costs you the least for chat-style use.
- Use Q4_K_M. 4-bit quantization is the default for a reason; 8-bit roughly doubles the file size for a small quality gain, which is the wrong trade on an 8 GB card.
-
Offload a few layers. Slower, but it runs.
ollama psshows whether the model is 100% GPU, 100% CPU, or split, so you can see the moment you cross the line. - Pick a smaller model. A 3B/4B model at 8 GB is comfortable and leaves context headroom.
If you are buying a card
An 8 GB card such as the RTX 3070 is an entry point: it runs 7B/8B models well and will not grow into 14B. A 12 GB card (RTX 3060 12 GB or RTX 4070) gives 14B at 4-bit some room. A 16 GB card (RTX 4060 Ti 16 GB or RTX 4070 Ti Super) is comfortable for 14B and some 32B at low context. 24 GB opens 32B properly.
Match the card to the model you actually want to run, not the other way around. The most common expensive mistake I see is buying a 24 GB card for a 7B model, then discovering that the workload never needed it; the second most common is buying 8 GB and expecting 14B.
Quantization is where most of the budget comes from
The difference between a model that runs and one that does not is usually the quantization level, not the parameter count. A 7B model at 8-bit is roughly 8 GB before cache, which does not fit on an 8 GB card at any useful context. The same model at 4-bit is about 4.4 GB and leaves room for a real conversation. In practice, Q4_K_M is the level most people should start with on 8 GB, and Q5_K_M is worth trying only when you have confirmed at least 1.5 GB of headroom after loading, because the KV cache still has to fit in whatever is left.
There is a quality cost, but it is smaller than the memory saving suggests for chat and summarization work. Where 4-bit hurts most is long-context reasoning and code generation, and that is exactly where an 8 GB card is already tight because of the cache. If your workload depends on 32K-token code context, the honest conclusion is that 8 GB is the wrong card, not that you need a cleverer quantization.
Running more than one thing
A single 8B model at 4K context leaves about 1 GB of headroom, which is not enough to keep a second model resident. If you want an embedding model alongside a chat model, budget for it up front: a small embedding model at 4-bit is a few hundred MB, which fits, but two generation models generally do not. The same rule applies to background GPU work such as a browser with hardware acceleration or a video call. On 8 GB, the desktop is a roommate that does not pay rent, and it will take memory the moment you stop watching.
Common questions
Can an 8 GB card run a 14B model? At 4-bit the file is about 9 GB, so it does not fit in 8 GB without offload. A 12β16 GB card is the practical upgrade if 14B is the goal.
Is 8 GB enough for Llama 3.1 8B? At 4-bit the file is about 4.9 GB. It fits with a few thousand tokens of context; longer context adds cache and may push you into offload.
Should I buy an 8 GB or 16 GB card? If local AI is the main use and 14B is plausible later, 16 GB is the safer buy. 8 GB is fine for 7Bβ8B at short context, and it is a reasonable first card if the budget is fixed.
Why does my model load but feel slow? Check ollama ps. If it reports a split, layers are running on the CPU, and no amount of prompt tuning will fix that. Lower context or switch to a smaller model before changing anything else.
Practical rules of thumb
- 7B/8B at 4-bit is the sweet spot for 8 GB. Expect it to work, expect 4K context, and expect to keep the desktop light.
- 14B at 4-bit needs 12 GB to be pleasant. It will technically run on 8 GB with offload, but slowly enough that you will stop using it.
- Context is the hidden cost. A 4K conversation is cheap; a 32K conversation can double your memory footprint without changing the model file at all.
-
Measure, do not guess.
ollama psand a tokens/s number tell you more than any spec sheet, including this one.
Bottom line
An 8 GB card is a good entry point for 7Bβ8B local models at 4-bit and modest context. It is not a 14B card and no amount of quantization will make it one. If you already own one, start with Llama 3.1 8B or Qwen2.5 7B, keep context at 4K, and check ollama ps before you blame the model. If you are buying, 12β16 GB buys meaningfully more useful life for local AI, and the jump from 8 GB to 16 GB is the one upgrade that changes which models you can run rather than just how fast they run.
Model file sizes were checked against the Ollama library in November 2026. The budget table is an illustrative example, not a measurement of a specific card; real memory use depends on runtime, quantization, context, and concurrency. NVIDIA's RTX 3070 (8 GB) and RTX 4090 (24 GB) specification pages were used as the reference points for the tiers above.
Originally published by Dev.to AI. Aggregated on AIWithGhost for educational purposes β full credit and traffic to the original publisher.