LoRA fine-tuning on one GPU: from JSONL to a working adapter

I wanted a model that answered in my own format — not a general chatbot, just something that reliably turned messy input into the labels I cared about. Retraining a full model was out of reach, but LoRA on a single consumer card turned out to be very doable. Here is the process I followed, including the parts that broke.

This is written for someone with one GPU (8–24 GB of VRAM), a Linux or Windows box, and Python installed. You don't need a cluster or a big budget. You do need to be comfortable with a terminal.

What LoRA actually changes

LoRA keeps the base model frozen and bolts small adapter matrices onto selected layers. During training, only those adapters update, which is why the memory footprint stays small: you're not storing gradients and optimizer states for billions of weights, only for a thin slice of them.

The result is a checkpoint measured in tens of megabytes instead of gigabytes. You keep the base model around and load the adapter on top of it at inference time.

The models that fit this workflow comfortably are small language models — anything from a few million up to roughly 10B parameters. In practice I'd start with Qwen3.5-0.8B or a small Gemma (2B); a 16–24 GB card gives you room for Gemma 9B or Phi-4 14B, and Gemini Nano-1 (1.8B) / Nano-2 (3.25B) are also on the table.

Requirements before you touch any code

• One consumer GPU with at least 8 GB of VRAM. Phi-4 14B will want a lot more than that.
• Python 3.10+.
• transformers, peft and datasets installed.
• A base model picked by name, so you're not guessing later.
• A train.jsonl file with a few hundred clean examples.
• Admin rights for CUDA drivers and around 20 GB of free disk for checkpoints.

pip install transformers peft datasets

That's the whole environment for a first run. Resist the urge to add a dozen more packages before you've seen the loss go down once.

Prepare the data, then pick the model

I do data first because it's the cheapest thing to get wrong and the most annoying thing to debug mid-training. One JSON object per line, one instruction/output pair each:

{"instruction": "Classify the ticket", "output": "billing"}
{"instruction": "Classify the ticket", "output": "technical"}

The trainer expects plain text pairs, not a database dump. For a narrow task, 100–300 clean examples are usually enough to see a style or format shift — quality beats volume here.

Model choice is mostly a VRAM decision. For an 8 GB card, Qwen3.5-0.8B or Gemma 2B leaves headroom for gradients and optimizer states. On 16–24 GB you can move up to Gemma 9B or Phi-4 14B. Smaller models are not a downgrade for this workflow; they're the reason it fits at all.

Load the base model and attach an adapter

Standard Hugging Face loading, nothing exotic. The model downloads once and is cached locally afterwards.

from transformers import AutoModelForCausalLM, AutoTokenizer
model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3.5-0.8B")
tok = AutoTokenizer.from_pretrained("Qwen/Qwen3.5-0.8B")

Then wrap it with peft. I used a small rank and targeted the attention projections:

from peft import LoraConfig, get_peft_model
cfg = LoraConfig(r=8, lora_alpha=16, target_modules=["q_proj", "v_proj"])
model = get_peft_model(model, cfg)
model.print_trainable_parameters()

print_trainable_parameters() is the sanity check I now run every single time. You want to see something in the neighbourhood of "0.5% of parameters trainable" and a non-zero trainable count. If it says zero, your target_modules don't match the architecture and nothing useful will happen during training.

Train for a couple of epochs, then test

One to three epochs is plenty for a narrow task. I kept the batch size at 1–4 and used gradient accumulation to get an effective batch that fits in memory. What you're watching is the loss: on my run it moved from around 2.5 down toward 1.0. A gentle downward slope is the signal; a flat line is not.

Once training finishes, save the adapter:

model.save_pretrained("my-lora-adapter")

Then load the base model plus the adapter and ask it something from your own domain. The comparison that matters is behaviour, not loss: do answers follow the style and format of your dataset? If yes, you're done with the first iteration.

Before you ship a fine-tuned model anywhere, evaluate it properly rather than eyeballing three prompts.

When it breaks

CUDA out of memory on the first steps. Drop to a smaller model (Gemma 9B → Gemma 2B or Qwen3.5-0.8B), set batch size to 1, or lower the LoRA rank from 16 to 8. In that order — the model swap usually fixes it fastest.

Loss stays flat after hundreds of steps. Validate the JSONL first: every line must be a complete, valid JSON object with both instruction and output. Then nudge the learning rate up slightly and re-check that the adapter targets the right modules.

The fine-tuned model outputs gibberish. This is almost always too many epochs on too little data. Retrain for 1–2 epochs. Also confirm that the tokenizer used at inference is the same one used in training — a mismatch produces exactly this kind of broken text.

A pre-flight list I keep next to me: the model fits in VRAM with headroom, train.jsonl holds at least 100–300 clean examples, peft and transformers are installed, an output directory exists for the adapter, and my test questions were written down before training started so I can't rationalise the results afterwards.

A few questions I had

Can Phi-4 14B be trained on one GPU? Yes, but it needs a high-VRAM consumer card. On anything smaller, start with Qwen3.5-0.8B or Gemma 2B.

Does a fine-tuned SLM replace a large LLM? No. Small models have their own niche — narrow tasks and agentic systems — and a 0.8B model with a LoRA adapter is not a general-purpose replacement.

How much data do I need? For a narrow task, a few hundred clean JSONL examples are typically enough to see the format change you're after.

The takeaway: LoRA is less about compute and more about discipline. Pick a model that fits with room to spare, keep the dataset small and clean, watch print_trainable_parameters() and the loss curve, and train for fewer epochs than you think you need. If you haven't run a model locally at all yet, get that working first — fine-tuning builds directly on it.

This article was originally published on AI Release — https://ai-release.org. More guides like this land in the channel: https://t.me/ai_release1.