Low-Rank Adaptation: Why Freezing Most Weights Still Fine-Tunes a Large Model

Full fine-tuning of a large language model is a heavy operation. Every weight is updated, every optimiser state must be stored, and a separate full copy of the model is needed for each task. Low-Rank Adaptation, usually called LoRA, asks a simpler question: does the update itself need to be that large, or can a much smaller change do the same work?

The answer, in many practical cases, is that a small change is enough. That is why freezing most weights still fine-tunes a large model.

The problem with full fine-tuning

A pretrained weight matrix already contains most of what the model knows. Fine-tuning is not meant to rebuild that knowledge. It is meant to steer it towards a new task, a new style, or a new set of facts.

When every weight is updated, three costs grow together:

• Memory. The model, its gradients, and its optimiser states must all fit in memory.
• Storage. Each fine-tuned model is a full copy of the original model.
• Risk. A large update can overwrite useful pretrained behaviour, especially on a small dataset.

LoRA keeps the pretrained weights fixed and learns only a small, structured update to selected matrices.

The low-rank idea

Consider one pretrained weight matrix. During fine-tuning, it changes from its original value to a new value. LoRA writes that change as the product of two smaller matrices:

Here, W_0 is the frozen pretrained matrix, B and A are the trainable low-rank factors, and the rank r is much smaller than the dimensions of W_0. If W_0 has shape d by k, then A has shape r by k and B has shape d by r. The update BA has the same shape as W_0, but it is built from far fewer parameters.

The parameter saving is easy to see. A full update needs one parameter for every entry of W_0. A low-rank update needs only the entries of A and B:

For example, a 4096 by 4096 matrix has about 16.8 million entries. With rank 8, the low-rank factors have only about 65,000 entries. That is a reduction of more than 99 percent, before any other saving is counted.

Why a low-rank update can be enough

The key claim is not that the model itself is low-rank. It is that the change needed to adapt the model to a new task often has low intrinsic rank. In other words, the useful directions of change live in a much smaller subspace than the full weight space.

This makes sense if fine-tuning is mainly a steering operation. The model already knows language, reasoning patterns, and much of the world. A new task usually requires emphasising some of those patterns and suppressing others, rather than learning an entirely new representation from scratch. A low-rank update can express that kind of change: it adds a small number of new directions to the output of a layer, while leaving the rest of the pretrained behaviour untouched.

Rank is therefore a capacity control. A larger rank can express a richer update, but it also costs more parameters and can overfit more easily. A smaller rank is cheaper and more regularised, but it may be too limited for a task that really does require a large change. In practice, ranks such as 8, 16, or 32 are common starting points, and the right value depends on the task and on how much data is available.

How LoRA is applied in practice

In a transformer, LoRA is usually applied to the attention weight matrices, and often to the feed-forward matrices as well. The original weights are frozen, so they receive no gradient updates. Only A and B are trained.

A is typically initialised with small random values, while B is initialised to zero. That detail matters: at the start of training, the update BA is zero, so the model behaves exactly like the pretrained model. Training then grows the update gradually from that safe starting point.

The forward pass adds the low-rank term to the frozen computation, scaled by a factor that depends on the rank and on a scaling constant alpha:

Here, x is the input, h is the output, and alpha divided by r controls the size of the update relative to the frozen output. A higher alpha makes the update larger; a higher rank spreads the same alpha over more directions.

A minimal implementation makes the structure clear:

import torch
import torch.nn as nn

class LoRALinear(nn.Module):
    def __init__(self, base, r=8, alpha=16):
        super().__init__()
        self.base = base
        for param in self.base.parameters():
            param.requires_grad = False
        in_features = base.in_features
        out_features = base.out_features
        self.A = nn.Parameter(torch.randn(r, in_features) * 0.01)
        self.B = nn.Parameter(torch.zeros(out_features, r))
        self.scaling = alpha / r

    def forward(self, x):
        return self.base(x) + self.scaling * (x @ self.A.T @ self.B.T)

At inference time, the low-rank update can be merged back into the frozen weights, giving a single matrix W_0 plus BA. That removes the extra computation of the separate low-rank path, and the merged model can be used exactly like a normally fine-tuned model. Alternatively, the adapter can be kept separate, which makes it easy to switch between tasks by swapping adapters while sharing one base model.

Where LoRA wins, and where it does not

LoRA wins most clearly when the base model is large, the task data is limited, and many adapted versions of the same model are needed. One frozen base model plus many small adapters is far cheaper to store and serve than many full copies of the model. Training is also cheaper, because gradients and optimiser states are needed only for the adapter parameters.

It does not win everywhere. Some limits are worth stating plainly:

• Rank is a bottleneck. If a task genuinely needs a high-rank change, a very small rank will underfit, no matter how long training runs.
• Module choice matters. Adapting only a few matrices may be enough for some tasks and too narrow for others. Attention-only adapters are a common default, not a universal rule.
• Learning rates need care. Adapter parameters are trained from a different starting point than a full model, so the learning rate that works for full fine-tuning is not always the right one for LoRA.
• Freezing is not forgetting-proof. A low-rank update can still shift behaviour in unwanted ways, and it can still overfit a small or biased dataset.
• Merging is one-way in practice. Once an adapter is merged into the base weights, it is no longer a separate, swappable module unless the original weights are kept.

A useful way to judge LoRA is to ask what the task requires. If the task is mainly about steering, formatting, or specialising an already capable model, a low-rank update is a natural fit. If the task requires teaching the model a large body of new knowledge from scratch, the rank, the data, and the modules adapted all need to be checked more carefully.

The takeaway

Freezing most weights works because fine-tuning, in the common case, does not need to rewrite a model. It needs to adjust it. A low-rank update captures that adjustment in a compact form: two small matrices, a controlled rank, and a scaling factor that keeps the change proportional to what the task actually needs.

That is the core trade that makes LoRA practical. The pretrained model keeps its knowledge, the adapter carries the task-specific change, and the result is a fine-tuned model that costs far less to train, store, and switch between than a full copy would.

Connect with me on LinkedIn: linkedin.com/in/umeshtharukaofficial