Hello, I'm Shrijith Venkatramana, and I'm building LiveReview — a blast-radius aware AI code review built for your business-critical systems. Star us to help devs discover the project, give it a try, and share your feedback to help improve the product.

A 70B-parameter model in FP16 needs roughly 140 GB just to store its weights.

Put the same weights in 8-bit and you get roughly 70 GB.

That sounds like a solved problem.

It isn't.

The interesting part of LLM quantization is not reducing 16 bits to 8 bits. It is deciding which 8-bit numbers are allowed to represent which parts of the model.

This is where things get strange.

A single activation can be 50x or 100x larger than its neighbors. If you use one INT8 scale for the whole tensor, that one value can determine the scale for thousands of ordinary values.

Then researchers discovered something even more useful: these outliers are often concentrated in particular feature dimensions.

That observation led to a sequence of ideas involving Tim Dettmers, Song Han's group at MIT, and engineers from NVIDIA, Intel, Arm and others:

INT8 → find the outliers → isolate them or move them → choose better scaling → eventually use floating-point 8-bit formats.

The important lesson for developers is that bit width, number representation, and scaling granularity are three separate decisions.

Let's build the intuition from the ground up.

1. Quantization is really a problem of representing a range of numbers

Suppose you have this activation vector:

[-0.8, 0.3, -0.2, 0.7, 0.1, 50.0]

You want to represent it with signed INT8.

INT8 gives you 256 possible bit patterns, usually treated as approximately:

-127 ... 0 ... +127

A simple symmetric quantizer chooses

scale = max(abs(x)) / 127

Here:

scale = 50 / 127
      ≈ 0.394

Every value gets rounded onto a grid separated by about 0.394.

So:

 0.1 / 0.394 ≈ 0.25  -> 0
 0.3 / 0.394 ≈ 0.76  -> 1
-0.2 / 0.394 ≈ -0.51 -> -1
 0.7 / 0.394 ≈ 1.78  -> 2

After dequantization:

0.1 -> 0.0
0.3 -> 0.394
0.2 -> 0.394
0.7 -> 0.788

The small values have become rather crude.

Now imagine that 50.0 was absent.

The scale becomes:

0.8 / 127 ≈ 0.0063

Suddenly the quantization grid is about 62x finer.

That is the fundamental problem with absmax quantization:

One extreme value can consume the dynamic range that the other 99.9% of the tensor wanted to use.

This is why "INT8" by itself tells you surprisingly little.

You also need to ask:

What is the scaling granularity?

2. Per-tensor scaling is cheap; outliers make it painful

The simplest scheme is per-tensor scaling.

You look at the entire tensor and compute one scale:

s = max(abs(X)) / 127

Then:

X_int8 = round(X / s)
X_hat  = X_int8 * s

The hardware story is attractive.

One tensor.

One scale.

One INT8 GEMM.

Very little metadata.

But LLM activations have a peculiar distribution.

In 2022, Tim Dettmers, Mike Lewis, Younes Belkada and Luke Zettlemoyer published LLM.int8() and investigated what was causing ordinary INT8 quantization to fail as Transformer models grew.

Their measurements of OPT models found that large-magnitude activation features emerged systematically as models became larger.

The particularly memorable result was at the 6.7B scale.

They reported roughly 150,000 outlier values per sequence, but those outliers were concentrated into only about six feature dimensions across the Transformer. Those dimensions represented only around 0.1% of the feature values, yet removing them badly damaged model behavior.

This is an important observation.

The problem was not simply:

"There are a few large numbers."

It was closer to:

"There are a few special dimensions that repeatedly produce
large numbers, and the model actually uses them."

That distinction changes the engineering solution.

3. Per-channel scaling looks like the obvious solution

Suppose the activation matrix is:

X =

token 1: [ 0.2   0.3   0.1   60.0 ]
token 2: [ 0.1   0.2   0.3   55.0 ]
token 3: [ 0.3   0.1   0.2   49.0 ]

Column 4 is clearly behaving differently.

With per-tensor scaling, the 60.0 controls the scale for everything.

With per-channel scaling, every column gets its own scale:

channel 1 -> based on max(0.2, 0.1, 0.3)
channel 2 -> based on max(0.3, 0.2, 0.1)
channel 3 -> based on max(0.1, 0.3, 0.2)
channel 4 -> based on max(60, 55, 49)

Now the first three channels can use a much finer INT8 grid.

So why not simply quantize activations per channel?

Because the matrix multiplication is:

Y = XW

and the activation channels are the reduction dimension of the matrix multiplication.

If you scale every input channel independently, your computation becomes something like:

Y = (X * channel_scales) W

The scale is now entangled with every multiplication contributing to each output.

The problem is therefore partly numerical and partly architectural:

the quantization scheme you would like is not necessarily the quantization scheme your fast GEMM kernel wants.

This is one of the recurring themes of quantization:

The numerically nicest scheme is not necessarily the computationally cheapest scheme.

The SmoothQuant paper from Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth and Song Han made this tension particularly clear.

For their experiments, activation quantization with finer channel granularity could preserve accuracy, while conventional INT8 GEMM implementations favored coarser activation scaling.

The trick was to move the problem somewhere else.

4. SmoothQuant: move the difficulty from activations into weights

This is one of the nicest pieces of algebra in practical LLM optimization.

Suppose:

Y = XW

Pick a diagonal matrix of per-channel scales:

S = diag(s1, s2, ..., sn)

Then:

Y = XW
  = (X S^-1)(S W)

Nothing has changed mathematically.

You have simply moved a scale factor from one side of the matrix multiplication to the other.

Define:

X' = X S^-1
W' = S W

Then:

X'W' = XW

Now suppose one activation channel contains huge values.

Choose sj to be large.

The corresponding activation channel gets divided by sj:

large activation -> smaller activation

while the corresponding weight channel gets multiplied by sj:

weight -> somewhat larger weight

Why is that useful?

Because weights tend to be considerably easier to quantize than activations.

SmoothQuant exploits this asymmetry.

Its smoothing factor can be expressed approximately as:

sj = max(|Xj|)^alpha / max(|Wj|)^(1-alpha)

where alpha controls how much quantization difficulty gets moved toward the weights.

At:

alpha = 0

you move essentially none of the activation difficulty.

At:

alpha = 1

you push the problem aggressively toward the weights.

A common starting point is:

alpha = 0.5

which roughly balances the ranges in each channel.

Consider a toy channel:

max activation = 100
max weight     = 0.01

With alpha = 0.5:

s = sqrt(100 / 0.01)
  = sqrt(10000)
  = 100

So the transformed ranges become approximately:

activation: 100 / 100 = 1
weight:       0.01 * 100 = 1

You have turned:

activation range = 100
weight range     = 0.01

into:

activation range ≈ 1
weight range     ≈ 1

The matrix multiplication still computes the same function.

The distribution has simply been rearranged into a form that is friendlier to quantization.

This is why SmoothQuant is more interesting than "use smaller numbers."

It is an algebraic transformation that changes where quantization error is paid.

The original paper reported up to 1.56x speedup and 2x memory reduction for their evaluated models while maintaining close accuracy to higher precision baselines.

5. LLM.int8() took a different route: keep the bad dimensions in higher precision

Dettmers' approach was conceptually different.

Instead of trying to eliminate the outliers, LLM.int8() observed:

99.9%+ of the values -> ordinary INT8 computation
tiny set of important dimensions -> higher precision

So the matrix multiplication is decomposed.

Conceptually:

Y = X_outlier W_outlier
  + X_regular W_regular

The outlier dimensions are computed in FP16, while the rest use INT8.

This is a very practical compromise.

Suppose the hidden dimension is 4096 and only a handful of feature dimensions are problematic.

You do not need to make all 4096 dimensions expensive just because six of them are troublesome.

This is analogous to designing a network where one pathological flow gets special handling instead of upgrading the entire network.

And there is an operational advantage:

you preserve the bulk of the INT8 computation.

The LLM.int8() paper reported that more than 99.9% of values could still participate in 8-bit multiplication while the problematic dimensions were handled at higher precision.

That work was also an important moment historically.

Before it, "8-bit inference" often sounded like a relatively straightforward compression exercise.

The experience with billion-parameter Transformers showed that scaling the model changed the statistical behavior of the activations.

Quantization became a problem about understanding the model's internal structure, not merely reducing storage.

6. FP8 changes the number system itself

Now we get to FP8.

INT8 gives you a fixed-point-like grid after scaling.

FP8 is fundamentally different.

Instead of using all eight bits to represent an integer, you split them into:

sign + exponent + mantissa

The 2022 FP8 proposal from Paulius Micikevicius and collaborators defined two formats:

E4M3
1 sign bit + 4 exponent bits + 3 mantissa bits

E5M2
1 sign bit + 5 exponent bits + 2 mantissa bits

The tradeoff is exactly what you would expect:

more exponent bits -> more range
more mantissa bits  -> more precision

A rough mental model is:

INT8:
values lie on an approximately uniform grid

FP8:
values are distributed approximately logarithmically
across orders of magnitude

Imagine the numbers:

0.125
0.25
0.5
1
2
4
8
16

A floating-point representation naturally gives you useful coverage across such scales.

An integer representation needs a scale to move the whole grid around.

That makes FP8 much better suited to distributions with wide dynamic range.

But there is a subtle point:

FP8 does not eliminate scaling.

A common FP8 computation still looks conceptually like:

x_fp8 = FP8(x / scale)
x_hat = FP8_value * scale

The format gives you a wider range structure inside the 8 bits, but the scale still determines which region of that format your tensor occupies.

And there are different FP8 formats for different numerical jobs.

E4M3 provides more mantissa precision and less range.

E5M2 sacrifices mantissa precision for more exponent range.

That makes them useful for different parts of training. A common arrangement is E4M3 for forward-pass values and E5M2 for gradients.

This was the broader significance of the 2022 FP8 work: INT8 and FP8 were no longer simply two ways to store "small numbers."

They represented two different numerical philosophies:

INT8:
"Give me a scale, then use a uniform integer grid."

FP8:
"Give me a scale, then let the exponent encode dynamic range."

That distinction matters enormously for LLM activations.

7. So which one should a developer use?

The useful answer is: look at the whole inference system, not just the datatype.

There are at least four variables:

1. Datatype
   INT8 vs FP8 vs FP16/BF16

2. Scaling granularity
   per-tensor vs per-token vs per-channel vs block-wise

3. Quantization location
   weights, activations, KV cache, or some combination

4. Hardware/kernel support
   what your actual GPU or CPU can execute efficiently

This produces a very different engineering decision than:

"INT8 is smaller, therefore INT8 is better."

Consider a 70B model.

Very roughly:

FP16 weights:
70B * 2 bytes
≈ 140 GB

INT8 / FP8 weights:
70B * 1 byte
≈ 70 GB

You have saved approximately:

70 GB

of parameter memory.

That can determine whether a model fits on a given machine.

But total inference memory is closer to:

weights
+ KV cache
+ temporary activations
+ workspace
+ runtime overhead

So 70 GB of weights does not mean a model is going to fit comfortably into a 70 GB device.

There is also an economic dimension.

Suppose your deployment needs enough GPU memory for:

5 x 80 GB GPUs

at FP16, primarily because the weights are too large.

If an 8-bit representation reduces the weight footprint enough to move the deployment to:

3 x 80 GB GPUs

the economic effect can be larger than the numerical effect.

You have potentially removed:

2 GPUs

from every replica.

At scale, that changes:

GPU rental
rack capacity
power
network bandwidth
failure surface
deployment density

But this is where benchmarking matters.

If your hardware has highly optimized FP8 Tensor Core kernels and your INT8 path requires awkward conversions, FP8 may win despite both using exactly one byte per stored value.

On another machine, INT8 may have better kernel support.

Even within one datatype, the difference between:

per-tensor
per-token
per-channel
block-wise

can change both accuracy and runtime.

A useful profiling equation is therefore:

effective cost
≈ memory traffic
+ compute time
+ scaling overhead
+ kernel inefficiency
+ synchronization overhead

The cheapest-looking representation on paper can lose once all five terms are included.

Conclusion: 8 bits is only the beginning

The interesting story of LLM quantization is not:

16 bits -> 8 bits

It is:

How do we spend the limited precision budget?

The history makes this progression clear.

Dettmers and colleagues encountered systematic activation outliers and separated a tiny set of problematic dimensions from the bulk of the computation.

Xiao, Lin, Seznec, Wu, Demouth and Han showed that the algebra of the matrix multiplication could be exploited to move quantization difficulty from activations into weights.

Micikevicius and collaborators then pushed the industry toward a floating-point 8-bit representation that gave hardware a better numerical tradeoff for deep learning.

For developers, the mental model I find most useful is:

INT8 vs FP8
        |
        +-- What numbers can the format represent?
        |
        +-- What scale maps my tensor into that range?
        |
        +-- How many elements share that scale?
        |
        +-- Where do the outliers live?
        |
        +-- Can the hardware execute that representation efficiently?

Once you think this way, "quantize the model to 8-bit" stops being a single operation.

It becomes a small numerical systems-design problem.

And that is probably the more useful way to approach the next generation of LLM inference.

When you deploy an LLM, which tradeoff would you optimize first: numerical accuracy, GPU memory, or raw tokens/second?

Your team's attention is limited, and the deluge of AI-generated code is making it harder to keep production reliable and secure without slowing you down.

I'm building LiveReview, a blast-radius aware AI code review built for your business-critical systems.

Instead of presenting every diff with equal emphasis, LiveReview scores each change by blast radius — how far its impact reaches through your call graph — so you can focus attention where it actually matters.

Spend code review effort where business risk is highest — not spread evenly across every diff.

⭐ Star it on GitHub:

HexmosTech / LiveReview

Blast-Radius Aware AI Code Review for Business-Critical Systems

     

LiveReview: Blast-Radius Aware AI Code Review for Business-Critical Systems

LiveReview is an AI code reviewer that scores every hunk of a diff by blast radius: how far a change reaches through your call graph, how much persistent state it touches, and how well-tested it is. A 3-line change to a shared auth check can outrank a 300-line UI tweak. Your team's attention goes to the highest-risk code first, not spread evenly across every diff.

blast-radius-demo.mp4

LiveReview's Blast Radius & Review Priority scoring, live in the diff viewer.

The exact math, not a black box	Visualize blast radius at a glance	Every factor that feeds the score

How does Blast Radius scoring work? (a more technical explanation)

Here's the goal:

• A 3-line fix in a function used by 40 other files, that also writes to a database, should score high.
• A 300-line UI change in one file, fully covered by…

View on GitHub

Click below to try LiveReview with your codebase: