Hello, I'm Rijul, and I'm building LiveReview — a blast-radius aware AI code review built for your business-critical systems. Star us to help devs discover the project, give it a try, and share your feedback to help improve the product.

Imagine you've trained a huge AI model that is very good at its job, such as recognizing objects in photos or answering questions.

But there's a problem. The model is enormous. It needs powerful servers, costs a lot to run, and may be too slow or too large for a phone, smartwatch, or car.

You want a smaller model that performs almost as well but runs faster and costs less.

One option is to train a small model from scratch using your original data. However, it may not reach the same level of performance as the larger model.

Knowledge distillation offers another approach. Instead of learning only from the original data, the smaller model also learns from the larger model, much like a student learning from a teacher rather than studying a textbook alone.

Let's see how it works.

1. The Teacher and the Student

Think about two ways of learning mathematics.

In the first, you get a textbook containing problems and their final answers.

In the second, you also have a tutor who explains how they approach each problem. The tutor might show you that one problem resembles another, even though the answers are different.

The second approach gives you additional information that can help you learn.

Knowledge distillation applies a similar idea to AI models:

• Teacher model: The larger model that already performs well.
• Student model: The smaller model that learns from the teacher.

The teacher provides additional learning signals that help the student reproduce some of its capabilities.

2. Learning from More Than the Correct Answer

Let's say we have an AI model that classifies animal photos into three categories:

• Cat
• Dog
• Fox

When the model receives a photo of a cat, it produces a score for each category. These scores are called logits. A function called softmax can convert them into probabilities that add up to 100%.

For example, the teacher might produce these probabilities:

The teacher predicts cat because it assigns the highest probability to that class.

But notice something else: it considers dog more likely than fox.

Those smaller probabilities provide additional information about how the teacher distinguishes between the categories.

Hard labels vs. soft labels

In ordinary supervised classification, the training target might be a hard label:

Cat = 100%, Dog = 0%, Fox = 0%

This tells the model which category is correct, but it doesn't directly communicate how the other categories compare.

The teacher's output, on the other hand, gives us a distribution:

Cat = 90%, Dog = 9%, Fox = 1%

These are called soft labels. They provide information about the teacher's relative confidence across all three categories.

The teacher is effectively saying, "This is a cat. Dog is less likely, but fox is even less likely."

This information can help the student learn distinctions that a simple correct-or-incorrect label doesn't express.

Geoffrey Hinton and his colleagues popularized the term dark knowledge for the useful information contained in a teacher's output probabilities, particularly the information in the less likely alternatives.

The key idea is:

Train the student to learn from the teacher's full probability distribution, not just the correct answer.

3. Making the Hidden Information Easier to See: Temperature

There's a problem with the example above.

Imagine the teacher is extremely confident and produces these probabilities:

The probabilities for dog and fox are tiny. The difference between them may provide a useful learning signal, but the teacher's output is dominated by the cat prediction.

This is where temperature comes in.

Temperature, usually represented by (T), controls how soft or sharp the probability distribution becomes when softmax is applied to the teacher's logits.

• T = 1: The usual softmax distribution.
• Higher temperature: The probabilities become more evenly distributed, making the differences between less likely classes easier to observe.
• Lower temperature: The distribution becomes sharper, giving more weight to the highest-scoring class.

For example, with a higher temperature, the teacher might produce a distribution like this:

These numbers are illustrative, not a guaranteed result of a particular temperature.

The important point is that the ranking remains the same, while the probability distribution becomes softer.

The student can now receive a more informative signal about how the teacher relates the different classes.

4. How the Student Actually Learns

The student needs to learn two things:

1. How to imitate the teacher.
2. How to predict the correct answers from the original training data.

During training, these goals can be combined into a single loss function.

A. Match the teacher

First, the student produces its own probability distribution using the same temperature.

The training process compares the student's softened probabilities with the teacher's softened probabilities.

One way to measure this difference is KL divergence. It measures how different two probability distributions are.

The student is trained to reduce this difference, making its outputs more similar to the teacher's.

B. Match the correct labels

The student is also trained using the actual labels in the dataset.

For example, if the image contains a cat, the student should still learn that cat is the correct category.

This is commonly done using cross-entropy loss, a standard method for training classification models.

C. Combine both learning signals

The two losses can be combined using a weighted sum:

• Distillation loss: Encourages the student to imitate the teacher.
• Label loss: Encourages the student to predict the correct answers.

The weights determine how much each objective contributes to training.

This combination helps the student learn from both the teacher's behavior and the original training labels.

5. The Knowledge Distillation Process

Here's the overall workflow:

1. Prepare the teacher: Train a large model or use an existing one.
2. Choose the student: Select a smaller model architecture suitable for your use case.
3. Prepare the training data: Run the examples through the teacher to obtain its outputs.
4. Train the student: Use the teacher's outputs and the original labels to calculate the training loss.
5. Deploy the student: Once training and evaluation are complete, use the smaller model for inference. The teacher usually isn't needed during deployment.

The result is a student model that aims to retain as much of the teacher's useful behavior as possible while requiring fewer resources.

6. Does Knowledge Distillation Really Work?

Yes. One well-known example is DistilBERT, a smaller version of BERT developed by Hugging Face.

According to its creators, DistilBERT is approximately 40% smaller and 60% faster than BERT while retaining around 97% of its language-understanding capabilities on the evaluated benchmarks.

These figures describe that particular model and evaluation setup. Results vary depending on the teacher, student architecture, training data, and task.

Knowledge distillation is also relevant to large language models.

A smaller language model can be trained using outputs generated by a larger model. This is one form of learning from a teacher, although the exact methods differ from probability-based distillation used in classification.

For example, a larger model might generate answers to a collection of questions. Those answers can then be used as training examples for a smaller model.

This can help transfer some of the larger model's capabilities into a model that is cheaper to run.

However, distillation doesn't guarantee that the student will match the teacher. The quality of the training data, the student architecture, and the distillation method all matter.

Conclusion

Knowledge distillation is a way to transfer useful behavior from a larger AI model to a smaller one.

Instead of learning only from the correct answers, the student can also learn from the teacher's output probabilities or other signals. This gives the student additional information about how the teacher approaches a task.

The goal is to preserve as much useful performance as possible while reducing the cost and resources needed to run the model.

In simple terms, knowledge distillation helps a smaller model learn to do more with less by learning from a larger model.

Your team's attention is limited, and the deluge of AI-generated code is making it harder to keep production reliable and secure without slowing you down.

I'm building LiveReview, a blast-radius aware AI code review built for your business-critical systems.

Instead of presenting every diff with equal emphasis, LiveReview scores each change by blast radius — how far its impact reaches through your call graph — so you can focus attention where it actually matters.

Spend code review effort where business risk is highest — not spread evenly across every diff.

⭐ Star it on GitHub:

HexmosTech / LiveReview

Risk-Aware AI Code Review for Business-Critical Systems

     

LiveReview: Risk-Aware AI Code Review for Business-Critical Systems

LiveReview is the org-wide risk-aware harness for AI-written code. It covers two different jobs: deciding which pull requests are safe to skip entirely, and prioritizing what's risky inside the ones that still need a human.

Auto-Approve: Ship the Safe 50% Without a Human

Auto-Approve Upto 50% of Your Pull Requests, Safely. Risk scoring decides which MRs/PRs are safe enough to ship without a human, so your team only spends review time on risky and critical changes.

demo.mp4

LiveReview auto-approving a low-risk change while routing a risky change to a human reviewer. Click to play.

The risk score blends three signals:

• Semantic LLM analysis
• A custom ML model trained on your production risk signals
• Static graph-based codebase analysis

Inspired by Meta's RADAR paper: 50x fewer production incidents, 3x fewer rollbacks, 35% faster reviews.

How it works:

1. Track outcomes - trace production incidents and…

View on GitHub

Click below to try LiveReview with your codebase: