On-policy distillation loses to GRPO on compute efficiency

Researchers compared vanilla reverse-KL on-policy distillation with GRPO across task difficulty, student and teacher size, domain, and optimisation choices, measuring held-out accuracy against training FLOPs. With an external post-trained teacher, OPD generally lost to GRPO on accuracy per training FLOP at matched settings.

Why it matters

It challenges the assumption that dense teacher supervision is cheaper than RL, informing how labs allocate training compute for smaller models.

Analysis

The study directly tests OPD against GRPO on FLOPs, not just final accuracy, and finds the dense signal does not compensate for teacher inference cost in the tested settings. That matters because distillation is often assumed to be efficient. The main uncertainty is whether results generalise beyond vanilla reverse-KL OPD, the tested teachers, tasks, and optimisation choices.

Source: arXiv

AI disclosure: this article was produced with AI assistance and reviewed for accuracy.