Effective Teacher Support for Reasoning Distillation
Summary
- A new study explores distilling language models from the teacher's highest-probability vocabulary fraction instead of computing dense scores over the full vocabulary.
- Experiments evaluated Qwen3 4B to 1.7B, Qwen3 8B to 4B, and Nemotron Nano 12B to 9B pairs, testing vocabulary support reductions from 100% down to 0.01%.
- For Qwen3 settings, retaining 0.01% of the vocabulary caused no observed MMLU-Pro performance degradation compared to full-vocabulary distillation.
- Caching retained next-token scores at 0.01% support cut training step time by 38% and restricted student output computation to those tokens, cutting peak memory by 24%.
Why it matters
Reducing vocabulary support during reasoning distillation significantly improves training efficiency and reduces memory overhead without degrading benchmark performance on models like Qwen3.