Beyond On-Policy vs. Off-Policy: At What Level Should LLM Agent Distillation Follow the Student?
Summary
- Researchers introduce a hierarchical agent distillation framework for LLM agents that separates the policy inducing interaction history from the policy generating action token prefixes.
- The study examines self-distillation across three models and four interactive environments, exploring pairing, mixture proportions, and training schedules.
- Results show that student-induced histories paired with teacher-generated action prefixes provide the strongest standalone training signal, while following the student at both levels performs worst.
- Front-loading fully teacher-generated contexts and delaying fully on-policy contexts improves performance over static mixtures.
Why it matters
This research provides practical guidelines for optimizing LLM agent distillation by demonstrating how separating and scheduling on-policy and off-policy levels impacts training effectiveness.