Long-Context Generation Is a Sampling Problem (When You Turn Up the Heat)
Summary
- Researchers show that long-context generation degradation in language models is heavily influenced by sampling methods rather than training deficits alone.
- The study tests 29 open-weight models from 1B to 122B parameters across ten benchmarks and needle retrieval up to one million tokens.
- Samplers like top-nsigma that keep candidate token pools small and steady as temperature rises maintain accuracy, while standard samplers like top-p degrade.
- At temperature 2, coding agents using top-nsigma resolved 46 to 62 GitHub issues, compared to 0 for agents using top-p, top-k, or Top-H.
Why it matters
Understanding how samplers affect long-context token generation under high temperatures helps developers drastically improve the reliability of coding agents and reasoning tasks without requiring additional training.