Long-Context Generation Is a Sampling Problem (When You Turn Up the Heat)

Impact 8 arXiv cs.AI

Summary

  • Researchers show that long-context generation degradation in language models is heavily influenced by sampling methods rather than training deficits alone.
  • The study tests 29 open-weight models from 1B to 122B parameters across ten benchmarks and needle retrieval up to one million tokens.
  • Samplers like top-nsigma that keep candidate token pools small and steady as temperature rises maintain accuracy, while standard samplers like top-p degrade.
  • At temperature 2, coding agents using top-nsigma resolved 46 to 62 GitHub issues, compared to 0 for agents using top-p, top-k, or Top-H.

Why it matters

Understanding how samplers affect long-context token generation under high temperatures helps developers drastically improve the reliability of coding agents and reasoning tasks without requiring additional training.

Who should care