StateBench: Testing how language models track state changes over long documents
Summary
- Researchers introduced StateBench, a new benchmark designed to evaluate how language models track entity state changes over long documents across four complexity levels.
- The benchmark levels—r-trivial, aperiodic, soluble, and general finite—are grounded in Krohn-Rhodes theory and correspond to different types of variable assignments.
- Evaluations show that while frontier and pretrained transformer models perform well on simpler levels, they break down on harder state-tracking tasks as inputs grow longer.
- Models trained directly on StateBench reveal that recurrent and hybrid architectures, such as Gated DeltaNet, generalize much farther than transformers on complex state tracking.
Why it matters
Understanding the architectural limits of language models in long-context state tracking helps guide the development of future recurrent and hybrid systems capable of handling complex reasoning tasks.