SciGame: Let Agents Play the Scientific Discovery Games in the Real and Counterfactual World
Summary
- Researchers introduced SciGame, a controlled benchmark converting scientific literature into structured cards with real and counterfactual environments.
- SciGame features a Historical collection of canonical laws and a live Temporal collection from papers published after model knowledge cutoffs.
- Experiments across five frontier models show that familiar pretrained priors can hurt agent performance when governing relations are altered.
- The benchmark demonstrates that recovering exact symbolic forms remains a major bottleneck, even when agents correctly interpret evidence.
Why it matters
SciGame provides a rigorous evaluation framework for LLM-based scientific discovery agents by separating empirical evidence from pretrained priors and testing adaptability in counterfactual environments.