SciGame: Let Agents Play the Scientific Discovery Games in the Real and Counterfactual World

Impact 7 arXiv cs.AI

Summary

  • Researchers introduced SciGame, a controlled benchmark converting scientific literature into structured cards with real and counterfactual environments.
  • SciGame features a Historical collection of canonical laws and a live Temporal collection from papers published after model knowledge cutoffs.
  • Experiments across five frontier models show that familiar pretrained priors can hurt agent performance when governing relations are altered.
  • The benchmark demonstrates that recovering exact symbolic forms remains a major bottleneck, even when agents correctly interpret evidence.

Why it matters

SciGame provides a rigorous evaluation framework for LLM-based scientific discovery agents by separating empirical evidence from pretrained priors and testing adaptability in counterfactual environments.

Who should care