Summary
- Researchers introduce Science Prediction Universe (SPU), a benchmark featuring 100 public verified tasks and 16 private unpublished wet-lab tasks to evaluate AI prediction of experimental outcomes.
- Evaluation of 13 agent configurations across 12 frontier models shows task-adjusted Spearman correlations ranging from 0.227 to 0.478 on the public benchmark.
- Analysis indicates performance is bottlenecked by scientific judgment rather than tool execution, with agents frequently relying on shared monotone priors instead of experimental reality.
Why it matters
The SPU benchmark highlights current limitations in frontier AI models regarding scientific judgment and experimental outcome prediction, pointing to key areas for improvement in AI-driven scientific discovery.