PersonaScienceBench: Evaluating Scientific Agents with Diverse Simulated Researchers
Summary
- Researchers introduced PersonaScienceBench, a new benchmark featuring 200 tasks across five benchmarks and four scientific domains to evaluate scientific AI agents.
- The benchmark uses simulated researchers with diverse persona profiles to provide context and feedback based on their preferences during agent interaction.
- Evaluation across eighteen model configurations revealed distinct differences in task performance, fact recovery, self-repetition, and workflow revision.
- Analysis demonstrates that agent behaviors and interaction traces can differ significantly even when final task scores are identical.
Why it matters
PersonaScienceBench provides a rigorous framework for evaluating how scientific AI agents collaborate with humans who have diverse preferences and expertise, moving beyond simple static task completion metrics.