SlackBench: Benchmarking Agents on Collaborative Projects Grounded in Real Code Repositories
Summary
- Researchers introduce SlackBench, a new benchmark for evaluating AI agents on realistic collaborative research and engineering workflows.
- The benchmark is grounded in privacy-preserving reenactments of real academic research projects, combining multi-party dialogues with GitHub repositories and Google Docs.
- SlackBench features 140 queries covering tasks like agentic search, abstention on unanswerable questions, and goal-directed summarization.
- Testing 33 frontier models and agent harnesses revealed that the top system scores around 80%, while most score below 66%.
Why it matters
SlackBench addresses a gap in AI evaluation by testing how agents handle complex, multi-party dynamics and joint reasoning across workplace communications and evolving code repositories.