AppSci-Bench: Can LLMs Build AI Agents and Make Them Better?

Impact 7 arXiv cs.AI

Summary

  • Researchers introduced AppSci-Bench to evaluate how effectively large language models can build AI agents for businesses.
  • The benchmark covers seven realistic domains including slides generation, order-to-cash, compliance review, and customer call insights.
  • Testing seven frontier models revealed significant limitations, with the best model, Claude Opus 5.0, achieving a mean-minimum score of 59.1%.
  • Models struggle particularly with eliciting task requirements and constructing representative internal evaluations.

Why it matters

AppSci-Bench highlights critical limitations in current frontier LLMs regarding end-to-end agent development, showing that automated agent engineering remains a major challenge for AI practitioners.

Who should care