Summary
- Anthropic released a new evaluation tool for Claude Code via its claude-api plugin, featuring build_eval and hill-climb commands.
- A review by Hamel Husain and Isaac Flath highlighted strong out-of-the-box issue discovery capabilities for voice agents and human handoff.
- Drawbacks included pushing users to create evals before inspecting data, inadequate context for validating judgments, and overly broad evaluator scope.
- The reviewers recommended holding off on the current plugin version until workflow and data review interfaces improve.
Why it matters
Understanding first-party AI evaluation tools helps engineering teams assess automated testing workflows and determine when to adopt native versus custom evaluation frameworks.