ReplayRCA: Measuring Post-hoc Diagnosis and Live Oversight of Agent Failures by Counterfactual Replay
Summary
- Researchers introduce ReplayRCA, a benchmark that measures post-hoc diagnosis and live oversight of language-model agent failures via counterfactual replay.
- The benchmark scores diagnoses by intervention in OpsWorld, a deterministic back-office environment, evaluating models across post-hoc autopsy and live oversight tracks.
- Results show Grok 4.7 achieves the highest causal repair score of +51.3 points among 13 frontier models, while DeepSeek V4.1 Flash performs strongly as a live supervisor.
- The study demonstrates that post-hoc diagnostic performance can be a poor proxy for live oversight effectiveness.
Why it matters
Evaluating AI agent failures through counterfactual intervention provides a more rigorous measure of reliability than annotation agreement, highlighting disparities between post-hoc diagnosis and real-time supervision.