Counting the Bandages: On measuring the time that AI saves
Summary
- This research paper examines how to accurately measure the time AI assistance actually saves, contrasting a simple stopwatch view with a ledger view that accounts for errors and defects.
- The author introduces a model-free pair of quantities—time saved and excess escaped defects—and a break-even defect cost decision number.
- Replication of METR's developer study shows reported slowdowns ranging from 19% to 24% depending on the chosen estimand.
- A small pre-specified experiment demonstrated that AI reviewers adjusted checking effort based on perceived drafter error rates, where lower perceived error led to faster work but increased error rates.
Why it matters
Properly evaluating AI productivity gains requires accounting for verification overhead and the cost of escaped errors. This paper provides formal models and empirical re-evaluations that challenge simplistic stopwatch-based metrics.