Evaluating Governed LLM-Driven ML Experimentation: Lifecycle, Provenance, and Failure Metrics

Impact 7 arXiv cs.AI

Summary

  • Autonomous agents running machine learning experiments are typically evaluated only by their best score, obscuring their experimental process.
  • A new governed harness uses an external control plane to own trial lifecycles, check edit scopes, and record trials in an append-only ledger.
  • Governance metrics like invalid-action rates, repeated-bad rates, and provenance completeness are calculated directly from this ledger.
  • Across six ML tasks and 1,445 trials, the ledger successfully exposed hidden failures, such as unrecorded failed trials and reset issues.
  • The author released the ledgers for all 1,445 trials to allow complete recomputation of reported figures.

Why it matters

Evaluating LLM-driven experimentation processes rather than just outcomes provides necessary oversight, reproducibility, and failure detection for autonomous AI agents in engineering pipelines.

Who should care