The most misunderstood graph in AI just got a major update

Impact 8 MIT Technology Review AI

Summary

  • A new report from the AI evaluation nonprofit METR updates its time horizon plot, showing AI capabilities continuing on an exponential trajectory.
  • The updated plot incorporates more open-ended tasks and challenges requiring judgment, such as designing AI evaluations and scoring philosophical arguments.
  • While frontier models excel at verifiable domains like coding and math, they currently fall short of human performance on these open-ended, judgment-laden tasks.
  • Evaluating open-ended tasks is difficult and costly, often relying on estimated completion times and subjective ratings rather than automated checks.
  • Experts emphasize that while adding open-ended evaluations is a welcome step, predicting progress on automating complex AI research remains challenging.

Why it matters

Understanding AI capabilities on open-ended and judgment-heavy tasks is crucial for determining how close the industry is to automating AI research and development, though current evaluation methods face practical scaling hurdles.

Who should care