The most misunderstood graph in AI just got a major update
Summary
- A new report from the AI evaluation nonprofit METR updates its time horizon plot, showing AI capabilities continuing on an exponential trajectory.
- The updated plot incorporates more open-ended tasks and challenges requiring judgment, such as designing AI evaluations and scoring philosophical arguments.
- While frontier models excel at verifiable domains like coding and math, they currently fall short of human performance on these open-ended, judgment-laden tasks.
- Evaluating open-ended tasks is difficult and costly, often relying on estimated completion times and subjective ratings rather than automated checks.
- Experts emphasize that while adding open-ended evaluations is a welcome step, predicting progress on automating complex AI research remains challenging.
Why it matters
Understanding AI capabilities on open-ended and judgment-heavy tasks is crucial for determining how close the industry is to automating AI research and development, though current evaluation methods face practical scaling hurdles.