New Artificial Analysis index widens the gap between open and closed models
Summary
- Artificial Analysis overhauled its Intelligence Index to version 5, utilizing 60 percent private test sets to prevent benchmaxxing.
- The update widened the performance gap between open models and the top of the leaderboard from 3 points to 12 points.
- Kimi K3.1 Pro dropped five spots to ninth place, while Grok 4.7 lost 12 points in the updated rankings.
- AI researcher Nathan Lambert noted that reinforcement learning infrastructure gaps also contribute to performance disparities across domains.
Why it matters
Changes in AI benchmarking methodologies heavily influence how open and closed models are perceived, highlighting the growing reliance on private test sets to measure true generalization.