CWE-bench v2: More coverage, quality, and transparency
Summary
- Collinear AI released CWE-bench v2, a public benchmark of 100 tasks covering 91 different Common Weakness Enumerations to test AI model cyber capabilities.
- The benchmark uses real open-source repositories and programmatic verification to deterministically evaluate whether models can find and fix security weaknesses.
- Gemini 4 Argon leads the evaluation with 67.8% pass@1, narrowly outperforming Claude Opus 5.5, while DeepSeek V4.1 Flash offers a lower-cost Pareto frontier option.
- The tasks are formatted as self-contained Dockerized environments via Harbor, enabling both model evaluation and reinforcement learning training for defensive agents.
Why it matters
CWE-bench v2 provides rigorous, reproducible evaluation and training environments for assessing frontier AI models on critical cybersecurity tasks, highlighting current limitations in automated vulnerability remediation.