AI Benchmark Credibility Crisis Emerges as SWE-bench Verification Fails

SWE-bench Verified is effectively dead as a benchmark after OpenAI discontinued reporting on it in February 2026, with the AI community questioning what reliable coding evaluations even exist anymore. AWS research reveals that ReAct-style agents waste 90.8% of retry budgets on hallucinated tool calls across 200-task benchmarks, with the issue stemming from architectural flaws rather than model mistakes. A comprehensive analysis shows top benchmarks can be gamed, leading to debates about the true measure of agent reliability and the need for new evaluation standards.