Benchmark Credibility Crisis: SWE-bench Contamination Alleged; 'SusVibes' Exposes Vibe-Coding Security Risks in Agentic Pipelines

A new arXiv paper alleges SWE-bench contamination via 'instance-specific memorization,' prompting practitioners to call for its retirement as a meaningful coding benchmark. Separately, a new 'SusVibes' benchmark exposed serious security vulnerabilities in agentic coding workflows like SWE-agent, sparking heated debate about the safety of autonomous developer pipelines as they proliferate in enterprise settings. The two stories together are accelerating a broader practitioner-led push to question whether current leaderboards have any meaningful correlation with real-world deployment quality. Developer polls on X continue to show Claude (Opus 4, Sonnet 4.5) dominating for real-world coding tasks, while Grok 4 draws mixed reactions despite strong reasoning benchmark scores.