OpenAI flags reliability issues in SWE-Bench Pro coding benchmark

OpenAI published an analysis arguing that SWE-Bench Pro—a widely cited coding benchmark—has reliability problems that muddy how the industry evaluates AI coding models. The stated goal is to 'separate signal from noise' in coding evaluations, distinguishing genuine capability improvements from benchmark artifacts.

The timing is pointed: this same week, xAI claimed Grok 4.5 is 'Opus-class' partly on benchmark results, and community threads touted Claude Sonnet 5's reported SWE-Bench leap (92.4% vs Opus 4.6's 80.8%). By questioning benchmark reliability, OpenAI both hedges against rivals' benchmark boasts and stakes out a methodological high ground—though critics will note it's convenient for OpenAI to cast doubt on benchmarks precisely when competitors are winning them.

Mechanically, coding benchmarks like SWE-Bench Pro can suffer from data contamination (test problems leaking into training data), ambiguous grading, and environment flakiness—all of which inflate or distort scores. OpenAI's analysis reportedly probes these failure modes. Competitively, this feeds a genuine developer concern echoed in HN threads about over-reliance on leaderboard numbers, and it's adjacent to Cognition's SWE-1.7 buzz (257 HN points) about models nearing GPT-5.5/Opus intelligence. Skeptics will read it as benchmark-politics. Watch whether the benchmark's maintainers respond and whether the industry adopts more robust coding evals as a result.