news
Separating signal from noise in coding evaluations
July 8, 2026
OpenAI published a new analysis arguing that SWE-Bench Pro, a popular coding benchmark, has reliability and accuracy issues in how it evaluates AI models. The finding matters because benchmark flaws can distort model rankings and make coding gains look stronger or weaker than they really are.
A new analysis from OpenAI reveals issues in SWE-Bench Pro, a popular coding benchmark, raising concerns about reliability and accuracy in evaluating AI models.
Source: openai.com