# SWE-bench Verified

> Fixing real bugs in open-source Python projects so the tests pass.

Source: https://www.superintelligence.com.au/benchmarks/swe-bench-verified/
Data updated 2026-10-07. Benchmark data from Epoch AI (CC BY) and official leaderboards.

Best score today: 84% (+7 pts in 12 months).
Human comparison: There is no published human score. Each task is a real fix that a human developer already made, so in a sense people have solved all of them, but nobody has timed a group of engineers sitting the whole set.

## What it tests

Real bug reports from popular open-source Python projects. The model must change the code so the project’s own tests pass. The Verified set is a subset of 500 tasks that human engineers checked were clear and solvable.

## Why it matters

It has been the most widely quoted measure of whether AI can do a software engineer’s everyday work.

## Caveats

Scores depend heavily on the tools and scaffolding around the model, not just the model itself. Every task comes from a public repository, so some solutions may have been in a model’s training data. In February 2026 OpenAI, which produced the Verified set, stopped reporting scores on it. Its audit found many tasks whose tests reject correct fixes, and signs that leading models had seen the solutions in training. Read the top of this chart with that in mind.

## Best score by quarter

| Quarter | Best score | Record held by |
| --- | --- | --- |
| Q4 2024 | 31% | GPT-4o (Nov 2024) |
| Q1 2025 | 61% | claude-3-7-sonnet-20250219 |
| Q2 2025 | 71% | Claude Opus 4 |
| Q3 2025 | 74% | GPT-5 (high) |
| Q4 2025 | 77% | Claude Opus 4.5 (no thinking) |
| Q1 2026 | 79% | Claude Opus 4.6 (no thinking) |
| Q2 2026 | 84% | Claude Opus 4.7 (max) |
| Q3 2026 | 84% | Claude Opus 4.7 (max) |
| Q4 2026 | 84% | Claude Opus 4.7 (max) |

## Top scores right now

| Rank | Model | Developer | Score | Released |
| --- | --- | --- | --- | --- |
| 1 | Claude Opus 4.7 (max) | Anthropic | 84% | 2026-04-16 |
| 2 | GPT-5.5 (xhigh) | OpenAI | 81% | 2026-04-23 |
| 3 | Gemini 3.5 Flash | Google DeepMind | 79% | 2026-05-19 |
| 4 | Claude Opus 4.6 (no thinking) | Anthropic | 79% | 2026-02-05 |
| 5 | GLM-5.2 | Z.ai (Zhipu AI) | 79% | 2026-06-16 |
| 6 | DeepSeek v4 Pro (max) | DeepSeek | 78% | 2026-04-24 |
| 7 | Qwen3.7 Max | Alibaba | 77% | 2026-05-19 |
| 8 | GPT-5.4 (high) | OpenAI | 77% | 2026-03-05 |
| 9 | Claude Opus 4.5 (no thinking) | Anthropic | 77% | 2025-11-24 |
| 10 | Kimi K2.6 | Moonshot | 77% | 2026-04-20 |
| 11 | Qwen 3.6 Max (Preview) | Alibaba | 77% | 2026-04-20 |
| 12 | Gemini 3.1 Pro Preview | Google DeepMind | 76% | 2026-02-19 |
| 13 | Gemini 3 Flash Preview | Google DeepMind | 75% | 2025-12-17 |
| 14 | Claude Sonnet 4.6 (no thinking) | Anthropic | 75% | 2026-02-17 |
| 15 | GPT-5.3 Codex (high) | OpenAI | 75% | 2026-02-05 |
| 16 | GLM-5.1 | Z.ai (Zhipu AI) | 74% | 2026-04-07 |
| 17 | GPT-5.2 (high) | OpenAI | 74% | 2025-12-11 |
| 18 | Kimi K2.5 | Moonshot | 74% | 2026-01-27 |
| 19 | GPT-5 (high) | OpenAI | 74% | 2025-08-07 |
| 20 | claude-opus-4-1-20250805 | Anthropic | 73% | 2025-08-05 |
| 21 | Gemini 3 Pro Preview | Google DeepMind | 73% | 2025-11-18 |
| 22 | GLM-5 | Z.ai (Zhipu AI) | 72% | 2026-02-11 |
| 23 | GPT-5 (medium) | OpenAI | 72% | 2025-08-07 |
| 24 | Claude Sonnet 4.5 (no thinking) | Anthropic | 71% | 2025-09-29 |
| 25 | Claude Opus 4 | Anthropic | 71% | 2025-05-22 |
| 26 | GPT-5.1 (high) | OpenAI | 68% | 2025-11-13 |
| 27 | GPT-5 mini (medium) | OpenAI | 65% | 2025-08-07 |
| 28 | o3 (medium) | OpenAI | 62% | 2025-04-16 |
| 29 | claude-3-7-sonnet-20250219 | Anthropic | 61% | 2025-02-24 |
| 30 | Qwen 3.6 Plus (2026-04-02) | Alibaba | 58% | 2026-03-31 |
| 31 | gemini-2.5-pro | Google DeepMind | 58% | 2025-06-17 |
| 32 | GPT-4.1 | OpenAI | 49% | 2025-04-14 |
| 33 | GPT-4o (Nov 2024) | OpenAI | 31% | 2024-11-20 |

Download all data: https://www.superintelligence.com.au/data/benchmarks.csv