How AI progress is measured

What a benchmark is

Think of a benchmark as an exam that never changes. Researchers write a set of questions or tasks, each with an answer that can be checked, and every AI model sits the same exam. That makes it possible to compare models from different companies, and to compare this year’s models with last year’s.

Some benchmarks test knowledge, like graduate-level science. Others test skills, like fixing real software bugs or solving puzzles the model has never seen before.

See it in the explorerWatch five benchmarks climb, quarter by quarter.

The site tracks five of them, each aimed at a different kind of ability.

• GPQA Diamond (details) asks hard science questions written by experts, designed to be difficult even with a web search.
• SWE-bench Verified (details) gives a model real software problems from open projects and checks whether its fix works.
• FrontierMath (details) uses new, difficult mathematics problems written by professional mathematicians.
• Humanity’s Last Exam (details) collects very hard questions from experts across many subjects.
• ARC-AGI-2 (details) tests whether a model can work out a new pattern from a few examples.

Why tests wear out

When a new benchmark launches, models usually score poorly. Over the following years scores climb, often faster than the people who wrote the test expected. Once the best models get nearly everything right, the benchmark is called saturated. It can no longer separate a good model from a great one, so researchers move on to something harder.

This is a sign that the tool did its job, not that it failed. But it means no single benchmark can follow progress for long. The field keeps building new tests, which is why the explorer tracks several at once and why newer ones often carry ambitious names.

There is a second cause of wear. Once a test is famous, companies have reason to do well on it, and the score starts to matter for its own sake. Researchers sometimes describe this as a measure ceasing to be a good measure once it becomes a target.

A recurring pattern is that a test thought to be far out of reach turns out to be closer than expected. This is part of why forecasts about AI have been so hard, and why people who follow the field pay attention to how quickly each new test is being used up.

The contamination problem

Imagine a student who has seen the exam questions and answers before the test. Their score would be high, but it would say little about what they understand. Something similar can happen with AI models, and it is called data contamination.

Models learn from enormous amounts of text collected from the internet. If a benchmark’s questions or answers are published online, they may end up in that training material, and the model may then reproduce them rather than work them out. Nobody has to cheat on purpose for this to happen.

Test designers respond in several ways:

• Keeping part of the question set private, so it cannot leak into training data.
• Writing fresh problems that did not exist when the model was trained.
• Checking whether a model’s answers look memorised rather than reasoned.

These steps help, though none is perfect. It is one reason to give more weight to tests with hidden or newly written questions.

Reading a score well

A score is only meaningful with context. Before reading much into a number, it helps to ask a few plain questions.

• What does the test actually measure? A high mark on a science quiz does not mean a model can do a scientist’s job.
• How was it run? Results can change depending on how many attempts the model gets, whether it can use tools, and how much thinking time it is given.
• Who reported it? A result checked by an independent group is usually more reliable than one a company reports about its own model.
• What is the trend? One number is a snapshot. A line over time across several tests is more informative.

Scores also do not translate neatly into real-world usefulness. A model can do well on a test and still fail at an ordinary task, or the reverse. The benchmark explorer shows live numbers, so this page does not quote scores that would go out of date.

It is also fair to keep in mind what a benchmark leaves out. Most are narrow, run on a computer and marked automatically. They rarely capture judgement, teamwork, reliability over weeks, or knowing when to ask for help, all of which matter in real work.

Comparing with people

Many benchmarks include a human baseline, which is how well people perform on the same test. It gives the numbers a reference point, such as whether a model has passed the typical expert or only the typical student.

Baselines need care too. The people tested might be experts, volunteers or crowd workers, and they may have had limited time or no tools. A different group could produce a different baseline. Treat it as a useful landmark rather than a fixed line.

What to take from this

Measurement in AI is imperfect, and researchers say so themselves. Even so, tests are the best shared way we have to see whether systems are getting better and where they still struggle. Used carefully, across several benchmarks and over time, they give a fair picture.

If you want to understand why scores have climbed as they have, the next guide, Scaling, simply, explains the main ingredient behind that progress.

In practice that means being wary of two opposite mistakes. One is treating a single impressive score as proof that a system can do anything. The other is dismissing all testing because it is imperfect. A steady, sceptical reading sits between them.