# Benchmark saturation

> When top models score close to the maximum on a test, so it can no longer tell them apart.

Source: https://www.superintelligence.com.au/glossary/benchmark-saturation/

Benchmark saturation happens when the best AI models score so close to the top mark that the test stops being useful. If several models all get 95% or more, the remaining differences may be noise, mistakes in the questions, or luck, rather than real differences in ability.

Picture a school spelling test that every student in the class passes with full marks. It no longer shows who the strongest speller is. The same thing has happened repeatedly with AI tests: a benchmark that once looked hard becomes easy within a few years, and researchers have to build a harder one.

This matters for how progress is judged. Saturation can make progress look like it has stalled when the test has simply run out of room. It can also hide real gaps, because a high score on an old test says little about tasks the test never covered. Some researchers also worry that saturation is sped up when test questions leak into training material. There is no single agreed threshold for calling a test saturated, and people judge it differently.

Related: https://www.superintelligence.com.au/glossary/benchmark.md, https://www.superintelligence.com.au/glossary/human-baseline.md, https://www.superintelligence.com.au/glossary/data-contamination.md, https://www.superintelligence.com.au/glossary/evaluation.md