Methodology
How every number in the benchmark explorer is produced, from the source file to the chart.
Source
All scores come from Epoch AI’s Benchmarking Hub, downloaded as a single file from https://epoch.ai/data/benchmark_data.zip. Epoch runs some benchmarks itself and collects others from official leaderboards. The data is published under CC BY 4.0, and so is the processed version on this site.
Suggested citation for the source data:
Epoch AI, 'Capabilities & benchmarking'. Published online at epoch.ai. Retrieved from 'https://epoch.ai/benchmarks' [online resource].
The five benchmarks
- GPQA Diamond: Graduate-level biology, chemistry and physics questions that are hard to look up. Series starts Q1 2023. About the test.
- SWE-bench Verified: Fixing real bugs in open-source Python projects so the tests pass. Series starts Q4 2024. About the test.
- FrontierMath: Unpublished research-level maths problems with answers a computer can check. Series starts Q1 2024. About the test.
- Humanity’s Last Exam: Thousands of expert questions across 100+ subjects, chosen to stump AI. Series starts Q3 2024. About the test.
- ARC-AGI-2: Visual grid puzzles that are easy for people and new to every model. Series starts Q3 2024. About the test.
How “best score” is worked out
- Every result for a benchmark is placed in the calendar quarter in which the model was released.
- For each quarter, the line shows the highest score any model released up to the end of that quarter had achieved. It is a running record, so it can stay flat but never falls.
- The line runs to the current quarter. If no new model beat the record this quarter, the previous record carries forward.
- Scores are stored to one decimal place and shown as whole percentages.
- The 12-month change compares today’s record with the record four quarters earlier. If the series is shorter than that, the explorer says so instead of showing a change.
Why release date, not test date
Epoch often re-tests older models on newer versions of a benchmark. If the chart used the date a test was run, a model released in 2024 could appear in 2026 and the history would flatten out. Using the model’s release date shows what AI could do at each point in time. The leaderboard’s “Released” column uses the same date.
Human comparisons
The dashed “Compare with humans” line appears only where a benchmark’s authors published a single human score for the same questions, and the card links to that source. Where no such figure exists, the card says why rather than estimating one.
Updates
The data refreshes automatically every Monday. The site is rebuilt only when the numbers have changed, and a failed or malformed download leaves the last good data in place. Last updated .
Download
benchmarks.csv has one row per benchmark per quarter, with the columns benchmark, quarter, quarter_end, best_score_pct and record_model. Please credit Epoch AI as the original source.
Limits
- A score reflects the setup used to test the model, including any tools and how much computing it was allowed. Different setups can give different results for the same model.
- Some results are reported by the developer and not independently checked.
- Models that were never tested on a benchmark do not appear in its history.