# Evaluation

> The process of testing what an AI system can do and how safely it behaves, often shortened to an eval.

Source: https://www.superintelligence.com.au/glossary/evaluation/

An evaluation, often called an eval, is any structured test of an AI system. It might measure knowledge or skill, such as maths and coding, or behaviour, such as whether the model refuses dangerous requests, tells the truth, or can be tricked into breaking its rules. A benchmark is one kind of evaluation with fixed questions and scoring.

Evaluations matter because they are how anyone, inside or outside a company, learns what a model can do before and after release. Some are run by the developers, some by independent groups, and some by government-backed institutes. Safety evaluations try to detect risky abilities early, for example help with building weapons or hacking systems.

They are hard to do well. A model may perform differently depending on how it is asked, so a poor result can reflect a poor test rather than a poor model. A clean result may simply mean the test missed something. Researchers also disagree about how far evaluations can predict real-world behaviour, and about whether outsiders should be given access to test models before they are released.

Related: https://www.superintelligence.com.au/glossary/benchmark.md, https://www.superintelligence.com.au/glossary/human-baseline.md, https://www.superintelligence.com.au/glossary/data-contamination.md, https://www.superintelligence.com.au/glossary/frontier-lab.md