Evals

Systematic tests and benchmarks that measure AI model performance, safety and behaviour across tasks, inputs and scenarios.

Also known as: Evaluations, Evaluation suite, Evaluation harness

In plain English

Evals are like exams or product reviews for AI: they check how well a model performs on specific tasks. Companies or researchers run them using example questions, user-style prompts or simulated problems to find strengths and weaknesses.

In practice

In procurement and deployment, teams use evals to compare vendors, set acceptance thresholds and detect risks such as bias or hallucinations. Evaluation results inform integration decisions, monitoring plans, SLAs and whether further mitigation or fine-tuning is required.

Under the hood

Evals combine labelled datasets, scenario suites, automated harnesses and metrics (accuracy, F1, calibration, safety scores) to quantify behaviour. They can be black-box or white-box, include human annotation and adversarial cases, and use statistical analysis to estimate reliability and uncertainty. Continuous evaluation (CI/CD), benchmark comparisons and human-in-the-loop judgement are common parts of an evaluation workflow.

Example

Let's add those new safety tests to our evals before we move the model to production.

Related terms

Hallucination · Human-in-the-loop · Prompt Engineering

•