Systematic tests and benchmarks that measure AI model performance, safety and behaviour across tasks, inputs and scenarios.
Also known as: Evaluations, Evaluation suite, Evaluation harness
In plain English
Evals are like exams or product reviews for AI: they check how well a model performs on specific tasks. Companies or researchers run them using example questions, user-style prompts or simulated problems to find strengths and weaknesses.
In practice
In procurement and deployment, teams use evals to compare vendors, set acceptance thresholds and detect risks such as bias or hallucinations. Evaluation results inform integration decisions, monitoring plans, SLAs and whether further mitigation or fine-tuning is required.
Under the hood
Evals combine labelled datasets, scenario suites, automated harnesses and metrics (accuracy, F1, calibration, safety scores) to quantify behaviour. They can be black-box or white-box, include human annotation and adversarial cases, and use statistical analysis to estimate reliability and uncertainty. Continuous evaluation (CI/CD), benchmark comparisons and human-in-the-loop judgement are common parts of an evaluation workflow.
Example
Let's add those new safety tests to our evals before we move the model to production.