Make the model prove it.
A beginner-friendly tour of openai/evals: the open-source framework and registry for testing language models and systems built with them.
A model can feel better and still regress.
Human spot-checks are useful, but they are inconsistent, slow, and easy to steer with a handful of impressive examples.
An eval is a small contract: fixed inputs, a defined system under test, and a scoring rule you can run again after a prompt, model, tool, or retrieval change.
Five moving parts. One loop.
Think of Evals as a test harness around an LLM system. Click a node to unpack its job.
The repo is a map, not a monolith.
The important split is between the engine, the registry, and your test material. Click a path.
Runtime primitives
Core interfaces, metrics, records, task state, and the CLI live here. The repository can run a registered eval against a completion function.
Start with the simplest signal that works.
When the answer is constrained
Match, Includes, FuzzyMatch, and JSON matching cover cases where the desired output has little variation.
Inspect completions first. A clean-looking metric can hide a badly specified task.
# conceptual scoring sample["ideal"] = "42" sample["input"] = "What is 6 × 7?" Match: output starts with an ideal answer Includes: output contains an ideal answer FuzzyMatch: either answer contains the other
YAML is the wiring diagram.
A registry entry binds a friendly name to a versioned eval, its implementation, and the data or parameters it needs. Click a line.
oaieval gpt-… arithmetic resolves through this friendly name.The run is a traceable event loop.
Every sample moves through the same small sequence. That makes failure inspection possible instead of leaving you with one mysterious score.
Resolve eval name, class, and args.
Read one JSONL object per datapoint.
Generate a completion from the model or system.
Capture sampling, match, error, and extra events.
Return metrics such as accuracy.
Same eval. Different brains.
A Completion Function standardises the input and output contract so the same eval can exercise a plain model, a retrieval system, or a tool-using agent.
Direct model
Receives a string or chat conversation, returns a list of text completions. That is enough for Evals to run it against a registered task.
A good metric is not enough.
The repository’s contribution guidance points to four quality checks. Together they turn a test from a demo into a useful instrument.
Consistent
Samples revolve around one use case, domain, or failure mode.
Challenging
There is room for models to fail, and humans could still reason about the task.
Directionally clear
References or rubrics make “better” behaviour legible.
Carefully crafted
Prompts, templates, and results have been inspected and iterated.
Product leadership lens: an eval suite is a living definition of quality. It should evolve with the product, not sit beside it as a one-off benchmark.
From zero to first signal.
The shortest useful path is six deliberate steps. Use an existing template first; custom code is a later move.
Put the runner in your environment.
For running existing evals, install the package. For creating evals, clone the repository and use an editable install so changes are reflected immediately.
# run existing evals pip install evals # or develop from a clone git clone https://github.com/openai/evals cd evals pip install -e .
Data + YAML + CLI.
A minimal eval is just a dataset, a registry entry, and a run command. This is the “no custom code yet” path.
For basic evals, samples usually need input and ideal. For model-graded evals, the required keys are determined by the prompt and its arguments.
# data: samples.jsonl {"input": "6 × 7?", "ideal": "42"} # registry: arithmetic.yaml arithmetic: id: arithmetic.dev.match-v1 metrics: [accuracy] arithmetic.dev.match-v1: class: evals.elsuite.basic.match:Match args: samples_jsonl: arithmetic/samples.jsonl # run oaieval gpt-3.5-turbo arithmetic
The sharp edges are part of the design.
The README warns that API calls made during eval runs incur costs. Track run cost alongside quality if the suite will run often.
If you change prompts, samples, or scoring rules, bump the eval version so comparisons remain meaningful.
A top-line accuracy can hide formatting failures, invalid model-graded choices, or a narrow dataset. Local JSONL logs are available by default.
Public contributions require rights to upload the data and are licensed under the repository’s MIT terms. Use private evals for sensitive workflow data.
Current-repo note: the README says Evals can now be configured and run directly in the OpenAI Dashboard, while this repository documents the open-source CLI and registry workflow.
Quality is a habit, not a vibe.
Start with one failure mode you care about. Make it measurable. Run it on every meaningful change. Then let the failures teach you what the product actually needs.
This field guide is grounded in the repository’s README and docs at the cloned snapshot, commit 8eac7a7 (14 April 2026).