Benchmarks

What we test models on

Every verification score comes from open, publicly licensed datasets that anyone can download and check. Each question has one correct answer.

Verification score datasets

Verification suite v2: 80 questions in total, the same for every model, latest run 27 Oct 2026. See also scores by subject.

ODC-BY 1.0

Graduate-level multiple-choice questions across 285 disciplines, written and checked by experts. We use questions marked hard.

MIT

A harder, reasoning-focused successor to MMLU with ten answer options per question across 14 subjects.

Each question lists ten options, A to J. A model guessing at random would score about 10%.
The model is asked to end with ANSWER: <letter>. We compare that letter with the dataset's answer key.
A missing answer, an unreadable answer or an API error counts as wrong.
Questions were sampled once, with fixed random seeds, stratified across subjects. The same questions go to every model.
With 80 questions, scores carry roughly ±10 points of statistical uncertainty. Treat small gaps between models as ties.

Output speed and latency: every model streams the same fixed prompt 3 times; we report the median tokens per second after the first token and the median time to the first token.
End-to-end response time: the median time from sending that prompt to receiving the last token.
Cost per question: what OpenRouter actually billed for the verification questions, divided by the number of questions, reasoning tokens included.
Settings: temperature 0; reasoning effort low for models that support it.

Full details are on the methodology page.