Data as of · 78 models, 80 questions · reviewed by LLM Verifications team · methodology

Methodology

How every number is produced

Nothing on this site is copied from a vendor. Each figure is computed from requests we sent ourselves, and every one of those requests is kept as a record.

Reviewed by LLM Verifications team · Data as of

1. Calling the models

Our benchmark harness (version 0.1.0) is a single Python script that uses only the standard library. It calls every model through OpenRouter's public chat completions API, the same endpoint any developer can use, with streaming switched on. The model IDs we test are the ones listed in OpenRouter's model catalogue, and each model page on this site shows the exact ID.

Every model gets the same request settings. Quality questions are sent at temperature 0, so the model picks its most likely answer rather than sampling. Models that accept a reasoning setting are sent effort low; models that don't accept one get no reasoning parameter at all. Each quality answer may use up to 4000 output tokens, reasoning included, and each speed run is capped at 700 output tokens. A request that runs longer than the harness's time limit is stopped and recorded as a failure.

For every request the harness records the exact request body, the full streamed response (answer text and any visible reasoning), the time each part arrived, the token counts and which host served it. A moment after each request ends, it fetches OpenRouter's generation record, which holds the native token counts and the amount actually charged.

2. The question suite

Every model answers the same 80 questions from Verification suite v2. The questions are not written by us. They are sampled, with fixed random seeds so the draw can be repeated, from open datasets that anyone can download:

  • SuperGPQA (paper): 50 questions, difficulty = hard, stratified by discipline. By M-A-P, ByteDance Seed and collaborators. Licence ODC-BY 1.0.
  • MMLU-Pro (paper): 30 questions, test split, stratified by subject. By TIGER-Lab, University of Waterloo. Licence MIT.

Every question is multiple choice with ten options, labelled A to J, and exactly one correct answer from the dataset's own answer key. With ten options, a model that guessed at random would score about 10%. The questions fall into 6 subject groups: science (30), engineering (15), medicine (11), economics (9), law (6), humanities & social science (9). Each subject holds only a handful of questions, so subject results are shown as counts (for example 7/11), not percentages. The benchmarks page describes each dataset in more detail.

How an answer is graded

Each prompt ends with the instruction to give the final answer on the last line in the form ANSWER: <letter>. The harness takes the text after the last "ANSWER:" in the reply (so a model can reason first and change its mind), or the last non-empty line if there is none. It then normalises that text: lower case, surrounding quotes, asterisks and backticks removed, spaces collapsed, trailing punctuation dropped. The answer counts as correct only if it matches the answer key's letter. There is no partial credit, no human judging and no second model acting as a grader, so anyone with the response text can re-check the verdict.

A sample question

This is one of the 80 questions exactly as it is asked, from MMLU-Pro (Humanities & social science). The expected answer is E.

Lukianoff and Haidt argue that the disinvitation of certain speakers

A. is a form of cognitive distortion.
B. will eventually lead to a decline in student enrollment.
C. will encourage students to disregard positive aspects of a person's history.
D. will lead to a decline in university reputation.
E. will only serve to promote intellectual homogeneity on college and universities.
F. All of the above.
G. is an example of mental filtering.
H. is a violation of the speaker's freedom of speech.
I. is a proactive measure to safeguard student's mental health.
J. will foster a culture of intolerance and narrow-mindedness on campuses.

Choose the single best option.

Give your final answer on the last line in the form: ANSWER: <letter>

3. Verification score and its uncertainty

The verification score is simply the share of questions a model answered correctly. A sample of 80 questions is enough to separate strong models from weak ones, but not to rank models that are a few points apart. We show that uncertainty instead of hiding it.

The usual way to put a range on a proportion like this is the Wilson score interval. With 80 questions, a model scoring 50% has a 95% interval of 39.3% to 60.7%, about ±10.7 points. At the median score of the models in the latest run, 62.5%, the interval is 51.5% to 72.3%, about ±10.4 points. In plain terms: if two models are within a few points of each other, the order between them could easily flip on a different set of questions.

When we call a winner

The comparison pages only say one model is more accurate than another when a two-proportion z-test on the two scores gives |z| of at least 1.96, the usual threshold for 95% confidence on a two-sided test. Below that, the verdict says the gap is too close to call. For output speed, time to first token and price we use a simpler rule: a model wins only if it is at least 10% better, and anything closer is reported as similar. Rankings in tables still sort by the raw numbers, but the written verdicts follow these rules.

4. Output speed and latency

Each model streams the same fixed prompt 3 times. The prompt asks for about 300 words of plain prose, which gives a long enough answer to measure a steady generation rate:

Explain how a household refrigerator keeps food cold. Cover the refrigerant cycle, the compressor, the condenser and the evaporator. Write about 300 words of plain prose without headings or lists.

Time to first token is measured from sending the request to the first streamed token of any kind, reasoning included, so a model that thinks for ten seconds before answering shows a ten-second latency. Output speed is the completion token count divided by the time between the first token and the end of the stream. We use the native token count from OpenRouter's generation record when it is available, because it counts what the model actually produced in its own tokenizer. End-to-end time is request sent to last token. For each measure we publish the median of the runs, which is less sensitive to a single slow request than the mean.

Some models think without streaming their thinking, then deliver the whole answer within about a second. Dividing hundreds of tokens by a near-zero stream time would produce an impossible speed. The site flags a run as a burst when nothing arrives for at least five seconds, then 300 or more tokens land in under 1.5 seconds. When most of a model's runs are bursts, we leave its output speed blank and still publish its latency and end-to-end time, which remain real measurements.

5. Cost

Cost per question is what OpenRouter actually billed for the quality questions, read from each request's usage or generation record, added up and divided by the number of graded questions. Reasoning tokens are included because you pay for them too, which is why a model with a low list price can still be expensive per question if it thinks at length.

We also check the bill against the list price. For every request, the harness takes the listed per-token price of the host that served it, multiplies it by the native token counts and compares the result with the amount charged. Model pages show both the billed cost and the listed price, and the leaderboard's cost column is always the billed figure. List prices come from OpenRouter's catalogue and can change after a run.

6. Failures, retries and partial runs

  • Retries. A request is attempted up to three times when it fails with HTTP 429 or a 5xx error, or with a host error such as "provider returned error", a timeout, a reset connection or an overloaded host. The harness waits 4 seconds before the second attempt and 8 seconds before the third. Every attempt is counted, and only the final result is graded.
  • Host failures are left out. A request that still fails says nothing about the model's knowledge, so it is excluded from the score rather than counted as wrong. A model that answers but gives no recognisable answer line is counted as wrong. Model pages show how many questions were asked and how many failed.
  • No score for a broken run. If more than 20% of a model's questions fail, or the run stops before every question was asked, the model gets no verification score at all. Its speed, latency and price can still be shown.
  • Budget cap. Each run has a spending limit. Once it is reached the harness starts no new requests; any model left with an incomplete suite gets no score, as above.
  • Merged runs. Not every model is re-tested in every run. Results are merged, and each model keeps the date of the run that actually tested it, shown on its page. If the question suite changes (detected by its hash), older scores from the previous suite are removed, so every score on the site comes from the same questions.

7. The evidence trail

  1. Every request and response is written to its own JSON file: the request body and its SHA-256, the streamed text, the timings, the token counts, the host and the generation ID.
  2. The SHA-256 hash of every one of those files goes into a run manifest, so changing any file afterwards would show.
  3. The question suite and the harness script are hashed too, so anyone can confirm which questions and which code produced a run.

Latest run: 20261027T120517Z, suite SHA-256 cd444ee142dc6b87e74967c9925ea9b7b171b90fe0e18b5a214c4587d964121a, harness SHA-256 6d522eab4cfc8cdb4e18d42fbeef8d7040b83cd3d661100aea022a03321d4625.

8. Reproducing the results

The questions are public: every one comes from the open datasets linked above, under their own licences. The models are public too, through OpenRouter, so anyone with an OpenRouter API key can send the same prompts with the same settings and compare the answers. Because sampling is at temperature 0, most models give the same or a very similar answer on a repeat run, though hosts can differ slightly and models are sometimes updated behind the same ID.

The proof files, the run manifest and the harness itself are not published on this site. They are kept by the LLM Verifications team with each run. If you want to check a specific number, a question's result, or the harness hash above, contact us at support@globesword.com and tell us which model and run you are looking at.

9. Limitations

  • Sample size. 80 questions give the uncertainty described above. Small gaps between models are noise, and subject results with only a few questions each are very noisy.
  • One kind of task. These are expert-level multiple-choice questions. They say a lot about knowledge and reasoning, and nothing directly about coding agents, long documents, writing style or tool use.
  • One location. Speed and latency are measured from one client location. Network distance to the host affects time to first token, so your own numbers may be higher or lower.
  • Routing and hosts. OpenRouter may send a request to any host that serves the model. Hosts differ in speed, and occasionally in quantisation or configuration. We record which host served each request; model pages list them.
  • Prices change. Billed costs are what we paid on the test date. List prices are refreshed from OpenRouter's catalogue and may have moved since.

10. What we don't do

  • No paid placement. Vendors can't pay to be listed or ranked.
  • No numbers copied from model cards or vendor announcements.
  • No hand-editing of results. If a number is wrong, we fix the cause and re-run; see our editorial standards.