Score vs. cost per question
OpenAI models in color, every other model in grey
Provider
13 models tested, compared with 65 models from other labs.
OpenAI#3GPT-5.3-Codex82.5% verification score95 tok/s1.4 s first token$0.01 / questionOpenAI#12GPT-6 Astra76.2% verification score49 tok/s1.3 s first token$0.013 / questionOpenAI#12GPT-6.1 Terra Pro76.2% verification score— tok/s8.5 s first token$0.005 / questionOpenAI#12GPT-6.1 Sol76.2% verification score41 tok/s2.1 s first token$0.0029 / questionOpenAI#15GPT-5.6 Terra75.0% verification score101 tok/s1.1 s first token$0.0063 / questionOpenAI#18GPT-6.1 Terra73.8% verification score122 tok/s2.4 s first token$0.0013 / questionOpenAI#18GPT-6 Sol73.8% verification score92 tok/s0.9 s first token$0.0024 / questionOpenAI#21o372.5% verification score163 tok/s1.2 s first token$0.0088 / questionOpenAI#21GPT-5.472.5% verification score55 tok/s1.7 s first token$0.011 / questionOpenAI#21GPT-6 Luna72.5% verification score137 tok/s1.2 s first token$0.00025 / questionOpenAI#41gpt-oss-120b58.8% verification score82 tok/s0.5 s first token$0.00013 / questionOpenAI#46gpt-oss-20b51.2% verification score103 tok/s0.7 s first token$0.000066 / questionOpenAI models in color, every other model in grey
OpenAI models in color, every other model in grey
OpenAI models highlighted against every model in this run · higher is better
OpenAI models highlighted against every model in this run · higher is better
OpenAI models highlighted against every model in this run · lower is better
OpenAI models highlighted against every model in this run · lower is better
OpenAI models highlighted against every model in this run · lower is better
OpenAI models highlighted against every model in this run · higher is better
| # | ||||||||
|---|---|---|---|---|---|---|---|---|
| 1 | GPT-5.3-Codex OpenAI · served by OpenAI | 82.5 | 66 / 80 | 95 | 1.4 s | $0.01 | $1.75 / $14 | 400k |
| 2 | GPT-6 Astra OpenAI · served by OpenAI | 76.2 | 61 / 80 | 49 | 1.3 s | $0.013 | $10 / $50 | 1.1M |
| 3 | GPT-6.1 Terra Pro OpenAI · served by OpenAI | 76.2 | 61 / 80 | — | 8.5 s | $0.005 | $1 / $5 | 1.1M |
| 4 | GPT-6.1 Sol OpenAI · served by OpenAI | 76.2 | 61 / 80 | 41 | 2.1 s | $0.0029 | $2 / $10 | 1.1M |
| 5 | GPT-5.6 Terra OpenAI · served by OpenAI | 75.0 | 60 / 80 | 101 | 1.1 s | $0.0063 | $2 / $12 | 1.1M |
| 6 | GPT-6.1 Terra OpenAI · served by OpenAI | 73.8 | 59 / 80 | 122 | 2.4 s | $0.0013 | $1 / $5 | 1.1M |
| 7 | GPT-6 Sol OpenAI · served by OpenAI | 73.8 | 59 / 80 | 92 | 0.9 s | $0.0024 | $2 / $10 | 1.1M |
| 8 | o3 OpenAI · served by OpenAI | 72.5 | 58 / 80 | 163 | 1.2 s | $0.0088 | $2 / $8 | 200k |
| 9 | GPT-5.4 OpenAI · served by OpenAI | 72.5 | 58 / 80 | 55 | 1.7 s | $0.011 | $2.5 / $15 | 1.1M |
| 10 | GPT-6 Luna OpenAI · served by OpenAI | 72.5 | 58 / 80 | 137 | 1.2 s | $0.00025 | $0.1 / $0.5 | 1.1M |
| 11 | gpt-oss-120b OpenAI · served by CoreWeave | 58.8 | 47 / 80 | 82 | 0.5 s | $0.00013 | $0.15 / $0.6 | 131k |
| 12 | gpt-oss-20b OpenAI · served by Darkbloom | 51.2 | 41 / 80 | 103 | 0.7 s | $0.000066 | $0.018 / $0.09 | 131k |
| 13 | GPT-5.5 Pro OpenAI · served by OpenAI | — | 0 / 0 | — | 33.8 s | — | $30 / $180 | 1.1M |