Score vs. cost per question
Google models in color, every other model in grey
Provider
6 models tested, compared with 72 models from other labs.
Google#2Gemini 4 Argon84.8% verification score69 tok/s3.3 s first token$0.0063 / questionGoogle#3Gemini 3.8 Flash82.5% verification score142 tok/s3.2 s first token$0.0037 / questionGoogle#9Gemini 3.1 Pro Preview77.5% verification score124 tok/s3.4 s first token$0.017 / questionGoogle#15Gemini 3.5 Flash Lite75.0% verification score409 tok/s1.1 s first token$0.0034 / questionGoogle#64Gemma 4 31B30.0% verification score58 tok/s0.6 s first token$0.0014 / questionGoogle#71Gemma 4 26B A4B15.0% verification score172 tok/s0.7 s first token$0.0011 / questionGoogle models in color, every other model in grey
Google models in color, every other model in grey
Google models highlighted against every model in this run · higher is better
Google models highlighted against every model in this run · higher is better
Google models highlighted against every model in this run · lower is better
Google models highlighted against every model in this run · lower is better
Google models highlighted against every model in this run · lower is better
Google models highlighted against every model in this run · higher is better
| # | ||||||||
|---|---|---|---|---|---|---|---|---|
| 1 | Gemini 4 Argon Google · served by Google | 84.8 | 67 / 79 | 69 | 3.3 s | $0.0063 | $2 / $10 | 1M |
| 2 | Gemini 3.8 Flash Google · served by Google | 82.5 | 66 / 80 | 142 | 3.2 s | $0.0037 | $0.825 / $4.125 | 1M |
| 3 | Gemini 3.1 Pro Preview Google · served by Google | 77.5 | 62 / 80 | 124 | 3.4 s | $0.017 | $2 / $12 | 1M |
| 4 | Gemini 3.5 Flash Lite Google · served by Google | 75.0 | 60 / 80 | 409 | 1.1 s | $0.0034 | $0.3 / $2.5 | 1M |
| 5 | Gemma 4 31B Google · served by CoreWeave | 30.0 | 24 / 80 | 58 | 0.6 s | $0.0014 | $0.2 / $0.4 | 262k |
| 6 | Gemma 4 26B A4B Google · served by Darkbloom | 15.0 | 12 / 80 | 172 | 0.7 s | $0.0011 | $0.09 / $0.3 | 262k |