Score vs. cost per question
NVIDIA models in color, every other model in grey
Provider
3 models tested, compared with 75 models from other labs.
NVIDIA#38Nemotron 3 Ultra60.0% verification score136 tok/s0.8 s first token$0.0057 / questionNVIDIA#51Nemotron 3.5 Lightning43.8% verification score116 tok/s0.5 s first token$0.00049 / questionNVIDIA#58Nemotron 3.5 Super VL40.0% verification score715 tok/s0.6 s first token$0.0013 / questionNVIDIA models in color, every other model in grey
NVIDIA models in color, every other model in grey
NVIDIA models highlighted against every model in this run · higher is better
NVIDIA models highlighted against every model in this run · higher is better
NVIDIA models highlighted against every model in this run · lower is better
NVIDIA models highlighted against every model in this run · lower is better
NVIDIA models highlighted against every model in this run · lower is better
NVIDIA models highlighted against every model in this run · higher is better
| # | ||||||||
|---|---|---|---|---|---|---|---|---|
| 1 | Nemotron 3 Ultra NVIDIA · served by BaseTen | 60.0 | 48 / 80 | 136 | 0.8 s | $0.0057 | $0.6 / $2.4 | 262k |
| 2 | Nemotron 3.5 Lightning NVIDIA · served by Io Net | 43.8 | 35 / 80 | 116 | 0.5 s | $0.00049 | $0.07 / $0.2 | 262k |
| 3 | Nemotron 3.5 Super VL NVIDIA · served by Darkbloom | 40.0 | 32 / 80 | 715 | 0.6 s | $0.0013 | $0.15 / $0.5 | 1M |