Verdict
Built from the measurements below
- Nemotron 3 Ultra is more accurate: 60.0% against 43.8% on the same 80 questions, a 16.2-point gap that is statistically significant.
- Nemotron 3.5 Lightning is 12× cheaper per question as billed ($0.00049 vs $0.0057), reasoning tokens included.
- Nemotron 3 Ultra generates text 1.2× faster: 136 tokens per second against 116.
- Nemotron 3.5 Lightning starts answering sooner: 0.5 s to the first token, against 0.8 s for Nemotron 3 Ultra.
Which should you choose?
Choose Nemotron 3.5 Lightning for the lower cost per correct answer ($0.0011 against $0.0094) and a quicker first token. Choose Nemotron 3 Ultra for the most accurate answers of the two and faster streaming. The widest subject gap is in humanities & social science: Nemotron 3 Ultra answered 9 of 9 correctly, Nemotron 3.5 Lightning 6 of 9. With so few questions per subject, treat this as a lead to test on your own prompts, not a result.
Accuracy gap -16.2 points (z = -2.06). We call a winner only when z is at least 1.96, the usual 95% threshold. Speed, latency and price need a 10% margin.