Verdict
Built from the measurements below
- Accuracy is too close to call: Mercury 2.5 scored 58.2% and Nemotron 3.5 Lightning 43.8% on the same 79 questions, a gap within what chance alone could produce.
- Mercury 2.5 is 4.3× cheaper per question as billed ($0.00011 vs $0.00049), reasoning tokens included.
- Mercury 2.5 generates text 1.4× faster: 157 tokens per second against 116.
- Nemotron 3.5 Lightning starts answering sooner: 0.5 s to the first token, against 2.3 s for Mercury 2.5.
Which should you choose?
Choose Mercury 2.5 for the lower cost per correct answer ($0.00019 against $0.0011), faster streaming and shorter outputs (about 673 tokens per question against 2,846). Choose Nemotron 3.5 Lightning for a quicker first token, open weights you can run yourself and more hosts to choose from on OpenRouter (4 against 1). The widest subject gap is in law: Mercury 2.5 answered 3 of 6 correctly, Nemotron 3.5 Lightning 1 of 6. With so few questions per subject, treat this as a lead to test on your own prompts, not a result.
Accuracy gap +14.5 points (z = 1.83). We call a winner only when z is at least 1.96, the usual 95% threshold. Speed, latency and price need a 10% margin.