Verdict
Built from the measurements below
- Accuracy is too close to call: Hermes 4 405B scored 26.2% and Qwen3.5 397B A17B 32.5% on the same 80 questions, a gap within what chance alone could produce.
- Hermes 4 405B is 1.3× cheaper per question as billed ($0.01 vs $0.013), reasoning tokens included.
- Qwen3.5 397B A17B generates text 5.9× faster: 240 tokens per second against 41.
- Hermes 4 405B starts answering sooner: 0.3 s to the first token, against 0.6 s for Qwen3.5 397B A17B.
- Qwen3.5 397B A17B accepts more context: 262k tokens against 131k.
Which should you choose?
Choose Hermes 4 405B for a quicker first token. Choose Qwen3.5 397B A17B for faster streaming, a 262k-token context window and more hosts to choose from on OpenRouter (10 against 1). The widest subject gap is in humanities & social science: Qwen3.5 397B A17B answered 8 of 9 correctly, Hermes 4 405B 6 of 9. With so few questions per subject, treat this as a lead to test on your own prompts, not a result.
Accuracy gap -6.3 points (z = -0.87). We call a winner only when z is at least 1.96, the usual 95% threshold. Speed, latency and price need a 10% margin.