밀집 모델 대결: 느린 게 더 빠르다?

Qwen3.6 27B vs Qwen3.5 27B vs Gemma 4 31B: Accuracy, Latency, Memory, and Token Efficiency Tested Qwen3.6 improves on Qwen3.5, but Gemma 4 remains surprisingly competitive. Benjamin Marie May 05, 2026 ∙ Paid 6 Share In a previous article, I found Gemma 4 31B to be superior or comparable to Qwen3.5 27B in most areas, with similar or better accuracy and lower latency. Gemma 4 31B vs Qwen3.5 27B: Inference Speed, Token-Efficiency, Accuracy, and Memory Consumption Benjamin Marie · Apr 15 Read full story However, the Qwen3.6 update likely changes that conclusion. On benchmarks, it appears to be significantly stronger than Qwen3.5. The best accuracy in this model class is therefore probably now achieved by Qwen3.6, but at what cost? And how does that trade-off vary depending on the task? The Kaitchup – AI on a Budget is a reader-supported publication. To receive new posts and support my work, consider becoming a free or paid subscriber. Subscribe I’ll answer these questions in this article. I measured accuracy, latency, and token efficiency, both with thinking enabled and disabled, and compared the results with the numbers I previously obtained for Qwen3.5 27B and Gemma 4 31B. I used the exact same setup to ensure that all results are directly comparable. Acknowledgments This article would not have been possible without the compute sponsorship generously provided by Verda , whose B200 and RTX Pro 6000 GPUs I used throughout this work. Verda provides access to high-end GPUs such as the B200 and B300, with GB300 support coming soon, as well as smaller GPUs such as the RTX 6000 Ada, which are among the most affordable per hour on the market. Verda is a European, AI-focused cloud and GPU infrastructure provider with sovereignty, sustainability, data privacy, and performance at its core. You can check them out here . Qwen3.6 27B: Much Better Accuracy than Gemma 4 and Qwen3.5? Before diving into the results, I should note that all the benchmarks I used are non-agentic and do not use tool calls. This setup is not particularly favorable to Qwen3.6, which appears to have been fine-tuned specifically for stronger agentic performance. And indeed, on several of the benchmarks I ran, Qwen3.6 performs behind both Qwen3.5 and Gemma 4, while for a few benchmarks, it’s significantly better. Hard math questions: AIME On hard math questions, as measured by AIME, Qwen3.6 significantly outperforms both Qwen3.5 and Gemma 4. It also performs better than Qwen3.5 on simpler math benchmarks such as Math 500. Single-turn coding: LiveCodeBench On single-turn coding tasks, measured with LiveCodeBench, Qwen3.6 improves over Qwen3.5 but still slightly underperforms Gemma 4. World knowledge: MMLU Pro Qwen3.6 also shows stronger world knowledge. On MMLU Pro, it answers more accurately than both Qwen3.5 and Gemma 4. The strange results Some results were more surprising. First, Qwen3.6 is significantly worse than Qwen3.5 at following instructions, as measured by IFBench. Second, Qwen3.6 is also significantly worse on GPQA Diamond. This was unexpected, since Qwen reports a 2.3-point improvement over Qwen3.5 on this benchmark. Was something wrong with my setup? Whenever I doubt my own results, I cross-check them against Artificial Analysis, since they also run some of these benchmarks. In this case, they found the same thing: Qwen3.5 performs better than Qwen3.6 on GPQA Diamond. This likely means Qwen ran something different: different hyperparameters, a different version of the benchmark, different post-processing, or some other variation in the evaluation setup. It is a useful reminder that benchmark scores published by different groups are not directly comparable. Overall, Qwen3.6 is only slightly better than Qwen3.5 on average, and it still underperforms Gemma 4 on this type of non-agentic task. Of course, benchmark scores only tell part of the story. I ran additional analyses to better understand these results. One analysis I now run systematically is CoDeC scoring, which identifies benchmarks where a model appears more comfortable, suggesting possible special training or fine-tuning on benchmark-like data. Did the Model See the Benchmark During Training? Detecting LLM Contamination Benjamin Marie · Feb 2 Read full story My CoDeC run confirms that Qwen3.6 is much more comfortable with AIME-style benchmarks. It reaches a score above 62, which is very rare. By contrast, AIME-like data appears to be much newer to Gemma 4 31B IT. This helps explain why Qwen3.6 improves so much over Qwen3.5 on AIME. I also examined how Qwen3.6 behaves on LiveCodeBench, since this is one of the benchmarks where Gemma 4 remains significantly ahead. Interestingly, the accuracy gap can be closed by leveraging random sampling, that is, by looking at pass@k rather than pass@1. At pass@4, Gemma 4 31B and Qwen3.6 27B achieve nearly the same accuracy. How to Reduce LLM Inference Cost and Improve Accuracy with Pass@k and Majority Voting Benjamin Marie · Apr 27 Read full story However, as we will see later, this does not make Qwen3.6 the better choice in practice. Because of its much higher cost, using Gemma 4 31B remains substantially cheaper when targeting the same level of coding accuracy. Qwen3.6 27B: Token Efficiency and Latency This post is for paid subscribers Subscribe Already a paid subscriber? Sign in Previous