Gemma 4 31B GGUF 양자화 모델 KL 발산 성능 순위

Gemma 4 31B GGUF quants ranked by KL divergence (unsloth, bartowski, lmstudio-community, ggml-org) oobabooga Apr 07, 2026 7 Share The plot above shows the KL divergence measured for 52 GGUF quants of Gemma 4 31B, from the following uploaders on Hugging Face: unsloth (20 quants) bartowski (27 quants) ggml-org (2 quants) lmstudio-community (3 quants) Lower KL divergence is better. It measures how different the quantized model’s token probability distribution is relative to the original model. A KL of 0 would mean the quant is identical to the original. Methodology All measurements were collected using the OpenAI-compatible API in text-generation-webui . llama.cpp doesn’t natively support extracting logprobs from the prompt, so I patched it. The patch is included in the llama.cpp binary shipped with the latest text-generation-webui release. The source can be found in this commit on my llama.cpp fork. For the reference logprobs, I used the BF16 GGUF model by unsloth . The evaluation works in three steps: The Jinja2 chat template is extracted from the model. The same template is used for every quant to keep the comparison identical across uploaders. Each sample’s messages are converted into a prompt string using the /v1/internal/chat-prompt endpoint with the instruction_template_str key. Each prompt is sent to the /v1/completions endpoint with echo: true and logprobs: 40 , which returns the top-40 token log-probabilities for every token in the prompt. The KL divergence is then computed token-by-token between the reference and quantized distributions. Dataset Most KL divergence benchmarks use Wikipedia with a context length of 2048 or similar. I wanted to measure KL divergence across real-world use cases, so I built a dataset with ~250,000 tokens across 6 categories: Coding General chat Tool calling Science Non-Latin scripts Long documents Each sample is a full OpenAI-compatible input. Tool calling samples include tool definitions and tool_calls , which get rendered into the prompt via the model’s chat template. Results Here is the plot above in log scale: For every quant type where both unsloth and bartowski provide a file (Q6_K, Q5_K_M, Q4_K_M, etc.), bartowski’s file is up to 1.5 GB larger and has slightly lower KL. On the Pareto frontier, neither dominates the other: they alternate depending on the size range. Unsloth’s UD- variants use a custom quantization scheme and tend to beat standard quants in their size range. For example, UD-Q3_K_XL (15.3 GB, KL 0.87) outperforms bartowski’s Q3_K_L (16.8 GB, KL 0.97) despite being 1.5 GB smaller. At higher bit rates the advantage shrinks: UD-Q6_K_XL (27.5 GB, KL 0.20) is essentially tied with bartowski’s Q6_K_L (27.1 GB, KL 0.20). The lmstudio-community and ggml-org quants are worse at the same file size. Their Q4_K_M files (18.7 GB) are similar in size to unsloth’s (18.3 GB) but have a KL of 0.76 vs unsloth’s 0.61, suggesting a less optimized quantization process. Q8_0 is identical across all uploaders at KL = 0.16. Notably, unsloth’s UD-Q8_K_XL (35.0 GB) is both larger and slightly worse (KL = 0.16) than Q8_0 (32.6 GB). Pareto frontier The Pareto frontier is the set of quants for which no smaller quant exists with lower KL divergence. These are the optimal choices depending on how much memory you have: Below Q5_K_S, quality drops fast. Top-1 agreement falls from 84% at Q5_K_S to under 80% at Q4_K_S. Per-category breakdown KL divergence is not uniform across tasks. Here is the breakdown for Q8_0, Q6_K, and Q5_K_S: Even Q8_0 shows a KL of 0.45 on long documents and 0.24 on non-Latin scripts. All categories roughly double from Q8_0 to Q5_K_S, but science and tool use remain the lowest throughout (0.07 and 0.08 at Q8_0). Per-category plots Here is the full KL divergence plot broken down by category: 7 Share