구형 제온 서버 174번 재부팅하며 찾아낸 최적 LLM 추론 옵션

Published on June 15, 2026 I restarted a 10 year old Xeon 174 times to delete twelve flags and gain four tokens a second 22 minutes read A follow-up to running Gemma 4 on a 2016 Xeon. I took that 25-flag config apart one flag at a time, to find which ones actually do the work, which are harmful, and which are just pitfalls. Most of my tuned config will do nothing for the typical user. A couple of weeks ago I got Gemma 4, a 26-billion-parameter model, running at reading speed on a 2016 Xeon with no GPU and 128 GB of DDR3 . That post spent about eight hours on the front page of Hacker News, which means a lot of people now have a 25-flag command sitting in a terminal somewhere, copied from a blog, with no real idea which of those flags is doing the work. I have some bad news about that command. You’re likely holding it wrong. What I said in that post in passing, is that half of those flags would not take just by being present. Some need the right hardware. Some need the right host setup. Some only help on the right workload. The engine accepts all of them and tells you almost nothing about which ones actually fired. So this post is me going back and finding out what will actually work for YOU. The way you find out is an ablation. You take the working config, switch off exactly one flag, measure what changes, put it back, and do the same for every flag in turn. The word is borrowed from neuroscience by way of machine learning, where it normally means knocking out a piece of the model itself, an attention head or a layer, to see what it was for. I am using it a little improperly here, for inference flags rather than model internals, but the idea carries over: turn one thing off, measure, repeat, and the differences tell you what each piece was worth. It was a lot of work. Like a lot. One hundred and seventy-four runs, each one a fresh server reloading twenty-five gigabytes of weights off a spinning disk before it can answer a single token. Three prompts per launch, several repetitions each, and one entire overnight run I had to throw in the bin over a deadlock I will get to. I am telling you the count because the count is the point. The reason nobody knows which of these flags matter is that finding out is slow and tedious, so almost nobody does it. I did, so here is the answer. The setup The box is the same one from the Xeon post. A Xeon E5-2620 v4, eight physical cores, sixteen threads, 128 GB of DDR3, no GPU, no swap. The engine is ik_llama.cpp on the feat/gemma-4-mtp branch. The verifier is gemma-4-26B-A4B-it at Q8_0, paired with its MTP drafter at Q8_0. I’ll use 3 test prompts: a short chat turn a roughly five-thousand-token document to summarize a code generation request. Greedy decoding, fixed seed, 256 new tokens, three repetitions, median reported. Two things to clear out of the way before the numbers. I ran the benchmark under llama-server rather than the llama-cli from the original command. The server is where the per-request speculative-decoding telemetry lives, which is the only clean way to check whether the drafter actually fired on a given request, and it is what the upstream pull request drives its own benchmarks through. Nothing about the config changes, just the harness around it. And every number below is the full config with one flag changed, so each delta is that flag’s contribution given everything else is still on, not its effect in a vacuum. Flags interact. Speculation changes how much the thread count matters. Repacking changes what flash attention has to read. The deltas do not add up to the gap between this config and a naive one, and I will not pretend they do. The whole board Here is every lever that moves the needle, in one place, before I walk through them. Decode speed, tokens per second, median of the repetitions, each row the full config with exactly one thing changed. The percentage is against the published config, so a flag that helps shows up as a loss when you remove it. lever changed chat long doc code published config (autotune drafter) 12.3 6.8 15.6 drafter, fixed draft 1 15.6 (+27%) 9.8 (+44%) 15.9 (+2%) drafter, fixed draft 2 16.1 (+31%) 7.4 (+8%) 17.8 (+14%) drafter, fixed draft 3, autotune off 13.9 (+13%) 7.1 (+4%) 18.3 (+17%) drafter off entirely 12.2 (wash) 10.5 (+54%) 12.2 (−22%) flash attention off 6.7 (−46%) 4.3 (−37%) 7.5 (−52%) threads -t 4 7.9 (−35%) 5.0 (−28%) 9.5 (−39%) threads -t 16 10.8 (−12%) 7.1 (+3%) 16.2 (+4%) run-time-repack off 11.8 (−4%) 9.2 (+35%) 13.7 (−12%) Everything else I tried, --mla-use , --cpu-moe , --merge-up-gate-experts , --no-kv-offload , the -sm graph cluster, lands within a few percent of the published config on chat and code, which is to say inside the noise. The long-document column is messier than the other two and I will come back to why in the repack section. Read the negatives as the interesting part. Most of these levers do nothing or cost you, and the work is concentrated in flash attention, the physical-core thread count, and the drafter once it is configured the right way. The drafter is workload-dependent, and one setting was wrong Speculative decoding is the most tuned part of the config, and it is where I found the most. The first finding is that the drafter is not a free win across the board. Measured against turning it off entirely: On chat it is roughly a wash at the default setting. On code it is worth about twenty-eight percent . A real win. On the long document it makes things slower. Turning the drafter off is fifty-four percent faster. Summarization is hard to predict, the drafter’s acceptance rate sits down around 0.15. A draft that gets rejected most of the time is mostly overhead. So the drafter wants to be gated by workload. On for code, off for long-context work. The config has it on globally, which quietly costs you about a third of your throughput every time you summarize something. This was not something I had clocked when I wrote the original post, and it is the single most useful thing in this one. The same flag being a win, a wash, and a loss depending on the prompt is worth sitting with, because it is a tiny hand-measured example of a routing gap. The model already routes internally: a mixture-of-experts gate sends every token to the experts it judges most useful, learned and conditional, per token. But that gate optimizes for the next token, not for which inference machinery pays off on a given kind of work. The choice I am making by hand here, speculate on code, do not speculate on the summary, is a second and coarser routing decision sitting one level up, per request instead of per token, and the config hardcodes one answer to it and applies that everywhere. You could imagine a cheap classifier in front of the engine that picks the decoding policy the way the expert gate picks experts, and --spec-autotune is a clumsy first stab at that idea, aimed at a single knob and, on this workload, aimed badly. I am not saying the expert routing is broken. I am saying there is a routing decision above it that my numbers say is worth making, and I made this one with a shell script and a stopwatch… metaphorically. The stopwatch is a metaphor to be clear. The second finding is a setting I had wrong. In the Xeon post I described --spec-autotune as the way to squeeze the most speed out of speculation. Over these generations it is the worst speculation setting I tested. Every fixed --draft-max beats it. chat: autotune 12.3 tokens per second, a fixed draft of 2 gets 16.1. Twenty-five flags, and the biggest single win in the file was deleting one of them and typing a number. code: autotune 15.6, a fixed draft of 3 with autotune off gets 18.3. long document: autotune 6.9, a fixed draft of 1 gets 9.8. The mechanism is acceptance rate. A shorter, fixed draft gets accepted by the verifier more often. A draft of 1 on chat sits at 0.74 acceptance against autotune’s 0.37, and acceptance is the entire currency speculation trades in. The autotuner spends the generatio