단일 RTX 3090으로 85 TPS·12만5천 컨텍스트 구현

An Overnight Stack for Qwen3.6–27B: 85 TPS, 125K Context, Vision — on One RTX 3090 Wasif Basharat 18 min read · Just now -- Listen Share A day after Qwen3.6 dropped: a consumer 24 GB card, a CUDA-graph bug, four patches, and the stack that made it sing. The headline Qwen3.6-27B on 1× RTX 3090 (24 GB, 230W cap) ───────────────────────────────────────────── Throughput 85 TPS sustained / 106 TPS peak Context 125 K tokens Vision Enabled (MoonViT BF16) VRAM 21.3 / 24 GB Server vLLM · full OpenAI API Tools ✅ Prefix cache ✅ Spec-decode MTP n=3 · AL 3.4–3.8 · accept 97/95/91% Lorbus’s own model card quotes ~60 TPS on an RTX 5090 for the same model. We did 85 TPS sustained — on a card with 25% less VRAM and ~40% less raw compute — at 230W (quiet and cool). The write-up below is how. 85 TPS sustained. 106 TPS peak. Past the 100 TPS barrier on a single 3090. Why this matters Qwen3.6–27B shipped yesterday. As of this morning, an open-source stack exists that serves it at datacenter-adjacent throughput on a GPU you can buy used for less than a MacBook Air. That’s worth pausing on before the deep dive. Your prompts stop leaving your machine. Regulated workloads, personal notes, customer data, internal codebases — the privacy tradeoff you’ve been making for GPT-class latency evaporates. The model reads your files on your RAID. Inference is free after the card pays for itself. At 85 TPS sustained on a 230W cap, a 3090 serves millions of tokens a day on a power bill that rounds to coffee money. API spend becomes a design choice, not an operating cost. Latency is the wire. No round-trip, no rate limit, no queuing behind the rest of the planet. First token arrives at the speed of a local socket. Agent loops that felt sluggish against an API stop feeling sluggish — and tool-calling loops, where you’re paying the round-trip every turn, get dramatically snappier. You own the weights. Qwen shipped these under a permissive license. They don’t get deprecated, silently re-tuned, or price-hiked. If the model works for your use case on Tuesday, it works on Friday and next year. Fine-tuning becomes a weekend project. Once your serving stack is validated, swapping in a LoRA for a domain-specialized version is hours, not a capex conversation. Vision-capable base + your domain data = your own private multimodal model. Vision + 125K context + tool calling + 85 TPS on a used consumer GPU is, as of this week, a new category of capability. The rest of this write-up is how we got it onto ours — and how you can put it on yours by tonight. The model, and why the problem is hard Qwen3.6–27B landed on 2026–04–22 as a dense vision-language model from Alibaba’s Qwen team. It’s part of the Qwen3-Next family — meaning it has a hybrid attention architecture : interleaved DeltaNet (Gated Linear Attention, recurrent state) and standard softmax attention layers. Plus a native Multi-Token Prediction (MTP) head for speculative decoding. On paper, it’s a dream model for a consumer GPU: 27B parameters = ~17 GB in INT4, comfortably on 24 GB Vision tower is a compact MoonViT that stays BF16 (~0.9 GB) MTP head = free speculative decoding if the inference engine plays nice In practice, every one of those “on paper” assumptions hits an invisible wall on consumer hardware. The three big ones: DeltaNet’s recurrent state breaks every standard speculative-decoding pipeline. Rejecting a draft token means rolling the state back — and linear-attention recurrences can’t be rolled back. vLLM, SGLang, llama.cpp mainline — none of them handled it correctly out of the box. Most INT4 quants drop the MTP head or quantize it to INT4 too. Either way, vLLM’s Qwen3_5MTP loader can't find the weight it's expecting ( mtp.fc.weight ). Result: MTP silently loads with zero parameters → 0% draft acceptance → you're paying the overhead with none of the speedup. TurboQuant KV cache (the thing that makes 125K context possible) refuses to load on hybrid models. vLLM raises NotImplementedError at arg_utils.py:1652 the moment it sees a Mamba-like layer. And even when you work around that , the backend has a subtler bug that crashes CUDA graph capture specifically when speculative decoding is enabled. Each of these stopped us cold on first contact. This is the story of getting past all three. The journey False start: “The model is broken” First symptom, 2026–04–22. We downloaded three Qwen3.6–27B GGUF quants via aria2c -i url-list to max throughput. Everything loaded. Everything produced multi-script gibberish — Chinese + Latin + Korean + Greek + Malayalam token salad, or infinite loops of " is is is is is..." . We spent ~6 hours hypothesizing: new attention variant? Broken sampling? Flash-attention version mismatch? Chat template drift? Upstream llama.cpp bug on qwen35 arch? The actual answer was embarrassingly mundane. aria2c silently corrupted the files. All three GGUFs came down at exactly the advertised size but with wrong bytes mid-file. Size checks passed; SHA256 checks didn't — which nobody had run yet. Re-downloaded via hf download with HF_HUB_ENABLE_HF_TRANSFER=1 and verified against HF's x-linked-etag header. Model worked immediately. Lesson (filed permanently under LEARNINGS.md): after any download ≥2 GB, sha256sum against HF's linked-etag. Never trust size alone. The first hypothesis on garbage output should always be file integrity, not the model. Baseline: “Vanilla INT4, no spec-dec, ~38 TPS” First real working config was cyankiwi AWQ-INT4 (19 GB) on vLLM :latest : A1: 38 TPS / 8K ctx / 21.4 GB VRAM Coherent output (“Capital of France → Paris”), full OpenAI API, nothing exotic. This was the first working path after the GGUF fiasco and gave us a baseline. Then we tried to turn on MTP: A3: FAIL — OOM MTP head needs 2.37 GB Base model weights 21.24 GB 24 GB card: nope. vLLM had allocated a fresh BF16 MTP head — 2.37 GB of parameters for a fused linear prediction layer — because cyankiwi’s quant didn’t include one. The Qwen3_5MTP loader defaults to instantiating BF16 when the quant omits the head tensors. Dead on a 24 GB card. Breakthrough #1: Lorbus’s clever quant Enter Lorbus/Qwen3.6-27B-int4-AutoRound — 19 GB total, AutoRound INT4 calibration, group_size 128, symmetric packing. Standard stuff. Except Lorbus's README had a paragraph that turned out to be the whole game: A plain auto-round run on a Qwen3.5/3.6 model packs mtp.fc as INT4. In that form, vLLM skips loading the layer entirely (param name mismatch between fc.qweight and the expected fc.weight ), which makes MTP speculative decoding produce 0% acceptance. This release dequantizes mtp.fc back to BF16 after AutoRound finishes. Translation: Lorbus manually dequantized the one weight that matters for MTP, costing ~100 MB . Everything else stays INT4. vLLM’s loader now finds exactly what it expects, and the MTP head runs at its intended BF16 precision alongside the INT4 base. We loaded it with --speculative-config '{"method":"mtp","num_speculative_tokens":1}' : L2: 54.6 TPS (narr) / 59.0 TPS (code) / 8K ctx / 20.3 GB VRAM Mean AL 1.9 · per-pos accept 78–96% First working speculative decoding on a Qwen3-Next hybrid on our stack. Ever. +31% narrative, +43% code over no-spec. And the MTP head footprint? 280 MiB. Not 2.37 GB. Because Lorbus packed the transformer-layer portions INT4 along with the base , leaving only the final fc projection in BF16. The single-card fit is a design decision, not luck. Why does this work where the DeltaNet rollback problem doesn’t bite? Because num_speculative_tokens=1 is a single-step propose/verify. There's no multi-step state rewind — accept or reject happens atomically per token, and DeltaNet state advances exactly once per committed token. The invariant holds. The MTP sweep: n=3 is the sweet spot With MTP working, we asked: how many speculative tokens should we draft per step? n=1: 54.6 / 59.0 TPS AL 1.9 per-pos 96% n=2: 61.0 / 69.7 TPS AL 2.4 per-pos 82/56% n=3: 63.8 / 79.7 TPS AL 3.4