vLLM v0.28.0 릴리스

Uh oh! There was an error while loading. Please reload this page . vllm-project / vllm Public Uh oh! There was an error while loading. Please reload this page . Notifications You must be signed in to change notification settings Fork 21.4k Star 90.4k v0.28.0 Latest Latest Compare Choose a tag to compare Sorry, something went wrong. Filter Loading Sorry, something went wrong. Uh oh! There was an error while loading. Please reload this page . No results found View all tags khluu released this 26 Aug 09:46 &middot; 522 commits to main since this release v0.28.0 2cf0a69 v0.28.0 Highlights This release features 584 commits from 270 contributors (76 new)! Kimi-K3 performance push : a major optimization effort for Kimi-K3 across the stack — Decode Context Parallel (DCP) support ( #50484 ), fused FlashKDA decode and prefill kernels ( #50654 , #51311 , #52458 ), SiTU activation support for MegaMoE ( #50510 ), GEMM-RS for sequence parallelism ( #52079 ), combined all-gathers with 1.5~3x kernel-level speedup ( #51070 ), an adaptive speculative token budget delivering ~60% better DSpark TTFT ( #51725 ), and optional shared-expert sharding saving ~17 GiB of memory per GPU ( #50912 ). Kimi-K3 also now runs on ROCm with the V2 model runner ( #51653 ). DeepSeek V4 : sparse MLA now works end-to-end for plain decode, MTP, and DSpark speculative decoding ( #51538 ), joined by AMD Quark NVFP4 support ( #47972 ), reasoning-effort prompts and mappings ( #50580 ), sparse top-k metadata kernel optimizations ( #52084 , #51967 ), narrowed eager CUDA graph regions ( #51430 , #52401 ), and ROCm enablement on gfx11 and gfx950 ( #47017 , #52212 ). Speculative decoding advances : DFlash2 with local convolution and a candidate selector ( #52816 ), DSpark confidence-scheduled verification ( #47808 ), and async scheduling auto-enabled for draft models ( #48341 ). Model Runner V2 maturation : E/P/D disaggregation ( #38390 ), weight offloading ( #51413 ), multi-layer MTP KV cache support ( #50062 ), encoder CUDA graphs ( #49852 ), decoder token-wise pooling ( #50931 ) plus Transformers pooling models ( #52425 ), attention-free models ( #52374 ), and thinking_token_budget support ( #46727 ). Tiered KV cache offloading : disk offloading support ( #49644 ), out-of-tree secondary tier managers via module_path ( #51007 ), partial secondary-tier load results ( #50321 ), tiering metrics ( #48798 ), and a canonical CPU layout for parallelism-agnostic offload ( #48414 ). Rust frontend & gRPC : a standalone renderer ( #50289 ), multimodal image inference over gRPC ( #50368 ), explicit data-parallel rank routing ( #51178 ), and RL lifecycle control ( #51316 ), with protobuf schemas now published to Buf ( #51276 ). New defaults : max_num_batched_tokens raised from 8192 to 16384 ( #51726 ), prefix caching enabled by default for Mamba models ( #50991 ), and the Blackwell CUDA graph capture default raised to 1024 ( #49390 ). Breaking changes : bitsandbytes support migrated to an out-of-tree plugin ( #43529 ); Transformers bumped to 5.15.0 ( #51668 ); the deprecated calculate_kv_scales runtime KV scale calculation was removed ( #49389 ); override_attention_dtype was removed ( #48684 ). Release Artifacts Python Wheels Platform Install PyPI (CUDA 13.0) pip install vllm PyPI (CUDA 13.0, uv) uv pip install vllm --torch-backend=auto ROCm pip install vllm --extra-index-url https://wheels.vllm.ai/rocm/0.28.0/rocm722 Docker Images Platform Docker Image CUDA 13.0 (Default) docker pull vllm/vllm-openai:v0.28.0 ( v0.28.0-cu130 also works) CUDA 12.9 docker pull vllm/vllm-openai:v0.28.0-cu129 CUDA 13.0 + Ubuntu 24.04 docker pull vllm/vllm-openai:v0.28.0-ubuntu2404 CUDA 12.9 + Ubuntu 24.04 docker pull vllm/vllm-openai:v0.28.0-cu129-ubuntu2404 ROCm docker pull vllm/vllm-openai-rocm:v0.28.0 CPU docker pull vllm/vllm-openai-cpu:v0.28.0 XPU docker pull vllm/vllm-openai-xpu:v0.28.0 Other Artifacts Pre-built release artifacts are available in the Assets section at the bottom of this page, including: Source distribution tarball CUDA 12.9 Python wheels for x86_64 and arm64 CUDA 13.0 Python wheels for x86_64 and arm64 CPU Python wheels for x86_64, arm64, and macOS Model Support New models : Muse Glimmer ( #51655 ), Ling 3.0 Flash with BF16, MTP, and parser support ( #51045 ) plus an FP8 variant ( #51265 ) and hybrid MXFP4 routed experts ( #52114 ), Dots3 NOTE native multimodal support ( #51255 ), and Interns2mobius ( #51149 ). Qwen : Qwen3.8 enabled on AMD ROCm ( #50068 ), fused CUDA post-conv MTP decode kernel for Qwen3.5 GDN ( #51674 ), GDN gates aligned with speculative tokens ( #51812 ), and Qwen3.5 fixes for text-only checkpoints ( #50734 , #50355 ). Transformers modeling backend : MLA support ( #48250 ), hardware-agnostic model definition ( #49458 ), fully generalized input embedding handling ( #51247 ), logit softcapping ( #52173 ), and a hardened multimodal path ( #51408 , #51657 ). LoRA : vision tower LoRA for Gemma4 ( #42662 ), tower/connector LoRA for Keye ( #51780 ) and Ultravox ( #48215 ). Vision encoders : ViT full CUDA graph for Kimi-K2.5 ( #50929 ) and Ernie-4.5-VL ( #45254 , #51461 ), torch.compile for the Qwen3-VL encoder ( #40116 ), and long-blocking H2D copies avoided in ViT ( #51841 ). MoE : extended EPLB support for Mistral Large 3 and additional MoE backends ( #48355 ), CuTe DSL skinny GEMM extended to GLM-5.2 ( #49791 ). Speculative decoding coverage : EAGLE3 support declared on KimiLinear ( #52171 ), Qwen3.6 dSpark acceptance coverage ( #51310 ). Correctness : MiniMax-M3 NVFP4 inference ( #48929 ) and compressed-tensors FP8 MoE SwiGLU params ( #46845 ), Gemma3n/Gemma4 variable-length audio batch padding ( #50958 ), Gemma 4 compatibility with the upcoming Transformers version ( #49797 ), and a Qwen3-Omni crash on video without an audio track ( #48420 ). Multimodal performance : fused on-device multimodal preprocess normalization ( #50411 ), faster placeholder and token-match scanning ( #50716 ), and repeated prompt-update scans avoided ( #51774 ). Engine Core Speculative decoding : DSpark confidence-scheduled verification ( #47808 ), top-k DSpark Markov projection ( #49969 ), DFlash2 with local convolution and a candidate selector ( #52816 ), async scheduling auto-enabled for draft models ( #48341 ), fused MTP trailing all-reduce with local-argmax draft tokens ( #49793 ), and an adaptive budget for speculative scheduled input tokens ( #51725 ). KV cache & scheduling : per-request scheduling for MLA chunked context ( #50613 ), partial-tail prefix reuse with fine-grained prefix matching ( #50507 ), backend-published KV packing in the KV-cache layout refactor ( #51612 , #51704 ), LIFO free-block reuse order restored when prefix caching is off ( #51482 ), and silent request skipping in priority scheduling fixed ( #49206 ). Performance : continued elimination of GPU<->CPU syncs on the execution path ( #51458 , #51738 , #52369 ) now guarded by a CI sync check ( #43107 ), new JIT warmup infrastructure with predicate filtering ( #49315 ), the top-k/top-p Triton sampler launched with 8 warps ( #51507 ), detokenization skipped in offline beam search ( #50333 ), Mask Replay ( #49577 ), optimized long-context MLA cache gathers ( #51739 ), and HF revisions resolved to a commit hash once per model load ( #49990 ). Hybrid/Mamba : prefix caching on by default ( #50991 ), the final part of the Mamba attention module refactor ( #44857 ), 3D-grid tiling of the state-copy Triton kernels ( #49436 ), and Mamba alignment applied before encoder caps ( #51603 ). RL workflows : stateful trainer send over NCCL and sparse NCCL ( #50902 ), CuMemAllocator.discard() for tag-selective GPU memory release ( #52514 ), level-2 sleep/wake/reload fixed with LoRA enabled ( #39935 ), and rewritten weight-transfer docs with standardized examples ( #51729 ). Startup robustness : file:// rendezvous for single-node executors eliminates startup port races ( #50999 , #51652 ), frontend processes are watched during engine startup ( #43417 ), a