AMD GPU에서 vLLM 추론 디코딩 활용하기

Table of Contents TL;DR: Speculative decoding allows vLLM to verify multiple drafted tokens in a single target-model pass. In our experiments, its effect on output-token throughput varied across drafting methods and proposal lengths, and also depended on the model family, draft checkpoint, workload, and acceptance behavior. Introduction Large language models support a wide range of applications, but serving them at scale requires careful optimization. Standard autoregressive decoding is the baseline used by most LLM serving systems: the model generates one token, appends it to the sequence, and then uses the updated sequence to generate the next token. This process is simple and reliable, but the serving loop still advances one committed token at a time because output tokens must be produced in strict left-to-right order. Speculative decoding [1] builds on this baseline through a draft-and-verify mechanism. A lightweight draft component proposes candidate future tokens, and the target model verifies those candidates before they are committed. When several draft tokens are accepted, the system can commit multiple output tokens from a single target-model verification step while preserving the target model's output behavior. This post explores how speculative decoding works in vLLM and shares measurements from our test environment. We first review the autoregressive decoding baseline and the draft-and-verify process. We then examine five speculative-drafting approaches: native MTP, Gemma 4 MTP, EAGLE-3, DFlash, and DSpark. These methods differ in how the draft component receives information from the target model and whether candidate tokens are generated sequentially, autoregressively, in parallel, or through a hybrid approach. Finally, we show how to enable the methods tested in our environment, report measurements from our experiments on AMD Instinct™ MI300X and MI355X GPUs using the ROCm™ open software platform, and discuss practical tuning and observability considerations. The autoregressive decoding baseline In standard autoregressive decoding, each decode step produces and commits one new token. For example, generating four output tokens requires four sequential decode steps: Step 1: context → model → T1 Step 2: context + T1 → model → T2 Step 3: context + T1 T2 → model → T3 Step 4: context + T1 T2 T3 → model → T4 After each step, the generated token is appended to the sequence and becomes part of the input for the next step. This makes the decoding loop straightforward, but it also requires one model decode step for every output token. During long generations, this token-by-token loop can dominate latency and limit serving throughput. The key question behind speculative decoding is therefore: Can we preserve the output behavior of the original model while reducing how often generation advances by only one token at a time? Speculative decoding addresses this by separating proposal from verification. A draft component first proposes several candidate future tokens. The original model, acting as the target model, then verifies those candidates before they are committed. Core idea of speculative decoding Speculative decoding does not replace the original model. Instead, it keeps the original model as the target model, which remains responsible for the final output, and adds a faster proposal stage in front of it. The process has two parts: Draft: propose several candidate future tokens. Verify: use the target model to check those candidates. During each speculative decoding round, as illustrated in Figure 1, a lightweight draft component proposes one or more future tokens. These tokens are only candidates and are not committed immediately. The target model then evaluates the candidate token sequence in one verification pass. Verification proceeds from left to right. Each draft token is checked using the target model's result at the corresponding position. Accepted tokens are committed to the output sequence. When a draft token is rejected, later candidates from the same proposal are no longer accepted. If a draft token is rejected, the target model provides the next token. The remaining draft tokens are discarded, and generation continues from the updated sequence. Conceptually, standard autoregressive decoding advances like this: target model → T1 target model → T2 target model → T3 target model → T4 Speculative decoding instead allows several candidate positions to be evaluated together: draft proposes T1 T2 T3 T4 model verifies ✓ ✓ ✗ stop commit T1 T2 replacement token - This can reduce the number of target-model decoding rounds when multiple candidates are accepted. When the draft component produces tokens that the target model accepts, several output tokens can be committed from one target-model verification step. When a proposal is rejected, the target-side result determines how generation continues. A simple accept/reject example Figure 2 gives an example of one speculative decoding round. Green boxes are draft tokens that survive verification, the red box marks the first rejected draft token, and the gray box is a later draft token that is discarded. The blue token in the output comes from the target model, not from the draft proposal. Suppose the current prompt is: The weather today is The draft component proposes several future tokens: sunny and warm outside The target model verifies the draft tokens from left to right: draft proposes sunny and warm outside model verifies ✓ ✓ ✗ stop commit sunny and clear - The first two draft tokens, sunny and and , are accepted. At the third position, the draft proposes warm , but the target model selects clear . The remaining candidate, outside , is discarded because it follows the first rejected position. The next decoding round therefore continues from: The weather today is sunny and clear How the drafting methods work Although all speculative decoding methods follow the same overall draft-and-verify process, they differ in how the draft component is designed and how it works with the target model. The main differences are: The type of information received from the target model. How this information is incorporated into the drafting process. Whether candidate tokens are generated sequentially or in parallel. Based on these differences, the drafting methods discussed in this post can be grouped into three broad categories: native MTP modules, separate MTP drafters, and dedicated target-conditioned draft networks. Native MTP modules: built directly into the target-model architecture; use a model-native auxiliary prediction path; generate candidate tokens sequentially. Separate MTP drafters: use a separate checkpoint paired with a specific target model; use target-model activations and shared KV-cache information during inference; generate candidate tokens sequentially. Dedicated target-conditioned draft networks: use separate speculator models trained for a specific target model, including EAGLE-3, DFlash, and DSpark. EAGLE-3 drafts autoregressively from target-model hidden states, DFlash drafts parallel blocks from target-model hidden states, and DSpark adds lightweight causal correction and confidence-based prefix selection. These categories describe the draft component architecture, not the target-model family. A target model may support native MTP while also having separately trained EAGLE-3, DFlash, or DSpark draft models. The draft component does not operate entirely on its own. Depending on the method, the draft component may receive: A hidden representation from the target model. Hidden states from several selected target layers. The target model's KV cache. Features produced by combining multiple target-model representations. The following sections explain how each method uses this information and how it generates candidate tokens. Native MTP Multi-Token Prediction, or MTP, refers to a family of model-native mechanisms for predicting tokens beyond the immediate next token. In vLL