엔비디아, 기억 수정 최적화 선형 어텐션 모델 공개

Artificial Intelligence AI Infrastructure Tech News AI Paper Summary Applications Technology Editors Pick Language Model Large Language Model Machine Learning New Releases Open Source Physical AI Software Engineering Staff Linear attention replaces the unbounded KV cache of softmax attention with a fixed-size recurrent state. This cuts sequence mixing to linear time and decoding to constant memory. The hard part is not what to forget. It is how to edit a compressed memory without scrambling existing associations. NVIDIA has released Gated DeltaNet-2 , a linear attention layer that targets that bottleneck. The model decouples the active memory edit into two channel-wise gates. It is trained at 1.3B parameters on 100B FineWeb-Edu tokens. It outperforms Mamba-2, Gated DeltaNet, KDA, and Mamba-3 across the researchs benchmark suite. The scalar gate problem in delta-rule models A recurrent linear attention layer stores a matrix state S t and reads it with the query. DeltaNet adds an active edit by subtracting the value currently associated with the current key. It uses a scalar step size β t to control how much to overwrite. Mamba-2 adds a data-dependent scalar decay α t for global forgetting. Gated DeltaNet combined both operations, but both gates remained scalar per head. Kimi Delta Attention (KDA) refines the decay side. It replaces the scalar α t with a channel-wise vector. KDA still keeps a single scalar β t for the active edit. That scalar controls two different things at once. It decides how much old content to erase on the key side. It also decides how much new content to commit on the value side. These two decisions act on different axes of the state. Tying them together is a modeling restriction, not a property of the delta rule. Gated Delta Rule-2: two gates instead of one Gated DeltaNet-2 separates the two decisions through Gated Delta Rule-2. It introduces a channel-wise erase gate b t ∈ [0,1] d k on the key axis. It also introduces a channel-wise write gate w t ∈ [0,1] d v on the value axis. Both gates are produced by sigmoid projections of the token representation. The update applies decay before the active edit. Written compactly, the recurrence is: S t = (I − k t (b t ⊙ k t ) ⊤ ) D t S t−1 + k t (w t ⊙ v t ) ⊤ Here D t = Diag(α t ) is the channel-wise decay carried over from KDA. The left factor of the erase matrix stays k t , preserving the delta-rule write direction. The right factor becomes b t ⊙ k t , making the read direction channel-selective. The write term k t z t ⊤ uses z t = w t ⊙ v t , making the value update channel-selective. When both gates collapse to the same scalar β t , the update recovers KDA exactly. When the decay α t also collapses to a scalar, it recovers Gated DeltaNet. Both prior models are preserved as tied subspaces of the new update. In the fast-weight view, Gated Delta Rule-2 is one online gradient step on a local regression loss. The decayed state stays close to memory, while the residual edit uses gated read and gated write targets. Chunkwise training and gate-aware backward The recurrence admits a chunkwise WY form that matches the structure used by KDA. Cumulative channel-wise decay is absorbed into the two factors of each rank-one erase. The per-chunk update becomes a product of asymmetric matrices of the form I − k̄ r ē r ⊤ . The implementation uses chunk size C = 64 with fused Triton kernels. For the backward pass, the scalar shortcut used by KDA no longer applies. The write side contains a different diagonal gate over value channels. The erase side contains a different diagonal gate over key channels. So the gate factors must appear inside the dot products that accumulate gradients. The paper derives this gate-aware vector-Jacobian product explicitly. On Hopper GPUs, the fused WY backward kernel is restricted to two and four warps to avoid a Triton WGMMA layout assertion. Block design and hybrid model Gated DeltaNet-2 is used as the recurrent token mixer in a standard Transformer-style block. Query and key paths use linear projection, short causal convolution, SiLU, and L2 normalization. The value path uses linear projection, short convolution, and SiLU. The decay α t , erase gate b t , and write gate w t come from separate linear branches. The recurrent output is RMS-normalized, multiplied by a SiLU output gate, and projected back. A hybrid variant inserts Sliding-Window Attention (SWA) after the recurrent mixer. A repeated cell contains Gated DeltaNet-2, an MLP, SWA, and another MLP. SWA handles exact local interactions, while the recurrent mixer compresses long histories. The hybrid retains linear sequence scaling with a bounded attention cache. Results at 1.3B parameters All models are 1.3B parameters trained on 100B FineWeb-Edu tokens. Parameter count and recurrent state size are matched across models. The recurrent state holds 262,144 floats per layer per batch element. Training length is 4K tokens, and hybrid models use a 2K SWA window. The Mamba-3 MIMO baseline uses rank R = 4 . On language modeling and commonsense reasoning, Gated DeltaNet-2 has the best average in both settings. The recurrent model averages 53.11 across LAMBADA and the reasoning suite. That sits above Mamba-3 MIMO at 52.39 and KDA at 52.28. In the hybrid setting, Gated DeltaNet-2 averages 53.97 against Mamba-3 MIMO at 52.72. Since recurrent state size is matched, the gain points to the update rule, not more memory. The clearest gains appear on RULER long-context retrieval. In the recurrent setting, S-NIAH-2 at 4K rises from 89.0 (KDA) to 93.0. S-NIAH-3 at 2K jumps from 63.2 (KDA) to 89.8. MK-NIAH-1 at 4K climbs from 28.0 (KDA) to 37.8. On real-world retrieval (SWDE, SQuAD, FDA, TriviaQA, NQ, DROP), Gated DeltaNet-2 also leads both settings. The recurrent average is 29.88 and the hybrid average is 42.28. Marktechpost’s Visual Explainer Gated DeltaNet-2 · Quickstart 01 / 08 NVIDIA · 2026 Gated DeltaNet-2 Decoupling Erase and Write in Linear Attention. A delta-rule recurrent attention layer with channel-wise erase and write gates. PyTorch Triton kernels 1.3B params 100B FineWeb-Edu tokens Authors Ali Hatamizadeh, Yejin Choi, Jan Kautz Repo github.com/NVlabs/GatedDeltaNet-2 License NVIDIA Source Code License-NC Step 01 · The Idea Two gates instead of one scalar Linear attention compresses an unbounded KV cache into a fixed-size recurrent state. Editing this memory without scrambling existing associations is the hard part. The Problem Prior delta-rule models (Gated DeltaNet, KDA) tie erasing old content and writing new content to one scalar gate β_t . The Fix Split it: a channel-wise erase gate b_t on the key axis, and a channel-wise write gate w_t on the value axis. Erase gate picks which key-side coordinates of the decayed state are read and removed. Write gate picks which value-side coordinates of the new content are committed. Channel-wise decay is inherited from KDA for fine-grained global forgetting. Step 02 · The Update Rule The Gated Delta Rule-2 With erase gate b_t ∈ [0,1]^{d_k} , write gate w_t ∈ [0,1]^{d_v} , and channel-wise decay D_t = Diag(α_t) , the recurrent state evolves as: S_t = (I − k_t (b_t ⊙ k_t) ⊤ ) D_t S_{t−1} + k_t (w_t ⊙ v_t) ⊤ Recovers KDA exactly when both gates collapse to the same scalar. Recovers Gated DeltaNet when the decay also collapses to a scalar. Trains efficiently via a chunkwise WY form with channel-wise decay absorbed into asymmetric erase factors. Step 03 · Get the Code Clone the repo and build the environment The official PyTorch implementation ships with a Dockerfile, training scripts, and the lit_gpt model definitions. git clone https://github.com/NVlabs/GatedDeltaNet-2.git cd GatedDeltaNet-2 # build the environment from the provided Dockerfile docker build -t gdn2 . docker run --gpus all -it —ipc=host -v $PWD:/workspace gdn2 Repo layout lit_gpt/ model code · scripts/ launchers &midd