연속 확산 언어 모델(CDLM)의 부활

A flurry of recent activity in the space of continuous diffusion models for language , after a few years of relative dormancy, suggests that this approach is making something of a comeback. Fully discrete diffusion methods had largely supplanted earlier attempts to make continuous diffusion work for language, but the tide is starting to turn. In this post, I want to take a closer look at what’s going on, and why it is happening now. The recent influx of new research in this space inspired me to write up some of my thoughts. I have written about diffusion language models before, so this mainly serves as an update to cover everything that’s happened since then. This will be a fairly subjective account – other perspectives and dissenting opinions are very welcome in the comments and elsewhere! I’ll discuss some technical aspects of continuous diffusion for language later on, but first, some historical context. Challenging the autoregressive hegemony Modern language models are, by and large, autoregressive : they generate sequences one token at a time. This is a natural decomposition of a difficult generation task into smaller, easier sequential steps. All steps are instances of the same underlying task (predict a token given preceding tokens), which enables parameter sharing across the sequence dimension. In spite of this inherently sequential generative process, the Transformer architecture 1 admits efficient parallel training across all sequence positions using teacher forcing 2 . This has turned out to be an extremely scalable recipe 3 , which has brought us large language models (LLMs). However, autoregression is not the only way to construct an iterative generative process for sequences. Inspired by early successes in the audiovisual domain, researchers sought to apply diffusion to language generation instead. Rather than generating a sequence one element at a time, the generative process of diffusion models is defined by reversing a corruption process, which gradually destroys information. The canonical way to do this is to add Gaussian noise little by little, until it completely overpowers the signal. 2021: early discrete diffusion models After early successes in image generation in 2019 4 and 2020 5 6 , the first attempts to apply this idea to language arrived in 2021, and involved replacing a continuous corruption process with a discrete one to enable modelling of categorical data: multinomial diffusion 7 , D3PM 8 and SUNDAE 9 . Back then, the dominance of autoregression was not as well-established as it is today: GPT-3 10 had turned some heads, but the ‘ChatGPT moment’ wouldn’t come until late 2022. At the time, discrete diffusion seemed to address some real theoretical flaws in the autoregressive paradigm, like exposure bias due to teacher forcing and the relative difficulty of applying it to infilling and constrained generation tasks. Note that there had been some exploration of non-autoregressive and any-order autoregressive approaches in the preceding years 11 12 (especially for machine translation 13 14 ), but not yet from a diffusion perspective. 2022: continuous diffusion for discrete data In 2022, several attempts to apply continuous diffusion to language modelling appeared, starting with Diffusion-LM 15 . This approach addresses the incompatibility between categorical data and corruption with Gaussian noise in a different way: simply represent the discrete categories with continuous embedding vectors, which are perfectly amenable to Gaussian noise corruption. That way, the Gaussian diffusion mechanism, which works so well for images, can be applied without any changes. Diffusion-LM touted the advantages of this alternative generative paradigm for controllable text generation in particular. In the last few months of 2022, quite a few other papers using variations of this approach were published, including DiffuSeq 16 , SSD-LM 17 , Difformer 18 , SeqDiffuSeq 19 , GENIE 20 , LD4LG 21 and also two papers that I worked on: self-conditioned embedding diffusion (SED) 22 and continuous diffusion for categorical data (CDCD) 23 . At the time, the allure of these continuous methods was that they could benefit from all the insights, tools and machinery that were being discovered and developed for continuous diffusion, as it completely took over audiovisual generation. For example, applying some of the sampling and distillation techniques developed for continuous diffusion models to discrete diffusion was often much less straightforward, or even downright impossible. Late 2023: the continuous extinction Then, something interesting happened: after 2023, virtually all new research in this space used discrete diffusion , and continuous diffusion for language went extinct. A diagram from a 2025 survey paper 24 about diffusion language models clearly shows this: New survey on diffusion language models: https://t.co/SHicf69gxV (via @NicolasPerezNi1 ). Covers pre/post-training, inference and multimodality, with very nice illustrations. I can't help but feel a bit wistful about the apparent extinction of the continuous approach after 2023🥲 pic.twitter.com/RYvLHuLHWH — Sander Dieleman (@sedielem) August 19, 2025 Continuous methods are marked in yellow, discrete methods in green. The transition from 2023 to 2024 is quite stark! It is difficult to say for certain why this happened, but I can think of a few potential factors: one is the ChatGPT moment, which gradually shifted the focus of language diffusion research from theoretical advantages and elegance to raw performance. Now, the goal was to try and match powerful autoregressive models at scale , or even outcompete them in specific settings. It seems that people felt closing the performance gap would be easier to achieve with fully discrete methods, perhaps because they are conceptually more closely related to autoregression. Another factor could be that the science of scaling language diffusion models had started to be explored, and initial observations for continuous methods weren’t looking promising. In May 2023, Gulrajani & Hashimoto 25 quantified the training efficiency gap for a likelihood-based continuous diffusion language model (Plaid-1B): 64x less efficient . (Note that the diagram above marks Plaid as discrete, but it is a continuous method.) At a time when the LLM community was still very much focused on the pareto frontier of training compute versus perplexity (Chinchilla-optimality 26 ), any modelling approach whose training efficiency was almost two orders of magnitude worse than an autoregressive baseline was difficult to take seriously. The first LLaMA 27 model, which challenged this training efficiency focus and argued for taking the inference budget into account, had only just been released a few months earlier (February 2023), so I believe it is plausible that the community had not yet internalised this shift. Needless to say, this is all highly speculative. Perhaps it was just a coincidence, and discrete methods ended up having more momentum around that time purely by chance. If you have any thoughts about what could have caused the late-2023 continuous extinction event, I’d be keen to hear them in the comments! Personally, I had stopped working on diffusion language models by that point (I got too busy building image and video generation models: Imagen and Veo , and later on, Nano Banana and Omni ), so I was just observing this evolution from the sidelines. I found it somewhat suprising, because I believed continuous diffusion has a few key advantages, like an ability to represent uncertainty at the individual token level, and a rich toolbox of sampling algorithms and tricks to draw on. Giving those up seemed like it could be a mistake, but the research community as a whole clearly figured that this was the way to go. Adapting continuous diffusion to discrete data We will talk about what’s been happening in the diffusion language modelling space more recently in the next se