Most modern language models are autoregressive, meaning they generate text one token at a time from left to right [S1, S6]. While this approach scales well, it is inherently sequential during inference [1].
Diffusion models offer a different path. Instead of sequential generation, they start with a corrupted version of a full sequence and gradually refine it into clean text [S5, S6]. This allows for parallel generation and bidirectional controllability [3].
Within this field, a divide exists between discrete and continuous methods. Discrete diffusion works directly on the token vocabulary, often using masks or random entries [S6, S7]. Continuous diffusion instead maps tokens into real-valued embedding vectors and applies Gaussian noise to those vectors [S1, S6].
The Rise and Fall of Continuous Methods
Continuous diffusion for language gained early momentum around 2022 with models like Diffusion-LM [1]. The appeal was the ability to leverage tools and machinery already perfected for image and video generation [1].
However, after 2023, the field shifted almost entirely toward discrete methods [1]. This “extinction” was driven by a focus on raw performance and training efficiency [S1, S6]. Early continuous models showed a steep efficiency gap; one analysis indicated a likelihood-based continuous model required far more compute than an autoregressive baseline to achieve similar results [6].
Discrete diffusion appeared conceptually closer to the categorical nature of text and the structure of autoregressive models [1]. As researchers sought to match the performance of large-scale models, discrete methods became the preferred route [1].
Why Continuous Diffusion is Returning
Recent research suggests that the previous underperformance of continuous models was not an intrinsic limitation but a result of underoptimized training recipes and evaluation protocols [4].
Continuous models offer two primary advantages over their discrete counterparts:
- Few-step generation: Continuous diffusion can map different noise points to different data samples, making one-step generation possible [4]. In contrast, masked discrete diffusion often collapses its prior distribution into a single masked state, which can lead to a “parallel decoding dilemma” and degraded quality during few-step generation [4].
- Controllability: Similar to image generation, continuous latents can be semantically manipulated using guidance mechanisms [4]. Discrete masked diffusion is generally limited to token-level editing, such as clozing [4].
New frameworks like LangFlow have demonstrated that by optimizing noise schedules, using self-conditioning, and refining the negative log-likelihood (NLL) bound, continuous diffusion can rival discrete methods and even outperform autoregressive models on certain zero-shot benchmarks [4].
Overcoming Technical Bottlenecks
Bridging the gap between categorical tokens and continuous space requires specific technical machinery [6].
One challenge is the geometry of the embedding space [6]. If embeddings are learned jointly with the denoiser, the model might reduce error by collapsing token vectors together or allowing their norms to grow [6]. Researchers use regularization, normalization, or extra classification losses to prevent this [6].
Noise scheduling is another critical factor [6]. Many noise levels provide little training signal because they either leave the token identity too intact or destroy it completely [6]. Modern systems now adapt the schedule to the model’s entropy or embedding geometry, spending more training time where the denoiser faces meaningful uncertainty [6].
The Impact of Flow Maps and Consistency
The latest revival of continuous diffusion is closely tied to flow maps and consistency models [6].
Traditional diffusion requires many small denoising steps to produce high-quality output [5]. A flow map learns to approximate the result of that entire sequence in a single larger step [6]. This approach is particularly useful for distillation [6].
Continuous trajectories can carry sequence-level relationships through the path from noise to text [6]. This gives flow-map distillation more room to preserve structure than discrete models, which often assume conditional independence among tokens sampled in the same round [6].
These advancements make diffusion-based language models more practical by reducing the number of refinement passes needed, potentially lowering energy costs and increasing generation speed [5].
If you are tracking the evolution of non-autoregressive generation, explore the latest research on Riemannian Diffusion Language Models (RDLM) to see how manifold geometry is being used to unify these approaches [S2, S3].
Sources
- Continuous diffusion language models - Sander Dieleman
- Continuous diffusion language models make a comeback
- Continuous Diffusion Model for Language Modeling - arXiv.org
- ICML Poster Consistent Diffusion Language Models
- Continuous Diffusion Rivals Discrete in Language Modeling
- [2502.11564] Continuous Diffusion Model for Language Modeling
- A Technical Overview of Continuous and Discrete Diffusion-based …
- Continuous Diffusion Language Models Were Held Back by a Habit … - Medium
