Every LLM You've Used Generates One Token at a Time
GPT, Claude, Gemini — every major model generates text token by token. That's the autoregressive architecture, and it's been the standard since GPT-1 in 2018.
On July 2, NVIDIA open-sourced Nemotron-Labs-TwoTower, a language model that generates text using diffusion. In plain terms: instead of producing words one at a time, it processes the entire output sequence in parallel.
The result: 2.42x faster throughput, retaining 98.7% of baseline quality.
The Two-Tower Architecture: Freeze Half, Speed Up Half
NVIDIA's engineers didn't train a new model from scratch. They took Nemotron-3-Nano-30B — a 30B autoregressive model — and split it into two "towers":
- Frozen Context Tower: the original 30B parameters, untouched, responsible for understanding input — doesn't participate in generation
- Denoiser Tower: 30B new parameters, dedicated to diffusion-based text generation
Total parameters reach 60B, but the compute doesn't double. The context tower is frozen, so it doesn't require gradient updates during inference.
The elegance of this design: it doesn't throw away autoregressive capability. It layers diffusion on top. You can run in autoregressive mode or diffusion mode. Both work.
Where the Speed Comes From
Autoregressive models bottleneck on waiting — every token must wait for the previous one to finish. Diffusion models generate the entire sequence in parallel, eliminating that wait entirely.
NVIDIA's benchmarks: generating 256 tokens with 1 block shows roughly 1.8x speedup. Generating 8K tokens with 32 blocks hits over 4x speedup.
The longer the text, the bigger the gain.
Three Decoding Modes, All Available
- Pure diffusion: fastest, minor noise possible
- Autoregressive + diffusion hybrid: diffusion builds the skeleton, autoregressive refines
- Speculative decoding: autoregressive quickly drafts, diffusion verifies
Not Perfect Yet
98.7% quality retention sounds solid, but what's in the other 1.3%? Simple tasks barely show a difference, but highly structured text — code, mathematical derivations, strictly formatted documents — occasionally produces minor deviations.
Also, diffusion mode currently only supports batch generation. Interactive chat scenarios still need autoregressive. But NVIDIA is already working on streaming diffusion output.
Why This Matters
If diffusion LLMs work at scale, the entire inference cost structure gets rewritten.
Right now, running a large model to generate a response spends most of its cost on "decode one token after another." If diffusion can scale up, the same hardware serves more users — or the same experience costs half as much.
And NVIDIA open-sourced it under Apache 2.0. That means other labs can build on TwoTower directly. The barrier to experimenting with diffusion LLMs just dropped.
This isn't about replacing autoregressive models. It's about adding a new option to the LLM architecture toolkit. For high-throughput workloads — batch text generation, code completion, content translation — that option is worth a lot.