What Does a Discrete Diffusion Model Learn?
A unifying theory of discrete diffusion: denoiser, score, and bridge parameterizations are one object in different coordinates — and the wrong choice makes the uniform-noise ELBO diverge at initialization.
**The gist** The paper proves the **Oracle Distance theorem**: a discrete diffusion model's negative ELBO exactly equals data entropy plus the path KL to the oracle reverse process — an identity, not a bound. The optimizer has three interchangeable coordinates — **denoiser, cavity (bridge plug-in), and score** — with closed-form conversions, and the framework recovers **MDM, UDM, SEDD, and GIDD** as special cases.
**Why it matters** Directly relevant only if you train or evaluate diffusion language models, but one result is a concrete footgun: a **denoiser parameterization** makes the **uniform-diffusion ELBO diverge at initialization** while the bridge plug-in stays finite. Reading a network in the wrong coordinate changes the process you actually train and sample, so pick the parameterization deliberately, not by convention.
**The gist** The paper proves the **Oracle Distance theorem**: a discrete diffusion model's negative ELBO exactly equals data entropy plus the path KL to the oracle reverse process — an identity, not a bound. The optimizer has three interchangeable coordinates — **denoiser, cavity (bridge plug-in), and score** — with closed-form conversions, and the framework recovers **MDM, UDM, SEDD, and GIDD** as special cases. **Why it matters** Directly relevant only if you train or evaluate diffusion language models, but one result is a concrete footgun: a **denoiser parameterization** makes the **uniform-diffusion ELBO diverge at initialization** while the bridge plug-in stays finite. Reading a network in the wrong coordinate changes the process you actually train and sample, so pick the parameterization deliberately, not by convention. **Watch out** Everything is verified on an **exactly solvable model**, not trained systems at scale, and the theory says every noising process shares the same **best achievable ELBO** — so gains must come from parameterization and optimization, not clever noise schedules.