{
  "schema_version": "1.0",
  "id": "archive:https://arxiv.org/abs/2607.05381v1",
  "slug": "2607-05381v1-18rb1yh",
  "url": "https://feed7.dev/p/2607-05381v1-18rb1yh",
  "title": "What Does a Discrete Diffusion Model Learn?",
  "why_included": "A unifying theory of discrete diffusion: denoiser, score, and bridge parameterizations are one object in different coordinates — and the wrong choice makes the uniform-noise ELBO diverge at initialization.",
  "summary": "**The gist** The paper proves the **Oracle Distance theorem**: a discrete diffusion model's negative ELBO exactly equals data entropy plus the path KL to the oracle reverse process — an identity, not a bound. The optimizer has three interchangeable coordinates — **denoiser, cavity (bridge plug-in), and score** — with closed-form conversions, and the framework recovers **MDM, UDM, SEDD, and GIDD** as special cases.",
  "practical_implication": "**Why it matters** Directly relevant only if you train or evaluate diffusion language models, but one result is a concrete footgun: a **denoiser parameterization** makes the **uniform-diffusion ELBO diverge at initialization** while the bridge plug-in stays finite. Reading a network in the wrong coordinate changes the process you actually train and sample, so pick the parameterization deliberately, not by convention.",
  "agent_context": "**The gist** The paper proves the **Oracle Distance theorem**: a discrete diffusion model's negative ELBO exactly equals data entropy plus the path KL to the oracle reverse process — an identity, not a bound. The optimizer has three interchangeable coordinates — **denoiser, cavity (bridge plug-in), and score** — with closed-form conversions, and the framework recovers **MDM, UDM, SEDD, and GIDD** as special cases.\n\n**Why it matters** Directly relevant only if you train or evaluate diffusion language models, but one result is a concrete footgun: a **denoiser parameterization** makes the **uniform-diffusion ELBO diverge at initialization** while the bridge plug-in stays finite. Reading a network in the wrong coordinate changes the process you actually train and sample, so pick the parameterization deliberately, not by convention.\n\n**Watch out** Everything is verified on an **exactly solvable model**, not trained systems at scale, and the theory says every noising process shares the same **best achievable ELBO** — so gains must come from parameterization and optimization, not clever noise schedules.",
  "source": {
    "name": "arXiv",
    "url": "https://arxiv.org/abs/2607.05381v1",
    "published_at": null
  },
  "source_class": "blog_post",
  "content_type": "Paper",
  "layer": "model",
  "domains": [
    "research"
  ],
  "topics": [],
  "verification": {
    "status": "needs_review",
    "label": "Needs Review",
    "method": "unverified",
    "verified_at": null
  },
  "uncertainty": [
    "Everything is verified on an **exactly solvable model**, not trained systems at scale, and the theory says every noising process shares the same **best achievable ELBO** — so gains must come from parameterization and optimization, not clever noise schedules."
  ],
  "lifecycle": "Current",
  "published_at": null,
  "modified_at": null,
  "supersedes": [],
  "expires_at": null,
  "formats": {
    "html": "https://feed7.dev/p/2607-05381v1-18rb1yh",
    "json": "https://feed7.dev/p/2607-05381v1-18rb1yh.json",
    "markdown": "https://feed7.dev/p/2607-05381v1-18rb1yh.md"
  }
}