{"id":14282,"date":"2026-05-14T18:15:05","date_gmt":"2026-05-14T17:15:05","guid":{"rendered":"https:\/\/www.blopig.com\/blog\/?p=14282"},"modified":"2026-05-14T18:15:12","modified_gmt":"2026-05-14T17:15:12","slug":"a-timeline-of-sampling-methods-of-diffusion-models","status":"publish","type":"post","link":"https:\/\/www.blopig.com\/blog\/2026\/05\/a-timeline-of-sampling-methods-of-diffusion-models\/","title":{"rendered":"A timeline of sampling methods of diffusion models"},"content":{"rendered":"\n

When approaching the methods used in de-novo protein design, one is quickly confronted with a plethora of overlapping formulations of what looks superficially like “the same thing”. One paper trains an \u03f5<\/mi><\/mrow>\\boldsymbol{\\epsilon}<\/annotation><\/semantics><\/math>-prediction network with a simple<\/em> MSE loss; another trains a score network with a stochastic-differential-equation justification; a third trains a clean-data predictor under yet another schedule. Each formulation carries its own notation, its own variance schedule, and its own sampler. Qualitatively, this zoo of formulations is doing the same thing: it starts from some unstructured noise and iteratively refines it to eventually produce a protein structure similar (but different!) to other proteins we have experimentally determined in the past. What is not immediately obvious to a newcomer is that all of these formulations are historical descendants of a small number of foundational ideas, and that essentially every architectural and algorithmic decision in a modern protein-design diffusion model has a specific paper of origin and a specific motivation for being there.<\/p>\n\n\n\n

This post is my attempt to put these formulations onto a single timeline. I trace the trajectory of the field through four foundational works: DDPM (Ho et al., 2020<\/a>), DDIM (Song et al., 2021a<\/a>), the score-based SDE unification (Song et al., 2021b<\/a>), and EDM (Karras et al., 2022), explaining at each step what specific problem with the previous formulation the next paper was attacking<\/em> and how the new formulation generalises or simplifies the old one<\/em>. The goal is coherent motivation rather than exhaustive coverage; the reader interested in implementation details is referred to the original papers and the references at the end.<\/p>\n\n\n\n

The problem we want to solve<\/h3>\n\n\n\n

Before diving into any specific method, let us be precise about the problem all of them are trying to solve. As mentioned above, we have a target distribution<\/em> p<\/mi>data<\/mtext><\/msub><\/mrow>p_{\\text{data}}<\/annotation><\/semantics><\/math> on R<\/mi>d<\/mi><\/msup><\/mrow>\\mathbb{R}^d<\/annotation><\/semantics><\/math> we are interested in sampling from. For our purposes, think the distribution over plausible protein backbones, side-chain configurations, or full structures. We cannot evaluate p<\/mi>data<\/mtext><\/msub><\/mrow>p_{\\text{data}}<\/annotation><\/semantics><\/math> analytically and we cannot sample from it directly; what we have is a finite dataset of samples {<\/mo>x<\/mi>(<\/mo>i<\/mi>)<\/mo><\/mrow><\/msup>}<\/mo>i<\/mi>=<\/mo>1<\/mn><\/mrow>N<\/mi><\/msubsup>\u223c<\/mo>p<\/mi>data<\/mtext><\/msub><\/mrow>\\{\\mathbf{x}^{(i)}\\}_{i=1}^{N} \\sim p_{\\text{data}}<\/annotation><\/semantics><\/math>. We want a generative model that produces fresh samples from p<\/mi>data<\/mtext><\/msub><\/mrow>p_{\\text{data}}<\/annotation><\/semantics><\/math> at inference time.<\/p>\n\n\n\n

The strategy that diffusion and score-based models all share is the same: introduce a tractable prior<\/em> p<\/mi>prior<\/mtext><\/msub><\/mrow>p_{\\text{prior}}<\/annotation><\/semantics><\/math> (typically an isotropic Gaussian N<\/mi>(<\/mo>0<\/mn>,<\/mo>I<\/mi>)<\/mo><\/mrow>\\mathcal{N}(\\mathbf{0}, \\mathbf{I})<\/annotation><\/semantics><\/math>) that we can sample from trivially, and learn a map<\/em> that transforms samples from p<\/mi>prior<\/mtext><\/msub><\/mrow>p_{\\text{prior}}<\/annotation><\/semantics><\/math> into samples from p<\/mi>data<\/mtext><\/msub><\/mrow>p_{\\text{data}}<\/annotation><\/semantics><\/math>. The methods differ in how they construct this map: as the reverse of a noising Markov chain (DDPM), or as the time-integral of a reverse-time stochastic differential equation (score SDE), with the deterministic probability-flow ODE as a natural sibling. But the goal is identical, and the underlying object (a parameterised, time-dependent transport from a tractable prior to an intractable data distribution) is identical across all of them.<\/p>\n\n\n\n
\"\"<\/a><\/figure>\n\n\n\n

A useful equation to keep in mind throughout is the Gaussian probability path<\/em><\/p>\n\n\n\n

x<\/mi>t<\/mi><\/msub>\u2005\u200a<\/mtext>=<\/mo>\u2005\u200a<\/mtext>\u03b1<\/mi>t<\/mi><\/msub>\u2009<\/mtext>x<\/mi>0<\/mn><\/msub>\u2005\u200a<\/mtext>+<\/mo>\u2005\u200a<\/mtext>\u03c3<\/mi>t<\/mi><\/msub>\u2009<\/mtext>\u03f5<\/mi>,<\/mo><\/mspace>\u03f5<\/mi>\u223c<\/mo>N<\/mi>(<\/mo>0<\/mn>,<\/mo>I<\/mi>)<\/mo>,<\/mo><\/mrow>\\mathbf{x}_t \\;=\\; \\alpha_t\\,\\mathbf{x}_0 \\;+\\; \\sigma_t\\,\\boldsymbol{\\epsilon},\\qquad \\boldsymbol{\\epsilon}\\sim\\mathcal{N}(\\mathbf{0},\\mathbf{I}),<\/annotation><\/semantics><\/math><\/div>\n\n\n\n

shared by all the methods we will discuss. The differences between DDPM, DDIM, VP\/VE-SDE, and EDM can largely be read off as different choices of (<\/mo>\u03b1<\/mi>t<\/mi><\/msub>,<\/mo>\u03c3<\/mi>t<\/mi><\/msub>)<\/mo><\/mrow>(\\alpha_t, \\sigma_t)<\/annotation><\/semantics><\/math> and different ways of turning a learnt regression target back into a sample.<\/p>\n\n\n\n
<\/div>\n\n\n\n

1. DDPM: Denoising as a Hierarchical VAE<\/h2>\n\n\n\n

The modern diffusion era begins with Ho, Jain & Abbeel’s “Denoising Diffusion Probabilistic Models”<\/a><\/strong> ([Ho et al., 2020<\/a>](#ref-ho2020)), which built on the variational construction of Sohl-Dickstein et al. (2015)<\/a> and simplified it into a recipe that could be trained reliably at the scale of CIFAR-10 and LSUN. The contribution is conceptually two-part: a clever choice of variational objective that turns generative modelling into a denoising problem, and an empirical finding that a simplified<\/em> version of that objective trains better than the principled one.<\/p>\n\n\n\n

The forward (noising) process<\/h3>\n\n\n\n

DDPM defines a fixed, parameter-free Markov chain that gradually corrupts a data point x<\/mi>0<\/mn><\/msub>\u223c<\/mo>q<\/mi>(<\/mo>x<\/mi>0<\/mn><\/msub>)<\/mo><\/mrow>\\mathbf{x}_0 \\sim q(\\mathbf{x}_0)<\/annotation><\/semantics><\/math> into Gaussian noise over T<\/mi><\/mrow>T<\/annotation><\/semantics><\/math> discrete steps. With a variance schedule<\/em> {<\/mo>\u03b2<\/mi>t<\/mi><\/msub>}<\/mo>t<\/mi>=<\/mo>1<\/mn><\/mrow>T<\/mi><\/msubsup>\u2282<\/mo>(<\/mo>0<\/mn>,<\/mo>1<\/mn>)<\/mo><\/mrow>\\{\\beta_t\\}_{t=1}^{T}\\subset(0,1)<\/annotation><\/semantics><\/math>,<\/p>\n\n\n\n
q<\/mi>(<\/mo>x<\/mi>t<\/mi><\/msub>\u2223<\/mo>x<\/mi>t<\/mi>\u2212<\/mo>1<\/mn><\/mrow><\/msub>)<\/mo>\u2005\u200a<\/mtext>=<\/mo>\u2005\u200a<\/mtext>N<\/mi>\u2009\u2063<\/mtext>(<\/mo>x<\/mi>t<\/mi><\/msub>;<\/mo>1<\/mn>\u2212<\/mo>\u03b2<\/mi>t<\/mi><\/msub><\/mrow><\/msqrt>\u2009<\/mtext>x<\/mi>t<\/mi>\u2212<\/mo>1<\/mn><\/mrow><\/msub>,<\/mo>\u2009<\/mtext>\u03b2<\/mi>t<\/mi><\/msub>I<\/mi>)<\/mo>.<\/mi><\/mrow>q(\\mathbf{x}_t \\mid \\mathbf{x}_{t-1}) \\;=\\; \\mathcal{N}\\!\\bigl(\\mathbf{x}_t;\\sqrt{1-\\beta_t}\\,\\mathbf{x}_{t-1},\\,\\beta_t\\mathbf{I}\\bigr).<\/annotation><\/semantics><\/math><\/div>\n\n\n\n

In the standard DDPM the schedule is linear<\/strong>: \u03b2<\/mi>t<\/mi><\/msub><\/mrow>\\beta_t<\/annotation><\/semantics><\/math> grows linearly from \u03b2<\/mi>1<\/mn><\/msub>=<\/mo>10<\/mn>\u2212<\/mo>4<\/mn><\/mrow><\/msup><\/mrow>\\beta_1 = 10^{-4}<\/annotation><\/semantics><\/math> to \u03b2<\/mi>T<\/mi><\/msub>=<\/mo>2<\/mn>\u00d7<\/mo>10<\/mn>\u2212<\/mo>2<\/mn><\/mrow><\/msup><\/mrow>\\beta_T = 2\\times 10^{-2}<\/annotation><\/semantics><\/math> over T<\/mi>=<\/mo>1000<\/mn><\/mrow>T = 1000<\/annotation><\/semantics><\/math> steps (Ho et al., 2020<\/a>).<\/p>\n\n\n\n

It is worth pausing to understand what this conditional is doing. The new sample x<\/mi>t<\/mi><\/msub><\/mrow>\\mathbf{x}_t<\/annotation><\/semantics><\/math> is drawn from a Gaussian centred at 1<\/mn>\u2212<\/mo>\u03b2<\/mi>t<\/mi><\/msub><\/mrow><\/msqrt>\u2009<\/mtext>x<\/mi>t<\/mi>\u2212<\/mo>1<\/mn><\/mrow><\/msub><\/mrow>\\sqrt{1-\\beta_t}\\,\\mathbf{x}_{t-1}<\/annotation><\/semantics><\/math> with variance \u03b2<\/mi>t<\/mi><\/msub>I<\/mi><\/mrow>\\beta_t \\mathbf{I}<\/annotation><\/semantics><\/math>. Two observations are crucial here:<\/p>\n\n\n\n
    \n

  1. The mean of x<\/mi>t<\/mi><\/msub><\/mrow>\\mathbf{x}_t<\/annotation><\/semantics><\/math> equals the mean of x<\/mi>t<\/mi>\u2212<\/mo>1<\/mn><\/mrow><\/msub><\/mrow>\\mathbf{x}_{t-1}<\/annotation><\/semantics><\/math> multiplied by 1<\/mn>\u2212<\/mo>\u03b2<\/mi>t<\/mi><\/msub><\/mrow><\/msqrt><<\/mo>1<\/mn><\/mrow>\\sqrt{1-\\beta_t} < 1<\/annotation><\/semantics><\/math>.<\/strong> Iterating this forward, the mean of x<\/mi>t<\/mi><\/msub><\/mrow>\\mathbf{x}_t<\/annotation><\/semantics><\/math> conditional on x<\/mi>0<\/mn><\/msub><\/mrow>\\mathbf{x}_0<\/annotation><\/semantics><\/math> is \u220f<\/mo>s<\/mi>\u2264<\/mo>t<\/mi><\/mrow><\/msub>1<\/mn>\u2212<\/mo>\u03b2<\/mi>s<\/mi><\/msub><\/mrow><\/msqrt>\u22c5<\/mo>x<\/mi>0<\/mn><\/msub><\/mrow>\\prod_{s\\le t} \\sqrt{1-\\beta_s}\\cdot\\mathbf{x}_0<\/annotation><\/semantics><\/math>, a product of factors strictly less than one. Such a product converges to zero as t<\/mi>\u2192<\/mo>T<\/mi><\/mrow>t\\to T<\/annotation><\/semantics><\/math>. In words: the forward process gradually forgets<\/em> the starting point and pushes the conditional mean toward the origin.<\/p>
    <\/li>\n\n\n\n

  2. The variance accumulates from zero towards one.<\/strong> A short recursive calculation (below) shows that if V<\/mi>a<\/mi>r<\/mi><\/mrow>(<\/mo>x<\/mi>0<\/mn><\/msub>)<\/mo>\u2264<\/mo>1<\/mn><\/mrow>\\mathrm{Var}(\\mathbf{x}_0) \\le 1<\/annotation><\/semantics><\/math>, the conditional variance of x<\/mi>t<\/mi><\/msub>\u2223<\/mo>x<\/mi>0<\/mn><\/msub><\/mrow>\\mathbf{x}_t \\mid \\mathbf{x}_0<\/annotation><\/semantics><\/math> grows monotonically from 0<\/mn><\/mrow>0<\/annotation><\/semantics><\/math> at t<\/mi>=<\/mo>0<\/mn><\/mrow>t=0<\/annotation><\/semantics><\/math> to a value close to 1<\/mn><\/mrow>1<\/annotation><\/semantics><\/math> at t<\/mi>=<\/mo>T<\/mi><\/mrow>t = T<\/annotation><\/semantics><\/math>.<\/p>
    <\/li>\n<\/ol>\n\n\n\n

    Putting these together: regardless of where x<\/mi>0<\/mn><\/msub><\/mrow>\\mathbf{x}_0<\/annotation><\/semantics><\/math> lives, q<\/mi>(<\/mo>x<\/mi>T<\/mi><\/msub>\u2223<\/mo>x<\/mi>0<\/mn><\/msub>)<\/mo><\/mrow>q(\\mathbf{x}_T \\mid \\mathbf{x}_0)<\/annotation><\/semantics><\/math> is approximately N<\/mi>(<\/mo>0<\/mn>,<\/mo>I<\/mi>)<\/mo><\/mrow>\\mathcal{N}(\\mathbf{0}, \\mathbf{I})<\/annotation><\/semantics><\/math> for any reasonable \u03b2<\/mi><\/mrow>\\beta<\/annotation><\/semantics><\/math> schedule. This is the property that makes the whole construction work.<\/p>\n\n\n\n

    The crucial algebraic fact about Gaussian Markov chains, and what makes diffusion tractable in the first place, is that the marginal at any<\/em> step t<\/mi><\/mrow>t<\/annotation><\/semantics><\/math> conditioned on x<\/mi>0<\/mn><\/msub><\/mrow>\\mathbf{x}_0<\/annotation><\/semantics><\/math> admits a closed form. Setting \u03b1<\/mi>t<\/mi><\/msub>=<\/mo>1<\/mn>\u2212<\/mo>\u03b2<\/mi>t<\/mi><\/msub><\/mrow>\\alpha_t = 1-\\beta_t<\/annotation><\/semantics><\/math> and \u03b1<\/mi>\u02c9<\/mo><\/mover>t<\/mi><\/msub>=<\/mo>\u220f<\/mo>s<\/mi>=<\/mo>1<\/mn><\/mrow>t<\/mi><\/msubsup>\u03b1<\/mi>s<\/mi><\/msub><\/mrow>\\bar{\\alpha}_t = \\prod_{s=1}^{t}\\alpha_s<\/annotation><\/semantics><\/math>,<\/p>\n\n\n\n
    q<\/mi>(<\/mo>x<\/mi>t<\/mi><\/msub>\u2223<\/mo>x<\/mi>0<\/mn><\/msub>)<\/mo>\u2005\u200a<\/mtext>=<\/mo>\u2005\u200a<\/mtext>N<\/mi>\u2009\u2063<\/mtext>(<\/mo>x<\/mi>t<\/mi><\/msub>;<\/mo>\u03b1<\/mi>\u02c9<\/mo><\/mover>t<\/mi><\/msub><\/msqrt>\u2009<\/mtext>x<\/mi>0<\/mn><\/msub>,<\/mo>\u2009<\/mtext>(<\/mo>1<\/mn>\u2212<\/mo>\u03b1<\/mi>\u02c9<\/mo><\/mover>t<\/mi><\/msub>)<\/mo>I<\/mi>)<\/mo>\u2005\u200a<\/mtext>\u2005\u200a<\/mtext>\u27fa<\/mo>\u2005\u200a<\/mtext>\u2005\u200a<\/mtext>x<\/mi>t<\/mi><\/msub>\u2005\u200a<\/mtext>=<\/mo>\u2005\u200a<\/mtext>\u03b1<\/mi>\u02c9<\/mo><\/mover>t<\/mi><\/msub><\/msqrt>\u2009<\/mtext>x<\/mi>0<\/mn><\/msub>\u2005\u200a<\/mtext>+<\/mo>\u2005\u200a<\/mtext>1<\/mn>\u2212<\/mo>\u03b1<\/mi>\u02c9<\/mo><\/mover>t<\/mi><\/msub><\/mrow><\/msqrt>\u2009<\/mtext>\u03f5<\/mi>,<\/mo><\/mspace>\u03f5<\/mi>\u223c<\/mo>N<\/mi>(<\/mo>0<\/mn>,<\/mo>I<\/mi>)<\/mo>.<\/mi><\/mrow>q(\\mathbf{x}_t \\mid \\mathbf{x}_0) \\;=\\; \\mathcal{N}\\!\\bigl(\\mathbf{x}_t;\\sqrt{\\bar{\\alpha}_t}\\,\\mathbf{x}_0,\\,(1-\\bar{\\alpha}_t)\\mathbf{I}\\bigr)\n\\;\\;\\Longleftrightarrow\\;\\;\n\\mathbf{x}_t \\;=\\; \\sqrt{\\bar{\\alpha}_t}\\,\\mathbf{x}_0 \\;+\\; \\sqrt{1-\\bar{\\alpha}_t}\\,\\boldsymbol{\\epsilon},\\quad \\boldsymbol{\\epsilon}\\sim\\mathcal{N}(\\mathbf{0},\\mathbf{I}).<\/annotation><\/semantics><\/math><\/div>\n\n\n\n

    This is the reparameterisation<\/em>: rather than simulating t<\/mi><\/mrow>t<\/annotation><\/semantics><\/math> forward steps, we can sample x<\/mi>t<\/mi><\/msub><\/mrow>\\mathbf{x}_t<\/annotation><\/semantics><\/math> directly from x<\/mi>0<\/mn><\/msub><\/mrow>\\mathbf{x}_0<\/annotation><\/semantics><\/math> in one go, using a single Gaussian noise draw and a precomputed \u03b1<\/mi>\u02c9<\/mo><\/mover>t<\/mi><\/msub><\/mrow>\\bar{\\alpha}_t<\/annotation><\/semantics><\/math>.<\/p>\n\n\n\n
    \n

    Intuition:<\/em> The forward process gradually destroys information by adding noise. The signal \u03b1<\/mi>\u02c9<\/mo><\/mover>t<\/mi><\/msub><\/msqrt>\u2009<\/mtext>x<\/mi>0<\/mn><\/msub><\/mrow>\\sqrt{\\bar{\\alpha}_t}\\,\\mathbf{x}_0<\/annotation><\/semantics><\/math> is rescaled down<\/em> as t<\/mi><\/mrow>t<\/annotation><\/semantics><\/math> grows, while the noise 1<\/mn>\u2212<\/mo>\u03b1<\/mi>\u02c9<\/mo><\/mover>t<\/mi><\/msub><\/mrow><\/msqrt>\u2009<\/mtext>\u03f5<\/mi><\/mrow>\\sqrt{1-\\bar{\\alpha}_t}\\,\\boldsymbol{\\epsilon}<\/annotation><\/semantics><\/math> is rescaled up<\/em>; together they keep the total variance at unity (when V<\/mi>a<\/mi>r<\/mi><\/mrow>(<\/mo>x<\/mi>0<\/mn><\/msub>)<\/mo>=<\/mo>1<\/mn><\/mrow>\\mathrm{Var}(\\mathbf{x}_0) = 1<\/annotation><\/semantics><\/math>). This is what we will later call a variance-preserving<\/em> path.<\/p>\n

    It is worth pausing on how nontrivial this is. No matter where you start<\/strong>, a protein structure, an image of a cat, a noisy sketch, applying small Gaussian perturbations enough times always lands you at the same easy-to-sample distribution: an isotropic Gaussian. The forward direction is universal. The generative problem reduces to inverting this “universal contraction”<\/strong>: if we can learn to undo the corruption step-by-step, we can sample new data points by starting at the Gaussian prior and reversing the chain.<\/p>\n<\/div>\n\n\n\n

    A latent-variable view, and why we need a variational bound<\/h3>\n\n\n\n

    Suppose we want to use the reverse direction to draw samples. We define a generative model p<\/mi>\u03b8<\/mi><\/msub><\/mrow>p_\\theta<\/annotation><\/semantics><\/math> that parameterises a reverse<\/em> Markov chain,<\/p>\n\n\n\n
    p<\/mi>\u03b8<\/mi><\/msub>(<\/mo>x<\/mi>0<\/mn>:<\/mo>T<\/mi><\/mrow><\/msub>)<\/mo>\u2005\u200a<\/mtext>=<\/mo>\u2005\u200a<\/mtext>p<\/mi>(<\/mo>x<\/mi>T<\/mi><\/msub>)<\/mo>\u2009<\/mtext>\u220f<\/mo>t<\/mi>=<\/mo>1<\/mn><\/mrow>T<\/mi><\/munderover>p<\/mi>\u03b8<\/mi><\/msub>(<\/mo>x<\/mi>t<\/mi>\u2212<\/mo>1<\/mn><\/mrow><\/msub>\u2223<\/mo>x<\/mi>t<\/mi><\/msub>)<\/mo>,<\/mo><\/mrow>p_\\theta(\\mathbf{x}_{0:T}) \\;=\\; p(\\mathbf{x}_T)\\,\\prod_{t=1}^{T} p_\\theta(\\mathbf{x}_{t-1}\\mid\\mathbf{x}_t),<\/annotation><\/semantics><\/math><\/div>\n\n\n\n

    where the prior p<\/mi>(<\/mo>x<\/mi>T<\/mi><\/msub>)<\/mo>=<\/mo>N<\/mi>(<\/mo>0<\/mn>,<\/mo>I<\/mi>)<\/mo><\/mrow>p(\\mathbf{x}_T) = \\mathcal{N}(\\mathbf{0},\\mathbf{I})<\/annotation><\/semantics><\/math> matches the limiting marginal of the forward process, and each reverse transition is a Gaussian with parameters (<\/mo>\u03bc<\/mi>\u03b8<\/mi><\/msub>(<\/mo>x<\/mi>t<\/mi><\/msub>,<\/mo>t<\/mi>)<\/mo>,<\/mo>\u03a3<\/mi>\u03b8<\/mi><\/msub>(<\/mo>x<\/mi>t<\/mi><\/msub>,<\/mo>t<\/mi>)<\/mo>)<\/mo><\/mrow>(\\boldsymbol{\\mu}_\\theta(\\mathbf{x}_t,t), \\boldsymbol{\\Sigma}_\\theta(\\mathbf{x}_t,t))<\/annotation><\/semantics><\/math> predicted by a neural network. To train, we want to maximise the likelihood p<\/mi>\u03b8<\/mi><\/msub>(<\/mo>x<\/mi>0<\/mn><\/msub>)<\/mo><\/mrow>p_\\theta(\\mathbf{x}_0)<\/annotation><\/semantics><\/math> of the observed data. Why the joint<\/em> and not just p<\/mi>\u03b8<\/mi><\/msub>(<\/mo>x<\/mi>0<\/mn><\/msub>)<\/mo><\/mrow>p_\\theta(\\mathbf{x}_0)<\/annotation><\/semantics><\/math> directly? Because we cannot evaluate p<\/mi>\u03b8<\/mi><\/msub>(<\/mo>x<\/mi>0<\/mn><\/msub>)<\/mo><\/mrow>p_\\theta(\\mathbf{x}_0)<\/annotation><\/semantics><\/math> in closed form: it is the marginal of the joint over all intermediate states:<\/p>\n\n\n\n
    p<\/mi>\u03b8<\/mi><\/msub>(<\/mo>x<\/mi>0<\/mn><\/msub>)<\/mo>\u2005\u200a<\/mtext>=<\/mo>\u2005\u200a<\/mtext>\u222b<\/mo>p<\/mi>\u03b8<\/mi><\/msub>(<\/mo>x<\/mi>0<\/mn>:<\/mo>T<\/mi><\/mrow><\/msub>)<\/mo>\u2009<\/mtext>d<\/mi>x<\/mi>1<\/mn>:<\/mo>T<\/mi><\/mrow><\/msub>.<\/mi><\/mrow>p_\\theta(\\mathbf{x}_0) \\;=\\; \\int p_\\theta(\\mathbf{x}_{0:T})\\,\\mathrm{d}\\mathbf{x}_{1:T}.<\/annotation><\/semantics><\/math><\/div>\n\n\n\n

    The integral on the right is over T<\/mi><\/mrow>T<\/annotation><\/semantics><\/math> high-dimensional latent variables and is intractable. This is the standard situation in latent-variable models, and the standard fix, due to the VAE literature, is to lower-bound the log-likelihood with the evidence lower bound (ELBO)<\/strong>. We introduce a variational distribution<\/em> q<\/mi>(<\/mo>x<\/mi>1<\/mn>:<\/mo>T<\/mi><\/mrow><\/msub>\u2223<\/mo>x<\/mi>0<\/mn><\/msub>)<\/mo><\/mrow>q(\\mathbf{x}_{1:T}\\mid\\mathbf{x}_0)<\/annotation><\/semantics><\/math> over the latents, multiply and divide by it inside the integral, and apply Jensen’s inequality to the resulting log of an expectation:<\/p>\n\n\n\n
    log<\/mi>\u2061<\/mo>p<\/mi>\u03b8<\/mi><\/msub>(<\/mo>x<\/mi>0<\/mn><\/msub>)<\/mo>\u2005\u200a<\/mtext>=<\/mo>\u2005\u200a<\/mtext>log<\/mi>\u2061<\/mo>\u222b<\/mo>q<\/mi>(<\/mo>x<\/mi>1<\/mn>:<\/mo>T<\/mi><\/mrow><\/msub>\u2223<\/mo>x<\/mi>0<\/mn><\/msub>)<\/mo>\u2009<\/mtext>p<\/mi>\u03b8<\/mi><\/msub>(<\/mo>x<\/mi>0<\/mn>:<\/mo>T<\/mi><\/mrow><\/msub>)<\/mo><\/mrow>q<\/mi>(<\/mo>x<\/mi>1<\/mn>:<\/mo>T<\/mi><\/mrow><\/msub>\u2223<\/mo>x<\/mi>0<\/mn><\/msub>)<\/mo><\/mrow><\/mfrac>\u2009<\/mtext>d<\/mi>x<\/mi>1<\/mn>:<\/mo>T<\/mi><\/mrow><\/msub>\u2005\u200a<\/mtext>\u2265<\/mo>\u2005\u200a<\/mtext>E<\/mi>q<\/mi><\/msub>\u2009\u2063<\/mtext>[<\/mo>log<\/mi>\u2061<\/mo>p<\/mi>\u03b8<\/mi><\/msub>(<\/mo>x<\/mi>0<\/mn>:<\/mo>T<\/mi><\/mrow><\/msub>)<\/mo><\/mrow>q<\/mi>(<\/mo>x<\/mi>1<\/mn>:<\/mo>T<\/mi><\/mrow><\/msub>\u2223<\/mo>x<\/mi>0<\/mn><\/msub>)<\/mo><\/mrow><\/mfrac>]<\/mo><\/mrow>.<\/mi><\/mrow>\\log p_\\theta(\\mathbf{x}_0)\n\\;=\\; \\log \\int q(\\mathbf{x}_{1:T}\\mid\\mathbf{x}_0)\\,\\frac{p_\\theta(\\mathbf{x}_{0:T})}{q(\\mathbf{x}_{1:T}\\mid\\mathbf{x}_0)}\\,\\mathrm{d}\\mathbf{x}_{1:T}\n\\;\\ge\\; \\mathbb{E}_{q}\\!\\left[\\log\\frac{p_\\theta(\\mathbf{x}_{0:T})}{q(\\mathbf{x}_{1:T}\\mid\\mathbf{x}_0)}\\right].<\/annotation><\/semantics><\/math><\/div>\n\n\n\n

    The expectation on the right is the evidence lower bound<\/em>. Its negative is the variational lower bound loss<\/strong>,<\/p>\n\n\n\n

    L<\/mi>VLB<\/mtext><\/msub>\u2005\u200a<\/mtext>=<\/mo>\u2005\u200a<\/mtext>\u2212<\/mo>\u2009<\/mtext>E<\/mi>q<\/mi><\/msub>\u2009\u2063<\/mtext>[<\/mo>log<\/mi>\u2061<\/mo>p<\/mi>\u03b8<\/mi><\/msub>(<\/mo>x<\/mi>0<\/mn>:<\/mo>T<\/mi><\/mrow><\/msub>)<\/mo><\/mrow>q<\/mi>(<\/mo>x<\/mi>1<\/mn>:<\/mo>T<\/mi><\/mrow><\/msub>\u2223<\/mo>x<\/mi>0<\/mn><\/msub>)<\/mo><\/mrow><\/mfrac>]<\/mo><\/mrow>,<\/mo><\/mrow>\\mathcal{L}_{\\text{VLB}} \\;=\\; -\\,\\mathbb{E}_{q}\\!\\left[\\log\\frac{p_\\theta(\\mathbf{x}_{0:T})}{q(\\mathbf{x}_{1:T}\\mid\\mathbf{x}_0)}\\right],<\/annotation><\/semantics><\/math><\/div>\n\n\n\n

    which we minimise to push the model joint toward the data joint induced by q<\/mi><\/mrow>q<\/annotation><\/semantics><\/math>.<\/p>\n\n\n\n

    What makes diffusion special compared to a generic VAE is that we fix<\/em> the variational distribution q<\/mi><\/mrow>q<\/annotation><\/semantics><\/math> to be the known forward process, not learnt, not parameterised, just the simple Gaussian chain we constructed above. This is the conceptual key: a diffusion model is a hierarchical VAE with T<\/mi><\/mrow>T<\/annotation><\/semantics><\/math> latents whose encoder is frozen by design and whose decoder is a learnable reverse Markov chain.<\/p>\n\n\n\n
    \"\"<\/a><\/figure>\n\n\n\n

    The diagram above summarises the construction we have just walked through: a frozen forward Markov chain q<\/mi><\/mrow>q<\/annotation><\/semantics><\/math> that destroys structure, and a learnable reverse chain p<\/mi>\u03b8<\/mi><\/msub><\/mrow>p_\\theta<\/annotation><\/semantics><\/math> that reconstructs it.<\/p>\n\n\n\n

    Decomposing L<\/mi>VLB<\/mtext><\/msub><\/mrow>\\mathcal{L}_{\\text{VLB}}<\/annotation><\/semantics><\/math><\/h3>\n\n\n\n

    The integral defining L<\/mi>VLB<\/mtext><\/msub><\/mrow>\\mathcal{L}_{\\text{VLB}}<\/annotation><\/semantics><\/math> is still high-dimensional, but it admits a clean per-step decomposition. Start by expanding the ratio inside the log, using the factorisations of p<\/mi>\u03b8<\/mi><\/msub>(<\/mo>x<\/mi>0<\/mn>:<\/mo>T<\/mi><\/mrow><\/msub>)<\/mo><\/mrow>p_\\theta(\\mathbf{x}_{0:T})<\/annotation><\/semantics><\/math> and q<\/mi>(<\/mo>x<\/mi>1<\/mn>:<\/mo>T<\/mi><\/mrow><\/msub>\u2223<\/mo>x<\/mi>0<\/mn><\/msub>)<\/mo><\/mrow>q(\\mathbf{x}_{1:T}\\mid\\mathbf{x}_0)<\/annotation><\/semantics><\/math>:<\/p>\n\n\n\n
    log<\/mi>\u2061<\/mo>p<\/mi>\u03b8<\/mi><\/msub>(<\/mo>x<\/mi>0<\/mn>:<\/mo>T<\/mi><\/mrow><\/msub>)<\/mo><\/mrow>q<\/mi>(<\/mo>x<\/mi>1<\/mn>:<\/mo>T<\/mi><\/mrow><\/msub>\u2223<\/mo>x<\/mi>0<\/mn><\/msub>)<\/mo><\/mrow><\/mfrac>\u2005\u200a<\/mtext>=<\/mo>\u2005\u200a<\/mtext>log<\/mi>\u2061<\/mo>p<\/mi>(<\/mo>x<\/mi>T<\/mi><\/msub>)<\/mo>+<\/mo>\u2211<\/mo>t<\/mi>=<\/mo>1<\/mn><\/mrow>T<\/mi><\/munderover>log<\/mi>\u2061<\/mo>p<\/mi>\u03b8<\/mi><\/msub>(<\/mo>x<\/mi>t<\/mi>\u2212<\/mo>1<\/mn><\/mrow><\/msub>\u2223<\/mo>x<\/mi>t<\/mi><\/msub>)<\/mo><\/mrow>q<\/mi>(<\/mo>x<\/mi>t<\/mi><\/msub>\u2223<\/mo>x<\/mi>t<\/mi>\u2212<\/mo>1<\/mn><\/mrow><\/msub>)<\/mo><\/mrow><\/mfrac>.<\/mi><\/mrow>\\log\\frac{p_\\theta(\\mathbf{x}_{0:T})}{q(\\mathbf{x}_{1:T}\\mid\\mathbf{x}_0)} \\;=\\; \\log p(\\mathbf{x}_T) + \\sum_{t=1}^{T}\\log\\frac{p_\\theta(\\mathbf{x}_{t-1}\\mid\\mathbf{x}_t)}{q(\\mathbf{x}_t\\mid\\mathbf{x}_{t-1})}.<\/annotation><\/semantics><\/math><\/div>\n\n\n\n

    The forward conditionals q<\/mi>(<\/mo>x<\/mi>t<\/mi><\/msub>\u2223<\/mo>x<\/mi>t<\/mi>\u2212<\/mo>1<\/mn><\/mrow><\/msub>)<\/mo><\/mrow>q(\\mathbf{x}_t\\mid\\mathbf{x}_{t-1})<\/annotation><\/semantics><\/math> are not directly comparable to the reverse conditionals p<\/mi>\u03b8<\/mi><\/msub>(<\/mo>x<\/mi>t<\/mi>\u2212<\/mo>1<\/mn><\/mrow><\/msub>\u2223<\/mo>x<\/mi>t<\/mi><\/msub>)<\/mo><\/mrow>p_\\theta(\\mathbf{x}_{t-1}\\mid\\mathbf{x}_t)<\/annotation><\/semantics><\/math>: they go in opposite directions. To make them comparable, rewrite q<\/mi>(<\/mo>x<\/mi>t<\/mi><\/msub>\u2223<\/mo>x<\/mi>t<\/mi>\u2212<\/mo>1<\/mn><\/mrow><\/msub>)<\/mo><\/mrow>q(\\mathbf{x}_t \\mid \\mathbf{x}_{t-1})<\/annotation><\/semantics><\/math> for t<\/mi>\u2265<\/mo>2<\/mn><\/mrow>t \\ge 2<\/annotation><\/semantics><\/math> using Bayes’ rule, conditioning everything on x<\/mi>0<\/mn><\/msub><\/mrow>\\mathbf{x}_0<\/annotation><\/semantics><\/math>:<\/p>\n\n\n\n
    q<\/mi>(<\/mo>x<\/mi>t<\/mi><\/msub>\u2223<\/mo>x<\/mi>t<\/mi>\u2212<\/mo>1<\/mn><\/mrow><\/msub>)<\/mo>\u2005\u200a<\/mtext>=<\/mo>\u2005\u200a<\/mtext>q<\/mi>(<\/mo>x<\/mi>t<\/mi><\/msub>\u2223<\/mo>x<\/mi>t<\/mi>\u2212<\/mo>1<\/mn><\/mrow><\/msub>,<\/mo>x<\/mi>0<\/mn><\/msub>)<\/mo>\u2005\u200a<\/mtext>=<\/mo>\u2005\u200a<\/mtext>q<\/mi>(<\/mo>x<\/mi>t<\/mi>\u2212<\/mo>1<\/mn><\/mrow><\/msub>\u2223<\/mo>x<\/mi>t<\/mi><\/msub>,<\/mo>x<\/mi>0<\/mn><\/msub>)<\/mo>\u2009<\/mtext>q<\/mi>(<\/mo>x<\/mi>t<\/mi><\/msub>\u2223<\/mo>x<\/mi>0<\/mn><\/msub>)<\/mo><\/mrow>q<\/mi>(<\/mo>x<\/mi>t<\/mi>\u2212<\/mo>1<\/mn><\/mrow><\/msub>\u2223<\/mo>x<\/mi>0<\/mn><\/msub>)<\/mo><\/mrow><\/mfrac>.<\/mi><\/mrow>q(\\mathbf{x}_t\\mid\\mathbf{x}_{t-1}) \\;=\\; q(\\mathbf{x}_t\\mid\\mathbf{x}_{t-1},\\mathbf{x}_0) \\;=\\; \\frac{q(\\mathbf{x}_{t-1}\\mid\\mathbf{x}_t,\\mathbf{x}_0)\\,q(\\mathbf{x}_t\\mid\\mathbf{x}_0)}{q(\\mathbf{x}_{t-1}\\mid\\mathbf{x}_0)}.<\/annotation><\/semantics><\/math><\/div>\n\n\n\n

    The first equality uses the Markov property of the forward chain; the second is Bayes. Substituting back and using a telescoping argument on the marginals (full derivation in [Ho et al., 2020<\/a>, appendix A](#ref-ho2020)), one obtains<\/p>\n\n\n\n

    L<\/mi>VLB<\/mtext><\/msub>\u2005\u200a<\/mtext>=<\/mo>\u2005\u200a<\/mtext>D<\/mi>K<\/mi>L<\/mi><\/mrow><\/msub>\u2009\u2063<\/mtext>(<\/mo>q<\/mi>(<\/mo>x<\/mi>T<\/mi><\/msub>\u2223<\/mo>x<\/mi>0<\/mn><\/msub>)<\/mo>\u2009<\/mtext>\u2225<\/mi>\u2009<\/mtext>p<\/mi>(<\/mo>x<\/mi>T<\/mi><\/msub>)<\/mo>)<\/mo><\/mrow>\u23df<\/mo><\/munder>L<\/mi>T<\/mi><\/msub><\/munder>\u2005\u200a<\/mtext>+<\/mo>\u2005\u200a<\/mtext>\u2211<\/mo>t<\/mi>=<\/mo>2<\/mn><\/mrow>T<\/mi><\/munderover>E<\/mi>q<\/mi><\/msub>\u2009\u2063<\/mtext>[<\/mo>D<\/mi>K<\/mi>L<\/mi><\/mrow><\/msub>\u2009\u2063<\/mtext>(<\/mo>q<\/mi>(<\/mo>x<\/mi>t<\/mi>\u2212<\/mo>1<\/mn><\/mrow><\/msub>\u2223<\/mo>x<\/mi>t<\/mi><\/msub>,<\/mo>x<\/mi>0<\/mn><\/msub>)<\/mo>\u2009<\/mtext>\u2225<\/mi>\u2009<\/mtext>p<\/mi>\u03b8<\/mi><\/msub>(<\/mo>x<\/mi>t<\/mi>\u2212<\/mo>1<\/mn><\/mrow><\/msub>\u2223<\/mo>x<\/mi>t<\/mi><\/msub>)<\/mo>)<\/mo>]<\/mo><\/mrow><\/mrow>\u23df<\/mo><\/munder>L<\/mi>t<\/mi>\u2212<\/mo>1<\/mn><\/mrow><\/msub><\/munder>\u2005\u200a<\/mtext>\u2212<\/mo>\u2009<\/mtext>E<\/mi>q<\/mi><\/msub>\u2009\u2063<\/mtext>[<\/mo>log<\/mi>\u2061<\/mo>p<\/mi>\u03b8<\/mi><\/msub>(<\/mo>x<\/mi>0<\/mn><\/msub>\u2223<\/mo>x<\/mi>1<\/mn><\/msub>)<\/mo>]<\/mo><\/mrow><\/mrow>\u23df<\/mo><\/munder>L<\/mi>0<\/mn><\/msub><\/munder>.<\/mi><\/mrow>\\mathcal{L}_{\\text{VLB}} \\;=\\; \\underbrace{D_{\\mathrm{KL}}\\!\\bigl(q(\\mathbf{x}_T\\mid\\mathbf{x}_0)\\,\\Vert\\,p(\\mathbf{x}_T)\\bigr)}_{L_T} \\;+\\; \\sum_{t=2}^{T}\\underbrace{\\mathbb{E}_{q}\\!\\left[D_{\\mathrm{KL}}\\!\\bigl(q(\\mathbf{x}_{t-1}\\mid\\mathbf{x}_t,\\mathbf{x}_0)\\,\\Vert\\,p_\\theta(\\mathbf{x}_{t-1}\\mid\\mathbf{x}_t)\\bigr)\\right]}_{L_{t-1}} \\;\\underbrace{-\\,\\mathbb{E}_{q}\\!\\left[\\log p_\\theta(\\mathbf{x}_0\\mid\\mathbf{x}_1)\\right]}_{L_0}.<\/annotation><\/semantics><\/math><\/div>\n\n\n\n

    This decomposition has a beautiful structure. L<\/mi>T<\/mi><\/msub><\/mrow>L_T<\/annotation><\/semantics><\/math> is essentially constant: it measures how close the forward marginal q<\/mi>(<\/mo>x<\/mi>T<\/mi><\/msub>\u2223<\/mo>x<\/mi>0<\/mn><\/msub>)<\/mo><\/mrow>q(\\mathbf{x}_T\\mid\\mathbf{x}_0)<\/annotation><\/semantics><\/math> is to the prior p<\/mi>(<\/mo>x<\/mi>T<\/mi><\/msub>)<\/mo>=<\/mo>N<\/mi>(<\/mo>0<\/mn>,<\/mo>I<\/mi>)<\/mo><\/mrow>p(\\mathbf{x}_T) = \\mathcal{N}(\\mathbf{0},\\mathbf{I})<\/annotation><\/semantics><\/math>, and is approximately zero by construction. L<\/mi>0<\/mn><\/msub><\/mrow>L_0<\/annotation><\/semantics><\/math> is a final-step log-likelihood that becomes negligible at high T<\/mi><\/mrow>T<\/annotation><\/semantics><\/math>. The interesting terms are the per-step KL divergences L<\/mi>t<\/mi>\u2212<\/mo>1<\/mn><\/mrow><\/msub><\/mrow>L_{t-1}<\/annotation><\/semantics><\/math>, which compare the true posterior<\/em> q<\/mi>(<\/mo>x<\/mi>t<\/mi>\u2212<\/mo>1<\/mn><\/mrow><\/msub>\u2223<\/mo>x<\/mi>t<\/mi><\/msub>,<\/mo>x<\/mi>0<\/mn><\/msub>)<\/mo><\/mrow>q(\\mathbf{x}_{t-1}\\mid\\mathbf{x}_t,\\mathbf{x}_0)<\/annotation><\/semantics><\/math>, a Gaussian with closed form thanks to Bayes’ rule on Gaussians, against the model’s reverse conditional p<\/mi>\u03b8<\/mi><\/msub>(<\/mo>x<\/mi>t<\/mi>\u2212<\/mo>1<\/mn><\/mrow><\/msub>\u2223<\/mo>x<\/mi>t<\/mi><\/msub>)<\/mo><\/mrow>p_\\theta(\\mathbf{x}_{t-1}\\mid\\mathbf{x}_t)<\/annotation><\/semantics><\/math>.<\/p>\n\n\n\n

    The true posterior is<\/p>\n\n\n\n

    q<\/mi>(<\/mo>x<\/mi>t<\/mi>\u2212<\/mo>1<\/mn><\/mrow><\/msub>\u2223<\/mo>x<\/mi>t<\/mi><\/msub>,<\/mo>x<\/mi>0<\/mn><\/msub>)<\/mo>\u2005\u200a<\/mtext>=<\/mo>\u2005\u200a<\/mtext>N<\/mi>(<\/mo>x<\/mi>t<\/mi>\u2212<\/mo>1<\/mn><\/mrow><\/msub>;<\/mo>\u2009<\/mtext>\u03bc<\/mi>~<\/mo><\/mover>t<\/mi><\/msub>(<\/mo>x<\/mi>t<\/mi><\/msub>,<\/mo>x<\/mi>0<\/mn><\/msub>)<\/mo>,<\/mo>\u2009<\/mtext>\u03b2<\/mi>~<\/mo><\/mover>t<\/mi><\/msub>I<\/mi>)<\/mo>,<\/mo><\/mrow>q(\\mathbf{x}_{t-1}\\mid\\mathbf{x}_t,\\mathbf{x}_0) \\;=\\; \\mathcal{N}\\bigl(\\mathbf{x}_{t-1};\\,\\tilde{\\boldsymbol{\\mu}}_t(\\mathbf{x}_t,\\mathbf{x}_0),\\,\\tilde{\\beta}_t\\mathbf{I}\\bigr),<\/annotation><\/semantics><\/math><\/div>\n\n\n\n

    with closed-form posterior mean and variance<\/p>\n\n\n\n

    \u03bc<\/mi>~<\/mo><\/mover>t<\/mi><\/msub>(<\/mo>x<\/mi>t<\/mi><\/msub>,<\/mo>x<\/mi>0<\/mn><\/msub>)<\/mo>\u2005\u200a<\/mtext>=<\/mo>\u2005\u200a<\/mtext>\u03b1<\/mi>\u02c9<\/mo><\/mover>t<\/mi>\u2212<\/mo>1<\/mn><\/mrow><\/msub><\/msqrt>\u2009<\/mtext>\u03b2<\/mi>t<\/mi><\/msub><\/mrow>1<\/mn>\u2212<\/mo>\u03b1<\/mi>\u02c9<\/mo><\/mover>t<\/mi><\/msub><\/mrow><\/mfrac>\u2009<\/mtext>x<\/mi>0<\/mn><\/msub>\u2005\u200a<\/mtext>+<\/mo>\u2005\u200a<\/mtext>\u03b1<\/mi>t<\/mi><\/msub><\/msqrt>\u2009<\/mtext>(<\/mo>1<\/mn>\u2212<\/mo>\u03b1<\/mi>\u02c9<\/mo><\/mover>t<\/mi>\u2212<\/mo>1<\/mn><\/mrow><\/msub>)<\/mo><\/mrow>1<\/mn>\u2212<\/mo>\u03b1<\/mi>\u02c9<\/mo><\/mover>t<\/mi><\/msub><\/mrow><\/mfrac>\u2009<\/mtext>x<\/mi>t<\/mi><\/msub>,<\/mo><\/mspace>\u03b2<\/mi>~<\/mo><\/mover>t<\/mi><\/msub>\u2005\u200a<\/mtext>=<\/mo>\u2005\u200a<\/mtext>1<\/mn>\u2212<\/mo>\u03b1<\/mi>\u02c9<\/mo><\/mover>t<\/mi>\u2212<\/mo>1<\/mn><\/mrow><\/msub><\/mrow>1<\/mn>\u2212<\/mo>\u03b1<\/mi>\u02c9<\/mo><\/mover>t<\/mi><\/msub><\/mrow><\/mfrac>\u2009<\/mtext>\u03b2<\/mi>t<\/mi><\/msub>.<\/mi><\/mrow>\\tilde{\\boldsymbol{\\mu}}_t(\\mathbf{x}_t,\\mathbf{x}_0) \\;=\\; \\frac{\\sqrt{\\bar{\\alpha}_{t-1}}\\,\\beta_t}{1-\\bar{\\alpha}_t}\\,\\mathbf{x}_0 \\;+\\; \\frac{\\sqrt{\\alpha_t}\\,(1-\\bar{\\alpha}_{t-1})}{1-\\bar{\\alpha}_t}\\,\\mathbf{x}_t,\\qquad \\tilde{\\beta}_t \\;=\\; \\frac{1-\\bar{\\alpha}_{t-1}}{1-\\bar{\\alpha}_t}\\,\\beta_t.<\/annotation><\/semantics><\/math><\/div>\n\n\n\n

    Here we can notice how the new mean of the gaussian is an interpolation between the noisy state in which we are at point t<\/mi><\/mrow>t<\/annotation><\/semantics><\/math> and the clean\/unnoised sample at time zero!<\/p>\n\n\n\n

    From KL to MSE on means<\/h3>\n\n\n\n

    A key fact, originally due to Feller (1949)<\/a>, is that when the forward step q<\/mi>(<\/mo>x<\/mi>t<\/mi><\/msub>\u2223<\/mo>x<\/mi>t<\/mi>\u2212<\/mo>1<\/mn><\/mrow><\/msub>)<\/mo><\/mrow>q(\\mathbf{x}_t \\mid \\mathbf{x}_{t-1})<\/annotation><\/semantics><\/math> is Gaussian with sufficiently small variance \u03b2<\/mi>t<\/mi><\/msub><\/mrow>\\beta_t<\/annotation><\/semantics><\/math>, the exact<\/em> reverse step q<\/mi>(<\/mo>x<\/mi>t<\/mi>\u2212<\/mo>1<\/mn><\/mrow><\/msub>\u2223<\/mo>x<\/mi>t<\/mi><\/msub>)<\/mo><\/mrow>q(\\mathbf{x}_{t-1} \\mid \\mathbf{x}_t)<\/annotation><\/semantics><\/math> is also approximately Gaussian. This is the formal justification for parameterising the model’s reverse conditional as a Gaussian with fixed covariance: p<\/mi>\u03b8<\/mi><\/msub>(<\/mo>x<\/mi>t<\/mi>\u2212<\/mo>1<\/mn><\/mrow><\/msub>\u2223<\/mo>x<\/mi>t<\/mi><\/msub>)<\/mo>=<\/mo>N<\/mi>(<\/mo>x<\/mi>t<\/mi>\u2212<\/mo>1<\/mn><\/mrow><\/msub>;<\/mo>\u2009<\/mtext>\u03bc<\/mi>\u03b8<\/mi><\/msub>(<\/mo>x<\/mi>t<\/mi><\/msub>,<\/mo>t<\/mi>)<\/mo>,<\/mo>\u2009<\/mtext>\u03c3<\/mi>t<\/mi>2<\/mn><\/msubsup>I<\/mi>)<\/mo><\/mrow>p_\\theta(\\mathbf{x}_{t-1}\\mid\\mathbf{x}_t) = \\mathcal{N}(\\mathbf{x}_{t-1};\\,\\boldsymbol{\\mu}_\\theta(\\mathbf{x}_t,t),\\,\\sigma_t^2\\mathbf{I})<\/annotation><\/semantics><\/math> with \u03c3<\/mi>t<\/mi>2<\/mn><\/msubsup>\u2208<\/mo>{<\/mo>\u03b2<\/mi>t<\/mi><\/msub>,<\/mo>\u03b2<\/mi>~<\/mo><\/mover>t<\/mi><\/msub>}<\/mo><\/mrow>\\sigma_t^2 \\in \\{\\beta_t, \\tilde{\\beta}_t\\}<\/annotation><\/semantics><\/math>. The variance is fixed<\/em> (not learned) to one of the two endpoints corresponding to the upper and lower bounds on the true reverse variance derived in Ho et al. (2020)<\/a>. The KL divergence between two Gaussians with the same covariance \u03c3<\/mi>t<\/mi>2<\/mn><\/msubsup>I<\/mi><\/mrow>\\sigma_t^2\\mathbf{I}<\/annotation><\/semantics><\/math> has a particularly simple form, it reduces to a scaled squared distance between the means:<\/p>\n\n\n\n
    D<\/mi>K<\/mi>L<\/mi><\/mrow><\/msub>\u2009\u2063<\/mtext>(<\/mo>N<\/mi>(<\/mo>\u03bc<\/mi>~<\/mo><\/mover>t<\/mi><\/msub>,<\/mo>\u03c3<\/mi>t<\/mi>2<\/mn><\/msubsup>I<\/mi>)<\/mo>\u2009<\/mtext>\u2225<\/mi>\u2009<\/mtext>N<\/mi>(<\/mo>\u03bc<\/mi>\u03b8<\/mi><\/msub>,<\/mo>\u03c3<\/mi>t<\/mi>2<\/mn><\/msubsup>I<\/mi>)<\/mo>)<\/mo>\u2005\u200a<\/mtext>=<\/mo>\u2005\u200a<\/mtext>1<\/mn>2<\/mn>\u03c3<\/mi>t<\/mi>2<\/mn><\/msubsup><\/mrow><\/mfrac>\u2009<\/mtext>\u2225<\/mo>\u03bc<\/mi>~<\/mo><\/mover>t<\/mi><\/msub>\u2212<\/mo>\u03bc<\/mi>\u03b8<\/mi><\/msub>\u2225<\/mo>2<\/mn><\/msup>.<\/mi><\/mrow>D_{\\mathrm{KL}}\\!\\bigl(\\mathcal{N}(\\tilde{\\boldsymbol{\\mu}}_t,\\sigma_t^2\\mathbf{I})\\,\\Vert\\,\\mathcal{N}(\\boldsymbol{\\mu}_\\theta,\\sigma_t^2\\mathbf{I})\\bigr) \\;=\\; \\frac{1}{2\\sigma_t^2}\\,\\lVert\\tilde{\\boldsymbol{\\mu}}_t – \\boldsymbol{\\mu}_\\theta\\rVert^2.<\/annotation><\/semantics><\/math><\/div>\n\n\n\n

    So the entire per-step loss reduces to an MSE between the model’s predicted mean and the true posterior mean<\/strong>. The strategic question is then: how should we parameterise the network’s output to make this MSE easy to learn?<\/em><\/p>\n\n\n\n

    Using the reparameterisation x<\/mi>0<\/mn><\/msub>=<\/mo>(<\/mo>x<\/mi>t<\/mi><\/msub>\u2212<\/mo>1<\/mn>\u2212<\/mo>\u03b1<\/mi>\u02c9<\/mo><\/mover>t<\/mi><\/msub><\/mrow><\/msqrt>\u2009<\/mtext>\u03f5<\/mi>)<\/mo>\/<\/mi>\u03b1<\/mi>\u02c9<\/mo><\/mover>t<\/mi><\/msub><\/msqrt><\/mrow>\\mathbf{x}_0 = (\\mathbf{x}_t – \\sqrt{1-\\bar{\\alpha}_t}\\,\\boldsymbol{\\epsilon})\/\\sqrt{\\bar{\\alpha}_t}<\/annotation><\/semantics><\/math>, that is, expressing x<\/mi>0<\/mn><\/msub><\/mrow>\\mathbf{x}_0<\/annotation><\/semantics><\/math> in terms of the noise \u03f5<\/mi><\/mrow>\\boldsymbol{\\epsilon}<\/annotation><\/semantics><\/math> that was used to construct x<\/mi>t<\/mi><\/msub><\/mrow>\\mathbf{x}_t<\/annotation><\/semantics><\/math>, and substituting into \u03bc<\/mi>~<\/mo><\/mover>t<\/mi><\/msub><\/mrow>\\tilde{\\boldsymbol{\\mu}}_t<\/annotation><\/semantics><\/math>, mechanical algebra gives<\/p>\n\n\n\n
    \u03bc<\/mi>~<\/mo><\/mover>t<\/mi><\/msub>(<\/mo>x<\/mi>t<\/mi><\/msub>,<\/mo>\u03f5<\/mi>)<\/mo>\u2005\u200a<\/mtext>=<\/mo>\u2005\u200a<\/mtext>1<\/mn>\u03b1<\/mi>t<\/mi><\/msub><\/msqrt><\/mfrac>\u2009\u2063<\/mtext>(<\/mo>x<\/mi>t<\/mi><\/msub>\u2212<\/mo>\u03b2<\/mi>t<\/mi><\/msub>1<\/mn>\u2212<\/mo>\u03b1<\/mi>\u02c9<\/mo><\/mover>t<\/mi><\/msub><\/mrow><\/msqrt><\/mfrac>\u2009<\/mtext>\u03f5<\/mi>)<\/mo><\/mrow>.<\/mi><\/mrow>\\tilde{\\boldsymbol{\\mu}}_t(\\mathbf{x}_t,\\boldsymbol{\\epsilon}) \\;=\\; \\frac{1}{\\sqrt{\\alpha_t}}\\!\\left(\\mathbf{x}_t – \\frac{\\beta_t}{\\sqrt{1-\\bar{\\alpha}_t}}\\,\\boldsymbol{\\epsilon}\\right).<\/annotation><\/semantics><\/math><\/div>\n\n\n\n

    This suggests parameterising the model’s mean by the same<\/em> expression but with a learnt \u03f5<\/mi>\u03b8<\/mi><\/msub>(<\/mo>x<\/mi>t<\/mi><\/msub>,<\/mo>t<\/mi>)<\/mo><\/mrow>\\boldsymbol{\\epsilon}_\\theta(\\mathbf{x}_t,t)<\/annotation><\/semantics><\/math> in place of the true \u03f5<\/mi><\/mrow>\\boldsymbol{\\epsilon}<\/annotation><\/semantics><\/math>:<\/p>\n\n\n\n
    \u03bc<\/mi>\u03b8<\/mi><\/msub>(<\/mo>x<\/mi>t<\/mi><\/msub>,<\/mo>t<\/mi>)<\/mo>\u2005\u200a<\/mtext>=<\/mo>\u2005\u200a<\/mtext>1<\/mn>\u03b1<\/mi>t<\/mi><\/msub><\/msqrt><\/mfrac>\u2009\u2063<\/mtext>(<\/mo>x<\/mi>t<\/mi><\/msub>\u2212<\/mo>\u03b2<\/mi>t<\/mi><\/msub>1<\/mn>\u2212<\/mo>\u03b1<\/mi>\u02c9<\/mo><\/mover>t<\/mi><\/msub><\/mrow><\/msqrt><\/mfrac>\u2009<\/mtext>\u03f5<\/mi>\u03b8<\/mi><\/msub>(<\/mo>x<\/mi>t<\/mi><\/msub>,<\/mo>t<\/mi>)<\/mo>)<\/mo><\/mrow>.<\/mi><\/mrow>\\boldsymbol{\\mu}_\\theta(\\mathbf{x}_t,t) \\;=\\; \\frac{1}{\\sqrt{\\alpha_t}}\\!\\left(\\mathbf{x}_t – \\frac{\\beta_t}{\\sqrt{1-\\bar{\\alpha}_t}}\\,\\boldsymbol{\\epsilon}_\\theta(\\mathbf{x}_t,t)\\right).<\/annotation><\/semantics><\/math><\/div>\n\n\n\n

    Substituting \u03bc<\/mi>~<\/mo><\/mover>t<\/mi><\/msub><\/mrow>\\tilde{\\boldsymbol{\\mu}}_t<\/annotation><\/semantics><\/math> and \u03bc<\/mi>\u03b8<\/mi><\/msub><\/mrow>\\boldsymbol{\\mu}_\\theta<\/annotation><\/semantics><\/math> into the KL expression and simplifying gives, after cancellation,<\/p>\n\n\n\n
    L<\/mi>t<\/mi>\u2212<\/mo>1<\/mn><\/mrow><\/msub>\u2005\u200a<\/mtext>=<\/mo>\u2005\u200a<\/mtext>\u03b2<\/mi>t<\/mi>2<\/mn><\/msubsup>2<\/mn>\u2009<\/mtext>\u03c3<\/mi>t<\/mi>2<\/mn><\/msubsup>\u2009<\/mtext>\u03b1<\/mi>t<\/mi><\/msub>\u2009<\/mtext>(<\/mo>1<\/mn>\u2212<\/mo>\u03b1<\/mi>\u02c9<\/mo><\/mover>t<\/mi><\/msub>)<\/mo><\/mrow><\/mfrac>\u2009<\/mtext>E<\/mi>x<\/mi>0<\/mn><\/msub>,<\/mo>\u03f5<\/mi><\/mrow><\/msub>\u2009\u2063<\/mtext>[<\/mo>\u2225<\/mo>\u03f5<\/mi>\u2212<\/mo>\u03f5<\/mi>\u03b8<\/mi><\/msub>(<\/mo>x<\/mi>t<\/mi><\/msub>,<\/mo>t<\/mi>)<\/mo>\u2225<\/mo>2<\/mn><\/msup>]<\/mo><\/mrow>.<\/mi><\/mrow>L_{t-1} \\;=\\; \\frac{\\beta_t^2}{2\\,\\sigma_t^2\\,\\alpha_t\\,(1-\\bar{\\alpha}_t)}\\,\\mathbb{E}_{\\mathbf{x}_0,\\boldsymbol{\\epsilon}}\\!\\left[\\lVert\\boldsymbol{\\epsilon} – \\boldsymbol{\\epsilon}_\\theta(\\mathbf{x}_t,t)\\rVert^2\\right].<\/annotation><\/semantics><\/math><\/div>\n\n\n\n

    Each per-step VLB term has thus reduced to a weighted MSE<\/em> between the true noise (drawn when we constructed x<\/mi>t<\/mi><\/msub><\/mrow>\\mathbf{x}_t<\/annotation><\/semantics><\/math>) and the network’s predicted noise. This is, structurally, vanilla supervised regression: predict \u03f5<\/mi><\/mrow>\\boldsymbol{\\epsilon}<\/annotation><\/semantics><\/math> given x<\/mi>t<\/mi><\/msub><\/mrow>\\mathbf{x}_t<\/annotation><\/semantics><\/math>.<\/p>\n\n\n\n

    Ho et al.<\/a> then made the empirical observation that dropping the time-dependent weight<\/em> in front of this MSE produces strictly better samples than keeping the principled VLB weight. The resulting simple objective<\/strong> is the workhorse of modern diffusion:<\/p>\n\n\n\n

    L<\/mi>simple<\/mtext><\/msub>(<\/mo>\u03b8<\/mi>)<\/mo>\u2005\u200a<\/mtext>=<\/mo>\u2005\u200a<\/mtext>E<\/mi>t<\/mi>,<\/mo>x<\/mi>0<\/mn><\/msub>,<\/mo>\u03f5<\/mi><\/mrow><\/msub>[<\/mo>\u2009<\/mtext>\u2225<\/mo>\u03f5<\/mi>\u2212<\/mo>\u03f5<\/mi>\u03b8<\/mi><\/msub>(<\/mo>x<\/mi>t<\/mi><\/msub>,<\/mo>t<\/mi>)<\/mo>\u2225<\/mo>2<\/mn><\/msup>\u2009<\/mtext>]<\/mo>,<\/mo><\/mspace>x<\/mi>t<\/mi><\/msub>=<\/mo>\u03b1<\/mi>\u02c9<\/mo><\/mover>t<\/mi><\/msub><\/msqrt>\u2009<\/mtext>x<\/mi>0<\/mn><\/msub>+<\/mo>1<\/mn>\u2212<\/mo>\u03b1<\/mi>\u02c9<\/mo><\/mover>t<\/mi><\/msub><\/mrow><\/msqrt>\u2009<\/mtext>\u03f5<\/mi>.<\/mi><\/mrow>\\mathcal{L}_{\\text{simple}}(\\theta) \\;=\\; \\mathbb{E}_{t,\\mathbf{x}_0,\\boldsymbol{\\epsilon}}\\bigl[\\,\\lVert \\boldsymbol{\\epsilon} – \\boldsymbol{\\epsilon}_\\theta(\\mathbf{x}_t,t)\\rVert^2\\,\\bigr],\\qquad \\mathbf{x}_t = \\sqrt{\\bar{\\alpha}_t}\\,\\mathbf{x}_0 + \\sqrt{1-\\bar{\\alpha}_t}\\,\\boldsymbol{\\epsilon}.<\/annotation><\/semantics><\/math><\/div>\n\n\n\n

    The dropped weighting is largest for small t<\/mi><\/mrow>t<\/annotation><\/semantics><\/math> (mild noise) and effectively upweights<\/em> the harder high-noise levels, which empirically gives sharper samples (Ho et al., 2020<\/a>).<\/p>\n\n\n\n
    \n

    This objective is extraordinarily efficient and scalable<\/em>, and this is, in my view, the single most important reason diffusion models took off so quickly. Look at what training requires at each step: pick a data point x<\/mi>0<\/mn><\/msub><\/mrow>\\mathbf{x}_0<\/annotation><\/semantics><\/math> from the training set, sample a timestep t<\/mi><\/mrow>t<\/annotation><\/semantics><\/math> uniformly, sample a single Gaussian noise vector \u03f5<\/mi><\/mrow>\\boldsymbol{\\epsilon}<\/annotation><\/semantics><\/math>, form x<\/mi>t<\/mi><\/msub>=<\/mo>\u03b1<\/mi>\u02c9<\/mo><\/mover>t<\/mi><\/msub><\/msqrt>\u2009<\/mtext>x<\/mi>0<\/mn><\/msub>+<\/mo>1<\/mn>\u2212<\/mo>\u03b1<\/mi>\u02c9<\/mo><\/mover>t<\/mi><\/msub><\/mrow><\/msqrt>\u2009<\/mtext>\u03f5<\/mi><\/mrow>\\mathbf{x}_t = \\sqrt{\\bar{\\alpha}_t}\\,\\mathbf{x}_0 + \\sqrt{1-\\bar{\\alpha}_t}\\,\\boldsymbol{\\epsilon}<\/annotation><\/semantics><\/math> using precomputed<\/em> \u03b1<\/mi>\u02c9<\/mo><\/mover>t<\/mi><\/msub><\/mrow>\\bar{\\alpha}_t<\/annotation><\/semantics><\/math> tables, and regress with one network forward-backward pass. There is no need to simulate the trajectory.<\/em> There is no MCMC inner loop, no ODE solver in training, no need to even store any state between iterations. The training cost per step is identical to that of a vanilla supervised regressor. This is the property that made diffusion scale from CIFAR-10 to ImageNet to text-conditional generation to protein design without much architectural rethinking.<\/p>\n<\/div>\n\n\n\n

    Ancestral sampling<\/h3>\n\n\n\n

    At inference time we want new samples from the trained model, so we must run the Markov chain backward. Start from x<\/mi>T<\/mi><\/msub>\u223c<\/mo>N<\/mi>(<\/mo>0<\/mn>,<\/mo>I<\/mi>)<\/mo><\/mrow>\\mathbf{x}_T\\sim\\mathcal{N}(\\mathbf{0},\\mathbf{I})<\/annotation><\/semantics><\/math> and for t<\/mi>=<\/mo>T<\/mi>,<\/mo>T<\/mi>\u2212<\/mo>1<\/mn>,<\/mo>\u2026<\/mo>,<\/mo>1<\/mn><\/mrow>t = T, T-1, \\dots, 1<\/annotation><\/semantics><\/math> apply the ancestral sampler<\/strong>:<\/p>\n\n\n\n
    x<\/mi>t<\/mi>\u2212<\/mo>1<\/mn><\/mrow><\/msub>\u2005\u200a<\/mtext>=<\/mo>\u2005\u200a<\/mtext>1<\/mn>\u03b1<\/mi>t<\/mi><\/msub><\/msqrt><\/mfrac>\u2009\u2063<\/mtext>(<\/mo>x<\/mi>t<\/mi><\/msub>\u2212<\/mo>\u03b2<\/mi>t<\/mi><\/msub>1<\/mn>\u2212<\/mo>\u03b1<\/mi>\u02c9<\/mo><\/mover>t<\/mi><\/msub><\/mrow><\/msqrt><\/mfrac>\u2009<\/mtext>\u03f5<\/mi>\u03b8<\/mi><\/msub>(<\/mo>x<\/mi>t<\/mi><\/msub>,<\/mo>t<\/mi>)<\/mo>)<\/mo><\/mrow>\u2005\u200a<\/mtext>+<\/mo>\u2005\u200a<\/mtext>\u03c3<\/mi>t<\/mi><\/msub>\u2009<\/mtext>z<\/mi>,<\/mo><\/mspace>z<\/mi>\u223c<\/mo>N<\/mi>(<\/mo>0<\/mn>,<\/mo>I<\/mi>)<\/mo>,<\/mo>\u2005\u200a<\/mtext>\u2005\u200a<\/mtext>\u03c3<\/mi>t<\/mi>2<\/mn><\/msubsup>\u2208<\/mo>{<\/mo>\u03b2<\/mi>t<\/mi><\/msub>,<\/mo>\u03b2<\/mi>~<\/mo><\/mover>t<\/mi><\/msub>}<\/mo>.<\/mi><\/mrow>\\mathbf{x}_{t-1} \\;=\\; \\frac{1}{\\sqrt{\\alpha_t}}\\!\\left(\\mathbf{x}_t – \\frac{\\beta_t}{\\sqrt{1-\\bar{\\alpha}_t}}\\,\\boldsymbol{\\epsilon}_\\theta(\\mathbf{x}_t,t)\\right) \\;+\\; \\sigma_t\\,\\mathbf{z},\\qquad \\mathbf{z}\\sim\\mathcal{N}(\\mathbf{0},\\mathbf{I}),\\;\\;\\sigma_t^2\\in\\{\\beta_t,\\tilde{\\beta}_t\\}.<\/annotation><\/semantics><\/math><\/div>\n\n\n\n

    We can observe here as well how the new less noisy sample at time t<\/mi>\u2212<\/mo>1<\/mn><\/mrow>t-1<\/annotation><\/semantics><\/math> is indeed our sample at time t<\/mi><\/mrow>t<\/annotation><\/semantics><\/math> minus the noise that the networks recognizes in the sample. Heuristically, we are doing a step towards the denoised sample (away from the noise).<\/p>\n\n\n\n

    A later refinement worth flagging here: [Nichol & Dhariwal (2021)<\/a>](#ref-nichol2021)<\/strong> showed that a cosine schedule<\/strong>,<\/p>\n\n\n\n

    \u03b1<\/mi>\u02c9<\/mo><\/mover>t<\/mi><\/msub>\u2005\u200a<\/mtext>=<\/mo>\u2005\u200a<\/mtext>f<\/mi>(<\/mo>t<\/mi>)<\/mo><\/mrow>f<\/mi>(<\/mo>0<\/mn>)<\/mo><\/mrow><\/mfrac>,<\/mo><\/mspace>f<\/mi>(<\/mo>t<\/mi>)<\/mo>\u2005\u200a<\/mtext>=<\/mo>\u2005\u200a<\/mtext>cos<\/mi>\u2061<\/mo><\/mrow>2<\/mn><\/msup>\u2009\u2063<\/mtext>(<\/mo>t<\/mi>\/<\/mi>T<\/mi>+<\/mo>s<\/mi><\/mrow>1<\/mn>+<\/mo>s<\/mi><\/mrow><\/mfrac><\/mstyle>\u22c5<\/mo>\u03c0<\/mi>2<\/mn><\/mfrac><\/mstyle>)<\/mo>,<\/mo><\/mspace>s<\/mi>=<\/mo>0.008<\/mn>,<\/mo><\/mrow>\\bar{\\alpha}_t \\;=\\; \\frac{f(t)}{f(0)},\\qquad f(t) \\;=\\; \\cos^2\\!\\Bigl(\\tfrac{t\/T+s}{1+s}\\cdot\\tfrac{\\pi}{2}\\Bigr),\\qquad s = 0.008,<\/annotation><\/semantics><\/math><\/div>\n\n\n\n

    avoids the pathology of the linear schedule destroying low-resolution signal too quickly; \u03b1<\/mi>\u02c9<\/mo><\/mover>t<\/mi><\/msub><\/mrow>\\bar{\\alpha}_t<\/annotation><\/semantics><\/math> stays near 1 for longer in the early steps and then collapses smoothly. They also showed that learning<\/em> a log-interpolation between \u03b2<\/mi>t<\/mi><\/msub><\/mrow>\\beta_t<\/annotation><\/semantics><\/math> and \u03b2<\/mi>~<\/mo><\/mover>t<\/mi><\/msub><\/mrow>\\tilde{\\beta}_t<\/annotation><\/semantics><\/math> for the reverse covariance improves log-likelihood. Both choices are now standard.<\/p>\n\n\n\n

    DDPM in the wild: the original RFDiffusion sampler<\/h3>\n\n\n\n

    The original RFDiffusion (Watson et al., 2023<\/a>), one of the model that launched de-novo protein design as we know it today, is in this DDPM lineage. Its inner sampling loop in rfdiffusion\/inference\/utils.py<\/code><\/a> implements exactly<\/em> the ancestral update we have just derived. The key helper, get_mu_xt_x0<\/code>, computes the posterior mean \u03bc<\/mi>~<\/mo><\/mover>t<\/mi><\/msub>(<\/mo>x<\/mi>t<\/mi><\/msub>,<\/mo>x<\/mi>0<\/mn><\/msub>)<\/mo><\/mrow>\\tilde{\\boldsymbol{\\mu}}_t(\\mathbf{x}_t, \\mathbf{x}_0)<\/annotation><\/semantics><\/math> and posterior variance \u03b2<\/mi>~<\/mo><\/mover>t<\/mi><\/msub><\/mrow>\\tilde{\\beta}_t<\/annotation><\/semantics><\/math> from cached \u03b2<\/mi><\/mrow>\\beta<\/annotation><\/semantics><\/math> and \u03b1<\/mi>\u02c9<\/mo><\/mover><\/mrow>\\bar{\\alpha}<\/annotation><\/semantics><\/math> schedules:<\/p>\n\n\n\n
    def get_mu_xt_x0(xt, px0, t, beta_schedule, alphabar_schedule, eps=1e-6):\n    \"\"\"Given xt, predicted x0 and the timestep t, give mu of x(t-1).\"\"\"\n    t_idx = t - 1\n    # (1) Posterior variance: sigma = ((1 - alphabar_{t-1}) \/ (1 - alphabar_t)) * beta_t\n    #     This is exactly tilde{beta}_t.\n    sigma = (\n        (1 - alphabar_schedule[t_idx - 1]) \/ (1 - alphabar_schedule[t_idx])\n    ) * beta_schedule[t_idx]\n    xt_ca  = xt[:, 1, :]    # C-alpha coordinates at time t\n    px0_ca = px0[:, 1, :]   # network's prediction of clean C-alpha coordinates\n    # (2) First term of the posterior mean: coefficient on x_0.\n    a = (\n        (torch.sqrt(alphabar_schedule[t_idx - 1] + eps) * beta_schedule[t_idx])\n        \/ (1 - alphabar_schedule[t_idx])\n    ) * px0_ca\n    # (3) Second term of the posterior mean: coefficient on x_t,\n    #     with sqrt(1 - beta_t) = sqrt(alpha_t).\n    b = (\n        (torch.sqrt(1 - beta_schedule[t_idx] + eps) * (1 - alphabar_schedule[t_idx - 1]))\n        \/ (1 - alphabar_schedule[t_idx])\n    ) * xt_ca\n    mu = a + b\n    return mu, sigma\n<\/code><\/pre>\n\n\n\n
    And the helper that drives one ancestral step, get_next_ca<\/code>, samples x<\/mi>t<\/mi>\u2212<\/mo>1<\/mn><\/mrow><\/msub><\/mrow>\\mathbf{x}_{t-1}<\/annotation><\/semantics><\/math> from the Gaussian N<\/mi>(<\/mo>\u03bc<\/mi>~<\/mo><\/mover>t<\/mi><\/msub>,<\/mo>\u03b2<\/mi>~<\/mo><\/mover>t<\/mi><\/msub>I<\/mi>)<\/mo><\/mrow>\\mathcal{N}(\\tilde{\\boldsymbol{\\mu}}_t, \\tilde{\\beta}_t\\mathbf{I})<\/annotation><\/semantics><\/math> in one line:<\/p>\n\n\n\n
    def get_next_ca(xt, px0, t, ..., noise_scale=1.0):\n    # Compute posterior mean and variance from the cached schedules.\n    mu, sigma = get_mu_xt_x0(xt, px0, t, beta_schedule, alphabar_schedule)\n    # (4) Ancestral sampling: x_{t-1} ~ N(mu, sigma * I).\n    sampled_crds = torch.normal(mu, torch.sqrt(sigma * noise_scale))\n    delta = sampled_crds - xt[:, 1, :]\n    ...\n    out_crds = xt + delta[:, None, :]\n    return out_crds \/ crd_scale, delta \/ crd_scale\n<\/code><\/pre>\n\n\n\n
    Each numbered line is a line-for-line transcription<\/em> of an equation from Section 1.<\/p>\n\n\n\n