{"id":14543,"date":"2026-07-29T15:09:12","date_gmt":"2026-07-29T14:09:12","guid":{"rendered":"https:\/\/www.blopig.com\/blog\/?p=14543"},"modified":"2026-07-29T16:11:28","modified_gmt":"2026-07-29T15:11:28","slug":"diffusion-models-cant-give-you-a-likelihood-so-how-do-we-score-inverse-folded-sequences","status":"publish","type":"post","link":"https:\/\/www.blopig.com\/blog\/2026\/07\/diffusion-models-cant-give-you-a-likelihood-so-how-do-we-score-inverse-folded-sequences\/","title":{"rendered":"Diffusion Models Can’t Give You a Likelihood… So How Do We Score Inverse-Folded Sequences?"},"content":{"rendered":"\n

I’ve been working with inverse folding models for sequence design for a while now, and one question kept coming up. How would you score a sequence’s likelihood under these models? It turns out the answer depends heavily on whether the model is autoregressive or diffusion-based. <\/p>\n\n\n\n

Autoregressive models, like GPT-style language models, make this straightforward. You can calculate the probability of a sequence directly, by breaking it into one prediction per position and multiplying them together. Diffusion models define a probability distribution over outputs too, but getting the likelihood of a particular output means accounting for many unobserved intermediate states, and that turns out to be much harder. This is why diffusion models typically rely on the Evidence Lower Bound, or ELBO,<\/a> a computable estimate, rather than the exact likelihood itself.<\/p>\n\n\n\n

\"\"<\/a><\/figure>\n\n\n\n

The easy case: autoregressive models<\/strong><\/p>\n\n\n\n

An autoregressive model breaks the probability of a sequence into a product of conditional probabilities:<\/p>\n\n\n\n

p<\/mi>(<\/mo>x<\/mi>1<\/mn>:<\/mo>n<\/mi><\/mrow><\/msub>)<\/mo>=<\/mo>\u220f<\/mo>i<\/mi>=<\/mo>1<\/mn><\/mrow>n<\/mi><\/munderover><\/mrow>p<\/mi>(<\/mo>x<\/mi>i<\/mi><\/msub>|<\/mo>x<\/mi><<\/mo>i<\/mi><\/mrow><\/msub>)<\/mo><\/mrow>p(x_{1:n}) = \\prod_{i=1}^{n} p(x_i \\mid x_{<i})<\/annotation><\/semantics><\/math><\/div>\n\n\n\n\n\n\n\n

Each term asks: given everything that came before position i<\/em>, what probability does the model assign to the observed value xi<\/sub><\/em>?<\/p>\n\n\n\n

Taking logarithms turns the product into a sum:<\/p>\n\n\n\n

log<\/mi>\u2061<\/mo><\/mspace><\/mrow>p<\/mi>(<\/mo>x<\/mi>1<\/mn>:<\/mo>n<\/mi><\/mrow><\/msub>)<\/mo>=<\/mo>\u2211<\/mo>i<\/mi>=<\/mo>1<\/mn><\/mrow>n<\/mi><\/munderover><\/mrow>log<\/mi>\u2061<\/mo><\/mspace><\/mrow>p<\/mi>(<\/mo>x<\/mi>i<\/mi><\/msub>|<\/mo>x<\/mi><<\/mo>i<\/mi><\/mrow><\/msub>)<\/mo>.<\/mi><\/mrow>\\log p(x_{1:n})\n=\n\\sum_{i=1}^{n}\n\\log p(x_i \\mid x_{<i}).<\/annotation><\/semantics><\/math><\/div>\n\n\n\n

To score a sequence, we read off the probability assigned to the observed token at each position and add the log-probabilities. Once the decoding order is fixed, the likelihood is directly available.<\/p>\n\n\n\n

One caviat is Inverse folding models like ProteinMPNN could use different residue-decoding orders. A sequence score calculated using one order is exact for that particular order. Averaging over every possible decoding order would require an additional sum over those orders.<\/p>\n\n\n\n

The harder case: diffusion models<\/strong><\/p>\n\n\n\n

A standard diffusion model introduces a sequence of noisy states:<\/p>\n\n\n\n

x<\/mi>0<\/mn><\/msub>,<\/mo>x<\/mi>1<\/mn><\/msub>,<\/mo>\u2026<\/mo>,<\/mo>x<\/mi>T<\/mi><\/msub>.<\/mi><\/mrow>x_0,x_1,\\ldots,x_T.<\/annotation><\/semantics><\/math><\/div>\n\n\n\n

Here, x0<\/sub><\/em> is the clean data point, x1<\/sub><\/em> is a slightly noisy version of x0<\/sub><\/em>, xT<\/sub><\/em> is approximately pure noise, and T<\/em> is the total number of diffusion steps. <\/p>\n\n\n\n

Generation runs backwards, beginning with noise at xT and gradually producing cleaner states.<\/p>\n\n\n\n

At each step, <\/p>\n\n\n\n

x<\/mi>t<\/mi>\u2212<\/mo>1<\/mn><\/mrow><\/msub>\u223c<\/mo>p<\/mi>\u03b8<\/mi><\/msub>(<\/mo>x<\/mi>t<\/mi>\u2212<\/mo>1<\/mn><\/mrow><\/msub>|<\/mo>x<\/mi>t<\/mi><\/msub>)<\/mo>.<\/mi><\/mrow>x_{t-1}\n\\sim\np_\\theta(x_{t-1} \\mid x_t).<\/annotation><\/semantics><\/math><\/div>\n\n\n\n

The symbol ~ means “is sampled from”, while the vertical bar means “given”. Therefore, this expression means: Given the current noisy state xt<\/sub><\/em>, the model produces a probability distribution for the next, slightly cleaner state xt-1<\/sub><\/em>, and samples from it. <\/p>\n\n\n\n

The symbol \u03b8 represents the learned parameters of the neural network. <\/p>\n\n\n\n

A Three-step example <\/strong><\/p>\n\n\n\n

Suppose the model uses only three diffusion steps: <\/p>\n\n\n\n

x<\/mi>0<\/mn><\/msub>,<\/mo>x<\/mi>1<\/mn><\/msub>,<\/mo>x<\/mi>2<\/mn><\/msub>,<\/mo>x<\/mi>3<\/mn><\/msub>.<\/mi><\/mrow>x_0,x_1,x_2,x_3.<\/annotation><\/semantics><\/math><\/div>\n\n\n\n

Genration might follow the path <\/p>\n\n\n\n

1.2<\/mn>\u27f6<\/mo>0.8<\/mn>\u27f6<\/mo>0.4<\/mn>\u27f6<\/mo>0.1.<\/mn><\/mrow>1.2\n\\longrightarrow\n0.8\n\\longrightarrow\n0.4\n\\longrightarrow\n0.1.<\/annotation><\/semantics><\/math><\/div>\n\n\n\n

In this example, <\/p>\n\n\n\n

x<\/mi>3<\/mn><\/msub>=<\/mo>1.2<\/mn><\/mrow>x_3=1.2\n<\/annotation><\/semantics><\/math><\/div>\n\n\n\n

is the starting noise, while<\/p>\n\n\n\n

x<\/mi>0<\/mn><\/msub>=<\/mo>0.1<\/mn><\/mrow>x_0=0.1\n<\/annotation><\/semantics><\/math><\/div>\n\n\n\n

is the final clean output. <\/p>\n\n\n\n

The model generates the path using<\/p>\n\n\n\n

x<\/mi>2<\/mn><\/msub>\u223c<\/mo>p<\/mi>\u03b8<\/mi><\/msub>(<\/mo>x<\/mi>2<\/mn><\/msub>|<\/mo>x<\/mi>3<\/mn><\/msub>)<\/mo>,<\/mo><\/mrow>x_2\\sim p_\\theta(x_2\\mid x_3),\n<\/annotation><\/semantics><\/math><\/div>\n\n\n\n

followed by <\/p>\n\n\n\n

x<\/mi>1<\/mn><\/msub>\u223c<\/mo>p<\/mi>\u03b8<\/mi><\/msub>(<\/mo>x<\/mi>1<\/mn><\/msub>|<\/mo>x<\/mi>2<\/mn><\/msub>)<\/mo>,<\/mo><\/mrow>x_1\\sim p_\\theta(x_1\\mid x_2),\n<\/annotation><\/semantics><\/math><\/div>\n\n\n\n

and finally<\/p>\n\n\n\n

x<\/mi>0<\/mn><\/msub>\u223c<\/mo>p<\/mi>\u03b8<\/mi><\/msub>(<\/mo>x<\/mi>0<\/mn><\/msub>|<\/mo>x<\/mi>1<\/mn><\/msub>)<\/mo>.<\/mi><\/mrow>x_0\\sim p_\\theta(x_0\\mid x_1).\n<\/annotation><\/semantics><\/math><\/div>\n\n\n\n

The notation <\/p>\n\n\n\n

x<\/mi>0<\/mn>:<\/mo>T<\/mi><\/mrow><\/msub>x_{0:T}\n<\/annotation><\/semantics><\/math><\/div>\n\n\n\n

is shorthand for the complete collection of states<\/p>\n\n\n\n

x<\/mi>0<\/mn><\/msub>,<\/mo>x<\/mi>1<\/mn><\/msub>,<\/mo>\u2026<\/mo>,<\/mo>x<\/mi>T<\/mi><\/msub>.<\/mi><\/mrow>x_0,x_1,\\ldots,x_T.\n<\/annotation><\/semantics><\/math><\/div>\n\n\n\n

For T<\/em> = 3, <\/p>\n\n\n\n

x<\/mi>0<\/mn>:<\/mo>3<\/mn><\/mrow><\/msub>=<\/mo>(<\/mo>x<\/mi>0<\/mn><\/msub>,<\/mo>x<\/mi>1<\/mn><\/msub>,<\/mo>x<\/mi>2<\/mn><\/msub>,<\/mo>x<\/mi>3<\/mn><\/msub>)<\/mo>.<\/mi><\/mrow>x_{0:3}\n=\n(x_0,x_1,x_2,x_3).<\/annotation><\/semantics><\/math><\/div>\n\n\n\n

The probability density of one comeplete path is <\/p>\n\n\n\n

p<\/mi>\u03b8<\/mi><\/msub>(<\/mo>x<\/mi>0<\/mn>:<\/mo>3<\/mn><\/mrow><\/msub>)<\/mo>=<\/mo>p<\/mi>(<\/mo>x<\/mi>3<\/mn><\/msub>)<\/mo><\/mspace>p<\/mi>\u03b8<\/mi><\/msub>(<\/mo>x<\/mi>2<\/mn><\/msub>|<\/mo>x<\/mi>3<\/mn><\/msub>)<\/mo><\/mspace>p<\/mi>\u03b8<\/mi><\/msub>(<\/mo>x<\/mi>1<\/mn><\/msub>|<\/mo>x<\/mi>2<\/mn><\/msub>)<\/mo><\/mspace>p<\/mi>\u03b8<\/mi><\/msub>(<\/mo>x<\/mi>0<\/mn><\/msub>|<\/mo>x<\/mi>1<\/mn><\/msub>)<\/mo>.<\/mi><\/mrow>p_\\theta(x_{0:3})\n=\np(x_3)\\,\np_\\theta(x_2\\mid x_3)\\,\np_\\theta(x_1\\mid x_2)\\,\np_\\theta(x_0\\mid x_1).<\/annotation><\/semantics><\/math><\/div>\n\n\n\n

Suppose the density values for this path are <\/p>\n\n\n\n

p<\/mi>(<\/mo>x<\/mi>3<\/mn><\/msub>)<\/mo>=<\/mo>0.2<\/mn>,<\/mo><\/mrow>p(x_3)=0.2,<\/annotation><\/semantics><\/math><\/div>\n\n\n\n
p<\/mi>\u03b8<\/mi><\/msub>(<\/mo>x<\/mi>2<\/mn><\/msub>|<\/mo>x<\/mi>3<\/mn><\/msub>)<\/mo>=<\/mo>0.5<\/mn>,<\/mo><\/mrow>p_\\theta(x_2\\mid x_3)=0.5,<\/annotation><\/semantics><\/math><\/div>\n\n\n\n
p<\/mi>\u03b8<\/mi><\/msub>(<\/mo>x<\/mi>1<\/mn><\/msub>|<\/mo>x<\/mi>2<\/mn><\/msub>)<\/mo>=<\/mo>0.4<\/mn>,<\/mo><\/mrow>p_\\theta(x_1\\mid x_2)=0.4,<\/annotation><\/semantics><\/math><\/div>\n\n\n\n

and <\/p>\n\n\n\n

p<\/mi>\u03b8<\/mi><\/msub>(<\/mo>x<\/mi>0<\/mn><\/msub>|<\/mo>x<\/mi>1<\/mn><\/msub>)<\/mo>=<\/mo>0.6.<\/mn><\/mrow>p_\\theta(x_0\\mid x_1)=0.6.\n<\/annotation><\/semantics><\/math><\/div>\n\n\n\n

The density of this particular path is then<\/p>\n\n\n\n

0.2<\/mn>\u00d7<\/mo>0.5<\/mn>\u00d7<\/mo>0.4<\/mn>\u00d7<\/mo>0.6<\/mn>=<\/mo>0.024.<\/mn><\/mrow>0.2\\times 0.5\\times 0.4\\times 0.6\n=\n0.024.<\/annotation><\/semantics><\/math><\/div>\n\n\n\n

For T<\/em> steps, this is written as <\/p>\n\n\n\n

p<\/mi>\u03b8<\/mi><\/msub>(<\/mo>x<\/mi>0<\/mn>:<\/mo>T<\/mi><\/mrow><\/msub>)<\/mo>=<\/mo>p<\/mi>(<\/mo>x<\/mi>T<\/mi><\/msub>)<\/mo>\u220f<\/mo>t<\/mi>=<\/mo>1<\/mn><\/mrow>T<\/mi><\/munderover><\/mrow>p<\/mi>\u03b8<\/mi><\/msub>(<\/mo>x<\/mi>t<\/mi>\u2212<\/mo>1<\/mn><\/mrow><\/msub>|<\/mo>x<\/mi>t<\/mi><\/msub>)<\/mo>.<\/mi><\/mrow>p_\\theta(x_{0:T})\n=\np(x_T)\n\\prod_{t=1}^{T}\np_\\theta(x_{t-1}\\mid x_t).<\/annotation><\/semantics><\/math><\/div>\n\n\n\n

The product symbol means that we multiply together the density of every denoising transition.<\/p>\n\n\n\n

Why the Likelihood Is Difficult<\/strong><\/p>\n\n\n\n

The calculation above gives the density of one particular path.<\/p>\n\n\n\n

However, the same final output x0<\/sub><\/em> could be reached through many different intermediate states. For example, several paths might end at x0<\/sub><\/em> = 0.1: <\/p>\n\n\n\n

1.2<\/mn>\u27f6<\/mo>0.8<\/mn>\u27f6<\/mo>0.4<\/mn>\u27f6<\/mo>0.1<\/mn>,<\/mo><\/mrow>1.2\\longrightarrow0.8\\longrightarrow0.4\\longrightarrow0.1,<\/annotation><\/semantics><\/math><\/div>\n\n\n\n
\u2212<\/mo>0.5<\/mn>\u27f6<\/mo>0.6<\/mn>\u27f6<\/mo>0.3<\/mn>\u27f6<\/mo>0.1<\/mn>,<\/mo><\/mrow>-0.5\\longrightarrow0.6\\longrightarrow0.3\\longrightarrow0.1,<\/annotation><\/semantics><\/math><\/div>\n\n\n\n

or <\/p>\n\n\n\n

0.9<\/mn>\u27f6<\/mo>0.2<\/mn>\u27f6<\/mo>0.15<\/mn>\u27f6<\/mo>0.1.<\/mn><\/mrow>0.9\\longrightarrow0.2\\longrightarrow0.15\\longrightarrow0.1.\n<\/annotation><\/semantics><\/math><\/div>\n\n\n\n

When we are given only x<\/em>0<\/sub>, we do not know which intermediate path was taken. <\/p>\n\n\n\n

The states<\/p>\n\n\n\n

x<\/mi>1<\/mn><\/msub>,<\/mo>\u2026<\/mo>,<\/mo>x<\/mi>T<\/mi><\/msub><\/mrow>x_1,\\ldots,x_T\n<\/annotation><\/semantics><\/math><\/div>\n\n\n\n

are called latent variables. They are part of the model, but they are not observed. <\/p>\n\n\n\n

To calculate the density of x0<\/sub><\/em>, we must combine the contributions of all possible latent paths: <\/p>\n\n\n\n

p<\/mi>\u03b8<\/mi><\/msub>(<\/mo>x<\/mi>0<\/mn><\/msub>)<\/mo>=<\/mo>\u222b<\/mo>p<\/mi>\u03b8<\/mi><\/msub>(<\/mo>x<\/mi>0<\/mn>:<\/mo>T<\/mi><\/mrow><\/msub>)<\/mo><\/mspace>d<\/mi>x<\/mi>1<\/mn><\/msub>\u22ef<\/mo>d<\/mi>x<\/mi>T<\/mi><\/msub>.<\/mi><\/mrow>p_\\theta(x_0)\n=\n\\int\np_\\theta(x_{0:T})\n\\,dx_1\\cdots dx_T.<\/annotation><\/semantics><\/math><\/div>\n\n\n\n

For a discrete model, this would be a sum rather than an integral. <\/p>\n\n\n\n

The difficulty is not evaluating one denoising step such as <\/p>\n\n\n\n

p<\/mi>\u03b8<\/mi><\/msub>(<\/mo>x<\/mi>t<\/mi>\u2212<\/mo>1<\/mn><\/mrow><\/msub>|<\/mo>x<\/mi>t<\/mi><\/msub>)<\/mo>.<\/mi><\/mrow>p_\\theta(x_{t-1}\\mid x_t).<\/annotation><\/semantics><\/math><\/div>\n\n\n\n

The model can evaluate that quantity when xt<\/sub><\/em> and x<\/em>t-1<\/em> <\/sub>are known. <\/p>\n\n\n\n

The difficulty is combining the contributions of every possible value of <\/p>\n\n\n\n

x<\/mi>1<\/mn><\/msub>,<\/mo>\u2026<\/mo>,<\/mo>x<\/mi>T<\/mi><\/msub>.<\/mi><\/mrow>x_1,\\ldots,x_T.\n<\/annotation><\/semantics><\/math><\/div>\n\n\n\n

For a standard diffusion model, this marginalisation is generally intractable.<\/p>\n\n\n\n

The Forward Noising Process <\/strong><\/p>\n\n\n\n

To make the problem manageable, diffusion models define a fixed forward process. <\/p>\n\n\n\n

The forward process begins with a knonw clean data point and gradually adds noise: <\/p>\n\n\n\n

x<\/mi>0<\/mn><\/msub>\u27f6<\/mo>x<\/mi>1<\/mn><\/msub>\u27f6<\/mo>\u22ef<\/mo>\u27f6<\/mo>x<\/mi>T<\/mi><\/msub>.<\/mi><\/mrow>x_0\n\\longrightarrow\nx_1\n\\longrightarrow\n\\cdots\n\\longrightarrow\nx_T.<\/annotation><\/semantics><\/math><\/div>\n\n\n\n

Its distribution is written as<\/p>\n\n\n\n

q<\/mi>(<\/mo>x<\/mi>1<\/mn>:<\/mo>T<\/mi><\/mrow><\/msub>|<\/mo>x<\/mi>0<\/mn><\/msub>)<\/mo>=<\/mo>\u220f<\/mo>t<\/mi>=<\/mo>1<\/mn><\/mrow>T<\/mi><\/munderover><\/mrow>q<\/mi>(<\/mo>x<\/mi>t<\/mi><\/msub>|<\/mo>x<\/mi>t<\/mi>\u2212<\/mo>1<\/mn><\/mrow><\/msub>)<\/mo>.<\/mi><\/mrow>q(x_{1:T}\\mid x_0)\n=\n\\prod_{t=1}^{T}\nq(x_t\\mid x_{t-1}).<\/annotation><\/semantics><\/math><\/div>\n\n\n\n

The two distributions have different roles: q<\/em> is the fixed forward process that adds noise; p\u03b8<\/sub><\/em> is the learned reverse process that removes noise. <\/p>\n\n\n\n

In a standard Gaussian diffusion model, a noisy state can be sampled directly from x0<\/sub><\/em>: <\/p>\n\n\n\n

x<\/mi>t<\/mi><\/msub>=<\/mo>\u03b1<\/mi>\u203e<\/mo><\/mover>t<\/mi><\/msub><\/msqrt><\/mspace>x<\/mi>0<\/mn><\/msub>+<\/mo>1<\/mn>\u2212<\/mo>\u03b1<\/mi>\u203e<\/mo><\/mover>t<\/mi><\/msub><\/mrow><\/msqrt><\/mspace>\u03f5<\/mi>,<\/mo><\/mspace>\u03f5<\/mi>\u223c<\/mo>\ud835\udca9<\/mi>(<\/mo>0<\/mn>,<\/mo>I<\/mi>)<\/mo>.<\/mi><\/mrow>x_t\n=\n\\sqrt{\\bar{\\alpha}_t}\\,x_0\n+\n\\sqrt{1-\\bar{\\alpha}_t}\\,\\epsilon,\n\\qquad\n\\epsilon\\sim\\mathcal{N}(0,I).<\/annotation><\/semantics><\/math><\/div>\n\n\n\n

Here, x0<\/sub><\/em> is the clean data point; \u03f5 is random Gaussian noise; \u03b1t <\/sub>control how much of the original data remains; and 1- \u03b1t<\/sub> controls how much noise is added. So the equation says take some part of the clean data, add some amount of random Gaussian noise, and you get the noisy sample at timestep t<\/em>. <\/p>\n\n\n\n

For a simple example, suppose <\/p>\n\n\n\n

x<\/mi>0<\/mn><\/msub>=<\/mo>10<\/mn>,<\/mo><\/mspace>\u03b1<\/mi>\u203e<\/mo><\/mover>t<\/mi><\/msub>=<\/mo>0.64<\/mn>,<\/mo><\/mspace>\u03f5<\/mi>=<\/mo>\u2212<\/mo>0.5.<\/mn><\/mrow>x_0=10,\n\\qquad\n\\bar{\\alpha}_t=0.64,\n\\qquad\n\\epsilon=-0.5.<\/annotation><\/semantics><\/math><\/div>\n\n\n\n

Then <\/p>\n\n\n\n

x<\/mi>t<\/mi><\/msub>=<\/mo>0.64<\/mn><\/msqrt>(<\/mo>10<\/mn>)<\/mo>+<\/mo>1<\/mn>\u2212<\/mo>0.64<\/mn><\/mrow><\/msqrt>(<\/mo>\u2212<\/mo>0.5<\/mn>)<\/mo>.<\/mi><\/mrow>x_t\n=\n\\sqrt{0.64}(10)\n+\n\\sqrt{1-0.64}(-0.5).<\/annotation><\/semantics><\/math><\/div>\n\n\n\n

Since <\/p>\n\n\n\n

0.64<\/mn><\/msqrt>=<\/mo>0.8<\/mn><\/mrow>\\sqrt{0.64}=0.8\n\n<\/annotation><\/semantics><\/math><\/div>\n\n\n\n

and <\/p>\n\n\n\n

0.36<\/mn><\/msqrt>=<\/mo>0.6<\/mn>,<\/mo><\/mrow>\\sqrt{0.36}=0.6,<\/annotation><\/semantics><\/math><\/div>\n\n\n\n

we obtain <\/p>\n\n\n\n

x<\/mi>t<\/mi><\/msub>=<\/mo>0.8<\/mn>(<\/mo>10<\/mn>)<\/mo>+<\/mo>0.6<\/mn>(<\/mo>\u2212<\/mo>0.5<\/mn>)<\/mo>=<\/mo>7.7.<\/mn><\/mrow>x_t\n=\n0.8(10)+0.6(-0.5)\n=\n7.7.<\/annotation><\/semantics><\/math><\/div>\n\n\n\n

The forward process has therefore turned the clean values 10 into a noisy values 7.7 <\/p>\n\n\n\n

This process does not make the exact likelihood easy to calculate. Instead, it gives an easy way to sample plausible noisy paths starting from a known x0<\/sub><\/em>. <\/p>\n\n\n\n

This is what allows us to contruct the ELBO<\/a>. <\/p>\n\n\n\n

The ELBO<\/strong><\/p>\n\n\n\n

The ELBO (Evidence Lower Bound)<\/a>, provides a computable lower bound on the true log-likelihood. <\/p>\n\n\n\n

To understand it, I use a simple discrete example with only two possible hidden paths, path A and path B. <\/p>\n\n\n\n

Suppose the model assigns the joint probabilities: <\/p>\n\n\n\n

p<\/mi>\u03b8<\/mi><\/msub>(<\/mo>x<\/mi>0<\/mn><\/msub>,<\/mo>A<\/mtext>)<\/mo>=<\/mo>0.06<\/mn><\/mrow>p_\\theta(x_0,\\text{A})=0.06<\/annotation><\/semantics><\/math><\/div>\n\n\n\n

and <\/p>\n\n\n\n

p<\/mi>\u03b8<\/mi><\/msub>(<\/mo>x<\/mi>0<\/mn><\/msub>,<\/mo>B<\/mtext>)<\/mo>=<\/mo>0.04.<\/mn><\/mrow>p_\\theta(x_0,\\text{B})=0.04.<\/annotation><\/semantics><\/math><\/div>\n\n\n\n

The true probability of x0<\/sub><\/em> is obtained by adding the probability of every path: <\/p>\n\n\n\n

p<\/mi>\u03b8<\/mi><\/msub>(<\/mo>x<\/mi>0<\/mn><\/msub>)<\/mo>=<\/mo>0.06<\/mn>+<\/mo>0.04<\/mn>=<\/mo>0.10.<\/mn><\/mrow>p_\\theta(x_0)\n=\n0.06+0.04\n=\n0.10.<\/annotation><\/semantics><\/math><\/div>\n\n\n\n

The true log-likelihood is then<\/p>\n\n\n\n

log<\/mi>\u2061<\/mo><\/mspace><\/mrow>p<\/mi>\u03b8<\/mi><\/msub>(<\/mo>x<\/mi>0<\/mn><\/msub>)<\/mo>=<\/mo>log<\/mi>\u2061<\/mo><\/mrow>(<\/mo>0.10<\/mn>)<\/mo>\u2248<\/mo>\u2212<\/mo>2.303.<\/mn><\/mrow>\\log p_\\theta(x_0)\n=\n\\log(0.10)\n\\approx\n-2.303.<\/annotation><\/semantics><\/math><\/div>\n\n\n\n

In this example, adding both paths is easy. In a real diffusion model, there are too many possible trajectories to combine directly. <\/p>\n\n\n\n

Instead of considering every path, the ELBO uses the tractable forward distribution<\/p>\n\n\n\n

q<\/mi>(<\/mo>x<\/mi>1<\/mn>:<\/mo>T<\/mi><\/mrow><\/msub>|<\/mo>x<\/mi>0<\/mn><\/msub>)<\/mo>.<\/mi><\/mrow>q(x_{1:T}\\mid x_0).<\/annotation><\/semantics><\/math><\/div>\n\n\n\n

In the two-path example, suppose the forward process assigns equal probability to both paths: <\/p>\n\n\n\n

q<\/mi>(<\/mo>A<\/mtext>|<\/mo>x<\/mi>0<\/mn><\/msub>)<\/mo>=<\/mo>0.5<\/mn><\/mrow>q(\\text{A}\\mid x_0)=0.5<\/annotation><\/semantics><\/math><\/div>\n\n\n\n

and <\/p>\n\n\n\n

q<\/mi>(<\/mo>B<\/mtext>|<\/mo>x<\/mi>0<\/mn><\/msub>)<\/mo>=<\/mo>0.5.<\/mn><\/mrow>q(\\text{B}\\mid x_0)=0.5.<\/annotation><\/semantics><\/math><\/div>\n\n\n\n

the ELBO is <\/p>\n\n\n\n

ELBO<\/mi>\u2061<\/mo>(<\/mo>x<\/mi>0<\/mn><\/msub>)<\/mo>=<\/mo>\u2211<\/mo>paths<\/mtext><\/munder><\/mrow>q<\/mi>(<\/mo>path<\/mtext>|<\/mo>x<\/mi>0<\/mn><\/msub>)<\/mo><\/mspace>log<\/mi>\u2061<\/mo><\/mspace><\/mrow>p<\/mi>\u03b8<\/mi><\/msub>(<\/mo>x<\/mi>0<\/mn><\/msub>,<\/mo>path<\/mtext>)<\/mo><\/mrow>q<\/mi>(<\/mo>path<\/mtext>|<\/mo>x<\/mi>0<\/mn><\/msub>)<\/mo><\/mrow><\/mfrac>.<\/mi><\/mrow>\\operatorname{ELBO}(x_0)\n=\n\\sum_{\\text{paths}}\nq(\\text{path}\\mid x_0)\n\\log\n\\frac{\np_\\theta(x_0,\\text{path})\n}{\nq(\\text{path}\\mid x_0)\n}.<\/annotation><\/semantics><\/math><\/div>\n\n\n\n

Here, path means the whole hidden noisy trajectory x1 ,…, xT<\/sub><\/em>. Then q (path <\/em>| x0<\/sub>)<\/em> is the forward noising process, given the clean data x0<\/sub><\/em>. p\u03b8<\/sub> (x0<\/sub><\/em>, path) is the models joint probability of the clean output and that path, and the whole thing is an avegare over paths of that log ratio. <\/p>\n\n\n\n

Essentially, ELBO is how good the model is on average over noisy paths that start from x0<\/sub><\/em>. <\/p>\n\n\n\n

For Path A, <\/p>\n\n\n\n

p<\/mi>\u03b8<\/mi><\/msub>(<\/mo>x<\/mi>0<\/mn><\/msub>,<\/mo>A<\/mtext>)<\/mo><\/mrow>q<\/mi>(<\/mo>A<\/mtext>|<\/mo>x<\/mi>0<\/mn><\/msub>)<\/mo><\/mrow><\/mfrac>=<\/mo>0.06<\/mn>0.5<\/mn><\/mfrac>=<\/mo>0.12.<\/mn><\/mrow>\\frac{\np_\\theta(x_0,\\text{A})\n}{\nq(\\text{A}\\mid x_0)\n}\n=\n\\frac{0.06}{0.5}\n=\n0.12.<\/annotation><\/semantics><\/math><\/div>\n\n\n\n

For path B, <\/p>\n\n\n\n

p<\/mi>\u03b8<\/mi><\/msub>(<\/mo>x<\/mi>0<\/mn><\/msub>,<\/mo>B<\/mtext>)<\/mo><\/mrow>q<\/mi>(<\/mo>B<\/mtext>|<\/mo>x<\/mi>0<\/mn><\/msub>)<\/mo><\/mrow><\/mfrac>=<\/mo>0.04<\/mn>0.5<\/mn><\/mfrac>=<\/mo>0.08.<\/mn><\/mrow>\\frac{\np_\\theta(x_0,\\text{B})\n}{\nq(\\text{B}\\mid x_0)\n}\n=\n\\frac{0.04}{0.5}\n=\n0.08.<\/annotation><\/semantics><\/math><\/div>\n\n\n\n

The ELBO is therefore <\/p>\n\n\n\n

ELBO<\/mi>\u2061<\/mo>(<\/mo>x<\/mi>0<\/mn><\/msub>)<\/mo>=<\/mo>0.5<\/mn><\/mspace>log<\/mi>\u2061<\/mo><\/mrow>(<\/mo>0.12<\/mn>)<\/mo>+<\/mo>0.5<\/mn><\/mspace>log<\/mi>\u2061<\/mo><\/mrow>(<\/mo>0.08<\/mn>)<\/mo>.<\/mi><\/mrow>\\operatorname{ELBO}(x_0)\n=\n0.5\\log(0.12)\n+\n0.5\\log(0.08).<\/annotation><\/semantics><\/math><\/div>\n\n\n\n

Numerically, <\/p>\n\n\n\n

ELBO<\/mi>\u2061<\/mo>(<\/mo>x<\/mi>0<\/mn><\/msub>)<\/mo>\u2248<\/mo>\u2212<\/mo>2.323.<\/mn><\/mrow>\\operatorname{ELBO}(x_0)\n\\approx\n-2.323.<\/annotation><\/semantics><\/math><\/div>\n\n\n\n

The true log-likelihood was <\/p>\n\n\n\n

log<\/mi>\u2061<\/mo><\/mspace><\/mrow>p<\/mi>\u03b8<\/mi><\/msub>(<\/mo>x<\/mi>0<\/mn><\/msub>)<\/mo>\u2248<\/mo>\u2212<\/mo>2.303.<\/mn><\/mrow>\\log p_\\theta(x_0)\n\\approx\n-2.303.<\/annotation><\/semantics><\/math><\/div>\n\n\n\n

Therfore, <\/p>\n\n\n\n

\u2212<\/mo>2.323<\/mn>\u2264<\/mo>\u2212<\/mo>2.303.<\/mn><\/mrow>-2.323\n\\leq\n-2.303.<\/annotation><\/semantics><\/math><\/div>\n\n\n\n

The ELBO is slightly smaller than the true log-likelihood, which is why it is called a a lower bound. <\/strong><\/p>\n\n\n\n

Why divide by q<\/em>?<\/strong><\/p>\n\n\n\n

The forward distribuition determines how often each path is sampled. <\/p>\n\n\n\n

Dividing by <\/p>\n\n\n\n

q<\/mi>(<\/mo>path<\/mtext>|<\/mo>x<\/mi>0<\/mn><\/msub>)<\/mo><\/mrow>q(\\text{path}\\mid x_0)<\/annotation><\/semantics><\/math><\/div>\n\n\n\n

corrects for that sampling probability. <\/p>\n\n\n\n

The ratio <\/p>\n\n\n\n

p<\/mi>\u03b8<\/mi><\/msub>(<\/mo>x<\/mi>0<\/mn><\/msub>,<\/mo>path<\/mtext>)<\/mo><\/mrow>q<\/mi>(<\/mo>path<\/mtext>|<\/mo>x<\/mi>0<\/mn><\/msub>)<\/mo><\/mrow><\/mfrac>\\frac{\np_\\theta(x_0,\\text{path})\n}{\nq(\\text{path}\\mid x_0)\n}<\/annotation><\/semantics><\/math><\/div>\n\n\n\n

compares how much probability the generative model gives the path with how often the forward process samples it. <\/p>\n\n\n\n

The connection to KL divergence<\/strong><\/p>\n\n\n\n

The KL divergence measures the difference between two full probability distributions over hideen paths.<\/p>\n\n\n\n

the exact relationship between the true log-likelihood and the ELBO is<\/p>\n\n\n\n

log<\/mi>\u2061<\/mo><\/mspace><\/mrow>p<\/mi>\u03b8<\/mi><\/msub>(<\/mo>x<\/mi>0<\/mn><\/msub>)<\/mo>=<\/mo>ELBO<\/mi>\u2061<\/mo>(<\/mo>x<\/mi>0<\/mn><\/msub>)<\/mo>+<\/mo>D<\/mi><\/mtext>KL<\/mi><\/mrow><\/msub>(<\/mo>q<\/mi>(<\/mo>x<\/mi>1<\/mn>:<\/mo>T<\/mi><\/mrow><\/msub>|<\/mo>x<\/mi>0<\/mn><\/msub>)<\/mo>\u2225<\/mo>p<\/mi>\u03b8<\/mi><\/msub>(<\/mo>x<\/mi>1<\/mn>:<\/mo>T<\/mi><\/mrow><\/msub>|<\/mo>x<\/mi>0<\/mn><\/msub>)<\/mo>)<\/mo><\/mrow>.<\/mi><\/mrow>\\log p_\\theta(x_0) = \\operatorname{ELBO}(x_0) + D_{\\mathrm{KL}} \\left( q(x_{1:T}\\mid x_0) \\parallel p_\\theta(x_{1:T}\\mid x_0) \\right).<\/annotation><\/semantics><\/math> <\/p>\n\n\n\n

In the two-path example, the models true posterioir probability of path A is <\/p>\n\n\n\n

p<\/mi>\u03b8<\/mi><\/msub>(<\/mo>A<\/mtext>|<\/mo>x<\/mi>0<\/mn><\/msub>)<\/mo>=<\/mo>0.06<\/mn>0.10<\/mn><\/mfrac>=<\/mo>0.6.<\/mn><\/mrow>p_\\theta(\\text{A}\\mid x_0)\n=\n\\frac{0.06}{0.10}\n=\n0.6.<\/annotation><\/semantics><\/math><\/div>\n\n\n\n

Similarly, <\/p>\n\n\n\n

p<\/mi>\u03b8<\/mi><\/msub>(<\/mo>B<\/mtext>|<\/mo>x<\/mi>0<\/mn><\/msub>)<\/mo>=<\/mo>0.04<\/mn>0.10<\/mn><\/mfrac>=<\/mo>0.4.<\/mn><\/mrow>p_\\theta(\\text{B}\\mid x_0)\n=\n\\frac{0.04}{0.10}\n=\n0.4.<\/annotation><\/semantics><\/math><\/div>\n\n\n\n

The models true posterior over paths is therefore <\/p>\n\n\n\n

(<\/mo>0.6,0.4<\/mn>)<\/mo>.<\/mi><\/mrow>(0.6,0.4).\n<\/annotation><\/semantics><\/math><\/div>\n\n\n\n

The forward distribution was <\/p>\n\n\n\n

(<\/mo>0.5,0.5<\/mn>)<\/mo>.<\/mi><\/mrow>(0.5,0.5).\n<\/annotation><\/semantics><\/math><\/div>\n\n\n\n

Thes distributions are close, but not identical, so the mismatch prodcues a positive KL divergence, making the ELBO slightly smaller than the true log-likelihood. <\/p>\n\n\n\n

Because KL divergenece cannot be negative, <\/p>\n\n\n\n

D<\/mi><\/mtext>KL<\/mi><\/mrow><\/msub>(<\/mo>q<\/mi>\u2225<\/mo>p<\/mi>)<\/mo>\u2265<\/mo>0<\/mn>,<\/mo><\/mrow>D_{\\mathrm{KL}}(q\\parallel p)\\geq 0,<\/annotation><\/semantics><\/math><\/div>\n\n\n\n

we always have <\/p>\n\n\n\n

ELBO<\/mi>\u2061<\/mo>(<\/mo>x<\/mi>0<\/mn><\/msub>)<\/mo>\u2264<\/mo>log<\/mi>\u2061<\/mo><\/mspace><\/mrow>p<\/mi>\u03b8<\/mi><\/msub>(<\/mo>x<\/mi>0<\/mn><\/msub>)<\/mo>.<\/mi><\/mrow>\\operatorname{ELBO}(x_0)\n\\leq\n\\log p_\\theta(x_0).<\/annotation><\/semantics><\/math><\/div>\n\n\n\n

If the forward distributions matched the model’s true posterior exactly, then <\/p>\n\n\n\n

D<\/mi><\/mtext>KL<\/mi><\/mrow><\/msub>(<\/mo>q<\/mi>\u2225<\/mo>p<\/mi>)<\/mo>=<\/mo>0<\/mn><\/mrow>D_{\\mathrm{KL}}(q\\parallel p)=0<\/annotation><\/semantics><\/math><\/div>\n\n\n\n

Applications to Inverse Folding diffusion models<\/strong><\/p>\n\n\n\n

Inverse folding is the task of predicting a sequence that folds into a target structure. Rather than modelling an unconditional distribution over sequences, an inverse folding model conditions everything on fixed structure S<\/em>: <\/p>\n\n\n\n

p<\/mi>\u03b8<\/mi><\/msub>(<\/mo>x<\/mi>0<\/mn><\/msub>|<\/mo>S<\/mi>)<\/mo>.<\/mi><\/mrow>p_\\theta(x_0 \\mid S).<\/annotation><\/semantics><\/math><\/div>\n\n\n\n

The structure is held constant throughout, both the forward noising process and the reverse denoising process are conditioned on it. The forward noising process and the reverse denoising process are conditioned on it. The forward process becomes <\/p>\n\n\n\n

q<\/mi>(<\/mo>x<\/mi>1<\/mn>:<\/mo>T<\/mi><\/mrow><\/msub>|<\/mo>x<\/mi>0<\/mn><\/msub>,<\/mo>S<\/mi>)<\/mo>=<\/mo>\u220f<\/mo>t<\/mi>=<\/mo>1<\/mn><\/mrow>T<\/mi><\/munderover><\/mrow>q<\/mi>(<\/mo>x<\/mi>t<\/mi><\/msub>|<\/mo>x<\/mi>t<\/mi>\u2212<\/mo>1<\/mn><\/mrow><\/msub>,<\/mo>S<\/mi>)<\/mo>,<\/mo><\/mrow>q(x_{1:T}\\mid x_0, S)\n=\n\\prod_{t=1}^{T}\nq(x_t\\mid x_{t-1}, S),<\/annotation><\/semantics><\/math><\/div>\n\n\n\n

and the reverse process becomes <\/p>\n\n\n\n

x<\/mi>t<\/mi>\u2212<\/mo>1<\/mn><\/mrow><\/msub>\u223c<\/mo>p<\/mi>\u03b8<\/mi><\/msub>(<\/mo>x<\/mi>t<\/mi>\u2212<\/mo>1<\/mn><\/mrow><\/msub>|<\/mo>x<\/mi>t<\/mi><\/msub>,<\/mo>S<\/mi>)<\/mo>.<\/mi><\/mrow>x_{t-1}\n\\sim\np_\\theta(x_{t-1}\\mid x_t, S).<\/annotation><\/semantics><\/math><\/div>\n\n\n\n

Everything derived in the previous section is carried over unchnaged once S<\/em> is added as a conditioning variable throughout. In particular, the exact decomposition still holds. <\/p>\n\n\n\n

log<\/mi>\u2061<\/mo><\/mspace><\/mrow>p<\/mi>\u03b8<\/mi><\/msub>(<\/mo>x<\/mi>0<\/mn><\/msub>|<\/mo>S<\/mi>)<\/mo>=<\/mo>ELBO<\/mi>\u2061<\/mo>(<\/mo>x<\/mi>0<\/mn><\/msub>|<\/mo>S<\/mi>)<\/mo>+<\/mo>D<\/mi><\/mtext>KL<\/mi><\/mrow><\/msub>(<\/mo>q<\/mi>(<\/mo>x<\/mi>1<\/mn>:<\/mo>T<\/mi><\/mrow><\/msub>|<\/mo>x<\/mi>0<\/mn><\/msub>,<\/mo>S<\/mi>)<\/mo>\u2225<\/mo>p<\/mi>\u03b8<\/mi><\/msub>(<\/mo>x<\/mi>1<\/mn>:<\/mo>T<\/mi><\/mrow><\/msub>|<\/mo>x<\/mi>0<\/mn><\/msub>,<\/mo>S<\/mi>)<\/mo>)<\/mo><\/mrow>.<\/mi><\/mrow>\\log p_\\theta(x_0 \\mid S)\n=\n\\operatorname{ELBO}(x_0 \\mid S)\n+\nD_{\\mathrm{KL}}\n\\left(\nq(x_{1:T}\\mid x_0, S)\n\\parallel\np_\\theta(x_{1:T}\\mid x_0, S)\n\\right).<\/annotation><\/semantics><\/math><\/div>\n\n\n\n

Scoring\/Ranking<\/strong> a Candidate sequence<\/strong><\/p>\n\n\n\n

The main use of this scoring, rather than generating. Given a candidate sequence x0<\/sub><\/em> and a fixed strucure S<\/em>, the question is not what sequence should the model produce, but how plausible does the model consider this specific sequence, given this specific structure. <\/p>\n\n\n\n

Because the exact log-likelihood is intractible for the same reasons discussed above, the ELBO is used instead. In practice this is computed by taking the known sequence x0<\/sub><\/em>, corrupting it to a range of noise levels t<\/em>, and evaluating how well the model reconstructs the true sequence identity at each position, conditioned on S<\/em>. Summing these reconstruction terms across timesteps, following the same construction as before, prodcues ELBO (x<\/em>0<\/em> <\/sub>| S<\/em>). <\/p>\n\n\n\n

Because the ELBO is a lower bound rather the exact values, it is less informative than its use as a ranking signal. Two candidate sequences x<\/em>0<\/sub>(1)<\/sup> and x0<\/sub>(2)<\/sup> for the same structure S<\/em> can be comaped using <\/p>\n\n\n\n

ELBO<\/mi>\u2061<\/mo>(<\/mo>x<\/mi>0<\/mn>(<\/mo>1<\/mn>)<\/mo><\/mrow><\/msubsup>|<\/mo>S<\/mi>)<\/mo><\/mspace>versus<\/mtext><\/mspace>ELBO<\/mi>\u2061<\/mo>(<\/mo>x<\/mi>0<\/mn>(<\/mo>2<\/mn>)<\/mo><\/mrow><\/msubsup>|<\/mo>S<\/mi>)<\/mo>.<\/mi><\/mrow>\\operatorname{ELBO}(x_0^{(1)} \\mid S)\n\\quad\\text{versus}\\quad\n\\operatorname{ELBO}(x_0^{(2)} \\mid S).<\/annotation><\/semantics><\/math><\/div>\n\n\n\n

In practice this can be used to rank candidate sequences or mutations by their structural plausibility under a fixed backbone, since no tractable exact alternative exists.<\/p>\n\n\n\n

A note on discrete sequences<\/strong><\/p>\n\n\n\n

The Gaussian forward process intorduced earlier, with <\/p>\n\n\n\n

x<\/mi>t<\/mi><\/msub>=<\/mo>\u03b1<\/mi>\u203e<\/mo><\/mover>t<\/mi><\/msub><\/msqrt><\/mspace>x<\/mi>0<\/mn><\/msub>+<\/mo>1<\/mn>\u2212<\/mo>\u03b1<\/mi>\u203e<\/mo><\/mover>t<\/mi><\/msub><\/mrow><\/msqrt><\/mspace>\u03f5<\/mi>,<\/mo><\/mrow>x_t\n=\n\\sqrt{\\bar{\\alpha}_t}\\,x_0\n+\n\\sqrt{1-\\bar{\\alpha}_t}\\,\\epsilon,<\/annotation><\/semantics><\/math><\/div>\n\n\n\n

applied to continuous data, Amino acid sequences are categorical rather than continuous, so a discrete diffusion model instead defines a forward process that corrupts a categorical identity directly, for example by replacing a residue with a randomly drawn amino acid with some probability that increases with t<\/em>, rather than adding Gaussian noise to a continuous embedding . <\/p>\n\n\n\n

The overall logic of the ELBO is unaffected by this distinction. The reverse process remains intractable, and the forward process remains tractable to sample from directly, regardless of whether the underlying noise model is Gaussian or categorical. <\/p>\n\n\n\n

Conclusion<\/strong><\/p>\n\n\n\n

For an autoregressive model, the likelihood is directly calculated:<\/p>\n\n\n\n

p<\/mi>(<\/mo>x<\/mi>)<\/mo>=<\/mo>\u220f<\/mo>i<\/mi>=<\/mo>1<\/mn><\/mrow>n<\/mi><\/munderover><\/mrow>p<\/mi>(<\/mo>x<\/mi>i<\/mi><\/msub>|<\/mo>x<\/mi><<\/mo>i<\/mi><\/mrow><\/msub>)<\/mo>.<\/mi><\/mrow>p(x)\n=\n\\prod_{i=1}^{n}\np(x_i \\mid x_{<i}).<\/annotation><\/semantics><\/math><\/div>\n\n\n\n

For a standard diffusion model, the likelihood requires integrating over intermediate states:<\/p>\n\n\n\n

p<\/mi>\u03b8<\/mi><\/msub>(<\/mo>x<\/mi>0<\/mn><\/msub>)<\/mo>=<\/mo>\u222b<\/mo>p<\/mi>\u03b8<\/mi><\/msub>(<\/mo>x<\/mi>0<\/mn>:<\/mo>T<\/mi><\/mrow><\/msub>)<\/mo><\/mspace>d<\/mi>x<\/mi>1<\/mn><\/msub>\u22ef<\/mo>d<\/mi>x<\/mi>T<\/mi><\/msub>.<\/mi><\/mrow>p_\\theta(x_0)\n=\n\\int\np_\\theta(x_{0:T})\n\\,dx_1\\cdots dx_T.<\/annotation><\/semantics><\/math><\/div>\n\n\n\n

That integral is the central difficulty. The ELBO avoids calculating it directly. It uses the tractable forward noising process to produce a lower bound on the true log-likelihood.<\/p>\n\n\n\n

In a real diffusion model, the hidden variable is the complete noisy trajectory<\/p>\n\n\n\n

x<\/mi>1<\/mn><\/msub>,<\/mo>x<\/mi>2<\/mn><\/msub>,<\/mo>\u2026<\/mo>,<\/mo>x<\/mi>T<\/mi><\/msub>.<\/mi><\/mrow>x_1,x_2,\\ldots,x_T.<\/annotation><\/semantics><\/math><\/div>\n\n\n\n

The ELBO is written as<\/p>\n\n\n\n

ELBO<\/mi>\u2061<\/mo>(<\/mo>x<\/mi>0<\/mn><\/msub>)<\/mo>=<\/mo>\ud835\udd3c<\/mi>q<\/mi>(<\/mo>x<\/mi>1<\/mn>:<\/mo>T<\/mi><\/mrow><\/msub>|<\/mo>x<\/mi>0<\/mn><\/msub>)<\/mo><\/mrow><\/msub>[<\/mo>log<\/mi>\u2061<\/mo><\/mspace><\/mrow>p<\/mi>\u03b8<\/mi><\/msub>(<\/mo>x<\/mi>0<\/mn>:<\/mo>T<\/mi><\/mrow><\/msub>)<\/mo><\/mrow>q<\/mi>(<\/mo>x<\/mi>1<\/mn>:<\/mo>T<\/mi><\/mrow><\/msub>|<\/mo>x<\/mi>0<\/mn><\/msub>)<\/mo><\/mrow><\/mfrac>]<\/mo><\/mrow>.<\/mi><\/mrow>\\operatorname{ELBO}(x_0)\n=\n\\mathbb{E}_{q(x_{1:T}\\mid x_0)}\n\\left[\n\\log\n\\frac{\np_\\theta(x_{0:T})\n}{\nq(x_{1:T}\\mid x_0)\n}\n\\right].<\/annotation><\/semantics><\/math><\/div>\n\n\n\n

The main idea is that the exact likelihood requires combining every possible hidden trajectory. The ELBO instead uses trajectories from an easy forward process to calculate a guaranteed lower bound on the log-likelihood. This is why likelihood evaluation is immediate for autoregressive models, while diffusion models often require a variational bound or another specialised estimation method.<\/p>\n","protected":false},"excerpt":{"rendered":"

I’ve been working with inverse folding models for sequence design for a while now, and one question kept coming up. How would you score a sequence’s likelihood under these models? It turns out the answer depends heavily on whether the model is autoregressive or diffusion-based. Autoregressive models, like GPT-style language models, make this straightforward. You […]<\/p>\n","protected":false},"author":131,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"nf_dc_page":"","wikipediapreview_detectlinks":true,"_monsterinsights_skip_tracking":false,"ngg_post_thumbnail":0,"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_feature_clip_id":0,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_post_was_ever_published":false},"categories":[633,920,189],"tags":[],"ppma_author":[817],"class_list":["post-14543","post","type-post","status-publish","format-standard","hentry","category-ai","category-diffusion","category-machine-learning"],"jetpack_featured_media_url":"","jetpack_sharing_enabled":true,"authors":[{"term_id":817,"user_id":131,"is_guest":0,"slug":"king","display_name":"King Ifashe","avatar_url":"https:\/\/secure.gravatar.com\/avatar\/47fee3b332544ca46a19ab03b6c5b8c35e9a4dbf1247beb5f9b773ed20efcad7?s=96&d=mm&r=g","author_category":"","user_url":"","last_name":"Ifashe","first_name":"King","job_title":"","description":""}],"_links":{"self":[{"href":"https:\/\/www.blopig.com\/blog\/wp-json\/wp\/v2\/posts\/14543","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.blopig.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.blopig.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.blopig.com\/blog\/wp-json\/wp\/v2\/users\/131"}],"replies":[{"embeddable":true,"href":"https:\/\/www.blopig.com\/blog\/wp-json\/wp\/v2\/comments?post=14543"}],"version-history":[{"count":5,"href":"https:\/\/www.blopig.com\/blog\/wp-json\/wp\/v2\/posts\/14543\/revisions"}],"predecessor-version":[{"id":14582,"href":"https:\/\/www.blopig.com\/blog\/wp-json\/wp\/v2\/posts\/14543\/revisions\/14582"}],"wp:attachment":[{"href":"https:\/\/www.blopig.com\/blog\/wp-json\/wp\/v2\/media?parent=14543"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.blopig.com\/blog\/wp-json\/wp\/v2\/categories?post=14543"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.blopig.com\/blog\/wp-json\/wp\/v2\/tags?post=14543"},{"taxonomy":"author","embeddable":true,"href":"https:\/\/www.blopig.com\/blog\/wp-json\/wp\/v2\/ppma_author?post=14543"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}