Diffusion Models Can’t Give You a Likelihood… So How Do We Score Inverse-Folded Sequences?

I’ve been working with inverse folding models for sequence design for a while now, and one question kept coming up. How would you score a sequence’s likelihood under these models? It turns out the answer depends heavily on whether the model is autoregressive or diffusion-based.

Autoregressive models, like GPT-style language models, make this straightforward. You can calculate the probability of a sequence directly, by breaking it into one prediction per position and multiplying them together. Diffusion models define a probability distribution over outputs too, but getting the likelihood of a particular output means accounting for many unobserved intermediate states, and that turns out to be much harder. This is why diffusion models typically rely on the Evidence Lower Bound, or ELBO, a computable estimate, rather than the exact likelihood itself.

The easy case: autoregressive models

An autoregressive model breaks the probability of a sequence into a product of conditional probabilities:

p(x1:n)=∏i=1np(xi|x<i)p(x_{1:n}) = \prod_{i=1}^{n} p(x_i \mid x_{<i})

Each term asks: given everything that came before position i, what probability does the model assign to the observed value xi?

Taking logarithms turns the product into a sum:

log⁡p(x1:n)=∑i=1nlog⁡p(xi|x<i).\log p(x_{1:n}) = \sum_{i=1}^{n} \log p(x_i \mid x_{<i}).

To score a sequence, we read off the probability assigned to the observed token at each position and add the log-probabilities. Once the decoding order is fixed, the likelihood is directly available.

One caviat is Inverse folding models like ProteinMPNN could use different residue-decoding orders. A sequence score calculated using one order is exact for that particular order. Averaging over every possible decoding order would require an additional sum over those orders.

The harder case: diffusion models

A standard diffusion model introduces a sequence of noisy states:

x0,x1,…,xT.x_0,x_1,\ldots,x_T.

Here, x0 is the clean data point, x1 is a slightly noisy version of x0, xT is approximately pure noise, and T is the total number of diffusion steps.

Generation runs backwards, beginning with noise at xT and gradually producing cleaner states.

At each step,

xt−1∼pθ(xt−1|xt).x_{t-1} \sim p_\theta(x_{t-1} \mid x_t).

The symbol ~ means “is sampled from”, while the vertical bar means “given”. Therefore, this expression means: Given the current noisy state xt, the model produces a probability distribution for the next, slightly cleaner state xt-1, and samples from it.

The symbol θ represents the learned parameters of the neural network.

A Three-step example

Suppose the model uses only three diffusion steps:

x0,x1,x2,x3.x_0,x_1,x_2,x_3.

Genration might follow the path

1.2⟶0.8⟶0.4⟶0.1.1.2 \longrightarrow 0.8 \longrightarrow 0.4 \longrightarrow 0.1.

In this example,

x3=1.2x_3=1.2

is the starting noise, while

x0=0.1x_0=0.1

is the final clean output.

The model generates the path using

x2∼pθ(x2|x3),x_2\sim p_\theta(x_2\mid x_3),

followed by

x1∼pθ(x1|x2),x_1\sim p_\theta(x_1\mid x_2),

and finally

x0∼pθ(x0|x1).x_0\sim p_\theta(x_0\mid x_1).

The notation

x0:Tx_{0:T}

is shorthand for the complete collection of states

x0,x1,…,xT.x_0,x_1,\ldots,x_T.

For T = 3,

x0:3=(x0,x1,x2,x3).x_{0:3} = (x_0,x_1,x_2,x_3).

The probability density of one comeplete path is

pθ(x0:3)=p(x3)pθ(x2|x3)pθ(x1|x2)pθ(x0|x1).p_\theta(x_{0:3}) = p(x_3)\, p_\theta(x_2\mid x_3)\, p_\theta(x_1\mid x_2)\, p_\theta(x_0\mid x_1).

Suppose the density values for this path are

p(x3)=0.2,p(x_3)=0.2,
pθ(x2|x3)=0.5,p_\theta(x_2\mid x_3)=0.5,
pθ(x1|x2)=0.4,p_\theta(x_1\mid x_2)=0.4,

and

pθ(x0|x1)=0.6.p_\theta(x_0\mid x_1)=0.6.

The density of this particular path is then

0.2×0.5×0.4×0.6=0.024.0.2\times 0.5\times 0.4\times 0.6 = 0.024.

For T steps, this is written as

pθ(x0:T)=p(xT)∏t=1Tpθ(xt−1|xt).p_\theta(x_{0:T}) = p(x_T) \prod_{t=1}^{T} p_\theta(x_{t-1}\mid x_t).

The product symbol means that we multiply together the density of every denoising transition.

Why the Likelihood Is Difficult

The calculation above gives the density of one particular path.

However, the same final output x0 could be reached through many different intermediate states. For example, several paths might end at x0 = 0.1:

1.2⟶0.8⟶0.4⟶0.1,1.2\longrightarrow0.8\longrightarrow0.4\longrightarrow0.1,
−0.5⟶0.6⟶0.3⟶0.1,-0.5\longrightarrow0.6\longrightarrow0.3\longrightarrow0.1,

or

0.9⟶0.2⟶0.15⟶0.1.0.9\longrightarrow0.2\longrightarrow0.15\longrightarrow0.1.

When we are given only x0, we do not know which intermediate path was taken.

The states

x1,…,xTx_1,\ldots,x_T

are called latent variables. They are part of the model, but they are not observed.

To calculate the density of x0, we must combine the contributions of all possible latent paths:

pθ(x0)=∫pθ(x0:T)dx1⋯dxT.p_\theta(x_0) = \int p_\theta(x_{0:T}) \,dx_1\cdots dx_T.

For a discrete model, this would be a sum rather than an integral.

The difficulty is not evaluating one denoising step such as

pθ(xt−1|xt).p_\theta(x_{t-1}\mid x_t).

The model can evaluate that quantity when xt and xt-1 are known.

The difficulty is combining the contributions of every possible value of

x1,…,xT.x_1,\ldots,x_T.

For a standard diffusion model, this marginalisation is generally intractable.

The Forward Noising Process

To make the problem manageable, diffusion models define a fixed forward process.

The forward process begins with a knonw clean data point and gradually adds noise:

x0⟶x1⟶⋯⟶xT.x_0 \longrightarrow x_1 \longrightarrow \cdots \longrightarrow x_T.

Its distribution is written as

q(x1:T|x0)=∏t=1Tq(xt|xt−1).q(x_{1:T}\mid x_0) = \prod_{t=1}^{T} q(x_t\mid x_{t-1}).

The two distributions have different roles: q is the fixed forward process that adds noise; pθ is the learned reverse process that removes noise.

In a standard Gaussian diffusion model, a noisy state can be sampled directly from x0:

xt=α‾tx0+1−α‾tϵ,ϵ∼𝒩(0,I).x_t = \sqrt{\bar{\alpha}_t}\,x_0 + \sqrt{1-\bar{\alpha}_t}\,\epsilon, \qquad \epsilon\sim\mathcal{N}(0,I).

Here, x0 is the clean data point; ϵ is random Gaussian noise; αt control how much of the original data remains; and 1- αt controls how much noise is added. So the equation says take some part of the clean data, add some amount of random Gaussian noise, and you get the noisy sample at timestep t.

For a simple example, suppose

x0=10,α‾t=0.64,ϵ=−0.5.x_0=10, \qquad \bar{\alpha}_t=0.64, \qquad \epsilon=-0.5.

Then

xt=0.64(10)+1−0.64(−0.5).x_t = \sqrt{0.64}(10) + \sqrt{1-0.64}(-0.5).

Since

0.64=0.8\sqrt{0.64}=0.8

and

0.36=0.6,\sqrt{0.36}=0.6,

we obtain

xt=0.8(10)+0.6(−0.5)=7.7.x_t = 0.8(10)+0.6(-0.5) = 7.7.

The forward process has therefore turned the clean values 10 into a noisy values 7.7

This process does not make the exact likelihood easy to calculate. Instead, it gives an easy way to sample plausible noisy paths starting from a known x0.

This is what allows us to contruct the ELBO.

The ELBO

The ELBO (Evidence Lower Bound), provides a computable lower bound on the true log-likelihood.

To understand it, I use a simple discrete example with only two possible hidden paths, path A and path B.

Suppose the model assigns the joint probabilities:

pθ(x0,A)=0.06p_\theta(x_0,\text{A})=0.06

and

pθ(x0,B)=0.04.p_\theta(x_0,\text{B})=0.04.

The true probability of x0 is obtained by adding the probability of every path:

pθ(x0)=0.06+0.04=0.10.p_\theta(x_0) = 0.06+0.04 = 0.10.

The true log-likelihood is then

log⁡pθ(x0)=log⁡(0.10)≈−2.303.\log p_\theta(x_0) = \log(0.10) \approx -2.303.

In this example, adding both paths is easy. In a real diffusion model, there are too many possible trajectories to combine directly.

Instead of considering every path, the ELBO uses the tractable forward distribution

q(x1:T|x0).q(x_{1:T}\mid x_0).

In the two-path example, suppose the forward process assigns equal probability to both paths:

q(A|x0)=0.5q(\text{A}\mid x_0)=0.5

and

q(B|x0)=0.5.q(\text{B}\mid x_0)=0.5.

the ELBO is

ELBO⁡(x0)=∑pathsq(path|x0)log⁡pθ(x0,path)q(path|x0).\operatorname{ELBO}(x_0) = \sum_{\text{paths}} q(\text{path}\mid x_0) \log \frac{ p_\theta(x_0,\text{path}) }{ q(\text{path}\mid x_0) }.

Here, path means the whole hidden noisy trajectory x1 ,…, xT. Then q (path | x0) is the forward noising process, given the clean data x0. pθ (x0, path) is the models joint probability of the clean output and that path, and the whole thing is an avegare over paths of that log ratio.

Essentially, ELBO is how good the model is on average over noisy paths that start from x0.

For Path A,

pθ(x0,A)q(A|x0)=0.060.5=0.12.\frac{ p_\theta(x_0,\text{A}) }{ q(\text{A}\mid x_0) } = \frac{0.06}{0.5} = 0.12.

For path B,

pθ(x0,B)q(B|x0)=0.040.5=0.08.\frac{ p_\theta(x_0,\text{B}) }{ q(\text{B}\mid x_0) } = \frac{0.04}{0.5} = 0.08.

The ELBO is therefore

ELBO⁡(x0)=0.5log⁡(0.12)+0.5log⁡(0.08).\operatorname{ELBO}(x_0) = 0.5\log(0.12) + 0.5\log(0.08).

Numerically,

ELBO⁡(x0)≈−2.323.\operatorname{ELBO}(x_0) \approx -2.323.

The true log-likelihood was

log⁡pθ(x0)≈−2.303.\log p_\theta(x_0) \approx -2.303.

Therfore,

−2.323≤−2.303.-2.323 \leq -2.303.

The ELBO is slightly smaller than the true log-likelihood, which is why it is called a a lower bound.

Why divide by q?

The forward distribuition determines how often each path is sampled.

Dividing by

q(path|x0)q(\text{path}\mid x_0)

corrects for that sampling probability.

The ratio

pθ(x0,path)q(path|x0)\frac{ p_\theta(x_0,\text{path}) }{ q(\text{path}\mid x_0) }

compares how much probability the generative model gives the path with how often the forward process samples it.

The connection to KL divergence

The KL divergence measures the difference between two full probability distributions over hideen paths.

the exact relationship between the true log-likelihood and the ELBO is

log⁡pθ(x0)=ELBO⁡(x0)+DKL(q(x1:T|x0)∥pθ(x1:T|x0)).\log p_\theta(x_0) = \operatorname{ELBO}(x_0) + D_{\mathrm{KL}} \left( q(x_{1:T}\mid x_0) \parallel p_\theta(x_{1:T}\mid x_0) \right).

In the two-path example, the models true posterioir probability of path A is

pθ(A|x0)=0.060.10=0.6.p_\theta(\text{A}\mid x_0) = \frac{0.06}{0.10} = 0.6.

Similarly,

pθ(B|x0)=0.040.10=0.4.p_\theta(\text{B}\mid x_0) = \frac{0.04}{0.10} = 0.4.

The models true posterior over paths is therefore

(0.6,0.4).(0.6,0.4).

The forward distribution was

(0.5,0.5).(0.5,0.5).

Thes distributions are close, but not identical, so the mismatch prodcues a positive KL divergence, making the ELBO slightly smaller than the true log-likelihood.

Because KL divergenece cannot be negative,

DKL(q∥p)≥0,D_{\mathrm{KL}}(q\parallel p)\geq 0,

we always have

ELBO⁡(x0)≤log⁡pθ(x0).\operatorname{ELBO}(x_0) \leq \log p_\theta(x_0).

If the forward distributions matched the model’s true posterior exactly, then

DKL(q∥p)=0D_{\mathrm{KL}}(q\parallel p)=0

Applications to Inverse Folding diffusion models

Inverse folding is the task of predicting a sequence that folds into a target structure. Rather than modelling an unconditional distribution over sequences, an inverse folding model conditions everything on fixed structure S:

pθ(x0|S).p_\theta(x_0 \mid S).

The structure is held constant throughout, both the forward noising process and the reverse denoising process are conditioned on it. The forward noising process and the reverse denoising process are conditioned on it. The forward process becomes

q(x1:T|x0,S)=∏t=1Tq(xt|xt−1,S),q(x_{1:T}\mid x_0, S) = \prod_{t=1}^{T} q(x_t\mid x_{t-1}, S),

and the reverse process becomes

xt−1∼pθ(xt−1|xt,S).x_{t-1} \sim p_\theta(x_{t-1}\mid x_t, S).

Everything derived in the previous section is carried over unchnaged once S is added as a conditioning variable throughout. In particular, the exact decomposition still holds.

log⁡pθ(x0|S)=ELBO⁡(x0|S)+DKL(q(x1:T|x0,S)∥pθ(x1:T|x0,S)).\log p_\theta(x_0 \mid S) = \operatorname{ELBO}(x_0 \mid S) + D_{\mathrm{KL}} \left( q(x_{1:T}\mid x_0, S) \parallel p_\theta(x_{1:T}\mid x_0, S) \right).

Scoring/Ranking a Candidate sequence

The main use of this scoring, rather than generating. Given a candidate sequence x0 and a fixed strucure S, the question is not what sequence should the model produce, but how plausible does the model consider this specific sequence, given this specific structure.

Because the exact log-likelihood is intractible for the same reasons discussed above, the ELBO is used instead. In practice this is computed by taking the known sequence x0, corrupting it to a range of noise levels t, and evaluating how well the model reconstructs the true sequence identity at each position, conditioned on S. Summing these reconstruction terms across timesteps, following the same construction as before, prodcues ELBO (x0 | S).

Because the ELBO is a lower bound rather the exact values, it is less informative than its use as a ranking signal. Two candidate sequences x0(1) and x0(2) for the same structure S can be comaped using

ELBO⁡(x0(1)|S)versusELBO⁡(x0(2)|S).\operatorname{ELBO}(x_0^{(1)} \mid S) \quad\text{versus}\quad \operatorname{ELBO}(x_0^{(2)} \mid S).

In practice this can be used to rank candidate sequences or mutations by their structural plausibility under a fixed backbone, since no tractable exact alternative exists.

A note on discrete sequences

The Gaussian forward process intorduced earlier, with

xt=α‾tx0+1−α‾tϵ,x_t = \sqrt{\bar{\alpha}_t}\,x_0 + \sqrt{1-\bar{\alpha}_t}\,\epsilon,

applied to continuous data, Amino acid sequences are categorical rather than continuous, so a discrete diffusion model instead defines a forward process that corrupts a categorical identity directly, for example by replacing a residue with a randomly drawn amino acid with some probability that increases with t, rather than adding Gaussian noise to a continuous embedding .

The overall logic of the ELBO is unaffected by this distinction. The reverse process remains intractable, and the forward process remains tractable to sample from directly, regardless of whether the underlying noise model is Gaussian or categorical.

Conclusion

For an autoregressive model, the likelihood is directly calculated:

p(x)=∏i=1np(xi|x<i).p(x) = \prod_{i=1}^{n} p(x_i \mid x_{<i}).

For a standard diffusion model, the likelihood requires integrating over intermediate states:

pθ(x0)=∫pθ(x0:T)dx1⋯dxT.p_\theta(x_0) = \int p_\theta(x_{0:T}) \,dx_1\cdots dx_T.

That integral is the central difficulty. The ELBO avoids calculating it directly. It uses the tractable forward noising process to produce a lower bound on the true log-likelihood.

In a real diffusion model, the hidden variable is the complete noisy trajectory

x1,x2,…,xT.x_1,x_2,\ldots,x_T.

The ELBO is written as

ELBO⁡(x0)=𝔼q(x1:T|x0)[log⁡pθ(x0:T)q(x1:T|x0)].\operatorname{ELBO}(x_0) = \mathbb{E}_{q(x_{1:T}\mid x_0)} \left[ \log \frac{ p_\theta(x_{0:T}) }{ q(x_{1:T}\mid x_0) } \right].

The main idea is that the exact likelihood requires combining every possible hidden trajectory. The ELBO instead uses trajectories from an easy forward process to calculate a guaranteed lower bound on the log-likelihood. This is why likelihood evaluation is immediate for autoregressive models, while diffusion models often require a variational bound or another specialised estimation method.

Author