PaperlayerView exact revision on arXiv

Source-linked AI summary

Probability Density Geodesics in Image Diffusion Latent Space

Qingtao Yu, Jaskirat Singh, Zhaoyuan Yang, Peter Henry Tu, Jing Zhang, Hongdong Li, Richard Hartley, Dylan Campbell

arXiv:2504.06675v2cs.CV

TL;DR

Studying image-space structure through probability-aware paths requires accurate score estimates and efficient geodesic solvers in extremely high-dimensional latent spaces. The paper develops diffusion-latent geodesic methods and applies them to video analysis and training-free image-sequence tasks. The authors report performance comparable to or better than existing state-of-the-art approaches.

  • Problem

    Studying image-space structure through probability-aware paths requires accurate score estimates and efficient geodesic solvers in extremely high-dimensional latent spaces.

    Source: §1 · p. 1, §4 · p. 4

  • Method

    The paper develops initial- and boundary-value solvers for probability-density geodesics in diffusion latent space and applies them to video analysis and training-free image-sequence tasks.

    Source: §1 · pp. 1–2

  • Results

    The training-free approaches for image interpolation and extrapolation perform comparably or better than existing state-of-the-art methods, with evaluation across five datasets.

    Source: §1 · p. 2, §6 · p. 8

  • Takeaways & Limitations

    The authors expect these techniques to be useful for generative-modeling tasks and for studying image-space distributions.

    Source: §6 · p. 8

    The method struggles with large camera motions or domain gaps, and its image-extrapolation performance is unreliable.

    Source: §5.3 · p. 8

Abstract

from arXiv · show

Diffusion models indirectly estimate the probability density over a data space, which can be used to study its structure. In this work, we show that geodesics can be computed in diffusion latent space, where the norm induced by the spatially-varying inner product is inversely proportional to the probability density. In this formulation, a path that traverses a high density (that is, probable) region of image latent space is shorter than the equivalent path through a low density region. We present algorithms for solving the associated initial and boundary value problems and show how to compute the probability density along the path and the geodesic distance between two points. Using these techniques, we analyze how closely video clips approximate geodesics in a pre-trained image diffusion space. Finally, we demonstrate how these techniques can be applied to training-free image sequence interpolation and extrapolation, given a pre-trained image diffusion model.

1. Introduction

Probability-density geodesics in diffusion latent space favor paths through probable image representations, providing a way to study the learned space. The work develops solvers and analysis quantities, then applies them to video analysis and training-free image-sequence tasks.

1. Introduction

  • A probability-density-weighted metric makes geodesics shorter through high-density, probable image latents than through low-density regions.The diffusion model’s score function supplies the gradient of log probability used to define this spatially varying geometry.Source: §1 · p. 1, Figure 1 · p. 1
  • The paper develops initial- and boundary-value solvers for Stable Diffusion’s 16,384-dimensional latent space, addressing score estimation, conditional densities, and computational efficiency.The methods also compute relative path probability, geodesic distance, and geodesic gradient norm.Source: §1 · pp. 1–2
  • The training-free approach achieves state-of-the-art results against interpolation methods across five datasets.The paper also reports performance comparable to or better than existing state-of-the-art approaches.Source: §1 · p. 2
  • The methods are used to assess how closely video clips approximate geodesics and to perform training-free image-sequence interpolation and extrapolation.Both applications use a pre-trained image diffusion model.Source: §1 · p. 2

2. Related Work

Image interpolation has developed from manually guided pixel transformations and constrained automation toward generative models. This work uses diffusion models in a training-free approach that also applies to extrapolation and provides a theoretical framework.

Text-to-image diffusion models

  • The work explores pre-trained Stable Diffusion for video analysis as well as image interpolation and extrapolation.The model is used to explore probability-density geodesics.Source: §2 · p. 2

Image interpolation

  • Traditional pixel-based interpolation often requires labeled correspondences and human involvement, while automated methods generally target controlled image pairs.GAN approaches have had limited success in open-world settings because input pairs must be in-distribution with training data.Source: §2 · p. 2
  • Diffusion-based methods generate smooth intermediate images for image pairs through textual embedding inversion and interpolation.The cited works include [11] [13] [53] [55] [56].Source: §2 · p. 2
  • The paper’s diffusion approach is training-free, extends to image extrapolation, and offers a theoretical framework for these tasks.The authors contrast it with prior works [11] [55] [56].Source: §2 · p. 2

3. Probability Density Geodesics

The section defines geodesics by minimizing path length under a probability-dependent metric, so high-density regions count as shorter routes. It then derives conditions for stationary paths and describes estimates of relative probability and geodesic distance.

Geodesic equations

  • A geodesic is a locally shortest path on a Riemannian manifold, obtained by minimizing curve length; stationary paths satisfy the Euler–Lagrange equations.The path is parameterized at constant speed.Source: §3 · pp. 2–3

Probability-density metric

  • The metric makes paths through high-probability regions shorter, and reduces to standard curve length when the probability distribution is uniform.The path-length functional is defined for curves joining two endpoints.Source: §3 · pp. 2–3

Geodesic equations

  • For the chosen Lagrangian, the optimality condition gives a nonlinear second-order ODE relating scaled acceleration to the log-probability gradient.The gradient of log probability is the Stein score function estimated by score-based diffusion models.Source: §3 · p. 3

Functional derivative

  • The path-length functional’s variation can be approximated by representing the curve as a polygonal line with an increasing number of segments.This derivation applies to curves with constant-speed parameterization.Source: §3 · p. 3

Probability density along the path

  • Relative probability along a path is computed from the line integral of the log-probability gradient and approximated at sampled points with the trapezoidal rule.The method assumes a conservative score field, which the paper says is a good approximation for standard diffusion models, though not strictly true.Source: §3 · p. 3

Geodesic distance

  • Geodesic distance is estimated from the probabilities computed along sampled points of the geodesic.The section describes this as a relative geodesic distance.Source: §3 · p. 3

4. Geodesics in Diffusion Latent Space

This section adapts geodesic computation to Stable Diffusion latents and addresses conditional densities, score reliability in low-density regions, and efficient solving in extremely high-dimensional space.

4. Geodesics in Diffusion Latent Space

  • The method computes geodesics in Stable Diffusion’s 64×64×4 latent space, using a noise predictor proportional to the score of the smoothed density.The latent is encoded from an image, and deterministic DDIM forward and backward processes provide the noised representation.Source: §4 · p. 3

4.1. Conditional probability density geodesics

  • Conditional geodesics use a conditioning signal ζ that matches endpoint conditions; experiments primarily use a latent-independent linear interpolation between endpoint embeddings.ζ(t) = (1 − t)z0 + tz1.Source: §4.1 · p. 4

4.2. Score estimation

  • At a nonzero diffusion timestep, noised data improves score estimates in low-density regions, while increasing the timestep makes the density more Gaussian.The method computes geodesics and queries the diffusion model at the selected timestep τ.Source: §4.2 · p. 4
  • For optimized latents that are initially out of distribution, the method uses noise-free score distillation, retaining domain correction while omitting the denoising direction.The passage notes that the denoising residual is generally nonzero and noisy, producing an averaging effect in optimized latents.Source: §4.2 · p. 4

4.3. Efficient IVP and BVP solvers

  • The solver setup assigns image interpolation to a boundary-value problem and image extrapolation to an initial-value problem, with the IVP solved using RK4.The IVP starts from an image and its text description and uses a target prompt to formulate the initial velocity.Source: §4.3 · p. 5
  • To constrain computation in the high-dimensional space, the BVP solver optimizes spherical control points with spline-derived velocities and accelerations, using a coarse-to-fine point schedule.The number of control points progresses through k ∈ [1, 3, 7, 15], and updates are projected back onto the sphere.Source: §4.3 · pp. 4–5

5. Experiments

Experiments test whether CLEVR videos approximate geodesics and evaluate geodesic solvers for image interpolation and extrapolation. Many video paths are approximately geodesic, and the training-free interpolation approach performs on par with the best methods.

5.1. Geodesic analysis of videos

  • Many CLEVR video paths are approximately geodesic: their gradient norms align with optimized paths, while perturbed or smoothed paths have larger norms.Source: §5.1 · p. 6, Figure 3 · p. 6

5.2. Applications

  • Interpolation evaluation uses MorphBench, Animals and Humans, and Web, reporting FID, PPL, PDV, and weighted TOPIQ on 17-frame sequences.Source: §5.2 · p. 6
  • Across interpolation datasets, the training-free approach is on par with the best methods, balancing high directness, input-distribution fidelity, and smoothness against slightly lower image quality.The comparison includes optimized geodesic paths and an unoptimized great-circle initialization.Source: §5.2 · p. 7, Table 1 · p. 7
  • On 25 randomly selected image pairs, optimization increases path log probability and decreases path length; the validation ablation compares text inversion, positive prompts, and negative prompts.Source: §5.2 · p. 7, Table 2 · p. 7

5.3. Limitations

  • Interpolation metrics struggle to capture smoothness, directness, and realism; FID is unreliable on these small image sets, while other metrics capture only individual aspects.Source: §5.3 · pp. 7–8
  • The method works well for local image morphing but struggles with large camera motions or domain gaps, and extrapolation is unreliable because good initial velocities are challenging to determine.The authors hypothesize that great-circle initialization can leave the optimizer in nearby local optima for difficult cases.Source: §5.3 · p. 8

6. Conclusion

The paper presents probability-density geodesic theory and algorithms for diffusion latent space, along with analysis quantities and training-free interpolation and extrapolation applications. The appendix derives weighted path length and its optimality conditions.

6. Conclusion

  • The methods compute probability-density geodesics and related path quantities, and the training-free interpolation and extrapolation approaches perform comparably or better than existing state-of-the-art.The paper identifies generative modeling and image-space distribution study as potential uses.Source: §6 · p. 8

A. Derivations

  • The action functional S is defined as the weighted path length of a path γ, using a Lagrangian L.The path has fixed endpoints xa and xb.Source: A. Derivations · p. 11
  • A path is stationary for S if and only if it satisfies the Euler–Lagrange equations, yielding a relationship between perpendicular components of scaled acceleration and the gradient of log weight.For constant-speed parameterization, acceleration along the path is zero.Source: A. Derivations · p. 11
  • The appendix derives the path-length functional derivative by approximating a curve with a polygonal line as the number of segments grows arbitrarily large.It also gives the expression for constant-speed parameterization.Source: A. Derivations · p. 11
  • With weight inversely proportional to probability density, the constant-speed optimality condition becomes a second-order ODE for high-probability geodesics.The appendix also gives the corresponding functional derivative.Source: A. Derivations · pp. 11–12

B. Further Implementation Details

Implementation details specify score-function normalization, conditioning and sampling choices, and how the initial velocity is set for the initial value problem.

B. Further Implementation Details

  • The score-function output is normalized by 1 + σ, and the experiments use a uniform weight function w(τ) = 1.The same negative text prompt is used across experiments.Source: B. Further Implementation Details · p. 12
  • Image interpolation uses perceptually uniform sampling with histogram equalization, while DDIM forward and backward processes use CFG 1 and the text-inverted positive conditional embedding.After optimization, each path point is projected back onto the sphere by scaling its norm to the sphere’s radius.Source: B. Further Implementation Details · p. 12
  • For the IVP, the initial velocity is the optimized pseudo-target displacement from the initial latent, projected onto the sphere’s tangent space.The pseudo-target uses a 0.8 mixture of target and source text embeddings, with 300 optimization iterations.Source: B. Further Implementation Details · p. 12

C. Further Details on Evaluation Metrics

The evaluation metric weights TOPIQ to emphasize the sequence’s middle frames, which tend to be farther from the source images.

C. Further Details on Evaluation Metrics

  • The weighted TOPIQ score emphasizes middle frames because they tend to be farther from the source images and more indicative of overall perceptual quality.The score uses a weighted rather than simple average.Source: C. Further Details on Evaluation Metrics · p. 12
  • The metric’s stated rationale is that middle frames are more indicative of overall perceptual quality than frames near the source images.The passage contrasts this weighted calculation with a simple average.Source: C. Further Details on Evaluation Metrics · p. 12
  • The weighting function is w(λ) = λ for λ ≤ 0.5 and w(λ) = 1−λ for λ > 0.5.This gives greater weight toward the sequence midpoint than the endpoints.Source: C. Further Details on Evaluation Metrics · p. 12

D. Additional Ablation Study on the Conditioning Signal

The ablation compares constant and time-linear conditioning alongside geodesic optimization, finding different advantages for the two conditioning choices.

D. Additional Ablation Study on the Conditioning Signal

E. Sensitivity Analysis of the Hyperparameters

The hyperparameters govern a trade-off between path directness and alignment with high-probability regions, image detail, and the strength of guidance during path optimization.

E. Sensitivity Analysis of the Hyperparameters

  • Larger β favors higher-probability regions over directness, while smaller β keeps the path more direct.β controls the trade-off between path directness and alignment with high-probability regions.Source: E. Sensitivity Analysis of the… · p. 12
  • Higher τ tends to morph high-level features but can lose fine details, while very low τ can degrade image quality through insufficient denoising.Source: E. Sensitivity Analysis of the… · p. 12
  • A sampling range ∆τ can help escape local minima versus zero range, but an overly large range weakens guidance and yields smoother paths with lower FID and TOPIQ scores.Source: E. Sensitivity Analysis of the… · p. 12

F. Further Qualitative Results

The appendix presents additional qualitative interpolation results and failure cases, including comparisons across methods and examples where smooth interpolation is limited.

F. Further Qualitative Results

Loading 2504.06675v2…