PaperlayerView exact revision on arXiv

Source-linked AI summary

Coarse-Guided Visual Generation via Weighted h-Transform Sampling

Yanghao Wang, Ziqi Jiang, Zhen Wang, Long Chen

arXiv:2603.12057v2cs.CVcs.AI

TL;DR

Coarse-guided visual generation must use degraded references without the costs of paired-data training, known forward operators, or unstable guidance–quality trade-offs. The paper proposes Weighted h-Transform Sampling, which approximates endpoint-directed h-transform guidance and weights it according to noise level. Experiments across image and video tasks show effectiveness and generalization, while the approximation error becomes increasingly problematic as noise decreases.

  • Problem

    Coarse-guided generation is constrained by paired-data training costs, requirements for known forward operators, and unstable balance between guidance faithfulness and generation quality.

  • Method

    Weighted h-Transform Sampling modifies sampling transitions with an approximate h-function based on the coarse sample and smoothly de-weights it as approximation error increases.

  • Results

    Extensive experiments across diverse image and video generation tasks demonstrate the method’s effectiveness and generalization.

  • Takeaways & Limitations

    The method provides training-free, operator-free, stable coarse-guided generation while maintaining guidance adherence and improving synthetic quality.

  • Takeaways & Limitations

    The coarse-sample approximation introduces unavoidable error that becomes increasingly problematic as the noise level decreases during sampling.

Abstract

from arXiv · show

Coarse-guided visual generation, which synthesizes fine visual samples from degraded or low-fidelity coarse references, is essential for various real-world applications. While training-based approaches are effective, they are inherently limited by high training costs and restricted generalization due to paired data collection. Accordingly, recent training-free works propose to leverage pretrained diffusion models and incorporate guidance during the sampling process. However, these training-free methods either require knowing the forward (fine-to-coarse) transformation operator, e.g., bicubic downsampling, or are difficult to balance between guidance and synthetic quality. To address these challenges, we propose a novel guided method by using the h-transform, a tool that can constrain stochastic processes (e.g., sampling process) under desired conditions. Specifically, we modify the transition probability at each sampling timestep by adding to the original differential equation with a drift function, which approximately steers the generation toward the ideal fine sample. To address unavoidable approximation errors, we introduce a noise-level-aware schedule that gradually de-weights the term as the error increases, ensuring both guidance adherence and high-quality synthesis. Extensive experiments across diverse image and video generation tasks demonstrate the effectiveness and generalization of our method.

1 Introduction

Coarse-guided generation is limited by training costs, known-operator requirements, and unstable guidance–quality trade-offs. Weighted h-Transform Sampling addresses these constraints with an approximate h-transform guidance mechanism and noise-level-aware weighting.

  • Challenges: Paired-data translation networks incur high training and collection costs and generalize poorly across different coarse-sample types.Different coarse inputs may require separate models and corresponding paired data.
  • Challenges: Training-free inverse-problem methods typically require a known fine-to-coarse forward operator, while alternative denoising methods struggle to balance guidance faithfulness and generation quality.The introduction identifies these constraints as central limitations of existing approaches.
  • Method: Weighted h-Transform Sampling modifies transition probabilities during sampling to add traction toward the ideal refined result.The method uses an h-transform perspective to incorporate coarse guidance without training a new model.
  • Method: Because the ideal h-function is intractable, the method approximates it with a coarse-sample-based term and smoothly reduces its weight as approximation error increases.The schedule is tied to noise level and is intended to preserve both coarse-guidance adherence and sample quality.
  • Results: Extensive experiments across image and video generation tasks demonstrate the method’s effectiveness and generalization.The supplied introduction identifies this as the main experimental conclusion.

2 Background

Diffusion models learn score-guided transport from a prior distribution toward target data, using forward and reverse stochastic processes. Doob’s h-transform modifies transition dynamics so sampling can be directed toward a fixed desired endpoint.

  • Diffusion Models: Diffusion models transport samples from a prior distribution p_T, such as Gaussian noise, toward the target data distribution p_0.A forward diffusion process defines the stochastic transport from data to the prior.
  • Diffusion Models: The forward diffusion process is represented by an SDE with drift f, diffusion coefficient g, time index t, and Brownian motion w.These terms specify the stochastic dynamics of the forward process.
  • Diffusion Models: A reversed SDE and an equivalent probability-preserving ODE describe sampling from the prior back toward the data distribution using the score ∇x_t log p_t(x_t).The reverse SDE changes the drift using the score, while the ODE uses a one-half diffusion correction.
  • Diffusion Models: A trained network s_θ predicts the unknown score and can replace it when solving the reverse SDE or ODE.The network is trained against tractable conditional scores induced by the forward process.
  • Doob’s h-Transform: Doob’s h-transform adds an h-function-dependent drift adjustment that changes transition probabilities and can guarantee termination at a fixed endpoint y.The h-function is defined as ∇x_t log p_t(x_T = y|x_t).

3 Weighted h-Transform Sampling

Weighted h-Transform Sampling uses a tractable coarse-sample h-function to approximate ideal guidance, then reduces its influence as approximation error grows during sampling.

  • Guided Generation with h-Transform: Given a coarse sample, the method modifies a pretrained diffusion model’s sampling dynamics to generate the corresponding fine visual sample.The ideal h-function would guarantee the endpoint equals the unknown fine sample, but it is unavailable because the ground truth is unknown.
  • Tractable Approximation: The method approximates the untractable ideal h-function with a tractable h-function based on the given coarse sample.Bayes’ rule and the closed-form conditional diffusion distribution yield an approximation involving the coarse sample, noise schedule, and trained score predictor.
  • Approximation Error Analysis: The approximation error is negatively correlated with noise level σ_t, diverging as σ_t →0 and vanishing as σ_t →1.For the variance-preserving diffusion example, the error is expressed using the coarse–fine sample difference and the noise schedule.
  • Sampling Algorithm: The sampling algorithm starts from Gaussian noise and repeatedly updates the latent using the pretrained score, coarse-sample guidance, noise schedules, and weight function.Algorithm 1 presents an Euler-solver implementation that iterates for M steps with step size Δt.
  • Weighted Approximation for Error Restriction: A σ_t-related weight function gradually reduces the approximate guidance term as approximation error increases.The scheduler keeps the weight near 1 when error is small and drives it toward 0 when error becomes large.

4 Related Work

The related work situates the paper within diffusion and flow-matching models, diffusion bridges, and conditional visual generation guided by modalities such as visual signals.

  • Diffusion Models and Bridges: Diffusion and flow-matching models generate visual samples through progressive transport between a prior distribution and a target distribution.The passage identifies DDPM, score matching, and optimal-transport flow matching as different perspectives on this model family.
  • Diffusion Models and Bridges: Large-scale pretrained models support noise-to-visual generation, while diffusion bridges extend visual generation to visual-to-visual translation.Doob’s h-transform is described as a typical mechanism for linking paired samples and learning a joint transport distribution.
  • Conditional Visual Generation: Conditional visual generation uses modalities such as text, action, and visual samples to constrain the generation range.The related work highlights coarse samples, strokes, and pose maps as examples of visual conditions.

5 Experiments

Experiments evaluate Weighted h-Transform Sampling on coarse-image restoration and camera-controlled video generation, with ablations and qualitative comparisons. The method generally improves guidance fidelity and generation quality across the reported settings.

  • Coarse-Image Guided Generation: On FFHQ 256x256, the study evaluates super-resolution, inpainting, motion deblur, and Gaussian deblur using FID and LPIPS.The dataset contains 1000 clean human-face images, and coarse inputs are created with forward operators and additive noise.
  • Coarse-Image Guided Generation: Compared with SDEdit, the method outperforms it across six of eight metrics and provides a better balance between guidance adherence and image quality.The qualitative comparison uses SDEdit t0 = 500 and the proposed method α = 5, with other settings held constant.
  • Coarse-Image Guided Generation: The method outperforms most operator-required baselines and performs competitively with DPS without knowing the forward operator.Compared baselines include ADMM-TV, ILVR, PnP-ADMM, MCG, DDRM, and DPS.
  • Coarse-Video Guided Generation: For camera-controlled video generation, the method uses a warped coarse video rendered from the first frame, depth, camera intrinsics, and prescribed camera poses.The evaluation uses 160 DL3DV-10K samples, retaining 49 frames resized to 720 x 480 resolution.
  • Coarse-Video Guided Generation: On DL3DV, the method achieves the best performance across all reported metrics except CLIP Cons. and shows better ground-truth alignment with decent image quality.The authors report that its CLIP Cons. is 0.972, while ground truth achieves 0.974 on that metric.
  • Ablation on Hyperparameter α: The ablation finds that σ_t-related weighting generally performs better across tasks and metrics, while α around 5 balances approximation error and guidance.Small α causes unclear images from larger approximation error, whereas excessively large α weakens coarse-image guidance.
  • Compatibility: The method is compatible with both score-based CogVideoX and flow-based Wan2.2.

6 Conclusion

The paper concludes that Weighted h-Transform Sampling injects coarse guidance during sampling through an approximate h function and controls approximation error with a weight scheduler. Extensive comparisons and ablations support its effectiveness and generalization.

  • Weighted h-Transform Sampling approximately drives generation toward the underlying ideal sample by injecting guidance during sampling.
  • A weight scheduler restricts approximation error to improve synthetic results.
  • Extensive comparisons and ablations support the method’s effectiveness and generalization.

Appendix

The appendix supplies theoretical proofs, derivations, implementation details, additional ablations and qualitative results, and an image-editing extension. It also covers formulations for both OT-FM and VP-SDE.

  • Section A.1 proves equivalent marginal distributions between the reversed SDE and its corresponding probability-flow ODE.
  • Sections A.2 and A.3 derive h_x0=ỹ and provide the weighted h-transform inference equation for OT-FM and VP-SDE.
  • Section A.4 documents coarse-video construction, video-generation text prompts, and related hyperparameter settings.
  • Section A.5 provides additional results, including ablations of the weight function λ and further qualitative comparisons.
  • Section A.6 extends the method to image editing.

A.1 Equivalence Proof of Marginal Distributions

The appendix proves that the reverse-time SDE and its corresponding ODE share the same marginal distributions. It does so by rewriting the Fokker–Planck evolution as a continuity equation and matching the resulting deterministic drift.

  • The proof starts from the reverse-time SDE drift and its Fokker–Planck equation for the latent density q_t(x_t).The reverse process includes the additional h-transform drift term h_x0=y.
  • Using Δq_t = ∇·(q_t∇log q_t), the density evolution is rewritten into continuity-equation form.
  • The resulting deterministic ODE drift is v_ODE = μ_SDE + 1/2 g^2(t)∇log q_t(x_t).
  • Substituting the score identity and simplifying yields the same drift as the ODE in Eq. (7).
  • Because both processes obey the same continuity equation, they have the same marginal distribution q_t(x_t) at every time t.

A.2 Derivation of hx0=ey

This section derives the tractable approximation h_x0=ẽ_y by differentiating the conditional Gaussian density p_t(x_t|x_0=ẽ_y). The resulting gradient is substituted into the h-transform expression to obtain the final guidance term.

  • The derivation starts from the conditional density p_t(x_t|x_0=ẽ_y).
  • Taking the logarithm and gradient yields (α_tẽ_y−x_t)/σ_t^2.
  • The derived gradient is substituted into Eq. (9) to obtain the final tractable h_x0=ẽ_y expression.

A.3 Weighted h-Transform Inference for Various SDEs

The method extends Weighted h-Transform Inference beyond score matching by deriving equivalent guided formulations for OT-FM velocity prediction and VP-SDE ϵ-prediction. These derivations express guidance through model-specific drift, score, or noise parameterizations.

  • Weighted h-Transform Inference is claimed to be compatible with DDPM and Flow Matching.
  • OT-FM with v-prediction: For OT-FM, the method rewrites the guided drift using the velocity field v_θ and a linear α_t, σ_t schedule.
  • OT-FM with v-prediction: The OT-FM derivation simplifies the guidance contribution to a term involving (x_t−ẽ_y)/σ_t.
  • VP-SDE with ϵ-prediction: For VP-SDE, the score is parameterized through ϵ_θ, and the condition defines a pseudo-target noise used inside the guided formulation.
  • VP-SDE with ϵ-prediction: The guided VP-SDE formulation combines ϵ_θ with the conditional pseudo-target through the weight λ_σ.
  • VP-SDE with ϵ-prediction: The resulting guided PF-ODE uses the linearly interpolated noise estimate ˆϵ_θ.

A.4 Implementation details

The implementation constructs coarse warped videos from a reference image, estimated depth, and camera motion. It resolves scale ambiguity and rendering artifacts through feature matching, depth ordering, hole filling, and mask refinement.

  • A pretrained DepthPro model estimates the first-frame depth map for coarse video construction.
  • The depth map is reprojected into a point cloud and rendered under specified camera motion poses.
  • SIFT feature matching with k-nearest-neighbor refinement and a 0.75 Lowe ratio threshold aligns depth scale with camera trajectories.
  • Z-buffering handles self-occlusions, while nearest-neighbor filling and 5∗5 morphological opening address deocclusion holes and projection noise.

The Implementation of Using Wan2.2 for Camera-controlled Video

The Wan2.2 implementation evaluates camera-controlled video generation with altered video dimensions and separate valid/invalid-region weighting. Additional experiments examine weight-function choices, restoration, video, and text-based editing results.

  • The Implementation of Using Wan2.2 for Camera-controlled Video: Wan2.2 videos use 81 frames at 832∗480 resolution, with λ_σ=σ^α and α=16 for valid regions and α=20 for invalid regions.
  • Ablation of the Weight Function: The weight-function ablation compares λ choices and α values on image super-resolution and inpainting tasks.
  • Ablation of the Weight Function: Weight functions related to σ_t achieve generally better performance across tasks and metrics, supporting the noise-level-aware design.
  • Additional Qualitative Results: Additional qualitative comparisons cover SDEdit for four image-restoration tasks and TTM and GWTF for camera-controlled video generation.
  • Image Editing: The editing setup uses the source image as ẽ_y and a target-prompt-conditioned pretrained text-to-image score.
  • Image Editing: On PIEBench image editing, the method outperforms ODE-Inv, SDEdit, and iRFDS while remaining competitive with FlowEdit and FlowAlign.
Loading 2603.12057v2…