PaperlayerView exact revision on arXiv

Source-linked AI summary

Self-Refining Video Sampling

Sangwon Jang, Taekyung Ki, Jaehyeong Jo, Saining Xie, Jaehong Yoon, Sung Ju Hwang

arXiv:2601.18577v2cs.CVcs.LG

TL;DR

Modern video generators struggle with complex physical dynamics, while external verification and additional training can be costly and limited. This paper uses a pretrained generator as an inference-time self-refiner through denoising-autoencoder-style refinement, adding uncertainty-aware regional updates. Across state-of-the-art models, the method improves motion coherence and physical realism, with over 70% human preference on Wan2.2 over the default sampler.

  • Problem

    Video generators still struggle with complex physical dynamics, while existing external-verifier and additional-training approaches are costly or limited in capturing fine-grained motion.

  • Method

    The method reuses a pretrained video generator as a self-refiner, iteratively refining latents through denoising-autoencoder-style sampling without external models or additional training.

  • Results

    Across state-of-the-art video generators, the method improves motion coherence, physical plausibility, and spatial consistency, achieving more than 70% human preference on Wan2.2 over the default sampler.

  • Takeaways & Limitations

    Inference-time self-refinement provides a practical approach for improving the motion coherence, physical plausibility, and overall quality of existing pretrained video generators.

  • Takeaways & Limitations

    Tasks requiring discrete or semantic correctness may show little or no improvement, and cases where the generator fails almost entirely may require external verifiers.

Abstract

from arXiv · show

Modern video generators still struggle with complex physical dynamics, often falling short of physical realism. Existing approaches address this using external verifiers or additional training on augmented data, which is computationally expensive and still limited in capturing fine-grained motion. In this work, we present self-refining video sampling, a simple method that uses a pre-trained video generator trained on large-scale datasets as its own self-refiner. By interpreting the generator as a denoising autoencoder, we enable iterative inner-loop refinement at inference time without any external verifier or additional training. We further introduce an uncertainty-aware refinement strategy that selectively refines regions based on self-consistency, which prevents artifacts caused by over-refinement. Experiments on state-of-the-art video generators demonstrate significant improvements in motion coherence and physics alignment, achieving over 70% human preference compared to the default sampler and guidance-based sampler.

1. Introduction

Video generators remain unreliable for complex physical dynamics, motivating inference-time self-refinement without external models or additional training. The proposed method iteratively refines latent samples and selectively targets uncertain regions, improving physical realism and motion quality.

  • Current video generators struggle with complex physical dynamics, and small visual errors can undermine applications such as robot manipulation.
  • External verifiers and additional training address these limitations but can be inefficient, domain-specific, or computationally expensive.
  • The method uses a pretrained video generator as its own self-refiner during inference, without external models or additional training.
  • Predict-and-Perturb iteratively noises and denoises video latents at a fixed noise level by reusing the generator's denoising-autoencoder behavior.The predicted clean endpoint is perturbed back to the same noise level, steering latents toward higher-density regions of the learned video distribution.
  • Uncertainty-aware refinement selectively updates uncertain spatio-temporal regions, mitigating artifacts from over-refinement while leaving stable regions largely unchanged.The strategy uses self-consistency in model predictions to gate refinement without extra computation.
  • More than 70% human preference on Wan2.2 was achieved against the default sampler, alongside improvements in motion coherence, physical plausibility, and spatial consistency.

2. Related Works

Prior work improves video motion and physical fidelity through self-reflective sampling, training, guidance, or specialized data. These approaches can require substantial computation or still struggle with complex motion.

  • Self-refinement in generative models uses internal signals to improve outputs without external evaluators, teachers, verifiers, or additional training.
  • Prior self-refinement methods include iterative language-model critique, MCMC-based sampling, and diffusion sampling that alternates denoising with inversion.
  • Video motion-coherence methods include joint optical-flow training and training-free guidance to reduce temporal variance.
  • Existing approaches can require substantial training or inference computation and still struggle with complex motions.
  • Physical-fidelity approaches train models on curated physics, domain-specific, or synthetic CGI datasets, including physics-focused mixture-of-experts models.

3. Preliminaries: Flow Matching in Video Diffusion Models

Modern video generators commonly apply flow matching in a compressed VAE latent space. Flow matching learns a time-dependent vector field that transports Gaussian prior samples toward the video data distribution through an ODE solved over discretized timesteps.

  • Video generators encode RGB videos into compressed VAE latents, reducing computational cost while preserving essential spatio-temporal structure.
  • Flow matching learns a time-dependent vector field that transforms samples from a Gaussian prior distribution into the target data distribution.
  • The learned vector field generates samples by solving an ordinary differential equation across discretized timesteps.
  • Training uses straight paths between paired prior and data samples, with the model learning the target displacement vector.

4. Self-Refining Video Sampling

The method reinterprets flow matching as a time-conditioned denoising autoencoder, enabling iterative Predict-and-Perturb refinement during inference. An uncertainty-aware extension selectively refines low-confidence regions to improve samples while limiting over-refinement artifacts.

  • Self-refinement formulation: Flow matching can be interpreted as training a time-conditioned denoising autoencoder across all noise levels.At inference, denoising at a fixed timestep acts as reconstruction at that noise level.
  • Predict-and-Perturb: Predict-and-Perturb iteratively reconstructs and corrupts video latents at a fixed noise level, using the generator itself without an external verifier.Predict maps a noisy state to a clean-latent prediction, while Perturb adds noise for local resampling.
  • Sampling integration: P&P can be integrated into existing ODE solvers in a plug-and-play manner, with refinement applied during selected inference timesteps.Applying P&P only at early noise levels, t<0.2, is reported to suffice because coarse motion and structure are determined early.
  • Over-refinement limitation: Applying more than 3 P&P updates with classifier-free guidance can cause over-saturation or simplification in static regions such as the background.Repeated amplified guidance accumulates more strongly in regions that remain unchanged across refinements.
  • Uncertainty-aware refinement: Uncertainty-aware P&P selectively refines locally uncertain regions while preserving confident regions to prevent over-refinement artifacts.The uncertainty map compares reconstructed predictions, and uncertain regions are refined while stable regions remain largely unchanged.
  • Uncertainty-aware refinement: Uncertainty maps align higher values with moving objects, using τ = 0.25 to separate regions for refinement from static background regions.The maps are computed at t = 0.1T and visualize masks over videos generated by Wan2.2-A14B T2V.

5. Experiments

Experiments evaluate self-refining sampling across challenging motion, robotics, physics-aligned, spatial-consistency, and visual-reasoning settings. Across these evaluations, the method improves motion, physical plausibility, spatial consistency, and selected temporal-refinement tasks, while discrete or semantic tasks show little improvement.

  • Motion Coherence for Challenging Motions: The method is evaluated on Dynamic-bench and VideoJAM-bench using Wan2.1 and Wan2.2 T2V, against UniPC, doubled NFE, CFG-Zero, and FlowMo.Dynamic-bench contains 120 prompts covering multi-object interactions, complex human motions, and physics-driven dynamics.
  • Motion Coherence for Challenging Motions: The method produces more realistic complex motions, including plausible gymnast poses and hand–pommel interactions, where the ODE sampler generates duplicated arms and implausible poses.The comparison includes the doubled-NFE ODE baseline, which still fails on the illustrated motion.
  • Motion Coherence for Challenging Motions: 73% favor the method over the default sampler and 70% over FlowMo in tie-adjusted human evaluation of motion quality.The evaluation used 20 evaluators.
  • Physical Realism in Robotics Videos: On robotics I2V, grasp success improves by +11.0% on Cosmos and +8.4% on Wan versus the base ODE sampler, while Robot-QA accuracy is highest.The method also outperforms best-of-4 verifier-based rejection sampling with moderate computational overhead.
  • Physics Alignment in the Wild: For physics alignment, 84% favor the method over the default sampler and 74% over the doubled-NFE baseline in human evaluation.Automated evaluation also finds the method best on the physics commonsense metric across both benchmarks, with larger gains on PhyWorldBench.
  • Spatial Consistency and Visual Reasoning: The method improves spatial consistency under large camera motions and raises success on graph traversal from 0.1 to 0.8, but maze solving remains near zero.Tasks depending on discrete or semantic correctness show little or no improvement, and excessive refinement can cause over-saturation without uncertainty awareness.

6. Discussion

The discussion explains why iterative P&P can refine video motion while behaving differently from image refinement, and why uncertainty-aware updates limit over-refinement artifacts.

  • Video-specific refinement: Cross-frame consistency makes repeated P&P updates produce controlled changes in video temporal structures, unlike the larger shifts observed for images.Videos preserve neighboring-frame layouts and motion trajectories during perturbation, so refinement acts as local latent search.
  • Mode-seeking behavior: Iterative P&P concentrates samples in stable, high-density modes, reducing temporal variance in videos rather than collapsing them to identical content.In videos, this temporal mode-seeking removes jittering and flickering artifacts.
  • Relation to prior samplers: Unlike ALD, P&P performs stochastic perturbation and correction at a fixed noise level and is not a strict MCMC sampler.ALD instead traverses annealed noise scales to approximate a target distribution, whereas P&P exhibits mode-seeking behavior.
  • Uncertainty-aware refinement: Multiple P&P updates without uncertainty awareness can cause over-saturation and motion misalignment, motivating selective refinement.The ablation attributes these artifacts to repeated updates, particularly where regions are already stable.
  • Relation to prior samplers: Unlike Restart and FreeInit, P&P locally resamples intermediate latents within one trajectory rather than restarting from earlier states or rerunning full denoising.This placement of refinement is described as more compute-efficient than repeating FreeInit’s full denoising process.

7. Conclusion

The paper concludes that self-refining sampling reuses a pretrained video generator to refine latents during inference, with uncertainty-aware updates targeting uncertain regions. Experiments report consistent gains in motion coherence, physical plausibility, and overall quality across video-generation tasks.

  • Conclusion: P&P reuses a pretrained video generator as a self-refiner by revisiting flow matching as a generalized denoising autoencoder.The method refines latents at each inference timestep without additional supervision or retraining.
  • Conclusion: Uncertainty-aware P&P selectively refines uncertain regions using self-consistency signals from the model itself.This strategy is presented as an extension of the core self-refining sampling method.
  • Conclusion: P&P consistently improves motion coherence, physical plausibility, and overall quality across diverse video-generation tasks.The conclusion characterizes the approach as practical and broadly applicable to existing pretrained video generators.

Impact Statement

The work targets more reliable and physically consistent video generation through training-free inference-time refinement. It frames improved temporal stability and physical plausibility as steps toward more trustworthy video generation while noting existing misuse risks.

  • Impact: The method aims to improve video-generation reliability and physical consistency by enhancing motion and temporal coherence during inference.It does so without additional supervision or retraining.
  • Impact: Principled sampling complements advances in large-scale video generators without requiring additional supervision or retraining.The stated scope is inference-time refinement rather than changes to model training.
  • Impact: Improved temporal stability and physical plausibility are presented as an important step toward more trustworthy video generation.The statement remains within the paper’s stated impact scope.
  • Risks: The method does not introduce risks beyond those already present in existing video-generation systems, which may be misused to create misleading content.This is the paper’s explicit risk boundary.

A.2. Implementation Details

The implementation specifies model-, task-, and timestep-dependent P&P settings, evaluation procedures, and baselines across video-generation benchmarks. Automated metrics, human comparisons, physics-alignment criteria, and compute settings define the experimental protocol.

  • Compute and baselines: Experiments run on a single NVIDIA H100 80GB GPU, while P&P increases NFE but keeps memory usage identical to the base sampler.The implementation also uses gradient checkpointing for FlowMo to reduce hardware requirements.
  • Uncertainty-aware implementation: The uncertainty mask is accumulated across P&P iterations so regions identified as certain are frozen in later refinement steps.This implementation operationalizes selective refinement over the sampling trajectory.
  • Hyperparameters: P&P refinement strength is controlled by the number of iterations assigned to inference-step ranges during the motion stage.For motion-enhanced generation, the example plan applies three iterations at steps 3–6 and one at steps 7–14, adding 20 NFEs.
  • Hyperparameters: Early inference steps are left unrefined to establish coarse spatial layout, while later steps receive lighter refinement for efficiency.The stated example avoids P&P at the earliest steps and applies one iteration in later steps.
  • Evaluation: The evaluation covers motion coherence, physical commonsense, text alignment, grasp success, and physics-related benchmark categories.Robot grasp success counts scores of 4 or 5 as successful, while VideoPhy2 and PhyWorldBench use their stated motion-focused evaluations.
  • Evaluation: VideoJAM-bench results show the strongest performance on VBench metrics, although those automated metrics are largely saturated.The paper therefore notes limited sensitivity to fine-grained motion quality.
  • Evaluation: Human evaluations compare same-seed video pairs against baselines, include ties, and report results on Dynamic-Bench and VideoPhy2 hard subset.Evaluators assess motion quality, text alignment, or physical commonsense depending on the evaluation.

A.7. Ablation Studies on Hyperparameters

The ablations show that refinement strength, background preservation, and timing jointly determine P&P’s effectiveness. Earlier inference stages better correct motion, while later stages contribute less.

  • P&P iterations: Increasing Kf strengthens refinement but adds NFEs and increases deviation from base ODE samples.Kf =1 is insufficient to adequately refine large motion.
  • Confidence threshold: Higher τ makes refinement more conservative, preserving background structure and color tone while slightly weakening refinement strength.When Kf is large, jointly increasing τ mitigates saturation artifacts by restricting refinement to uncertain regions.
  • Default setting: The default setting Kf =3 and τ =0.25 balances sample quality and computational cost.
  • Inference timing: Applying P&P earlier is more effective for correcting motion errors, whereas later inference stages contribute marginally.Applying P&P only late reduces artifacts but cannot fully correct motion because of strong cross-frame consistency.

B.1. Cross-Frame Consistency of Video

Video refinement benefits from strong cross-frame consistency: iterative perturbations accumulate coherently, while excessive refinement can cause mode-seeking or artifacts. The method also extends beyond flow matching, but its reasoning gains depend on the task.

  • Cross-frame consistency: Strong temporal correlations make videos more robust to perturbation than images, requiring larger early-stage perturbations and iterative refinement for effective motion correction.
  • Cross-frame consistency: For videos, refinement effects accumulate consistently across iterations, motivating early-stage iterative refinement.The L2 distance between intermediate and final refined predictions decreases near-linearly for videos, unlike oscillatory image behavior.
  • Other domains: P&P reduces text-related artifacts in FLUX.1-dev, with two iterations adding only 4% NFEs and selectively refining challenging text regions.
  • Uncertainty-aware refinement: Uncertainty-aware refinement mitigates over-refinement artifacts by selectively modifying uncertain motion-related regions.
  • Visual reasoning: Graph-traversal success rises from 0.1 to 0.8 with self-refinement, whereas maze-solving success remains near zero.Motion or temporal consistency can be refined, but discrete semantic correctness is not substantially improved when the model lacks the needed knowledge.
  • Mode-seeking behavior: Excessive P&P can concentrate samples in modes and reduce diversity, while uncertainty-aware video sampling primarily refines motion and reduces flickering.
  • Other domains: P&P also applies to diffusion-based video models, correcting artifacts including truncated lightsabers and teddy-bear mouth distortions.

C. Limitations and Future Work

The method trades additional inference cost and refinement control against robustness and diversity. Its local, stochastic refinement is limited for tasks requiring global planning or precise semantic decisions.

  • Risk of Over-Refinement: Conservative uncertainty thresholds can weaken refinement or require more P&P iterations, leaving the balance between refinement strength and diversity unresolved.
  • Local-Search Behavior: P&P’s local-search behavior is inadequate for maze solving, where global search or external verifiers may be necessary.
  • Refinement Model Choice: The same model need not perform refinement; different or fine-tuned refinement models remain a future direction.
  • Stochasticity in Refinement: More refinement iterations increase improvement opportunities but remain dependent on stochastic noise, whose control is left for future work.
  • Reducing Inference Time: The default method increases inference time by approximately 40%, although matched-NFE variants remain more effective than standard ODE sampling.

D. Dynamic Bench

The Dynamic-Bench prompt set covers multi-object interactions, complex human motions, and physics-driven dynamics. It includes 120 scenario prompts spanning these categories.

  • Benchmark composition: The benchmark organizes prompts into multi-object interactions, complex human motions, and physics-driven dynamics.Prompts 1–40 cover multi-object interactions, 41–80 complex human motions, and 81–120 physics-driven dynamics.
  • Multi-object interactions: The benchmark includes interactions such as bowling, collisions, fluid mixing, deformation, and momentum transfer.
  • Complex human motions: It also covers demanding human motions including gymnastics, skating, dancing, parkour, and acrobatics.
Loading 2601.18577v2…