PaperlayerView exact revision on arXiv

Source-linked AI summary

How I Warped Your Noise: a Temporally-Correlated Noise Prior for Diffusion Models

Pascal Chang, Jingwei Tang, Markus Gross, Vinicius C. Azevedo

arXiv:2504.03072v1cs.CVcs.LG

TL;DR

Video diffusion methods often fail to preserve temporal correlations, producing flicker or texture sticking. This paper represents noise as an integrated continuous field and transports it with a variance-correcting warp. The resulting temporally correlated noise supports restoration, editing, rendering, and conditional generation, with effectiveness depending on the diffusion pipeline.

  • Problem

    Standard noise priors in video diffusion do not systematically preserve correlations between successive frames, leading to flickering or texture-sticking artifacts.

  • Method

    The method represents pixels as integrals of an underlying white-noise field and warps that field with a distribution-preserving transport algorithm.

  • Results

    The proposed noise warping method creates temporally correlated, distribution-preserving noise fields usable for video restoration, editing, surrogate rendering, and conditional video generation.

  • Takeaways & Limitations

    Integral noise provides a noise prior that preserves motion-induced temporal correlations while retaining the original noise properties.

  • Takeaways & Limitations

    The method is computationally less efficient than simpler correlation-preservation techniques, and its benefit depends on the diffusion model's training data and pipeline.

Abstract

from arXiv · show

Video editing and generation methods often rely on pre-trained image-based diffusion models. During the diffusion process, however, the reliance on rudimentary noise sampling techniques that do not preserve correlations present in subsequent frames of a video is detrimental to the quality of the results. This either produces high-frequency flickering, or texture-sticking artifacts that are not amenable to post-processing. With this in mind, we propose a novel method for preserving temporal correlations in a sequence of noise samples. This approach is materialized by a novel noise representation, dubbed $\int$-noise (integral noise), that reinterprets individual noise samples as a continuously integrated noise field: pixel values do not represent discrete values, but are rather the integral of an underlying infinite-resolution noise over the pixel area. Additionally, we propose a carefully tailored transport method that uses $\int$-noise to accurately advect noise samples over a sequence of frames, maximizing the correlation between different frames while also preserving the noise properties. Our results demonstrate that the proposed $\int$-noise can be used for a variety of tasks, such as video restoration, surrogate rendering, and conditional video generation. See https://warpyournoise.github.io/ for video results.

1 INTRODUCTION

Video diffusion pipelines struggle to preserve natural correlations across frames because standard noise sampling discards motion information, causing flickering or texture sticking. The paper proposes warping noise to preserve motion-induced correlations while retaining useful noise properties.

  • Standard frame-wise noise sampling discards motion information and produces high-frequency hallucinations that are difficult to remove.
  • Fixed noise artificially enforces correlations but causes texture-sticking artifacts that post-processing cannot readily remove.
  • The paper asks how to construct a noise prior that preserves correlations present in an image sequence.
  • The proposed approach warps noise according to motion vectors or optical flow to preserve induced temporal correlations.
  • Integral noise interprets each pixel value as the integral of an underlying infinite-resolution noise field rather than a discrete value.
  • The method combines a distribution-preserving transport equation with a tailored algorithm and supports video restoration, editing, surrogate rendering, and conditional video generation.

2 METHOD

The method represents discrete Gaussian noise as integrals over an underlying white-noise field, then transports that field through deformations while correcting variance changes. Its discretization preserves temporal correlation, noise distribution, and pixel independence better than ordinary interpolation.

  • 2.1 THE-NOISE REPRESENTATION: Discrete Gaussian pixels are reinterpreted as integrals of an underlying infinite-resolution white-noise field.
  • 2.1 THE-NOISE REPRESENTATION: Conditional sampling refines each coarse pixel into higher-resolution Gaussian sub-pixels while preserving the observed coarse value.
  • 2.2 TEMPORALLY-CORRELATED DISTRIBUTION-PRESERVING NOISE TRANSPORT: The transport pipeline back-warps subdivided pixel contours, triangulates and rasterizes them on a finer grid, then sums and rescales covered sub-pixels.
  • 2.2 TEMPORALLY-CORRELATED DISTRIBUTION-PRESERVING NOISE TRANSPORT: The transport equation uses the deformation's Jacobian determinant to rescale samples for local stretching and the associated variance change.
  • 2.2 TEMPORALLY-CORRELATED DISTRIBUTION-PRESERVING NOISE TRANSPORT: The method balances interpolation-induced correlation preservation with the original noise distribution and neighboring-pixel independence.
  • 2.3 TOY EXAMPLE IN 1-D: Linear interpolation has variance σ²_z = α² + (1 − α)² < 1 for α ∈ (0, 1), whereas integral-noise warping restores unit variance through stochastic sampling.

3 EXPERIMENTS AND RESULTS

The method is evaluated across four video tasks and compared with random, fixed, and interpolation-based noise priors. Results indicate improved temporal coherency while preserving competitive visual quality.

  • Validation: The proposed prior simultaneously maximizes correct correlation between warped and original samples while maintaining pixel independence within each sample.
  • Applications: The evaluation covers appearance transfer, video restoration and super-resolution, pose-to-person generation, and fluid simulation super-resolution.
  • Baselines: Random noise causes flickering or incoherent details, whereas fixed noise produces texture-sticking artifacts across video applications.
  • Appearance transfer: The method improves temporal coherency in appearance transfer while avoiding unnatural sticking artifacts in denoised sequences.
  • Fluid super-resolution: Warping noise along with fluid density produces smoother transitions between frames in fluid simulation super-resolution.
  • Quantitative evaluation: Table 1 reports better temporal coherency than existing noise priors, competitiveness with traditional interpolation, and visual quality on par with standard Gaussian priors.

4 ABLATION STUDIES AND DISCUSSIONS

The comparisons show that preserving noise distribution and independence is important for retaining detail during warping. The method’s behavior also depends on resolution and its application to latent diffusion models.

  • Interpolation methods: Bilinear and bicubic warping blur high-frequency details, while nearest-neighbor warping creates duplicate-sample artifacts under spatial stretching.
  • Interpolation methods: The proposed method retains high-frequency details and complete pixel independence in warped noise samples.
  • Upsampling factor: A smaller upsample factor k increases the chance that warped polygons cover no sub-pixels, producing undefined output pixels.
  • Latent diffusion models: Noise warping has limited impact in latent diffusion models because their lower-resolution noise mainly controls composition and low-frequency structure.

5 RELATED WORK

Related work applies diffusion models to image and video editing through noise inversion, attention, and fine-tuning. These approaches address temporal consistency through conditioning, modified latent representations, or learned temporal modules.

  • Diffusion models: Diffusion models generate samples by reversing a learned diffusion process from Gaussian noise, with sampled noises and integrator parameters forming a latent representation.
  • Noise inversion: Image editing methods commonly invert denoising to recover noise that reconstructs an image under a textual condition.
  • Video editing: Video editing methods often invert noise frame by frame and use cross-attention features from an anchor frame to improve temporal coherency.
  • Fine-tuning: Fine-tuning approaches learn task-specific conditions or temporal-attention modules for preserving structures during image and video editing.

6 LIMITATIONS & CONCLUSION

The paper introduces an integral-noise representation and transport procedure for temporally correlated, distribution-preserving noise. Its effectiveness depends on the diffusion pipeline, and the algorithm is computationally less efficient than simpler alternatives.

  • Conclusion: Integral noise represents samples as integrals of a higher-resolution white-noise field, enabling variance-preserving continuous interpolation.
  • Conclusion: The noise transport equation accounts for both deformation of underlying pixel shapes and the variance rescaling required during transport.
  • Conclusion: The resulting samples are temporally correlated and distribution-preserving, supporting applications including video restoration and editing.
  • Limitations: The algorithm is more computationally inefficient than simpler correlation-preservation techniques, and stronger temporal correlation does not always improve video coherency.

7 REPRODUCIBILITY STATEMENT

The reproducibility materials identify implementation resources, mathematical notation, and foundational white-noise definitions used in the paper.

  • Proofs and pseudocode are provided in Appendices B and C, with additional experiments in Appendix D.
  • Appendix E provides detailed analysis of the method in latent diffusion models.
  • The symbol definitions include noise dimensions, pixel coordinates, discrete Gaussian samples, Gaussian variables, Borel sets, and Lebesgue measure.
  • The appendix defines continuous white noise through a random set function on a σ-finite measure space.
  • White-noise values are Gaussian with variance equal to set measure, and values on disjoint sets are independent.

B.2 CONDITIONAL PROBABILITY DERIVATION

The appendix derives conditional sampling of higher-resolution white noise from a discrete pixel sample, then describes its unit-variance implementation and the transport-equation setup.

  • A discrete pixel is modeled as a constant value over a unit square that is subdivided into N × N sub-pixels.
  • The sub-pixel values form a multivariate Gaussian vector whose covariance follows from the areas shared with the original pixel.
  • Conditioning the sub-pixel vector on the original pixel yields a Gaussian conditional sampling procedure.
  • Unit variance scaling: Multiplying the conditional upsampling expression by N produces higher-resolution noise with unit variance.
  • Unit variance scaling: Algorithm 1 upsamples an H × W noise sample by factor N using random noise, block means, nearest-neighbor expansion, and mean correction.
  • The transport derivation treats continuous white noise and relates it to Brownian motion and Itô integration.

C.2 NOISE TRANSPORT EQUATION DERIVATION

This appendix derives a transport equation for continuously defined white noise and discretizes it into a distribution-preserving pixel-warping procedure.

  • Standard pointwise transport is insufficient because white noise is defined over continuously integrated pixel areas.
  • The proposed transport uses a diffeomorphic deformation and area-aware integration to account for spatial distortion.
  • Naively transporting white noise produces an integrated variance that is not proportional to area, violating white-noise properties.
  • The noise transport equation adds variance rescaling and is proved to produce a continuously defined representation with white-noise characteristics.
  • Discretization rasterizes warped pixel polygons, while locally constant Jacobian and polygon approximations affect deformation accuracy rather than noise values.
  • With disjoint rasterized adjacent triangles, the resulting pixel samples remain standard normal and independent.
  • The practical algorithm uses anchor-frame warping first, previous-frame upsampling for missing pixels, and random noise for any remaining gaps.

C.5 1-D TOY EXAMPLE: PROOF

The one-dimensional example explains how integral-noise interpolation preserves unit variance and spatial correlations, while experiments compare the method with alternative warping and runtime schemes.

  • 1-D toy example: In the 1-D example, an interpolated variable is represented by white noise over a shifted unit interval.
  • 1-D toy example: Its covariance with neighboring pixels is 1 − α and α, respectively, reflecting the overlap of their intervals.
  • 1-D toy example: The conditional interpolation has mean (1 − α)x0 + αx1 and variance 1 − (α^2 + (1 − α)^2).
  • Interpolation baselines: Bilinear and bicubic interpolation blur high-frequency details, nearest neighbor creates duplicate-sample artifacts, and root-bilinear introduces spatial correlations.
  • Interpolation baselines: The proposed noise method outperforms the tested warping baselines by transporting noise while retaining Gaussian properties.
  • Runtime: The method requires approximately 1 second per frame, whereas simple noise sampling is efficient on GPU and some alternatives are 3 orders of magnitude slower.
  • Runtime: Warping accuracy and efficiency are controlled by the noise upsampling factor k and polygon subdivision steps s.

D.1 PHOTOREALISTIC APPEARANCE TRANSFER WITH SDEDIT

The method improves temporal coherency in appearance transfer and video restoration by transporting noise with scene motion rather than using fixed or standard interpolated noise. It also preserves perceptual quality within the range of Gaussian noise priors while avoiding artifacts associated with interpolation.

  • Noise warping transports scene details with object motion, whereas fixed noise leaves the bedsheet fold fixed in image space after bed rotation.
  • The appearance-transfer evaluation measures temporal coherency using masked MSE between each frame and the warped previous frame.
  • Standard interpolation methods fail to account for spatial deformations, producing visible denoising artifacts and lower temporal coherency than noise warping.
  • FID evaluates both sample quality and diversity, so random noise can score better through increased frame diversity despite weaker temporal coherence.
  • For I2SB restoration, noise warping produces temporally smoother and spatially sharper results, while interpolation is blurrier and random noise is noisier.

E.1 UNDERSTANDING THE LIMITATIONS

Latent diffusion limits the benefit of noise warping because temporal consistency in image space does not necessarily survive encoding into latent space. The latent experiments nevertheless examine how motion-aware noise behaves under increasingly constrained generation pipelines.

  • Noise warping improves latent-model temporal coherency less than expected because coherent images need not yield coherent autoencoder latents.
  • In latent diffusion, noise contributes less directly to image-space fine details, while autoencoder reconstruction error independently affects temporal coherency.
  • The continuous Gaussian-noise construction depends on structural self-similarity that does not hold in latent diffusion models.
  • The latent setup uses translating synthetic discs, Stable Diffusion, ControlNet scribbles, and a prompt to test motion-aware noise.

E.3 RESULTS AND ANALYSIS

Across increasingly constrained latent-diffusion experiments, noise warping improves the temporal movement of generated fine details, with its effect diminishing as other temporal constraints dictate structure.

  • Noise warping is better when warped properly across all three increasingly constrained experiments.
  • With frame-by-frame ControlNet generation, warped noise makes synthesized details move with the translating pizza instead of flickering or remaining fixed.
  • With cross-frame attention, fixed noise still causes sticking artifacts, while warped noise produces more natural ingredient motion.
  • With feature-map injection, warped noise yields pixel-perfect pizza translation, whereas fixed noise retains fine-detail sticking artifacts.
  • The noise scheme matters less as cross-frame attention and feature maps impose more constraints, leaving noise responsible mainly for fine high-frequency details.
Loading 2504.03072v1…