PaperlayerView exact revision on arXiv

Source-linked AI summary

Tabula Rasa: Monte Carlo estimation of unit-variance noise with controlled spatio-temporal correlation

Tobias Ritschel, Yang Zhou, Nick Milef, Mikhail Dereviannykh, Chen Liu, Christophe Hery, Carl Marshall

arXiv:2610.11653v1cs.GRcs.CV

TL;DR

Maintaining temporal correlation in Gaussian noise can affect its variance, although both properties matter for downstream video tasks. The paper jointly estimates Monte Carlo pixel reconstruction and variance with a hashing-based sketch. It reports white-spectrum, unit-variance noise with intricate temporal correlations, state-of-the-art speed, and parallelism over time.

  • Problem

    Generating noise with desired temporal correlation can affect its variance, so the method must control both properties.

    Source: §1 · p. 1

  • Method

    The method extends Monte Carlo pixel reconstruction by jointly estimating the pixel integral and variance, using a compact sketch based on two levels of hashing and a histogram.

    Source: §1 · p. 1

  • Results

    The method produces white-spectrum, unit-variance noise with intricate temporal correlations, and the paper reports state-of-the-art speed and parallelism over time.

    Source: §6 · p. 8

  • Takeaways & Limitations

    The noise is presented for video generation, stylized shading, animated texture synthesis, and correlated multi-view generation.

    Source: §5.2 · p. 7, §5.3 · p. 7, §5.5 · p. 8

    The method is biased, including because of nonlinear estimator transforms and the finite histogram.

    Source: §6 · p. 8

Abstract

from arXiv · show

We suggest a method to generate time-varying Gaussian noise with controlled variance and controlled temporal correlation. This noise is used in several downstream tasks for temporal control and temporal coherence. The core technical idea is to phrase this problem as joint Monte-Carlo estimation of both a classic pixel reconstruction and estimation of variance using the concept of "sketching" from the database literature. We demonstrate that our method allows temporal control for downstream tasks with simpler and faster code than previous methods.

1 Introduction

The paper targets noise that preserves both unit variance and spatio-temporal correlation, using Monte Carlo reconstruction augmented with variance estimation. Its approach combines sub-pixel decomposition with a compact hashing-based sketch, then uses prefiltering to address sampling difficulties.

1 Introduction

  • The method targets unit-variance noise with light-transport-induced spatio-temporal correlation, while avoiding the sequential computation and 2D-flow limitations of earlier approaches.Earlier methods used rasterization [Chang et al. 2024], Brownian bridges [Deng et al. 2025], or controlled pixel expansion and collapse [Burgert et al. 2025]; the paper reports simpler code, 3D coherence, and random access for its method.Source: §1 · p. 1, §2 · p. 2

3.1 Monte Carlo solution without controlled variance

  • Monte Carlo pixel reconstruction estimates a seeded random field over a pixel footprint, but combining its values does not generally preserve unit variance.The paper frames this as estimating the variance of the reconstructed random variable, distinct from the variance of the Monte Carlo estimator.Source: §3.1 · p. 2, §3 · p. 2

3.2.1 Sub-pixel decomposition.

  • Under the finite piecewise-constant field assumption, a pixel is a convex combination of uncorrelated subdomain values, with variance determined by their area weights.The areas are nonnegative and sum to one; the method does not require knowing the number or shapes of the subdomains.Source: §3.2 · p. 2, §3.2.2 · p. 3, §3.2.1 · p. 3

3.2.3 Estimating variance.

  • Because subdomain count and shape are unknown, the method estimates pixel reconstruction and subdomain areas jointly using a compact sketch.The sketch combines two levels of hashing with a histogram to make practical variance estimation possible.Source: §3.2.3 · p. 3, §3.2.2 · p. 3

Lattice.

  • A high-resolution lattice maps continuous coordinates to discrete positions, while the noise hash and global key determine spatial values and seed-dependent realizations.The lattice may be 3D and infinite; the described ray-tracing setup requires a unique hit point and excludes distribution effects or bounces.Source: Lattice · p. 3, Hashing · p. 3, §3.2.3 · p. 3

Hashing.

  • A smaller hash compresses lattice values into a per-thread histogram for estimating variate weights, trading storage for possible collisions.The described 16 × 16 repeating pattern ignores depth; collisions are reported only at extreme minification or magnification around 16-to-1.Source: Hashing · pp. 3–4

3.3 Prefiltering

  • With minification, counting many variables from few samples makes the variance estimate noisy, so prefiltering reduces the values that must be counted.Clamping discards noise bands that will be averaged away; the method blends the two most relevant levels and accounts for their independent variables in the variance.Source: Problem · p. 4, Solution intuition · p. 4, Clamping · p. 4

3.4 Summary

  • The paper summarizes the resulting path-tracing method in Algorithm 1 and also provides a scatter-rasterization variant with prefiltering.The alternative is described in the supplemental section.Source: §3.4 · p. 4

4 Analysis

The analysis establishes estimator consistency and bounds its bias, then evaluates noise quality, computational scaling, and comparison with prior methods. Experiments report near-white spectra and identify bin-count effects on variance, while the comparison finds non-inferior quality with improved global temporal consistency.

4.1 Analytical

  • The estimator is consistent and its bias is O(m^-1); the consistency argument assumes no Hs hash collisions, and the analysis generalizes to prefiltering.The paper also notes that the independent-sample assumption used in the analysis does not strictly hold for QMC, though observed convergence is similar or faster.Source: §4.1 · pp. 4–5

4.2 Empirical

  • MSE decreases with sample count, with QMC converging faster than MC; spectral deviation from white noise is about 3% of signal amplitude but grows with sample count as hash collisions increase.QMC converges slightly more slowly for ray tracing than for simpler zooming or optical-flow tasks.Source: §4.2 · pp. 5–6
  • At fixed m=1024, one bin yields variance around 0.1, while variance approaches 1 as bin count increases, with a sweet spot around 64 bins; Gaussianity seems unaffected by bin count.The Shapiro-Wilk test is used to assess Gaussianity.Source: §4.2 · p. 6

4.4 Comparison

  • Across the first four metrics, the method is reported as non-inferior to Burgert et al.; it has a small lead in PSNR and WRP-E and noticeable gains in 3D WRP-E.The authors describe the method as more general and random-access in time; they report no edge over trivial baselines for LPIPS and SSIM.Source: Table 1 · p. 6, §4.4 · p. 7

4.3 Computational performance

  • Compute time scales sublinearly with sample count and image resolution over the tested ranges; bin-count changes have only a mild timing relation as memory and atomic-contention effects offset.Prefiltered grids cost more to compute than simple grids, which the authors say pays off in higher quality.Source: §4.3 · p. 6
  • WebGPU has similar timing on game-optimized hardware and scales better with sample count when rendering geometry; its hardware rasterization supplies footprint derivatives more simply than JAX's jacfwd.The reported 96-GPU cluster produces a 96-frame animation in 1 × 2.44 ms, compared with 96×2.1 ms for Burgert et al. and 96×500 ms for Chang et al.Source: §4.3 · p. 6

5 Results

The paper applies spatio-temporally correlated unit-variance noise to diffusion-based video generation. Its described pipeline uses Go-with-the-Flow and downsamples the input noise to the model's latent resolution with a variance-preserving method.

5.1 Video generation

  • The application is controlling diffusion-based generative video models with spatio-temporally correlated unit-variance noise.The paper shows still frames and refers readers to a supplemental video for full sequences.Source: §5.1 · p. 7
  • The pipeline uses Go-with-the-Flow and downsamples 720 × 480 × 16 noise by a factor of 8 to the 90 × 60 × 16 VAE latent space.It uses variance-preserving uniform downsampling to keep the noise distribution compatible with the model's assumptions.Source: §5.1 · p. 7

13 noise

The paper applies its correlated noise to diffusion video generation, stylized shading, animated texture synthesis, steganography, and inter-view image generation. These examples show how controlled correlations affect motion, pattern consistency, and correspondence across views.

13 noise

  • When prompt and noise correlation conflict, the generated video can follow the prompt’s objects while retaining motion from the correlated spheres.The example uses falling-sphere correlations with an ice-cube prompt and produces cubes and chickens with sphere-like motion; experiments use 200 inference steps, guidance scale 6.0, and 20% random-noise mixing.Source: Conflicts · p. 7, §13 · p. 7

Conflicts.

  • Prompt-correlation matches tend to give the best results across diffuse, mirror, and glass prompts paired with diffuse, reflective, and refractive correlations.The best-performing combinations are the diagonal of the prompt-and-correlation comparison.Source: Light transport-consistency · p. 7

5.2 Stylized shading design

  • Multi-level variance-corrected noise keeps hatching pattern size and density constant across object positions and camera zoom.Regular 3D noise instead produces patterns that are too coarse up close and too dense at a distance, while its reduced variance causes hatching strokes to fade.Source: §5.2 · p. 7

5.3 Animated texture synthesis

  • Initializing frame-by-frame texture synthesis with correlated noise yields the desired texture and motion, while the random-noise baseline lacks coherence.The example uses correlations from two spheres falling and bouncing according to a physics simulation; the epipolar slices compare the correlated-noise animation with random noise.Source: §5.3 · pp. 7–8, Figure 5 · p. 7

5.4 Steganography

  • The animation can encode a user-provided flow or 3D scene as a message hidden in a single frame but perceptible across multiple frames.The example hides a complex 3D structure in noise without spatial structure in a single frame.Source: §5.4 · p. 8

5.5 Inter-view correlation

  • Generating correlated noise from multiple views supports corresponding images for lightfields and stereo pairs.The example uses a prompt for a glass ball and Tarot cards, inspired by the Stanford lightfield repository image.Source: §5.5 · p. 8

6 Conclusion

The conclusion presents the method as a fast, time-parallel extension of Monte Carlo path tracing for spatially white, unit-variance noise with intricate temporal correlations. It also notes estimator bias and identifies unbiased estimation and other spatial-correlation targets as future directions.

6 Conclusion

  • The estimator is biased by nonlinear estimator transforms and a finite histogram; unbiased estimation and targeting blue or golden noise remain future work.The paper identifies unit variance with temporal correlation as its current target property.Source: §6 · p. 8
  • The final method is simple: track accessed random variables in Monte Carlo image generation and rescale variance by a square root and division.The derivations are described as involved, but the final procedure is described as simple.Source: §6 · p. 8
  • The method generates noise that is white-spectrum and unit variance in space while following intricate temporal correlations.It extends Monte Carlo path tracing and can also be implemented in a rasterizer, with state-of-the-art speed and parallelism over time.Source: §6 · p. 8

Supplemental Material for Tabula Rasa: Monte Carlo estimation of unit-variance noise with controlled spatio-temporal correlation

The supplement outlines additional derivations and implementations for the paper’s variance-controlled noise estimation method. It also previews further analysis and NPR results.

Supplemental material

1 Primer on variance and correlation

The primer explains why combining random variables changes variance and why this method estimates and rescales it. It distinguishes unwanted spatial correlation from the controlled temporal and 3D correlations the method seeks to introduce.

1 Primer on variance and correlation

  • Averaging unit-variance random variables reduces variance, and filtering changes variance because it forms a weighted sum of pixel values.For an average of n variates, variance decreases inversely with n.Source: §1 · p. 11
  • The method estimates the changed variance and divides by its square root to restore unit variance.The primer describes this rescaling as restoring image contrast.Source: §1 · p. 11
  • The method aims to avoid unwanted spatial correlation in its output while introducing controlled temporal correlation.Downstream methods add spatial correlation, so the primer says they need to be able to assume it is absent in the input.Source: Correlation · p. 11

Correlation.

  • Unlike prior 2D flow-based correlation, direct sampling of a 3D random field supports projections unaffected by occlusion or clipping.The approach also extends to reflection and refraction of rays computed by Whitted-style ray casting.Source: Correlation · pp. 11–12

2 Rasterization pseudo-code

The supplement provides a rasterization-shader variant alongside the main paper’s ray-tracing approach, then derives the remainder bound used in Main Thm. 2. The proof uses smoothness and independent samples to establish an O(m^-2) expected remainder.

2 Rasterization pseudo-code

  • The rasterization shader provides a variant of the method that complements the main paper’s ray-tracing implementation.It is presented as Algorithm 1.Source: §2 · p. 12, §3 · p. 12

3 Additional details on the proof of Main Thm. 2

  • E[R(ba)] = O(m^-2), because independence and zero-mean centered samples leave only diagonal terms in the third-moment sum.The proof concludes that the sum has m constant terms divided by m^3.Source: §3 · pp. 12–13
  • The proof bounds the Taylor remainder using the mean value theorem and bounded third derivatives of the smooth function 𝜙.The derivatives are bounded over a neighborhood covering the point on the segment between a and ba.Source: §3 · p. 12

4 Spatial correlation

Resampling a pixel grid from a non-aligned texel grid leaves residual spatial correlation when pixels share texels. The supplement relates that correlation to pixel weights and shows it can shrink with increasing texel resolution, while reconstruction filters and integrators can introduce further considerations.

4 Spatial correlation

  • Non-aligned resampling leaves residual spatial correlation because neighboring pixels can share texels, but increasing texel resolution can make it arbitrarily low.The shared texels are the overlap between pixel reconstructions from the texel grid.Source: §4 · p. 13, Figure 4 · p. 13

4.1 Correlation pixels due to a shared texel

  • The Pearson correlation between two pixels is the cosine similarity of their normalized texel-weight vectors, with shared texels supplying the overlap.This analysis assumes perfect estimates and no Monte Carlo noise.Source: §4.1 · pp. 13–14

4.2 Example

  • In the 1D example with 10 texels per pixel, spatial correlation peaks at approximately 0.026 when the shared texel covers both pixels equally.The shared texel’s overlap length w is 0.05 at the peak.Source: §4.2 · p. 14
  • For n×n texels per pixel, the shared edge strip contains approximately n texels, and residual correlation vanishes linearly as resolution grows.This estimate assumes equal texel weights and each shared texel split exactly in half.Source: §4.2 · p. 14, Figure 6 · p. 14

4.3 Discussion

  • Changing reconstruction filters does not eliminate the effect; non-overlapping filters appear to minimize it but do not guarantee zero correlation.The authors had experimented only with box filters and identify exploring the filter–decorrelation continuum as future work.Source: §4.3 · p. 14
  • Other correlations can arise from light transport or the integrator, including QMC; the authors report no observed issues from the formally correct residual correlation.They note scrambling as a possible way to defeat integrator-induced correlation when necessary.Source: §4.3 · pp. 14–15

5 Additional results of non-photorealistic rendering

The supplemental NPR results compare variance-corrected noise with regular 3D noise for stippling and hatching. The authors report consistent pattern size and density across object position and camera zoom, and identify animation and blue-noise sampling as possible extensions.

5 Additional results of non-photorealistic rendering

  • The NPR examples use binary-thresholded noise for point stippling and anisotropically stretched noise layers for hatching.Color stippling independently thresholds CMYK channels using different noise seeds.Source: §5 · p. 15
  • Compared with regular 3D noise, multi-level variance-corrected noise gives consistent pattern size and density regardless of object position and camera zoom.Source: §5 · p. 15
  • The authors identify UV-space noise for object animation and blue noise for improved sample uniformity as potential future extensions.Source: §5 · p. 15

6 2D vs 3D flow

The section distinguishes types of flow and describes which conversions or flow inputs the methods can use. Absolute and relative flows are interchangeable up to numerics, while reversing forward or backward flow is harder in practice.

6 2D vs 3D flow

  • The section identifies a substantive distinction between 2D and 3D flow, while noting that this method can also take 2D flow as input.It says this method can take more advanced input than previous work, without treating the flow types as assumptions.Source: §6 · p. 15, 2D vs 3D · p. 15

Absolute vs relative.

  • Absolute and relative flow can be converted into one another by simple additions, and all previous methods and this method can use either up to numerics.Source: Absolute vs relative · p. 15

Forward vs backward.

  • All methods, including this one, can consume forward or backward flow, but conversion is harder because practical finite grids limit smooth reverse mapping.Source: Forward vs backward · p. 15

2D vs 3D.

The paper uses stable 3D-flow identifiers to preserve recognized point identity across space and time, including through occlusion. It also describes trajectory splatting and mode selection for resolving mixed motion at disocclusion edges.

2D vs 3D.

  • 3D flow gives each pixel a unique, stable identifier, preserving the identity of points recognized by the flow algorithm across occlusion.Source: 2D vs 3D · p. 15
  • Stretching and blending the noise means it is no longer unit-variance Gaussian, although its qualitative advantage remains evident.Source: 2D vs 3D · p. 15
  • Unlike coordinate-only prior methods, the flow can use time to represent points after occlusion, which those methods lose when a point disappears.Source: 2D vs 3D · p. 15

Implementation.

  • At disocclusion edges, additive trajectory splatting can average foreground and background motion; clustering contributors and selecting the dominant mode avoids that unwanted mix.Source: Implementation · p. 15
  • The 3D warping error replaces adjacent-frame flow with 3D optical flow to measure each pixel's difference from its birth pixel.Source: 3D Warping Error · p. 15

3D Warping Error.

The 3D warping error compares pixels with their birth pixels using 3D optical flow to assess global temporal consistency. The section also reports consistent pattern scale in NPR renders and identifies a table comparing optical-flow modalities.

3D Warping Error.

  • The 3D warping error uses reference-video 3D flow to measure each pixel's RGB difference from its birth pixel, capturing global temporal consistency.The protocol reduces RGB-channel error and averages across pixels in a video.Source: 3D Warping Error · p. 15
  • Ours maintains consistent pattern scale in NPR stippling and hatching renders compared with the world-space noise baseline.Source: Figure 7 · p. 16

Optical Flow Support.

Loading 2610.11653v1…