PaperlayerView exact revision on arXiv

Source-linked AI summary

SyncVid: World-Equivariant Diffusion for Synchronized Video Generation

Farhad G. Zanjani, Animesh Karnewar, Fatih Porikli, Amirhossein Habibian

arXiv:2610.31765v1cs.CV

TL;DR

Generating synchronized videos from a reference requires coherent dynamic content across viewpoints, including unseen regions, while existing methods rely on camera conditioning, cross-view architecture, or rendered geometry. SyncVid rasterizes canonical 3D noise into target views and whitens it before using it as the prior for a LoRA-adapted pretrained video diffusion model. It reports strong cross-view synchronization and perceptual-quality results, with camera-pose estimation remaining a limitation.

  • Problem

    Synchronized generation must keep dynamic content coherent across viewpoints, while camera-control approaches primarily use architectural coupling, rendered conditioning, or specialized training data.

    Source: §1 · pp. 1–2

  • Method

    SyncVid assigns noise to canonical 3D scene primitives and unobserved space, rasterizes it into target views, and uses view-consistent whitening with a LoRA-adapted pretrained diffusion model.

    Source: §1 · p. 2, §3.2 · p. 4, §3.3 · p. 5

  • Results

    SyncVid provides the strongest overall cross-view synchronization, with 1,332 matched pixels in dynamic regions, 1,834 in static regions, cross-view FVD of 146.99, and cross-view CLIP similarity of 91.66.

    Source: §4.3 · p. 9

  • Takeaways & Limitations

    The results suggest view-consistent noise improves cross-view alignment and content fidelity without substantially compromising temporal or background stability.

    Source: §4.3 · p. 10, §5 · p. 11

    SyncVid does not outperform state-of-the-art methods in camera-pose estimation; explicit camera-pose conditioning is proposed as a possible improvement.

    Source: §A.7 · p. 24

Abstract

from arXiv · show

Generating multiple videos of a dynamic scene from a single reference video requires synthesizing both observed and unobserved content consistently across viewpoints. Existing camera-control methods for video diffusion typically rely on architectural mechanisms, including camera embeddings, cross-view attention, and geometric conditioning. Although effective for static scene content, these approaches often struggle with the fine-grained, dynamic geometry that must remain coherent across views and over time. We synchronize the diffusion process itself by sampling the Gaussian noise prior in a canonical 3D space derived from the source video and rasterizing it into target cameras, rather than sampling independently in each 2D image plane. Our method, SyncVid, lifts a monocular video into a canonical 4D representation, associates noise with scene geometry and unseen space, and rasterizes the resulting structured noise into arbitrary views. Because rendering can introduce local correlations, we further propose a cross-view-consistent whitening procedure that decorrelates neighboring pixels while preserving correspondence across views. The structured noise serves as the diffusion prior for a LoRA-adapted pretrained video diffusion model, requiring no architectural modifications. By enforcing spatiotemporal and cross-view coherence at the source of stochasticity, SyncVid generates synchronized videos and improves both perceptual quality and cross-view consistency in our experiments. Project page: https://qualcomm-ai-research.github.io/SyncVid-Webpage.

1 Introduction

Synchronized video generation must keep dynamic scene content coherent across viewpoints, including when views reveal or occlude different regions. SyncVid coordinates the diffusion prior in canonical 3D space, rather than relying only on architectural coupling or 2D noise warping.

1 Introduction

  • Camera-controlled video methods produce useful visual and geometric agreement but primarily rely on architectural coupling, rendered conditioning, or specialized training data.Synchronized views matter for virtual reshooting, AR/VR content creation, and training data for 3D and 4D vision.Source: §1 · p. 1
  • Existing structured-noise approaches use 2D warping or target-video optical flow, which can degrade quality for out-of-plane rotation, occlusion, and reappearance.The target video’s optical flow may be unavailable before generation.Source: §1 · p. 2
  • SyncVid samples noise in canonical 3D scene space and rasterizes it into target views, tying the diffusion prior to shared scene geometry instead of independent image-plane noise.The canonical representation spans both the reconstructed scene and surrounding unobserved space.Source: §1 · p. 2
  • View-consistent whitening reduces rasterization-induced local correlations while preserving cross-view correspondence and keeping noise close to the pretrained model’s expected marginal distribution.The method is designed to avoid relying on 2D warping or editing of image-plane noise.Source: §1 · p. 2
  • SyncVid also establishes an automated benchmarking pipeline for synchronized video generation on the Mixkit subset of Open-Sora.The stated goal is reproducible comparison without manual or subjective selection of evaluation samples.Source: §1 · p. 2

2 Related Work

Prior structured-noise methods improve temporal consistency or control motion, while synchronized camera-control methods couple views through architecture, rendered geometry, or training data. SyncVid instead projects a canonical 3D noise field into each view.

2.1 Structured Noise for Synchronized Video Generation

  • PYoCo [9], How I Warped Your Noise [10], Go-with-the-Flow [11], and EquiVDM [12] structure noise to support temporal consistency or motion control, largely through 2D motion warping.These approaches suit consecutive frames from one camera but do not represent out-of-plane camera rotation or newly revealed and occluded surfaces.Source: §2.1 · p. 3

2.2 Synchronized Camera-Controlled Video Generation

  • Synchronized camera-control methods use cross-view interaction modules [6] [7] [8], rendered geometric conditions [4] [5] [13] [15], or specialized synchronized training data [4].These methods address synchronization through camera conditioning, architectural coupling, or geometry rather than a shared diffusion prior.Source: §2.2 · p. 3
  • SyncVid samples a canonical 3D noise field and projects it into each view, providing geometric conditioning and view-consistent diffusion initialization.This contrasts with independently sampled image-plane priors and the related methods’ 2D motion-warping limits.Source: §2.2 · p. 3, §2.1 · p. 3

3 Method

SyncVid builds a canonical noise representation from dynamic scene geometry, rasterizes it into target views, and applies position-dependent whitening to retain correspondence while reducing local correlation. A LoRA-adapted video diffusion model uses these structured priors without changing its backbone.

3.1 Preliminary: Noise Sampling in Video Diffusion Models

  • The standard video-diffusion prior assumes independent standard-Gaussian entries, so strongly correlated replacement noise can create a train–test mismatch even with LoRA adaptation.The method’s whitening step addresses the local correlations introduced by rasterization.Source: §3.1 · p. 4

3.2 Preliminary: Feed-Forward Dynamic Gaussian Splatting

  • A feed-forward dynamic Gaussian Splatting model supplies time-varying 4D primitives with canonical anchors, and target-pose rendering produces conditioning video.An enclosing box provides explicit 3D surface points for background pixels outside reconstructed regions.Source: §3.2 · p. 4

3.3 Canonical 3D Noise Latent

  • Independent noise vectors attached to canonical primitive identities yield shared cross-view and temporal structure when rasterized, while an enclosing cubemap supplies noise for unobserved regions.Pixels viewing the same scene region depend on the same canonical noise vectors; nearby pixels may remain correlated because they share primitives.Source: §3.3 · p. 5

3.4 View-Consistent Whitening

  • A spatially varying projection indexed by canonical 3D position approximately whitens rasterized noise while retaining view and temporal correspondence.This is approximate, not exact covariance whitening; higher Fourier frequencies improve local decorrelation but increase sensitivity to reconstruction and rasterization errors.Source: §3.4 · pp. 6–7, Figure 2 · p. 6

3.5 Synchronized Video Generation

  • The diffusion model replaces i.i.d. Gaussian noise with the rasterized, whitened prior and is fine-tuned through LoRA while its pretrained backbone remains frozen.Source-video and rendered-video conditioning tokens provide context but are discarded before the output head and do not enter the training objective.Source: §3.5 · pp. 7–8

4 Experiments

Experiments evaluate SyncVid on curated real-world benchmark videos across synchronization, temporal consistency, camera accuracy, visual quality, and human preference, alongside ablations of its design choices. SyncVid leads on several synchronization and perceptual metrics, while the ablations favor whitening, joint source/rendered-view conditioning, LoRA 256, and the larger backbone.

4.1 Datasets

  • Training uses 36K synthetic MultiCamVideo pairs and 2K real-world DL3DV videos, and videos are standardized to 5 seconds at 15 fps and 480P.DL3DV’s static-scene videos are converted into pseudo-synchronized pairs by temporal folding, yielding about 13.5K pairs.Source: §4.1 · p. 8
  • The 200-clip Multi-ViewBench uses automatic clip screening and recorded judgments to create reproducible standard and complex splits; complex scenes have mean richness 71.2 versus 30.9.The benchmark is drawn from the Mixkit subset of Open-Sora-Plan; the standard split contains richest single-subject scenes and complex contains richest multi-subject scenes with cluttered geometry.Source: §4.1 · p. 8

4.3 Main results

  • SyncVid records 1,332 matched pixels in dynamic regions, 1,834 in static regions, FVD-V 146.99, CLIP-V 91.66, and CLIP-F 97.87; ArtiFixer leads camera accuracy.The static-region matched-pixel count is second-highest, behind Reshoot-Anything; SyncVid remains competitive on camera accuracy.Source: Table 1 · p. 9, §4.3 · pp. 9–10
  • VBench scores are highest for SyncVid on aesthetic quality (52.07), imaging quality (56.03), and subject consistency (88.66).Temporal-flickering and motion-smoothness scores are close to the best reported values; background consistency (91.24) is second only to Reshoot-Anything.Source: Table 2 · p. 9, §4.3 · p. 10

4.2 Metrics

  • Evaluation covers cross-view synchronization, temporal consistency, camera accuracy, six no-reference VBench scores, and blind pairwise preferences.Camera accuracy uses rotation and translation errors; the user study compares SyncVid with ArtiFixer and Reshoot-Anything on two disjoint sets of 25 scenes.Source: §4.2 · p. 9

4.4 Ablations

  • Whitening improves all four reported metrics over standard and unwhitened noise; joint source/rendered-view conditioning gives the lowest FVD-V (461.44).Source-view conditioning instead gives the highest Mat. Pix. (319), CLIP-V (91.41), and CLIP-F (97.95).Source: §4.4 · p. 10, Table 4 · p. 11
  • LoRA 256 is best across all four metrics, and the larger DiT improves almost every metric; the final model combines both choices with whitening and joint-view conditioning.The ablations use a held-out set of OpenSora videos and the lighter Wan2.2-5B backbone for computational affordability.Source: §4.4 · p. 10

5 Conclusion

SyncVid coordinates diffusion for synchronized novel-view video generation by attaching persistent noise to a canonical 4D representation and rasterizing it into target cameras. View-consistent whitening supports integration with a pretrained model through lightweight adaptation, without changing its backbone.

5 Conclusion

  • SyncVid coordinates diffusion for synchronized novel-view video generation by attaching persistent noise to a canonical 4D representation and rasterizing it into target cameras.The same stochastic ingredients follow content across viewpoints and time, while the model can synthesize newly revealed regions.Source: §5 · p. 11
  • View-consistent whitening keeps the structured prior compatible with the pretrained model's expected distribution.This enables lightweight adaptation without modifying the diffusion backbone.Source: §5 · p. 11
  • SyncVid makes scene correspondence a property of the diffusion prior rather than relying on image-plane warping or additional cross-view architecture.The paper concludes that synchronized generation can benefit from treating noise as a world-level representation.Source: §5 · p. 11

A.1 Automated Evaluation Set Curation for OpenSora

Multi-ViewBench is a 200-clip real-world evaluation set curated through recorded model judgments and deterministic decisions. The procedure is intended to make clip selection reproducible and limit bias in a setting without ground-truth target views.

A.1 Automated Evaluation Set Curation for OpenSora

  • Multi-ViewBench contains 200 clips and makes each accept-or-reject decision a recorded model judgment or a deterministic function of recorded judgments.The full path from corpus to final CSV can be replayed offline from the released judgments.Source: §A.1 · p. 14
  • Because real-video reshooting has no ground-truth target view, benchmark behavior can only be measured on the clips the set contains.The passage notes that hand-selected composition can bias results and hinder reproducibility.Source: §A.1 · p. 14
  • Curation uses two stages: a vision-language model judges each candidate once, then filtering, de-duplication, and ranking use only recorded fields.Stage 2 does not call a model again.Source: §A.1 · p. 14

A.1.1 Source corpus

The source corpus is the Mixkit subset of Open-Sora-Plan v1.1.0, whose overlapping category folders and heterogeneous footage shape the curation pipeline.

A.1.1 Source corpus

  • The Mixkit subset contains 9,194 stock-footage clips filed under 40 topical category folders.The categories include forest, Street, Couple, and Bicycle.Source: §A.1.1 · p. 14
  • Category folders are editorial tags rather than a partition, so the same physical clip can appear in several categories.The corpus therefore contains substantially fewer distinct videos than files.Source: §A.1.1 · p. 14
  • The footage mixes locked-off tripod shots, time-lapses, portrait clips, and heavy optical effects with the moving-camera material sought for camera-controlled generation.Source: §A.1.1 · p. 14

A.1.2 Stage 1: structured vision-language judgement

Stage 1 records structured vision-language judgments for each candidate clip. The schema captures admissibility, composition, semantic informativeness, and the model's reasoning.

A.1.2 Stage 1: structured vision-language judgement

  • Each clip is judged independently using 12 uniformly spaced frames, rescaled to at most 512 pixels on the longest side, with a fixed prompt at temperature 0.2.Responses follow a JSON schema, and one JSON file is written per clip so an interrupted pass can resume.Source: §A.1.2 · p. 14
  • Six boolean gates encode hard admissibility requirements, while four semantic scores rate properties that make a clip informative for view synthesis.Source: §A.1.2 · p. 14
  • A composition bucket assigns each clip to the simple or complex split: simple denotes one subject with smooth tracking and a clean background, while complex includes multiple subjects, clutter, depth layers, or dynamic backgrounds.Source: §A.1.2 · p. 14
  • Three free-text reasoning fields record why the model answered as it did; Stage 2 consults them in one narrow place.The reasoning fields are released.Source: §A.1.2 · p. 14

A.1.3 The richness index

The richness index orders clips by how strongly they exercise synchronized generation. It combines semantic indicators of viewpoint-dependent geometry and nonrigid motion, with a product form that prevents a low factor from being offset by a high one.

A.1.3 The richness index

  • The index combines geometric parallax, occlusion potential, camera-motion weighting, and nonrigid-motion weighting into a richness score from 1 to 100.Geometric parallax and occlusion potential are scored from 1 to 5.Source: §A.1.3 · p. 15
  • Camera-motion weights are 2.0 for dynamic, 1.2 for optical-effect, and 1.0 for static labels; nonrigid-motion weights range from 1.0 for none to 2.0 for high.The nonrigid-motion weights are 1.0, 1.2, 1.5, and 2.0 for none, low, mid, and high, respectively.Source: §A.1.3 · p. 15
  • The product form penalizes clips with weak factors because depth without occlusion, or occlusion without camera motion, is considered uninformative for synchronized generation.Moving-camera clips are required at Stage 2, so the camera-motion weight is 2.0 throughout the final pool.Source: §A.1.3 · p. 15

A.1.4 Stage 2: deterministic filtering

Stage 2 applies five filters in a fixed order to reduce the scored corpus to a candidate pool. The filters enforce admissibility and landscape format, deduplicate clips, and remove clips without genuine viewpoint motion or with lens-driven apparent motion.

A.1.4 Stage 2: deterministic filtering

  • The five Stage 2 filters run in a fixed order, and Table 6 reports how many clips survive each step.At benchmark freeze, 7,561 of 9,194 corpus clips had been scored.Source: §A.1.4 · p. 15, Table 6 · p. 16
  • Clips must pass all six gates and have frame width greater than height; the landscape check is the only filter that reads pixels.Frame dimensions are cached so reruns do not repeat the pixel check.Source: §A.1.4 · p. 15
  • Deduplication keeps one row per unique filename, selecting the highest-richness copy and breaking ties deterministically by category and judgement filename.Discarded copies' categories are retained in an alias_categories field.Source: §A.1.4 · pp. 15–16
  • Static-source removal retains physically moving cameras, using reasoning text only to adjudicate ambiguous optical-effect labels or demote a dynamic clip explicitly described as stationary.Unambiguous labels are not overruled by the reasoning text.Source: §A.1.4 · p. 16
  • Any remaining clip labeled afocal_effects, aperture_effects, or sensor_effects is removed because lens-driven apparent motion is not viewpoint change.Source: §A.1.4 · p. 16

A.1.5 Why the filter order matters

This section details benchmark curation, camera-trajectory synthesis, experimental setup, and theoretical scope for whitening and cross-view noise consistency. It also covers pixel-correspondence evaluation, qualitative reshoots, reconstruction-related failure modes, and a camera-pose estimation gap.

A.1.5 Why the filter order matters

  • De-duplication follows the gates so rejected copies cannot re-enter and precedes static-source classification so each physical clip gets one verdict; static removal also precedes optical-effect filtering.The filters overlap because adjudication already reclassifies most optical-effect clips as static, leaving only four for the explicit optical filter; reversing the order could discard optical-effect clips with real camera motion.Source: §A.1.5 · p. 16

A.1.6 Constructing the two splits

  • The standard split takes the 100 richest simple clips; the complex split first uses max-min water-filling to balance 1,000 clips across 40 categories, then selects the 100 richest.The complex final 100 spans 19 categories, so balancing makes its candidate pool category-fair but does not make the released split uniform.Source: §A.1.6 · p. 17

A.1.7 Resulting statistics

  • Every complex clip scores at least 64, while only one standard clip reaches 64; complex clips also have high non-rigid motion and parallax ratings of at least 4.The splits are disjoint, contain 100 clips each, and together form Multi-ViewBench; the standard split has a mean richness score of 30.9 and is concentrated at parallax 2–3.Source: §A.1.7 · p. 17, Table 7 · p. 18

A.1.8 Reproducibility and release

  • The release provides the curation trail and frozen CSVs, allowing the benchmark to be audited and the splits to be recovered exactly.It includes code and prompts, per-clip judgments, pool and split CSVs, clip identities and source paths, and a SHA-256 manifest; deterministic Stage 2 is seed-free, and splits were fixed before baseline evaluation.Source: §A.1.8 · pp. 17–18

A.2 Novel-Trajectory Synthesis

  • Novel trajectories compose time-varying local SE(3) offsets after the moving source pose, begin aligned with it, and diverge across six arc, dolly, and pedestal-tilt variants.Scene depths set trajectory magnitudes, with median scene distance as fallback; this co-moving design avoids drift from anchoring offsets to a fixed world pose when source footage is handheld or drone footage.Source: §A.2 · pp. 18–19

A.3 Local Decorrelation of the Whitening Projection

  • Under the generic-bases assumption, distinct canonical anchors yield near-orthogonal projection bases, suppressing cross-pixel correlation while each pixel’s self-covariance remains exactly identity.The argument is approximate for nearby anchors relative to the Fourier wavelengths and can fail at blended occlusion boundaries; it does not prove exact whitening of the full rasterized field, and the bases are deterministic in practice.Source: §A.3 · pp. 20–21

A.4 Noise Rasterization and View Consistency

  • When one primitive dominates each corresponding pixel, any two same-time camera poses rasterize that primitive’s identical noise, without requiring a relationship between the poses.Pure rotation additionally gives a depth-independent pixel correspondence, unlike translation; blending at occlusion boundaries can break the guarantee, which concerns noise construction rather than generated-frame consistency.Source: §A.4 · pp. 21–22

A.5 Experimental Details

  • SyncVid uses a Wan2.2-14B backbone trained for 10K iterations, with 50 denoising steps at inference and a NoPo4D representation plus a finite cubemap proxy for unobserved regions.The effective training batch size is 32; the cubemap uses six 1024×1024 faces with three channels, and 4D representation construction takes 4.18 seconds per input clip and can be amortized across reshoots.Source: §A.5 · p. 23
  • Pixel matching uses RoMA-v2 [25] correspondences with uncertainty below 0.5, averaged separately for dynamic and static regions.SAM3 [26] provides a source-video segmentation mask to partition correspondences for each source–target frame pair.Source: §A.5 · p. 23

A.6 Qualitative Comparison

  • Qualitative comparisons show six equally spaced frames from source clips and generated reshoots under Dolly-out, Arc-left, and Arc-right trajectories.The supplement points readers to an HTML viewer for more detailed comparisons.Source: §A.6 · p. 23, Figure 5 · p. 25, Figure 6 · p. 26

A.7 Limitations and Failure Examples

  • SyncVid remains limited by monocular reconstruction errors, which can cause subject deformation or temporal drift, while weakly constrained disocclusions may be hallucinated.The examples include severe foreground inconsistencies in an Arc-right reshoot and unrealistic newly exposed content in a Dolly-out reshoot, despite retaining the main subject and setting.Source: §A.7 · p. 24, Figure 8 · p. 28
  • SyncVid does not outperform state-of-the-art methods in camera-pose estimation, despite excelling on synchronization metrics and preserving dynamic regions.The authors speculate that the gap may partly reflect the absence of explicit camera-pose conditioning, such as ArtiFixer’s Plücker embeddings; adding it is left for future work as orthogonal to structured noise.Source: §A.7 · p. 24
Loading 2610.31765v1…