PaperlayerView exact revision on arXiv

Source-linked AI summary

SymRegFlow: Symmetry-Regularized Flow Matching for Video World Models

Xi Ye, Yuzhu Wang, Xiaoyang Liu, Jiayi Wang, Yangyang Xu, Ruyu Wang, Wenlin Chen, Duo Su, Jun Zhu

arXiv:2610.02726v1cs.CV

TL;DR

Multi-view video world models are commonly limited to fixed camera rigs, and paired video targets for continuous viewpoints are unavailable in fixed-camera datasets. SymRegFlow trains continuous-view generation from noisy warped anchors using masked dual-anchor supervision and cross-anchor consistency; on nuScenes, it achieves the lowest FVD and FVMD among evaluated baselines.

  • Problem

    Multi-view world models support finite training-rig viewpoints, while fixed-camera datasets leave intermediate viewpoints and novel camera heights or baselines without paired video targets.

    Source: §1 · p. 1

  • Method

    SymRegFlow warps observed views into noisy novel-view anchors and combines masked dual-anchor supervision with cross-anchor denoising consistency while retaining observed-view supervision.

    Source: §1 · pp. 1–2

  • Results

    On nuScenes, SymRegFlow achieves the lowest FVD and FVMD among evaluated baselines, over 31% lower FVD than the best baseline; source-conditioned inference has the best FID and instance preservation.

    Source: §1 · p. 2, §5 · p. 10

  • Takeaways & Limitations

    The reported results support continuous-view video generation without ground-truth novel-view RGB supervision on nuScenes.

    Source: §1 · p. 2

    Performance depends on anchor quality: depth, projection, and visibility errors can corrupt supervision, especially near occlusions or under large viewpoint changes.

    Source: §5 · p. 10

Abstract

from arXiv · show

Flow-matching-based multi-view world models generate realistic videos, but are commonly restricted to fixed camera rigs. Extending them to continuously varying camera poses requires paired pose--video observations with dense pose coverage, which are costly to acquire. We introduce \emph{SymRegFlow}, a symmetry-regularized flow-matching framework for multi-view-consistent video generation across continuous viewpoints without ground-truth novel-view RGB supervision. For each target pose, SymRegFlow geometrically warps source views into noisy anchors and combines masked dual-anchor supervision with cross-anchor denoising-output consistency to mitigate anchor-specific errors. Under an affine Gaussian surrogate, we prove that suitable consistency regularization recovers the clean-reference optimum at fixed noise levels, strictly outperforming single- and merged-anchor baselines. Experiments on Cosmos-Drive-Dreams and nuScenes demonstrate high-quality, multi-view-consistent autonomous-driving video generation: on nuScenes, SymRegFlow achieves the lowest FVD and FVMD among the evaluated baselines, reducing FVD by over 31\% relative to the best baseline, and source-conditioned inference also attains the best FID and instance preservation.

1 INTRODUCTION

Existing multi-view world models generate only from the finite viewpoints represented by their training rigs, leaving continuous viewpoints without paired video targets. SymRegFlow uses noisy warped anchors and cross-anchor consistency to train continuous-view generation without ground-truth novel-view RGB supervision.

1 INTRODUCTION

  • Existing models support only finite training-rig viewpoints, while fixed-camera datasets lack paired targets for intermediate viewpoints and novel camera heights or baselines.Unreal Engine multi-camera data offers synchronized views but incurs high generation costs, a sim-to-real gap, and unmodeled simulator artifacts.Source: §1 · p. 1
  • SymRegFlow warps observed source views to sampled novel poses as noisy anchors, then combines masked dual-anchor supervision with cross-anchor denoising consistency.The anchors can contain geometry errors, occlusions, and resampling artifacts; observed-view supervision is retained to preserve pretrained generative capabilities.Source: §1 · pp. 1–2
  • The framework is a fine-tuning objective that retains flow-matching training and inference while adding continuous-view conditioning, directed cross-view attention, and source-latent replacement for source-conditioned inference.It does not require ground-truth novel-view videos.Source: §1 · p. 2
  • On nuScenes, SymRegFlow achieves the lowest FVD and FVMD among evaluated baselines, with over 31% lower FVD than the best baseline; source-conditioned inference also has the best FID and instance preservation.The contribution summary reports these benchmark comparisons.Source: §1 · p. 2

2 METHOD

SymRegFlow fine-tunes a pretrained multi-view flow-matching model for continuous novel poses using geometry-warped anchors and cross-anchor consistency. A toy study and a Gaussian surrogate analyze its performance and the conditions under which regularization recovers the clean-reference optimum.

2.1 PRELIMINARIES

  • Camera-pose equivariance preserves scene content while RGB observations reproject, and flow matching learns denoising predictions from noisy multi-view inputs.The latent forward path interpolates between clean latents and Gaussian noise; predicted clean latents are decoded into video.Source: §2.1 · p. 2

2.2 SYMREGFLOW FROM NOISY MULTI-VIEW ANCHORS

  • Each source view is lifted and rendered at the requested pose to form a partial anchor with a visibility/confidence mask; warping errors generally grow with viewpoint distance.The anchors can be noisy and incomplete because of geometry errors, scene dynamics, and occlusions.Source: §2.2 · p. 3
  • Training combines observed-view and masked anchor supervision with cross-anchor denoising consistency over the union of anchor support.The two branches share noise, noise level, condition, and observed-view inputs; at inference, the requested pose is specified without anchors or warping, while DCVA protects observed-view fidelity.Source: §2.2 · pp. 3–4

2.3 SE(2) TOY EXAMPLE

  • In the SE(2) toy study, SymRegFlow achieves the lowest FD and highest SSIM at all nine evaluated poses and five distortion levels.It also outperforms Merged anchor at every distortion level, with mean FD advantages of 0.254 at Eb = 0.2 and 0.434 at Eb = 0.8.Source: §2.3 · p. 5

2.4 WHY SYMREGFLOW CAN IMPROVE NOISY-ANCHOR FINE-TUNING

  • Under a scalar Gaussian surrogate, suitable consistency regularization recovers the clean-reference optimum pointwise in noise level and improves on single-anchor and fixed convex-merge optima.The analysis assumes independent centered Gaussian anchor errors and shared branch noise.Source: §2.4 · p. 5

2.5 EXTENSION TO LATENT FLOW MATCHING

  • For latent flow matching, the method warps source RGB before encoding and uses the resulting latent anchors with downsampled visibility masks and the same three loss components.The video implementation retains supervised velocity losses and evaluates consistency on clean predictions.Source: §2.5 · p. 5

3 EXPERIMENTS

Experiments test continuous novel-view generation on Cosmos-Drive-Dreams and midpoint-view quality and consistency on nuScenes, then ablate anchor strategy, consistency weight, and attention design.

3.1 QUALITATIVE GENERATION ON COSMOS-DRIVE-DREAMS

  • SymRegFlow generates continuously specified novel views with coherent appearance across views and frames, while different seeds yield varied background trees and cloud patterns.The top-row novel views use separate inference runs with the same seed and scene conditioning; the bottom rows compare seeds 0 and 42.Source: §3.1 · p. 6, Figure 3 · p. 6

3.2 QUANTITATIVE COMPARISON ON NUSCENES

  • 158.62 FVD and 2519.17 FVMD are the joint-generated mode’s lowest scores among evaluated baselines; source-conditioned inference reaches the highest instance-preservation score, 99.44.Both inference modes outperform all baselines in FVD and FVMD; the table evaluates midpoint views across 30 nuScenes validation scenes.Source: §3.2 · p. 6, Table 1 · p. 7
  • SymRegFlow produces more uniform nuScenes frames and better preserves thin structures than baselines that show source-camera color discontinuities and distortions.The reported examples include street lamps; the authors describe the result as consistent with cross-anchor consistency regularization.Source: §3.2 · p. 7, Figure 4 · p. 7

3.3 ABLATION STUDIES

  • 545.14 novel-view FVD for C0 improves on AL’s 562.23, while C0.05’s 498.94 improves on merged-anchor AM’s 704.32.AM also exhibits artifacts from mismatched RGB merging, including misaligned overlaps and residual gray pixels where inpainting fails.Source: §3.3 · p. 8, Table 2 · p. 8
  • AL and AR copy anchor-visible content into missing regions, but SymRegFlow avoids both duplicated content and AM’s fusion seam and geometric discontinuity.In the controlled example, queries on duplicated workers attend to source-supported counterparts at 13.82× and 12.22× uniform attention, respectively.Source: §3.3 · p. 8, Figure 5 · p. 8
  • λc = 0.05 lowers MEt3R to 0.2660 and novel-view FVD to 498.94; larger weights worsen FVD, reaching 701.95 at λc = 2.The 0.05 setting also maximizes both VBench consistency scores.Source: §3.3 · p. 9
  • Replacing DCVA with full attention raises existing-view FVD by 24.85% and worsens novel-view FVD, while lowering MEt3R by 24.96% and requiring 1.47× the backbone forward/backward time.The VBench scores remain similar; the authors characterize DCVA as trading some cross-view agreement for efficiency and view fidelity.Source: §3.3 · p. 9

4 RELATED WORK

Prior work imposes symmetry through specialized equivariant architectures or training and sampling schemes, while camera-control methods trade among paired-data training, inference-time control, and rig-bound joint generation. SymRegFlow instead adds symmetry through fine-tuning while retaining flow matching and supporting continuously pose-conditioned targets.

4.1 SYMMETRY AND EQUIVARIANCE IN WORLD MODELS

  • Equivariant denoisers, vector fields, and latent memories encode symmetry in model structure, while architecture-agnostic alternatives use group-averaged targets or symmetry-structured noise.SymRegFlow instead aligns denoising outputs from independently warped noisy anchors over their union of valid supports through its fine-tuning objective.Source: §4.1 · p. 9

4.2 CAMERA-CONTROLLED VIDEO WORLD MODELS

  • Training-free camera-control methods avoid paired training data but typically modify sampling or refine trajectories; SymRegFlow fine-tunes and retains the standard sampler for arbitrary target poses.Source-conditioned inference additionally uses source-latent replacement.Source: §4.2 · p. 9
  • Pose-annotated or synchronized-pair methods amortize camera control, while ReCapture fits trajectory-specific LoRAs without coupling anchors or enforcing cross-view symmetry; SymRegFlow uses dataset-level fine-tuning and anchor agreement.Joint-view generators remain tied to training-rig slots and poses, whereas SymRegFlow introduces a continuously pose-conditioned target slot without novel-view RGB supervision.Source: §4.2 · p. 9

5 CONCLUSION AND DISCUSSION

SymRegFlow combines masked dual-anchor supervision with cross-anchor denoising consistency to generate multi-view-consistent video across continuous viewpoints without novel-view RGB supervision. On nuScenes it leads evaluated baselines on FVD and FVMD, while its guarantees and remaining limitations depend on anchor quality and an affine Gaussian surrogate.

5 CONCLUSION AND DISCUSSION

  • Masked dual-anchor supervision and cross-anchor denoising consistency mitigate source-dependent biases without requiring novel-view RGB supervision.Under an affine Gaussian surrogate, suitable regularization recovers the clean-reference optimum at each noise level.Source: §5 · pp. 9–10
  • On nuScenes, SymRegFlow achieves the lowest FVD and FVMD among evaluated baselines; source-conditioned inference also has the best FID and instance preservation.Ablations also confirm improved novel-view quality and reduced source-dependent attention bias.Source: §5 · p. 10
  • Depth, projection, and visibility errors can corrupt anchors, especially near occlusions or under large viewpoint changes; shared errors and jointly unseen regions remain uncorrected.Better anchors and geometric-uncertainty modeling remain open directions.Source: §5 · p. 10

A TRAINING AND INFERENCE ALGORITHMS

Training prepares warped anchor views and masks, then optimizes masked supervision and cross-anchor consistency with shared noise. Inference supports both source-conditioned novel-view synthesis and joint three-view generation.

A TRAINING AND INFERENCE ALGORITHMS

  • Training encodes source clips and warped anchors, builds latent masks, and samples one shared noise, time, and condition for both anchor passes.The loss combines source supervision, masked anchor supervision, and clean-prediction consistency; updates train LoRA and the view embedder.Source: A TRAINING AND INFERENCE ALGORITHMS · p. 14
  • The video implementation retains velocity prediction, equivalent to clean-prediction supervision weighted by ω(t) = t−2; consistency is computed on clean predictions without that factor.Both passes reuse the same conditioning dropout.Source: A TRAINING AND INFERENCE ALGORITHMS · p. 14
  • DCVA restricts source queries to source keys while novel queries attend to all three views; its four-stream implementation preserves the two-pass objective and gradients.The implementation uses source indices S = {1, 2} and slot order (1, n, 2).Source: A TRAINING AND INFERENCE ALGORITHMS · p. 14
  • Source-conditioned inference replaces source inputs during the first Nrep steps, then relies on the scheduler-maintained state; the novel slot follows one sampling trajectory.The reported source-conditioned results use Nrep = 30 and N = 35.Source: A TRAINING AND INFERENCE ALGORITHMS · p. 15
  • Joint-generated inference initializes and denoises three views from noise without observed RGB inputs or source-slot substitution.Both sampling modes use fine-tuned models with DCVA and no anchors or geometric warping.Source: A TRAINING AND INFERENCE ALGORITHMS · p. 15

B PROOF AND NUMERICAL ILLUSTRATION OF PROPOSITION 1

For the stated affine Gaussian surrogate, suitable consistency strength recovers the clean-reference optimum at a fixed noise level, while excess risk rises when regularization is too strong. The guarantee is scoped to unit masks and fixed source predictions, and the numerical illustration evaluates the surrogate formulas.

B PROOF AND NUMERICAL ILLUSTRATION OF PROPOSITION 1

  • At λ⋆c, SymRegFlow reaches the clean-reference predictor and zero excess risk under the affine Gaussian surrogate.The result assumes unit masks and fixed source predictions; it does not establish the same guarantee for the full neural network or one constant λc across diffusion times.Source: B PROOF AND NUMERICAL ILLUSTRATION OF… · pp. 16–17
  • Excess risk strictly decreases below λ⋆c and strictly increases above it as the optimizer passes through the clean-reference predictor.Zero excess risk means attaining minimum total risk q/D(t), not zero reconstruction error.Source: B PROOF AND NUMERICAL ILLUSTRATION OF… · p. 17
  • Consistency penalizes sensitivity to anchor disagreement, countering the inflated anchor variance q + ν fitted by supervision alone.The two anchor branches use the same diffusion noise, canceling the shared clean sample and noise in their difference.Source: B PROOF AND NUMERICAL ILLUSTRATION OF… · pp. 16–17
  • The proof holds for independent affine errors with unit masks and fixed source predictions; it does not compare union and intersection masking choices.The analytic comparison is on clean-reference risk, not the numerical values of different training objectives.Source: B PROOF AND NUMERICAL ILLUSTRATION OF… · pp. 16–17
  • In the numerical illustration, λ⋆c = 4 and SymRegFlow strictly improves on all three baselines in a neighborhood of that strength.The figure plots clean-reference excess risk for α ∈ {0.35, 0.50, 0.65}; these are surrogate-formula evaluations, not additional MNIST measurements.Source: Figure 6 · p. 17, B PROOF AND NUMERICAL ILLUSTRATION OF… · p. 17

C.1 IMPLEMENTATION DETAILS

The video experiments adapt Cosmos-Transfer2.5 with continuous-view conditioning and evaluate generated novel views using distributional, consistency, motion, and instance-preservation measures. The protocols include both a 30-scene comparison and a four-scene ablation.

C.1 IMPLEMENTATION DETAILS

  • Video experiments use Cosmos-Transfer2.5 with 29-frame, 1280×720 clips, rank-96 LoRA, gray fill for invalid warp regions, and default DCVA with λc = 0.05.Quantitative nuScenes runs use 16 GPUs, global batch size 32, and checkpoints evaluated at step 2,048.Source: §C.1 · p. 18
  • Novel-view pose conditioning renders map polylines and 3D object boxes at target poses; continuous view embeddings interpolate endpoint camera identities using a ring coordinate.This evaluated interpolation design can be replaced by other pose representations, including camera matrices or Plücker rays.Source: §C.1 · p. 18
  • Novel-view FID and FVD compare generated clips with unpaired captured source-view distributions because captured midpoint RGB is unavailable; world-scenario renderings are not RGB ground truth.FID uses 2,048-dimensional Inception features and FVD uses 400-dimensional I3D features.Source: §C.1 · p. 18

C.2 EVALUATION PROTOCOLS

  • Table 1 pools eight adapters over 30 nuScenes scenes and evaluates 720 clips per distribution for FVD and FID; FVMD and instance preservation use 180 sequences per model.MEt3R uses generated existing views for joint-generated outputs and captured views for other rows, measuring internal consistency and agreement with observations, respectively.Source: §C.2 · pp. 18–19
  • The four-scene ablation trains one adapter on scenes 0012–0015 and evaluates 96 novel clips, with paired captured endpoints used for existing-view metrics.This ablation uses joint-generated inference only for MEt3R comparisons against generated endpoints.Source: §C.2 · p. 19

C.3 ADDITIONAL ABLATION RESULTS

C0.05 leads the consistency-weight sweep on the five-metric hybrid profile, while a reduced-resolution probe compares DCVA and full-attention backbone costs. The analysis also specifies the checkpoint and variants used.

C.3 ADDITIONAL ABLATION RESULTS

  • All 11 reported variants were evaluated at step 2,048; the radar displays the seven Cγ sweep models, while the configured C0.001 run was omitted for lacking checkpoint outputs.AL and AR use individual anchors, AM merges both anchors, and CFA replaces DCVA with full attention at λc = 0.05.Source: §C.3 · p. 19, Figure 7 · p. 20
  • 92.4 was C0.05’s largest mean hybrid profile in the consistency-weight sweep, ahead of C0 at 71.3 and C0.1 at 61.2.The five equally weighted spokes summarize existing-view FVD and four novel-view measures; the mean is descriptive, not polygon area or a replacement for individual metrics.Source: §C.3 · p. 19, Figure 7 · p. 20
  • DCVA took 3.07 seconds versus 4.51 seconds for full attention (1.47×) in a reduced-resolution forward/backward probe.The probe excludes several components, including the control network, VAE, data loading, optimizer, and distributed communication, so it is not an end-to-end training benchmark.Source: §C.3 · p. 20

C.4 ATTENTION-BIAS ANALYSIS

In a controlled one-scene diagnostic, SymRegFlow reduces source-aligned cross-view route imbalance and the single-anchor preference for visible source regions. The section also describes a continuous camera embedding that preserves pretrained endpoint embeddings.

C.4 ATTENTION-BIAS ANALYSIS

  • SymRegFlow reduces the cross-view route gap to 0.307, a 57.8% reduction from AL’s 0.729 and 64.5% from AR’s 0.865.The single-anchor preferences align with their fine-tuning anchors; the reported diagnostic averages three relative-offset axes.Source: Table 3 · p. 20, §C.4 · p. 20
  • Figure 8 profiles show that single-anchor adaptation raises compatibility toward its fine-tuning source, while SymRegFlow brings the novel-to-left and novel-to-right routes closer.The profiles report post-RoPE, pre-softmax QK compatibility over relative latent width, height, and time offsets.Source: Figure 8 · p. 21, §C.4 · p. 21
  • Figure 9’s signed zero-offset route gap reverses with the selected single anchor, whereas SymRegFlow remains closer to zero throughout denoising.These attention diagnostics support a routing-bias explanation but do not by themselves causally establish the repeated-content artifact.Source: Figure 9 · p. 21, §C.4 · p. 21
  • AL and AR assign 20.6% and 11.4% more invisible-to-visible attention mass, respectively, than SymRegFlow under the corresponding paired visibility masks.This within-frame preference is consistent with cross-view routing bias toward source-supported content.Source: Table 3 · p. 20, §C.4 · p. 21
  • Figure 5 marks invisible-region queries in red and their strongest visible-anchor attention matches in cyan; AM and SymRegFlow are clean RGB controls without attention measurements.The panels show generated frame 28 at denoising step 17 and DiT block 20.Source: §C.4 · p. 22

C.5 GLOBAL CONTINUOUS NOVEL-VIEW ENCODING

  • The continuous-view encoding places the six rig cameras on a fixed circle and combines pretrained endpoint embeddings with a learned residual that vanishes at both endpoints.The interpolation factor is disambiguated by its position on the rig rather than represented in a pair-local space.Source: §C.5 · p. 22

D ADDITIONAL SE(2) TOY STUDY

In the toy study, exact-target FD selects λc = 2, although pseudo-target metrics continue improving through λc = 4. The study compares single- and dual-anchor variants under controlled anchor corruption and evaluation conditions.

D ADDITIONAL SE(2) TOY STUDY

  • All continuous-pose variants start from a shared paired-MNIST checkpoint and are fine-tuned for 2,000 iterations; λc is selected as 2 by mean exact-target FD.The sweep uses λc ∈ {0, 0.1, 0.25, 0.5, 1, 2, 4} over 45 pose–distortion cells.Source: D ADDITIONAL SE(2) TOY STUDY · p. 22
  • The A1-only and A2-only baselines use their respective single pseudo-examples, while Merged anchor uses the mask-weighted fusion of both anchors.SymRegFlow uses both pseudo-examples for paired forward passes.Source: D ADDITIONAL SE(2) TOY STUDY · p. 22
  • 2.213 was the minimum exact-target FD at λc = 2, down from 2.754 at λc = 0 and below 2.265 at λc = 4.PSNR and SSIM on exact targets continued increasing through λc = 4, reaching 15.79 and 0.382; pseudo-target metrics also improved through λc = 4.Source: Figure 10 · p. 23, D ADDITIONAL SE(2) TOY STUDY · p. 23
  • Anchor errors increase with source–target pose distance and include heterogeneous morphology, scale, shear, angle, translation, smooth-grid, and pixel perturbations.Confidence and loss masks decrease with distance; the two anchors are not identical or independent in every pixel.Source: D ADDITIONAL SE(2) TOY STUDY · p. 23
  • Evaluation covers nine interior poses and five distortion levels, with 256 samples per cell and shared Gaussian samples across methods and poses.Each method–distortion setting uses one fine-tuning seed, so variation across distortion levels measures robustness to anchor quality rather than optimization uncertainty.Source: D ADDITIONAL SE(2) TOY STUDY · p. 23

E ADDITIONAL NUSCENES QUALITATIVE EXAMPLES

The appendix adds qualitative novel-view comparisons on nuScenes across several scenes and camera pairs. The panels show the evaluated methods and generated-video frames without reporting a quantitative comparison.

E ADDITIONAL NUSCENES QUALITATIVE EXAMPLES

  • The qualitative panels place DiST-4D, OmniRe, GEN3C, FreeVS, and SymRegFlow in rows, with generated-video frames in columns.The comparisons are presented as novel-view examples rather than quantitative results.Source: E ADDITIONAL NUSCENES QUALITATIVE… · p. 24
  • Scenes 0012 and 0013 compare novel views between the front and front-right cameras.The captions identify the scene and camera pair but state no comparative outcome.Source: Figure 11 · p. 24, Figure 12 · p. 24
  • Scene 0014 includes front-to-front-right and back-to-back-left novel-view comparisons.These captions identify additional camera pairs without stating a winner.Source: Figure 13 · p. 25, Figure 14 · p. 25

Related papers

Loading 2610.02726v1…