PaperlayerView exact revision on arXiv

Source-linked AI summary

MVTrack4Gen: Multi-View Point Tracking as Geometric Supervision for 4D Video Generation

JoungBin Lee, Jaewoo Jung, Jongmin Lee, Tongmin Kim, Hyunsung Kim, Takuya Narihira, Kazumi Fukuda, Jahyeok Koo, Jisang Han, Yuki Mitsufuji, Seungryong Kim

arXiv:2606.26087v1cs.CV

TL;DR

Novel-view video generation must preserve camera control, scene geometry, motion, and visual quality, while existing approaches trade off these requirements. MVTrack4Gen supervises camera-conditioning-only diffusion models with multi-view point tracking and correspondence objectives, achieving state-of-the-art geometric consistency and competitive camera accuracy across benchmarks.

  • Problem

    Novel-view video generation must jointly preserve camera control, geometric consistency, motion consistency, and photorealistic quality when changing viewpoint from monocular video.

  • Method

    MVTrack4Gen jointly trains camera-conditioning-only diffusion models with multi-view point tracking and attention-level correspondence supervision to strengthen cross-view geometric and temporal motion correspondences.

  • Results

    Across benchmarks, MVTrack4Gen achieves state-of-the-art geometric consistency and competitive camera accuracy, while improving visual and motion consistency across views.

  • Takeaways & Limitations

    Multi-view tracking supervision can strengthen correspondence features in camera-conditioning-only diffusion models without requiring explicit 3D reconstruction at inference.

  • Takeaways & Limitations

    The method is limited to 480×832 resolution and 81-frame sequences and requires ground-truth multi-view point correspondences during training.

Abstract

from arXiv · show

Synthesizing a novel-view video from a monocular reference video along a target camera trajectory requires both geometric consistency and motion fidelity with respect to the reference video. Existing methods based on explicit 3D representations are limited by the accuracy of off-the-shelf reconstruction modules, which often produce inaccurate geometry for dynamic objects in monocular videos. In contrast, camera-conditioning-only methods can achieve high visual quality but often struggle to preserve geometric and motion consistency. In this work, we introduce MVTrack4Gen (Multi-View point Tracking for Novel-View Generation), a motion-aware training framework that leverages multi-view point tracking as an additional geometric and motion supervision signal for camera-conditioning-only novel-view video diffusion models. Our key finding is that specific attention layers encode strong correspondence cues, where query features attend to key features at geometrically corresponding locations across views and over time, and the misalignment of these correspondences causes motion inconsistency. Based on this observation, we route these features into an auxiliary multi-view tracking head and jointly train the diffusion model with a point-tracking objective. By explicitly strengthening these motion-aware correspondences, MVTrack4Gen improves existing models to better follow the motion in the reference view and maintain cross-view geometric consistency. Across diverse benchmarks, our method achieves state-of-the-art geometric consistency and competitive camera accuracy.

1 Introduction

MVTrack4Gen improves camera-conditioning-only novel-view video generation by supervising attention-based multi-view correspondences with point-tracking objectives. Applied to ReCamMaster and Redirector on DAVIS and iPhone, it improves both backbones and achieves strong visual quality, geometric consistency, and camera accuracy.

  • Introduction: Novel-view video generation must preserve reference-scene geometry across synthesized views while accurately following a user-specified camera trajectory.The task targets applications including virtual cinematography, robotics, and immersive AR/VR.
  • Introduction: Camera-conditioning-only methods avoid error-prone explicit 3D reconstruction by jointly processing reference videos and target camera trajectories within diffusion models.They inject trajectories through camera embeddings such as Plücker coordinates and can be trained on multi-view video pairs.
  • Introduction: Attention-map analysis shows that incorrect cross-view correspondences in dominant layers are directly associated with geometric and motion inconsistencies.The identified correspondence errors occur in regions where dynamic objects exhibit geometric or motion inconsistencies.
  • Introduction: MVTrack4Gen adds ground-truth multi-view point tracks as auxiliary supervision to strengthen motion-aware correspondences without reintroducing explicit 3D conditioning.It builds a tracking head on local 4D correlation volumes from selected attention-layer query and key features, jointly training it with the diffusion model and a multi-view correspondence loss.
  • Introduction: On DAVIS and iPhone, MVTrack4Gen consistently improves ReCamMaster and Redirector, attaining best scores on most VBench visual-quality metrics and state-of-the-art geometric consistency.Its camera accuracy is comparable to reconstruction-based and camera-conditioning-only baselines.

2 Related Work

Prior novel-view video generation methods either reconstruct explicit 3D representations or rely only on camera conditioning and video diffusion priors. Point-tracking research has progressed toward occlusion-aware, temporally refined, and multi-view geometrically consistent correspondences, motivating their use for supervision.

  • Novel-View Generation Using Explicit 3D Representation: Explicit-3D approaches estimate geometric cues from the input video and condition video diffusion models to synthesize newly visible regions from target viewpoints.The passage characterizes this as one major line of novel-view generation work.
  • Novel-View Generation Using Camera Conditioning Only: Camera-conditioning-only methods bypass explicit geometry, using camera poses and video diffusion priors to implicitly reason about 3D structure.GCD is identified as an early camera-controlled dynamic novel-view method trained on the multi-view synthetic Kubric video dataset.
  • Point Tracking: Point tracking generalizes optical flow to long-term pixel correspondences while explicitly handling occlusion, with later methods improving temporal refinement and joint tracking.TAPIR combines global matching-based initialization with temporal refinement, while CoTracker jointly tracks points using a transformer with cross-track attention.
  • Point Tracking: Multi-view trackers aggregate information across views or fuse multi-view features to produce geometrically consistent correspondences beyond monocular tracking, motivating auxiliary supervision for video diffusion models.MV-TAP estimates robust 2D trajectories through cross-view attention, whereas MVTracker targets feed-forward 3D point tracking.

3 Preliminaries: Video Diffusion Transformer for Novel-View Generation

This section introduces camera-conditioning-only novel-view video generation, which synthesizes a target-view video from a monocular reference using a VAE and Diffusion Transformer. The concatenated reference and target latents enable 3D attention to capture temporal and cross-view correspondences, with training performed via flow matching.

  • Camera-conditioning-only frameworks synthesize a video from a novel camera viewpoint given a monocular reference video.
  • A typical model contains a VAE and Diffusion Transformer (DiT), with the VAE encoding the reference video into latent representations.
  • The VAE downsamples inputs by 4× temporally and 16× spatially, while DiT receives concatenated reference and noisy target latents.
  • DiT predicts the velocity field conditioned on the text caption and target camera trajectory, and the model is trained via flow matching.
  • 3D Attention Across Reference and Target Latent Tokens: Because reference and target latents are concatenated, 3D attention jointly captures intra-video temporal and inter-video cross-view correspondences.

4 Analysis

The analysis shows that camera-conditioning-only models encode temporal and cross-view geometric correspondences in their 3D attention maps. In ReCamMaster, specific intermediate layers provide strong matching cues across views and over time, with a dominant peak around the 18th diffusion layer.

  • Emergent correspondences: 3D attention maps contain token-level correspondences across layers and denoising timesteps.The analysis identifies temporal correspondences within reference and target views, plus cross-view correspondences between them.
  • Analysis setup: The analysis compares attention-derived matches with pseudo ground-truth point tracks to measure correspondence accuracy.Pseudo ground-truth tracks are obtained using MV-TAP on synchronized multi-view recordings from MultiCamVideo.
  • Correspondence measurement: Correspondences are extracted by selecting the highest-attention key feature and retaining only cycle-consistent forward matches.A match is reliable when backward matching from the destination returns to the original query token.
  • Findings: Attention maps exhibit correspondence cues within each view over time and across reference and target views.These are termed intra-video temporal correspondences and inter-video cross-view correspondences, respectively.
  • Findings: A dominant peak around the 18th diffusion layer indicates strong matching performance in specific intermediate layers.The same layer range supports both motion tracking within views and geometric alignment across views, while the harmonic mean follows the matching-accuracy layer–timestep trend.

5 Methodology

MVTrack4Gen jointly trains a camera-controlled video diffusion model with a multi-view tracking module that reuses 3D-attention query and key features. Multi-scale temporal and cross-view correlations, together with tracking and correspondence supervision, promote motion fidelity and geometric consistency.

  • Framework: The framework synthesizes a target-view video from reference and target camera trajectories by jointly training diffusion and multi-view tracking modules sharing 3D-attention features.The shared features are directly supervised to connect generation with geometric and motion consistency.
  • Camera Conditioning: Both reference and target camera extrinsics and intrinsics condition the diffusion backbone through dense 6D Plücker ray maps injected into every DiT layer.The method uses ReCamMaster and Redirector as backbones with richer camera conditioning than target extrinsics alone.
  • Correlation Volumes: Query–key similarities in 3D attention form multi-scale local 4D correlation volumes that encode intra-view temporal motion for the tracking module.Query points are sampled across all video frames so newly appearing points can also contribute motion supervision.
  • Multi-View Tracking Head: A transformer tracking head with factorized temporal and multi-view attention predicts residual point, visibility, and confidence updates from correlation features.Its input tokens concatenate current visibility, confidence, and local 4D correlation volumes, with visibility and confidence initialized to zero.
  • Training Objectives: The total objective combines diffusion, multi-view point-tracking, and multi-view correspondence losses, with λtrack = λcorr = 0.01.Tracking uses ground-truth trajectories, while correspondence supervision applies cross-entropy to attention weights at visible tracked locations.

6 Experiments

Experiments evaluate MVTrack4Gen across in-the-wild and handheld dynamic-scene benchmarks against explicit 3D lifting and camera-conditioning baselines. Results show improved geometric consistency, strong visual quality and camera accuracy, while ablations validate attention-level correspondence supervision.

  • Datasets: Evaluation uses DAVIS for diverse monocular motion and iPhone for casual handheld dynamic-scene captures.Training combines Kubric and MultiCamVideo.
  • Baselines and setup: The study compares against GEN3C, TrajectoryCrafter, CogNVS, NeoVerse, ReCamMaster, and Redirector across explicit 3D lifting and camera-conditioning-only paradigms.The backbones are ReCamMaster and Redirector, with 3D attention layers and the camera encoder fine-tuned while other parameters remain frozen.
  • Evaluation metrics: Metrics cover generation quality, geometric consistency, and camera accuracy, using dataset-appropriate image, video, geometry, and camera measures.iPhone uses PSNR, SSIM, and LPIPS against ground-truth views; DAVIS uses VBench measures when ground-truth views are unavailable.
  • Main results: On DAVIS, MVTrack4Gen achieves the best overall performance, leading most visual-quality metrics and attaining the lowest MEt3R, MEt3Rdynamic, mTransErr, and mCamMC while remaining competitive on mRotErr.The reported visual-quality leaders are Subject Consistency, Background Consistency, Aesthetic Quality, Imaging Quality, and Motion Smoothness.
  • Qualitative results: Qualitative results show stronger cross-view alignment, whereas baselines exhibit viewpoint inaccuracies, geometric distortions, depth errors, non-rigid warping, or texture degradation.These comparisons cover DAVIS and challenging real-world iPhone scenes, including dynamic objects and reconstructed point clouds.
  • Ablation study: Ablation results show that adding multi-view correspondence loss to the 3D attention map substantially improves geometric consistency and camera accuracy while leaving image quality largely unchanged.Variants are progressively added to ReCamMaster and fine-tuned for 10k iterations.

7 Conclusion · A Correspondence in 3D Attention Map · A.1 Dataset for Analysis and Pseudo Ground-Truth Generation

MVTrack4Gen strengthens motion and geometric correspondence for dynamic novel-view video diffusion by exploiting correspondence-specialized attention layers. Its analysis uses synchronized multi-view data and MV-TAP-derived tracks as pseudo ground truth.

  • 7 Conclusion: MVTrack4Gen strengthens both motion and geometric correspondence in a video diffusion model for dynamic novel-view generation.
  • A Correspondence in 3D Attention Map: Correspondence-specialized attention layers jointly encode intra-video temporal and inter-video cross-view correspondences in novel-view diffusion models.
  • A.1 Dataset for Analysis and Pseudo Ground-Truth Generation: The MultiCamVideo dataset contains synchronized multi-view recordings of dynamic scenes rendered via Unreal Engine 5 with large camera movements and diverse camera trajectories.
  • A.1 Dataset for Analysis and Pseudo Ground-Truth Generation: Because scenes are captured by multiple cameras at identical timestamps, the dataset provides paired reference–target videos for evaluating temporal and multi-view consistency.
  • A.1 Dataset for Analysis and Pseudo Ground-Truth Generation: MV-TAP jointly reasons over synchronized views through cross-view attention to estimate dense point correspondences across temporal and cross-view axes.
  • A.1 Dataset for Analysis and Pseudo Ground-Truth Generation: Query points are initialized on the first frame in a regular grid, and MV-TAP runs jointly over the 10 synchronized view videos.
  • A.1 Dataset for Analysis and Pseudo Ground-Truth Generation: For each tracked point, the representation records its 2D location in view v at frame i.
  • A.1 Dataset for Analysis and Pseudo Ground-Truth Generation: A binary visibility indicator records whether each point is visible, and the resulting tracks serve as pseudo ground truth.

A.2 Details of Attention-based Correspondence Evaluation … B.4 Training Data

The paper evaluates attention-based correspondences with forward–backward cycle consistency and PCK, then extends the analysis across backbones and details camera encoding, multi-scale tracking features, layer selection, and training data. These components identify correspondence-specialized attention structure and provide geometric and motion supervision for novel-view generation.

  • A.2 Details of Attention-based Correspondence Evaluation: Forward matches select the highest-attention spatial location, while backward mapping through reverse attention tests whether the correspondence returns to the original query.The mapping operator F is applied in both directions to evaluate attention-based correspondences.
  • A.2 Details of Attention-based Correspondence Evaluation: A correspondence is reliable only when forward–backward cycle consistency holds, using threshold δ = 16 and tracks rescaled to latent resolution.This enforces mutual best matching between the two attention directions.
  • A.2 Details of Attention-based Correspondence Evaluation: PCK counts reliable matches within δ pixels of co-visible ground truth, using latent-grid matching, original-pixel distances, and averaging across correspondence types, frames, and scenes.The three evaluated types are reference-view temporal, target-view temporal, and cross-view correspondences.
  • A.3 Generalization to Another Backbone: Redirector reproduces ReCamMaster’s layer–timestep specialization: temporal correspondences peak in intermediate ranges, while cross-view correspondence localizes near the middle 18th layer.Attention and confidence analyses show temporal signals strengthening in early or later layers and cross-view signals in intermediate layers.
  • B.1 Plücker Ray Camera Encoding: Camera conditioning uses dense 6D Plücker ray maps for reference and target views, encoded by a lightweight convolutional module and projected into every DiT layer.The ray direction is normalized for scale invariance, and injection follows ReCamMaster.
  • B.2 Multi-Scale Local 4D Correlation: The tracking head builds multi-scale local 4D correlations by sampling query and key features around points across frames and concatenating correlation volumes over scales.The feature pyramid uses S ∈ {1, 2, 3, 4} scales and local neighborhoods around query and estimated match points.
  • B.3 Layer Ablation: The 18th DiT layer is selected for tracking-head features because its 3D attention most reliably encodes inter-video cross-view correspondence and yields the best downstream generation.The matching layer is the DiT layer whose query and key features are routed into the tracking head.
  • B.4 Training Data: Training combines Kubric and MultiCamVideo to diversify scenes, camera motions, and visual domains, while sequence reversal augments object-motion and camera-trajectory diversity.Kubric supplies precise calibration and dense tracks; MultiCamVideo supplies 10 synchronized calibrated viewpoints and 1,560 tracked points per frame.

B.5 Training Objective Details

The training objective combines rectified-flow diffusion supervision with multi-view point-tracking and correspondence losses. Diffusion loss trains target-view generation, while tracking supervision shapes temporal and cross-view attention toward geometrically correct correspondences.

  • Diffusion Loss: The rectified-flow diffusion loss predicts velocity from noisy target latents conditioned on reference and target latents, timestep, conditioning signal, and target camera.The target is ϵ − x0 with xt = (1 − t)x0 + tϵ, and squared error is evaluated only on target-view tokens.
  • Tracking Loss: The tracking objective supervises the multi-view point-tracking head using ground-truth trajectories across reference and target views.Its sequence term is a visibility-weighted Huber loss, restricting coordinate regression to visible points and avoiding noisy gradients from occlusions.
  • Tracking Loss: The tracking terms use λseq = 0.05 and λconf = λvis = 1.0.These weights specify the relative contributions of sequence, confidence, and visibility components in the tracking objective.
  • Tracking Loss: The confidence loss calibrates per-point confidence against realized regression error, assigning higher confidence to predictions within a small ground-truth radius.This encourages reliability estimates that downstream consumers can use to filter unreliable tracks.
  • Multi-View Correspondence Loss: The multi-view correspondence loss applies visibility-weighted cross-entropy to attention rows, supervising each query against co-visible ground-truth matches across latent frames and both views.Averaging over co-visible target frames jointly shapes intra-video temporal and inter-video cross-view correspondences to enforce geometric consistency.

C More Generation Results

On DAVIS in-the-wild monocular videos with diverse object and camera motion, MVTrack4Gen produces novel views that remain faithful to the reference while following the target camera trajectory and preserving dynamic-subject and surrounding-scene geometry. These qualitative results support strengthening multi-view correspondence learning in video diffusion without explicit 3D representations at inference.

  • DAVIS qualitative comparisons: MVTrack4Gen generates visually faithful novel views that accurately follow the prescribed target camera trajectory across diverse DAVIS scenes.The comparisons cover in-the-wild monocular videos with diverse object and camera motion.
  • Correspondence learning: The qualitative results align with Tab. 1’s quantitative trends, confirming the effectiveness of multi-view correspondence learning for in-the-wild novel-view synthesis.The approach achieves this without any explicit 3D representation at inference.

D Attention Visualization Results

Attention visualization identifies the 18th layer as correspondence-specialized, with queries attending to geometrically corresponding locations across views. Synthesis errors are tightly coupled with breakdowns in this cross-view alignment.

  • Attention Visualization Results: The 18th layer most strongly captures cross-view geometric correspondences, as each query attends to its corresponding location across views.The attention map at this layer is visualized because it acts as the correspondence-specialized layer.
  • Attention Visualization Results: Synthesis errors are tightly coupled with correspondence misalignment when query points fall on incorrectly generated regions.The passage links incorrect generated regions to a breakdown of attention alignment.

E Limitations

MVTrack4Gen achieves state-of-the-art geometric consistency and camera-pose accuracy for dynamic novel-view generation, but its applicability remains constrained by the ReCamMaster backbone. In particular, fixed resolution and sequence-length limits restrict use on higher-resolution and longer videos.

  • MVTrack4Gen still has limitations despite achieving state-of-the-art geometric consistency and camera-pose accuracy on dynamic novel-view generation.
  • The inherited ReCamMaster backbone fixes the output resolution at 480×832, restricting applicability to higher-resolution videos.
  • The inherited ReCamMaster backbone limits sequences to 81 frames, restricting applicability to longer videos.
Loading 2606.26087v1…