Source-linked AI summary
Track2View: 4D-Consistent Camera-Controlled Video Generation via Paired 3D Point Tracks
Feng Qiao, Zhaochong An, Zhexiao Xiong, Serge Belongie, Nathan Jacobs
TL;DR
Re-rendering video from a novel viewpoint requires prescribed camera trajectories while preserving scene appearance and dynamics across frames. Track2View uses paired 3D point tracks to provide continuous source-target correspondences, achieving state-of-the-art visual quality, synchronization, and camera accuracy on 400 videos, with rotation error reduced by 30–65%.
Problem
Novel-view video re-rendering must control camera trajectories while preserving scene appearance and dynamics across every frame, but existing approaches lack explicit temporally continuous source-target correspondences.
Method
Track2View conditions a video diffusion transformer on paired 3D point tracks and uses a dual-view conditioner to transfer source context through geometric operations and temporal aggregation.
Results
30–65% rotation-error reduction and 61–72% translation-error reduction relative to leading baselines accompany state-of-the-art visual quality, view synchronization, and camera accuracy on 400 videos.
Takeaways & Limitations
Track2View provides explicit, temporally continuous correspondences for camera-controlled video re-generation across static and dynamic scenes.
Takeaways & Limitations
Performance depends on upstream 3D point-tracking quality, and behavior under extreme out-of-distribution conditions remains uncharacterized.
Abstract
from arXiv · showhide
Re-rendering an existing video from a novel camera viewpoint requires the output to follow the prescribed camera trajectory while preserving the appearance and dynamics of the original scene across every frame. Existing methods rely on per-frame pose embeddings, noisy point-cloud renderings, or implicit learned correspondences, none of which provides an explicit, temporally continuous link between source and target pixels. We propose Track2View, which conditions a video diffusion transformer on paired 3D point tracks: sparse trajectories of scene points projected into both the source and target camera views. These tracks provide explicit spatiotemporal correspondences that are temporally continuous by construction, encoding what content should appear where and when. At the core of Track2View is a dual-view track conditioner that transfers visual context from source to target view through parameter-free geometric operations and learned temporal aggregation, ensuring generalization to arbitrary camera trajectories without memorizing specific motions. We further introduce a data curation pipeline that extracts one-to-one track correspondences by running a 3D point tracker on temporally concatenated multi-camera view pairs. On a 400-video benchmark spanning static and dynamic scenes, Track2View achieves state-of-the-art results across visual quality, view synchronization, and camera accuracy, reducing rotation error by 30-65% and translation error by 61-72% relative to leading baselines. Project page is available at this https URL: https://qjizhi.github.io/track2view
1 Introduction
Track2View addresses the challenge of camera-controlled video-to-video re-rendering by conditioning video diffusion on paired 3D point tracks that provide explicit, temporally continuous correspondences between source and target views. Its dual-view conditioner combines parameter-free geometric operations with learned temporal aggregation to support arbitrary camera trajectories.
- Motivation: Precisely steering camera trajectories remains difficult in video generation, especially when video-to-video re-rendering must preserve an existing scene across a novel viewpoint.The paper frames camera control as important for filmmaking, virtual reality, and robotics simulation.
- Track2View: Track2View re-renders existing videos from novel viewpoints using paired 3D point tracks projected into source and target camera sequences.The tracks encode camera-induced and object-induced displacement as explicit 4D correspondences.
- Track2View: 3D point tracks provide temporally continuous spatiotemporal correspondences and remain valid when the source camera moves because they are defined in a global coordinate system.This avoids relying on per-frame relative camera parameterizations tied to a static source camera.
- Track2View: The dual-view track conditioner uses parameter-free bilinear sampling and scattering, while learned temporal aggregation captures cross-frame context.The parameter-free geometric operations are designed to generalize to arbitrary camera trajectories.
2 Related Work
Video generation has progressed from image-diffusion extensions to latent diffusion and Diffusion Transformers, while camera-controlled novel-view generation remains challenged by spatiotemporal consistency. Recent point-tracking and feed-forward 4D reconstruction advances make track-based conditioning more practical.
- Video Generation: Video generation evolved from image diffusion backbones with temporal modules to latent video diffusion and attention-based Diffusion Transformers.Diffusion Transformers unify spatial and temporal dependencies and scale to large datasets, higher resolutions, and longer videos.
- Camera-Controlled Generation: Camera-controlled novel-view video generation includes monocular reshooting systems and camera-only conditioning approaches, but full spatiotemporal consistency remains difficult.GCD and ReCapture establish monocular reshooting from a single input video; camera-only methods inject poses or motion descriptors into pretrained video generators.
- Track-Based Conditioning: CoTracker and CoTracker3 improve point tracking under occlusion, while SpatialTracker variants lift tracking into 3D by modeling geometry, camera ego-motion, and object motion.Feed-forward reconstruction methods additionally recover dense point maps without per-scene optimization.
3 Method
Track2View re-renders a source video from a prescribed target camera trajectory by conditioning video diffusion on paired 3D point tracks that preserve temporal correspondence across views. Its conditioner geometrically transfers source visual context, aggregates it across time, and injects view-specific depth before target-latent denoising.
- Task formulation: The task generates a synchronized target video that preserves the source scene’s content and dynamics under a user-specified camera trajectory.Each target frame corresponds to the same time step as its source frame.
- Track-based conditioning: Source 3D point tracks are estimated with SpatialTrackerV2 and reprojected under target camera poses to obtain paired source-target tracks.At inference, target tracks require no additional tracking because they are obtained by reprojection.
- Training and implementation: Track2View uses a clean source latent, a noisy target latent, and target-only flow-matching loss, with N=1152 default queries and LoRA adaptation of pretrained WAN-2.1 DiT blocks.The model generates 81-frame videos at 480 × 832 resolution; LoRA uses rank r=64 and α=64.
- Dual-view track conditioner: The dual-view conditioner samples source-token features at projected track locations, aggregates each track temporally, encodes inverse depth, and scatters features into dense source and target grids.Bilinear sampling and scattering are parameter-free, while temporal aggregation and feature projections are learned.
- Dual-view track conditioner: Shared temporal features and view-specific depth encodings let the conditioner propagate visible-point context while reasoning about parallax, occlusion, and viewpoint-consistent rendering.The resulting track tokens are added element-wise to concatenated dual-view video tokens before the DiT blocks.
- Paired track extraction: Paired training tracks are extracted by reversing one camera video, concatenating it with another at their shared first frame, and tracking the resulting 161-frame sequence.The recovered tracks preserve point identities across views; source and target cameras are randomly selected from MultiCamVideo.
4 Experiments
Track2View is evaluated on the 400-video RealCam-Vid benchmark across visual quality, view synchronization, camera accuracy, and VBench perceptual dimensions. It consistently outperforms camera-controlled video-generation baselines, with especially substantial gains in camera accuracy, while ablations show benefits from broader track queries and deeper temporal aggregation.
- Evaluation Setup: 400 videos comprise 200 static RealEstate10K scenes and 200 dynamic MiraData scenes in the RealCam-Vid evaluation set.FID reference features come from 33,388 real test frames, with benchmark captions used for text conditioning.
- Comparison: Track2View consistently outperforms Trajectory Attention, TrajectoryCrafter, Gen3C, and ReCamMaster across visual quality, view synchronization, and camera accuracy.Track2View is also evaluated at 25 and 49 frames to match baseline output lengths.
- Visual Quality: 26.82 vs. 30.32 for Gen3C and 33.85 for ReCamMaster is Track2View’s FID at 81 frames, while CLIP-T and CLIP-F remain comparable or better.This indicates lower visual-distribution distance without sacrificing semantic fidelity or temporal consistency.
- Camera Accuracy: 30–65% lower RotErr and 61–72% lower TransErr are achieved relative to respective baselines across frame lengths.At 25 frames, RotErr is 1.24° vs. 3.54° (−65%); at 81 frames, reductions remain 30% for RotErr and 61% for TransErr against the strongest baseline.
- Ablations: 2.24° to 1.55° (−31%) RotErr and 1.681 to 0.818 (−51%) TransErr result from increasing temporal aggregation depth from 2 to 8 layers.The 8-layer configuration is used by default; the “All” query strategy also yields best Mat.Pix. (0.685) and tied-best RotErr (2.24°).
5 Conclusion
Track2View uses paired 3D point tracks to provide explicit, temporally continuous correspondences for camera-controlled video re-generation. Its main limitations are dependence on upstream tracking quality and uncharacterized behavior under extreme out-of-distribution conditions.
- Conclusion: Track2View conditions a video diffusion transformer on paired 3D point tracks for camera-controlled video re-generation.The tracks are sparse scene-point trajectories projected into both source and target screen spaces.
- Conclusion: Sparse trajectories in source and target screen spaces provide explicit, temporally continuous correspondences absent from pose-based and rendering-based approaches.
- Limitations: The method depends on upstream 3D point-tracker quality, so tracking failures may degrade conditioning and generation quality.
- Limitations: Although trained entirely on synthetic Unreal-Engine-5 data and transferable to RealEstate10K and MiraData videos, behavior under extreme out-of-distribution conditions remains uncharacterized.