Source-linked AI summary
Generative Camera Dolly: Extreme Monocular Dynamic Novel View Synthesis
Basile Van Hoorick, Rundi Wu, Ege Ozguroglu, Kyle Sargent, Ruoshi Liu, Pavel Tokmakov, Achal Dave, Changxi Zheng, Carl Vondrick
TL;DR
Dynamic novel view synthesis seeks to generate a video of a scene from another viewpoint, but existing methods often require multi-view recordings or limit camera changes. GCD fine-tunes a pretrained video diffusion model on paired simulated views, reporting state-of-the-art monocular synthesis and generalization to several real-world domains.
Problem
Dynamic novel view synthesis aims to generate the same dynamic scene from another viewpoint, but prior methods often require contemporaneous multi-view video or restrict camera changes to a handful of degrees.
Source: §1 · pp. 1–2Method
Results
GCD achieves state-of-the-art monocular DVS and generalizes to driving, robot manipulation, and heavily occluded in-the-wild videos.
Source: §1 · p. 3Takeaways & Limitations
The framework generates synchronized novel-view video from monocular input, with a virtual camera capable of up to six degrees of freedom.
Source: §1 · p. 3, §3 · p. 5The task assumes output frames are synchronized with the input and camera intrinsics remain constant across time and pose changes.
Source: §3 · p. 5
Abstract
from arXiv · showhide
Accurate reconstruction of complex dynamic scenes from just a single viewpoint continues to be a challenging task in computer vision. Current dynamic novel view synthesis methods typically require videos from many different camera viewpoints, necessitating careful recording setups, and significantly restricting their utility in the wild as well as in terms of embodied AI applications. In this paper, we propose $\textbf{GCD}$, a controllable monocular dynamic view synthesis pipeline that leverages large-scale diffusion priors to, given a video of any scene, generate a synchronous video from any other chosen perspective, conditioned on a set of relative camera pose parameters. Our model does not require depth as input, and does not explicitly model 3D scene geometry, instead performing end-to-end video-to-video translation in order to achieve its goal efficiently. Despite being trained on synthetic multi-view video data only, zero-shot real-world generalization experiments show promising results in multiple domains, including robotics, object permanence, and driving environments. We believe our framework can potentially unlock powerful applications in rich dynamic scene understanding, perception for robotics, and interactive 3D video viewing experiences for virtual reality.
1 Introduction
Dynamic novel view synthesis seeks a video of the same scene from another viewpoint, but existing methods often depend on contemporaneous multi-view video or small camera changes. GCD fine-tunes a pretrained video model on paired simulated views and reports state-of-the-art monocular DVS with generalization to real-world scenes.
1 Introduction
- GCD achieves state-of-the-art monocular DVS and generalizes to driving, robot manipulation, and heavily occluded in-the-wild videos.The figure illustrates detailed synthesis even for extreme camera transformations.Source: Figure 1 · p. 2, §1 · p. 3
- Prior methods typically require contemporaneous multi-view video or restrict camera changes to a handful of degrees, limiting their fit for in-the-wild use cases.Those use cases include street scenarios, robot manipulation, mixed-reality passthrough, and viewing recorded videos from new angles.Source: §1 · p. 2
- GCD uses paired dynamic-scene videos from simulation to fine-tune a pretrained video generative model for dynamic view synthesis.The approach leverages pretrained models' learned representations and priors.Source: §1 · pp. 2–3
- The paper presents GCD as an end-to-end video-to-video framework for generating a dynamic scene from novel viewpoints.Its virtual camera can move with up to six degrees of freedom and reveal unseen regions, including objects behind occlusions.Source: §1 · p. 3
2 Related Work
Dynamic scene view synthesis has often relied on synchronized multi-view recordings or per-scene representations, while newer single-view and diffusion-based methods offer different trade-offs. GCD proposes a broader video-to-video approach that aims to synthesize dynamic scene content, including occluded objects and non-object regions.
2 Related Work
- Dynamic scene reconstruction has largely relied on synchronized multi-view videos, while single-camera methods include volumetric rendering and diffusion-prior distillation into hybrid 4D representations.DynIBaR aggregates features from nearby views; DpDy distills diffusion priors into static and dynamic NeRF components [71].Source: §2 · p. 3
- Per-scene optimization limits knowledge sharing and extrapolation from incomplete observations, while monocular inputs without strong multi-view cues can trigger failure modes.An exception performs dynamic scene completion from a single static RGB-D video.Source: §2 · p. 4
- GCD adopts Stable Video Diffusion [5], a public image-to-video model, while its framework is described as generalizable to other video-generation approaches.Other cited approaches use diffusion models [5] [24] [59] or autoregressive training.Source: §2 · p. 4
- Most generative 3D methods channel 2D diffusion priors into a single iteratively optimized 3D representation, while emerging 4D work remains mostly limited to single-object or animal animations [1] [35] [60] [90].Video-to-4D work is less explored; requires depth and is trained from scratch for narrow domains.Source: §2 · p. 4
- Object-centric completion methods address occluded object parts or track fully occluded instances, but they focus on objects rather than the full complexity of dynamic scenes [67].GCD's broader scene coverage includes “stuff” regions such as surfaces and liquids.Source: §2 · pp. 4–5
- A concurrent approach translates third-person to first-person video per frame, using hand-object interaction priors and focusing primarily on those scenarios [39].The GCD method figure instead describes input and output videos linked by relative camera extrinsics ΔE.Source: §2 · p. 5, Figure 2 · p. 5
3 Approach
GCD formulates monocular dynamic view synthesis as generating a synchronized video from a target camera pose, assuming fixed camera intrinsics. It adapts a pretrained video diffusion model with relative-pose control and time-varying input-video conditioning.
3 Approach
- The task maps a single-camera RGB video to a plausible video of the same dynamic scene from specified target extrinsics, with synchronized frames and constant intrinsics across time and poses.The virtual output camera uses the input camera's focal length.Source: §3 · p. 5
- GCD adapts Stable Video Diffusion to accept relative camera transformations as fine-grained controls, addressing the lack of video conditioning and camera control in pretrained video generators.The approach uses large-scale video generative models because novel view synthesis is under-constrained.Source: §3 · p. 5
3.1 Camera viewpoint control
- The model learns camera control from paired videos and relative extrinsics, decomposing each transformation into rotation and translation and embedding it into network features.The pose embedder is randomly initialized, while the other weights start from the public image-to-video checkpoint; the model is then trained end-to-end.Source: §3.1 · p. 6
3.2 Video conditioning
- Video conditioning substitutes the full source video for SVD's first-frame input so the model can observe scene dynamics over time.At each timestamp, contemporaneous source frames are attached to the output sample; temporal and spatial attention connect features across frames and locations.Source: §3.2 · pp. 6–7
- In the Kubric-4D qualitative ablation, the gradual, maximum-90° finetuned model preserves scene layout and dynamics better than blurrier ablations, especially for moving objects.It often hallucinates plausible backgrounds in unseen regions; the input and output sequences each contain 14 frames.Source: Figure 3 · p. 7
4 Datasets
The paper contributes two synthetic multi-view video datasets to address the relative scarcity of such data and support training and evaluation. They cover generic object interactions and photorealistic driving, with task setups that target novel viewpoints including a top-down driving view.
4 Datasets
- Multi-view video remains relatively sparse compared with conventional image or video datasets, motivating two synthetic datasets for training and evaluation.The datasets are shown in Figures 3, 4, and 5.Source: §4 · p. 7
4.1 Kubric-4D
- Kubric-4D provides 3,000 dynamic scenes with RGB-D from 16 fixed virtual cameras, including frequent complex occlusions.Scenes contain 7–22 objects and are 60 frames long at 24 FPS; arbitrary-view videos are rendered from merged per-frame point clouds.Source: §4.1 · p. 8
4.2 ParallelDomain-4D
- ParallelDomain-4D provides 1,533 photorealistic driving scenes with RGB, semantic and instance annotations, and depth from 19 fixed virtual cameras.Each scene has 50 frames at 10 FPS across varied locations, vehicles, traffic, and weather.Source: §4.2 · p. 8
4.3 Task details
- The task setup targets a destination camera pose by or before the generated video ends, with pose descriptions required to support convex interpolation and map to a valid 6-DoF transform.Kubric camera controls have three effective degrees of freedom; ParallelDomain maps forward-facing vehicle video to a fixed top-down view for a detailed overview of surroundings.Source: §4.3 · p. 9
5 Choice of camera trajectory
The section evaluates camera-trajectory choices, balancing rapid arrival at a target view against maintaining alignment with the pretrained video model’s source-view prior. On Kubric-4D, gradual trajectories and SVD finetuning are preferred, with a trade-off between training rotation range and achievable extremes.
5 Choice of camera trajectory
- The trajectory design seeks views as far as 180° around the scene, but moving away from the source too quickly risks distribution misalignment with SVD’s source-view-biased video prior.The source-view bias reflects that SVD commonly generates videos from nearly the input image’s perspective, with only minor panning or mild rotation.Source: §5 · p. 10
- Gradual trajectories interpolate the virtual camera pose from source to destination over the video, whereas direct trajectories use the destination pose at every frame.For Kubric, max 90° training limits azimuth variation to ±90°; max 180° training allows any 360°-surround viewpoint.Source: §5 · p. 11
- +1.17 dB average PSNR favors gradual over direct trajectories on Kubric-4D; finetuning beats scratch by +1.34 dB, while max 180° versus max 90° differs by +0.55 dB.The same overall findings hold on ParallelDomain-4D, where gradual, finetuned is the best model; gradual, max 90°, finetuned is used in further Kubric experiments because it is close to max 180°.Source: §5 · p. 11
6 Experiments
The experiments compare monocular dynamic view synthesis against per-scene, diffusion, and geometric baselines on two in-domain datasets, with additional real-world examples. GCD outperforms baselines on both datasets and sometimes generalizes to varied real-world scenes.
6 Experiments
- The evaluation reports numerical results on Kubric-4D and ParallelDomain-4D test splits and qualitative examples from in-the-wild videos.The experiments compare against state-of-the-art baselines and include examples from varied domains.Source: §6 · pp. 11–12
6.1 Implementation details
- Training uses 10k iterations on Kubric-4D and 13k on ParallelDomain-4D; inference uses 25 EDM sampling steps and takes roughly 10 seconds per video.The models use 14 frames at 384×256 resolution; reported training runs take roughly three days.Source: §6.1 · p. 12
- RGB evaluation averages PSNR, SSIM, and LPIPS across frames and examples; semantic evaluation reports mIoU over the 10 most common categories.Occluded regions are evaluated separately, and the first output frame is excluded because it aligns spatially with the input frame.Source: §6.1 · pp. 13–14
6.2 Baselines
- Baselines include per-scene methods, pretrained diffusion models, and depth-based reprojection; all methods use the same monocular input and sampled camera trajectories.The per-scene methods are described as limited to smaller camera changes in the one- or low-view regime and much slower at inference.Source: §6.2 · p. 14
6.3 Results
- On both datasets, GCD outperforms baselines by a large margin; HexPlane, Vanilla SVD, and ZeroNVS exhibit reconstruction, camera-following, or temporal-coherence failures.GCD mostly produces plausible geometry and motion under the desired viewpoints; qualitative Kubric-4D results show fewer flickering artifacts than baselines.Source: Figure 5 · p. 12, §6.3 · pp. 14–15
- GCD sometimes generalizes to driving, indoor, and robotic scenes; a fully occluded duck example preserves the hidden object's position and appearance at the final frame.The qualitative examples include BridgeData V2, TCOW Rubric, TRI-DDAD, and Berkeley DeepDrive.Source: Figure 6 · p. 13, §6.3 · p. 15
7 Discussion
The paper presents monocular dynamic view synthesis by finetuning a pretrained video diffusion model on synthetic data, while noting difficulty with significantly out-of-distribution videos such as those with moving humans.
7 Discussion
- The framework finetunes a large-scale pretrained video diffusion model on high-quality synthetic data to synthesize dynamic novel views from monocular video.Source: §7 · p. 15
- The model still struggles on significantly out-of-distribution examples, including videos with moving humans.Source: §7 · p. 15
- The authors characterize the work as meaningful progress in 4D-scene understanding and a first step toward zero-shot monocular dynamic view synthesis.Source: §7 · p. 15
Supplementary Material
The supplementary material describes analyses of HexPlane's view-count requirements and camera-control difficulty, followed by implementation details, failure cases, and video visualizations.
Supplementary Material
- Section B analyzes HexPlane's equivalent source-view requirements and how model metrics vary with camera-control difficulty.Source: A Overview · p. 22
- Section C covers architecture, training, dataset processing, evaluation, and baseline adaptations.Source: A Overview · p. 22
- Section D discusses failure cases; additional qualitative results are available as video visualizations.Source: A Overview · p. 22
B.1 Comparison to multi-view methods
The supplementary comparison finds that GCD's single-view results surpass HexPlane trained with 16 views, while also examining view count and camera-rotation difficulty.
B.1 Comparison to multi-view methods
- Figure 7 plots test-set SSIM against HexPlane's number of training views, averaged over 20 scenes.Source: Figure 7 · p. 22
- For camera rotation in Kubric-4D, the main difficulty occurs in roughly the first 80 degrees; performance is mostly flat afterward.PSNR is measured at the last output frame, when the desired horizontal azimuth is reached.Source: Figure 8 · p. 23
- GCD's single-view results give better quality than HexPlane's results from 16 input views.HexPlane is evaluated with 1, 4, 8, 16, or 32 training views.Source: §B.1 · pp. 22–23
B.2 Error as a function of rotation angle
The section describes how rotation-angle effects are evaluated, defines the spherical camera coordinates, and outlines camera-pose conditioning in the model architecture.
B.2 Error as a function of rotation angle
- Figure 8 plots test-set average PSNR against rightward horizontal rotations from 0° to 180° for the Kubric-4D gradual, max 180°, finetuned model.Elevation is fixed at 10° and radius at 15m to encourage obstructed objects from the input view.Source: §B.2 · p. 23
C.1 Coordinate system
- Kubric-4D camera poses use azimuth ϕ, elevation θ measured from the XY-plane, and radial distance r.The elevation convention differs from inclination measured from the Z-axis.Source: Figure 9 · p. 23, §C.1 · p. 24
C.2 Architecture
- The architecture encodes relative camera extrinsics as (∆ϕ, ∆θ, ∆r), Fourier-encodes the angles, and embeds them with an MLP.Camera-pose embeddings condition the network alongside CLIP attention and other microconditioning; input camera poses need not be known.Source: §C.2 · p. 24
C.3 Data and training
Training uses temporally synchronized video pairs and sampled camera trajectories, with dataset-specific pose and frame sampling. The model conditions diffusion on camera motion and trains with a focal latent loss.
C.3 Data and training
- Kubric-4D pairs synchronized 14-frame clips sampled from 60 frames at 24 FPS, while ParallelDomain-4D samples 14-frame clips from 50 frames at 10 FPS.Random frame strides yield Kubric-4D rates of 6, 8, 12, or 24 FPS and ParallelDomain-4D rates of 5 or 10 FPS; conditioning augmentation uses noise strength 0.02.Source: §C.3 · pp. 24, 26
- The model performs latent-space diffusion, conditioning the U-Net on per-frame CLIP embeddings and embeddings for frame rate, camera pose transformation, and motion value.At training, the estimated output video is supervised; inference performs multiple denoising steps.Source: Figure 10 · p. 25
- Kubric-4D pose ranges differ by variant: max 90° limits relative azimuth to ±90°, while max 180° permits ±180° and broader elevation variation.The max 90° bounds are |∆θ| ≤30° and |∆r| ≤3; max 180° uses |∆θ| ≤60° and |∆r| ≤3.Source: §C.3 · pp. 24–25
- Output camera trajectories interpolate from source to destination poses; ParallelDomain-4D instead uses a fixed destination and a sine-wave Euclidean trajectory.Kubric-4D samples elevation starting points uniformly in sin θ, while ParallelDomain-4D uses fixed source and destination viewpoints.Source: §C.3 · p. 25
- For Kubric-4D, the motion bucket value scales with the magnitude of relative camera rotation, signaling greater optical flow for larger relative angles.The value is scaled from 0 to 255.Source: §C.3 · pp. 25–26
C.4 Loss
- Training uses a focal L2 loss on latent feature maps, progressively focusing on the 10% of embeddings with the largest mismatch after 5,000 iterations.For ParallelDomain-4D semantic completion, vehicle and people categories receive 3× and 7× weights, respectively, to reduce false negatives, especially for small objects.Source: §C.4 · p. 26
C.5 Evaluation
Evaluation fixes shared test splits and randomized test conditions across methods. Baselines receive resolution or scale adjustments, and reported real-world failures identify generalization weaknesses.
C.5 Evaluation
- All methods use the same dataset-specific test split, with 14-frame subclips and fixed sampled Kubric-4D target poses; diffusion models generate four samples per trajectory.Test inputs use starting elevation θ₁ = 5° to increase occlusion, and probabilistic-model results average four generated samples.Source: §C.5 · p. 26
C.6 Baselines
- Vanilla SVD inputs and targets are center-cropped and resized to 1024×576, while ZeroNVS uses 256×256 and a visually tuned scale for each video.Vanilla SVD retains its default motion bucket of 127; ZeroNVS scale is tuned until output qualitatively aligns with ground truth.Source: §C.6 · pp. 26–27
D Failure cases
- The failure-case figure notes that deformable objects are absent from Kubric-4D finetuning data and that some examples may be out-of-distribution for ParallelDomain-4D.The authors connect deformable-object absence with occasional errors in shape, appearance, and motion, including objects blending together.Source: Figure 11 · p. 27
- Real-world failures include blurry people, a cut-off robot arm, missed highway signs, unreconstructed overpasses, and confusion between pedestrians and cars.The authors attribute the robot-arm case presumably to Kubric-4D lacking robots or robotic motion patterns.Source: D Failure cases · p. 27