ReCamDriving: LiDAR-Free Camera-Controlled Video Synthesis for Novel Trajectories

1 Introduction

• Multi-pass videos provide diverse viewpoints for 3D reconstruction and world-model training, but collecting them requires multiple synchronized vehicles.The paper presents single-pass supervision as an attractive, scalable alternative.Source: §1 · pp. 1–2
• Repair models can fail on out-of-distribution rendering artifacts, while sparse LiDAR cues leave occluded or distant regions geometrically inconsistent.Prior camera-transformation training from single-pass segments also models longitudinal motion, creating a gap for lateral inference.Source: §1 · pp. 2–3
• ReCamDriving uses novel-trajectory 3DGS renderings as structurally complete camera guidance in a purely vision-based framework.Its two-stage training first establishes coarse control from camera poses, then adds 3DGS renderings to guide geometry without treating them as restoration templates.Source: §1 · p. 3
• A 3DGS-based cross-trajectory curation strategy provides lateral-trajectory supervision from single-pass videos and supports ParaDrive, a dataset of 110K parallel-trajectory pairs.The strategy aligns training and inference camera-transformation patterns.Source: §1 · p. 3
• ReCamDriving achieves state-of-the-art performance in camera controllability and 3D consistency.The introduction reports this overall result without specifying benchmark values.Source: §1 · p. 3

2.1 Diffusion Priors for Repairing 3D Renderings.

• Diffusion-prior repair methods improve rendering fidelity but focus on local artifact correction, depend on training-distribution coverage, and can fail on out-of-distribution degradations.Examples include 3DGS-Enhancer, Difix3D+ [60], GSFixer [67], and Freesim.Source: §2.1 · p. 4

2.2 Camera-Controlled Video Generation.

• Pose-conditioned generation often lacks geometric consistency, while point-map and LiDAR approaches face reconstruction-quality or spatial-completeness limits; ReCamDriving instead uses dense 3DGS renderings.Prior camera-controlled methods include [2] [15], and LiDAR-based approaches include FreeVS [53] and StreetCrafter.Source: §2.2 · p. 4, Figure 2 · p. 5

3 Cross-trajectory Data Curation and ParaDrive Dataset

• Single-pass datasets and longitudinal source–target splits provide limited supervision for lateral camera transformations, creating a train–test gap for lateral-view generation.The proposed strategy uses laterally shifted 3DGS renderings as training sources and clean recorded videos as supervision.Source: §3 · p. 5
• Approximately 110K parallel-trajectory video pairs are constructed from about 1.6K 3DGS scenes reconstructed from WOD and NuScenes.Underfitted reconstructions supply renderings with varied artifact levels; fully optimized models render laterally shifted source trajectories.Source: §3 · pp. 5–6

4.1 Preliminary: Latent Diffusion Models with Flow Matching

• Latent diffusion performs generation in a compact autoencoder space, while Flow Matching learns a deterministic ODE velocity field to transport noise toward data.The training objective conditions the predicted velocity on camera information.Source: §4.1 · pp. 6–7

4.2 Training Stage 1: Relative Pose-Guided Coarse Camera Control

• Stage 1 conditions on relative camera pose to learn coarse viewpoint transformation, using relative rather than absolute poses for easier inference specification and greater robustness to calibration errors.The relative pose and identity pose are encoded separately, and the source and noisy video latents are concatenated along the frame dimension before DiT processing.Source: §4.2 · pp. 7–8

4.3 Training Stage 2: Fine-grained Camera Control via 3DGS Renderings

• Stage 2 adds Rendering Attention and Cross Attention; Cross Attention fuses 3DGS rendering features with diffusion latents after Stage 1 self-attention, whose modules remain frozen and whose latents have undergone coarse camera transformations.This coarse viewpoint transformation reduces geometric-alignment difficulty and mitigates the tendency to collapse into trivial restoration of 3DGS rendering artifacts.Source: §4.3 · p. 8

5.1 Experimental setup

• The evaluation uses WOD and NuScenes and measures visual quality, camera accuracy, and view consistency against reconstruction, repair, and camera-controlled baselines.DriveStudio is excluded from camera-accuracy evaluation because it renders using ground-truth poses.Source: §5.1 · pp. 9–11

5.2 Comparison Results

• Qualitative comparisons show ReCamDriving maintaining structural continuity where repair and sparse-LiDAR baselines produce view inconsistencies, incomplete geometry, or blurred distant backgrounds.The examples cover WOD and NuScenes; FreeVS is evaluated at a cropped resolution excluding sky regions.Source: Figure 4 · p. 10, Figure 5 · p. 10, §5.2 · p. 11
• ReCamDriving consistently outperforms all baselines across reported metrics on WOD and NuScenes, while remaining stable as lateral offsets increase.View consistency for FreeVS, StreetCrafter, and Difix3D+ degrades sharply with larger offsets.Source: Table 3 · p. 11, §5.2 · p. 11

5.3 Ablation Studies

• 3DGS conditioning improves camera control over pose-only conditioning and avoids the large-offset geometric inconsistencies seen with sparse LiDAR guidance.Adding LiDAR on top of the 3DGS-conditioned method yields no significant overall-metric improvement and incurs data-acquisition costs.Source: §5.3 · p. 12, Table 4 · p. 12
• A repair-only baseline often fails to correct artifacts that are not well covered by its training distribution.The comparison uses the same general architecture but omits rendering attention and camera-pose injection.Source: §5.3 · p. 12, Figure 7 · p. 13, Table 5 · p. 13
• Two-stage training produces clearer, more 3D-consistent results than jointly training all modules, which tends to yield artifacts and degraded visual performance.The comparison is conducted on WOD and NuScenes.Source: §5.3 · p. 13, Table 6 · p. 14, Figure 8 · p. 14
• Cross-trajectory curation improves camera accuracy and view consistency over longitudinal-only training for novel-trajectory prediction.The baseline uses separated segments of each recorded trajectory as source and supervision.Source: §5.3 · p. 13, Table 7 · p. 14
• Clean recorded-video sources at inference yield sharper backgrounds than blurred 3DGS-rendered sources, and the source sensitivity supports camera-control generation rather than mere rendering restoration.The study finds that the source-video mismatch does not degrade performance.Source: §5.3 · pp. 13–14, Figure 9 · p. 14

• Offline 3DGS preprocessing is required, although each reconstruction supports multiple trajectory generations.Source: §6 · pp. 14–15
• Feed-forward 3D reconstruction could reduce preprocessing time from hours to seconds as the techniques evolve.Source: §6 · p. 15
• The framework offers a pathway to use internet-scale datasets that lack expensive LiDAR annotations.Source: §6 · p. 15

• ReCamDriving combines 3DGS structural camera conditions with two-stage training for precise camera control and consistent geometry.Source: §7 · p. 15
• The cross-trajectory curation strategy yields ParaDrive, supporting scalable lateral-trajectory supervision from single-pass driving videos.Source: §7 · p. 15
• Extensive experiments show higher camera accuracy and visual fidelity than existing baselines.Source: §7 · p. 15

• 3DGS models are trained for 30,000 iterations on approximately 1.6K Waymo and NuScenes scenes, with lateral-offset renderings used as source conditioning inputs.Source: §8 · p. 21
• Training pairs use novel-trajectory renderings at 30,000 iterations as sources, intermediate original-trajectory renderings as conditions, and clean recorded videos as supervision.Source: Figure 10 · p. 21
• Intermediate 3DGS renderings at 100, 500, and 1,000 iterations provide degraded conditions intended to match inference artifacts; optimization uses default DriveStudio settings without SMPL refinement.Source: §8 · p. 21

• Evaluation uses 20 selected scenes from each of WOD and NuScenes, with the selected sequence names listed in Tables 8 and 9.Source: §9 · p. 21, Table 8 · p. 22, Table 9 · p. 22
• Each evaluation scene uses its recorded trajectory video as source input and 3DGS renderings of eight lateral offsets as camera conditions.The offsets are ±1 m, ±2 m, ±3 m, and ±4 m.Source: §9 · p. 22
• Table 9 lists NuScenes evaluation sequences with 10 sequence indices per row.Source: Table 9 · p. 22

10 Discussion on the Availability of Feed-forward 3DGS

• STORM is used to generate 20-frame 3DGS segments for evaluating feed-forward 3DGS during inference.Fig. 11 presents the effect of using the feed-forward 3DGS method during inference.Source: §10 · p. 22, Figure 11 · p. 23

11 Temporal Consistency Error

• For each consecutive frame pair, TCE samples points from a center-cropped region to focus on central dynamic subjects and mitigate out-of-bounds artifacts from camera motion.The forward optical flow predicts point locations in the next frame, and backward flow projects them back for a reliability check.Source: §11 · p. 23
• TCE is the arithmetic mean of the frame-pair consistency error across consecutive pairs in a video.The frame-pair error is the mean Euclidean distance between original points and their re-projected locations.Source: §11 · p. 23
• The calculation averages the pairwise consistency errors over T−1 consecutive-frame pairs.Source: §11 · p. 23

12 Comparisons of Different Training Stages

• The training-stage comparison visualizes synthesis at lateral offsets of −4 m and +4 m, while Table 10 compares methods by frame length.The framework uses progressive two-stage training for camera control and consistent content generation.Source: §12 · p. 23, Figure 12 · p. 24, Table 10 · p. 24

13 More Comparisons

• StreetCrafter and FreeVS use short temporal windows, and their rollout inference often introduces background inconsistencies and noticeable temporal jitter.StreetCrafter is trained on Vista, while FreeVS is built on Stable Video Diffusion.Source: §13 · p. 24

14 More Qualitative Results

• The section presents qualitative novel-trajectory results in Fig. 13 and Fig. 14, with two scenes shifted right by 3 m and 4 m.The accompanying text reports that the method maintains geometric consistency and visual fidelity while accurately generating novel trajectories.Source: §14 · p. 25, Figure 13 · p. 25, Figure 14 · p. 26