PaperlayerView exact revision on arXiv

Source-linked AI summary

PRG-Fusion: Orchestrating Generative Priors with Reconstruction Evidence for Driving View Synthesis

Sipeng He, Jialei Chen, Zhen Fang, Dongchun Ren, Feng Zhao

arXiv:2609.06948v1cs.CV

TL;DR

Collecting real-world data across diverse driving behaviors and conditions is costly and difficult to scale, motivating video synthesis along different trajectories from a single driving log. PRG-Fusion converts local reconstruction evidence into pixel-wise Preserve, Repair, and Generate labels that route 3DGS appearance, LiDAR geometry, and video priors by region. On Waymo, PRG-Fusion achieves improved visual quality, geometric fidelity, and view consistency; under a 6 m shift, it reduces FID by 8.81 compared with the second-best method.

  • Problem

    Collecting real-world data across diverse driving behaviors and conditions is costly and difficult to scale, motivating video synthesis along different trajectories from a single driving log.

    Source: §1 · p. 1

  • Method

    PRG-Fusion converts local reconstruction evidence into pixel-wise Preserve, Repair, and Generate labels that route 3DGS appearance, LiDAR geometry, and video priors by region.

    Source: §1 · pp. 1–2

  • Results

    On Waymo, PRG-Fusion achieves improved visual quality, geometric fidelity, and view consistency; under a 6 m shift, it reduces FID by 8.81 compared with the second-best method.

    Source: §4.2 · p. 7, §5 · p. 8

  • Takeaways & Limitations

    The reported experiments demonstrate improved visual quality, geometric fidelity, and view consistency for novel-trajectory synthesis.

    Source: §5 · p. 8

Abstract

from arXiv · show

Synthesizing photorealistic driving videos along specified trajectories is essential for scalable closed-loop simulation. Reconstruction-based methods leverage neural rendering to synthesize geometrically consistent views, but often exhibit diverse artifacts and missing content when the viewpoint deviates from the training trajectory. In contrast, generative models can synthesize realistic views along arbitrary trajectories from vehicle sensor data, yet often struggle to maintain temporal and geometric consistency across frames. To combine the strengths of both, we propose PRG-Fusion, a framework for driving view synthesis that uses reconstruction evidence to orchestrate generative priors across regions. Specifically, we extract region-wise degradation evidence from reconstructed driving scenes and convert it into Preserve, Repair, and Generate (PRG) labels. At inference, these labels serve as a unified routing policy for region-aware spatiotemporal synthesis, orchestrating 3DGS appearance preservation, LiDAR-guided structural correction, and video-prior-driven content completion across Preserve, Repair, and Generate regions, respectively. We then follow a two-stage training paradigm, first establish geometric control from sparse LiDAR projections and subsequently learning appearance control from dense 3DGS renderings. Extensive experiments on Waymo demonstrate that PRG-Fusion achieves state-of-the-art overall performance in novel trajectory video synthesis, with superior visual quality and geometric fidelity while maintaining competitive view consistency under large trajectory shifts.

1. Introduction

Synthesizing views from recorded driving logs offers a scalable alternative for closed-loop simulation, but reconstruction and generative methods each have limitations under trajectory shifts. PRG-Fusion uses pixel-wise Preserve, Repair, and Generate labels to route reconstruction and generative information according to local evidence.

1. Introduction

  • Reconstruction-based renderings degrade when target trajectories depart from captured viewpoints, producing artifacts, geometric distortions, and holes in newly exposed regions.Repeatedly collecting diverse real-world driving data is costly and difficult to scale, motivating synthesis from existing driving logs.Source: §1 · p. 1
  • Generative driving models support diverse instructed trajectories but can lack stable 3D consistency across frames, causing temporal drift in structures and spatial relationships.Balancing their trajectory generalization with reconstruction-based 3D consistency remains a challenge for closed-loop simulation.Source: §1 · p. 1
  • PRG-Fusion converts local reconstruction evidence into pixel-wise Preserve, Repair, and Generate labels that guide novel-trajectory video synthesis.The framework combines reconstruction geometry with the generalization capability of video generative models.Source: §1 · pp. 1–2
  • PRG labels retain dense 3DGS appearance in reliable regions, use LiDAR geometry for degraded regions with structure, and give the video prior more freedom where reconstruction support is insufficient.This routing adapts the use of appearance, geometry, and generative priors to each region's reconstruction state.Source: §1 · p. 2, Figure 1 · p. 2
  • On Waymo, PRG-Fusion outperforms existing methods in novel-trajectory synthesis, with superior visual quality, geometric fidelity, and view consistency.A two-stage strategy first establishes geometric control from LiDAR projections, then introduces 3DGS appearance cues to encourage complementary control.Source: §1 · p. 2

2. Related Work

Prior driving-scene reconstruction is limited by camera coverage, while generative driving methods offer controllability but can lack reliable geometric support. PRG-Fusion differs from reconstruction-centric uses of generative priors by routing them as part of video generation according to local reconstruction evidence.

2.1. Driving Scene Reconstruction

  • Driving-scene reconstruction methods including StreetSurf [10], EmerNeRF [36], Street Gaussians [34], DrivingGaussian [45], S3Gaussian, and OmniRe [7] produce faithful near recorded views, but large trajectory shifts expose distortions, artifacts, missing content, and temporal inconsistency.These methods extend neural fields to dynamic street scenes or model actors compositionally, yet rendering quality remains bounded by camera coverage.Source: §2.1 · pp. 2–3

2.2. Generative Models for Autonomous Driving

  • Driving video models such as MagicDrive [8], VISTA [9], Cosmos [1], and OmniNWM [17] enable controllable generation, but sparse or occluded geometric conditions leave some target structure and appearance difficult to control.FreeVS [29], StreetCrafter [35], and DiST-4D [11] use scene priors, LiDAR geometry, or metric depth, while unsupported regions may rely on generative priors and produce inconsistent content or structures.Source: §2.2 · p. 3

2.3. Scene Reconstruction with Generative Priors

  • Reconstruction-centric methods use generative priors to refine unreliable 3D content, whereas PRG-Fusion converts reconstruction evidence into region-aware control signals for video generation.Examples include ReconDreamer [24], ReconDreamer++ [43], DriveX, Difix3D+ [31], and FaithFusion [30].Source: §2.3 · p. 3

3. Method

PRG-Fusion synthesizes novel-trajectory driving videos by combining 3DGS appearance, LiDAR geometry, and video priors through pixel-wise Preserve, Repair, and Generate labels. Separate visual and geometry branches are trained progressively and routed according to local reconstruction state.

3. Method

  • PRG-Fusion derives pixel-wise Preserve, Repair, and Generate labels from reconstruction evidence to adaptively combine 3DGS renderings, LiDAR geometry, and video priors.The goal is temporally consistent synthesis that preserves scene-specific appearance and geometric faithfulness under viewpoint changes.Source: §3 · p. 3

3.1. Preliminaries

  • Flow Matching trains a conditional video model to transport Gaussian noise toward video latents by predicting velocity at sampled timesteps.The condition can include reference frames or other control signals.Source: §3.1 · p. 3

3.2. Reconstruction Evidence Extraction

  • Unreliability compares target-view Gaussian sensitivity with accumulated training-view sensitivity, while support uses visibility, viewing-direction coverage, and depth consistency.The observation and geometric support are combined using alpha-compositing weights.Source: §3.2 · pp. 4–5
  • Low-unreliability pixels become Preserve; among the remaining pixels, support distinguishes Repair from Generate.This hierarchy retains reliable reconstruction, uses evidence-guided correction where supported, and delegates unsupported content to the video prior.Source: §3.2 · p. 5

3.3. Dual-Branch Conditional Video Generation

  • A visual branch encodes 3DGS appearance and a geometry branch encodes LiDAR projections; progressive training learns geometry first, then appearance with the geometry branch frozen.The staged strategy addresses the risk that dense 3DGS cues dominate sparse LiDAR conditioning.Source: §3.3 · pp. 5–6

3.4. PRG-Guided Fusion

  • PRG labels route visual and geometry branch residuals token-wise: Preserve favors visual guidance, Repair geometry, and Generate reduces reconstruction guidance.The routed residuals are fused and injected into corresponding frozen DiT layers.Source: §3.4 · p. 6

4. Experiments

Experiments compare PRG-Fusion with reconstruction-, generation-, and restoration-based approaches on Waymo using novel-view quality and view-consistency metrics. Results favor PRG-Fusion across quantitative and qualitative comparisons, while ablations show the gains from PRG-guided region-adaptive fusion.

4.1. Experimental Setup

  • On Waymo, experiments use 114 scenes and 684 aligned triplets, comparing reconstruction-, generation-, and restoration-based methods with novel-view and view-consistency metrics.Source: §4.1 · pp. 6–7

4.2. Comparison Results

  • At 3 m, PRG-Fusion leads NTA-IoU, NTL-IoU, and FID; at 6 m, it keeps competitive NTA-IoU, leads NTL-IoU, cuts FID by 8.81, and leads FVD and CLIP-V.Source: Table 1 · p. 7, §4.2 · p. 7, Table 2 · p. 7
  • Across both offsets, PRG-Fusion better preserves vehicle appearance, lane structures, and scene layout; across frames it maintains identity, lane topology, and distant structure better than competitors.Source: §4.2 · pp. 7–8, Figure 5 · p. 7

4.3. Ablation Study

  • LiDAR conditioning chiefly improves NTL-IoU and road geometry, whereas 3DGS conditioning markedly improves NTA-IoU and scene-specific appearance; joint conditioning combines these strengths but uniform fusion can conflict.Source: §4.3 · p. 8, Figure 6 · p. 8
  • PRG-guided fusion achieves the best NTA-IoU and NTL-IoU among variants, with clearer lane structures and object boundaries while preserving vehicle and scene appearance.Source: Table 3 · p. 8, §4.3 · p. 8

5. Conclusion

PRG-Fusion uses pixel-wise Preserve, Repair, and Generate labels to coordinate 3DGS appearance, LiDAR geometry, and video priors according to local reconstruction evidence. Its current reliance on scene-specific 3DGS and LiDAR adds preprocessing overhead.

5. Conclusion

  • PRG-Fusion coordinates 3DGS appearance, LiDAR geometry, and video priors by local reconstruction state, improving visual quality, geometric fidelity, and view consistency.The framework uses a dual-branch control architecture and two-stage progressive training.Source: §5 · p. 8
  • The current framework relies on scene-specific 3DGS reconstruction and LiDAR geometry, adding preprocessing overhead; future work considers feed-forward reconstruction and predicted depth.Source: §5 · p. 8

A. Additional PRG Extraction Details

  • The method constructs pixel-wise PRG label sequences from a reconstructed 3DGS scene, training views, and a target camera trajectory.The labels encode Preserve, Repair, or Generate for each target-view pixel.Source: A. Additional PRG Extraction Details · p. 12

A.1. Gaussian-Level Evidence

  • Gaussian-level evidence combines visibility frequency with directional coverage, while global and local statistics capture persistent unreliability and trajectory-dependent changes.The local estimate uses 32-frame windows with a stride of 16; the global term stabilizes the estimate, and directional coverage provides bounded modulation.Source: §A.1 · p. 12
  • Rendering-gradient sensitivity measures how strongly target rendering depends on weakly constrained Gaussians, with sky gradients masked and responses robustly normalized.The responses are normalized using the 5th and 95th percentiles.Source: §A.1 · p. 12

A.2. Pixel-Level Evidence and PRG Assignment

  • Pixel-level support checks geometric consistency by back-projecting target pixels from rendered depth into training views and comparing relative depths.Invalid projections, non-positive depths, and invalid-opacity pixels receive zero support; the consistency check uses four-times downsampled resolution.Source: §A.2 · p. 12
  • PRG assignment preserves reliable pixels, repairs unreliable but sufficiently supported pixels, and generates content for weakly supported pixels.The decision rule excludes sky and the ego vehicle and uses unreliability and reconstruction-support thresholds of 0.25.Source: §A.2 · p. 13

B.1. Training Data and Condition Construction

Training data are constructed as aligned triplets of rendered 3DGS conditions, projected LiDAR geometry, and synchronized real video. The two control branches follow a two-stage progressive training strategy.

B.1. Training Data and Condition Construction

  • Each training example aligns a target-trajectory 3DGS rendering and projected LiDAR observations with the synchronized RGB video.The LiDAR observations are aggregated and colorized before projection into the target camera sequence.Source: §B.1 · p. 13
  • The training set comprises 684 aligned triplets from 114 Waymo scenes.Source: §B.1 · p. 13
  • The two control branches are trained using the two-stage progressive strategy described in Section 3.3.Source: §B.1 · p. 13

B.2. Training and Inference Settings

The setup uses a frozen video DiT backbone with specified clip and LiDAR inputs. PRG routing parameters are fixed across evaluation scenes and control layers, with branch and region roles specified.

B.2. Training and Inference Settings

  • The frozen Cosmos Transfer 2.5 video DiT processes 93-frame clips at 10 fps and 960×640 resolution, while LiDAR conditions aggregate five sweeps from five cameras.Source: §B.2 · p. 13

B.3. PRG Routing Parameters

  • PRG thresholds and routing parameters are fixed globally and shared across evaluation scenes and control layers.Source: §B.3 · p. 13
  • The routing-parameter matrix rows represent the visual and geometry branches, and its columns represent Preserve, Repair, and Generate.Source: §B.3 · p. 13

C.1. Evaluation Scenes

Evaluation uses eight Waymo validation scenes selected for substantial dynamic-agent motion, with scene identifiers and frame ranges reported for reproducibility.

C.1. Evaluation Scenes

  • The evaluation uses eight Waymo validation scenes with substantial dynamic-agent motion, and reports their identifiers and inclusive frame ranges for reproducibility.Source: §C.1 · p. 13

C.2. Novel-View Quality Metrics

Novel-view quality is assessed through agent and lane geometry and frame-level visual fidelity, while view consistency is measured at video-distribution and semantic levels.

C.2. Novel-View Quality Metrics

  • NTA-IoU compares detected agent boxes with projected Waymo 3D boxes, while NTL-IoU compares detected lanes with projected Waymo HDMap geometry.Higher values indicate better preservation of agent positions and extents or static road geometry, respectively.Source: §C.2 · p. 13
  • Figure 7 compares lane-change, 3 m lateral-shift, and 6 m lateral-shift examples; its GT column is the original-trajectory reference, not target-view pixel ground truth.FreeVS is shown at its officially released cropped resolution, excluding the sky region.Source: Figure 7 · p. 14
  • Lower FID indicates higher frame-level visual fidelity by comparing Inception feature distributions of synthesized frames and real evaluation-scene frames.This distribution-level metric does not require pixel-aligned RGB ground truth at shifted viewpoints.Source: §C.2 · p. 14

C.3. View Consistency Metrics

  • FVD-R3D compares source and synthesized video feature distributions, while CLIP-V averages cosine similarity between temporally corresponding frames; lower FVD-R3D and higher CLIP-V indicate stronger consistency.FVD-R3D uses R3D-18 features [27], and CLIP-V uses CLIP image features [25].Source: §C.3 · p. 14

D. Additional Qualitative Results

Figure 7 presents additional qualitative comparisons across all three novel-trajectory settings.

D. Additional Qualitative Results

Related papers

Loading 2609.06948v1…