PaperlayerView exact revision on arXiv

Source-linked AI summary

RECAST: From Log Replay to Closed-Loop Driving Simulation with View-Complete Actors

Zijun Zhao, Liewen Liao, Kang Shen, Songan Zhang, Ming Yang

arXiv:2609.31374v1cs.ROcs.CV

TL;DR

Driving simulation beyond recorded trajectories exposes vehicle views missing from sparse logs, where generated or reconstructed actors may render unrealistically. RECAST creates view-complete actors from single logged observations and integrates them into scenes for planner-in-the-loop simulation; compared with native actors, it improves reported scene-rendering and closed-loop outcomes.

  • Problem

    Sparse logged observations leave unseen vehicle views, and image-to-3D methods can produce unrealistic vehicles at those viewpoints that interfere with planner visual input.

    Source: §I · p. 1

  • Method

    RECAST generates a view-complete actor from a single segmented vehicle observation, registers it in the reconstructed scene, and uses RECAR-based adaptation for real-log inputs.

    Source: §I · pp. 1–2, B. View-Complete Actor Generation · p. 3

  • Results

    RECAST actors achieve a 63.0% NC rate versus 22.2% with native Street Gaussians actors, while scene FDincep is 112.10 versus 110.00 for Asset Harvester.

    Source: B. Q1: Closed-Loop Simulation with Ego… · p. 5, C. Q2: Scene-Level Rendering Quality · p. 6

  • Takeaways & Limitations

    With a fixed GTRS-Dense planner, closed-loop outcomes depend on rendered observations as well as the planner, underscoring actor rendering’s importance in simulation-based evaluation.

    Source: §V · p. 7

Abstract

from arXiv · show

Closed-loop driving simulation requires rendered observations to remain reliable as the ego vehicle and surrounding actors move beyond their recorded trajectories, exposing views absent from the source log. Existing data-driven simulators reconstruct dynamic actors from sparse observations, which can result in rendering artifacts under these viewpoint changes. We introduce RECAST (REconstructing Controllable Actors for Simulation and Testing), a 3D Gaussian Splatting framework that generates a view-complete actor from a single segmented vehicle observation in a driving log and registers the generated actor in the reconstructed scene. RECAST supports planner-in-the-loop rendering under controlled ego-actor interactions. To adapt an image-to-3D prior to real vehicles, we further introduce RECAR, a dataset of approximately 20K real vehicles with 600K background-free RGBA images spanning diverse vehicle colors and types. We use two-stage adaptation to improve vehicle generation from real driving-log observations. At the actor level, RECAST reduces $\mathrm{FD}_{\mathrm{incep}}$ from 9.788 to 7.992 relative to unadapted TRELLIS. At the scene level, under actor motion beyond logged trajectories, RECAST reduces $\mathrm{FD}_{\mathrm{incep}}$ from 129.35 to 112.10 and increases $\mathrm{CLIP}_{\mathrm{margin}}$ ($\times1000$) from 0.14 to 3.47 relative to Street Gaussians. We demonstrate planner-in-the-loop simulation with the image-conditioned planner GTRS-Dense. Compared with native Street Gaussians actors, RECAST increases the no-collision (NC) rate from 22.2% (12/54) to 63.0% (34/54) and the mean minimum predicted time-to-collision (TTC) from 0.798 s to 2.150 s. These experiments show that RECAST supports closed-loop planner evaluation under controlled ego-actor interactions beyond log replay. Visit our project page at https://zijunkr.github.io/RECAST/

I. INTRODUCTION

Driving-scene reconstructions are often evaluated near logged viewpoints, while trajectory departures expose unseen vehicle surfaces and rendering artifacts. RECAST generates view-complete actors from single logged observations and integrates them into scenes for controlled planner-in-the-loop simulation.

I. INTRODUCTION

  • Driving-scene reconstruction methods are typically evaluated under log replay or small camera translations, with actors remaining near recorded states.Source: §I · p. 1
  • Trajectory departures expose unseen vehicle sides, where image-to-3D methods can produce unrealistic views that interfere with a planner’s visual input.Sparse logs constrain observed vehicle views, motivating view-complete actors for interactive simulation.Source: §I · p. 1
  • RECAST generates a view-complete actor from one segmented logged vehicle observation and registers it in the reconstructed scene for simulation.The framework supports controllable ego and actor motion and renders observations from the current simulation state.Source: §I · p. 1
  • RECAST integrates generated actors into reconstructed real-world scenes for controlled ego–actor interactions and planner-in-the-loop execution.Source: §I · p. 2
  • RECAR provides approximately 20K real vehicles and 600K background-free RGBA images for a two-stage adaptation strategy targeting generation from a single segmented driving-log image.Source: §I · p. 2

II. RELATED WORK

Prior work generates vehicle assets or reconstructs driving scenes, but sparse observations leave unseen surfaces weakly constrained, while library insertion substitutes independently captured vehicles. RECAST adapts an image-to-3D prior with vehicle-specific supervision to generate target actors from single logged observations for controlled simulation.

A. 3D Vehicle Generation

  • General-purpose image-to-3D methods directly predict structured 3D representations, but are not explicitly adapted to sparse observations from real driving logs.Novel-view synthesis methods produce unseen views but require an additional lifting stage to obtain manipulable 3D actors.Source: A. 3D Vehicle Generation · p. 2
  • RECAST adapts a structured image-to-3D prior with vehicle-specific supervision to generate view-complete actors from single driving-log observations and integrate them into controlled simulation.Source: A. 3D Vehicle Generation · p. 2

B. Driving Scene Reconstruction and Simulation

  • NeRF and 3D Gaussian Splatting methods reconstruct driving scenes for novel-view rendering, with recent 3DGS approaches separating static backgrounds and individual actors.Source: B. Driving Scene Reconstruction and… · p. 2
  • Scene reconstruction methods preserve actor identity but leave unseen vehicle surfaces weakly constrained because actor representations are optimized from original-sequence observations.AutoSplat uses template initialization and reflected Gaussian consistency, but these priors provide limited instance-specific appearance cues for unseen surfaces.Source: B. Driving Scene Reconstruction and… · p. 2
  • HUGSIM and MADrive provide view-complete assets under large viewpoint changes but replace the target actor with an independently captured vehicle.RECAST instead generates a view-complete actor from an observation of the target vehicle.Source: B. Driving Scene Reconstruction and… · p. 2

A. Framework Overview

RECAST builds a view-complete Gaussian actor from one segmented driving-log observation, registers it in the reconstructed scene, and supports controllable closed-loop simulation. RECAR and two-stage adaptation provide vehicle imagery and training targets for generating actors from sparse observations.

A. Framework Overview

  • RECAST generates a view-complete actor from one segmented vehicle observation, registers it with the logged 3D box and LiDAR points, and inserts it into the reconstructed background.During simulation, the ego controller executes the planner’s predicted trajectory while the actor controller follows specified motion.Source: A. Framework Overview · p. 2, Figure 2 · p. 3

B. View-Complete Actor Generation

  • RECAR provides 20K real vehicles as 600K background-free RGBA images at 1200 × 800 resolution, organized into 360° sequences with fixed elevation and regular azimuth sampling.The regularly sampled, background-free views support multi-view pseudo-target construction.Source: B. View-Complete Actor Generation · p. 2
  • Stage I fine-tunes TRELLIS’s sparse-structure and SLAT flow transformers on vehicle geometry and appearance, using a single clean RGBA image as conditioning.Generated assets are selected using aesthetic scores above 4.5 and manual screening before their latent representations are cached as pseudo-targets.Source: B. View-Complete Actor Generation · p. 3
  • Stage II retains clean conditioning images with probability ρ and otherwise applies synthetic perturbations that mimic driving-log imagery and imperfect foreground masks.The perturbation changes the conditioning observation while cached pseudo-targets remain unchanged.Source: B. View-Complete Actor Generation · p. 3
  • At inference, the adapted model converts a single-view RGBA crop into sparse structure and latent features, then decodes them into a view-complete 3D Gaussian actor.Grounding DINO localizes the actor and box-prompted SAM provides its alpha mask.Source: B. View-Complete Actor Generation · pp. 3–4

C. Actor Registration and Insertion

RECAST aligns the generated actor to the logged vehicle using its 3D box and accumulated LiDAR points, then replaces the logged actor while retaining its world-pose mapping.

C. Actor Registration and Insertion

  • Constrained ICP aligns generated Gaussian centers to actor-local LiDAR points by estimating isotropic scale, yaw offset, and 3D translation.The logged 3D box provides initial scale and orientation; LiDAR points are aggregated across visible frames.Source: C. Actor Registration and Insertion · p. 4
  • RECAST replaces the logged actor geometry with the registered view-complete actor while retaining the actor-local-to-world pose mapping.A flattened black Gaussian primitive approximates the contact shadow and shares the actor’s registration transform.Source: C. Actor Registration and Insertion · p. 4

D. Closed-Loop Simulation

RECAST updates the scene from current ego and actor states, renders the planner’s requested input, and advances both vehicles over an execution interval. Actor motion is specified by a path and heading, with speed control tracking the desired speed.

D. Closed-Loop Simulation

  • For image-conditioned planning, the renderer produces camera images from the current scene and camera poses; state-conditioned planning uses ego history, actor states, the HD map, and navigation input.The planner input modality determines which information is supplied.Source: D. Closed-Loop Simulation · p. 4
  • The planner predicts an ego trajectory, and the ego controller executes its first K waypoints while advancing ego state and camera poses over Δt.The next planner input uses updated states, or rendered images with ego state for image-conditioned planning.Source: D. Closed-Loop Simulation · p. 4
  • The actor follows a specified path and heading, while an IDM-based speed controller tracks the desired speed and the simulator updates its state over the same interval as the ego.The actor state comprises position, speed, and heading.Source: D. Closed-Loop Simulation · p. 4

IV. EXPERIMENTS

The experiments examine whether RECAST enables closed-loop render–plan–update simulation, improves scene-level rendering fidelity under controlled ego–actor interactions, and benefits single-image vehicle generation from adaptation.

IV. EXPERIMENTS

  • Q1 asks whether RECAST supports closed-loop render–plan–update simulation with a view-complete actor generated from a logged image.Source: §IV · p. 4
  • Q2 compares scene-level rendering fidelity under controlled ego–actor interactions against native actors and alternative actor-generation methods using the same reconstructed background.Source: §IV · p. 4
  • Q3 asks whether vehicle-domain adaptation on RECAR and driving-log-robust adaptation improve single-image vehicle generation.Source: §IV · p. 4

A. Experimental Setup

The experiments use a controlled-interaction benchmark built from Waymo scenes and a filtered RECAR vehicle dataset. Scene-level rendering and adaptation use specified training, rendering, and rollout settings.

A. Experimental Setup

  • Nine Waymo scenes each contribute one target vehicle, with other dynamic actors removed so controlled target motion exposes the vehicle from different ego viewpoints.The logged sequence supplies reference observations.Source: A. Experimental Setup · p. 4
  • After quality filtering, RECAR contains 12,896 vehicle identities: 12,248 training instances and 648 held-out test instances.Source: A. Experimental Setup · p. 4
  • Scene experiments use Street Gaussians, render three forward-facing Waymo cameras at 960 × 640, and run 160-frame rollouts over 16 seconds at 0.1-second steps.Each scene is optimized for 30,000 iterations on one NVIDIA A40 GPU.Source: A. Experimental Setup · p. 4
  • Vehicle-domain adaptation trains TRELLIS structure and SLAT models, then continues training with driving-log corruptions such as blur, low light, fog, occlusion, and cropping.The follow-on adaptation runs for 12,000 steps at a learning rate of 2 × 10−6.Source: A. Experimental Setup · pp. 4–5

B. Q1: Closed-Loop Simulation with Ego Planners

With the same GTRS-Dense planner and matched interactions, RECAST actors improve several closed-loop outcomes over native Street Gaussians actors. The evaluation measures collision avoidance, drivable-area compliance, timing, route progress, and comfort.

B. Q1: Closed-Loop Simulation with Ego Planners

  • GTRS-Dense receives current multicamera renderings, predicts 40 waypoints over four seconds, and executes the first five before the next planning cycle.The evaluation uses three actor motions and two initial-distance and motion-parameter configurations, with matched target states across methods.Source: B. Q1: Closed-Loop Simulation with Ego… · p. 5
  • NC, DAC, and TTC assess safety, while route completion and comfort measure progress and motion smoothness.TTC is the minimum predicted collision time across 32 cycles, capped at four seconds and averaged over rollouts.Source: B. Q1: Closed-Loop Simulation with Ego… · p. 5
  • Paired 95% confidence intervals resample nine scenes 200,000 times while retaining all six matched rollouts per sampled scene.Source: B. Q1: Closed-Loop Simulation with Ego… · p. 5
  • 63.0% versus 22.2% NC with RECAST versus native Street Gaussians actors, a 40.7-percentage-point paired difference (95% CI: [25.93, 55.56]).TTC differs by 1.352 s (95% CI: [0.628, 2.046]) and COM by 4.7 percentage points (95% CI: [2.1, 7.5]), both higher with RECAST; route completion is 17.57% versus 16.82%.Source: B. Q1: Closed-Loop Simulation with Ego… · p. 5, Table II · p. 5
  • The planner’s outcomes vary with actor rendering because replacing the actor changes the visual observations used for its decisions, even when bounding boxes follow logged poses.Incomplete reconstructions can expose missing or distorted surfaces beyond logged trajectories, potentially altering planning decisions and complicating attribution of failures to the planner.Source: B. Q1: Closed-Loop Simulation with Ego… · p. 5

C. Q2: Scene-Level Rendering Quality

Under controlled Waymo interactions, RECAST improves several scene-level and vehicle-appearance measures over native Street Gaussians actors. The evaluation uses shared trajectories to isolate actor-rendering differences and distinguishes scene realism from reference-crop similarity.

C. Q2: Scene-Level Rendering Quality

  • The comparison includes four target-actor motion settings and uses cached ego and actor poses so methods share trajectories within each scenario.The evaluation retains three actor motions and adds an advancing motion in which the ego overtakes the actor.Source: C. Q2: Scene-Level Rendering Quality · pp. 5–6, Figure 4 · p. 6
  • FDincep assesses scene and actor-crop realism against real Waymo observations; CLIPimg and LPIPS compare reference vehicle crops without matched viewpoints.CLIPveh measures alignment with the prompt “vehicle,” while CLIPmargin and VQAmargin use positive-minus-negative prompt scores.Source: C. Q2: Scene-Level Rendering Quality · p. 6
  • With the Street Gaussians background fixed, RECAST reduces scene FDincep by 13.3% and target-crop FDincep by 35.6% relative to native actors.RECAST has the best CLIP and VQA margins, CLIPimg, CLIPveh, and LPIPS among compared actor representations; Asset Harvester has lower scene and target-crop FDincep values of 110.00 and 124.61, versus RECAST’s 112.10 and 124.74.Source: C. Q2: Scene-Level Rendering Quality · p. 6, Table III · p. 6
  • Scene FDincep compares image distributions, whereas reference-crop metrics compare individual vehicle appearances, so Asset Harvester’s lower FDincep does not establish a closer match to the target vehicle.Source: C. Q2: Scene-Level Rendering Quality · p. 6
  • RECAST’s reference-crop scores support appearance similarity, while qualitative examples show coherent vehicle shape across viewpoint changes.The viewpoint differences limit reference-crop scores as evidence of appearance similarity.Source: C. Q2: Scene-Level Rendering Quality · p. 6

D. Q3: Actor-Level Generation Quality

The section compares single-image vehicle generation methods and adaptation stages, finding that RECAST improves actor-level generation quality and appearance matching in views absent from the input.

D. Q3: Actor-Level Generation Quality

  • Each method receives one RGBA observation and renders four held-out views at azimuths 0°, 90°, 180°, and 270°.The target cameras use a 40° field of view and 15° pitch, with identical rendered and reference-view parameters.Source: D. Q3: Actor-Level Generation Quality · p. 6
  • RECAST outperforms evaluated baselines on all six metrics, reducing FDincep from 9.788 to 7.992 against unadapted TRELLIS.Table IV reports actor-level quantitative results on held-out data.Source: Table IV · p. 7, D. Q3: Actor-Level Generation Quality · p. 7
  • Stage I reduces FDincep from 9.788 to 8.176, and Stage II further reduces it to 7.992 while improving SSIM, LPIPS, CLIPimg, and FDdinov2 at unchanged PSNR.Stage I provides most gains on held-out RECAR; Stage II keeps pseudo-targets fixed while perturbing conditioning images, with fewer visible appearance errors in real driving-log examples.Source: D. Q3: Actor-Level Generation Quality · p. 7
  • Figure 5 shows more complete geometry and closer input-appearance matches; its first three examples are driving-log crops and last two are 3DRealCar.AH denotes Asset Harvester, and Ours w/o II denotes the model without driving-log-robust adaptation.Source: D. Q3: Actor-Level Generation Quality · p. 7, Figure 5 · p. 7

V. CONCLUSION

RECAST integrates view-complete vehicle actors generated from single driving-log images into reconstructed scenes for controlled planner-in-the-loop simulation. The paper reports improved actor generation and scene rendering, and notes that closed-loop outcomes depend on rendered observations as well as the planner.

V. CONCLUSION

  • RECAST integrates view-complete vehicle actors generated from single driving-log images into reconstructed scenes for controlled planner-in-the-loop simulation.Two-stage adaptation on RECAR improves actor generation quality, while scene-level experiments show improved rendering realism over native actors beyond logged trajectories.Source: §V · p. 7
  • With a fixed GTRS-Dense planner, closed-loop outcomes depend on rendered observations as well as the planner.The conclusion highlights actor rendering as important in simulation-based evaluation.Source: §V · p. 7
  • Future work will extend RECAST to pedestrians and other traffic actors and investigate training planners through controllable interactions in rare, safety-critical scenarios.The authors plan to assess whether this training improves robustness and transfers to real-world driving.Source: §V · p. 7

Related papers

Loading 2609.31374v1…