GeoSim: Realistic Video Simulation via Geometry-Aware Composition for Self-Driving

Motivation

• Existing image-editing methods improve visual realism but generally lack an underlying realistic 3D world, limiting direct geometric consistency.Physics-based rendering also requires substantial computation, manual asset creation, and physical modeling, while real-time engines retain a realism gap.Source: §1 · p. 1

Contribution

• GeoSim inserts dynamic objects into existing videos using a realistic image-manipulation framework that combines data-driven methods with computer graphics.The approach leverages bounding-box annotations and self-driving sensor data to build textured 3D assets while retaining physically grounded simulation.Source: §1 · pp. 1–2
• GeoSim constructs a large-scale, fully textured 3D asset bank from self-driving fleet data despite sparse observations, occlusions, limited viewpoints, and lighting changes.The reconstruction procedure enforces temporal consistency and uses a learned shape prior to regularize assets.Source: §1 · p. 2

Applications

• GeoSim supports long-range multi-camera video simulation and synthetic labeled-data generation for self-driving perception.The authors report realistic, dynamically plausible, geometrically consistent outputs and gains on both qualitative and quantitative realism metrics.Source: §1 · p. 2
• GeoSim-augmented images improve perception performance, supporting applications including data augmentation, safety verification, Sim2Real, augmented reality, and automatic video editing.These application areas are presented as potential uses suggested by the experiments.Source: §1 · p. 2

Simulation for Robot Learning

• Urban driving simulation is difficult because outdoor scenes involve large scale, weather and lighting variation, fast-moving objects, and large sensor-viewpoint changes.Manual 3D asset design also limits scene diversity and leaves a Real2Sim gap.Source: §2 · p. 2

Image Synthesis and Manipulation

• Image synthesis methods can produce high-resolution images but often show texture and object-shape artifacts, while purely network-based methods struggle with complex physics such as lighting changes.These methods commonly generate images from scene graphs, normals, semantic segmentations, styles, or trajectories.Source: §2 · p. 2
• GeoSim combines image-based rendering and neural in-painting with real-world data to adjust inserted-object appearance and produce 3D-layout-aware scene modifications.This design extends graphics-aware image manipulation toward controllable scene editing.Source: §2 · p. 3

Video Synthesis and Manipulation

• Video synthesis methods enforce temporal consistency using sequential masks, depth maps, or trajectories, but GeoSim targets dynamic-object insertion through geometry-aware composition.The related video-synthesis literature is framed as an alternative route to realistic video scenario generation.Source: §2 · p. 3

3D Reconstruction and View Synthesis

• GeoSim reconstructs dynamic-object 3D assets from camera imagery and LiDAR to synthesize novel views, drawing on established image-based rendering and 3D-reconstruction approaches.The reconstruction step connects sensor data, geometry, appearance, and novel-view rendering.Source: §2 · p. 3, Figure 2 · p. 3

3.1. Multi-Sensor 3D Asset Reconstruction

• GeoSim reconstructs realistic dynamic-object assets from multi-view camera images and LiDAR, avoiding artist-created assets and requiring no ground-truth 3D shapes.The method uses readily available 3D bounding-box annotations and stores mesh geometry, appearance images, and silhouettes for later novel-view warping.Source: §3 · p. 3, §3.1 · pp. 3–4
• The asset-reconstruction problem is challenging because objects are sparsely observed, occluded, viewed from limited angles, and distorted by lighting and shadows.GeoSim addresses these conditions with a learning-based multi-view, multi-sensor approach trained from weak annotations.Source: §3.1 · p. 3
• The reconstruction network processes camera crops and aggregated LiDAR points in two streams, fuses their features, and outputs the object’s 3D shape.The shape is represented as a mesh with vertices and faces, while camera images and silhouettes encode appearance.Source: §3.1 · p. 4

3.2. Self-Supervised Learning

• Self-supervised training enforces agreement between the predicted 3D mesh and camera silhouettes and LiDAR observations without direct shape supervision.The silhouette loss compares rendered masks with automatically generated silhouettes, while the LiDAR loss measures consistency with accumulated point clouds.Source: §3.2 · p. 4
• Shape regularizers impose local smoothness and discourage implausible geometry through Laplacian, normal, and edge penalties.These terms preserve local geometry, smooth surfaces, and penalize long edges.Source: §3.2 · p. 4

4.1. Scenario Generation

• GeoSim selects object placements from lane geometry, traffic flow, visibility, and collision constraints, then retrieves compatible assets using viewpoint and camera-distance similarity.Placement samples are drawn from lanes within the camera field of view, while retrieved object shapes undergo collision checking.Source: §4.1 · p. 5

4.2. Occlusion-Aware Neural Rendering

• The method warps a selected object from its source view into a target pose by rendering its 3D mesh at the target viewpoint and generating a texture through inverse warping.The rendered target depth and source image are combined using the source and target camera matrices.Source: §4.2 · p. 5
• Shadow generation renders scenes with and without inserted objects under a cloudy HDRI, then blends background intensities using the rendered-image ratio.This approximates shadows and ambient occlusion beyond the inserted object’s pixels.Source: §4.2 · p. 5
• Occlusion masks compare the inserted object’s rendered depth with a dense target-scene depth map estimated from RGB imagery and projected LiDAR.An object pixel is occluded when the target-scene depth is smaller than the corresponding object depth.Source: §4.2 · pp. 5–6
• A synthesis network blends the target background, rendered object, and binary silhouette to correct illumination, color, boundaries, and missing source-view regions.The network follows occlusion reasoning as the final composition stage.Source: §4.2 · p. 6, Figure 4 · p. 6

Experimental Details

• The evaluation uses a large reconstructed vehicle asset bank and compares GeoSim against holistic, retrieval-based, guided-editing, and CAD-based baselines.The asset bank contains approximately 8,000 vehicles, while the baselines require object masks or use CAD insertion.Source: §5.1 · pp. 6–7

Perceptual Quality Evaluation

• GeoSim outperforms competing image simulation methods on human realism judgments and FID, with qualitative results showing fewer appearance and geometric artifacts.Human A/B testing, automatic FID, and qualitative comparisons evaluate realism; the supplied table captions identify the perceptual evaluation scope but not cell values.Source: §5.2 · p. 7, Table 1 · p. 8
• The hybrid geometry-aware rendering approach significantly outperforms alternative rendering options on human scores, while shadows improve human-rated realism without improving FID.This ablation indicates a gap between computational perceptual metrics and human criteria.Source: §5.2 · pp. 7–8, Table 2 · p. 8

Video Simulation

• GeoSim generates highly realistic, temporally consistent video across multiple cameras and is evaluated on both UrbanData and Argoverse.Argoverse uses GeoSim assets built from UrbanData because its dataset is too small for diverse asset creation.Source: §5.1 · p. 6, §5.2 · p. 8

Limitations

• GeoSim failures include incorrect occlusions, irregular meshes, inaccurate poses, illumination mismatches, and blank pixels in long-range video simulation.The long-range artifacts arise when textures are inverse-warped from source viewpoints far from the target viewpoint.Source: §5.2 · p. 8

Downstream Perception Task

• Adding 9,879 GeoSim-generated training examples improves PSPNet car segmentation by 3.4% and overall mIoU by 1.8%, with consistent gains across two segmentation models.For DeepLabv3, the corresponding gains are 0.4% for cars and 0.2% for overall mIoU.Source: §5.3 · p. 8, Table 4 · p. 8

• GeoSim combines geometry-guided object placement and rendering to generate flexible synthetic scenes with greater visual realism than competing alternatives.The procedure accounts for physical realism in dynamic object placement and bypasses manual 3D asset creation.Source: §6 · p. 8
• The approach improves downstream semantic segmentation performance and motivates applications including sim2real and autonomous-system evaluation.Source: §6 · p. 8

3D reconstruction network

• The reconstruction network processes cropped multi-view images and LiDAR sweeps, extracting image features before combining them with point-cloud features.Images are resized to 256 × 256, processed by a convolutional backbone, and represented as feature vectors.Source: §A.1 · p. 12, Figure 8 · p. 13
• Multi-view image features are fused through max pooling, concatenation, and multilayer perceptrons before producing the image-based representation.Source: §A.1 · p. 12
• Mesh reconstruction is supervised by silhouette consistency, LiDAR consistency, and regularization terms enforcing smoothness and local geometric structure.Edge, Laplacian, and normal regularizers penalize long edges, preserve local geometry, and smooth neighboring surface normals.Source: §A.1 · p. 12

Segment retrieval

• Single-view retrieval removes candidates with viewpoint changes above 10° and samples remaining objects using inverse scores based on view angle and camera distance.Source: §A.2 · p. 12
• Multi-view retrieval filters source objects with view-range overlap below 20° and ranks candidates using overlap and minimum source distance.Sampling remains categorical and weighted by inverse score rather than selecting only the highest-ranked candidate.Source: §A.2 · pp. 12–13

Depth completion

• Depth completion produces a dense background depth map so rendered object depths can determine occlusion relations during composition.The depth model uses camera images, sparse projected depth, sparse-depth masks, and a dilated sparse-depth map as input.Source: §A.3 · p. 13

Shadow generation

• Shadows are generated by rendering a virtual scene containing the inserted object and a ground plane, approximating scene changes beyond the object pixels.The procedure uses image-based rendering because geometry is available for the inserted object.Source: §A.4 · p. 13, Figure 9 · p. 14

Synthesis network

• The synthesis network is trained with augmented object segments and masks that mimic color inconsistency, missing texture, and imperfect boundaries at inference.Inputs include a scene image, sampled vehicle mask, and corresponding RGB segment.Source: §A.5 · p. 14, Figure 10 · p. 14
• At inference, the network receives a warped and occlusion-reasoned composition crop, then returns a blended result pasted back into the original image location.The crop is at least 128 × 128 and at most 1024 × 1024, with the mask dilated before synthesis.Source: §A.5 · p. 15

Video simulation

• Video simulation augments five-second snippets with up to five inserted objects whose initial placements follow the camera field of view and local lane graph.Each snippet contains 50 consecutive frames sampled at 10Hz.Source: §A.6 · p. 15

Object retrieval

• Object retrieval filters candidates by view-range overlap and ranks them by overlap and camera distance before inverse-score sampling.Source: §A.6 · p. 15

Video simulation

• Heuristic behavior, lateral, and longitudinal models refine object paths into trajectories containing location, orientation, velocity, and acceleration for realistic braking and acceleration.The system uses a cost-based lane-change model, a lane-centering lateral heuristic, and the Intelligent Driver Model for longitudinal motion.Source: §A.6 · pp. 15–16

Dataset breakdown

• UrbanData contains roughly 16.5K labeled snippets, including 12K with manually annotated 2D segmentation maps from one frame per snippet.Source: §A.7 · p. 16
• The dataset is divided into multiple training and testing splits for GeoSim components and baselines, with task-specific assignments documented in the experimental setting.Source: §A.7 · p. 16

A/B evaluation

• The human study uses pairwise image comparisons in which users select the more realistic image while considering appearance, lighting, shadows, occlusion, and scene consistency.Source: §A.8 · p. 16, Figure 11 · p. 17
• GeoSim was preferred in 211 of 279 pairs in the 2D synthesis ablation, with a maximum human-evaluation p-value of 1.64e-18 across comparisons.Source: §A.8 · p. 16

• GeoSim produces more realistic and 3D-aware simulated images than other simulation methods.The comparisons highlight blurred textures, implausible placements, distorted shapes, and boundary artifacts in competing approaches.Source: §B.1 · p. 17

Post-processing components

• Removing occlusion reasoning, shadows, or SynNet significantly reduces synthesis realism.Without occlusion reasoning, vehicles fail to conform to existing scene elements; removing shadows makes vehicles appear off the ground.Source: §B.2 · p. 17

Lane-map ablation

• GeoSim is preferred by humans 95.5% of the time over random ground-location and orientation sampling.Random sampling produces useful edge cases but lacks temporal consistency, limiting its suitability for video simulation.Source: §B.2 · p. 17

Synthesis-network ablation

• Users prefer full GeoSim over the version without the image-synthesis network 95.0% of the time.Source: §B.2 · p. 17

Failure cases

• GeoSim’s main failure cases are incorrect occlusion relationships, irregular reconstructed meshes, inaccurate object poses, and illumination differences.Pose errors are usually caused by map errors, while illumination failures arise from mismatches between rendered segments and target scenes.Source: §B.3 · p. 17