Source-linked AI summary
Editable Scene Simulation for Autonomous Driving via Collaborative LLM-Agents
Yuxi Wei, Zi Wang, Yifan Lu, Chenxin Xu, Changxing Liu, Hao Zhao, Siheng Chen, Yanfeng Wang
TL;DR
Customized autonomous-driving perception data is costly to collect, while existing scene-simulation approaches have limitations in interaction, rendering, and asset integration. ChatSim combines collaborative LLM agents with multi-camera rendering and lighting estimation for language-driven scene editing. Experiments report photo-realistic customized scenes and improvements in photo-realism and lighting errors.
Problem
Real-world perception data collection is costly, and existing editable scene-simulation approaches are limited in interaction efficiency, multi-camera photo-realistic rendering, and external-asset integration.
Source: §1 · p. 1Method
ChatSim combines collaborative LLM agents for language-driven editing with McNeRF multi-camera rendering and McLight multi-camera lighting estimation.
Source: §1 · p. 2, §4.1 · p. 5, §4.2 · p. 6Results
ChatSim generates customized photo-realistic scenes from complex language commands; wide-angle rendering improves photo-realism by 4.5%, while intensity error and angular error fall by 57.0% and 9.9%.
Source: §1 · p. 2Takeaways & Limitations
Experiments show ChatSim can simulate customized perception data, including dangerous corner cases, through human language commands.
Source: §1 · p. 2
Abstract
from arXiv · showhide
Scene simulation in autonomous driving has gained significant attention because of its huge potential for generating customized data. However, existing editable scene simulation approaches face limitations in terms of user interaction efficiency, multi-camera photo-realistic rendering and external digital assets integration. To address these challenges, this paper introduces ChatSim, the first system that enables editable photo-realistic 3D driving scene simulations via natural language commands with external digital assets. To enable editing with high command flexibility,~ChatSim leverages a large language model (LLM) agent collaboration framework. To generate photo-realistic outcomes, ChatSim employs a novel multi-camera neural radiance field method. Furthermore, to unleash the potential of extensive high-quality digital assets, ChatSim employs a novel multi-camera lighting estimation method to achieve scene-consistent assets' rendering. Our experiments on Waymo Open Dataset demonstrate that ChatSim can handle complex language commands and generate corresponding photo-realistic scene videos.
1. Introduction
Customized perception data is costly to collect, while existing scene-simulation approaches do not jointly provide flexible editing, photo-realistic view consistency, and external-asset integration. ChatSim combines language-driven editing with collaborative LLM agents, multi-camera rendering, and lighting estimation.
1. Introduction
- Real-world perception data collection can be prohibitively expensive, and reproducing specific scenarios from real instances is often nearly impossible.Customized scene simulation is described as an alternative for modeling specific conditions without the costs and logistical complexity of real-world collection.Source: §1 · p. 1
- Existing approaches fail to jointly meet needs for sophisticated commands, photo-realistic view-consistent outcomes, and realistic integration of external digital assets.Graphics engines have limited realism; image-generation methods struggle with view consistency and asset import; rendering systems demand extensive code-based editing and lack external assets.Source: §1 · pp. 1–2
- ChatSim is presented as a system for editing photo-realistic 3D driving scenes through natural-language commands with external digital assets.Users issue commands conversationally without intervening in intermediate simulation steps.Source: §1 · p. 2
- ChatSim uses specialized collaborative LLM agents to decompose complex simulation demands into editing tasks.The agents support natural-language editing and distribute tasks across specialized roles.Source: §1 · p. 2
- McNeRF uses multi-camera inputs and alignment and exposure-time adjustments to address pose misalignment and brightness inconsistency in rendering.Exposure times are incorporated to recover scene radiance in HDR and mitigate color differences where camera images meet.Source: §1 · p. 2
- McLight estimates skydome and surrounding illumination to render external assets with scene-consistent lighting.It models sun behavior and location-specific effects such as tree shade.Source: §1 · p. 2
- ChatSim improves photo-realism by 4.5% with wide-angle rendering and reduces intensity error and angular error by 57.0% and 9.9%, respectively.The experiments are conducted on the Waymo autonomous driving dataset.Source: §1 · p. 2
2. Related Work
Prior scene-simulation approaches span graphics engines, image generation, and scene rendering, with different limitations in realism, consistency, editing, and asset integration. ChatSim combines language-driven editing and external assets with multi-camera rendering and lighting estimation.
2. Related Work
- Graphics engines face a domain gap, image-generation methods struggle to maintain scene consistency, and rendering methods require intermediate user editing and lack external-asset support.These limitations motivate ChatSim’s language-driven editing and integration of external digital assets.Source: §2 · pp. 2–3
- ChatSim integrates McNeRF, a multi-camera neural radiance field designed for high-fidelity scene rendering.It is presented as part of a system for automatic simulation editing through language commands.Source: §2 · p. 3
- Few lighting-estimation methods consider multi-camera input, which is common in driving scenarios; McLight combines with McNeRF to estimate wider-range and spatially varying lighting.The approach targets realistic illumination effects on integrated assets.Source: §2 · p. 3
- ChatSim applies collaborative LLM agents to language-command editing of 3D driving scenes.Prior collaborative-agent work is cited as background for the paper’s use of this framework.Source: §2 · p. 3
3. Collaborative LLM-Agents for Editing
ChatSim uses specialized LLM agents to interpret and execute distinct editing tasks, then combines their outputs into a scene video. The workflow also records editing configurations for possible multi-round edits.
3. Collaborative LLM-Agents for Editing
- The project manager decomposes user commands and dispatches instructions, while specialized agents handle distinct editing tasks.Agents use prompts to turn commands into structured configurations, which their role functions process.Source: §3.1 · p. 3, §3 · p. 3
3.1. Specific Agent’s Functionality
- The view-adjustment agent converts natural-language viewpoint instructions into movement parameters and updated camera extrinsics.Its role functions turn movement parameters into transformation matrices applied to the original camera parameters.Source: §3.1 · p. 3
- The vehicle-deleting agent identifies requested vehicles by attributes and removes them using per-frame inpainting; the asset manager selects or modifies assets to match user specifications.The asset manager maintains a digital-asset bank and can modify an imperfect match, such as by changing its color.Source: §3.1 · p. 4
- The vehicle-motion agent extracts position and movement attributes from requests, maps placement to lane nodes, and plans trajectories using Bezier curves.Trajectory tracking is added to fit vehicle dynamics.Source: §3.1 · p. 4
- The foreground-rendering agent combines camera extrinsics, selected 3D assets, motion information, and McLight illumination to render foreground objects.The estimated illumination is used by the Blender API.Source: §3.1 · p. 4
3.2. Agent Collaboration Workflow
- The workflow renders background and foreground images through separate agent teams, then composes them into a video for the user.The project manager records each agent’s editing configuration for possible multi-round editing.Source: §3.2 · p. 5
4. Novel Rendering Methods
ChatSim pairs McNeRF for multi-camera background rendering with McLight for lighting external foreground assets. McNeRF addresses camera alignment and exposure differences, while McLight combines skydome and scene-specific illumination.
4. Novel Rendering Methods
- McNeRF supports background rendering, while McLight estimates lighting for foreground assets, combining two techniques to enhance simulation photo-realism.Source: §4 · p. 5
4.1. McNeRF for Background Rendering
- McNeRF aligns asynchronous camera poses and incorporates exposure times into HDR radiance fields to improve brightness consistency across camera views.Aligned poses also provide ray origins and directions for rendering and facilitate foreground-rendering operations.Source: §4.1 · p. 5
4.2. McLight for Foreground Rendering
- McLight reconstructs skydome lighting with a residual connection for sun-peak intensity and fuses multi-camera cues using self-attention.The residual design restores the sun’s peak intensity, supporting strong shadows for virtual objects.Source: §4.2 · p. 6
- McLight samples McNeRF’s scene lighting at each virtual-object position and blends it with skydome illumination, capturing spatially varying effects.The blend weights skydome intensity by the transmittance of McNeRF’s final sampling point.Source: §4.2 · pp. 6–7
5. Experimental Results
ChatSim handles complex, abstract, and context-dependent language edits, and its simulated data improves downstream detection across different amounts of real data. Component experiments report strong results for multi-camera rendering, lighting estimation, and text-driven motion generation.
5.1. Datasets and implementation details
- Waymo Open Dataset provides multi-camera images and calibrations; McLight uses 449 HDRIs for autoencoder training and HoliCity for its second stage.Experiments use three front-facing cameras, sample 40 frames per scene at 10 Hz, hold out 1/8 of 120 images, and use GPT-4.Source: §5.1 · p. 7
5.2. System results
- Against Visual Programming, ChatSim significantly outperforms in two deletion and replacement samples; Visual Programming supports only single-frame 2D image editing.Source: Figure 8 · p. 7, §5.2 · p. 8
- ChatSim executes mixed complex, highly abstract, and context-dependent multi-round commands, including a rare dangerous wrong-way chase scenario.The project manager agent records context across rounds.Source: §5.2 · p. 7
- Simulation augmentation improves detection across real-data amounts, aiding rough AP30 with little real data and fine-grained AP70 as real data increases.Figure 9 compares detection performance with and without simulated data under different real-data training amounts.Source: Figure 9 · p. 8, §5.2 · p. 8
5.3. Component results
- McNeRF achieves state-of-the-art PSNR, SSIM, and LPIPS on 32 scenes and enables fast rendering.Source: Table 3 · p. 8, §5.3 · p. 8
- McLight produces more accurate peak behavior and higher user preferences, with vehicle insertions showing realistic reflections and scene-consistent shadows.Evaluation includes relative intensity error on HDRI, angular error on HoliCity, and a user study.Source: Table 4 · p. 8, §5.3 · p. 9, Figure 11 · p. 9
- ChatSim's motion generation outperforms GPT2Motion and GPT2Code in the user study while keeping trajectories within lane boundaries at a high rate.Source: Table 5 · p. 9, §5.3 · p. 9
6. Conclusions and Limitations
ChatSim combines collaborative LLM agents with McNeRF and McLight for editable, photo-realistic 3D driving-scene simulation. The authors identify weather changes as a future extension.
6. Conclusions and Limitations
A. Supplementary Explanation of Table 2
- Table 2 accuracy averages success rates across three trials of 15 category-specific commands, with success requiring correct operations, configurations, and parameter values.Source: A. Supplementary Explanation of Table 2 · p. 13
B.1. Supplementary Experiments of Lighting Estimation.
The supplementary lighting experiment compares methods using merged multi-view inputs and reports that McLight still outperforms the baselines.
B.1. Supplementary Experiments of Lighting Estimation.
- Lighting-estimation baselines receive merged multi-view inputs represented as a wide-angle image.Source: §B.1 · p. 13
- McLight still significantly outperforms the lighting-estimation baselines in this comparison.Source: §B.1 · p. 13
- Intensity evaluation without multi-view inputs remains unchanged from Table 4.Source: §B.1 · p. 13
B.2. Supplementary Experiments of Background Rendering.
The supplementary experiments report that multi-camera alignment improves rendering consistently and McNeRF with exposure outperforms the other aligned baselines. The appendices also detail how agents implement and coordinate complex, abstract, and multi-round language commands.
B.2. Supplementary Experiments of Background Rendering.
- Multi-camera alignment consistently improves rendering, and McNeRF with exposure outperforms the other aligned baselines.The passage refers to ablations in a table but supplies no numerical values.Source: §B.2 · p. 13
C.1. Agent Implement Details
- Each LLM agent uses a GPT-4 API component prompted with its function, actions, inputs, outputs, examples, and JSON-formatted responses.Missing output keys receive defaults; GPT-4 API parameters use official defaults.Source: §C.1 · p. 13
C.2. Reasoning Processes
- For a complex command, the project manager splits tasks among agents for vehicle removal, asset selection, motion, and view adjustment before background and foreground rendering are composed.The example coordinates a Porsche and a pursuing police car driving the wrong way.Source: §C.2 · p. 13
- The project manager expands “Create a traffic jam” into repeated car additions, while later commands can modify previously added cars and add vehicles relative to them.The motion agent revises an existing car’s trajectory when asked to turn left.Source: §C.2 · p. 14
D.1. HDRI dataset
The supplementary sections describe HDR skydome reconstruction and multi-camera lighting prediction, asset preparation, rendering, and text-guided vehicle motion. Together, these procedures specify how scene lighting and assets are prepared and integrated into simulations.
D.1. HDRI dataset
- The HDRI dataset contains 449 outdoor panorama HDRIs, with 357 randomly selected for training and the remainder for testing.The panoramas are licensed CC0.Source: §D.1 · p. 14
D.2. LDR to HDR Skydome Reconstruction
- An LDR-to-HDR autoencoder encodes sky panoramas as content, peak-direction, and intensity vectors, then decodes spatial maps and content features into an HDR panorama.Training uses peak-direction, peak-intensity, HDR-reconstruction, and LDR-reconstruction losses, with rotation, flipping, exposure, and white-balance augmentation.Source: §D.2 · pp. 14–15
- A spherical-Gaussian peak-intensity residual replaces decoded HDR values at the sun peak, preserving its high intensity for stronger inserted-object shadows.The method explicitly addresses over-smoothing of the decoded HDR panorama.Source: §D.2 · p. 14, Figure 12 · p. 15
D.3. Predict HDR Skydome from Multi-Camera Images
- The multi-camera predictor fuses rotated peak directions and averaged intensities with attention-fused sky content, then decodes the fused features into an HDR panorama.Its training uses simulated camera crops from HoliCity panoramas and pseudo-targets generated by the previous reconstruction stage.Source: §D.3 · p. 15
E. 3D Asset Bank
- The asset bank standardizes Blender models with meter-scale dimensions, a shared origin and orientation, a named car-paint material, and merged meshes.This normalization supports continued expansion of the asset bank.Source: E. 3D Asset Bank · p. 16
F. Blender Rendering Details
- The Blender workflow renders combined image, depth, and shadow outputs, then uses masks and depth information for occlusion and adds motion blur to match the background.A shadow-catching plane supports rendered vehicle shadows.Source: F. Blender Rendering Details · p. 16
G.1. Vehicle Placement
- Vehicle placement maps language-derived attributes to lane nodes, selects a compatible node, and uses its midpoint and lane direction for initial position and heading.The method assumes vehicles lie on lane centerlines and checks iteratively for placement conflicts.Source: §G.1 · pp. 16–17
G.2. Vehicle Motion Planning
- Motion planning chooses destinations from action and speed attributes, fits cubic Bezier curves between endpoints, and iteratively adjusts intermediate points to lane nodes to avoid off-road trajectories.A trajectory-tracking method provides vehicle-dynamics post-processing.Source: §G.2 · p. 17
H.1. Dataset Selection
Waymo experiments use three frontal cameras and 120 images per scenario. The alignment procedure expresses camera poses in Metashape’s unified space and scales that space to real-world units.
H.1. Dataset Selection
- Each Waymo scenario uses 120 images captured from three frontal cameras.The selected sequences are described in Table 6.Source: §H.1 · p. 17
H.2. Multi-Camera Alignment
- Camera extrinsics are aligned between vehicle and Metashape coordinates, with a scaling factor ensuring that aligned-space units match real-world units.The front camera is designated C0.Source: §H.2 · pp. 17–18
- The ablation figure contrasts McNeRF without pose alignment, without exposure adjustment, and with both, alongside target images.The caption identifies the variants and target row but does not state their outcomes.Source: Figure 16 · p. 18
I.1. Qualitative Ablation Study of Background Rendering
The qualitative ablation shows McNeRF avoids blur from pose misalignment and brightness changes at camera junctions. The supplementary results also describe occlusion handling, detection gains from simulated data, and rare-scene construction with external assets.
I.1. Qualitative Ablation Study of Background Rendering
- McNeRF avoids the significant blur and anomalies seen without pose adjustment and the brightness changes at camera junctions seen without exposure adjustment.The full method achieves the best qualitative rendering outcome among the displayed variants.Source: §I.1 · p. 18
I.2. Occlusion with Depth Test
- For foreground-background occlusion, the method combines SAM image patches with sparse point-cloud depth to estimate patch depths and compare them against rendered foreground depth.The authors note that sparse, noisy depth completion is unsuitable for pixel-level accuracy; Figure 17 also shows vehicle color adjustment.Source: §I.2 · p. 18, Figure 17 · p. 19
- Simulation data significantly enhances 3D detection performance, with gains tending to stabilize as the amount of simulated data increases.Table 7 compares detection performance across amounts of ChatSim simulation data.Source: Table 7 · p. 19, §I.2 · p. 19
I.3. Rare Cases Simulation
- External assets let ChatSim place uncommon elements such as bulldozers, fences, and excavators into reconstructed scenes to emulate rare cases.The authors describe these simulations as rare digital twins of collected data.Source: §I.3 · p. 19, Figure 18 · p. 19
I.4. Supplementary 3D Detection Augmentation Experiment
The supplementary experiment varies the amount of ChatSim simulation data while holding the real-data set at 4,200 frames, using Lift-Splat for 3D detection.
I.4. Supplementary 3D Detection Augmentation Experiment
- The experiment fixes the original real-data amount at 4,200 frames while augmenting it with varying quantities of ChatSim simulation data.It evaluates 3D detection augmentation under this setting.Source: §I.4 · p. 19
- Lift-Splat [54] is the detection model used in the augmentation experiments.Source: §I.4 · p. 19