PaperlayerView exact revision on arXiv

Source-linked AI summary

How Simulation Helps Autonomous Driving:A Survey of Sim2real, Digital Twins, and Parallel Intelligence

Xuemin Hu, Shen Li, Tingyu Huang, Bo Tang, Rouxing Huai, Long Chen

arXiv:2305.01263v2cs.ROcs.AI

TL;DR

Autonomous driving needs extensive simulation and real-world testing, yet differences in lighting, textures, dynamics, and agent behavior create a reality gap. This survey reviews sim2real, digital twins, and parallel intelligence, along with their applications, algorithms, models, simulators, development, and challenges. It organizes the field from transfer methods toward synchronized and parallel approaches while identifying unresolved generalization and integration challenges.

  • Problem

    Differences between simulated and real worlds make it difficult to transfer autonomous-driving knowledge reliably, despite the safety and cost advantages of simulation.

  • Method

    The survey classifies and reviews sim2real, digital twins, and parallel intelligence methods, applications, algorithms, models, simulators, and development trajectories.

  • Results

    The survey synthesizes state-of-the-art approaches and identifies challenges including limited task-independent portability, incomplete DT model-algorithm integration, and the absence of synchronous dynamic sim2real methods for DT parameters and noise.

  • Takeaways & Limitations

    The paper frames sim2real, digital twins, and parallel intelligence as progressively broader approaches for addressing the reality gap in autonomous driving.

Abstract

from arXiv · show

Safety and cost are two important concerns for the development of autonomous driving technologies. From the academic research to commercial applications of autonomous driving vehicles, sufficient simulation and real world testing are required. In general, a large scale of testing in simulation environment is conducted and then the learned driving knowledge is transferred to the real world, so how to adapt driving knowledge learned in simulation to reality becomes a critical issue. However, the virtual simulation world differs from the real world in many aspects such as lighting, textures, vehicle dynamics, and agents' behaviors, etc., which makes it difficult to bridge the gap between the virtual and real worlds. This gap is commonly referred to as the reality gap (RG). In recent years, researchers have explored various approaches to address the reality gap issue, which can be broadly classified into three categories: transferring knowledge from simulation to reality (sim2real), learning in digital twins (DTs), and learning by parallel intelligence (PI) technologies. In this paper, we consider the solutions through the sim2real, DTs, and PI technologies, and review important applications and innovations in the field of autonomous driving. Meanwhile, we show the state-of-the-arts from the views of algorithms, models, and simulators, and elaborate the development process from sim2real to DTs and PI. The presentation also illustrates the far-reaching effects and challenges in the development of sim2real, DTs, and PI in autonomous driving.

I. INTRODUCTION

Autonomous-driving research uses simulation to address safety and cost constraints, but must bridge the reality gap between simulated and real environments. This survey organizes bridging approaches into sim2real, digital twins, and parallel intelligence, and reviews their methods, applications, simulators, challenges, and opportunities.

  • Motivation: Simulation helps reduce the safety risks and real-vehicle costs of testing immature autonomous-driving algorithms before deployment.These costs include high-precision sensors, radars, cameras, and collision damage.
  • Approach taxonomy: Reality-gap methods are divided into sim2real transfer, digital-twin learning, and parallel-intelligence technologies.The three categories are presented as approaches for bridging simulation and reality.
  • Sim2real: Sim2real trains autonomous-driving systems in simulation and transfers learned strategies and knowledge to real vehicles using six method families.The families are curriculum learning, meta-learning, knowledge distillation, robust reinforcement learning, domain randomization, and transfer learning.
  • Digital twins: Digital twins map real-world physical entities into simulation using sensor data and physical models, supporting multi-scale modeling of environments and vehicles.Their purpose is to reflect the lifecycle of corresponding physical entities through data interaction.
  • Parallel intelligence: Parallel intelligence extends simulation-based approaches with description, prediction, and prescription functions while combining artificial and physical scenarios through parallel computation.The survey presents PI as a higher-level technology than digital twins.
  • Survey scope: The survey contributes a taxonomy, comprehensive reviews of reality-gap methods and applications, and discussion of key challenges and opportunities.It specifically emphasizes sim2real, digital twins, parallel intelligence, and the simulators used to implement them.

II. SIMULATION TO REALITY TRANSFER

Simulation-to-reality transfer addresses the mismatch between strong simulated performance and weaker real-world performance in autonomous driving. This section reviews curriculum-based and other transfer approaches for moving learned models from simulation into real driving domains.

  • Motivation: Autonomous-driving algorithms require extensive simulation testing because real-world testing is costly and safety-sensitive.The motivation is especially strong for reinforcement-learning-based methods.
  • Transfer strategy: Sim2real methods train models in simulation and transfer them to the real world to address the performance mismatch between the two environments.The section covers curriculum learning, meta-learning, knowledge distillation, robust reinforcement learning, domain randomization, and transfer learning.
  • Curriculum learning: Curriculum learning trains models on simple tasks before progressively introducing more complex ones.Self-paced learning dynamically adjusts the curriculum to the learner’s pace, and object detection can progress from simulated source images toward real-domain prediction.
  • Curriculum learning: Automatically Generated Curriculum uses deep reinforcement learning and candidate sets to optimize traffic efficiency at complex intersections and reduce training time.Its transfer effectiveness is not consistent across different driving situations when knowledge is pretrained on other tasks.
  • Curriculum learning: The section illustrates curriculum self-paced learning as a route from source-domain training through easy-to-hard tasks toward object detection in the real domain.The figure corresponds to the object-detection approach described for self-paced learning.

B. Meta-learning

Meta-learning enables autonomous-driving systems to reuse prior experience, adapt more quickly across tasks, and generate simulation data that narrows the simulation–reality distribution gap.

  • B. Meta-learning: Meta-learning lets models gain prior experience from related tasks and learn new knowledge more quickly.It has been applied to single-task, multitask, few-shot, and neural architecture search settings.
  • B. Meta-learning: Meta-learning addresses inefficient reinforcement-learning exploration by transferring knowledge across robotic systems and inferring task relationships from data.Probabilistic latent-variable formulations support this transfer across tasks.
  • B. Meta-learning: Online adaptive model-based reinforcement learning uses a meta-learned dynamic prior to address sim2real transfer on a vehicle.The approach adapts high-capacity dynamic models during training.
  • B. Meta-learning: Adaptive meta-reinforcement learning enables fast iterations for changing tasks and extends reinforcement learning to urban autonomous driving in CARLA.The method uses an embedded adaptive meta-reinforcement-learning controller.
  • B. Meta-learning: Meta-Sim automatically synthesizes labeled datasets related to downstream tasks to reduce labeled-data costs and bridge simulation–reality distribution differences.It optimizes a meta-objective for generating task-relevant synthetic images.

D. Robust reinforcement learning

Robust reinforcement learning models disturbances and uncertainty during training, while domain randomization varies simulated conditions to improve transfer and generalization to real driving environments.

  • D. Robust reinforcement learning: Robust reinforcement learning explicitly considers input disturbances and modeling errors to make control systems less sensitive to uncertainty.Its formulation is based on H∞ control theory and includes physical-parameter changes.
  • D. Robust reinforcement learning: Robust adversarial reinforcement learning trains against adversarial environmental and observational perturbations to improve behavior under system disturbances.Highway-ramp merging work models the environment as an adversarial agent and uses white-box and black-box attacks.
  • D. Robust reinforcement learning: Sequential robust reinforcement learning addresses limitations of zero-sum simultaneous-game formulations by modeling sequential deployment with a Stackelberg game.The formulation targets single-agent robot control and multi-agent highway merging.
  • E. Domain randomization: Domain randomization samples varied virtual-domain parameters, then adjusts the sampling strategy to improve generalization to the real domain and reduce the reality gap.The approach is used for sim2real data augmentation.
  • E. Domain randomization: Domain randomization has been used with randomized visual features and dynamics models to transfer learned strategies from simulation to real vehicles and robots.Applications include RGB-image transfer, calibrated physical parameters, complex road scenarios, and multiple motor skills.
  • E. Domain randomization: Experiments reported that domain randomization with pyramid consistency was much more effective than existing methods.The method uses consistency-forced training across domains to learn highly generalizable models.

F. Transfer learning

Transfer learning moves labeled data or learned driving knowledge from source domains, including simulation, to related target domains where data labeling is costly or unavailable.

  • F. Transfer learning: Transfer learning transfers labeled data or knowledge between related domains to improve learning in target tasks.It is especially useful when supervised-learning data are expensive and time-consuming to label.
  • F. Transfer learning: In autonomous driving, transfer learning commonly moves driving strategies learned in simulation to real environments.The approach can transfer from labeled source-domain data to unlabeled target-domain data.
  • F. Transfer learning: Simulation-based data collection is considered because acquiring information for autonomous driving is costly.Intersection simulations have been transferred to the real world, demonstrating robustness of the transfer process.
  • F. Transfer learning: A continuous end-to-end approach performs transfer in two steps, moving environment and information representations toward nearby vehicle-value domains.The staged process implements learning through successive transfer steps.
  • F. Transfer learning: Adaptive transfer methods can avoid manual data labeling and compensate for domain bias through weakly paired source–target data.Transfer learning is also combined with multitask policy gradients and robust reinforcement learning for complex behavioral planning.

III. DIGITAL TWINS TECHNOLOGIES AND APPLICATIONS

Digital twins extend simulation by linking virtual models to physical entities through real-time data exchange, while AR/MR technologies support immersive, interactive autonomous-driving environments. The section reviews how these technologies contribute to safer and more comprehensive simulation and testing.

  • Digital twins: Digital-twin methods address the time and expense of validating autonomous-driving algorithms in diverse physical scenarios.The paper motivates more comprehensive simulation to reduce the physical tuning required for design validation.
  • Digital twins: Digital twins differ from iterative and model-based simulation through visual synchronization of physical and digital data.The comparison includes iterative convergence, digital exchanges, and synchronized data between physical and digital spaces.
  • Digital twins: Digital twins are virtual copies of physical entities connected through real-time data exchange.The survey presents digital twins as an extension of augmented and mixed reality technologies.
  • AR and MR: AR systems combine tracking, display, and interaction so users can perceive and interact with virtual objects integrated into physical-world imagery.These requirements support real-time placement of virtual information from the user’s viewpoint.
  • AR and MR: AR-assisted driving interfaces can display detected vehicles, obstacles, road information, and autonomous-driving decisions to support drivers.The reviewed windshield prototype presents environmental and decision information in a virtual environment for emergency scenarios.
  • AR and MR: High-precision simulators can connect virtual traffic scenarios with real vehicles, while mixed-reality systems can support digital-twin modeling and tracking.The reviewed examples use online simulator connections and mixed-reality integration for autonomous-driving verification and digital-twin environments.

B. Digital twins in autonomous driving

Digital twins provide autonomous-driving systems with high-fidelity, data-connected virtual environments for modeling, testing, and synchronization with physical vehicles. Applications span simulator platforms, sensor and map models, traffic-flow logic, and communication-enabled testing systems.

  • Motivation and structure: Digital twins provide safer, diverse autonomous-driving testing environments while reducing testing time cost by at least 80%.The survey contrasts this capability with traditional simulators’ limited end-to-end equivalence and incomplete coverage of weather, climate, and lighting.
  • Motivation and structure: A typical autonomous-driving digital-twin system includes sensor-model, 3D-map, and traffic-flow logic twins.Researchers construct task-specific systems from these three components.
  • Frameworks and applications: Digital-twin frameworks generally combine physical, data, and virtual parts, with additional modules added for specific tasks.One framework separates the physical entity, digital-twin, and simulation-evaluation components.
  • Frameworks and applications: V2X-based digital-twin systems support high-fidelity transmission and simulation for autonomous-driving testing, including low-latency connectivity.Vehicle-edge-computing approaches model physical vehicle environments using heterogeneous information and V2I communication.
  • Frameworks and applications: Unity-based digital-twin integration with physical target-detector data significantly improves driving safety in the reported evaluation.The system depicts road traffic conditions by combining physical detections with digital-twin information.
  • Frameworks and applications: Digital-twin reinforcement-learning methods combine virtual and real traffic data, with domain randomization and adaptive techniques used to strengthen transfer robustness.These techniques are applied across development stages of the digital-twin lifecycle.

IV. PARALLEL INTELLIGENCE TECHNOLOGIES

Sim2real and digital-twin methods address the reality gap through transfer or synchronized real–virtual learning, but complex scenarios can exceed their dynamic-control simulation capabilities. Parallel intelligence responds by coupling artificial and real systems through computation, physics, and social interaction.

  • Parallel intelligence technologies: Parallel intelligence is presented as an approach for complex scenarios where sim2real and digital-twin dynamic-control simulations cannot adapt to a large reality gap.The proposed direction constructs an artificial system and uses parallel computation of artificial and real systems to handle uncertainty.

A. Parallel intelligence concept and ACP method

Parallel intelligence connects physical and artificial worlds through cyber-physical-social interaction and the ACP method. The survey reviews its extensions into parallel learning, vision, control, testing, driving, and planning for autonomous driving.

  • Parallel intelligence concept: Cyber-physical-social systems enable virtual and real systems to interact, provide feedback, and promote one another.Parallel intelligence is grounded in the evolution from cyber-physical systems to cyber-physical-social systems.
  • ACP method: The ACP method combines artificial societies, computational experiments, and parallel execution to model, evaluate, and interact with complex physical systems.Its stages are artificial-system modeling, computational-experiment training and evaluation, and interaction with the real system.
  • Applications: Parallel intelligence has been applied to knowledge transfer between simulation and reality through parallel learning, vision, control, testing, driving, and planning.The survey organizes these technologies as extensions of ACP and parallel-intelligence methods.
  • Parallel learning: Parallel learning consists of descriptive, predictive, and prescriptive learning stages.The framework extends parallel intelligence into machine learning through three interrelated stages.
  • Parallel vision: Parallel vision constructs artificial scenes resembling real driving scenarios to obtain accurate labels and generate large-scale visual data.The approach targets feature extraction challenges in complex environments faced by traditional computer-vision methods.

C. Parallel driving and parallel testing

Parallel driving connects physical vehicles, human drivers, and cognitive attributes across interconnected parallel worlds, while parallel testing uses cyclic co-evolution between real and virtual testing grounds. Parallel planning and control extend this framework to emergency-scenario learning and data-informed transportation management.

  • Parallel driving: Parallel driving projects physical vehicles, human drivers, and cognitive attributes into three interconnected parallel worlds.
  • Parallel driving: Integrated multi-ADAV modules learn driver behavior across reality and simulation scenarios to bridge the reality gap.
  • Parallel testing: Parallel testing cyclically updates real and virtual testing grounds through co-evolution, improving virtual–real interaction for autonomous-driving evaluation.
  • Parallel planning: Parallel planning generates emergency traffic scenarios in an artificial traffic world to support timely real-world planning during emergent events.
  • Parallel control: Parallel control combines synthetic data and real-world observations to enhance control effectiveness and autonomous-driving development.

V. AUTONOMOUS DRIVING SIMULATORS

Autonomous-driving simulators reduce the cost of testing and provide environments for rare-event and interactive evaluation, but simulator capabilities vary across rendering, dynamics, controllability, and sensor detail. The surveyed approaches still face broad challenges in transferability, data, digital-twin methodology, scalability, and efficient multimodal-data analysis.

  • Simulator motivation: Simulation supports low-cost autonomous-driving testing because real-world testing is expensive and rare-event data are difficult to collect.
  • Simulator capabilities: Game-engine simulators such as Unreal and Unity provide rendering but typically offer limited control over characters and weather.
  • Simulator capabilities: Driving simulators include interactive dynamic-scene tools such as RoadRunner Scenario, CARLA, and LGSVL, while robot-based environments provide detailed vehicle, pedestrian, and sensor modeling.
  • Challenges: Current methods often target specific scenarios, with unsatisfactory performance across different tasks and no general task-independent solution ensuring transfer effectiveness and portability.
  • Challenges: Challenges include scarce real data, limited digital-twin methodology, separated DT models and algorithms, increasing parallel-computing complexity, and redundant multimodal data.

VII. CONCLUSION

The survey addresses the need to transfer knowledge from simulation to reality despite limited and costly real-world data. It reviews sim2real, digital twins, parallel intelligence, simulators, applications, and future challenges in autonomous driving.

  • The paper reviews sim2real, digital twins, and parallel intelligence as approaches for addressing the simulation–reality gap in autonomous driving.
  • The survey covers methodologies, applications, virtual technologies, parallel-intelligence technologies, autonomous-driving simulators, and future challenges.
  • The paper’s scope reflects the need for substantial real-world data while simulated data remain important for reliable and robust autonomous-driving research.
Loading 2305.01263v2…