PaperlayerView exact revision on arXiv

Source-linked AI summary

CARLAverse: A Highly Modular, Distributed, and Multimodal Framework for Human-in-the-Loop Simulation

Patrick Rebling, Philipp Nenninger, Reiner Kriesten

arXiv:2609.11478v1cs.ROcs.HC

TL;DR

Autonomous-driving research needs safer and more realistic ways to study human interactions with vulnerable road users, which existing multi-agent platforms inadequately support under haptic synchronization constraints. CARLAverse integrates driving, cycling, and pedestrian simulators in a shared CARLA environment using distributed physics, with local ego dynamics and centralized NPC and traffic simulation. The framework is presented as a modular, multimodal ecosystem for connected HITL simulation.

  • Problem

    Mixed-traffic studies need authentic human behavior from vulnerable road users, but existing multi-agent platforms struggle to provide immersive, synchronized, heterogeneous HITL simulation.

  • Method

    CARLAverse combines DrivoCARLA, CycloCARLA, and WalkoCARLA in a shared CARLA infrastructure with local ego physics and centralized NPC and global traffic physics.

  • Results

    CARLAverse is introduced as a highly modular, distributed, and multimodal simulation ecosystem for shared multi-agent HITL experiments.

  • Takeaways & Limitations

    The framework provides an open architecture for integrating heterogeneous human-operated road-user simulators into centrally coordinated mixed-traffic scenarios.

Abstract

from arXiv · show

The development of autonomous driving demands comprehensive testing in mixed-traffic scenarios involving vulnerable road users (VRUs), where purely artificial agents often fail to capture authentic human social negotiations. While human-in-the-loop (HITL) simulators enable safe investigation of these interactions, existing multi-agent platforms struggle with the network latency and synchronization constraints required for high-fidelity haptic feedback. To resolve this, we present CARLAverse, an open-source, multimodal simulation ecosystem. Extending modular hardware abstraction, CARLAverse integrates driving (DrivoCARLA), cycling (CycloCARLA), and pedestrian (WalkoCARLA) simulators into a shared virtual environment. Its core methodological contribution is a distributed physics architecture: latency-critical ego dynamics and high-frequency force feedback are computed locally on client nodes, while a central CARLA server orchestrates non-player character (NPC) physics and global traffic. By decoupling haptic control loops from network bottlenecks, CARLAverse enables scalable, cross-institutional HITL experiments without compromising physical immersion. Code and documentation: https://git.ieem-ka.de/simulator-environments/carlaverse

I. INTRODUCTION

CARLAverse addresses the difficulty of studying human behavior in mixed traffic by integrating heterogeneous HITL simulators into a shared virtual environment. Its distributed physics architecture separates local, latency-critical ego dynamics from centrally coordinated NPC and traffic simulation.

  • I. INTRODUCTION: Mixed-traffic research requires realistic investigation of interactions with cyclists and pedestrians, because artificial agents can miss subtle human behaviors such as eye contact, yielding, body language, and hesitation.HITL simulation provides a safe and repeatable way to study these interactions without exposing participants to physical risk.
  • I. INTRODUCTION: CARLAverse integrates vehicle, bicycle, and pedestrian HITL nodes within a unified CARLA-based infrastructure for shared multimodal simulation.The supported domains are DrivoCARLA, CycloCARLA, and WalkoCARLA.
  • I. INTRODUCTION: The framework extends modular hardware abstraction into an open, multimodal, connected ecosystem intended to support heterogeneous simulator hardware and cross-domain interaction.The paper positions this architecture as an expansion of prior hardware-agnostic standalone simulator work.
  • I. INTRODUCTION: Its distributed physics architecture computes each participant’s ego physics locally while a central CARLA server manages NPC physics and globally coordinated traffic.This decoupling targets low-latency interaction and high-fidelity force feedback while retaining centralized environmental coordination.
  • I. INTRODUCTION: Existing multi-agent platforms remain constrained by cost, limited immersion, rigid hardware dependencies, or synchronization architectures unsuitable for geographically distributed haptic experiments.Centralized middleware can introduce serialization latency, while some systems rely on proprietary infrastructure and localized networks.

C. Distributed Architectures and Latency Management

Multi-agent HITL simulation must minimize high-frequency haptic latency while coordinating globally consistent traffic. CARLAverse addresses this through distributed local ego-physics and direct communication, distinguishing it from centralized or middleware-heavy approaches.

  • At up to 1000 Hz, haptic simulation workloads make latency and synchronization central constraints for multi-agent HITL systems.The passages identify update rates above 100 Hz and up to 1000 Hz for stable haptics.
  • ROS 2 and DDS can add up to 50 % more latency than raw sockets for small control payloads, with overhead worsening as node count increases.The cited comparison links this overhead to degraded mechanical-feedback fidelity.
  • Centralized physics architectures and intermediary middleware may bottleneck multi-agent expansion and introduce latency into haptic control loops.The literature synthesis contrasts these limitations with CARLAverse’s distributed design.
  • The ecosystem combines multiple human-operated vehicle, bicycle, and pedestrian nodes in a shared synchronous multimodal environment.The framework is presented as open-source and intended to capture human-human interactions in mixed traffic.
  • CARLAverse localizes latency-critical ego physics while centrally coordinating global traffic and collision elements.Local computation covers effects such as tire friction, gyroscopic forces, and force feedback, while global coordination remains centralized.

III. CARLAVERSE ARCHITECTURE

CARLAverse extends CARLA’s server-client model to heterogeneous simulator nodes that can participate in a shared virtual environment across institutions. Its architecture separates authoritative global simulation from latency-critical local physics.

  • CARLAverse connects heterogeneous driving, bicycle, and pedestrian simulator clients to a central CARLA server.Each local node hosts a specific multimodal setup while the central server runs the CARLA Unreal Engine environment.
  • The topology supports geographically distributed, cross-institutional co-simulation in which participants share an identical virtual environment over a WAN.The example includes desktop, vehicle, cyclist, and pedestrian participants at different institutions.
  • The central server acts as authoritative orchestrator for traffic management, traffic lights, NPC behavior and physics, weather, and global collisions.Centralizing these elements maintains global synchronization and consistency for connected participants.
  • Local nodes compute latency-critical ego physics, while exchanging lightweight world-state telemetry and discrete collision impulse events with the central server.This split supports local control loops while preserving a synchronized global world state.

C. Communication Protocols and Data Logging

CARLAverse bypasses ROS for active simulator control, using native CARLA RPC over TCP/IP and dedicated UDP streams for latency-critical telemetry. Shared tools and logging support modular setup, diagnostics, and experimental data acquisition.

  • CARLAverse bypasses ROS orchestration for active control and uses native CARLA API communication over TCP/IP for reliable transactional state updates.
  • Dedicated connectionless UDP streams carry high-frequency force-feedback and motion-cueing telemetry, supporting local control loops with less transmission overhead.
  • Tlocal = tinput + tphysics,local + tactuation excludes network transmission delays from the time-critical haptic loop.The localized loop is contrasted with a centralized loop containing transmission and reception network terms.
  • The architecture is designed to reduce transport delay and mitigate jitter-related force spikes or steering-wheel oscillations.
  • ROS remains available as a passive logging engine, recording human inputs, eye tracking, synchronized traffic states, and sensor data through rosbags.
  • carlaverse_tools, the launcher, and configuration managers standardize API usage, lifecycle control, YAML configuration, and hardware diagnostics across simulator nodes.The launcher also supports installation, configuration editing, and peripheral testing.

B. Rapid Prototyping and YAML-Based Orchestration

CARLAverse separates hardware-agnostic simulator modules from YAML-configured node instantiation, supporting rapid reconfiguration, reproducibility, dynamic interfaces, and shared spatial audio.

  • B. Rapid Prototyping and YAML-Based Orchestration: YAML configuration governs simulator-node instantiation while executable control, force-feedback, and visualization modules remain hardware-agnostic.This separation of concerns reduces source-code changes when moving between simulator hardware configurations.
  • B. Rapid Prototyping and YAML-Based Orchestration: A decoupled web-based HMI connects to the CARLA Client through WebSockets and drives separate touchscreen-input and dashboard-display browser interfaces.External configuration specifies the interfaces, while standard web technologies support rapid prototyping without simulator recompilation.
  • B. Rapid Prototyping and YAML-Based Orchestration: Archiving YAML configurations with rosbag logs preserves hardware states, physics models, environmental profiles, and control mappings for reconstruction.
  • B. Rapid Prototyping and YAML-Based Orchestration: The spatial audio engine generates a multi-channel soundscape for each connected client according to its absolute position in the virtual world.It combines context-aware environmental sounds and dynamic emitters, with local nodes rendering each client-specific mix.

E. Force Feedback Server

CARLAverse uses standalone, low-latency C++ servers to translate local simulation outputs into force-feedback and motion-platform commands while preserving hardware-level modularity and safety.

  • E. Force Feedback Server: The force-feedback pipeline streams locally computed steering resistance and counter-torques through a tightly packed 40-byte UDP structure to platform-specific input drivers.The server supports DirectInput on Windows and evdev on Linux, enabling layered spring, damper, friction, and vibration effects.
  • E. Force Feedback Server: The C++ force-feedback server is a shared core for automotive steering bases and bicycle front-fork motors, with standalone reuse in custom simulation environments.
  • E. Force Feedback Server: A hardware watchdog zeros active forces after a prolonged simulation crash or network dropout, preventing uncontrolled high-torque motor behavior.
  • E. Force Feedback Server: The motion-platform server receives low-latency UDP telemetry in a 21-byte binary payload and maps it through the D-BOX SDK to physical actuation.Replacing the hardware-specific backend preserves the client-side physics implementation and standard UDP protocol, although the D-BOX module requires a valid license.
  • E. Force Feedback Server: The shared visualization system supports simple, physical, and manual camera configurations, including multi-display offset compensation based on monitor geometry.Camera sensors can be positioned at the driver’s head location, with monitor-specific rotations producing a smooth, unshifted image.

V. MULTIMODAL SIMULATION NODES IN THE CARLAVERSE

CARLAverse combines specialized driving, cycling, and pedestrian simulator nodes within a connected multimodal ecosystem, with DrivoCARLA providing a scalable automotive implementation and shared data architecture.

  • V. MULTIMODAL SIMULATION NODES IN THE CARLAVERSE: The ecosystem’s simulator nodes reproduce road-user-specific visual, auditory, haptic, and vestibular cues across DrivoCARLA, CycloCARLA, and WalkoCARLA.
  • V. MULTIMODAL SIMULATION NODES IN THE CARLAVERSE: DrivoCARLA scales from single-monitor desktop setups to enclosed multi-display vehicle cabins while using ecosystem-wide visualization, spatial audio, and haptic interfaces.
  • V. MULTIMODAL SIMULATION NODES IN THE CARLAVERSE: Its local Simulation PC integrates vehicle interfaces, dynamic UI, motion cueing, and force-feedback abstractions while communicating bidirectionally with the central CARLA server.An asynchronous ROS-based logging container can capture sensor and driving telemetry.
  • V. MULTIMODAL SIMULATION NODES IN THE CARLAVERSE: A preliminary user study demonstrated the empirical utility and standalone viability of the simulator pipeline.
  • V. MULTIMODAL SIMULATION NODES IN THE CARLAVERSE: DrivoCARLA abstracts steering wheels, pedals, and button arrays through Pygame and maps YAML-defined raw inputs into nonlinear vehicle commands.External ADAS algorithms can inject pedal overrides or steering counter-torques for human-trust and takeover studies.
  • V. MULTIMODAL SIMULATION NODES IN THE CARLAVERSE: State machines coordinate lighting and indicator behavior, including interactions among turn signals, hazard lights, and vehicle operating states.

2) Steering Physics and Force Feedback:

DrivoCARLA computes realistic steering feedback from tire dynamics and uses speed-dependent safeguards, while motion cueing converts filtered vehicle telemetry into stabilized pitch, roll, and heave commands.

  • 2) Steering Physics and Force Feedback:: The steering model computes self-aligning torque Mz from slip angle, vertical tire load, and pneumatic trail using a simplified Pacejka Magic Formula.Mz = −Fz · tp · sin(1.6 · arctan(8.0 · α)).
  • 2) Steering Physics and Force Feedback:: Below 15 km/h, dynamic Pacejka feedback fades into a high-resistance parking spring and friction force to prevent low-speed steering oscillations.
  • 2) Steering Physics and Force Feedback:: The indicator state machine permits directional reset through the IR event only when Lane Change Assist is active.
  • 3) Motion Cueing:: The motion-cueing algorithm maps filtered longitudinal acceleration and braking inputs to pitch, lateral acceleration and steering to roll, and processed jerk signals to heave.Heave processing applies deadbanding, clamping, and low-pass filtering, with an additional emergency-braking impulse.
  • 3) Motion Cueing:: Washout filters return platform axes to neutral after acceleration events, adapting the return rate to vehicle speed for plausible and comfortable cues.
  • 2) Steering Physics and Force Feedback:: High-frequency velocity telemetry and latency-critical steering force feedback are handled locally in CycloCARLA through a microcontroller and DirectInput serial connection.

B. CycloCARLA: Bicycle Simulator

CycloCARLA replaces CARLA’s four-wheeled vehicle dynamics with a locally computed bicycle model and captures high-frequency rider input for responsive simulation.

  • CycloCARLA locally computes a custom single-track kinematic bicycle model and overrides CARLA’s default four-wheeled dynamics.The resulting movement is enforced by injecting absolute target velocity and angular velocity.
  • High-frequency wheel-speed capture from a microcontroller reduces longitudinal-acceleration latency compared with commercial trainers broadcasting telemetry at about 1 Hz.
  • A calibrated 2D lookup table with bilinear interpolation converts computed total force into smooth fractional resistance commands for proprietary smart-trainer hardware.

2) Steering and Haptics:

CycloCARLA provides force-feedback steering and comfort-oriented visualization, while WalkoCARLA uses modular, API-based VR control and rendering to support pedestrian simulation.

  • 2) Steering and Haptics:: The bicycle interface uses a force-feedback motor to reproduce trail-induced self-centering resistance and high-frequency steering vibrations.
  • 2) Steering and Haptics:: CycloCARLA primarily uses a forward monitor rather than VR to avoid visual-vestibular conflict and simulator sickness during prolonged studies.Its modular architecture still permits switching to VR rendering.
  • 2) Steering and Haptics:: WalkoCARLA uses the native CARLA Python API instead of hardcoded Unreal Engine modifications, preserving modularity and compatibility with newer CARLA releases.
  • 2) Steering and Haptics:: Standard OpenXR Action Bindings map VR-controller inputs to pedestrian direction vectors without dependence on proprietary manufacturer SDKs.
  • 2) Steering and Haptics:: WalkoCARLA addresses VR rendering demands through stereoscopic processing, asymmetric-FOV correction, hardware-based cropping, coordinate transformation, and GPU texture optimization.The rendering loop increased from seven to 36 FPS on an average desktop computer.

3) Pose Process:

WalkoCARLA estimates full-body pose from consumer webcam input, transforms landmarks into body-relative coordinates, and streams joint parameters to the pedestrian avatar without blocking VR rendering.

  • 3) Pose Process:: WalkoCARLA implements an accessible 6-DOF pose-estimation pipeline using vision tracking instead of specialized full-body tracking suits.
  • 3) Pose Process:: MediaPipe extracts 33 three-dimensional skeletal landmarks from a standard webcam before transforming them for avatar animation.
  • 3) Pose Process:: The system builds a torso-relative coordinate frame from shoulder and body-longitudinal vectors, using their cross product to determine the body-depth normal.
  • 3) Pose Process:: An inverse torso rotation isolates arm motion in local body coordinates, which are converted into shoulder Euler angles and elbow hinge angles.
  • 3) Pose Process:: Parallel Python multiprocessing prevents vision inference from bottlenecking VR rendering while synchronizing avatar gestures and gaze for traffic-interaction studies.

VI. CONCLUSION AND FUTURE WORK

CARLAverse combines modular vehicle, bicycle, and pedestrian nodes with distributed physics to support synchronized, low-latency multi-agent HITL research. The authors identify validation, hardware, rendering, large-scale experiments, and full integration as continuing work.

  • VI. CONCLUSION AND FUTURE WORK: CARLAverse integrates vehicle, bicycle, and pedestrian simulators into a shared centrally coordinated world for multi-agent human-in-the-loop research.
  • VI. CONCLUSION AND FUTURE WORK: Distributed physics separates central global traffic simulation from local latency-critical ego dynamics, supporting high-frequency control loops for haptics and motion cueing.
  • VI. CONCLUSION AND FUTURE WORK: The ecosystem is intended to enable scalable, low-latency studies of social negotiation, intent recognition, and trust dynamics beyond a single laboratory.
  • VI. CONCLUSION AND FUTURE WORK: The physical models for VRU nodes and CARLAcoustics remain subject to empirical validation, including effects on immersion and localization accuracy.
  • VI. CONCLUSION AND FUTURE WORK: Planned extensions include bicycle motion and wind hardware, an omnidirectional pedestrian treadmill, VR-rendering evaluation, and possible neural rendering.
  • VI. CONCLUSION AND FUTURE WORK: Future studies will conduct large-scale synchronous HITL experiments, while remaining work completes release of all CARLAverse components.
Loading 2609.11478v1…