PaperlayerView exact revision on arXiv

Source-linked AI summary

SIREN-Bench: Behavior-Driven Generation and Evaluation of Emergency-Vehicle Interactions

Yicheng Zhu, Tianmu Zhao, Haoxin Leng, Fan Zuo, Tao Li, Zilin Bian

arXiv:2608.24094v1cs.RO

TL;DR

SIREN addresses the lack of behavior-level, reproducible data for safety-critical emergency-vehicle interactions. It uses closed-loop SUMO–CARLA co-simulation to generate configurable EMV–civilian behavior and instantiates seven templates in SIREN-Bench-v1. Evaluations show task-specific weaknesses: traffic clearance challenges detection, privileged intersection traversal challenges prediction, and vision-language models struggle with higher-risk events.

  • Problem

    Existing datasets and benchmarks do not directly provide configurable EMV privileges, civilian responses, consistent sensing, and ground truth for evaluating safety-critical interactions.

  • Method

    SIREN couples SUMO traffic evolution and behavior logic with CARLA continuous control and synchronized sensing, generating seven parameterized templates across L1–L3 and three behavior families.

  • Results

    Evaluations reveal behavior-dependent failure modes across detection, prediction, and risk understanding: clearance is hardest for detection, intersection traversal for prediction, and higher-risk events for vision-language models.

  • Takeaways & Limitations

    Behavior-centered benchmarking distinguishes how EMV-induced traffic reorganization affects perception, prediction, and semantic understanding, establishing SIREN as an extensible evaluation platform.

  • Takeaways & Limitations

    SIREN-Bench-v1 uses one map and one base realization per template for detection and prediction, preventing separation of level, map, and realization effects.

Abstract

from arXiv · show

Emergency vehicles (EMVs) can reorganize surrounding traffic as civilian vehicles brake, change lanes, or form rescue corridors in response to their passage. Evaluating these safety-critical interactions requires behavior-level control over both EMV privileges and civilian responses, together with consistent sensing and ground truth. Existing datasets and simulation benchmarks do not directly provide this combination. We present \textbf{SIREN}, a behavior-driven SUMO--CARLA co-simulation platform for generating EMV--civilian interactions. SIREN couples SUMO's network-level traffic evolution and behavior logic with CARLA's continuous vehicle control and synchronized onboard sensing; depending on the active behavior, the interaction is controlled by SUMO, CARLA, or jointly. We instantiate the platform as \textbf{SIREN-Bench-v1}, comprising seven parameterized interaction templates across emergency levels L1--L3 and three behavior families, with synchronized sensor observations and simulator-native annotations. We demonstrate the benchmark through three representative tasks: 3D object detection, trajectory prediction, and vision-language risk understanding. Evaluations of nine trajectory predictors, four LiDAR-based detectors, and five vision-language models reveal behavior-dependent failure modes. Traffic-clearance interactions are hardest for detection, privileged intersection traversal is hardest for prediction, and no learned predictor outperforms the constant-velocity reference on average. Vision-language models perform substantially better on normal traffic than on near-miss and collision events. These results demonstrate the value of behavior-centered benchmarking and establish SIREN as an extensible data-generation and evaluation platform for autonomous-driving and transportation safety research.

1 Introduction

SIREN addresses the need for reproducible, behavior-level evaluation of emergency-vehicle interactions by coupling configurable EMV privileges and civilian responses with consistent simulation, sensing, and ground truth. SIREN-Bench-v1 instantiates this approach across parameterized interaction templates and three downstream AI tasks, exposing behavior-dependent weaknesses.

  • Motivation: EMV encounters reorganize surrounding traffic through coupled, asymmetric interactions that can increase mobility risk.Privileged EMVs may violate ordinary mobility rules while multiple civilian vehicles brake, change lanes, or yield.
  • Research gap: Existing datasets and simulators provide realistic observations or operational traffic analysis, but not direct control over EMV behavior, civilian response, sensing, and ground truth together.Naturalistic datasets capture recorded encounters, while SUMO- and VISSIM-based studies primarily target routing, signal priority, and response time.
  • Approach: SIREN couples SUMO network-level traffic evolution and behavior logic with CARLA continuous control and synchronized onboard sensing through state exchange and control transfer.Depending on the active behavior, EMV–civilian interactions are controlled by SUMO, CARLA, or both.
  • Benchmark: SIREN-Bench-v1 provides seven parameterized templates spanning emergency levels L1–L3 and traffic clearance, privileged traversal, and nonstandard road-space behaviors.The benchmark evaluates 3D object detection, trajectory prediction, and vision-language risk understanding.
  • Findings: Traffic clearance is hardest for detection, privileged intersection traversal is hardest for prediction, and risk-understanding models show behavior-dependent failure patterns.No learned trajectory predictor outperforms the constant-velocity reference on average; vision-language models perform better on normal traffic than near-miss and collision events.

2 Related Work

Prior work provides emergency-vehicle operations studies, configurable simulators, naturalistic datasets, and curated long-tail scenarios, but not behavior-driven generation of emergent EMV–civilian interactions. SIREN targets this missing mechanism by making EMV behavior and civilian response the source of new benchmark instances.

  • Emergency-vehicle simulation: SUMO-based emergency-vehicle studies model special rights and surrounding emergency-lane formation, mainly targeting response time, routing, or signal operations.These works treat emergency mobility primarily as a transportation-operations problem.
  • Autonomous-driving benchmarks: Existing autonomous-driving benchmarks combine real-world data, CARLA, accelerated simulation, or curated rare-event scenarios, but their routes and categories remain pre-specified.HiDrive includes emergency-vehicle yielding and red-light emergency-yielding scenarios without an EMV behavior model that generates new emergent interactions.
  • Positioning: The related-work gap is not the absence of EMV response content, but the absence of EMV behavior as the mechanism for generating new benchmark instances.This distinction motivates SIREN’s behavior-centered benchmark design.

3 The SIREN Platform

SIREN begins with privileged EMV and civilian behavior policies, resolves their interaction through synchronized SUMO–CARLA control, and converts simulator states and onboard observations into benchmark episodes. The platform organizes seven parameterized templates across emergency levels and behavior families while preserving closed-loop, state-dependent responses.

  • Generation pipeline: SIREN configures EMV and civilian behaviors before trajectories, then resolves their closed-loop interaction through SUMO–CARLA co-simulation.Simulator states and synchronized sensor observations are converted into task-specific annotations and benchmark episodes.
  • Platform design: The platform builds on a privileged EMV behavior model by adding level-dependent civilian response policies and event-based SUMO–CARLA control transfer.The underlying model supplies longitudinal and gap-acceptance parameters but does not specify civilian responses or sensing and rendering.
  • Emergency levels: L0 is a non-emergency replay setting, while L1–L3 retain privileged EMV motion and vary the emergency cue and induced civilian response.L1 has no induced yielding, L2 activates feasible lane clearing, and L3 activates rescue-corridor pull-over with vehicles holding clearance until passage.
  • Hybrid co-simulation: The hybrid controller uses a moving EMV-centered interaction region, with SUMO maintaining broader traffic and CARLA handling synchronized local control.Intent overrides provide temporary CARLA lane-change paths, while infeasible requests are deferred and reconsidered.
  • Benchmark construction: SIREN-Bench-v1 contains seven parameterized templates across traffic clearance, traffic-control traversal, and nonstandard road-space use.Template parameters include speeds, traffic volume, response distance, and maneuver timing, enabling different realizations of the same target interaction.
  • Data and annotations: Each rollout records poses, kinematics, controller states, signal phases, and calibrated onboard observations for downstream tasks.Detection labels use a single generic Car class for every vehicle, including the EMV.

4 Experiments

Experiments evaluate trajectory prediction, 3D object detection, and vision-language risk understanding on SIREN-Bench, revealing distinct behavior-dependent weaknesses and evaluation caveats.

  • Trajectory prediction: 1.88 m ADE was achieved by the best learned predictor, compared with 1.44 m for constant velocity and 1.86 m for IDM on the seven-template mean.No learned predictor surpassed either non-learned reference on average; source-domain shift, observation-history differences, and model-specific choices may contribute.
  • Trajectory prediction: S6, privileged traversal through a stop-controlled intersection with crossing traffic, was the highest-error scenario for seven of nine prediction methods.Mean learned-predictor ADE reached 4.82 m in S6 versus 1.94 m in S7, although S6 had the fewest scored samples.
  • 3D object detection: Mean detection mAP was about 0.42 in S1 and 0.41 in S2, versus 0.63 and 0.61 in the red-light templates S3–S4.Clearance episodes contain dense, off-center vehicle configurations with increased geometric overlap; one base episode per template limits behavior isolation.
  • 3D object detection: Mean mAP ordered the behavior groups as clearance (0.42) < stop-sign traversal (0.51) < shoulder use (0.59) < red-light traversal (0.62).NDS followed the same observed trend, while absolute values are not directly comparable with published nuScenes results because evaluation was zero-shot under different sensing and layout conditions.
  • Risk understanding: 70.48% accuracy and 82.68% Normal-class F1 made Qwen3.5-9B the accuracy leader, but its Near-Miss and Collision precision and F1 were zero.LLaVA-Llama3-8B instead favored Near-Miss, with 22.86% accuracy and 37.21% Near-Miss F1, but zero Normal and Collision scores.
  • Risk understanding: 40.45% was the highest Near-Miss F1, achieved by Blaifa-InternVL3.5-8B, while MiniCPM-V-4.5-8B was the only model with non-zero Collision performance.MiniCPM-V-4.5-8B achieved 6.25% Collision precision and 11.11% Collision F1, and no evaluated model obtained non-zero F1 for all three classes.

5 Conclusion

SIREN provides behavior-driven generation of emergency-vehicle interactions through closed-loop SUMO–CARLA co-simulation, instantiated as SIREN-Bench-v1 with seven parameterized templates. Results show that different EMV behaviors stress different model capabilities, while the initial benchmark has important evaluation-scope limits.

  • SIREN-Bench-v1 instantiates closed-loop SUMO–CARLA interaction generation with seven parameterized EMV templates.
  • Traffic-clearance episodes produce the lowest detection scores, while privileged intersection traversal creates greater difficulty for motion prediction.
  • Risk-understanding models exhibit class-dependent biases, with no evaluated model obtaining non-zero F1 across all three risk categories.
  • Each interaction template is evaluated at one emergency level and on one map, preventing separation of level, map, and realization effects.
  • Detection and trajectory prediction use one base realization per template, limiting analysis across repeated seeds.

A Appendix

The supplement documents two artifacts: the SIREN platform and the SIREN-Bench benchmark. It organizes platform details around co-simulation, sensing, behavior control, target speeds, and data conventions.

  • SIREN platform: Part A documents the SIREN platform’s co-simulation loop, control ownership, onboard sensors, behavior logic, target speeds, and data formats.
  • SIREN-Bench benchmark: Part B documents the SIREN-Bench benchmark as the second artifact introduced in the main paper.

A.1.1 Co-Simulation Loop and Control Ownership

SIREN advances SUMO and CARLA in lockstep, combining network-level traffic and route information with CARLA-focused continuous control inside a moving EMV-centered interaction region. Control ownership and civilian responses vary by emergency level.

  • Co-simulation loop: Base episodes run in synchronized SUMO–CARLA mode at a fixed 0.05 s step, while actor states are logged at 10 Hz.
  • Interaction region: The interaction region uses a 40 m inner and 50 m outer hysteresis band centered on the EMV.
  • State exchange and control: SUMO supplies routes, lane-change intent, rescue-lane status, and level-L3 corridor information, which CARLA converts into continuous throttle, brake, and steering.
  • Control execution: Only feasible lateral requests are executed; infeasible requests are deferred and reconsidered at later steps.
  • Control ownership: Levels L1–L3 share the privileged EMV policy but differ in emergency cue and induced civilian response.

A.1.2 Onboard Sensor Suite and Placement

SIREN uses a fixed EMV-mounted sensor suite expressed in the CARLA vehicle frame, combining LiDAR, GNSS, IMU, and four RGB cameras. Camera placement provides front, rear, and oblique rear-side coverage but no pure side-facing view.

  • Sensor placement: The sensor suite places LiDAR, GNSS, and IMU together at z = 1.93 m in the CARLA vehicle frame.
  • Camera coverage: Four RGB cameras provide front, rear, and left/right rear-facing coverage, with no pure side-facing camera.
  • Coverage limitation: Near-lateral civilian yielding and pull-over are therefore captured only obliquely.
  • Camera specifications: All cameras use 800 × 600 resolution and a 100° horizontal field of view.

A.1.3 Behavior Implementation: Per-Step Control Logic

SIREN implements behaviors as per-step control routines with persistent state, allowing interaction outcomes to emerge from synchronized traffic evolution rather than scripted trajectories. At level L3, prioritized EMV control includes gap escape, SUMO-state copying, forced lane changes, and route following.

  • Per-step control logic: Each actor evaluates control routines at every synchronous step over latched flags that persist across steps.Response delay, incomplete lane clearing, and corridor formation emerge from evolving traffic.
  • EMV priority order: L3 EMV control prioritizes gap_escape over copy_sumo, forced lane changes, and route following.Higher-priority branches can preempt or suppress lower-priority behavior.
  • Emergency levels: L1 and L2 use the same EMV routine as L3 but remove gap_escape and copy_sumo, retaining route following with optional intent-driven forced lane changes.This creates level-dependent control behavior without scripted trajectories.
  • Gap escape: The gap_escape routine activates after prolonged low speed with a detected blocker, plans waypoints through available gaps, and temporarily disables normal collision avoidance.Its trigger and release constants are specified in Table 8.

A.1.4 Controller Target Speeds

The controller assigns target speeds by operating mode through a shared waypoint navigator, while nominal civilian motion remains governed by CARLA Traffic Manager. Feasible yielding adds a temporary speed reduction during lane changes.

  • Target-speed mapping: Table 9 lists mode-dependent target speeds enforced by the shared WaypointNavigator in km/h.The navigator converts these values internally to m/s.
  • Civilian baseline: Nominal civilian motion uses CARLA Traffic Manager at the road speed limit rather than a fixed navigator target.The configured Traffic-Manager speed difference is 0%.
  • Feasible yield: The L2 feasible-yield maneuver sets civilian speed to 50% below the speed limit for the lane-change duration.This reduction applies specifically during feasible yielding.
  • L3 control state: At level L3, Algorithm 1 evaluates latched control guards from top to bottom, with gap escape as the highest-priority mode.The algorithm requires emergency_lvl=3 and preserves state across ticks.

A.1.5 Data Formats and Coordinate Conventions

The appendix defines released trajectory and detection data conventions, evaluation protocols, checkpoint provenance, and extended results for prediction, detection, and risk understanding. These details establish common grids and expose cross-domain and metric-specific caveats.

  • A.1.5 Data Formats and Coordinate Conventions: Trajectory logs store one row per actor and frame at 10 Hz, including pose, velocity, speed, actor type, and an EMV indicator in the CARLA world frame.The frame is left-handed: x forward, y right, and z up.
  • A.1.5 Data Formats and Coordinate Conventions: Detection exports KITTI-format point clouds and labels in the EMV sensor frame, dropping objects beyond 50 m to match the nuScenes evaluation range.Coordinate conversion rotates world coordinates by −ψ_EMV about the vertical axis.
  • A.1.5 Data Formats and Coordinate Conventions: Civilian yield control latches until the EMV passes, using feasible-yield lane changes at L2 and pull-over followed by braking hold at L3.Non-yielding civilians remain under Traffic Manager control.
  • A.1.5 Data Formats and Coordinate Conventions: Heading consistency requires negating the sensor-frame heading after lateral-axis reflection; omitting this produces approximately 2θ disagreement, whereas retaining it yields approximately 3° median orientation error.This convention aligns exported boxes and point clouds.
  • A.2.1 Evaluation Metrics / A.2.4 Extended Trajectory-Prediction Metrics: The appendix reports ADE/FDE and min-of-k variants for trajectory prediction, with RMSE emphasizing the largest per-step errors because it is root-mean-square.Tables 11 and 12 use the common 5-s, 0.2-s grid.
  • A.2.2 Evaluation Protocols and Model Setup: Trajectory evaluation uses a common 5-s future sampled every 0.2 s, while released checkpoints differ in history length and training domain, creating confounds between model families.All qualifying vehicles, including the EMV, are scored.
  • A.2.2 Evaluation Protocols and Model Setup / A.2.3 Model and Checkpoint Provenance: Four nuScenes-pretrained LiDAR detectors are evaluated zero-shot on SIREN-Bench, so absolute detection values are not comparable to published nuScenes results.The benchmark uses a denser 64-beam sensor and a different city layout.
  • A.2.5 Detection True-Positive Error Breakdown / A.2.6 Risk Understanding: Detection translation and scale errors remain small across detectors, while velocity and attribute errors are much larger and therefore make mAP the headline metric and NDS secondary.Risk understanding classifies videos as Normal, Near-Miss, or Collision, selecting the more severe intermediate prediction.
Loading 2608.24094v1…