Source-linked AI summary
Scalable Distributed Simulation-Based Testing for Automated Driving Systems
Christian Geller, Benedikt Haas, Lutz Eckstein
TL;DR
Large ADS simulation suites are difficult to execute repeatedly and at scale. The paper presents a DevOps-driven Kubernetes framework for packaging applications, composing environments, and orchestrating distributed scenario execution. On 200 scenarios, distributed configurations reduced end-to-end time by a factor of 3–8 versus a sequential baseline, while exposing batching and stability trade-offs.
Problem
Repeatedly testing evolving ADS software across large, diverse scenario suites remains difficult to automate and scale.
Method
The framework combines standardized ROS 2 Helm charts, declarative Helmfile environments, and Argo Workflows for batched, parallel scenario execution and artifact collection.
Results
Distributed configurations reduce end-to-end time and increase throughput by a factor of 3–8 compared with the sequential, single-simulator configuration s1-b1.
Takeaways & Limitations
The framework supports robust multi-node regression testing and connects heterogeneous scenario sources with downstream evaluation and metadata-based safety assessment.
Takeaways & Limitations
The ADS application is illustrative rather than a complete safety argument or final validation campaign.
Abstract
from arXiv · showhide
Virtual scenario-based testing is a key enabler for validating automated driving systems (ADS) and intelligent transport systems (ITS). However, executing large-scale test suites involving possibly thousands of scenarios remains labor-intensive and difficult to scale. This paper presents an end-to-end, DevOps-driven framework that automates build, deployment, and distributed execution of CARLA-based scenario tests of an ADS on a lightweight Kubernetes cluster. ROS 2 applications are packaged as standardized Kubernetes Helm charts generated from repository specifications, while entire simulation environments are composed declaratively via dynamic Helmfile manifests. The paper describes how a distributed testing workflow can be implemented in Argo Workflows to provision environments, aggregate and batch OpenSCENARIO test cases from configurable sources, execute scenarios in parallel across cluster nodes, and collect logs and resource metrics. In an evaluation on a multi-node K3s cluster running 200 scenarios, the best configuration speeds up end-to-end workflow time by more than a factor of eight compared to a sequential baseline. The results demonstrate significant gains in end-to-end execution time and quantify trade-offs between parallelism, orchestration overhead, and cluster stability. The framework is further demonstrated in a real-world ADS test application with connections to scenario sources and downstream evaluation modules. This demonstrates that the approach provides a strong foundation not only for scalable simulation testing, but also for generating traceable evidence that can support safety arguments.
1 Introduction
The paper addresses the need to repeatedly execute diverse ADS simulation tests at scale by proposing an automated, distributed framework spanning build, deployment, execution, and result collection.
- ADS require reliable operation under diverse conditions, creating frequent regression-testing demands as software and test suites evolve.
- Simulation complements real-world testing by enabling controlled and repeatable execution of challenging scenarios.
- Embedding simulation tests in continuous development can provide fast feedback and reproducible evidence for safety validation.
- The framework automates provisioning, execution, teardown, and result collection while distributing scenarios across a lightweight Kubernetes cluster.
- ROS 2 applications are packaged as standardized Helm charts, and complete simulation environments are composed declaratively with dynamic Helmfile manifests.
- The paper contributes an architecture, Kubernetes packaging design, empirical analysis of execution and resource utilization, and a real ADS testing application.
2 State of the Art
Existing scenario standards, simulators, and orchestration technologies support ADS testing, but do not by themselves provide a complete, scalable regression-testing workflow.
- Scenario-based validation extends ADS testing beyond roads and proving grounds, with OpenSCENARIO and OpenDRIVE supporting portable scenarios and maps.
- CARLA and CARLOS provide simulation and DevOps-oriented execution foundations, which this paper extends toward automated large-scale Kubernetes testing.
- Existing toolchains commonly cover only parts of the workflow, leaving environment composition, scalable execution, and systematic artifact collection to be combined.
- Prior distributed approaches use custom orchestration, Docker Compose, or Kubernetes, but adaptation beyond their target use cases remains effort-intensive.
3 Research Approach
The research approach targets the gap in lightweight, flexible end-to-end ADS simulation testing by combining reproducible environments, streamlined ROS 2 deployment, and controlled distributed execution.
- The paper identifies a lack of an open, lightweight, flexible end-to-end approach combining reproducible environment composition with large-scale scenario execution.
- The framework describes complete test environments as code, including the simulator, system under test, scenario engine, evaluator, and auxiliary services.
- ROS 2 integration is streamlined through consistent packaging and deployment across local development and continuous integration.
- Large suites are executed in parallel while minimizing repeated startup and teardown overhead and avoiding cluster-management overload.
- The framework is required to assemble configurable scenario suites, execute them reliably across nodes, persist logs, traces, and metrics, and tear down environments deterministically.
4 Architecture
The architecture packages ROS 2 components for Kubernetes, composes run-specific environments and scenario inputs declaratively, and orchestrates batched execution with artifact and metric collection.
- The architecture combines Helm-chart packaging, dynamic Helmfile environment composition, OpenSCENARIO specifications, and Argo-based distributed execution.
- K8s-ROS reuses a versioned Helm chart template adapted through repository-specific configuration to provide consistent Kubernetes deployment interfaces.
- CI builds, validates, tests, versions, and publishes ready-to-use ROS 2 Helm charts, with optional repository-specific overrides.
- Dynamic Helmfile configuration combines a stable base with externally supplied values so each run can vary scenarios, maps, and ADS versions reproducibly.
- OpenSCENARIO test cases use simulator-compatible road representations, online metrics and thresholds, and later trajectory-based offline analysis.
- The workflow aggregates scenarios from configurable sources, optionally batches them, provisions per-scenario environments, executes within resource constraints, and persists artifacts and metrics.
5 Evaluation
The evaluation measures distributed CARLA scenario testing across runtime, throughput, resource utilization, and stability, then demonstrates the framework in a realistic ADS testing application. Results show substantial speedups from parallelism and batching, alongside real-time and control-plane trade-offs.
- Experimental setup: 200 OpenSCENARIO scenarios were evaluated on a three-node K3s cluster using CARLA and a full ADS software stack.The experiments varied simulator instances per node and workflow batch size while measuring end-to-end workflow time, throughput, resource utilization, and stability symptoms.
- Parallelism: 58.7 min: With batch size 9, increasing simulator instances per node from s1 to s3 reduced workflow time from 136 min to 58.7 min.The speed-up was not strictly proportional because of orchestration overheads and resource contention.
- Parallelism: 100% GPU peaks: Running three simulators per node increased average GPU utilization from 22.46% to 37.81% but produced execution delays and reduced real-time fidelity.Higher throughput therefore approached resource saturation and could not guarantee real-time behavior.
- Batching: 67.7 min: Increasing batch size from 9 to 60 with two simulators per node reduced runtime from 95.0 min to 67.7 min, while batch size 200 improved it only marginally to 65.2 min.The largest batch degraded control-plane responsiveness and destabilized the framework, whereas medium batches provided most overhead reduction.
- Overall results: 3–8×: Distributed configurations reduced end-to-end time and increased throughput relative to the sequential s1-b1 baseline.The comparison covers all tested distributed configurations against the sequential, single-simulator configuration.
- ADS testing application: The application demonstration executed 81 initial test cases, forwarded 65 runs to a posteriori evaluation, decomposed them into 396 base scenarios, and applied up to four metric checks.Results and metadata supported overview, coverage, and developer-oriented analysis through the safety evaluation workflow.
6 Conclusion
The paper presents a DevOps-driven framework for distributed ADS scenario testing on lightweight Kubernetes, standardizing application packaging and orchestrating parallel execution with artifact collection. Experiments show practical gains in workflow time and resource utilization, while future work targets broader observability and resource modeling.
- The framework standardizes ROS 2 applications as Helm charts, composes simulation environments through dynamic Helmfile manifests, and orchestrates scenario suites with Argo Workflows.It supports parallel execution and artifact collection across a lightweight Kubernetes cluster.
- Distributed execution robustly runs large test-case collections on a multi-node K3s cluster while substantially reducing end-to-end workflow time and improving overall resource utilization.
- The workflow connects heterogeneous scenario sources with downstream evaluation pipelines for metadata-based analysis and scenario-level safety assessment.
- Future work will improve monitoring and observability, extend resource modeling from the simulator to all ADS modules, and integrate evaluation more tightly into the cluster.These changes are intended to improve scheduling and traceability from simulation execution to evaluation results.