PaperlayerView exact revision on arXiv

Source-linked AI summary

Generalizable Robustness Testing of DNN-Based Robotic Navigation Systems via XAI-Guided Search

Khizra Sohail, Miren Illarramendi, Aitor Arrieta

arXiv:2610.06862v1cs.RO

TL;DR

Small input perturbations can produce unsafe behavior in DNN-controlled robotic systems, while image-specific testing and simulation-only evaluation leave generalization and physical transfer uncertain. The paper combines multi-image evolutionary search with Integrated Gradients guidance and reports higher robustness-test success than baseline techniques. Its evaluation also checks a perturbation subset on a physical robot, where transfer cannot be assumed from simulation alone.

  • Problem

    Perturbations optimized for individual images may not generalize across operational observations, and simulation-based failures may not transfer to physical systems.

    Source: §1 · p. 2

  • Method

    The framework uses multi-objective evolutionary search to optimize sparse perturbations over representative images, with aggregated Integrated Gradients guiding mutations toward influential regions.

    Source: §1 · p. 2

  • Results

    70.0% median success rate versus 53.85% for baseline techniques, with median hypervolume increasing from 0.65 to 0.73.

    Source: §8 · p. 22

  • Takeaways & Limitations

    The evaluation pairs simulation testing with physical-robot validation to assess system-level failures and their transfer.

    Source: §1 · p. 2

    The evaluation covers one DNN controller, one robotic platform, and one navigation task, limiting generalization to other models, robots, and environments.

    Source: §6 · p. 21

Abstract

from arXiv · show

**Context:** Deep Neural Networks (DNNs) increasingly control Cyber-Physical Systems (CPSs), yet small input perturbations can cause unsafe system-level behavior. Existing approaches often optimize perturbations for individual images and evaluate them only in simulation, limiting their generalizability and practical validity. **Objectives:** This work aims to generate robustness tests that remain effective across operational observations and to evaluate whether the resulting failures transfer from simulation to a physical robot. **Methods:** We propose an explainability-guided multi-objective evolutionary approach that generates sparse perturbations over representative images selected through visual and behavioral clustering. Aggregated Integrated Gradients guide mutations toward influential image regions. We evaluate the approach on a DNN-controlled LeoRover in Gazebo, conduct an ablation study, and validate a stratified subset of perturbations on the physical robot. **Results:** The approach achieved a median success rate of 70.0%, compared with 53.85% for unguided search, and increased median hypervolume from 0.65 to 0.73. Multi-image optimization improved the success rate from 50.0% to 57.5%, while XAI guidance further increased it to 70.0%. In the sim-to-real evaluation, simulation achieved 0.95 precision and 0.67 recall, and simulated and physical failure times showed a significant positive correlation of 0.617. **Conclusion:** Combining multi-image optimization with explainability-guided search improves robustness testing for DNN-controlled robotic systems. Simulation effectively identifies and prioritizes transferable failures, but physical validation remains necessary because some real-world failures are not reproduced in simulation.

1 Introduction

DNN-based robotic systems can turn small input changes into unsafe behavior, while image-specific tests and simulation-only evaluation leave generalization and physical transfer uncertain. The paper proposes multi-image, explainability-guided search and evaluates it in simulation and on a physical robot.

1 Introduction

  • Small DNN input changes can propagate through cyber-physical systems into unsafe behavior, motivating system-level robustness testing.The introduction gives lane departures and collisions as examples.Source: §1 · pp. 1–2
  • Existing perturbations often target one image and may exploit image-specific features rather than generalize across observations [10].Robotic controllers process sequences of images, not isolated inputs.Source: §1 · p. 2
  • Multi-image evaluation raises search costs, limiting image diversity, while uninformed mutations may spend effort on regions with little controller impact.The paper motivates both representative image selection and attribution-guided mutation.Source: §1 · p. 2
  • Simulation-only assessment leaves open whether generated failures transfer to physical robots, while full physical search is costly, slow, difficult to reproduce, and potentially unsafe.The paper frames sim-to-real transfer as an open problem.Source: §1 · p. 2
  • The proposed framework optimizes sparse perturbations across representative images for steering effect and modification cost, using aggregated Integrated Gradients to probabilistically prioritize influential regions.The mutation mechanism retains randomness to explore other image regions.Source: §1 · p. 2
  • The evaluation uses a DNN-controlled LeoRover in Gazebo, ablates multi-image optimization and XAI guidance, and validates a representative perturbation subset on the physical robot.This tests both system-level failure induction and simulation-to-real transfer.Source: §1 · p. 2

2 Background

The background frames robotic navigation as a closed-loop process where prediction errors may accumulate into system-level failures. It also introduces search-based testing and multi-objective trade-offs as context for generating robustness tests.

2.1 DNN-Based Robotic Navigation Systems

  • In closed-loop navigation, each control command changes the robot state and subsequent camera input, so prediction errors may or may not accumulate into system-level failure.The controller maps image x_t to command u_t = f(x_t).Source: §2.1 · p. 3
  • Model-output changes alone do not establish robot failure; robustness testing should also measure system-level execution outcomes.Small deviations can accumulate, while a large output change may not cause failure.Source: §2.1 · p. 3

2.2 Search-based Software Testing

  • Search-based testing iteratively generates candidate inputs against fitness functions, while multi-objective search preserves non-dominated solutions representing trade-offs.For robustness tests, the objectives can balance system effect against perturbation size or localization; DNN search effectiveness also depends on representation, fitness, and guidance [8] [6].Source: §2.2 · p. 3

3 XAI-guided search-based robustness testing

The framework selects representative operational images, aggregates their attribution maps, and uses the resulting guidance in a multi-objective search for perturbations that generalize across scenarios. Its pipeline combines joint visual-behavioral clustering with archive-based optimization of output deviation and perturbation cost.

3.1 Phase 1: Data Clustering Pipeline

  • Joint visual-behavioral clustering selects representative images, preserving diverse operational situations while reducing redundant frames.The pipeline reduces the joint feature space, applies HDBSCAN, and samples one image per cluster.Source: §3.1 · p. 4

3.2 Phase II: Generation of Global Attribution Map

  • Aggregated absolute Integrated Gradients maps identify consistently influential regions and become a probability distribution that biases mutation locations.The guidance remains stochastic while favoring image locations more likely to affect the DNN’s steering output.Source: §3.2 · p. 5

3.3 Phase III: XAI-Guided Multi-Objective Search Framework

  • An archive-based search applies the same localized perturbation across representative images, using Add, Remove, Move, and Scale mutations.Parents are sampled uniformly from the archive to explore different effectiveness-cost trade-offs.Source: §3.3 · p. 6
  • Candidates trade off average DNN-output deviation across representative images against the number of modified pixels.The deviation objective is maximized, while the modified-pixel objective is minimized.Source: §3.3 · p. 7
  • Pareto dominance retains non-dominated perturbations, while crowding-distance pruning preserves trade-off diversity when the archive exceeds capacity.The archive is returned as a diverse set of non-dominated perturbations after the iteration limit.Source: §3.3 · p. 7

4 Experimental Setup

The experimental setup evaluates robustness testing through three questions: search performance, optimization choices, and simulation-to-physical transfer. It uses a DNN-controlled LeoRover, a representative image set, and repeated Gazebo searches before physical validation.

4 Experimental Setup

  • The study asks whether guided search outperforms an unguided baseline, how optimization formulation affects tests across scenarios, and whether simulation outcomes transfer to the physical robot.The transfer assessment includes both simulated failures and benign outcomes.Source: §4 · pp. 7–8

4.1 Case Study System

  • The LeoRover follows a colored-tape closed-loop track using a front-facing camera; its DNN outputs linear and angular velocity commands through a ROS control pipeline.Physical failure is defined as both front wheels crossing the circuit lines while the line is no longer visible to the camera.Source: §4.1 · p. 8

4.2 Dataset and Experimental Parameters

  • The 16,048-image driving dataset was reduced to 51 representative images through clustering to make search feasible while preserving diversity across operational scenarios.The clustering pipeline uses PCA on joint visual–behavioral features followed by HDBSCAN.Source: §4.2 · pp. 8–9
  • Each configuration runs for 1,000 iterations across 30 independent runs, with perturbations limited to at most 10 circular regions and less than 10% of image pixels.Integrated Gradients uses 50 interpolation steps; the attribution maps correlate 0.9846 with the m = 300 reference map and have a mean absolute difference of 0.000006.Source: §4.2 · p. 9

4.3 Unguided Search Baseline (RQ1)

The experimental comparisons isolate unguided search, single- versus multi-image optimization, and physical validation of simulation-generated perturbations. Hardware testing is limited to a stratified subset because physical runs require substantially more time than simulation.

4.3 Unguided Search Baseline (RQ1)

  • The unguided baseline shares the proposed method’s representation, operators, objectives, budget, and evaluation pipeline, but samples regions uniformly and lacks attribution guidance and an online Pareto archive.Pareto-optimal candidates are extracted after the search budget is exhausted.Source: §4.3 · p. 10

4.4 Ablation Study (RQ2)

  • Single-image optimization evaluates each candidate on one representative image, whereas multi-image optimization evaluates it across the representative dataset with uniformly sampled spatial mutations.The comparison separates the effect of multi-image evaluation from attribution-guided mutation.Source: §4.4 · p. 10

4.5 Sim-to-Real Transfer of the Generated Perturbations (RQ3)

  • Physical validation covers 60 of the 600 proposed-method perturbations, stratified across simulated failure-time or perturbation-cost levels; each selected perturbation is run once on the robot.Hardware validation is restricted to the proposed approach because setup, monitoring, resets, and nominal-condition checks make runs time-consuming.Source: §4.5 · pp. 10–12

4.6 Evaluation Metrics

The evaluation measures whether perturbations cause out-of-bounds failures and compares successful tests by failure time and modification cost. Pareto-front quality summarizes the trade-off between those two objectives.

Success Rate

  • For each perturbation, simulation records a binary outcome indicating whether the rover goes out of bounds, separating system-level failure evaluation from the DNN-output-deviation proxy used during search.Each archive is replayed twice because simulator execution is stochastic.Source: §4.6 · p. 12
  • Success Rate is the proportion of archived perturbations that induce an out-of-bounds event.A higher rate means a larger fraction of generated perturbations produce system-level failures.Source: §4.6 · p. 12

Failure Time

  • Failure time measures elapsed simulation time until an out-of-bounds event; lower times indicate earlier failures, and the metric is defined only for successful perturbations.Unsuccessful perturbations are accounted for through Success Rate.Source: §4.6 · p. 12

Perturbation Cost and Hypervolume

  • Perturbation cost counts modified pixels, and the Pareto front minimizes both this cost and failure time among successful solutions.Hypervolume summarizes the normalized front, with higher values indicating a better trade-off between low failure time and low pixel changes.Source: §4.6 · p. 13

4.7 Statistical Analysis

The analysis uses non-parametric tests for stochastic metrics, corrects for multiple pairwise comparisons, and reports effect sizes. Experiments run on a CPU laptop with 32 GB RAM.

4.7 Statistical Analysis

  • Because several metric distributions failed the normality assessment, configurations were compared with Kruskal-Wallis tests followed by Mann-Whitney U pairwise tests when significant.Bonferroni correction sets the adjusted threshold to αadj = 0.0083 for six comparisons per metric; Vargha–Delaney effect sizes are also reported.Source: §4.7 · pp. 13–14

4.8 Computational Resources For Experiment

  • Experiments ran on a personal laptop with an Intel Core Ultra 7 155H processor and 32.0 GB RAM, using the CPU in a 64-bit x64 environment.Source: §4.8 · p. 14

5 Analysis of the Results and Discussion

The experiments compare XAI-MOEA with unguided and ablated optimization variants, then assess how well simulation outcomes transfer to the physical LeoRover. XAI-MOEA improved success-rate and hypervolume results, while simulation was useful for prioritizing failures but missed some physical failures.

5.1 RQ1 – Sanity Check

  • 70.0% vs. 53.85% median success rate and 0.73 vs. 0.65 median hypervolume favored XAI-MOEA over Unguided Search (p < 0.00001).The hypervolume considers successful perturbations and summarizes the trade-off between perturbation cost and failure-inducing effectiveness.Source: §5.1 · p. 14, Figure 8 · p. 15, Figure 9 · p. 16
  • 5,915 seconds vs. 3,752 seconds was XAI-MOEA’s median runtime versus Unguided Search, with runs completing in approximately 1.6 hours on median.The authors characterize this additional computation as practical for offline robustness testing.Source: §5.1 · p. 14, Figure 10 · p. 17

5.2 RQ2 – Ablation of the Optimization Formulation

  • 50.0% to 57.5% median success rate followed the move from single-image Variant I to multi-image Variant II; XAI-MOEA reached 70.0%.XAI-MOEA’s success-rate improvement over both variants was significant after correction; Variant I versus Variant II was not significant after correction (p = 0.05).Source: §5.2 · pp. 15–16, Figure 11 · p. 18
  • 0.73 median hypervolume exceeded Variant I (0.69) and Variant II (0.68), but differences were not significant; median runtime ranged from 2,920 to 5,915 seconds.Variant II’s median runtime was approximately 4,556 seconds, and even the most expensive configuration completed in approximately 1.6 hours on median.Source: §5.2 · pp. 15–16, Figure 12 · p. 18, Figure 13 · p. 19

5.3 RQ3 – Sim-to-Real Transfer:

  • 0.9508 precision and 0.6744 recall came from 120 simulator-to-hardware comparisons; 28 false negatives were physical failures missed in simulation.The simulator produced 58 true positives, 3 false positives, and 31 true negatives; the authors describe simulation as useful for pre-screening, not as proof of safety.Source: §5.3 · pp. 17–18, Figure 14 · p. 20
  • For 27 perturbations failing in both environments, simulated and physical failure times correlated (r = 0.6170, p = 2.5 × 10−7), although physical failures generally occurred earlier.The tracks had different geometries, so absolute failure times were not directly comparable; the correlation reflects relative severity ordering.Source: §5.3 · pp. 18–19, Figure 15 · p. 20

6 Threats to Validity

The study identifies validity threats related to evaluation scope, parameter choices, stochasticity, and the use of a proxy objective. It describes measures taken to mitigate these threats, including physical evaluation, ablation, repeated runs, and multiple outcome metrics.

External validity

  • Evaluation on one DNN controller, one robotic platform, and one navigation task limits generalization to other models, robots, and operational environments.Evaluating perturbations in both simulation and on the physical system partially mitigates this external-validity threat.Source: §6 · p. 21

Internal validity

  • Selected clustering, attribution, perturbation, and search parameters may influence the results.The study used the same perturbation representation, objectives, and search budget across compared configurations and conducted an ablation study.Source: §6 · p. 21

Conclusion validity

  • Stochastic search and simulation may affect observed results; the authors mitigated this with 30 independent runs, duplicate archive replays, and corrected statistical tests.They also conducted effect-size analysis.Source: §6 · p. 21

Construct validity

  • DNN-output deviation is a proxy objective that does not necessarily imply system-level failure.The study also evaluates actual out-of-bounds executions, success rate, failure time, perturbation cost, and hypervolume.Source: §6 · p. 21

7 Related Work

Related work spans model-level adversarial perturbations, scenario-level CPS testing, and camera-input perturbations for autonomous systems. This work distinguishes itself by searching for localized sparse image perturbations and evaluating their system-level effects in simulation and on a physical robot.

Related work on adversarial robustness

  • Adversarial vision work established model vulnerabilities and perturbation-generation mechanisms, but typically evaluates classification, detection, or label outcomes rather than closed-loop robotic system failures.This paper focuses on whether DNN output changes propagate through control to unsafe behavior such as leaving the intended track.Source: §7 · p. 21

Related work on CPS testing

  • CPS scenario-testing approaches explore semantic or environmental changes, whereas this work fixes the operational scenario and searches for localized camera-input perturbations.The perturbations are applied consistently across representative camera images and assessed for robotic navigation failures.Source: §7 · p. 21

Related work on camera-input perturbations

  • Camera-perturbation studies often use image-specific optimization, task-specific formulations, or predefined operators; this work aggregates Integrated Gradients and uses them to guide multi-objective search for sparse universal structures.The generated perturbations are evaluated in both a simulator and on a physical robot.Source: §7 · pp. 21–22

8 Conclusion

The paper concludes that XAI-guided multi-image evolutionary search improves perturbation-based robustness testing for DNN-controlled robots. Simulation helps identify transferable failures, but its lower recall means targeted physical validation remains necessary.

Conclusion

  • The approach combines representative-image optimization, sparse perturbations, and a global Integrated Gradients attribution map to guide search.It achieved a median success rate of 70.0% versus 53.85% for baseline techniques and improved median hypervolume from 0.65 to 0.73.Source: §8 · p. 22
  • Multi-image optimization increased median success rate from 50.0% to 57.5%, and XAI guidance further increased it to 70.0%.The success-rate improvements over single-image and unguided multi-image variants were statistically significant, while hypervolume differences among evolutionary variants were not.Source: §8 · p. 22
  • Simulation reached 0.95 precision and 0.67 recall for physical failure transfer, so physical validation remains necessary because some real failures are missed.The median execution time was approximately 1.6 hours, which the authors considered practical for offline testing.Source: §8 · p. 22
  • Future work will evaluate additional robots, controllers, environments, attribution methods, and perturbation types.Source: §8 · p. 22

Replication Package

The replication package is publicly available through Zenodo, with the virtual machine image provided separately because of its large size.

Replication Package

  • The replication package is publicly available on Zenodo.Its record is https://doi.org/10.5281/zenodo.21496097.Source: Replication Package · p. 22
  • The virtual machine image is available through a separate Zenodo record.Its record is https://doi.org/10.5281/zenodo.21497744.Source: Replication Package · p. 22
  • The package and virtual machine use separate records because the virtual machine image is large.Source: Replication Package · p. 22
Loading 2610.06862v1…