PaperlayerView exact revision on arXiv

Source-linked AI summary

Causal Analysis of Failures of Cyber-Physical Systems with Continuous Inputs

Tom Yaacov, David A. Kelly, Khen Elimelech, Hana Chockler

arXiv:2610.30737v1cs.RO

TL;DR

Black-box CPSs are difficult to verify fully before deployment, and prior CPS failure explanations used binary causal inputs that could not represent continuous changes. This paper adds distance-weighted causal concepts and an explanation algorithm guided by distance-weighted responsibility; experiments report more effective guidance than discrete responsibility across autonomous-driving simulator environments.

  • Problem

    Black-box neural components and complex real-world scenarios make full pre-deployment verification unrealistic, while prior CPS failure explanations could not represent continuous input changes.

    Source: §I · p. 1

  • Method

    The paper extends actual causality with distance-weighted actual cause and responsibility, then uses distance-weighted responsibility to guide a system-agnostic failure-explanation algorithm.

    Source: §VI · pp. 7–8

  • Results

    Across environments from two autonomous-driving simulators, distance-weighted responsibility provided more effective guidance than discrete responsibility and substantially reduced the computational effort required to find explanations.

    Source: §VI · p. 8

  • Takeaways & Limitations

    The extension lets actual-causality analysis account for intervention magnitude rather than only cardinality, including in causal models beyond CPSs.

    Source: §VI · p. 7

Abstract

from arXiv · show

Modern autonomous Cyber-Physical Systems (CPSs), such as self-driving cars, operate in increasingly complex environments while expected to maintain high reliability. The black-box nature of many of these systems makes it impossible to fully verify their behavior prior to deployment. When a failure occurs, it is important to analyze it to guide improvements to system robustness. A recent work introduced the novel concept of leveraging the framework of actual causality for CPS failure explanation. However, that work considered only a discrete, binary causal model that cannot capture continuous changes in system inputs. In this paper, we address these limitations by extending the actual causality framework with the notion of distance to account for the continuous characteristics of the system. Specifically, we introduce the novel concepts of distance-weighted actual cause and distance-weighted degree of responsibility, enabling causal influence to account for the magnitude of changes required to avoid a failure, rather than only its cardinality. While motivated by the continuous nature of CPSs, this constitutes a general extension of the theory of actual causality and enables more fine-grained analysis of continuous inputs in any causal model. Building on these notions, we present an efficient, practical, system-agnostic explanation-derivation algorithm that uses distance-weighted responsibility to guide the explanation search. We experimentally evaluate our approach across a number of environments from two autonomous-driving simulators with black-box neural controllers. The results show that the proposed extension provides more effective guidance than the current discrete approach.

I. INTRODUCTION

Black-box CPSs are difficult to verify fully before deployment, and failure explanations can help analyze incidents and guide future mitigation. This paper extends actual causality to account for continuous input changes and uses the extension to guide failure-explanation search.

I. INTRODUCTION

  • Black-box neural components and complex real-world scenarios make pre-deployment robustness verification unrealistic, while failure analysis can help prevent recurrence or prepare users to mitigate failures.The desired explanation identifies scenario elements that led to the failure.Source: §I · p. 1
  • The earlier CPS framework integrated system testing with causal models and used responsibility to guide an efficient explanation search, but assumed binary inputs.The present work extends that framework to continuous causal models.Source: §I · p. 1
  • Prior CPS failure explanations used binary variables, so their responsibility measure treated small and large input modifications equally and provided less nuanced search guidance.Actual causality is backward-looking and explains specific past events, unlike general probabilistic causal models,.Source: §I · pp. 1–2
  • Distance-weighted actual cause and responsibility account for intervention magnitude, and an improved explanation algorithm uses distance-weighted responsibility to guide the search.The extension is presented as a contribution to actual causality generally, not only CPS analysis.Source: §I · p. 2
  • Experiments across autonomous-driving environments report that distance-weighted responsibility gives more effective and scalable guidance for explanation search than prior approaches.The evaluation also assesses how well it prioritizes elements across multiple possible explanations for a failure.Source: §I · p. 2
  • Related work applies actual causality to XAI and image classification, while CPS alternatives include sensitivity analysis and model-dependent debugging.Other approaches use Bayesian models, ontology-based reasoning,, or state-machine trace comparisons.Source: §I · p. 2

A. Testing Cyber-Physical Systems

The testing setup treats environmental variation as the source of execution variability and fixes the system's initial state. Outcomes distinguish successful task completion from user-defined failure types.

A. Testing Cyber-Physical Systems

  • The model assumes environment variability is the only execution variability and holds the initial state fixed, making the environment the input.An execution is represented by a time-parameterized trajectory.Source: A. Testing Cyber-Physical Systems · p. 2
  • A status predicate marks task completion as success and assigns negative values to different user-defined failure types.The controller is assumed to perform a specific task while remaining robust to environmental variation.Source: A. Testing Cyber-Physical Systems · p. 2
  • The causal model represents obstacle placements as continuous location variables and uses O to indicate the outcome under an intervention.This contrasts with binary present-or-absent test inputs.Source: Figure 2 · p. 3

B. Illustration: an autonomous car

The running example is a black-box-controlled autonomous car navigating an obstructed track. The setup fixes the track shape and varies obstacle locations.

B. Illustration: an autonomous car

  • The running example uses a black-box neural controller trained to steer an autonomous car to the track's end while avoiding collisions.The example is presented as representative of CPSs that connect computational controllers with physical environments.Source: B. Illustration: an autonomous car · p. 3
  • The track shape is fixed, while each circular obstacle has an x position across the track and a y position along it.Obstacle coordinates are specified relative to the track.Source: B. Illustration: an autonomous car · p. 3
  • The status predicate marks reaching the end without collision as success and uses the collided obstacle's location to distinguish failure types.Crashes on or near obstacle i receive status −i.Source: B. Illustration: an autonomous car · p. 3

C. Actual causality

Actual causality models observed outcomes through variables and their dependencies, then uses interventions to identify causes and explanations. For CPS failures, the section represents obstacles and outcomes in a binary causal model and relates responsibility to necessary changes in the outcome.

C. Actual causality

  • A sufficient explanation is a minimal subset of obstacles whose presence reproduces the failure in the binary causal model.The model treats each obstacle as a binary variable and includes an outcome variable indicating whether the masked execution matches the original scene.Source: C. Actual causality · p. 3
  • Explanation computation is intractable, so existing approaches use heuristic search guided by degree of responsibility.The causal model represents variables and their dependencies with structural equations; interventions replace selected equations to test outcome changes.Source: C. Actual causality · pp. 3–4
  • An actual cause must hold in the observed context, be sufficient under a counterfactual assignment to change the outcome, and be minimal.Multiple distinct minimal explanations can exist for the same failure.Source: C. Actual causality · pp. 3–4
  • Responsibility is the inverse size of the smallest actual cause containing a variable, and is zero when no such cause exists.The paper uses necessary-condition responsibility—obstacle subsets whose removal changes the outcome—because crashes are rare in its simulation set.Source: C. Actual causality · p. 4

III. EXTENDING ACTUAL CAUSALITY WITH DISTANCE

The paper extends actual causality with distance to account for intervention magnitude, and introduces a continuous CPS model whose variables represent obstacle coordinates. Distance-weighted causes and responsibility favor explanations requiring smaller changes from the original scenario.

Continuous CPS causal model

  • The revised CPS model represents each obstacle with separate x and y coordinate variables, with × denoting absence and the output variable indicating the controller’s execution outcome.The model retains the binary model’s depth-2 structure.Source: §III · p. 4

Distance-weighted definitions

  • Distance-weighted actual causes and responsibility measure interventions relative to the original scenario using a predefined distance metric, favoring explanations achieved with smaller deviations.The framework is general to causal models, although distance becomes degenerate for binary variables, motivating a model that reflects the CPS’s continuous structure.Source: §III · p. 4
  • A distance-weighted actual cause must hold in the observed scenario, admit an assignment that changes the outcome, and be minimal under the distance measure.Minimality rules out a subset that changes the outcome with a smaller distance from the original assignment.Source: §III · p. 4
  • Distance-weighted responsibility is 1/(1 + d(v_n, v_s)) for the closest distance-weighted actual cause containing a variable, and 0 when none contains it.The new definitions specialize to the earlier cardinality-based definitions when distance counts variables, and preserve responsibility values in [0, 1] under a nonnegative distance.Source: §III · p. 4

Motivating example

  • Distance-weighted responsibility distinguishes obstacles by how much their configurations must change to alter the outcome, unlike discrete responsibility, which assigns obstacles 0, 5, 7, 8, and 9 equal values of 1.With Euclidean distance and 200 samples, the respective distance-weighted values are 0.36, 0.38, 0.55, 0.90, and 0.66, prioritizing the three obstacles in the explanation.Source: §III · pp. 4–5

IV. RESPONSIBILITY-GUIDED EXPLANATION-DERIVATION ALGORITHM

The algorithm ranks obstacles by approximate distance-weighted responsibility and incrementally searches that ordering for a sufficient explanation. The ranking guides search efficiency, while a subsequent minimization step preserves explanation validity.

Motivation

  • Ranking higher-responsibility obstacles first is intended to reduce the number of simulation runs needed to find an explanation.The paper notes that precise explanation computation is intractable.Source: §IV · p. 5
  • Distance-weighted responsibility provides a finer-grained alternative to the binary, obstacle-removal-only ranking in prior CPS work.The paper reports that the new approach reduces the overall effort required to compute explanations.Source: §IV · p. 5

Responsibility approximation

  • The responsibility approximation samples random interventions, orders them by distance, identifies minimal interventions that change the outcome, and updates each intervened element’s responsibility.The update uses 1/(1 + d(env, env′)).Source: §IV · p. 5

Explanation search

  • The search incrementally evaluates prefixes of obstacles ordered by approximate responsibility until the original failure is reproduced, then searches the candidate’s powerset for a smaller explanation.The responsibility approximation is computed by Alg. 1.Source: §IV · p. 5

Guarantee

  • The ranking is a search heuristic: even if its approximation is poor, the algorithm eventually considers all obstacles and minimizes the resulting subset to return a valid explanation.Responsibility approximation therefore affects search efficiency, not the validity of the result.Source: §IV · p. 5

V. EXPERIMENTAL EVALUATION

The evaluation compares explanation-search efficiency and multiple-explanation ranking across two autonomous-driving simulators. Distance-weighted responsibility generally guides both tasks more effectively than discrete responsibility.

Experimental Setup

  • The evaluation uses LiteRacer and MetaDrive environments with varied track geometries, while increasing obstacle counts from 10 to 14.For each track type, the authors created 100 failing examples and used 20 examples at each obstacle-count level.Source: §V · p. 6, Figure 4 · p. 6

Comparison

  • The search compares exhaustive enumeration with discrete and Euclidean distance-weighted responsibility rankings, using responsibility-estimation budgets of 50 and 100.Intervention sampling perturbs selected obstacle locations with standard normal Gaussian noise; a Chamfer distance is also evaluated.Source: §V · p. 6

Results

  • Distance-weighted responsibility consistently outperforms discrete responsibility in simulation runs, often substantially, with the gap widening as obstacle counts increase.The comparison covers the Euclidean-distance results; the Chamfer-distance advantage is reported as largely preserved.Source: §V · pp. 6–7, Figure 5 · p. 7

Multiple explanation evaluation

  • For multiple explanations, the study measures the fraction of explanations fully covered as obstacles are considered in responsibility-ranking order.Coverage curves are summarized by area under the curve, with higher AUC indicating a more effective ranking.Source: §V · p. 7, Table I · p. 7
  • Distance-weighted ranking achieves higher explanation-coverage AUC in almost all settings, often by a substantial margin, across the evaluated obstacle counts.The advantage is particularly pronounced in LiteRacer environments.Source: §V · p. 7, Table I · p. 7

VI. CONCLUSION

The paper extends actual causality with distance-weighted cause and responsibility to account for continuous inputs and intervention magnitude. Across two autonomous-driving simulators, the resulting responsibility ranking provides more effective guidance than discrete responsibility for finding explanations.

VI. CONCLUSION

  • Distance-weighted actual cause and responsibility extend actual causality to account for continuous inputs and the magnitude of interventions.The definitions are presented as an extension of actual-causality theory beyond CPS applications.Source: §VI · pp. 7–8
  • A system-agnostic black-box algorithm uses distance-weighted responsibility to guide the search for failure explanations.Source: §VI · p. 8
  • Across environments from two autonomous-driving simulators, distance-weighted responsibility provides more effective guidance than discrete responsibility and substantially reduces explanation-search effort.Source: §VI · p. 8

APPENDIX I EVALUATED TRACKS VISUALIZATION

The appendix visualizes the evaluated track types and configurations. MetaDrive visualizations include lanes in both directions, but the vehicle drives only on the right side.

APPENDIX I EVALUATED TRACKS VISUALIZATION

APPENDIX II EXECUTION TIME ANALYSIS

The appendices report runtime behavior, Chamfer-distance evaluation results, and Euclidean-distance ranking curves for multiple explanations.

APPENDIX II EXECUTION TIME ANALYSIS

APPENDIX III EVALUATION RESULTS WITH CHAMFER DISTANCE

APPENDIX IV MULTIPLE EXPLANATION EVALUATION EUCLIDEAN

Loading 2610.30737v1…