PaperlayerView exact revision on arXiv

Source-linked AI summary

Sufficient and Necessary Causal Explanations for the Analysis of Autonomous Systems

Tom Yaacov, Khen Elimelech, Hana Chockler, Moshe Y. Vardi

arXiv:2610.29724v1cs.RO

TL;DR

Failure analysis must address both reproducing observed failures and finding changes to avoid them, while black-box components make system behavior difficult to interpret. The paper introduces necessary explanations and a responsibility-guided approximation within a causal framework; it reports improved efficiency and stronger avoidance results than existing methods.

  • Problem

    Failure analysis is difficult for autonomous systems with black-box components, and prior sufficient explanations support reproducing failures but do not address how to avoid them.

    Source: §I · p. 1

  • Method

    The paper formalizes necessary explanations within actual causality and develops an approximation algorithm based on causal responsibility.

    Source: §I · p. 1, §VI · p. 5

  • Results

    The responsibility-guided approach uses fewer simulation runs than exhaustive search across evaluated environments, and the paper reports higher success, lower computation time, and better quality than existing avoidance methods.

    Source: §I · p. 2, §VII · p. 6

  • Takeaways & Limitations

    Necessary and sufficient explanations provide a unified causal framework for addressing failure avoidance and reproduction.

    Source: §IX · p. 8

    In some MetaDrive configurations, responsibility ranking appears insufficient to offset the additional simulation budget needed to estimate it.

    Source: §VII · p. 6

Abstract

from arXiv · show

Understanding failures of autonomous systems is challenging, particularly when their behavior depends on black-box components such as neural networks. In this work, we make a distinction between two objectives of failure analysis: reproduction, which seeks to identify conditions sufficient to reproduce an observed failure, and avoidance, which seeks to identify the changes required to prevent it. Previous work on causal explanations of system failures is based on the concept of sufficient explanations and naturally supports reproduction. Nevertheless, we observe that it does not address the problem of avoiding the failure. In this paper, we introduce and formalize the concept of necessary explanations for system analysis as the correct formalism for failure avoidance. We place necessary explanations in the framework of actual causality and show that, together with sufficient explanations, they provide a unified causal framework for addressing both objectives of failure analysis. We present an efficient approximation algorithm for generating necessary explanations based on the notion of causal responsibility. We evaluate the efficiency of our approach across multiple environments. We demonstrate the practical utility of necessary explanations for failure avoidance through counterfactual generation and show that it outperforms existing methods.

I. INTRODUCTION

The introduction frames failure analysis as two distinct goals—reproducing a failure and avoiding it—and proposes necessary explanations to address the avoidance gap left by prior sufficient explanations. It presents a causal framework and an approximation algorithm, evaluated for efficiency and practical failure avoidance.

  • Failure analysis is difficult when autonomous systems rely on opaque components such as neural networks and operate in complex, variable environments.Source: §I · p. 1
  • Reproduction seeks conditions sufficient to recreate a failure, whereas avoidance seeks the minimal changes needed to change its outcome; neither objective can be deduced from the other.Source: §I · p. 1
  • Prior causal failure explanations based on sufficient explanations support reproduction, but changing or removing one sufficient subset does not guarantee avoidance because multiple such subsets may exist.Source: §I · p. 1
  • Necessary explanations identify failure-execution elements whose removal avoids the failure, and the paper combines them with sufficient explanations in a unified causal framework.The paper also develops a practical approximation algorithm based on causal degree of responsibility.Source: §I · p. 1
  • The paper reports higher success, lower computation time, and better quality than prior non-causal failure-avoidance methods, with benefits across two simulators of different fidelity.The evaluation uses an autonomous car with a black-box neural controller trained to avoid collisions.Source: §I · p. 2
  • Existing failure-avoidance approaches mainly generate nearby passing examples, but may emphasize downstream effects or changes unnecessary for avoiding failure,.A related root-cause approach relies on causal discovery, unlike this paper's reasoning within an abstract model.Source: §I · p. 2

II. PRELIMINARIES

The preliminaries model a system as producing a trajectory in an environment and assume it can be simulated without access to its explicit internal model. A task-specific testing model assigns outcomes, illustrated with an autonomous car navigating obstacles.

II. PRELIMINARIES

  • The formalism covers time-extended systems whose outputs are trajectories, while allowing a trajectory to consist of a single final state.Source: §II · p. 2
  • An environment captures external factors, and its variability determines trajectories when the initial state is treated as predetermined.The trajectory ξ concatenates the system states from the initial state through the end of execution.Source: §II · p. 2
  • The system may be a black box if it can be simulated from an environment specification, and analysis distinguishes optional environment elements from mandatory parameters.Source: §II · p. 2

Illustration: an autonomous car

  • A task-specific testing model maps an environment to an outcome based on the resulting trajectory, distinguishing task success from different failure types.In the car example, the track shape is a parameter, obstacles are elements, and the neural controller is tasked with reaching the track end without collisions.Source: §II · pp. 2–3, Illustration: an autonomous car · p. 3

III. ACTUAL CAUSALITY FOR FAILURE EXPLANATION: FUNDAMENTALS

The section introduces actual causality as a way to reason about observed outcomes and instantiates it for failures in a particular scene. Its causal model represents obstacle presence and uses simulation outcomes while treating the controller as a black box.

A. Actual causality

  • Actual causality represents variables and their dependencies with structural equations, with a causal network depicting the direction of dependence.Source: A. Actual causality · p. 3
  • A context assigns values to variables without incoming edges, and together with a causal model specifies the situation being analyzed.Source: A. Actual causality · p. 3
  • Interventions replace selected variable equations with assignments, enabling reasoning about how changes affect the outcome.Source: A. Actual causality · p. 3

B. Causal framework for failure explanation

  • For a crash scene, the model uses one variable per obstacle as a presence mask and a binary outcome variable indicating whether the masked scene's outcome matches the original.The model is specific to the scene; the controller remains a black box whose behavior is inferred from simulation runs.Source: B. Causal framework for failure… · p. 3

C. Sufficient explanations for test outcomes

A sufficient explanation is a minimal subset of scene obstacles that can reproduce the observed failure when the other obstacles are removed. The section connects this definition to understanding and reproducing failures.

  • A sufficient explanation is a minimal obstacle subset that produces the failure when assigned present on an otherwise empty track.Minimality means no strict subset also satisfies that condition.Source: C. Sufficient explanations for test… · p. 3
  • Because model variables correspond one-to-one with obstacles, a sufficient explanation maps directly to scene obstacles; more than one minimal sufficient explanation may exist.Source: C. Sufficient explanations for test… · pp. 3–4
  • Sufficient explanations support understanding and failure amplification by preserving the identified obstacles while varying the rest of the environment to generate further failures.This structured approach is particularly useful when failures are rare and additional failing executions are difficult to obtain.Source: C. Sufficient explanations for test… · p. 4

IV. IDENTIFYING EXPLANATION OBJECTIVES

Failure explanations serve two distinct objectives: reproducing an outcome and avoiding it. Necessary explanations extend actual causality to identify minimal changes that avoid failure, complementing sufficient explanations in a unified framework.

IV. IDENTIFYING EXPLANATION OBJECTIVES

  • Outcome avoidance targets changes to a failing scenario, while outcome reproduction identifies conditions for making an outcome happen again.Avoidance can support repair during or after a failure; reproduction can help study rare outcomes during debugging or falsification.Source: §IV · p. 4
  • Avoidance aligns with counterfactual generation, while reproduction aligns with sensitivity analysis.The paper argues that actual causality, with its proposed extension, can address both objectives.Source: §IV · p. 4, §V · p. 5

V. ACTUAL CAUSALITY FOR FAILURE EXPLANATION: EXTENSION FOR FAILURE AVOIDANCE

  • A necessary explanation is a minimal set of elements whose removal changes the failure outcome; multiple such explanations may exist.This contrasts with sufficient explanations, which identify elements enough to reproduce the failure.Source: §V · pp. 4–5
  • Sufficiency and necessity capture different intervention directions, and neither explanation type is automatically deducible from the other.A subset whose removal avoids failure need not itself be sufficient to cause failure, and multiple sufficient sets may exist.Source: §V · p. 4, §IV · p. 4
  • Necessary explanations can yield high-quality counterfactuals by changing the original scene minimally, while sufficient explanations can seed additional examples of the same failure.Together, the two explanation types provide a unified causal framework for failure analysis.Source: §V · p. 5

VI. GENERATING NECESSARY EXPLANATIONS

The algorithm estimates causal responsibility to rank obstacles, then searches for a minimal set whose removal changes the outcome. Its ranking guides the search for necessary explanations.

VI. GENERATING NECESSARY EXPLANATIONS

  • Responsibility ranks elements by their causal relevance, using the inverse size of the smallest necessary explanation containing each element.The algorithm uses approximate responsibility because its precise computation is intractable.Source: §VI · p. 5
  • The algorithm samples m interventions on repairable obstacles observed in the original trajectory and retains minimal outcome-changing subsets to estimate responsibility.For each element in a retained subset, responsibility is updated using the inverse subset size.Source: §VI · p. 5
  • It ranks obstacles by estimated responsibility, greedily tests removals, then removes redundant obstacles while preserving an outcome change.The resulting explanation is necessary and minimized by checking whether smaller subsets still change the outcome.Source: §VI · p. 5

VII. PERFORMANCE EVALUATION

Across the evaluated environments and obstacle counts, responsibility-guided search used fewer simulation runs than exhaustive search, with the size of the reduction varying by scenario. The evaluation covered LiteRacer and MetaDrive track configurations.

VII. PERFORMANCE EVALUATION

  • The evaluation used LiteRacer and MetaDrive simulators across multiple track geometries, with each simulator's default neural-network controller.LiteRacer covered sinusoidal, segmented, and lane-shift tracks; MetaDrive covered curve, straight, and mixed tracks.Source: §VII · p. 6
  • Each track configuration had 100 failing examples with 15–19 obstacles, and the crashing obstacle was fixed and excluded from repair.All obstacles were observed during execution and could potentially influence vehicle behavior.Source: §VII · p. 6
  • Responsibility-guided search required fewer simulation runs than exhaustive search across all evaluated environments and obstacle counts.The improvement was more pronounced in LiteRacer; in some MetaDrive configurations, responsibility estimation costs were not fully offset by the guidance.Source: Table I · p. 6, §VII · p. 6

VIII. COUNTERFACTUAL GENERATION EVALUATION

The section evaluates necessary-explanation-guided counterfactual generation for validity, similarity, and efficiency. Its approach is compared with DeCaF and random search across simulation budgets and environments.

Evaluation setup

  • The evaluation generates one passing counterfactual per failing example and measures validity, Euclidean-distance similarity, and generation time.The method spends one-third of the simulation budget identifying necessary explanations, then perturbs elements from a randomly selected explanation while leaving other elements unchanged.Source: §VIII · pp. 6–7

Results

  • At 400 runs, the necessary explanation-based approach achieves the highest validity rate across MetaDrive settings, while both it and random search reach 100% in LiteRacer settings.DeCaF fails to generate valid counterfactuals in several LiteRacer cases; no method is perfect across all MetaDrive settings.Source: §VIII · p. 7, Table II · p. 7

Evaluation setup

  • The comparison includes DeCaF, which learns system behavior from simulated passing and failing examples, and a random search baseline.DeCaF uses an M5 model tree and KD-Tree Nearest Neighbors; the random baseline returns the closest sampled perturbation that avoids failure.Source: §VIII · p. 7

Results

  • Necessary explanations produce the closest counterfactuals in almost every setting, with DeCaF generally second among cases where all methods succeed.The comparison uses average distance over a shared valid evaluation set.Source: Table III · p. 7, §VIII · p. 7
  • The approach exhibits more stable improvement: counterfactual distance decreases monotonically with the number of samples in almost all settings, while DeCaF has several cases where larger budgets worsen counterfactual quality.Results for settings with 15–18 obstacles are provided at.Source: §VIII · p. 7
  • At 400 runs, the approach has runtimes comparable to random search and is faster than DeCaF across both simulators and obstacle counts.The reported comparison is elapsed time in minutes.Source: §VIII · p. 7

IX. CONCLUSION

The conclusion frames failure analysis around reproduction and avoidance. It presents sufficient and necessary explanations in a unified actual-causality framework and summarizes the algorithm and evaluation.

Conclusion

  • Fig. 5 plots average counterfactual distance against simulation runs, while Table IV reports elapsed time for the compared methods.The figure covers 100–400 runs and identifies lower distance as better.Source: Figure 5 · p. 8, Table IV · p. 8
  • Sufficient explanations suit failure reproduction, while necessary explanations are formulated for failure avoidance and repair.Together, both explanation types can be computed in a unified actual-causality framework.Source: §IX · pp. 7–8
  • The paper presents an algorithm for necessary explanations and reports evaluation of its efficiency and comparison with existing counterfactual-generation methods.Source: §IX · p. 8

APPENDIX I EVALUATED TRACK VISUALIZATION

The appendix visualizes evaluated track types, including LiteRacer-style tracks and MetaDrive environments; the MetaDrive visualizations show lanes in both directions, with driving on the right side.

Sinusoidal tracks

Segmented tracks

Lane Shift tracks

MetaDrive tracks

APPENDIX II EXECUTION TIME ANALYSIS

The appendix reports that simulation-run counts dominate runtime in the responsibility-guided approach. It also identifies additional counterfactual validity and distance results across budgets and obstacle counts.

APPENDIX II EXECUTION TIME ANALYSIS

  • In the sinusoidal short narrow environment, computation beyond simulation runs is negligible relative to runtime for the responsibility-guided approach.The execution-time analysis compares exhaustive and responsibility-guided approaches across numbers of simulation runs.Source: APPENDIX II EXECUTION TIME ANALYSIS · p. 12, Figure 21 · p. 12

APPENDIX III ADDITIONAL COUNTERFACTUAL GENERATION EVALUATION RESULTS

Loading 2610.29724v1…