Source-linked AI summary
How Should a Robot Assess Risk? Towards an Axiomatic Theory of Risk in Robotics
Anirudha Majumdar, Marco Pavone
TL;DR
Robots need principled risk metrics for safe decision making under uncertainty, but robotics lacks a firm basis for determining which metrics are rational. The paper advocates axioms, characterizes the resulting distortion risk metrics, and discusses common-metric pitfalls and sequential consistency. It concludes that these ideas provide preliminary directions toward an axiomatic framework for quantifying risk in robotics.
Problem
Robotics lacks a firm theoretical foundation for choosing risk metrics that support rational and trustworthy assessments in safety-critical uncertain settings.
Method
The paper advocates axioms for robotics risk metrics, characterizes the metrics satisfying them, and examines their interpretation, common pitfalls, and sequential time consistency.
Results
The advocated axioms characterize distortion risk metrics, while sequential analysis identifies compounding metrics as time-consistent and locally property-satisfying under stated conditions.
Takeaways & Limitations
The paper provides preliminary directions toward an axiomatic theory for quantifying risk and safety in robotics applications.
Takeaways & Limitations
In sequential decision making, the composition required for time consistency is more difficult to interpret than the one-step metrics, creating a tension the paper leaves for future work.
Abstract
from arXiv · showhide
Endowing robots with the capability of assessing risk and making risk-aware decisions is widely considered a key step toward ensuring safety for robots operating under uncertainty. But, how should a robot quantify risk? A natural and common approach is to consider the framework whereby costs are assigned to stochastic outcomes - an assignment captured by a cost random variable. Quantifying risk then corresponds to evaluating a risk metric, i.e., a mapping from the cost random variable to a real number. Yet, the question of what constitutes a "good" risk metric has received little attention within the robotics community. The goal of this paper is to explore and partially address this question by advocating axioms that risk metrics in robotics applications should satisfy in order to be employed as rational assessments of risk. We discuss general representation theorems that precisely characterize the class of metrics that satisfy these axioms (referred to as distortion risk metrics), and provide instantiations that can be used in applications. We further discuss pitfalls of commonly used risk metrics in robotics, and discuss additional properties that one must consider in sequential decision making tasks. Our hope is that the ideas presented here will lead to a foundational framework for quantifying risk (and hence safety) in robotics applications.
1 Introduction
Robotics lacks a firm theoretical basis for choosing risk metrics, despite the importance of safe decision making under uncertainty. The paper proposes an axiomatic framework to identify sensible metrics and examines their application-specific and sequential implications.
- Motivation: Safe robot planning commonly summarizes stochastic costs with expected value, but this choice is not well justified for risk-sensitive, safety-critical settings.Shaping costs to capture tail risk can produce irrational behavior, while worst-case assessments may be excessively conservative.
- Problem: Risk metrics map cost random variables to real numbers, with expected cost representing risk neutrality and worst-case cost representing extreme risk aversion.The paper seeks metrics between these extremes that support rational and trustworthy risk assessment.
- Problem: Robotics has received little attention to the question of which risk metrics are sensible, leaving no firm theoretical foundation for application-specific selection.Existing alternatives include distributional robustness and chance-constrained programming, but the paper focuses on axiomatic evaluation of risk metrics.
- Contribution: The paper advocates axioms for robotics risk metrics and characterizes the resulting class as distortion risk metrics, paralleling axiomatic work in finance.Distortion risk metrics form a subset of coherent risk metrics and have been studied previously in finance.
- Significance: The framework is motivated by the possibility that irrational risk assessment could lead robots to behave harmfully toward themselves, humans, or other autonomous agents.The authors also anticipate that safety-critical robots may eventually face regulatory requirements for officially approved risk metrics.
2 Assessing Risk: Preliminaries
The paper models uncertain robot outcomes with monetary cost random variables and defines risk as a mapping from those variables to real numbers. A certification-deposit interpretation connects the abstract metric to risk attitudes and applications.
- Risk Metrics: A risk metric is a mapping ρ : Z →R from a cost random variable to a real number.The outcome space is finite, with probabilities assigned by P and costs assigned by Z : Ω→R.
- Interpretation: The framework assumes that costs are expressed in monetary terms, making them tangible and interpretable for risk assessment.This assumption also supports reasoning about insurance policies for safety-critical robots.
- Interpretation: Perceived risk ρ(Z) is interpreted as the deposit a robot owner must provide to cover potential operating costs before deployment.A risk-neutral agency may use ρ(Z) = E[Z], whereas a highly conservative agency may require the worst-case cost.
- Interpretation: The Robot Certification Agency is a pedagogical device for interpreting risk and motivating axioms; in practice, the robot’s decision-making system assesses risk.The example uses an autonomous car whose random costs may include fuel, time, crashes, and mechanical wear.
3 An Axiomatization of Risk Metrics for Robotics Applications
The paper proposes six axioms for sensible robotics risk metrics, characterizes metrics satisfying them, and uses examples to expose pitfalls of common alternatives.
- Axioms and their interpretations: The authors propose six axioms that risk metrics used in robotics should satisfy to be considered sensible assessments of risk.The axioms are presented with formal statements and intuitive interpretations.
- Axioms and their interpretations: Axioms A1–A6 require properties including monotonicity, translation invariance, positive homogeneity, subadditivity, comonotone additivity, and law invariance.Together, these properties capture essential requirements for the proposed interpretation of risk.
- Examples and pitfalls of commonly used risk metrics: The paper motivates distortion risk metrics as the general class that precisely characterizes risk metrics satisfying A1–A6 and supports generating new examples.This framework is presented as a principled alternative for robotics applications.
- Examples and pitfalls of commonly used risk metrics: Risk metrics satisfying A1–A6 include CVaR, expected cost, and worst-case assessment, spanning assessments from risk-neutral to worst-case.CVaR measures the expected cost in the upper tail and is interpreted as quantifying how bad bad outcomes are.
- Examples and pitfalls of commonly used risk metrics: Mean-variance risk can violate monotonicity: although controller π has lower costs in every disturbance outcome, β = 1 makes the robot prefer π′.The example shows that failing an advocated axiom can produce unreasonable controller selection.
- Examples and pitfalls of commonly used risk metrics: VaR fails subadditivity and can select costs Z′ even when a reasonable agent would prefer Z.VaRα(Z) ≤ 0 is equivalent to the chance constraint P[Z > 0] ≤ α.
4 Distortion Risk Metrics
Distortion risk metrics are defined by Axioms A1–A6 and characterized through representations involving CVaR, Choquet integrals, and risk envelopes. The paper advocates them for robotics while noting that some low-level tasks may require reconsidering particular axioms.
- Representation theorems: Coherent risk metrics can be represented as expectations under a worst-case probability mass function selected from a compact convex risk envelope.This representation also captures robustness to uncertainty in the underlying distribution.
- Representation theorems: Comonotonic risk metrics, satisfying Axioms A1–A5, are characterized using Choquet integrals based on monotone, normalized, and submodular set functions.The Choquet integral generalizes integration by allowing nonlinear dependence on the random variable.
- Distortion risk metrics: Risk metrics satisfying Axioms A1–A6 are called distortion risk metrics and are equivalent to risk metrics of the form (4).They are also equivalent to spectral risk measures.
- Representation theorems: Distortion risk metrics admit a CVaR representation obtained by choosing a probability measure ν on [0,1].The representation theorem provides a way to generate examples by selecting functions ν satisfying its assumptions.
- Robotics applications: The paper advocates distortion risk metrics for robotics risk evaluation, contrasting them with commonly used metrics such as mean-variance and VaR.The proposed class is presented as satisfying the advocated axioms while retaining useful representation theorems.
- Robotics applications: Subadditivity and comonotone additivity may be unclear for low-level control tasks whose cost components cannot be optimized independently.The paper leaves open whether these axioms should be abandoned or replaced in such applications.
5 Sequential Decision Making and Time Consistency
Sequential decision making requires risk assessments that remain locally relevant and temporally consistent. Dynamic risk metrics, constructed by compounding one-step metrics, provide the advocated approach, though their interpretation is less direct.
- 5 Sequential Decision Making and Time Consistency: Sequential robotics tasks require properties beyond static risk assessment, including local relevance and temporal consistency.The local property excludes future scenarios known to be impossible, while time consistency aligns assessments across decision stages.
- 5 Sequential Decision Making and Time Consistency: Failure of time consistency can make a single-policy optimization reject a solution that appears acceptable from every second-stage perspective.In the CVaR example, the optimized cost has τ⋆ > 0, although CVaRα(cN(xN)) ≤ 0 in every state at k = 1.
- 5 Sequential Decision Making and Time Consistency: Applying one static risk metric to the sum of all stage costs does not generally produce time-consistent risk assessments.The paper therefore introduces dynamic risk metrics that map future cost streams to assessments at multiple time steps.
- 5 Sequential Decision Making and Time Consistency: Compounding one-step risk metrics constructs time-consistent assessments, and under mild conditions every time-consistent metric has this form.The one-step metrics assess costs incurred at the next time step from the current time-step perspective.
- 5 Sequential Decision Making and Time Consistency: Compounded distortion risk metrics preserve one-step rationality while ensuring time consistency for sequential decision making.The paper advocates this construction for sequential tasks, while noting that the resulting composition is harder to interpret than a single metric on total cost.
6 Discussion and Conclusions
The paper presents preliminary directions toward an axiomatic theory of robotic risk, while emphasizing that suitable axioms and metric choices remain application-dependent and open to further work.
- The proposed axioms define distortion risk metrics as a general class for risk assessment in robotics.
- Further axioms remain an important direction for future work and may depend on the application domain.
- A4 and A5 may matter for high-level decisions with diversification but may not apply to low-level control where diversification is unavailable.
- A particular risk metric could be learned from human risk evaluations in the relevant application domain.The paper describes this as one possible approach and notes prior first steps using coherent risk metrics.
- The authors hope this work will support convergence on a standard class of risk metrics for robotics.