Source-linked AI summary
Explicability? Legibility? Predictability? Transparency? Privacy? Security? The Emerging Landscape of Interpretable Agent Behavior
Tathagata Chakraborti, Anagha Kulkarni, Sarath Sreedharan, David E. Smith, Subbarao Kambhampati
TL;DR
Interpretable-agent research uses overlapping notions for behavior that observers can understand, while related privacy and security work addresses hiding intentions. The paper develops a unified taxonomy of these concepts, compares their relationships across cooperative and adversarial settings, and identifies gaps including observer-model differences and limited longitudinal analysis.
Problem
Existing work on explicable, legible, predictable, transparent, private, and secure behavior lacks a coherent framework, despite addressing how observers infer agent intentions.
Method
The paper organizes these behaviors into a unified framework that models the agent, observer, observations, computational assumptions, and plan completions.
Results
The paper shows that interpretable behavior can be used dually to reveal or obfuscate agent intentions, while explicability, legibility, and predictability remain distinct and context-dependent.
Takeaways & Limitations
The taxonomy provides design directives for agents that account for observer models when choosing behavior, including behavior intended to communicate or conceal intentions.
Takeaways & Limitations
The survey focuses on behavior interpretability and identifies limited treatment of combined properties, evolving expectations, and experimental separation of expectations from preferences.
Abstract
from arXiv · showhide
There has been significant interest of late in generating behavior of agents that is interpretable to the human (observer) in the loop. However, the work in this area has typically lacked coherence on the topic, with proposed solutions for "explicable", "legible", "predictable" and "transparent" planning with overlapping, and sometimes conflicting, semantics all aimed at some notion of understanding what intentions the observer will ascribe to an agent by observing its behavior. This is also true for the recent works on "security" and "privacy" of plans which are also trying to answer the same question, but from the opposite point of view -- i.e. when the agent is trying to hide instead of revealing its intentions. This paper attempts to provide a workable taxonomy of relevant concepts in this exciting and emerging field of inquiry.
Introduction
The paper frames interpretable behavior around the observer’s model and organizes related notions—explicability, legibility, predictability, and transparency—within a unified taxonomy. It also highlights that these properties depend on interaction setting, observer computation, and whether plans reveal or obscure intentions.
- Model differences with the Observer: The central challenge is generating behavior that accounts for differences between the agent’s model and the observer’s expectations.A mismatch in beliefs, goals, capabilities, rewards, or computational assumptions can make behavior difficult to interpret.
- Model differences with the Observer: The framework represents the agent and observer through planning, observation, computation, and completion models.Plans solve a planning problem, emit observations, and are interpreted through the observer’s possible completions under a computational model.
- Interpretability? Plans versus Goals: Explicability measures alignment with observer expectations for a known goal, whereas legibility reduces ambiguity over possible goals.An explicable plan has a completion in both the agent and observer models; a legible plan minimizes observer models or goals compatible with the observations.
- Interpretability? Plans versus Goals: Predictability reduces ambiguity over possible plan completions given a goal, so an explicable plan can remain unpredictable when multiple completions are possible.Predictability adds a minimization over the cardinality of the possible plan set, and the paper treats explicability, legibility, and predictability as a spectrum.
- Online versus offline interactions.: Online and offline interaction settings change how explicability and predictability should be evaluated because these properties can vary as actions are observed.A plan prefix may become explicable or inexplicable after further actions, while offline predictability is defined over plan suffixes.
- Discussion: Existing research remains fragmented: observer-model learning can conflate explicability with predictability, experiments can mix expectations with preferences, and longitudinal effects are largely understudied.The paper identifies single-interaction studies and insufficiently separated human judgments as important limitations.
Turning the Tables
The paper turns from revealing intentions to adversarial behaviors that hide or distort them, organizing goal and plan obfuscation through distinctions among privacy, security, and deception. These concepts depend strongly on the observation and observer models, while existing work leaves broader agent–observer model differences largely unexplored.
- Goals versus Plans: Goal obfuscation is the inverse of legibility, while plan obfuscation is the inverse of predictability; either can be considered without necessarily obfuscating the other.The paper places both under a unified framework and notes that existing work has focused mainly on goal obfuscation.
- Online versus Offline: Partial or noisy observability makes offline obfuscation meaningful, and offline analysis can identify privacy-preserving prefixes even when fully obfuscating solutions are unavailable.The paper also notes that online obfuscation is especially relevant for evading pursuit or surveillance.
- The Observer and the Observation Model: Existing adversarial work varies in its observation and observer models but generally does not examine broader differences between agent and observer models.Some studies model a rational observer using plan-recognition probabilities, while others assume a complete planner.
- Security and Privacy: Privacy and obfuscation make multiple goals likely or possible from observations, whereas security requires all possible goals to produce the same observation tokens.The surveyed privacy formulations include pairwise goal likelihood and k-ambiguity, while security imposes the stronger consistency constraint.
- Security and Privacy: A secure plan is always private, because security additionally requires the same plan output across obfuscated goals.This independence-of-inputs constraint prevents an adversary from identifying the real goal by rerunning the planning algorithm.
- Obfuscation and Deception: A deceptive plan is always obfuscating, but it may or may not be dissimulating.Simulation hides the truth, whereas dissimulation makes a decoy goal more likely than the real goal.
Discussion
The discussion extends the plan-interpretability framework by considering richer observer models, observer preferences, active observers, and interactions among interpretability behaviors. It also identifies model reconciliation as a communication-based response when behavior cannot align with human expectations.
- Future Work: The survey connects plan interpretability with model reconciliation and identifies both as directions for extending the framework.These connections are presented as possible future-work directions.
- Communication and Model Reconciliation: When an explicable plan is infeasible, communication can establish common ground by expressing incapability, clarifying misunderstandings, augmenting goals, or even lying.Model reconciliation addresses differences between human and agent models until the plan is accepted as best under the circumstances.
- Further Generalizations: The framework should represent how an agent’s actions change the observer’s model, including epistemic effects beyond partial or noisy sensing.Explanatory actions illustrate how behavior can implicitly update the observer’s model.
- Further Generalizations: Interpretability objectives can optimize observer preferences over possible goals or plans, with cardinality treated as only one special case.This generalization applies to both legibility and obfuscation settings.
- An Active / Semi-Passive Observer: Existing work assumes passive observers, whereas semi-passive or active observers could alter observability, assist or impede execution, and change the relative importance of interpretability objectives.Surveillance and collaboration are examples where observer activity changes which behaviors matter most.
- Unified Approach to Interpretable Behavior: Because prior work studies interpretability notions largely in isolation, combining explicability, legibility, and predictability creates unresolved questions about goals and observer rationality.A behavior may be legible under one goal yet explicable under another, while explicability and legibility rely on different rationality assumptions.
Conclusion
The survey unifies diverse interpretable behaviors within a framework that accounts for the observer model and supports both revealing and obfuscating intentions. It also points toward broader privacy-preservation formulations in joint planning.
- Conclusion: The survey provides design directives for agents that account for observer models while deciding how to behave.It treats interpretability as relevant to the agent’s own decision-making process.
- Conclusion: Modeling interpretability is dual-use: agents can use it either to reveal or to obfuscate their intentions.The conclusion places privacy and obfuscation alongside interpretable behavior.
- Conclusion: The framework compares previously confused concepts and highlights gaps and future directions, including links to privacy preservation in multi-agent planning.The cited joint-planning work offers additional insight toward a more general formulation of privacy and obfuscation.