Source-linked AI summary
The Mirror Agent Model: a Bayesian Architecture for Interpretable Agent Behavior
Michele Persiani, Thomas Hellström
TL;DR
The paper addresses the lack of a unified framework for interpretable behavior and explanations. It presents the Mirror Agent Model, which mirrors the agent’s intentional model in the observer model and supports informative communication, legible behavior, and saliency-based explanations. Preliminary results illustrate informative intention communication, the legibility–optimality trade-off, and reward- and obstacle-focused saliency explanations.
Problem
Existing interpretability and explainability techniques are scattered across methods, algorithms, and theoretical frameworks, leaving a need for a consistent general computational framework.
Method
The Mirror Agent Model uses structurally mirrored Bayesian-network models of the agent and observer, applying Theory of Mind to interpretable behavior and explanations across agent types.
Results
The paper reports informative intention communication, legible behavior with an optimality trade-off, and preliminary saliency explanations that select reward regions and obstacles.
Takeaways & Limitations
Mirroring the agent’s intentional model provides a potential common architecture for targeted interpretability tasks and explanation generation.
Takeaways & Limitations
The mirror model builds on the agent’s implementation, so its obtained behavior depends on that implementation, including PDDL-based BDI and Deep Q-Network-based RL agents.
Abstract
from arXiv · showhide
In this paper we illustrate a novel architecture generating interpretable behavior and explanations. We refer to this architecture as the Mirror Agent Model because it defines the observer model, that is the target of explicit and implicit communications, as a mirror of the agent's. With the goal of providing a general understanding of this work, we firstly show prior relevant results addressing the informative communication of agents intentions and the production of legible behavior. In the second part of the paper we furnish the architecture with novel capabilities for explanations through off-the-shelf saliency methods, followed by preliminary qualitative results.
1 Introduction
Interpretable behavior and explanations aim to make agents’ intentions understandable to human observers, but existing approaches remain fragmented. The paper presents a Theory-of-Mind-based architecture intended to unify these tasks across agent types.
- Human-understandable behavior can improve trust, safety, interaction fluency, and acceptance by helping observers predict an agent’s actions.
- Interpretable behavior regularizes aspects of an agent’s behavior to match observer expectations and reduce ambiguity about its intentions.
- Explanations complement inexplicable behavior by communicating reasons or contrasts, such as why an agent chose one action rather than another.
- Existing methods span diverse algorithms and theoretical frameworks, motivating a general computational framework for interpretability and explainability.
- The proposed Theory-of-Mind architecture can address interpretable behavior or explanations and adapt to different kinds of agents.
2 Background
The Mirror Agent Model treats interpretable behavior and explanations as implicit and explicit communications aimed at aligning the observer’s inferred intention with the agent’s intention. It uses structurally mirrored Bayesian models to support informative communication and legibility across BDI and reinforcement-learning agents.
- Background: Interpretable behavior communicates intention implicitly through explicable actions, whereas explanations communicate explicitly to complement inexplicable behavior.Both pursue correct reproduction of the agent’s intention in the observer’s intentional model.
- The Mirror Agent Model: The agent models the observer’s understanding as a second-order theory of mind and balances optimal behavior, interpretability, and explanations while minimizing model divergence.The observer model represents how the observer infers the agent’s intention.
- The Mirror Agent Model: The Mirror Agent Model represents the agent and observer with Bayesian networks sharing the same structure but potentially different probability distributions.Structural mirroring can capture methods focused on parts of the agent model and avoid costly model alignment.
- The Mirror Agent Model: Its divergence measure H quantifies differences between intention distributions and can instantiate legibility, predictability, or explanation methods depending on which network components are compared.Lower divergence indicates that the agent’s behavior matches the observer’s mental simulations more closely.
- Informative Intention Verbalization: For informative communication, the BDI agent selects plan verbalizations by information gain, and a user study found the proposed strategy made participants guess the robot’s intention faster than incremental and decremental baselines.The study also found that actions directly affecting goal predicates were, on average, the most informative to communicate.
- Legible Behavior: For reinforcement-learning agents, mirror-based regularization improves legibility by steering actions away from regions associated with other policies, but increasing regularization decreases performance.The formulation extends legibility from goal-driven agents to arbitrary policies by using reward regions rather than goal locations.
3 Generating Explanations
The paper extends the Mirror Agent architecture to generate explanations through saliency masks that communicate the agent’s policy to an observer. Preliminary tunnel-environment results show that the agent selects masks highlighting rewards and obstacles while avoiding regions associated with other policies.
- 3.1 Background: The paper distinguishes interaction-oriented explanations from expert-facing feature visualizations that may not support common-sense reasoning for uninformed users.Saliency maps are presented as an understandable starting point, but the paper notes current shortcomings in common-sense reasoning.
- 3.1 Background: The explanation model adds dedicated Bayesian-network nodes so an RL agent can communicate decisions through saliency masks.The approach uses off-the-shelf perturbation-based saliency methods and represents explanations as masks over the input state.
- 3.2 Explanation model: The model selects a mask that minimizes the distance between the agent’s enacted policy and the observer’s inferred policy.The selected mask gives the observer the highest probability of the agent’s policy while avoiding communication of alternative policies.
- 3.2 Explanation model: A single best mask explains the current decision, while the full explanation distribution can produce fuzzy saliency maps by probability-weighted averaging.This allows explanations to represent uncertainty across multiple plausible masks.
- 3.3 Preliminary results: In the tunnel environment, the agent mostly selects reward regions and obstacles as explanations.It avoids other-colored regions because communicating them would lead the observer to infer policies rewarded on those cells.
4 Conclusions
The Mirror Agent Model unifies interpretable behavior and explanations by mirroring the agent’s intentional model in the observer. Its applicability spans agent types representable as Bayesian networks, but its behavior remains dependent on the underlying implementation.
- 4 Conclusions: The Mirror Agent Model uses structurally matched agent and observer models to address multiple interpretable-behavior tasks with targeted subnetworks.The same divergence measure can identify which parts of the agent’s intentional model are understood by the observer.
- 4 Conclusions: The model can be applied to varied agents represented as Bayesian networks, but its resulting behavior depends on the agent implementation.The paper demonstrates this dependence using BDI and RL agents implemented with PDDL and Deep Q-Networks.
- 4 Conclusions: The architecture is intended to unify interpretable behavior and explanation generation within a single computational framework.The authors identify future work as formulating relevant algorithms from both areas in the mirror setting.