PaperlayerView exact revision on arXiv

Source-linked AI summary

Discovering Agents

Zachary Kenton, Ramana Kumar, Sebastian Farquhar, Jonathan Richens, Matt MacDermott, Tom Everitt

arXiv:2208.08345v2cs.AIcs.LG

TL;DR

Identifying agents in causal models is difficult, although accurate models matter for safety analysis. The paper defines agents through counterfactual policy adaptation and develops causal discovery methods for inferring game graphs from experiments. It demonstrates the approach’s utility by improving safety analyses of several AI systems.

  • Problem

    Identifying agents in causal models is non-trivial, while such models are used to analyze machine-learning safety.

    Source: §1 · p. 1

  • Method

    The paper defines agents by counterfactual policy adaptation and gives algorithms for discovering game graphs from experiments and identifying decision and utility variables.

    Source: §1 · p. 2, §3 · p. 5, §3.2 · p. 7

  • Results

    The approach demonstrated utility by improving safety analyses of several AI systems.

    Source: §6 · p. 25

  • Takeaways & Limitations

    Causal discovery experiments can ground game-graph models and support more reliable safety and fairness analyses built on such models.

    Source: §6 · p. 25

    The choice of variable coarse-graining may affect whether agents are discovered.

    Source: §5.2 · p. 23

Abstract

from arXiv · show

Causal models of agents have been used to analyse the safety aspects of machine learning systems. But identifying agents is non-trivial -- often the causal model is just assumed by the modeler without much justification -- and modelling failures can lead to mistakes in the safety analysis. This paper proposes the first formal causal definition of agents -- roughly that agents are systems that would adapt their policy if their actions influenced the world in a different way. From this we derive the first causal discovery algorithm for discovering agents from empirical data, and give algorithms for translating between causal models and game-theoretic influence diagrams. We demonstrate our approach by resolving some previous confusions caused by incorrect causal modelling of agents.

1. Introduction

Agency is difficult to identify in systems whose behavior varies across contexts or emerges from multiple components, yet safety analysis depends on accurate causal models. The paper defines agency through counterfactual policy adaptation and proposes causal methods for discovering agents from experiments.

1. Introduction

  • Causal discovery matters because goal-directed AI can pose safety risks, and experimental grounding can add assurance that an agent model is not mistaken.The paper also points to potential use in inferring game graphs in settings such as software simulations.Source: §1 · pp. 1–2

1.1. Example

  • In the mouse example, changing the known probability of finding cheese changes the decision rule, illustrating how interventions can reveal agency and support inference of both graph representations.The example contrasts a game graph with a mechanised causal graph; the latter includes mechanisms for the mouse’s decision rule and environmental parameters.Source: §1.1 · pp. 2–3

1.2. Other Characterisations of Agents

  • The proposed definition treats a system as an agent if it would adapt its policy when its actions influenced the world differently, without prespecifying its decision-maker or objective.The authors present it as consistent with decision theory and related to intentional-stance and cybernetics characterisations.Source: §1.2 · p. 3
  • Unlike compression-based characterisation, counterfactual adaptation can cover simple policies and exclude accidentally optimal systems; it also tests for systematic, rather than incidental, goal-directed influence.The paper notes that compression may fail for a single binary decision and may classify an accidentally optimal rock as an agent.Source: §1.2 · p. 3

1.3. What do we consider an agent

  • Whether a thermostat or an RL system counts as an agent depends on including its creation or training process: a frozen thermostat or learned policy generally will not adapt after the relevant change.Designers might redesign the thermostat when informed of the changed heater, and RL systems may adapt through retraining in a different environment.Source: §1.3 · p. 4

1.4. Outline

  • The paper proceeds from technical background to agent-discovery algorithms, example applications, and discussion.The outline assigns these topics to Sections 2 through 5, respectively.Source: §1.4 · p. 4

2. Background

The background introduces structural causal models, causal games, and the timing distinction between interventions made before and after policies are selected.

2. Background

  • An SCM represents randomness with exogenous variables and relates endogenous variables through structural equations; its graph records dependencies in those equations.Interventions replace structural equations, while soft interventions replace them with potentially non-constant functions.Source: §2 · pp. 4–5
  • A causal game partitions variables into chance, decision, and utility variables, with agents choosing distributions over decisions based on available information.A game graph distinguishes these node types and represents decision information with edges into decision nodes.Source: §2 · p. 5
  • A policy combines decision rules that map available information to actions, and agents optimize expected utility; multi-agent settings require a solution concept such as Nash equilibrium.Decision rules can be combined with the game to form an SCM for computing expected utility.Source: §2 · p. 5
  • Pre-policy interventions occur before policies are chosen, so agents may adapt; post-policy interventions occur afterward, leaving policies fixed even if they are no longer rational.The distinction corresponds to whether agents are aware of the intervention when selecting policies.Source: §2 · p. 5

3. Algorithms for Discovering Agents

The section presents algorithms for constructing agent and game graphs from causal experiments and for converting between game graphs and mechanised causal graphs. It establishes correctness and gives criteria for identifying mechanism edges, with the conversions shown to be inverses under stated conditions.

3.1. Mechanised Structural Causal Model

  • Mechanised SCMs distinguish object-level variables from mechanism variables that parameterize their dependencies; object-level interventions leave mechanisms unchanged.The object-level subgraph is acyclic, but the mechanism-only subgraph may be cyclic; connecting mechanisms violates the usual independent causal mechanism assumption.Source: §3.1 · pp. 6–7

3.2. Edge-labelled mechanised causal graphs

  • Terminal mechanism edges distinguish variables valued for their own sake from variables whose mechanisms adapt for downstream reasons.Their labels test whether mechanism responses persist when structural mechanism interventions cut a variable’s effects on its children.Source: §3.2 · pp. 7–8

Leave-one-out causal discovery

  • Leave-one-out discovery recovers the causal graph of a potentially cyclic SCM, and Algorithm 1 recovers the edge-labelled mechanised graph when structural mechanism interventions are available for all nodes.Leave-one-out varies interventions on one node while intervening on the others; more efficient discovery methods may be substituted, with tradeoffs between speed and assumptions.Source: §3.3 · p. 8, Leave-one-out causal discovery · p. 10

3.3. Discovering Edge-labelled, Mechanised Causal Graphs

  • Algorithm 1 applies leave-one-out discovery to object-level and mechanism variables, then labels edges using structural mechanism interventions on object-level children.The procedure infers edge-labelled mechanised causal graphs from interventional data.Source: §3.3 · p. 8

3.4. Discovering game graphs

  • Under Assumptions 1–5, Algorithms 1 and 2 recover the correct game graph from a mechanised causal game.Algorithm 2 identifies decisions through incoming terminal mechanism edges and utilities through outgoing terminal mechanism edges.Source: §3.4 · pp. 10, 12

3.5. Mechanism Identification Procedure

  • A mechanism edge (eY, eD) is present exactly when Y is s-reachable from decision D; a terminal edge also requires a path from D to utility Y that passes through no other utility of the agent.These are necessary and sufficient conditions for the respective edges in Algorithm 1’s mechanised causal graph.Source: §3.5 · pp. 13–14
  • Algorithm 3 converts a game graph to a mechanised causal graph by adding function edges, mechanism edges for s-reachability, and terminal edges for qualifying decision-to-utility paths.The section presents the two graph types as physical and decision-theoretic representations of the same underlying system.Source: §3.5 · pp. 13–14

3.4. Discovering game graphs

  • Algorithm 2 maps decision and utility nodes to their types, with remaining nodes classified as chance; applying it after Algorithm 3 reproduces the game graph.The round-trip preserves node types, edges, and colouring.Source: §3.4 · p. 12, §3.5 · p. 15

3.5. Mechanism Identification Procedure

  • For C produced by a mechanised causal game satisfying Assumptions 1 to 5, and with every node having an incoming mechanism edge also having an incoming terminal edge, a3(a2(C)) = C.In general, mechanisms with non-terminal incoming mechanism edges are not codified in the game graph when using a2.Source: §3.5 · p. 15

4. Examples

These worked examples show how interventional mechanised graphs recover agent decisions and objectives, and expose incentive conclusions that ordinary game graphs can miss.

4.1. Simple example

  • Changing the mechanism for X from p > 0.5 to p < 0.5 changes optimal D from 1 to 0, letting the algorithms recover the game graph.Source: §4.1 · p. 16

4.2. Optimising a model of a human

  • With a user model trained on past data, changing how recommendations affect opinions can alter predicted clicks, making policy depend on human updating.The authors note that this interpretation depends on how the user model was obtained.Source: §4.2 · p. 17
  • Under one plausible interpretation, the discovered path implies the recommender adapts to changes in how recommendations influence users, challenging the claim that it lacks that incentive.Source: §4.2 · p. 17
  • Because interpretations of one graph can yield different conclusions, graphical incentive analysis is trustworthy only when all non-decision mechanisms lack incoming arrows.The authors leave generalization to dependent mechanisms for future work.Source: §4.2 · pp. 17–18

4.3. Actor-Critic

  • Algorithms 1 and 2 recover the actor and critic as separate agents, preventing the incorrect single-agent inference that the system does not try to influence state or reward.Source: §4.3 · p. 19
  • At a coarser graph, state and reward have an instrumental-control incentive; the detailed two-agent case instead calls for a multi-agent incentive concept capturing joint influence.Source: §4.3 · p. 19

4.4. Modified Action Markov Decision Process

The examples show the algorithms recovering the intended game graphs for an overridden-action MDP and a system with no agents, while violations of Assumption 1 can produce incorrect graphs. In the shared-utility and non-descendant-utility cases, Algorithm 2 misclassifies agents or variables.

4.4. Modified Action Markov Decision Process

  • In the modified-action MDP, Algorithm 2 recovers both decisions as belonging to one agent, whose decision rules are influenced by the mechanism for potentially overridden variables.The game graph represents the override through a chance variable; the decision rules are not shown as observing the modification itself.Source: §4.4 · pp. 19–20, Figure 5 · p. 20

4.5. Zero agents

  • With no decisions or utilities, Algorithm 2 identifies no agents and recovers the standard causal Bayesian network.The example has X causing Y, which causes Z; no mechanism has incoming edges to indicate a decision.Source: §4.5 · p. 20, Figure 6 · p. 20

4.6.1. Multiple agents with a shared utility

  • For an Assistance Game with a shared utility, Algorithm 2 incorrectly assigns all human and robot decisions to one agent, a failure under Assumption 1.The authors suggest sufficient recall as one possible basis for modifying the colouring logic to distinguish agents sharing a utility.Source: §4.6.1 · p. 21, Figure 7 · p. 21

4.6.2. Non-descendent utility

  • When a decision’s utility is not its descendant, Algorithm 2 misclassifies that decision and utility as chance variables, although the decision rule can still influence the utility indirectly.The example’s path runs through the other agent’s decision; Assumptions 2 and 3 rule out strategizing through this path, which the authors identify as a target for future work.Source: §4.6.2 · pp. 21–22, Figure 8 · p. 22

5. Discussion

The discussion emphasizes that discovered game-graph roles depend on which variables are modeled, so careful variable selection and causal assumptions matter. It also relates the approach to causal discovery and identifies limits on applying the ICM principle to systems with agents.

5.1. Relativism of variable types

  • Decision, utility, and chance labels depend on the modeled variables: omitted utilities can make their parents appear to be utilities, and omitted decisions can make their children appear to be decisions.The model’s variable set therefore defines a frame in which node types are interpreted.Source: §5.1 · p. 22

5.2. Modelling advice

  • Choose variables that capture relevant environmental aspects, can be measured and intervened on, have exhaustive mutually exclusive domains, and are logically independent.The mechanisms for the variables must also be clear enough to measure and intervene on.Source: §5.2 · pp. 22–23
  • Coarse-graining can affect whether agents are discovered, and distinguishing object-level variables from mechanisms helps avoid conflating policies with variables or utilities with instrumental variables.The authors leave marginalisation in game graphs for future work, beyond an example in Appendix B.Source: §5.2 · p. 23

5.3. Relationship to Causality Literature

  • The mechanised SCM framework permits cycles between mechanism variables and assumes a unique solution, while distinguishing its focus on discovering agents from influence diagrams used for causal modelling.Dawid’s parameter DAG is similar but does not consider links between parameter variables.Source: §5.3 · p. 23
  • The approach uses causal discovery to build mechanised SCMs for agents and translate them into game graphs, unlike causal discovery work usually focused on agent-agnostic physical systems.The paper assumes experimental data for discovery and leaves discovery from observational data as future work.Source: §5.3 · p. 24
  • The ICM principle’s fixed-mechanism assumptions do not generally hold for decision rules or variables strategically relevant to agents, limiting ICM-based methods in systems whose data include agents.The authors suggest using Algorithm 1 to identify mechanism edges that violate ICM, potentially improving the performance of ICM-based methods when applied to suitable systems.Source: §5.3 · p. 24

6. Conclusion

The paper formalizes agents causally as systems that adapt their behaviour when the consequences of their actions change, and develops an experimental process to identify them. The authors report that the approach improves safety analyses of AI systems by grounding their causal models in discovery experiments.

6. Conclusion

  • The paper formalizes agents as systems that adapt their behaviour when changes alter how their actions influence the world.The proposed experimental process can assess whether a system is an agent under stated assumptions, without prespecifying its decision maker or objective.Source: §6 · pp. 24–25
  • The formal definition improves on Orseau et al. (2018) by better handling systems with few actions and accidentally optimal systems.The authors describe the proposal as consistent with earlier informal characterisations of agents.Source: §6 · p. 25
  • The approach improves safety analyses of several AI systems by grounding causal models in discovery experiments.The authors say this could also improve the reliability of safety and fairness analyses that build on such modelling.Source: §6 · p. 25

A.1. Notation

The notation section establishes conventions for variables, outcomes, and their relationships. It also specifies finite outcome domains and the use of subscripts.

A.1. Notation

  • Roman capitals denote variables and lowercase letters denote their outcomes; bold type denotes vectors of variables or outcomes.The notation uses V and v for variables and outcomes, and bold V and v for vectors.Source: §A.1 · p. 29
  • Pa_V, Ch_V, Anc_V, and Desc_V denote a variable’s parents, children, ancestors, and descendants, while Fa_V denotes its family.Fa_V = Pa_V ∪ {V}.Source: §A.1 · p. 29
  • Variable outcome domains are assumed finite, and subscripts denote submodels and potential responses to interventions.dom(V) denotes the possible outcomes of V; dom(V) for a vector is the product of its component domains.Source: §A.1 · p. 29

A.2. Structural Causal Model

The section defines structural causal models through variables, structural equations, and an exogenous-variable distribution, with a graph encoding dependencies. Interventions replace structural equations, and the treatment of nonrecursive models assumes a unique solution.

A.2. Structural Causal Model

  • An SCM comprises endogenous variables, corresponding exogenous variables, structural equations, and a distribution over the exogenous variables.The model’s causal graph represents dependencies among endogenous variables in those equations.Source: §A.2 · p. 29
  • A causal graph has an edge from V to W exactly when W’s structural equation depends on V.The white-node subgraph in Fig. 1c is an example of a causal graph.Source: §A.2 · p. 29
  • An intervention replaces the targeted variables’ structural equations with constants, and the paper assumes a unique solution under all interventions.The potential response is the solution for the queried variables in the intervened submodel; the uniqueness assumption also applies to nonrecursive SCMs.Source: §A.2 · pp. 29–30

A.3. Structural Causal Game

A structural causal game represents agents, variables, causal mechanisms, and the information available at decisions, allowing policies and strategic behaviour to be analysed within a causal model.

A.3. Structural Causal Game

  • A structural causal game includes agents, endogenous decision, chance, and utility variables, exogenous variables, decision information, non-decision structural equations, and independent exogenous noise.The endogenous variables are partitioned into decisions, chance variables, and utilities.Source: §A.3 · p. 30
  • Its game graph is a DAG whose edges represent causal dependence or information available to a decision.The graph distinguishes decision, chance, utility, and exogenous nodes.Source: §A.3 · pp. 30–31
  • A policy assigns decision rules to an agent’s decisions, and a policy profile replaces decision equations to induce a distribution over variables.Acyclicity of the game graph ensures this induced distribution is well-defined.Source: §A.3 · p. 31
  • Strategic behaviour is characterized using best responses and Nash equilibrium, with subgame perfect equilibrium requiring Nash equilibrium in every subgame.In a subgame, the rational response is independent of variables outside it.Source: §A.3 · p. 31

B. Getting a mechanised SCM by Marginalisation and Merging

The bandit example constructs a mechanised causal graph by accounting for how policies change over time, marginalising earlier variables, and merging repeated mechanisms.

B. Getting a mechanised SCM by Marginalisation and Merging

  • The second-iteration policy depends on the arm and outcome at the first iteration.This dependence is represented by including the policy mechanisms in the graph.Source: B. Getting a mechanised SCM by… · p. 32
  • Marginalising the first arm and reward turns the path from the first reward mechanism to the second arm mechanism into a direct edge.That path was previously mediated by the first reward.Source: B. Getting a mechanised SCM by… · p. 32
  • Merging repeated mechanisms makes observation of the merged arm mechanism correspond to the second iteration, while intervention sets both arm mechanisms.The construction also includes a terminal edge because the second reward has no children.Source: B. Getting a mechanised SCM by… · p. 32
  • Applying Algorithm 2 to the example yields the expected game graph.The example represents a bandit algorithm choosing an arm and receiving a reward over repeated iterations.Source: B. Getting a mechanised SCM by… · p. 32

C. Example of relativism of variable types

The example shows that Algorithm 2’s decision and utility classifications depend on which variables the model includes, without making the alternative classifications contradictory.

C. Example of relativism of variable types

D. Supplementary Figures

The supplementary figures show full mechanised causal graphs for the MAMDP and Assistance Game examples.

D. Supplementary Figures

  • Figure 10 presents the full mechanised causal graph for the MAMDP example in Figure 5.The caption identifies the example but does not describe the graph’s individual components.Source: Figure 10 · p. 34
  • Figure 11 presents the full mechanised causal graph for the Assistance Game example in Figure 7.The caption identifies the example but does not describe the graph’s individual components.Source: Figure 11 · p. 34
Loading 2208.08345v2…