Source-linked AI summary
Artificial Empathy: Towards a Framework for Unsupervised Agency Detection and Policy Reconstruction
Peter Kuhn, Chris Pang, Sonakshi Chauhan
TL;DR
The paper addresses how an AI system can detect and reconstruct other agents’ policies from observation alone, a less-constrained problem than inverse reinforcement learning. It proposes artificial empathy, which uses an independently trained RL agent as a prior over agentic dynamics. In a simple gridworld with a random-walk decoy, the system imitates an expert, demonstrating feasibility in that setting.
Problem
Agency detection and policy reconstruction from observation alone remain underexplored, although such capabilities matter for cooperative behaviour and alignment.
Method
Artificial empathy uses an RL agent trained on an independent task as a prior to jointly detect agency and reconstruct another agent’s policy from environmental observations.
Results
The AE system learned to imitate an expert policy in a gridworld containing a random-walk decoy, suggesting that self-referential priors support unsupervised agency detection.
Takeaways & Limitations
The results suggest that self-referential priors can support feasibility-level unsupervised agency detection and policy reconstruction.
Takeaways & Limitations
Current evidence is limited to a small, fully observable gridworld with one decoy agent and a hand-selected structurally similar prior; multiple agents, partial observability, and prior-target divergence remain untested.
Abstract
from arXiv · showhide
We study how an AI system can identify and model other agents in its environment from observation alone, which is a capability necessary for cooperative behaviour in the real world. This problem is less constrained than inverse reinforcement learning and remains largely unexplored. We propose a framework that uses a reinforcement learning agent, trained on an independent task as a prior about agentic dynamics, to perform agency detection and policy reconstruction.
1 Introduction
The paper introduces artificial empathy as a framework for jointly detecting agency and reconstructing policies from behaviour alone. It uses an AI system’s own agentic dynamics as a prior for identifying and modelling other agents.
- Problem and contribution: Artificial empathy jointly addresses agency detection and policy reconstruction from observed behaviour.Agency detection distinguishes agentic behaviour from non-agentic activity, while policy reconstruction infers an agent’s policy from behaviour alone.
- Problem and contribution: The framework targets unsupervised identification of agents in environments where random physical events might otherwise be mistaken for agency.The proposed approach uses prior knowledge about agency to decide whether environmental dynamics can be modelled as agentic.
- Core idea: An AI system can use its own dynamics as a prior about what agency looks like.Agency detection and policy reconstruction are jointly approached by testing whether environmental changes are predictable from variations in the system’s own dynamics.
2 Prior Work
Prior work provides related ideas through simulation-based social understanding, self-other representational overlap, and inverse reinforcement learning. However, these approaches leave artificial empathy’s joint, observation-based task insufficiently addressed.
- Simulation-based approaches: Simulation theory explains social understanding as predicting others through one’s own cognition and decision-making.The approach is associated with putting oneself in another’s place to infer and explain behaviour.
- Alignment approaches: Self-other representational overlap has been used in LLM alignment, but depends on accurately identifying representations in activation space.The cited approach is described as primarily intuition-driven despite producing behaviour that can appear aligned.
- Inverse reinforcement learning: IRL infers reward functions from labelled observation-action pairs, but its reliance on preprocessed trajectories limits applicability when agent behaviour is not clearly identifiable.The paper therefore treats IRL and its variants as incomplete alignment strategies by themselves.
3 Method
The method uses a trained reinforcement-learning agent as an agency prior and models another agent’s policy by predicting environmental dynamics from observations. Its losses compare encoded future states with prior-conditioned predictions while encouraging committed action predictions.
- Overview and motivation: The AE task infers an other agent’s policy from an observation stream, while the broader framework combines self and other policies into a joint policy.The current task is purely epistemic, whereas the proposed joint policy is πΩ = f(πS, πO).
- Base agent architecture: The base architecture is a deep Q-learner augmented with a trainable world model.The model represents world states probabilistically and supports policy representation through Q-functions.
- AE architecture: The AE system uses a trained base agent from an independent task as a structurally similar prior for reconstructing an expert’s policy from reordered observations.The prior agent supplies the agency-related dynamics used while learning from the expert’s environment.
- AE architecture: AE marginalises over possible actions to obtain a probability distribution over next world states under the assumption of agentic behaviour.The resulting prior-conditioned distribution is compared with encoded world states using Kullback-Leibler divergence.
- AE architecture: The full AE loss adds an entropy penalty so predictions commit to specific actions rather than only producing smooth action distributions.The paper describes the loss as combining the KL term with additional weighted terms, including the entropy penalty.
4 A Simple Experiment
The experiment tests whether AE can imitate an expert in a fully observable gridworld while its observation stream contains a random-walk decoy. The system succeeds, supporting feasibility of unsupervised agency detection in this setting.
- Experimental setup: The evaluation trains agents on two tasks, uses one as an expert and the other as a prior, and measures imitation by cumulative reward on task A.The experiment uses a 15×15 gridworld with food and edge tasks, with fully observable trajectories.
- Results: A random-walk decoy is added to the observation stream to test whether AE can distinguish agentic dynamics from non-agentic activity.Successful imitation requires treating the decoy’s random dynamics as ill-suited to prediction through the prior.
- Results: AE learns to imitate the edge-game expert despite the decoy, suggesting that self-referential priors can support unsupervised agency detection.The authors characterize these as early, promising results rather than evidence of robustness or scale.
5 Limitations
The current evaluation demonstrates feasibility in a small, fully observable gridworld but does not establish robustness or scale. Several realistic settings and loss-weight sensitivities remain untested.
- The experiments use a small, fully observable gridworld with one confounding decoy agent, limiting conclusions to feasibility rather than robustness or scale.
- The AE system relies on a hand-selected, structurally similar prior agent, while degradation under architectural or task-structural divergence remains untested.
- Multiple genuine agents, partial observability, and non-stationary policies were not evaluated, despite being more representative of real-world alignment settings.
- Sensitivity to the weighted loss terms LKL, RH, Rp, Rq, and RQ has not been systematically characterised.
6 Conclusion and Future Work
The paper introduces artificial empathy as a framework that jointly detects agency and reconstructs policies using an RL agent’s own dynamics as a prior. In a gridworld imitation task, the system learned to imitate an expert despite a random-walk decoy, while broader scaling and joint-policy construction remain future work.
- 6 Conclusion and Future Work: Artificial empathy jointly performs agency detection and policy reconstruction by using an RL agent’s own dynamics as a prior over agentic behaviour.
- 6 Conclusion and Future Work: In the imitation game, AE performance after 35 episodes was far better than a random walk and approached the base agent’s performance.
- 6 Conclusion and Future Work: An AE system initialised from an unrelated prior task learned to imitate an expert policy in a gridworld with a random-walk decoy, suggesting self-referential priors can support unsupervised agency detection.
- 6 Conclusion and Future Work: Future work includes partial observability, multiple simultaneous agents, less structurally similar priors and targets, multi-game agency priors, and an explicit joint policy πΩ.
A Artificial Empathy System Diagram
The diagram contrasts action selection in the base agent and AE system: both predict future states and assign Q-values, but the base uses argmax while AE marginalises states with softmax.
- For each possible action, both architectures compute a predicted future state and assign that state a separate Q-value.
- The base architecture selects the maximum-Q state via argmax, whereas the AE system marginalises states through a softmax.