Source-linked AI summary
Language Models as Agent Models
Jacob Andreas
TL;DR
Language models are trained without direct access to authors’ internal states, raising questions about whether they can model goal-directed language use. This paper surveys evidence and argues that models infer approximate beliefs, desires, and intentions that influence generation, while emphasizing their narrow scope.
Problem
Because language models lack direct access to authors’ internal states and still produce coherence errors, whether they model goal-directed language use remains uncertain.
Method
The paper develops an agent-simulation framework and surveys empirical examples of language models encoding communicative intentions, beliefs, and desires.
Results
Language models sometimes infer approximate, partial representations of agents’ beliefs, desires, and intentions that predict relations between observations, states, and actions or utterances.
Takeaways & Limitations
Language models can serve as models of intentional communication in a narrow sense and provide a framework for understanding failures and guiding improvement.
Takeaways & Limitations
The evidence concerns narrow task-specific slices, relies on English experiments, and may not generalize to complex beliefs or lower-resource languages.
Abstract
from arXiv · showhide
Language models (LMs) are trained on collections of documents, written by individual human agents to achieve specific goals in an outside world. During training, LMs have access only to text of these documents, with no direct evidence of the internal states of the agents that produced them -- a fact often used to argue that LMs are incapable of modeling goal-directed aspects of human language production and comprehension. Can LMs trained on text learn anything at all about the relationship between language and use? I argue that LMs are models of intentional communication in a specific, narrow sense. When performing next word prediction given a textual context, an LM can infer and represent properties of an agent likely to have produced that context. These representations can in turn influence subsequent LM generation in the same way that agents' communicative intentions influence their language. I survey findings from the recent literature showing that -- even in today's non-robust and error-prone models -- LMs infer and use representations of fine-grained communicative intentions and more abstract beliefs and goals. Despite the limited nature of their training data, they can thus serve as building blocks for systems that communicate and act intentionally.
1 Introduction
The paper argues that language models can narrowly function as agent models: during next-word prediction, they sometimes infer representations of agents’ beliefs, desires, and intentions that causally shape generated text. This framework explains current failures and identifies directions for improvement without claiming that models are generally human-like.
- Motivation: Current language models sometimes produce grammatically or semantically acceptable text describing impossible situations, contradictions, or invalid inferences, reflecting failures to model communicative intent.They are trained on text alone and lack the social and perceptual context human learners use to connect utterances with speakers’ mental states.
- Evidence: Despite these failures, models can correctly generalize physical knowledge and infer that different individuals hold different beliefs, even in novel situations.The bowling-ball and leaf examples show belief-sensitive predictions about a vacuum-chamber demonstration.
- Evidence: Models can also relate beliefs and motivations to plans, representing multiple individuals’ differing goals alongside the environment in which they act.The lunchbox completion describes Syd’s theft, concealment strategy, motivation, and beliefs about Lou’s awareness.
- Core claims: The paper’s central claims are that next-word prediction can infer approximate, partial representations of agents’ beliefs, desires, and intentions, and that these representations causally influence prediction.This treats language-model prediction as approximate communicative-intention inference, or agent simulation.
- Scope and aims: These claims concern narrow, context-dependent simulation of goal-directed behavior, not general human-likeness; the framework aims to clarify failure modes and guide model improvement.The paper examines an idealized account of agent beliefs, desires, and intentions and cases where existing models approach it.
2 Case Study: An Incoherent Encyclopedia
A language model trained on a globally incoherent encyclopedia learned to represent individual authors and generate text consistent with their beliefs. However, its samples remained mutually inconsistent, so the model as a whole could not be treated as a coherent intentional agent.
- 2 Case Study: An Incoherent Encyclopedia: Although individual samples reflected coherent authors, pairs of samples could contradict one another, preventing the LM as a whole from expressing a coherent belief set or communicative intent.The experiment therefore demonstrates local author-like coherence without global coherence across samples.
- 2 Case Study: An Incoherent Encyclopedia: Because each document came from one coherent author, a reliable LM had to infer the prefix’s likely author to select propositions consistent with that author’s behavior.The encyclopedia mixed equal parts A-type authors believing A, B-type authors believing B ≠ A, and O-type authors believing all propositions in A ∪ B, including contradictions.
- 2 Case Study: An Incoherent Encyclopedia: The RNN recovered author identity with 98% accuracy from the representation of a document’s fifth token, while its samples were consistent with A-, B-, or O-type authors in 31%, 33%, and 36% of cases, respectively.The model was trained on 10,000 length-10 documents without author-identity labels.
- 2 Case Study: An Incoherent Encyclopedia: Post-hoc control of the initial hidden representation made the RNN generate A-type propositions 89% of the time, despite no training objective to control author type.The remaining generated samples were O-type.
- 2 Case Study: An Incoherent Encyclopedia: The case study is deliberately simplified: its corpus lists propositions rather than modeling the complex beliefs and goal-directed communication of real human authors.The authors present it as a sketch of how an LM can model individual text-generating agents.
3 Discussion: An Incoherent Internet
The section models language generation with a Belief–Desire–Intention framework in which agents form communicative intentions from beliefs and desires, then realize them as utterances. It argues that LMs trained on marginal text distributions can nevertheless maintain constraints on hypothetical authors’ beliefs, desires, and intentions, despite simplified assumptions and imperfect coherence.
- BDI framework: The Belief–Desire–Intention model represents agents as holding beliefs about world states and desires over possible outcomes, which shape communicative intentions and resulting utterances.The model describes agents acting under partial uncertainty to achieve desired environmental states through communication.
- BDI framework: A probabilistic corpus-generation process samples agents’ beliefs and desires, forms communicative intentions consistent with them, and realizes those intentions as utterances.This provides a general sequence for formalizing how intentional agents could generate text corpora.
- LM sampling: An LM observes samples from the marginal distribution p(U), so predicting continuations can require modeling other beliefs likely held by an author, even for false or infrequent beliefs.The rutabega example illustrates that continuation prediction involves more than grammaticality and basic world knowledge.
- Limitations and evidence: The framework is a simplification because corpora also contain emotional and non-intentional text, automated text, and other latent variables beyond belief.Real models also exhibit within-document coherence errors, though evidence suggests they encode at least aspects of intentions, beliefs, and desires that control generation.
- LM sampling: Although LM sampling introduces speaker information incrementally and stochastically rather than from fixed initial beliefs, effective models must maintain the beliefs, desires, and intentions constrained by coherent contexts.This latent-state acquisition differs from the corpus-generating process, while producing the same constraint-maintenance effect for subsequent generation.
4 Modeling Communicative Intentions: The Sentiment Neuron
Radford et al. (2017) found that an LSTM language model trained on product reviews represented authors’ sentiment intentions and used that representation to control generated text. The experiment supports intentional communication at a relatively low level, while showing that syntactic errors may limit finer-grained intention modeling.
- Motivation: Product-review datasets contain contradictory claims from heterogeneous authors, so intentions and beliefs should be modeled for individual agents in context rather than for an LM as a whole.This contrasts with treating the entire language model as encoding a single belief about a proposition.
- Evidence for (C1): 92% accuracy: a single LSTM neuron predicted binarized review ratings despite training without explicit star ratings, encoding authors’ product attitudes.The model was trained on 82 million Amazon reviews and evaluated on IMDB reviews.
- Evidence for (C2): Fixing the neuron to maximal or minimal values controlled generated-review sentiment, sometimes preserving topic coherence about the reviewed product.This intervention shows that the inferred intention representation was causally linked to generation.
- Model failures and counter-evidence: The sentiment and topic information persisted in a model that produced significant syntactic errors, indicating that low-level intention representations can coexist with poor surface generation.More recent language models make fewer such syntactic errors, but stronger syntax is likely needed for finer-grained communicative intentions.
- Contribution: These experiments support mapping between text and relatively low-level communicative intentions, leaving open whether LMs select intentions from general beliefs and goals.The next challenge is modeling intention selection conditioned on broader beliefs and serving broader goals.
5 Modeling Beliefs: Transformer Entity Representations
Li et al. (2021) found that transformer language models encode entity properties, relations, and uncertainty from text, including consequences not explicitly stated. Editing these representations changes generated actions, although entity tracking remains unreliable for negation, coreference, modality, and implication.
- Method: Li et al. (2021) tested pre-trained BART and T5 on documents interleaving agents’ observations with their actions, where accurate modeling required tracking changing entity states.They trained linear models to predict entity states from LM representations of entity mentions.
- Evidence for (C1): Up to 97% accuracy: across multiple datasets, LMs linearly encoded entities’ properties and relations, including unstated consequences, while distinguishing unspecified facts from false ones.This evidence came from linear probes of LM representations of entity mentions in text adventures and laboratory protocols.
- Evidence for (C2): Editing beaker representations changed whether generated actions treated them as empty or full, showing that entity states mediated subsequent text generation.Models never generated instructions to pour out a beaker edited to be empty.
- Model failures and counter-evidence: LMs do not always reliably infer entities’ states or existence, particularly under negation and coreference with distractors [Pandia and Ettinger (2021); Schuster and Linzen (2022)].Their representations may also lack machinery for complex beliefs involving modality and implication.
- Model failures and counter-evidence: Although a purely syntactic account remains possible, Meng et al. (2022) found that localized entity representations also encode background knowledge alongside contextual states.
6 Modeling Desires: Prompt Engineering for Truthfulness
Truth-oriented prompts increased LM truthfulness on TruthfulQA, while harmful prompts sharply reduced accuracy, suggesting that models can condition generation on an author’s communicative goals. However, substantial factual and goal-related failures remain, and the mechanism behind effective prompt engineering is still unclear.
- Modeling Desires: The results distinguish goal-sensitive generation from merely sampling mixed true and false answers, because textual descriptions of an author’s goals changed responses without supplying evaluation facts.The helpful and harmful prompts differed in their implied communicative goals while using unrelated questions.
- Modeling Desires: Truthful prompts raised truthful answers from roughly 38% to 58%, whereas harmful prompts reduced accuracy to less than 20%, supporting goal-conditioned control of LM truthfulness.The prompts used unrelated example questions rather than factual information about the evaluation items.
- Model failures and counter-evidence: Even with truthful prompting, 42% of questions were answered incorrectly, revealing gaps in factual knowledge and in relating facts to goals.Truthfulness improved at the example level but did not eliminate incorrect answers.
- Prompt engineering: The findings suggest that effective prompts create an accessible context representation from which an LM can predict an author’s future actions, despite unresolved prompt-design principles.The prompts could influence behavior even when they included extraneous biographical detail, such as Professor Smith’s description.
7 Why do models fail?
Models’ factuality and coherence errors can be understood as failures to infer or condition on latent agent-state representations. The paper identifies limitations in training data, context windows, and architectures that constrain these abilities, alongside corresponding remedies.
- Factuality and coherence errors arise when models fail to infer or condition on representations of agent state.This framing helps explain how such errors arise and how they might be addressed.
- Limitations of training datasets: Unsupervised learning of structured latent variables such as agent intent can converge to incorrect representations.Semi-supervised annotations about beliefs, goals, or social and perceptual context may improve language modeling; author-conditioned models provide supporting evidence (Keskar et al., 2019; Zellers et al., 2019).
- Limitations of context windows: Human agents’ complete beliefs, desires, and intentions cannot fit within today’s context windows, which restrict models to simpler state aspects.One proposed remedy is to factorize short-term and long-term context components relevant for prediction, as in preliminary work by Henaff et al. (2016) and Dai et al. (2019).
- Limitations of LM architectures: Current fixed-depth RNN and transformer architectures differ from planning algorithms that use branching search and potentially unbounded computation.This mismatch limits their ability to compute communicative intentions that accomplish modeled agents’ goals given their beliefs.
- Limitations of LM architectures: Scratchpads record intermediate computations in generated text, but consuming context this way leaves less room to represent and reason about the modeled agent (Camburu et al., 2018).Overcoming this tradeoff may require explicit algorithmic reasoning.
8 Building Agents
Building useful agents requires models of specific agents, not merely populations, and current language models still lack many necessary components. Text-only pre-training may nevertheless scaffold efficient grounded training, while NLP must improve architectures and mechanistically understand their goal-oriented reasoning.
- Current NLP systems often require models of specific agents, but surveyed failures show that language models lack many components needed for agent-like behavior.
- Text-only pre-training could provide approximate models linking beliefs, desires, intentions, and utterances that scaffold efficient interactive grounded training.Understanding when and how communicative intentions are encoded could make producing goal-directed language require only limited additional training.
- A hybrid training paradigm may be the only route to high-quality language modeling compatible with human timescales.Replicating intensive real-time interaction in silico would be enormously time-consuming, non-reproducible, and incompatible with the scientific workflow driving machine-learning progress.
- NLP must develop architectures that overcome current limitations and understand mechanistically how they infer and reason about goal-oriented behavior.Even vague representations of human beliefs, desires, and intentions could provide a first step toward agents that reason about others.
9 Limitations
Current language models only approximate intentional behavior modeling, with evidence limited to task-specific slices rather than general-purpose representations of beliefs, desires, or intentions. Their agent-state inferences may also weaken or disappear in lower-resource languages, limiting general-purpose use and equitable deployment.
- 9 Limitations: Current models provide only narrow, task-specific slices of belief, desire, and intention representations, so their findings do not establish general-purpose intentional-behavior modeling.The extent to which these findings scale to more complex agent behavior remains uncertain.
- 9 Limitations: Agent-state inferences are likely reduced or absent in lower-resource languages, challenging general-purpose deployment and equitable access to the technology.
10 Ethical Considerations
The paper’s core LM inferences can fail, causing unpredictable and undesirable behavior, and can also be deliberately exploited to simulate users with malign intentions.
- 10 Ethical Considerations: LM inference failures can produce unpredictable and undesirable behavior, including untruthful answers on TruthfulQA.These failures qualify the paper’s claims about LM inferences and their behavioral consequences.
- 10 Ethical Considerations: Successful LM inferences can also be prompted to simulate users with malign intentions, enabling deliberate harm.The passage frames goal conditioning as ethically consequential because inferred goals can be used maliciously.