PaperlayerView exact revision on arXiv

Source-linked AI summary

Implicit Representations of Meaning in Neural Language Models

Belinda Z. Li, Maxwell Nye, Jacob Andreas

arXiv:2106.00737v1cs.CL

TL;DR

The paper asks whether neural language models represent the world described by text or only model surface co-occurrence statistics. It probes BART and T5 representations for dynamic entity and situation states, finding that text-only training supports structured, linearly decodable, and editable meaning representations.

  • Problem

    The paper investigates whether neural language models build representations of meaning and the world described by text beyond lexical relations and syntactic structure.

  • Method

    The authors freeze language-model encoders and use probes to recover logical information states, including proposition truth values, from contextual representations.

  • Results

    In two domains, pretrained BART and T5 representations can be mapped with a linear probe to structured representations of entity and world states.

  • Takeaways & Limitations

    These representations are structured, interpretable, localized, and editable, with implications for improving factuality and coherence in language-model generation.

  • Takeaways & Limitations

    The experiments use the simplest parts of dynamic semantics and do not address interesting quantification or scope.

Abstract

from arXiv · show

Does the effectiveness of neural language models derive entirely from accurate modeling of surface word co-occurrence statistics, or do these models represent and reason about the world they describe? In BART and T5 transformer language models, we identify contextual word representations that function as models of entities and situations as they evolve throughout a discourse. These neural representations have functional similarities to linguistic models of dynamic semantics: they support a linear readout of each entity's current properties and relations, and can be manipulated with predictable effects on language generation. Our results indicate that prediction in pretrained neural language models is supported, at least in part, by dynamic representations of meaning and implicit simulation of entity state, and that this behavior can be learned with only text as training data. Code and data are available at https://github.com/belindal/state-probes .

1 Introduction

The paper argues that neural language models trained only on text can learn approximate, dynamic representations of entities and situations, not merely surface or syntactic patterns. Probing BART and T5 reveals representations that support semantic decoding and influence generation.

  • Motivation: NLMs encode logical descriptions of entities’ changing properties and relations in simple semantic domains.These representations model situations and entity states as discourse unfolds.
  • Motivation: Text alone can induce implicit state representations, including semantically necessary consequences not explicitly mentioned in a sentence.The paper illustrates this with a chest becoming empty after its key is picked up.
  • Approach: The paper formalizes probing for information-state representations and applies the procedure to BART and T5 trained on Alchemy and TextWorld text.The approach tests whether model representations encode situations like those depicted in Figure 1.
  • Results: Implicit meaning representations can be linearly decoded from NLM encodings of entity mentions.This is one of the paper’s principal empirical findings.
  • Results: The representations are primarily attributable to open-domain pretraining rather than in-domain fine-tuning.The finding is reported across the investigated models and domains.
  • Implications: The results are discussed as relevant to evaluating and improving factuality and coherence in NLMs.The paper also points to implications for understanding model representations.

2 Background

The background situates the work within probing and dynamic semantics. Its central distinction is recovering discourse-level situation representations rather than only sentence-level linguistic information.

  • Existing probing: Prior probing studies identify syntactic, lexical-semantic, predicate–argument, and related information in language-model representations.These studies typically train low-capacity probes to predict linguistic labels from fixed representations.
  • Novelty: This paper extends probing to recover representations of the situation described by a discourse.The target is broader than the semantic information expressed by individual sentences.
  • Dynamic semantics: Dynamic semantics represents meaning with information states: sets of possible world states consistent with a discourse.Each new sentence updates the set and can constrain facts about even unmentioned entities.
  • Hypothesis: The paper hypothesizes that text-trained LMs represent information states and decodes them through truth values assigned to logical propositions.The proposed representation uses values of true, false, or undetermined.
  • Research goal: The work focuses on understanding representations themselves rather than optimizing downstream-task performance.This distinguishes its objective from much prior research using language-model pretraining for supervised semantic tasks.

3 Approach

The approach formalizes discourse meaning as information states and trains probes to recover proposition truth values from frozen language-model representations. It evaluates whether semantic information is encoded, where it is localized, and how accurately it can be recovered.

  • Probe design: The probe recovers each proposition’s truth value from encoder representations associated with the preceding discourse.The target labels are true, false, or undetermined.
  • Information states: A situation completely specifies the properties and relations of every entity in a defined universe.Logical propositions such as P(x) and R(x, y) receive true or false values in each situation.
  • Information states: A discourse denotes a set of possible situations, and each sentence updates that information state.The resulting state captures what is known or uncertain about the described world.
  • Scope: The tasks use the simplest parts of dynamic semantics and do not investigate interesting quantification or scope.More detailed exploration of quantification in NLMs is left for future study.
  • Probe design: The semantic probe tests whether propositions are encoded linearly, where they are encoded, and how accurately they can be recovered.It recovers information state contents proposition by proposition from E(x1:i).
  • Probe components: A localizer extracts token representations as candidates for encoding a proposition, and a classifier predicts its truth value.The classifier maps an embedded proposition and localized representation to T, F, or ?.
  • Evaluation: The classifier is trained on labeled propositions and evaluated on held-out discourses to test generalization.The procedure labels relevant propositions with their truth or falsehood before training and testing.

4 Experiments

The experiments test whether BART and T5 encode evolving entity states and whether manipulating those representations changes generated language. Across Alchemy and TextWorld, probes recover semantic state information, with evidence that representations are localized to entity mentions and shaped more by pretraining than fine-tuning.

  • Experimental setup: The experiments probe information states in BART and T5 using Alchemy and TextWorld representations, classifiers, and causal interventions.The encoder representations are frozen during probe training, and the classifier maps representations and propositions to truth values.
  • Experimental setup: Alchemy supplies deterministic beaker-state supervision by prepending each initial-state declaration to sequences of drain, pour, and mix instructions.Each instruction sequence has a fully specified initial state and ground-truth resulting state.
  • Probe results: 14.3% of Alchemy information states and 53.8% of TextWorld information states are exactly recovered by the T5 probe, exceeding the no-change baseline’s 0% and 9.7%.The no-LM baseline achieves 0% in Alchemy and 1.8% in TextWorld.
  • Probe results: Pretraining contributes more than in-domain fine-tuning to recoverable semantic state, although both contribute to final probe accuracy.Semantic state remains recoverable from models without in-domain fine-tuning.
  • Locality and fact decoding: Entity state information is roughly localized to mentions of the target entity, though locality varies by dataset and model.In Alchemy, correct-beaker tokens have the highest entity EM; in TextWorld, remapped-entity decoding is only 1–3% worse than decoding from the correct entity.
  • Locality and fact decoding: Relations can be decoded from either side of a relation, while two-entity probes are broadly better and one entity still contains nontrivial relational information.The experiments also manipulate encoder representations, causing the model to behave as if it were in the inserted information state; mixed encodings produce generations consistent with both beakers being empty.

5 Limitations

The limitations discussion emphasizes both language-model output and implicit state representations.

  • The limitations discussion emphasizes both LM output and implicit state representations.
  • LM output is explicitly included among the issues considered in the limitations discussion.
  • Implicit state representations are explicitly included alongside LM output in the limitations discussion.

6 Conclusion

The paper concludes that text-only training can produce simple, structured meaning representations in neural language models, with potential uses for analyzing and editing generation.

  • Text-only training enables neural language models to encode simple representations of meaning.
  • Linear probes map internal text representations to representations of the described world across two domains and two pretrained models.
  • These representations are structured, interpretably localized, and editable.
  • Probing entity states may reveal training-data biases, while editing representations may correct generation errors.

Impact Statement

The paper connects meaning representations in language models to potential benefits and risks in real-world deployment. It highlights applications for factuality, coherence, bias, and toxicity while warning that interpretability is dual-use.

  • The paper proposes probing and modifying language models' underlying information state to improve factuality and coherence and reduce bias and toxicity.
  • Interpretability research can improve the effectiveness of models for generating false, misleading, or abusive language.
  • Learned semantic representations may misrepresent the world because of prediction errors and training-data errors.

A.1 Datasets Details (§4.1)

The study uses Alchemy and TextWorld data to train and evaluate language models and probes, with TextWorld incorporating separate worlds, multiple agent behaviors, and partial-information challenges.

  • Alchemy preserves the original 3657-train/245-dev split and adds a synthetic 3600-train/500-dev dataset for consistency evaluation.
  • TextWorld uses separate training and testing worlds and transcripts from one perfect and two semi-random agents.
  • TextWorld training samples 4000 sequences across 79 worlds, while development samples 500 sequences across 9 worlds.
  • Because players and language models may lack full state information, the dataset includes propositions that are known true, known false, or unknown.
  • A separately trained BART labeller predicts proposition truth values from transcripts and uses confidence to construct the unknown set.

A.2 Probe Details + Additional Results (§4.2)

The probing setup tests whether BART and T5 representations encode propositions about entities and situations, using proposition embeddings and localized language-model representations. Probes decode beaker states, entity properties, and relations from these representations.

  • Proposition embeddings are converted to natural-language descriptions and encoded with the same BART or T5 encoder being probed.
  • Alchemy Probe: Alchemy localization maps a beaker proposition to the eight tokens comprising that beaker’s initial-state declaration.The tokens encode “the,” the beaker identifier, “beaker,” “has,” amount, color, and punctuation.
  • Alchemy Probe: The Alchemy probe predicts each beaker’s final state by learning a linear projection between localized LM representations and embedded propositions.It selects the state maximizing the dot product over possible values and colors.
  • Additional Results: Synthetic Alchemy data permits accurate decoding from a wider token set, although those tokens still correspond to the relevant beaker.
  • Textworld Probe: For TextWorld, localization maps entity or entity-pair propositions to all argument mentions and averages their representations.
  • Textworld Probe: A bilinear probe classifies proposition–representation pairs as true, false, or unknown using three scores and chooses the highest-scoring label.

A.3 Localization Experiment Details + Additional Results (§4.3)

The localization experiments test where entity and situation information is encoded by varying which token representations are supplied to probes. Results compare mention-based localization, token windows, synthetic data, and first-versus-last mentions.

  • Mentions vs. Other Tokens (§4.3.1) – Alchemy: Alchemy localization probes extract propositions about a target beaker from tokens at varying offsets around its initial-state description.
  • Mentions vs. Other Tokens (§4.3.1) – Alchemy: Token-wise results evaluate beaker-state decoding within a 3-beaker, 24-token window around the target beaker.
  • Additional Results: In synthetic Alchemy, state information remains local to the target beaker’s initial-state description but is more distributed across amount, color, and mention tokens.
  • Mentions vs. Other Tokens (§4.3.1) – Textworld: TextWorld localization experiments exclude contexts that do not mention the remapped entity.
  • Which Mention? (§4.3.2) – first/last: First-versus-last localization replaces all-mention indices with indices for either the first or last instance of an entity.
  • Which Mention? (§4.3.2) – single- vs. both-entity probe: The both-mentions probe is slightly better for decoding relations and properties, while the single-entity probe outperforms all baselines.The authors note that the both-mentions advantage may reflect fewer candidate propositions per entity pair.

A.4 Proposition Embedder Ablations

The ablation shows that proposition embedding format substantially affects probe performance, while error analysis identifies action effects and relation decoding as important sources of difficulty.

  • Proposition Embedder Ablations: 75.0 versus 45.7: encoded natural-language representations significantly outperform featurized representations for proposition probing.The result suggests that fact-embedding form matters: encoding is linear in sentence-embedding space but nonlinear in human-grounded feature space.
  • Error Analysis: 53.5% state EM versus 7.55% for a linear decoder: a nonlinear BART decoder recovers substantially more final-state information.The authors caution that probe errors cannot be cleanly separated from representation errors, and the result suggests some state information is nonlinearly encoded.
  • Error Analysis: 44.6% error for acted-on beakers versus 13.3% for unchanged beakers, indicating that errors chiefly concern reasoning about action effects.Overall, the average number of incorrect beakers per sample is 25.0%, or 2.7 out of 7.
  • Error Analysis: TextWorld relations are harder to probe than properties, with locational relations among the proposition types the probe consistently misclassifies.Table 6 reports whole-state metrics, while Table 7 breaks error rates down by proposition type.
  • Error Analysis: 31.4% and 33.3% error rates for pour- and mix-type actions exceed the 25.3% rate for drain-type actions.The difference is partly attributed to the greater concentration of drain actions in shorter action sequences.
Loading 2106.00737v1…