PaperlayerView exact revision on arXiv

Source-linked AI summary

Monitoring Latent World States in Language Models with Propositional Probes

Jiahai Feng, Stuart Russell, Jacob Steinhardt

arXiv:2406.19501v2cs.CLcs.LG

TL;DR

Language models can produce responses unfaithful to their input, motivating tools that inspect whether their internal representations remain faithful. The paper introduces compositional propositional probes for decoding latent world states from activations, finding faithful propositions across complex and adversarial settings even when outputs are unfaithful.

  • Problem

    Language models may be unfaithful because of irrelevant examples, unintended tendencies, biases, prompt attacks, or training-data attacks, creating a need to identify and correct such behavior.

  • Method

    Propositional probes extract lexical concepts from activations and bind them into logical propositions using a binding subspace and similarity metric.

  • Results

    Probes generalize from simple templates to rewritten stories and Spanish translations, achieving a Jaccard Index within 10% of a prompting skyline, and produce more faithful propositions than outputs in prompt injections, backdoors, and gender bias.

  • Takeaways & Limitations

    The results suggest that language models can encode a faithful world model even when they decode it unfaithfully, motivating latent-state monitoring systems.

  • Takeaways & Limitations

    The binding-subspace method may be too noisy for contexts with more than two entities, because the current subspace may contain spurious directions or incompletely capture binding.

Abstract

from arXiv · show

Language models are susceptible to bias, sycophancy, backdoors, and other tendencies that lead to unfaithful responses to the input context. Interpreting internal states of language models could help monitor and correct unfaithful behavior. We hypothesize that language models represent their input contexts in a latent world model, and seek to extract this latent world state from the activations. We do so with 'propositional probes', which compositionally probe tokens for lexical information and bind them into logical propositions representing the world state. For example, given the input context ''Greg is a nurse. Laura is a physicist.'', we decode the propositions ''WorksAs(Greg, nurse)'' and ''WorksAs(Laura, physicist)'' from the model's activations. Key to this is identifying a 'binding subspace' in which bound tokens have high similarity (''Greg'' and ''nurse'') but unbound ones do not (''Greg'' and ''physicist''). We validate propositional probes in a closed-world setting with finitely many predicates and properties. Despite being trained on simple templated contexts, propositional probes generalize to contexts rewritten as short stories and translated to Spanish. Moreover, we find that in three settings where language models respond unfaithfully to the input context -- prompt injections, backdoor attacks, and gender bias -- the decoded propositions remain faithful. This suggests that language models often encode a faithful world model but decode it unfaithfully, which motivates the search for better interpretability tools for monitoring LMs.

1 INTRODUCTION

The paper develops propositional probes to extract symbolic latent world states from language-model activations and tests whether those states remain faithful when outputs become unfaithful. The probes generalize beyond simple training contexts and outperform model outputs in three adversarial settings.

  • Propositional probes extract logical propositions from internal activations, and in prompt injections, backdoors, and gender bias they remain more faithful than model outputs.For example, the probes decode LivesIn(Alice, Laos) and LivesIn(Bob, Peru) from the corresponding context.
  • The probes compose lexical concepts from token activations into propositions, using a binding subspace where tokens bound together have similar activations.The binding subspace enables composition across tokens, such as pairing Alice with Laos rather than Bob with Laos.
  • 10%: On rewritten stories and Spanish translations, probes achieve a Jaccard Index within 10% of a prompting skyline despite training only on simple templates.
  • The results motivate monitoring systems that inspect latent world models, especially as language models are deployed as autonomous agents.The authors propose extending the approach to more complex role-filler bindings and state changes.

2 RELATED WORK

The related work connects propositional probing to prior lexical and environment-specific probing research, while positioning the paper's contribution as compositional binding across lexical probes.

  • Probing: Propositional probing builds on studies decoding beliefs in Othello, Alchemy, and TextWorld, while exploiting compositionality to leverage broader probing research.
  • Representations of binding: The approach builds on research showing semantic-role and coreference representations in language models and on binding vectors discovered in activations.

3 TASK DEFINITION AND PRELIMINARIES

The task is to decode ground-truth logical propositions representing a model's beliefs from its internal activations. Evaluation uses a closed world with two people and tests probes trained on templated data against stories and Spanish translations.

  • The task predicts a set of propositions representing an input passage from the model's internal activations, assuming the model can answer related comprehension questions.
  • The model processes each passage into token-position activations, from which the proposition set is predicted.
  • Probes are trained only on SYNTH and evaluated on held-out, harder versions to measure easy-to-hard generalization.
  • Datasets progress from templated SYNTH inputs to GPT-3.5-turbo-rewritten PARA stories and Spanish TRANS translations, all labeled with ground-truth propositions.
  • The closed world contains names, countries, occupations, and foods linked by LivesIn, WorksAs, and LikesToEat predicates.Each predicate binds a name to one attribute domain.

4 PROPOSITIONAL PROBES

Propositional probes first classify lexical values in token activations and then bind predicted attributes to entities using a binding similarity metric. This composition relies on learned domain directions, thresholds, attribution-based supervision, and a metric that favors correctly bound pairs.

  • Domain probes: Domain probes classify each activation as a domain value or null using learned vectors and a threshold, with the highest dot product determining the prediction.
  • Domain probes: 44: Domain-probe vectors are learned from activations associated with each value and centered by subtracting the average domain vector.
  • Domain probes: A Grad-CAM-style attribution method assigns token- and layer-level supervision by estimating which activations contribute to lexical information.
  • Domain probes: Layer 20 of 40 is selected because middle-layer activations at the last token position are most informative.
  • Binding composition: The binding similarity metric should be higher for correctly bound pairs such as Laos–Alice than for unbound pairs such as Laos–Bob.
  • Propositional probes: Propositional probes bind predicted attributes to entities by selecting, for each attribute, the name with the highest binding similarity.

5 BINDING SUBSPACE

The paper identifies a binding subspace in language-model activations using a Hessian-based algorithm, then evaluates whether it captures and enables controlled swapping of entity–attribute bindings. The method succeeds in recovering binding structure, but current similarity measurements become noisy beyond two entities.

  • 5.2 HESSIAN-BASED ALGORITHM: The Hessian-based algorithm identifies a low-rank binding matrix H by measuring how coordinated activation perturbations change binding strength.It erases binding information, perturbs entity and attribute activations, and estimates which directions jointly restore correct attribute predictions.
  • 5.2 HESSIAN-BASED ALGORITHM: The method uses binding similarity d(Zk, Zl) to compose lexical domain probes into propositions by matching each attribute with the best-associated name.The binding subspace is intended to separate binding information from lexical content and support compositional proposition construction.
  • 5.3 EVALUATIONS OF THE HESSIAN-BASED ALGORITHM: Interchange interventions test whether swapping projected binding components changes pairings from (Ei, Ai) and (Ej, Aj) to (Ei, Aj) and (Ej, Ai).The evaluation applies all pairwise swaps in synthetic three-entity, three-attribute contexts.
  • 5.3 EVALUATIONS OF THE HESSIAN-BASED ALGORITHM: The top 50 Hessian dimensions out of 5120 capture binding information and enable swaps involving a third binding vector, whereas random subspaces fail without also changing content.The Hessian subspace also outperforms the skyline for swapping the second and third entity or attribute bindings.
  • 5.3 EVALUATIONS OF THE HESSIAN-BASED ALGORITHM: For three-entity contexts, the binding metric does not clearly distinguish the second and third entities, suggesting spurious directions or incomplete subspace recovery.The authors conclude that current methods may be too noisy beyond two entities, while also observing partial sensitivity to token order.

6 PROPOSITIONAL PROBES EVALUATIONS

Propositional probes match prompting on standard contexts and remain more faithful under prompt injections, backdoors, and gender bias, though performance depends on the learned binding metric.

  • 6.2 ADVERSARIAL SETTINGS: Across prompt-injected and backdoored settings, probing outperforms prompting while preserving access to the input-context propositions.Table 1 summarizes exact-match and Jaccard results for standard, prompt-injected, and backdoored conditions.
  • 6.1 STANDARD SETTINGS: Probing performs comparably with prompting on standard datasets by Jaccard index, but significantly worse on TRANS exact-match accuracy.The evaluation reports these comparisons in Table 1; the TRANS exact-match drop is attributed partly to weaker food-probe generalization.
  • 6.1 STANDARD SETTINGS: Replacing the Hessian-based binding metric with DAS or random subspaces significantly reduces performance, supporting its role in identifying a useful binding subspace.The ablations replace the learned binding subspace with 50-dimensional and 1-dimensional DAS subspaces or a random 50-dimensional subspace.
  • 6.2 ADVERSARIAL SETTINGS: Both prompting and probing show gender bias, but probing is significantly less biased between pro-stereotypical and anti-stereotypical contexts.The probing analysis tests whether the binding subspace preferentially associates occupations with male or female tokens and also reports calibrated accuracy.

7 CONCLUSION

The paper presents evidence that language models construct symbolic internal models of input contexts and that these models can remain faithful when outputs become unfaithful. It proposes propositional probes as a basis for monitoring such behavior, while noting that greater semantic complexity remains to be addressed.

  • 7 CONCLUSION: The work supports two hypotheses: language models construct symbolic models of input contexts, and those models may remain faithful despite unfaithful outputs.The conclusion frames these as hypotheses supported by the paper’s evidence.
  • 7 CONCLUSION: Propositional probes decode symbolic propositions from activations in a small closed world, enabled by a discovered binding mechanism.Scaling to larger worlds may require representations for role-filler binding and state changes.
  • 7 CONCLUSION: Because probes remain faithful when outputs are unfaithful, sufficiently complex probes could monitor models at inference time for attacks, unintended tendencies, and biases.The conclusion presents this as a prospective application rather than a demonstrated deployment capability.

A DATASETS

The evaluation uses a synthetic dataset of templated propositions, then tests generalization to GPT-rewritten stories and Spanish translations. The experiments use finite name, country, food, and occupation domains with controlled probing procedures.

  • A DATASETS: SYNTH contains 512 random template-generated examples plus a separate 512-example validation set used to select domain-probe thresholds.The template expresses two people’s countries, foods, and occupations.
  • A DATASETS: The closed-world domains contain 60 names, 16 countries, 41 foods, and 14 one-token occupations.The occupation list is restricted to one-token occupations from Winobias; foods are two-token items.
  • A DATASETS: PARA rewrites SYNTH examples into one-paragraph stories, while TRANS translates PARA examples into fluent Spanish while preserving their meaning.Both transformations are produced with GPT-3.5-turbo instructions.
  • A DATASETS: Binding evaluation averages intervention accuracy over 20 randomly constructed contexts, with layer-dependent activation scaling used to improve interchange-intervention accuracy.The experiments use an internal GPU cluster, with Hessian computation taking about five hours.

C.1 BINDING SUBSPACE DESIGN CHOICES

The binding-subspace design follows a two-stage circuit in which an entity’s binding identity is retrieved and later used to recover its attribute. The authors simplify perturbation placement and use a symmetric metric to accommodate binding-vector evolution across layers.

  • C.1 BINDING SUBSPACE DESIGN CHOICES: Resolving Bob’s country retrieves Bob’s binding ID around layer 15, then uses it around layer 30 to look up the matching attribute, Peru.The circuit motivates the binding-subspace parameterization and similarity design.
  • C.1 BINDING SUBSPACE DESIGN CHOICES: Bob’s and Peru’s binding vectors are not identical, so the design must account for their differing layer-specific representations.The empirical difference motivates shared parameterization and a symmetric binding metric.
  • C.1 BINDING SUBSPACE DESIGN CHOICES: The method shares Hessian parameters across layers and injects perturbations broadly, reducing hyperparameter tuning but potentially sacrificing precision.The authors expect restricting perturbed layers more carefully could improve binding-subspace accuracy.
  • C.1 BINDING SUBSPACE DESIGN CHOICES: The circuit-motivated design choices are ultimately validated by the performance of the resulting propositional probes.The implementation also trains Hessian and DAS components on templated names-and-countries datasets.

E QUALITATIVE HESSIAN ANALYSIS

The analyses examine where lexical information is localized and whether the binding metric tracks coreference and multi-entity relationships. Grad-CAM identifies informative layers and token positions, while qualitative binding analyses reveal successful coreference tracking but noise in three-entity discrimination.

  • Coreferred entities receive the same binding vectors as their referrents regardless of whether references appear in matching or reversed order.
  • The three-entity similarity metric does not strongly distinguish the second and third entity, although interchange interventions still work.The authors attribute this mismatch to possible spurious non-binding directions in the 50-dimensional subspace.
  • Grad-CAM attributes domain-value information mainly to the token position carrying that value and to middle layers.For two-token food values, information is concentrated in the second token position.
  • The probe layer is selected as l = 20, and per-layer AUC-PRC supports this choice.The attribution procedure estimates each position-layer contribution using a linear gradient approximation over contrast pairs.
  • The threshold h is selected using validation accuracy on PARAPHRASE and TRANSLATE subsets.

H GENDER BIAS EVALUATIONS

The gender-bias evaluation compares prompting and probing on synthetic occupation–country contexts, including reordered sentence structures and varying entity–attribute distances. Probes remain competitive under these variations and outperform prompting in anti-stereotypical cases, while order can reduce their accuracy.

  • Gender bias evaluation: The evaluation uses 400 synthetic contexts linking occupations, countries, and genders, with half the contexts swapping the final two sentences.
  • Gender bias evaluation: Prompting predicts gender from higher occupation-conditioned log probability, whereas probing selects the gender token with higher binding similarity to the occupation token.
  • Gender bias evaluation: Accuracy measures correct predictions, while calibrated accuracy removes paired-response mean log-probability preferences before applying the same decision rule.
  • Gender bias evaluation: Probes outperform prompting in both series and nested orders when occupations are anti-stereotypical, despite lower probe accuracy in nested order.
  • Order analysis: Probes are robust to studied data orderings except nested, indicating partial sensitivity of the extracted binding subspace to order.
  • Position analysis: Propositional probes do not degrade on LONG or MEDIUM contexts compared with SHORT contexts.The LONG and MEDIUM datasets insert noun and verb phrases between entity and attribute positions.

I.2 ORDER ANALYSIS

Order analysis tests propositional probes across several name–attribute arrangements, including coreference and nested constructions. Probes are robust to most orderings, retain advantages over prompting in adversarial settings, and show errors consistent with a mixture of semantic binding and order information.

  • Order analysis: Propositional probes remain robust across studied orderings except nested, where they achieve 44% exact-match accuracy.This exceeds the 25% random-binding expectation but differs from the 0% expected for purely order-based binding.
  • Order analysis: Constraining propositions to unique entities raises accuracy to 76%, while 78% of wrong outputs assign both countries to the same name.These errors support a continuous interaction between order information and semantic binding.
  • Order analysis: On the alternate nested dataset, prompting fails catastrophically whereas probes perform similarly to the original nested dataset.
  • Order analysis: Probes still outperform prompting on adversarial prompt-injection and backdoor versions of the nested dataset.
  • Order analysis: In gender-bias contexts, probes perform worse under nested order than series order but still outperform prompting for anti-stereotypical occupations.
  • Additional model results: The evaluated Llama-2-13b-chat results are mostly similar to the main results.
Loading 2406.19501v2…