PaperlayerView exact revision on arXiv

Source-linked AI summary

Do Language Models Have Beliefs? Methods for Detecting, Updating, and Visualizing Model Beliefs

Peter Hase, Mona Diab, Asli Celikyilmaz, Xian Li, Zornitsa Kozareva, Veselin Stoyanov, Mohit Bansal, Srinivasan Iyer

arXiv:2111.13654v1cs.CLcs.AIcs.LG

TL;DR

The paper asks how language models can be understood and evaluated in terms of beliefs, including their consistency and interdependencies. It develops belief-detection measures, sequential learned-optimizer updates, and belief graphs. Models show limited belief-like qualities, and SLAG improves learned optimizers in sequential updating, where baseline performance otherwise declines.

  • Problem

    Language models may store information without exhibiting the logical consistency and other structural properties associated with beliefs, motivating tests of belief-like qualities and their interdependencies.

    Source: §1 · pp. 1–2

  • Method

    The paper evaluates belief consistency, trains learned optimizers with SLAG for sequential updates, and visualizes inter-belief effects as belief graphs.

    Source: §1 · p. 2

  • Results

    Models exhibit limited belief-like qualities; off-the-shelf optimizers generally outperform learned methods for single updates, while SLAG lets learned optimizers outperform baselines sequentially.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The authors advocate evaluating updates by their ability to correct false beliefs, and use belief graphs to expose dependencies among model beliefs.

    Source: §1 · p. 2

    Treating an update as a change in model output misses changes in pθ(y|x) that leave the most probable output unchanged.

    Source: §7 · p. 10

Abstract

from arXiv · show

Do language models have beliefs about the world? Dennett (1995) famously argues that even thermostats have beliefs, on the view that a belief is simply an informational state decoupled from any motivational state. In this paper, we discuss approaches to detecting when models have beliefs about the world, and we improve on methods for updating model beliefs to be more truthful, with a focus on methods based on learned optimizers or hypernetworks. Our main contributions include: (1) new metrics for evaluating belief-updating methods that focus on the logical consistency of beliefs, (2) a training objective for Sequential, Local, and Generalizing model updates (SLAG) that improves the performance of learned optimizers, and (3) the introduction of the belief graph, which is a new form of interface with language models that shows the interdependencies between model beliefs. Our experiments suggest that models possess belief-like qualities to only a limited extent, but update methods can both fix incorrect model beliefs and greatly improve their consistency. Although off-the-shelf optimizers are surprisingly strong belief-updating baselines, our learned optimizers can outperform them in more difficult settings than have been considered in past work. Code is available at https://github.com/peterbhase/SLAG-Belief-Updating

1 Introduction

The paper studies language models through belief detection, updating, and visualization, treating belief as a useful lens on model behavior rather than claiming human-like beliefs. It finds belief-like qualities are limited, while update methods can improve consistency and belief graphs expose dependencies.

1 Introduction

  • Analyzing language models in terms of beliefs may help explain their behavior because internal informational states can support accurate behavioral inferences.The paper presents this as a useful lens, not as a claim that models have beliefs in the same sense as people.Source: §1 · p. 1
  • Belief detection tests paraphrase consistency, entailment consistency, and whether logically neutral beliefs remain independent.The entailment tests include contrapositives, and logically neutral pairs are constructed from Wikidata5m.Source: §1 · p. 2
  • SLAG improves learned optimizers by training them on harder data and multiple small edits, raising update success and reducing corruption of other beliefs.Sequential training also lets learned optimizers outperform off-the-shelf methods when updating multiple beliefs in a row.Source: §1 · p. 2
  • The authors advocate evaluating updates by whether they correct false or morally undesirable beliefs, rather than merely changing beliefs to plausible alternatives.They note that correcting incorrect beliefs is harder than changing predictions to similar outputs.Source: §1 · p. 2
  • Belief graphs represent beliefs as nodes and directed edges that indicate when changing one belief changes another.The paper also discusses graph metrics to summarize these dependencies.Source: §1 · p. 2
  • Models exhibit limited belief-like qualities, while belief graphs reveal nonsensical dependencies and highly connected beliefs.The authors report paraphrase consistency below 70% and mixed consistency under entailment.Source: §1 · p. 2

2 Related Work

Related work examines information stored in language models, consistency as a criterion for belief, and methods for making targeted model updates. This paper adopts prior consistency measures and builds on learned-optimizer approaches to editing model weights.

2 Related Work

  • Talmor et al. (2020) measure consistency under entailment, while Elazar et al. (2021) measure paraphrase consistency; the paper adopts both approaches.Kassner et al. (2021) concurrently examine both kinds of consistency and introduce BeliefBank for entailment.Source: §2 · p. 3
  • Post-training update methods include hand-crafted weight-editing algorithms, projected gradient descent, and learned optimizers that balance the desired output with limited changes elsewhere.The paper builds on De Cao et al. (2021), identifies difficulty with sequential belief updates, and introduces SLAG.Source: §2 · p. 3
  • Test-time information addition can improve entailment consistency without changing model weights or affecting outputs on other potentially relevant inputs.Kassner et al. (2021) use this approach; the passage compares it with retrieval-based methods.Source: §2 · p. 3

3 Updating Beliefs in Language Models

The paper formulates belief editing as a learned-optimizer problem and evaluates whether updates change the target belief while preserving consistency and unrelated outputs. It extends training and evaluation to sequential updates, a setting intended to better reflect repeated model editing.

3 Updating Beliefs in Language Models

  • A learned optimizer uses the current model weights, input, current prediction, and desired output to produce updated model parameters.The paper follows De Cao et al. (2021) and uses KNOWLEDGEEDITOR with the SLAG training objective.Source: §3 · pp. 4–5
  • The evaluation metrics cover desired changes to the main input and paraphrases, retained predictions elsewhere, accuracy changes, entailed data, and similar but logically neutral data.Local Neutral data is guaranteed to be logically independent of the Main Input while remaining similar to it.Source: §3 · p. 4
  • The authors argue that evaluating whether updates correct incorrect predictions is harder than changing them to similar alternatives, so prior methods’ efficacy may be overestimated.They propose focusing evaluation on making incorrect points correct.Source: §3 · p. 4
  • Sequential evaluation applies multiple belief updates before measurement, reflecting developers’ likely need to update many beliefs over time.For a test set of n points and r consecutive updates, the procedure yields floor(n/r) measurements.Source: §3 · p. 5
  • SLAG trains the optimizer across multiple datapoints in sequence, using the per-datapoint objective and model parameters produced by earlier updates.The training objective limits backpropagation through the model history to keep memory use constant in the number of sequential updates.Source: §3 · p. 5

4 Experiment Setup

The experiments use four factual or relational datasets to test belief-related properties and updates. They compare learned and off-the-shelf optimizers under task-specific model selection and evaluation procedures.

4.1 Datasets

  • FEVER uses its original 10,444-point test set, while zsRE uses 80/10/10 train/dev/test splits and measures paraphrase update success on 32.8% of questions.FEVER training data is randomly split into 94,469 train and 10,496 dev points.Source: §4.1 · p. 6
  • LeapOfThought uses filtered unique factual claims, and Wikidata5m supplies paired-relation sequence examples for evaluating local-neutral behavior.For Wikidata5m, paired inputs share an entity but use different relations and labels; the paired facts are designed to be logically independent while remaining local to the main input.Source: §4.1 · p. 6

4.2 Methods Evaluated

  • Five random-seed models are trained per dataset, using RoBERTa-base for binary tasks and BART-base for sequence-to-sequence tasks.Each model gets learned optimizers trained with SLAG and the De Cao et al. (2021) objective, listed as KE.Source: §4.2 · p. 6
  • Learned and off-the-shelf optimizers are selected using the same task-specific criterion, while ablations use one model seed per condition.The criterion averages Update Success Rate, Local Neutral Retain Rate, and All Data Δ-Acc; confidence intervals and hypothesis tests use block bootstrap over model seeds and data points.Source: §4.2 · p. 6

5 Experiment Results

Across detection and updating experiments, ~100M-parameter models showed limited belief-like consistency, while update results varied with the setting and method. The experiments also tested how individual terms in the learned-optimizer objective affected performance.

5.2 Can we update beliefs in LMs?

  • With single updates, off-the-shelf optimizers outperform learned optimizers on most metrics and binary-task ∆-Acc, while SLAG improves selected measures over KNOWLEDGEEDITOR.SLAG raises All Data Retain Rate on zsRE by 5.77 points and Wikidata5m Paraphrase Update Success by 11.92 points; its 3.02-point Entailed Data Update Success gain is not significant.Source: Table 4 · p. 7, §5.2 · pp. 7–8

5.1 Do LMs have beliefs about the world?

  • Belief-like qualities were limited: paraphrase consistency was 69.50% on zsRE and 25.84% on Wikidata5m, while LeapOfThought entailment and contrapositive accuracy were 85.63% and 16.51%.Entailment accuracy is computed only where the Main Input prediction is correct; contrapositive accuracy measures whether the model follows the contrapositive when its antecedent holds.Source: §5.1 · pp. 6–7

5.2 Can we update beliefs in LMs?

  • Positive ∆-Acc values may reflect test-split distribution shift: FEVER train and dev data are 73% True, compared with 50% True in test.On the dev split, AdamW achieves a negative ∆-Acc of -0.18 (±0.11).Source: §5.2 · p. 7
  • On zsRE, sequentially trained SLAG improves Main Input Update Success by 4.86 (±0.83; p=1e−4) and Paraphrase Update Success by 1.39 (±0.93; p=.004) under ten updates.Improvements on Wikidata5m followed the same direction and were statistically significant except for ∆-Acc; non-sequential training led to drastic performance drops.Source: Table 6 · p. 8, §5.2 · p. 8

5.3 How does the learned optimizer objective influence performance?

  • Objective-term ablations were mixed: the paraphrase objective reduced zsRE Main Input Update Success by 0.71 (±0.60; p=.021), while effects on Wikidata5m varied.On Wikidata5m, paraphrase Update Success rose 16.94 (±1.03; p<1e−4) points and Local Neutral Retain Rate rose 9.71 (±1.44; p<1e−4) points; the entailment objective reduced Entailed Data Update Success by 4.56 (±7.22; p=.213) points.Source: §5.3 · p. 8

6 Analysis

Belief updates improve paraphrase consistency and entailment accuracy, while some beliefs are harder to retain than others. Belief graphs reveal extensive interconnections, consequential corruption, and often-intransitive update effects.

6.1 Belief updates improve consistency

  • 33.14±1.46 on zsRE and 59.87±1.09 on Wikidata5m are the gains in paraphrase consistency after updating; entailment accuracy rises by 17.20±7.10 points.For Wikidata5m, post-update consistency is 90.1% when pre-update consistency is maximal, versus 77.1% when beliefs are initially totally inconsistent.Source: §6.1 · pp. 8–9, Table 7 · p. 9

6.2 Which beliefs are hard to retain when updating other beliefs?

  • Incorrect predictions are harder to retain: zsRE retain rates are about 96% for initially correct inputs and 75% for incorrect ones.On FEVER, incorrect beliefs change around 4% of the time, compared with 2.5% for correct beliefs.Source: §6.2 · p. 9
  • On Wikidata5m, Local Neutral data has a 47.85±0.96 Retain Rate, 15.66 points below the 61.51±1.33 rate on All Data.Local Neutral beliefs are harder to avoid changing than random data.Source: §6.2 · p. 9

6.3 Belief Graphs

  • Belief graphs connect datapoints with directed edges when updating one changes another, using off-the-shelf optimizers to reverse each prediction.The graphs contain 10,444 FEVER nodes and 8,642 LeapOfThought nodes, including Main Inputs and Entailed Data.Source: §6.3 · p. 9
  • The inspected graph subgraphs show no apparent reason why individual beliefs are connected, though incorrect and Local beliefs appear more likely to change.Figure 3 colors nodes by pre-update accuracy and labels them with ground truth.Source: §6.3 · p. 9, Figure 3 · p. 10
  • For 5% of the data, updates cause at least 211 FEVER points and 2,752 LeapOfThought points to become incorrectly predicted; updates are often intransitive.Every belief has an in-edge or out-edge, and some beliefs are sensitive to or influential over many others.Source: §6.3 · pp. 9–10, Table 8 · p. 10

7 Discussion and Conclusion

The discussion treats belief updates as changes in model outputs and notes both uncertainty-related scope limits and dual-use concerns. The learned optimizer generates low-rank transformations of model gradients to edit task-model weights.

7 Discussion and Conclusion

  • The data use declarative statements with full commitment, leaving uncertainty expressed in language or as model-assigned probabilities for future work.The paper proposes evaluating update methods by their ability to correct mistaken beliefs while noting possible malicious use.Source: §7 · p. 10
  • The paper counts a belief as updated when the model output changes, which misses changes in pθ(y|x) that leave the output mode unchanged.The authors also note that update methods could induce problematic beliefs, and that their effects on broader model behavior are uncertain.Source: §7 · p. 10

A Learned Optimizer Details

  • KNOWLEDGEEDITOR encodes an input and requested prediction change, then uses low-rank matrices to transform the model gradient and produce new weights.For Transformer models, it edits attention and feed-forward weights.Source: A Learned Optimizer Details · p. 12
  • For sequence-to-sequence update labels, the optimizer uses the correct label when a prediction is wrong and a randomly selected training label when it is right.This differs from prior methods that use model beam-search samples as update labels for all points.Source: A Learned Optimizer Details · p. 12

B.1 Compute Costs.

The appendices describe memory-efficient learned-optimizer training, hyperparameter selection, and dataset-specific procedures. Additional details include limitations of Wikidata5m paraphrases and LeapOfThought entailment training.

B.1 Compute Costs.

  • Stop-gradient and gradient accumulation keep memory constant in r and limit growth with K in the reported implementation.The implementation uses a BART-base model on zsRE with batch size 16; recomputing gradients at each K step is noted as a possible trade-off.Source: §B.1 · p. 13, Figure 5 · p. 15

B.2 Hyperparameters and Objective Terms.

  • Task models train for 10 binary-task epochs or 20 sequence-to-sequence epochs, while learned optimizers are selected using update success, Local Neutral Retain Rate, and Δ-Acc.Baseline optimizers are tuned separately for each dataset; the learned optimizer edits attention and feed-forward weights, while the baseline optimizes all trainable parameters.Source: §B.2 · pp. 13–14

B.3 Wikidata5m Additional Details.

  • Wikidata5m uses four template-based paraphrases per Main Input, and only 33.80% of dev entities and 37.18% of test entities appear in training.The authors report no better model performance on entities seen in training.Source: §B.3 · p. 14

B.4 LeapOfThought Additional Details

  • LeapOfThought test accuracy is 83.65 (± 1.05) on claims and 93.66 (±0.87) on facts, but contrapositive accuracy is low.Because fewer than 10% of training points permit the entailed-data objective, the authors oversample them to fit that objective.Source: §B.4 · pp. 14–15

C Noise in Datasets

The dataset examples include ambiguous, vague, or nonsensical statements, as well as questions whose answers can be one-to-many or time-dependent. These properties constrain how confidently their labels can be interpreted.

C Noise in Datasets

  • FEVER includes vague or ambiguous claims, including examples whose truth depends on interpretation or personal opinion.The paper notes that such ambiguity is a general language issue, not unique to this dataset.Source: C Noise in Datasets · p. 15
  • LeapOfThought contains nonsensical or vague statements for which the conditions making a claim true or false are unclear.Examples include “A sunlight is a good health” and “A detective desires equal opportunity.”Source: C Noise in Datasets · p. 15
  • zsRE questions can involve one-to-many or time-dependent relations despite having only one ground-truth answer.Spatial answers may validly name a city, region, country, province, or continent.Source: C Noise in Datasets · pp. 15–16
  • Wikidata5m examples can have misleading aliases or relation rephrasings, and some relations are one-to-many or time-dependent.The authors hope that allowing multiple valid answers addresses some of this issue when measuring correctness.Source: C Noise in Datasets · p. 16

D Metric Computation and Bootstrap Details

The section describes how update metrics and uncertainty are computed, including a caveat for interpreting update transitivity and evidence about sequential-update settings.

Metric computation

  • For Δ-Acc, each update is evaluated on 30 sampled points for zsRE and Wikidata5m and 200 for FEVER and LeapOfThought.Sampling preferentially selects evaluation data used less often; on LeapOfThought, Δ-Acc is computed for Main Inputs and Entailed Data.Source: D Metric Computation and Bootstrap… · p. 16
  • Update-Transitivity measures how often changing A changes C when updating A changes B and updating B changes C.The authors caution that a confounding belief D could determine the B–C relationship; absent reason to suspect one, they expect logically consistent models to show 100% transitivity.Source: D Metric Computation and Bootstrap… · pp. 16–17

Sequential-update ablation

  • On zsRE, the method outperforms the baseline at rtest = 10, while a non-sequential objective’s performance declines drastically as rtest rises.The figure caption says the gap is likely to increase at higher rtest.Source: Figure 6 · p. 17

Bootstrap computation

  • Block bootstrap resamples data points and model seeds 10,000 times, pairing data points across conditions for hypothesis tests.For metrics with multiple observations per Main Input, those observations are averaged rather than resampled; a full three-dimensional bootstrap was slower and gave similar results.Source: D Metric Computation and Bootstrap… · p. 17

E Additional Results

Additional results show that sequential training matters for repeated updates, training-label choice affects correction success, and updateability does not track initial paraphrase consistency. Further ablations examine training-set size and the number of update steps.

Ablation across num. sequential steps

  • Sequential training is critical when learned optimizers must update points in sequence, and it is the only setting where they outperform baselines across all relevant metrics.The comparison concerns sequential updating with sequence prediction tasks.Source: E Additional Results · p. 17

Choosing training labels for learned optimizers

  • Using correct or random desired labels instead of beam-search samples improves Main Input update success by 3.27 (±0.65; p<1e−4) points and paraphrase success by 2.38 (±1.05; p<1e−4) points.Δ-Acc rises by 0.15 (±0.18; p=.09); the authors recommend these labels when the goal is fixing wrong outputs.Source: E Additional Results · p. 18

Which beliefs are hard to update?

  • Initial paraphrase consistency is nearly uncorrelated with single-update success on zsRE (ρ = −.027) and Wikidata5m (ρ = .013).The authors report no variance to explain on LeapOfThought because update success there is already 100%.Source: E Additional Results · p. 18, Figure 7 · p. 18

Learning curve

  • Main Input Update Success Rate steadily rises with training-set size for a learned optimizer trained with SLAG on zsRE.Figure 8 presents the learning curve across training-set sizes.Source: E Additional Results · p. 18, Figure 8 · p. 18

Ablation by num. update steps

  • On zsRE, Main Input Update Success rises by over three points when Ktest increases from 1 to at least 5; matching Ktrain to Ktest adds about 0.5 points.The figure caption says training with the test-time K is useful.Source: E Additional Results · p. 18, Figure 9 · p. 19

Belief graphs

Loading 2111.13654v1…