PaperlayerView exact revision on arXiv

Source-linked AI summary

Beliefs and Behavior in Language Models

Alex Smolin, Bryan Wilder

arXiv:2609.07943v1cs.AIcs.LG

TL;DR

The paper addresses whether beliefs are useful descriptions of LLM behavior, a question relevant to making decisions legible and evaluating intent-related behavior. It infers a latent belief state from responses to varied prompts and evaluates its predictive relationships with outputs. Highly capable models are behaviorally coherent enough for belief to predict their behavior well, while the framework also reveals measurement, framing, and within-instance dynamics.

  • Problem

    The field lacks systematic evidence for whether attributing beliefs to LLMs predicts behavior, despite the relevance of such abstractions to legibility, evaluation, and alignment.

  • Method

    The paper infers a latent belief variable from many prompted outputs and evaluates whether it predicts responses across decision problems, framings, and held-out cases.

  • Results

    Highly capable models show behaviorally coherent beliefs, with one latent state predicting outputs across 16 prompts at AUCs above 0.90; verbalized probabilities are relatively high-quality measurements.

  • Takeaways & Limitations

    Belief is a useful predictive abstraction for highly capable LLMs in the studied settings and provides an empirical foundation for assessing belief-based descriptions and evaluations.

  • Takeaways & Limitations

    The framework evaluates predictive attribution and does not resolve the philosophical question of what ultimately licenses cognitive terms such as belief or desire.

Abstract

from arXiv · show

There is significant uncertainty about whether abstractions like beliefs or desires usefully describe the behavior of large language models (LLMs). In addition to the inherent scientific interest of this question, these latent quantities are often invoked to explain the behavior of LLMs to users or to define and evaluate harmful behaviors which are relative to intent. Nevertheless, we currently lack a means to systematically test whether concepts like "belief" are well-applied to LLMs, and hence whether they are likely to be fruitful ingredients of attempts to align models with human interests. We propose an approach for empirically studying such questions, asking whether a single latent variable inferred from the LLMs' outputs -- interpreted as a degree of belief -- allows an observer to make interpretable predictions of how the LLMs' will respond to new prompts. We find that highly capable models are usefully described as holding beliefs and that, generally, the predictability of model outputs based on an inferred latent belief tracks overall trends in model capability. Building on these findings, we provide empirical strategies to study how beliefs in LLMs can be measured, the extent to which LLMs comply with instructed decision rules or payoffs, and how beliefs evolve within individual instances of an LLM over the course of reasoning.

Introduction

The paper develops an empirical framework for testing whether latent beliefs usefully predict LLM behavior. Across several decision settings, highly capable models show coherent, belief-like behavior, while the framework also supports measuring beliefs and tracking their evolution.

  • The field lacks a systematic way to test whether attributing beliefs to LLMs improves prediction of behavior in new circumstances.The paper focuses on mathematically precise decision problems involving an unknown binary state.
  • The framework infers a latent belief from responses to many prompts and tests whether that state predicts outputs across prompts and cases.If behavior is not internally coherent, reconstructing such a shared latent state should be impossible.
  • A single latent state predicts highly capable models’ behavior across 16 prompts with AUCs above 0.90 in healthcare, Werewolf, and scientific forecasting tasks.These settings cover medical screening, strategic interaction under imperfect information, and forecasting social science experiments.
  • Verbalized probability judgments are relatively high-quality measurements of latent beliefs, especially for highly capable models.
  • Less capable models can sometimes exhibit a shared latent variable across decision framings, although its numerical scale may not be instruction-following or mutually consistent.
  • Models’ beliefs evolve meaningfully during deliberation, making individual instances distinguishable from the base model in some circumstances.The framework measures concordance between beliefs inferred from independently branched instances after an initial chain-of-thought phase.

Methods

The paper models belief as a latent variable linking an LLM’s responses across instructions, then tests whether this variable predicts held-out actions across several decision-making domains. The framework imposes interpretable decision-theoretic restrictions and evaluates predictive performance using repeated outputs from each case.

  • Latent variable model: The model represents belief about a binary unknown y with a latent z in [0,1], where higher z indicates greater belief that y = 1.Responses to different instructions are modeled as related through this shared latent state.
  • Latent variable model: Decision-theoretic restrictions require z to summarize outcome-relevant information and choice probabilities to vary monotonically with latent belief.The model evaluates outcome sufficiency and enforces monotonicity through the functional forms for the output distributions.
  • Latent variable model: The main specification uses simple functional forms that make binary choices monotone in z and interpret them as stochastic choices around decision thresholds.For probability outputs, a precision parameter controls stochasticity and anchors the latent scale to a probability.
  • Latent variable model: Neither a two-dimensional latent variable nor nonlinear output relationships detectably improves predictive power, so the paper retains the simpler interpretable model.The extensions are explored in supplementary analyses.
  • Inference and evaluation: Inference estimates model parameters on training cases, infers each held-out case’s latent z from observed responses, and predicts withheld outputs by integrating over its posterior.This tests whether responses to some instructions reveal a latent belief that predicts responses to other instructions.
  • Domains and action spaces: The evaluation covers chronic-kidney-disease screening, Werewolf games, and social-science experiment forecasting, using 16 outputs for each of 500 cases per domain.Outputs include probability assessments, binary decisions, and parameterized decisions involving thresholds, utilities, or bets.

Results

A single inferred latent belief predicts diverse LLM outputs, with stronger models generalizing more coherently across actions, framings, domains, and reasoning instances. Elicited probabilities are informative belief measurements, while decision behavior shows capability-dependent instruction following and framing effects.

  • Prediction: Highly capable models achieve concordance indices of 0.8-0.95 and R2 values of 60-90% when predicting held-out actions from an inferred latent belief.Less capable models perform substantially worse, and model-level predictive performance follows overall capability trends.
  • Prediction: Zero-shot predictions across unseen framings, domains, or both reach AUCs close to 1 and pooled R2 values of approximately 0.6 to 0.8 for the most capable models.Less capable models lose substantially more accuracy in these out-of-setting evaluations.
  • Measuring the sufficiency of elicited probabilities: A single numerical probability report captures over 90% of jointly inferred latent-belief predictive performance for the best models.For weaker models, elicited probabilities provide a significantly worse summary of the latent state.
  • Inferred decision thresholds: Decision thresholds are closest to instructed values for the strongest model, whereas weaker models show systematic framing effects across payoff-identical prompts.Bet framing often produces a higher threshold for taking action a = 1, though the authors do not classify framing effects as irrational.
  • Inferred decision thresholds: Less capable models can still be described as having degrees of belief, but their beliefs need not correspond directly to probabilities followed by an explicit decision rule.More capable models are significantly closer to that probability-and-rule behavior.
  • Distinguishing models and instances: Evidence from a different reasoning prefix retains nontrivial predictive success, while same-instance evidence generally improves reciprocal prediction between probability reports and decisions.This suggests both shared beliefs across instances of a base model and additional belief variation associated with the particular reasoning prefix.

Discussion

The paper finds that highly capable LLMs can be usefully described as holding behaviorally coherent beliefs, while framing this as an empirical rather than philosophical resolution. This behavioral lens may support safety and interpretability work, but its scope and relationship to human cognition remain limited.

  • Discussion: Highly capable LLMs are sufficiently behaviorally coherent for an inferred latent belief state to predict their behavior well.The framework provides an empirical basis for assessing whether belief is a useful behavioral description without resolving what ultimately licenses cognitive terminology.
  • Discussion: Belief- and intention-based categories such as deception are useful descriptions for AI safety only when actions are predictable from elicitable beliefs.The paper connects this requirement to safety behaviors defined in terms of beliefs and intentions.
  • Discussion: A behavioral lens complements mechanistic interpretability by specifying the behavior whose internal underpinnings should be located.The authors do not claim that beliefs must correspond to a particular activation pattern.
  • Discussion: The framework evaluates beliefs in purely predictive terms and focuses on single-shot decision problems, leaving broader philosophical evidence and other tasks for future work.Its design tests semantic and decision-theoretic properties and zero-shot generalization, but does not establish that LLMs literally hold beliefs.
  • Discussion: Attributing beliefs can help understand and predict LLM behavior, but their implementation and generalization need not resemble human cognition.The paper therefore cautions against assuming human-like mechanisms for belief formation or response to evidence.

A.1 Data construction and output families

The appendix constructs three-domain datasets with 500 cases each and sixteen temperature-1 outputs per case. It models these outputs as conditionally related to a scalar latent state, using probability, binary, and ordinal response families.

  • A.1 Data construction and output families: 500 cases in each of three domains—CKD screening, Werewolf, and scientific forecasting—provide the primary analysis data.For every case and model, the study elicits one numerical probability, thirteen binary decisions, and two five-category ordinal judgments.
  • A.1 Data construction and output families: Sixteen outputs per case comprise probability reports, binary decisions, and ordinal judgments collected at temperature 1.The probability category includes numerical, Likert, and probability-range responses; binary outputs include domain and parameterized decisions.
  • A.1 Data construction and output families: Strict complete-case parsing retains only cases for which all sixteen outputs parse, with retention reported separately by model and domain.The percentages are calculated from the 500 collected cases.
  • A.1 Data construction and output families: Parameterized threshold, betting, and asymmetric-error prompts are matched so a risk-neutral expected-value or expected-loss minimizer would choose the same action.These families vary the decision framing while preserving an implied common action rule.
  • A.1 Data construction and output families: The scalar latent state has a fitted prior and output-specific conditional distributions, with the prior represented by a piecewise-linear log density in logit space.Seven equally spaced control points determine the learned prior density, which is normalized numerically.
  • A.1 Data construction and output families: Numerical probability reports anchor the latent state's orientation and scale, while binary and ordinal models impose monotonicity through positive slopes.The probability report uses a Beta observation model with conditional mean equal to the latent state; ordinal probabilities use proportional odds.

A.3 Fitting, posterior inference, and held-out-group prediction

The model parameters are fit by maximum marginal likelihood, and held-out groups of outputs are predicted from posterior beliefs inferred using only complementary observations. Evaluation tests both directions between probability reports and decisions.

  • A.3 Fitting, posterior inference, and held-out-group prediction: Parameters are estimated by maximum marginal likelihood, with one-dimensional integrals evaluated by 128-point Gauss–Legendre quadrature.Primary optimization uses Adam, validation early stopping, and a fixed random seed.
  • A.3 Fitting, posterior inference, and held-out-group prediction: For each test case, the posterior latent belief is inferred from observed outputs O while the held-out target group H is excluded from posterior inference.The target responses are used only for scoring, and the observation model is not refitted.
  • A.3 Fitting, posterior inference, and held-out-group prediction: The primary analysis predicts all decisions from probability reports and all probability reports from decisions.Concordance is computed separately for each target and macro-averaged, with ties handled by the stated scoring rules.

A.4 Flexible-response and two-dimensional sensitivity analyses

Two sensitivity analyses make the latent model more expressive by allowing nonlinear responses or two latent dimensions. Across all model–domain cells, both variants achieve nearly the same predictive performance as the original model.

  • A.4 Flexible-response and two-dimensional sensitivity analyses: The sensitivity analyses test whether more expressive latent specifications improve predictive performance.They vary response shape and latent dimensionality while preserving the core framework.
  • A.4 Flexible-response and two-dimensional sensitivity analyses: The nonlinear-response model replaces binary and ordinal linear scores with monotone piecewise-linear functions of logit(z).It retains the anchored Beta probability observation and one-dimensional latent prior, using seven fixed knots.
  • A.4 Flexible-response and two-dimensional sensitivity analyses: The dimensionality sensitivity uses a two-dimensional latent state with a jointly fitted prior represented on a 7×7 control-point grid.The probability report continues to provide the model's numerical orientation and scale.
  • A.4 Flexible-response and two-dimensional sensitivity analyses: Nearly identical predictive performance across both new latent models and the original model is observed in all 27 model–domain cells.The comparison covers three domains crossed with nine models and uses target-by-target single-output evaluations.

A.5 Reasoning-prefix branching experiments

The branching experiments vary reasoning prefixes and downstream responses to assess how shared reasoning affects probability reports and behavioral coherence across tasks.

  • Experimental design: 30 cases per domain were evaluated with Gemma 3 12B and DeepSeek V3.2 at temperature 1.The first branching experiment sampled five qualitative reasoning prefixes per case and five probability responses from each prefix, producing 750 responses per model and domain.
  • Experimental design: The behavior-branching experiment sampled 20 reasoning prefixes per case and replayed each exact prefix before eliciting all downstream task responses.The same sixteen outputs were used in every domain, with three probability reports separated from decision outputs for reciprocal prediction analyses.
  • Behavior branching: Same-prefix comparisons use evidence and targets from one reasoning prefix, whereas different-prefix comparisons use evidence from another prefix for the same case.The analysis required complete sixteen-response vectors and averaged concordance across evaluable targets over 1,000 different-prefix draws.
  • Model sensitivity: Figure 7 compares held-out single-output prediction under one-dimensional linear, one-dimensional monotone-spline, and two-dimensional linear latent models.This target-by-target sensitivity analysis differs from the reciprocal group holdouts used in the primary analysis.
  • Prompt domains: The domain-specific prompts cover chronic-kidney-disease screening, Werewolf inference, and social-science experiment forecasting.The prompt sets elicit probability or likelihood reports for each domain’s underlying unknown.

B Held out framing and domain generalization with probability report inputs

Using only three probability or likelihood outputs to infer the latent belief produces qualitatively similar held-out decision predictions, with lower out-of-domain R2 when Werewolf is the target domain.

  • Domain generalization: Qualitatively similar held-out decision-prediction results arise when latent belief inference uses only three probability or likelihood outputs.The main difference is lower out-of-domain R2 when Werewolf is the target domain.

C Probability sufficiency under pooled R2

The pooled-R2 analysis compares predictions based on all other outputs with predictions based only on the numerical probability response across fifteen non-probability targets.

  • Analysis setup: Figure 9 evaluates both prediction series on all fifteen outputs other than the numerical probability report.The same fitted latent-variable model is used throughout the comparison.
  • Analysis setup: The gray series conditions on all other outputs, whereas the blue series conditions only on the numerical probability response.For each target, the corresponding latent posterior is integrated through the fitted observation model to generate predictions.
  • Results: The numerical probability alone retains most of the pooled predictive performance for the strongest models, while weaker models show a larger gap, especially in scientific forecasting.Pooled R2 aggregates squared errors and constant-predictor errors across the fifteen targets, using each target’s test-set mean as its constant predictor.

D Outcome sufficiency of inferred latent beliefs

The paper tests whether an inferred latent belief is sufficient to explain decisions and outcomes across healthcare, Werewolf, and scientific-forecasting settings.

  • Outcome sufficiency: The analysis tests whether decisions retain residual information about the true outcome after conditioning on the inferred latent belief.The latent state is the all-response posterior mean, and the outcome-sufficiency statistic measures outcome-decision dependence conditional on that state.
  • Outcome domains: The outcome domains include CKD status, Werewolf identity, and whether a social-science experiment has a positive effect.The corresponding prompt sets ask for probability reports, screening decisions, game-role probabilities, and comparative experimental predictions.
  • Outcome sufficiency: L1,j(Z) equals the expected decision-probability-weighted absolute difference between outcome probabilities under the two decisions, equivalently expected conditional covariance magnitude.The statistic is zero exactly when conditional independence holds almost surely for binary outcomes and decisions.
  • Estimation: The statistic is normalized by pooled test-row outcome prevalence and averaged equally over evaluable decision prompts.The reported prevalences are 0.182, 0.482, and 0.395 for the three domains.
  • Estimation: 15 of 351 model-domain-prompt cells are excluded because they have fewer than six observations in one decision class.The remaining estimates use Gaussian-kernel local-linear regressions, four-fold cross-validation for bandwidths, and joint case-bootstrap confidence intervals.
  • Results: Highly capable models have below 5% relative mean violations of the outcome-sufficiency condition across nine models and three domains.Figure 10 reports normalized L1 violations with jointly bootstrapped 95% percentile intervals.

E.3.2 experiment priority no threshold

Researchers must prioritize one of two conditions for a new experiment with equal cost, sample size, and measurement quality. The decision should favor the condition expected to produce the largest positive effect on the stated outcome.

  • The experiment carries forward only one of two conditions for testing.
  • Both conditions receive the same cost, sample size, and measurement quality in the proposed experiment.
  • The prioritization goal is to select the treatment expected to produce the largest positive effect on the stated outcome.
  • The response must provide reasoning first and then output exactly one listed multiple-choice answer.
Loading 2609.07943v1…