PaperlayerView exact revision on arXiv

Source-linked AI summary

Bayesian Fine-tuning Yields Language Models that are as Bayesian as their Beliefs Allow

Polina Tsvilodub, Andreas Waldis, Linlu Qiu, Tal Linzen, Michael Franke

arXiv:2610.00679v1cs.CL

TL;DR

Behavioral success alone cannot reveal whether an LM represents Bayesian beliefs or uses them to choose. The paper tests Bayesian- and oracle-supervised LMs across behavior, representations, and computations, finding that the Bayesian-trained model's policy is as Bayesian as its beliefs allow.

  • Problem

    Behavioral evaluation cannot distinguish errors in a model's beliefs from errors in how it transforms beliefs into recommendations.

    Source: §8 · p. 9

  • Method

    The paper tests increasingly demanding requirements for Bayesian decision making through behavioral evaluation, representation probing, and causal interventions on a flight recommendation task.

    Source: §1 · p. 1, §2 · p. 3

  • Results

    Across R1–R4, BAYESLM is more consistent with BAYESASSIST than ORACLELM, and its policy is as Bayesian as its beliefs allow.

    Source: §1 · p. 2, §8 · p. 9

  • Takeaways & Limitations

    Bayes-consistent uncertainty supervision yields better reasoning under uncertainty than standard tuning on oracle-correct answers.

    Source: §8 · p. 9

    Language models may only approximate posteriors, and how they form beliefs and turn them into choices in a forward pass remains open.

    Source: §8 · p. 9

Abstract

from arXiv · show

Language models (LMs) are increasingly used for tasks that require reasoning about hidden variables from a few observations, for which Bayesian inference is the normatively correct solution. While supervised fine-tuning of an LM on the outputs of an optimal $\textit{Bayesian}$ model leads to near-Bayesian behavior, standard supervised fine-tuning (SFT) on the true answers for the task falls short of it. But behavior alone does not tell us $\textit{why}$ tuning on a $\textit{Bayesian}$ or an $\textit{oracle}$ (true answers) signal differs: whether the resulting LM represents Bayesian beliefs, acts on them, or turns them into a choice the way Bayes' rule does. To compare them, we formulate increasingly demanding requirements for an LM to count as a Bayesian decision maker, spanning its behavior, representations, and computations, and test them on a flight recommendation task. The Bayes-trained LM acts Bayesian, encodes quantities of Bayes' rule in its middle layers, and uses the encoded belief for the recommendation to a certain extent. The oracle-trained LM differs from it both in the beliefs it holds and whether it reads beliefs out into recommendations. Exchanging beliefs between the LMs transfers a part of the Bayesian advantage. Bayes fine-tuning thus installs usable Bayesian beliefs in an LM for reasoning under uncertainty in a way standard SFT on oracle answers cannot, highlighting the advantage of nuanced supervision.

1 INTRODUCTION

The paper asks whether language models that behave like Bayesian decision makers also represent and use Bayesian beliefs as their decisions are formed. It evaluates increasingly demanding layers of Bayesian decision making and finds that Bayesian fine-tuning yields more consistent behavior, representations, and computations than oracle fine-tuning.

1 INTRODUCTION

  • Behavioral evaluations alone cannot reveal whether a wrong recommendation comes from an incorrect belief or from how a belief becomes a choice.The mappings from beliefs to policies and policies to choices are many-to-one, so Bayesian-looking choices need not imply deeper Bayesian computation.Source: §1 · p. 1
  • The paper tests four increasingly demanding requirements spanning Bayesian behavior, represented beliefs, causal use of beliefs, and belief-based choice policies.The framework combines behavioral evaluation, representation probing, and causal interventions.Source: §1 · p. 1, Figure 1 · p. 2
  • The comparison isolates training signal by testing a pretrained LM, a model fine-tuned on Bayesian recommendations, and one fine-tuned on true choices.The two fine-tuned models share the task, starting model, and procedure, differing only in whether training uses Bayesian recommendations or oracle answers.Source: §1 · p. 2
  • BAYESLM is more Bayes-consistent than ORACLELM across the four requirements, but its use of beliefs remains partial.BAYESLM’s beliefs improve ORACLELM’s Bayes accuracy when patched across, though not to BAYESLM’s level.Source: §1 · p. 2

2 EXPERIMENTAL SET-UP

The experiments use a five-round flight recommendation task in which an assistant infers a user’s fixed preferences from observed choices. Three related language models are evaluated against a Bayesian assistant across choice, policy, and belief measures and test sets designed to examine Bayes consistency.

2 EXPERIMENTAL SET-UP

  • The task asks an assistant to recommend among three flights while inferring the user’s fixed preferences from choices observed over five rounds.Each flight is represented by departure time, duration, number of stops, and price; the user’s preference weights are hidden from the assistant.Source: §2 · p. 2
  • The comparison uses a pretrained STARTLM, BAYESLM fine-tuned on Bayesian recommendations, and ORACLELM fine-tuned on true user choices.Comparing BAYESLM and ORACLELM isolates the Bayesian versus oracle training signal.Source: §2 · p. 3
  • The analysis separates the recommended action (L1), the choice policy (L2), and beliefs about preferences (L3), which are decoded from LM representations using linear probes.For LMs, the policy is the probability assigned to each flight-option token; the belief measures include decoded prior, likelihood, and posterior.Source: §2 · p. 3
  • The study measures Bayes correctness at L1 and KL-divergence from the assistant’s policy and beliefs at L2 and L3.Expected utility averages option utility over the preference belief, and the Bayesian choice policy applies softmax to expected utilities with rationality parameter β.Source: §2 · p. 3
  • Figure 2 compares Bayes accuracy for the three models across four test sets, while Figure 3 plots model policies and averages against BAYESASSIST rankings.Figure 2 marks chance agreement at 1/3; Figure 3 uses color to distinguish Bayes-correct from incorrect policies.Source: Figure 2 · p. 4, Figure 3 · p. 4
  • Across four test sets, BAYESLM performance on original and shuffled datasets confirms invariance to evidence order; shuffling reorders evidence within the first four rounds, while the original set is held out from fine-tuning.Source: §2 · pp. 3–4

3 ACTING BAYESIAN: BEHAVIORAL ANALYSIS

BAYESLM is more Bayes-accurate than ORACLELM on the best action, while both show consistency across test sets. BAYESLM also has a more Bayes-coherent choice policy and fails in a way that better tracks the assistant’s uncertainty.

3 ACTING BAYESIAN: BEHAVIORAL ANALYSIS

  • BAYESLM achieves Bayes accuracy of 0.84 versus ORACLELM’s 0.69 on the original set, and both are significantly less accurate on the high-noise set.Their accuracy does not significantly differ between original, shuffled, and low-noise sets; STARTLM is near chance across all sets.Source: §3 · p. 4
  • On Bayes-correct predictions, policy divergence is 0.10 for BAYESLM and 0.26 for ORACLELM; on incorrect predictions, β = 4 shifts it to 0.38 and 0.90, respectively.The corresponding incorrect-prediction divergences before scaling are 0.42 for BAYESLM and 0.66 for ORACLELM.Source: §3 · p. 4
  • When wrong, BAYESLM assigns 52% probability to the assistant’s second choice and 17% to its least-probable choice, while ORACLELM fails more uniformly at 48% and 26%.Both models fail more often when the assistant’s policy is more uniform, with a stronger dependence for BAYESLM.Source: §3 · pp. 4–5
  • BAYESLM’s policy entropy correlates more strongly with the assistant’s than ORACLELM’s does (Pearson r = 0.61 versus r = 0.30), indicating closer tracking of its uncertainty.The section concludes that both models meet consistency criteria across test sets, but BAYESLM’s lower policy divergence and more Bayesian failures drive its accuracy advantage.Source: §3 · p. 5

4 REPRESENTING BELIEFS: PROBING ANALYSIS

Both fine-tuned models encode decodable Bayes-rule quantities in middle layers, but BAYESLM’s representations are more Bayes-consistent than ORACLELM’s. Decoded policy quality tracks behavioral policy divergence, while probing alone cannot establish whether beliefs causally shape recommendations.

4 REPRESENTING BELIEFS: PROBING ANALYSIS

  • BAYESLM’s best-layer divergences are 0.25 for the prior and 0.35 for the posterior, with likelihood-update correlation r = 0.59; ORACLELM’s corresponding values are 0.40, 0.45, and 0.53.The quantities become decodable from layers 15–25 in fine-tuned models, while STARTLM shows none of this pattern.Source: §4 · pp. 5–6
  • Decoded-policy Bayes accuracy rises to 0.87 for BAYESLM and 0.71 for ORACLELM, compared with 0.37 for STARTLM.BAYESLM’s decoded choice policy approaches the assistant’s by layer 42, whereas ORACLELM stops short; the policies emerge during the same middle-layer window as the beliefs.Source: §4 · p. 6, Figure 5 · p. 6
  • Actual policy divergence increases with decoded-policy divergence (β = 0.19, p < 10^-4), and ORACLELM’s divergence is worse than BAYESLM’s (β = −0.11, p < 10^-4).Decoded-belief divergence has no significant effect on actual policy divergence (p > 0.5).Source: §4 · p. 6
  • Probing cannot establish whether beliefs causally shape predictions: high-entropy policies could reflect incorrect beliefs or close expected utilities under correct beliefs.The authors therefore turn to direct interventions on beliefs.Source: §4 · p. 6

5 ACTING ON BAYESIAN BELIEFS: INTERVENTION ANALYSIS

The intervention tests whether recommendations change in the direction implied by an injected prior. It edits the belief representation at the feedback token and across subsequent tokens and layers.

5 ACTING ON BAYESIAN BELIEFS: INTERVENTION ANALYSIS

  • The test injects a specific prior and checks whether the recommendation changes to the one implied by the resulting Bayesian-assistant policy.The intervention addresses whether the model uses its encoded belief, rather than merely carrying one in hidden states.Source: §5 · p. 6
  • The expected test is whether recommendations change when the Bayesian assistant’s policy changes under the injected belief, and remain unchanged otherwise.The passage describes the intervention criterion, not its outcome.Source: §5 · p. 6
  • For each feature and positive or negative value direction, the intervention builds a prototype hidden state by averaging 500 representative training samples.It patches the prototype into the residual stream at the round-four feedback token and next 32 tokens, across eight subsequent layers with α = 0.5.Source: §5 · p. 6

4. We use

Belief interventions steer BAYESLM’s recommendations in a limited layer range, while ORACLELM remains near chance. Attention masks also reduce BAYESLM’s accuracy, with the larger drop under the mask permitting sequential flow.

4. We use

  • Single-position, single-layer edits produced no behavioral effect, so the intervention spans tokens and layers; balanced accuracy averages change and no-change samples equally.The prototype is patched across eight subsequent layers, and results also hold across test sets.Source: §4 · p. 7
  • BAYESLM reaches balanced accuracy of 0.77 at layers 18–25, while ORACLELM’s 0.52 remains near chance; the effect peaks around layers 15–22 and falls to chance in upper layers.A random-vector control barely changes answers, supporting the conclusion that the injected belief drives BAYESLM’s recommendation changes.Source: §4 · p. 7
  • BAYESLM’s Bayes accuracy drops from 0.84 to 0.68 under the one-swoop mask and to 0.56 under the sequential-flow mask.Both masks reduce accuracy; the authors interpret the pattern as suggesting a distributed, more one-swoop-like implementation.Source: §4 · p. 7
  • BAYESLM partially uses encoded beliefs for recommendations, whereas ORACLELM’s predictions are less sensitive to belief interventions.The interventions do not identify which computations form the recommendation.Source: §4 · p. 7

6 FORMING A BAYESIAN POLICY FROM BELIEFS

The section tests how LMs turn represented beliefs into recommendations, using cross-model belief patching and an explicit expected-utility policy. BAYESLM’s belief read-out is closer to the normative policy than ORACLELM’s, but belief quality also matters.

6 FORMING A BAYESIAN POLICY FROM BELIEFS

  • BAYESLM beliefs raise ORACLELM balanced accuracy from 0.52 to 0.61, while ORACLELM beliefs lower BAYESLM accuracy from 0.77 to 0.61.The remaining gap after patching BAYESLM beliefs into ORACLELM indicates that belief content alone does not explain the models’ difference.Source: §6 · pp. 7–8, Figure 6 · p. 8
  • Neuro-symbolic accuracy is nearly identical to BAYESLM’s own (+0.02), while the ORACLELM-based neuro-symbolic model improves over ORACLELM by 0.09.Both neuro-symbolic policies diverge little from the Bayesian policy (0.009 for BAYESLM, 0.017 for ORACLELM), but more from their own models’ policies (0.106 and 0.354).Source: Table 1 · p. 8, §6 · p. 8
  • The explicit policy plugs probed beliefs into posterior-mean and expected-utility calculations to approximate how beliefs map to recommendations.The comparison uses each model’s best belief-probing hyperparameter setting.Source: §6 · pp. 7–8
  • The authors conclude that Bayesian fine-tuning gives an LM a more consistent probabilistic representation and a read-out that partly uses it, unlike standard oracle fine-tuning.They suggest behavioral gaps also reflect noise in policy components beyond beliefs, especially for ORACLELM.Source: §6 · p. 8

7 RELATED WORK

Prior work examines Bayesian inference and uncertainty in LMs through behavior, internal representations, and external scaffolding. This paper combines behavioral and internal analyses with a task requiring both latent-variable inference and decision-making.

7 RELATED WORK

8 DISCUSSION

Across four requirements, BAYESLM is more consistent with BAYESASSIST than ORACLELM, but it is not a fully genuine Bayesian decision maker. The discussion draws implications for supervision and evaluation while noting limits of the model and study.

8 DISCUSSION

  • Bayesian supervision yields better reasoning under uncertainty than standard oracle-answer tuning, because BAYESLM’s beliefs are more task-relevant and patching them improves ORACLELM’s Bayes accuracy.The explicit assistant can recover Bayes accuracy from decoded beliefs of either fine-tuned model.Source: §8 · p. 9
  • Across R1–R4, BAYESLM acts Bayesian and represents Bayes-rule quantities, uses decoded beliefs causally only partially, and maps beliefs to a choice policy as well as those beliefs allow.The authors locate BAYESLM’s remaining gap to BAYESASSIST in its beliefs rather than their read-out; its advantage over ORACLELM holds across backbones.Source: Table 2 · p. 9, §8 · p. 9
  • Behavioral evaluation alone cannot distinguish a wrong belief from a failure to transform a correct belief into a recommendation.The authors propose four operationalized requirements to examine these stages in LM decision-making.Source: §8 · p. 9
  • A one-pass LM may only approximate the posterior, can fail on high-entropy policies, and represents beliefs across layers and tokens rather than at one position.How beliefs form and become choices in the forward pass remains open.Source: §8 · p. 9

9 CONCLUSION

The authors apply four increasingly demanding tests of Bayesian decision-making to a flight recommendation task and conclude that Bayesian fine-tuning makes an LM as Bayesian as its beliefs allow. They also identify future extensions and disclose their use of generative AI tools.

9 CONCLUSION

  • The conclusion applies four requirements spanning behavior, representations, and computation to the flight recommendation task of Qiu et al. (2026).Source: §9 · pp. 9–10
  • Bayesian fine-tuning makes an LM as Bayesian as its beliefs allow, but not a fully genuine Bayesian decision maker.The authors report that beliefs are used causally across layers and tokens and turned into a policy as well as their quality allows.Source: §9 · p. 10
  • Future work should test other tasks with normative Bayesian solutions and examine how beliefs are updated and transformed into policies across options.Source: §9 · p. 10

AI USE STATEMENT

  • Generative AI tools were used for coding, analyses, visualizations, interpretation feedback, and drafting parts of the text; the authors reviewed the work and take responsibility for the final content.Source: AI USE STATEMENT · p. 10

A LIMITATIONS

Behavioral evaluations compare LM recommendations and policies with the Bayesian assistant; the accompanying sections detail regression analyses and limitations on the task, models, probes, and interventions.

A LIMITATIONS

  • Evidence is limited to one highly structured recommendation task, with assumptions including independent reward features; generalization to less structured or analytically unsolvable tasks remains open.The assistant’s training recommendations use softmax-rational decisions based on Bayesian inference over latent preferences, and models were trained on sequential, essentially in-context-learning data.Source: A LIMITATIONS · p. 13
  • Results focus on Gemma-2-9B-Instruct; other tested backbones show qualitatively similar patterns, but architectural effects and scaling remain unexamined.Qwen-2.5-7B showed lower Bayesian consistency.Source: A LIMITATIONS · p. 14
  • Linear probes provide only a lower bound on represented information, so belief divergence may partly reflect linear readout limits rather than the models’ representations.Controls indicate the probes are selective and recover much less from STARTLM, while nonlinear readouts could recover more.Source: A LIMITATIONS · p. 14
  • Interventions replace only the prior at a fixed BAYESLM-selected window; they show belief use but not how beliefs combine into expected utility across options.The transformation to a choice policy is tested externally rather than through intervention on the model.Source: A LIMITATIONS · p. 14

B STATISTICAL ANALYSES

  • Supplementary regressions analyze Bayes accuracy, policy divergence, and decoded-belief consistency using z-scored continuous predictors and Bonferroni-Holm correction.A model including both prior and decoded-policy divergences produced the same results.Source: B STATISTICAL ANALYSES · p. 14, Table 3 · p. 14, Table 4 · p. 14

C BEHAVIORAL EVALUATIONS

  • BAYESASSIST’s fifth-round Oracle accuracy is 0.811 on the original and shuffled sets, 0.683 on low noise, and 0.420 on high noise.Oracle accuracy measures agreement with the user’s true preference.Source: C BEHAVIORAL EVALUATIONS · p. 15
  • BAYESLM’s choice-policy entropy correlates with the Bayesian policy’s entropy at r = 0.61, compared with r = 0.30 for ORACLELM.For both models, disagreements cluster where the Bayesian assistant is most uncertain, but are more dispersed for ORACLELM.Source: Figure 9 · p. 16, C BEHAVIORAL EVALUATIONS · p. 16
  • At β = 4, policy divergence on Bayes-correct samples is 0.10 for BAYESLM and 0.26 for ORACLELM; on incorrect samples it falls for BAYESLM but rises for ORACLELM.Both models’ policies are more decisive than the assistant’s at β = 1: their chosen options have probabilities 0.74 and 0.72, versus 0.46 for BAYESASSIST.Source: C BEHAVIORAL EVALUATIONS · p. 16, Figure 8 · p. 16

D PROBING EXPERIMENTS

Probes find Bayes-related beliefs in the fine-tuned models’ middle-to-late layers, with BAYESLM generally closer to the Bayesian targets. Results across positions and noisy sets qualify how those decoded beliefs and choices should be interpreted.

D PROBING EXPERIMENTS

  • Decoded BAYESLM policies come closer to the Bayesian policy than ORACLELM’s, especially on instances with incorrect recommendations.The figure separates cases where the LM agrees with BAYESASSIST from those where it differs.Source: D PROBING EXPERIMENTS · p. 17
  • Choice representations develop at last-prompt and digit tokens but remain near chance at the feedback token, while belief representations are decodable across positions and depths.The passage treats the feedback-token pattern as a sanity check that choice is not committed before the new options appear.Source: D PROBING EXPERIMENTS · p. 17
  • Belief divergence is unchanged by evidence reordering, but decoded beliefs are lower-entropy than BAYESASSIST’s under noise; scaling BAYESLM beliefs by β ≈0.2 reduces divergence.For BAYESLM, divergence after scaling is 0.083 on low noise and 0.059 on high noise; the authors suggest noise-free probe training may contribute to the default mismatch.Source: D PROBING EXPERIMENTS · p. 17, Figure 11 · p. 18
  • Across all four flight features, BAYESLM’s prior-probe loss ends at about KL ≈0.25, below ORACLELM’s about 0.40; STARTLM stays near the uniform probe.The fine-tuned models improve between layers 17 and 25, and BAYESLM ends lowest on every feature.Source: D PROBING EXPERIMENTS · pp. 17–18

E ADDITIONAL RESULTS ON BELIEF INTERVENTION

These additional analyses test how encoded beliefs affect recommendations, how beliefs may be formed, and whether the findings hold across backbones. BAYESLM generally shows stronger Bayesian behavior than ORACLELM, while intervention effects vary by method and backbone.

E ADDITIONAL RESULTS ON BELIEF INTERVENTION

  • After the feedback token, CAA steering reaches at most 0.59 balanced accuracy versus 0.77 for prototype patching; digit-token steering never exceeds chance.Both methods work only in layers where the belief is decodable, but only prototype patching replaces it strongly enough to change most recommendations.Source: E ADDITIONAL RESULTS ON BELIEF… · p. 20, Figure 15 · p. 21
  • BAYESLM prior patching reaches 0.75–0.87 balanced accuracy across feature-direction settings, so no single feature drives the average.The selected prototype-patching configuration uses α = 0.5 across 32 tokens and 8 layers; evaluation covers 624 held-out instances.Source: E ADDITIONAL RESULTS ON BELIEF… · pp. 18, 19, 21, Table 6 · p. 20, Table 8 · p. 22
  • For BAYESLM, prior patching reaches 0.778 balanced accuracy on the shuffled set and 0.795 and 0.805 under low and high noise.The noisy sets score slightly higher because BAYESASSIST changes its recommendation more often, increasing the expected-change cases.Source: E ADDITIONAL RESULTS ON BELIEF… · p. 21, Table 8 · p. 22
  • Larger belief updates prompt more recommendation changes: 0.880 in the largest-KL quartile versus 0.664 in the smallest, with 0.83–0.85 accuracy when recommendations change.Intervention failures arise because the model does not change its recommendation, rather than changing it to the wrong option.Source: Figure 16 · p. 22, E ADDITIONAL RESULTS ON BELIEF… · p. 22

F MASKING METHODS

  • On BAYESLM, one-swoop masking reduces Bayes accuracy by 16%, while sequential masking reduces it by 28%.Randomly blocking the same number of tokens also lowers accuracy, to 28–38% depending on conversation length, supporting the relevance of the masked information flow.Source: F MASKING METHODS · p. 23, Figure 18 · p. 24

G NEURO-SYMBOLIC BEHAVIORAL POLICY

H RESULTS ACROSS BACKBONE MODELS

  • Across backbones, BAYESLM is more Bayes-accurate than ORACLELM; its prior is more strongly decodable, while prior-intervention effects vary by backbone.BAYESLM is invariant to evidence order and behaves as expected under noise; both masks reduce Bayes accuracy, with sequential masking reducing it more. The prior-intervention effect is largest for Gemma-2-9B, present for Llama-3.1-8B, and near chance for Qwen-2.5-7B.Source: H RESULTS ACROSS BACKBONE MODELS · p. 24, Figure 21 · p. 25, Figure 22 · p. 25
Loading 2610.00679v1…