PaperlayerView exact revision on arXiv

Source-linked AI summary

From Prompts to Posteriors: Can LLM Agents Perform Bayesian Inference in Context?

Linus Filbry, Luckeciano C. Melo, Yarin Gal

arXiv:2610.29331v1cs.AI

TL;DR

LLMs are increasingly entering scientific workflows, where belief updates must track accumulating evidence, but systems often rely on heuristics rather than probabilistic beliefs. This paper evaluates belief updating across scientific-discovery settings and finds that reasoning improves performance, while tool use supports approximation of intractable posteriors.

  • Problem

    Scientific agents need to revise beliefs as evidence accumulates, but current systems often use task-specific heuristics instead of probabilistic belief updates.

    Source: §1 · p. 1

  • Method

    The evaluation compares LLM belief updates with Bayesian references in sequential discovery settings, including a procedurally generated ProteinWorld environment.

    Source: §4.2 · p. 5, §5.1 · p. 5

  • Results

    Without reasoning, LLMs fail to maintain probabilistic consistency or reliably compute simple posteriors, while reasoning improves inference and tools enable effective approximation of analytically intractable posteriors.

    Source: §6 · pp. 8–9

  • Takeaways & Limitations

    Modern reasoning models can support belief updates closer to Bayesian-oracle performance, and code execution can extend them to intractable posteriors.

    Source: §6 · p. 9

    The tests do not establish universal within-dialogue coherence; their evidence supports treating outputs as valid probability assignments in the tested scientific-discovery setting.

    Source: §6 · p. 7

Abstract

from arXiv · show

Large language models (LLMs) are increasingly applied across the scientific discovery pipeline, from synthesizing literature to generating hypotheses to informing experimental design. A unifying principle behind these tasks is the need to continuously update beliefs as evidence accumulates, for which Bayesian inference provides a natural framework. However, existing work assumes that LLMs cannot carry out principled Bayesian belief updating in context, instead relying on simplifications and heuristics. To challenge this assumption, we present a principled evaluation of Bayesian reasoning capabilities in LLMs, analyzing both the model's internal consistency as well as its ability to recover an externally determined posterior. Specifically, we build an evaluation framework spanning qualitative epistemological examples to probe probabilistic coherence, quantitative benchmarks of Bayesian inference capabilities, and analyses of the impact of non-Bayesian updates on downstream task performance. Across different scenarios, we find that LLMs generally violate Bayes' rule when updating their beliefs in context. However, when equipped with modern reasoning abilities and code execution capabilities, their performance drastically improves, enabling even the close approximation of analytically intractable posteriors. Our evaluation shows that current LLMs with access to appropriate tools could already be used to make complex belief updates in scientific discovery, enabling rapid acceleration of the efficiency and scale of AI-powered scientific discovery.

1 INTRODUCTION

As LLMs move into scientific workflows, they must revise beliefs as evidence accumulates, yet current AI-scientist systems often use task-specific heuristics instead of probabilistic beliefs. This paper evaluates whether LLMs can perform Bayesian inference in context and how reasoning and tools affect that ability.

1 INTRODUCTION

  • LLMs are increasingly used for scientific tasks ranging from literature synthesis and hypothesis generation to experiment design and result interpretation.This trend motivates the vision of autonomous or semiautonomous “AI scientists.”Source: §1 · p. 1
  • AI-scientist systems often score hypotheses with task-specific heuristics rather than maintaining and updating probabilistic beliefs.Bayesian updating offers a framework for incorporating prior knowledge, while non-Bayesian updates risk incoherence and bias.Source: §1 · p. 1
  • The paper evaluates probabilistic coherence, posterior fidelity to a Bayesian oracle, and downstream experimental-design performance.The framework tests internal consistency and recovery of externally determined posteriors.Source: §1 · p. 2, Figure 1 · p. 2
  • The paper contributes ProteinWorld, a configurable environment for benchmarking iterative Bayesian inference in simulated scientific discovery.Its evaluation covers internal consistency and recovery of externally determined posteriors.Source: §1 · p. 2

2 RELATED WORK

Prior work studies Bayesian inference in purpose-trained and general-purpose LLMs, coherence and calibration, oracle-based update evaluation, and methods for enforcing better updates. This paper evaluates explicit belief distributions and tests reasoning and code execution as interventions.

Bayesian inference in LLMs

  • Purpose-trained Transformers can approximate Bayesian inference in context, while general-purpose LLM work has examined implicit inference and documented violations of exchangeability and position sensitivity.Some findings characterize general-purpose models as Bayesian only in expectation over context orderings.Source: §2 · p. 2

Probabilistic coherence and calibration

  • Prior coherence studies include two-event toy settings, while calibration scores stated confidence against outcomes as an external, orthogonal criterion.This work tests coherence to isolate internal-logic violations and compares cross-chat with single-dialogue elicitation.Source: §2 · pp. 2–3

Measuring belief updates against a Bayesian oracle

  • Earlier update evaluations use exact references for specified generative processes or proxies and self-elicited quantities when no ground-truth model is available.ProteinWorld adds procedural worlds, continuous latent attributes with engineered analytical intractability, and a downstream deviation-cost metric.Source: §2 · p. 3

Improving belief updates

  • Previous efforts have used strict harnesses or finetuning to force Bayesian updates; this work instead tests the premise and treats reasoning and code execution as interventions.Its protocol builds directly on the 20Q approach of Choudhury et al. (2026).Source: §2 · p. 3

3 BACKGROUND

Bayesian inference updates a prior over hypotheses as sequential evidence arrives, but practical posteriors can be analytically intractable. The paper also motivates Dutch Books as coherence checks and expected information gain as a basis for adaptive experiment choice.

Bayesian inference

  • Bayes’ rule combines a prior with experiment-conditioned likelihoods to produce a posterior, which becomes the prior for the next update.The sequential form assumes outcomes are conditionally independent given the hypothesis and experiment.Source: §3 · p. 3
  • Finding the posterior can be analytically intractable when the likelihood or hypothesis space makes the normalizing integral complex.Source: §3 · p. 3

Dutch Books

  • A Dutch Book is a set of bets based on stated probabilities that guarantees a loss, making it a test for incoherent beliefs.The construction uses the fair-price assumption for buying or selling event contracts.Source: §3 · pp. 3–4

Experimental Design

  • Expected Information Gain selects experiments by expected information about hypotheses, and adaptive design repeats selection, observation, and belief updating.Because future experiment choices use current beliefs, update quality affects long-term discovery performance.Source: §3 · p. 4

4 METHODOLOGY

The methodology evaluates in-context Bayesian inference across coherence, sequential posterior fidelity, and downstream experimental-design performance. It compares LLM beliefs with Bayesian references and examines the practical cost of belief-update deviations.

4.1 PROBABILISTIC COHERENCE

  • Coherence is tested by eliciting related ForecastBench event probabilities and checking extensionality, normalization, and monotonicity violations.The same events are queried independently across chats and jointly within one chat, testing cross-interaction and within-context coherence.Source: §4.1 · p. 4

4.2 SEQUENTIAL BELIEF UPDATING

  • Sequential updating uses a shared externally fixed prior and compares LLM-chosen experiments with passive and active Bayesian oracle trajectories.The passive oracle applies Bayes’ rule to the LLM trajectory’s experiments and results; the active oracle also chooses experiments from its own beliefs.Source: §4.2 · pp. 4–5
  • Posterior fidelity is measured with forward KL divergence between LLM and passive-oracle beliefs as evidence accumulates.The forward direction discourages early discarding of hypotheses still supported by the oracle; LLM probabilities are clipped below at 10^-6 and renormalized.Source: §4.2 · p. 5
  • Downstream performance compares LLM and active-oracle trajectories by the probability mass assigned to the environment’s true hypothesis.The posterior-mass gap is designed to be less affected by environment difficulty; independently generated histories make it an end-to-end performance difference.Source: §4.2 · p. 5

5 EXPERIMENTAL SETUP

The setup pairs two complementary sequential belief-updating environments with protocols for testing probabilistic coherence and posterior recovery. 20Q uses an open-ended domain, while ProteinWorld provides controlled finite worlds with specified distributions.

5.1 ENVIRONMENTS FOR SEQUENTIAL BELIEF UPDATING

  • 20Q uses unrestricted Yes/No questions over all existing animals, creating arbitrarily large, unstructured hypothesis and experiment spaces with binary outcomes.The setup makes no restriction on question type or animal guesses.Source: §5.1 · p. 5
  • ProteinWorld procedurally generates fictional, out-of-distribution protein worlds with finite classes and experiments, latent attributes, and potentially dependent attributes.The evaluator controls the number of classes, attribute dependencies, and observation-model complexity.Source: §5.1 · pp. 5–6
  • Evaluator-controlled distributions support simple instances with closed-form likelihoods and hard instances requiring probabilistic approximation, such as Monte Carlo simulation.Source: §5.1 · p. 6
  • For 20Q, the model evaluates hypothesis compatibility with the history, then normalizes probabilities over a dynamically filtered and extended subset with a uniform prior.The subset handles the infinite hypothesis space and is a necessary deviation from strict Bayesian updating.Source: §5.1 · p. 6
  • For ProteinWorld, the model receives the world’s exact distributions, observed history, and explicit prior, then outputs posterior probabilities across the finite hypothesis set.Source: §5.1 · p. 6
  • 20Q tests internal consistency against Bayesian updates derived from the LLM’s own marginal likelihood assessments; ProteinWorld tests recovery of an externally specified posterior.ProteinWorld likelihoods are calculated analytically for simple instances and with a Monte Carlo estimator for hard instances.Source: §5.1 · p. 6

5.2 MODELS AND EVALUATION PROTOCOL

  • 10 ForecastBench scenarios per violation type elicit Dutch Books on Gemini-2.5-Pro, Gemma-4-31B-it, and Qwen-2.5-32B-Instruct.The selection spans model sizes, release years, and reasoning and non-reasoning models.Source: §5.2 · p. 6
  • 40 20Q animals are tested for T=20; ProteinWorld uses 40 runs across five worlds at T=6, with five models, selected reasoning runs, and Gemini-2.5-Pro code execution on hard instances.Source: §5.2 · p. 6

6 RESULTS & DISCUSSION

LLMs without reasoning struggle to update beliefs accurately, while reasoning improves performance on simpler settings and code execution extends it to intractable posteriors. These gains remain bounded by the evaluated settings, and modern reasoning models may no longer need restrictive belief-updating harnesses.

6 RESULTS & DISCUSSION

  • Over 50%: cross-conversation violations exceeded 50% of scenarios for almost all model-constraint combinations; within dialogue, rates fell to 10% for Gemini and one of 30 scenarios for Gemma.Modern reasoning LLMs generally satisfy basic probability axioms when probabilities are elicited within a single dialogue, within the scope of these tests.Source: Figure 2 · p. 7, §6 · p. 7
  • Without reasoning, LLMs neither maintain probabilistic consistency over many experiments nor reliably compute the closed-form posterior in ProteinWorld - simple.The authors report substantial posterior divergence from the Bayesian oracle in both 20Q and ProteinWorld - simple.Source: §6 · pp. 7–8, Figure 3 · p. 7
  • At least 30%: LLM-based probability updates reduce final success rates by at least 30% across all models and both environments versus the active oracle trajectory.On 20Q, weaker models can collapse to close to 0% final accuracy.Source: Figure 4 · p. 8, §6 · p. 8
  • Reasoning reduces KL divergence on 20Q and ProteinWorld - simple; on the latter, modern models achieve near-zero KL and Gemma-4-31B-it matches the oracle.On 20Q, Gemma-4-31B-it also achieves mostly consistent updates for up to 10 experiments.Source: Figure 5 · p. 8, §6 · pp. 8–9
  • Reasoning alone remains insufficient for ProteinWorld - hard: models recognize the need for simulation but produce hallucinated outputs or overly broad numerical approximations.Gemini-2.5-Pro consequently shows high KL divergence and degraded performance relative to the oracle.Source: §6 · p. 9
  • With code execution, modern LLMs approximate even analytically intractable posteriors effectively in context, with strong posterior fidelity and downstream performance.On ProteinWorld - hard, Gemini-2.5-Pro matches the oracle using a numerically stable Monte Carlo estimator.Source: §6 · p. 9, Figure 6 · p. 9
  • For modern reasoning LLMs, restrictive structural harnesses may no longer be necessary because full model-generated belief updates reach comparable downstream performance.The paper favors less restrictive harnesses that use reasoning and tools while retaining an explicit belief state for coherence across conversations and human inspection.Source: §6 · p. 9

7 CONCLUSION

Across probabilistic coherence, posterior fidelity, and adaptive experimental design, LLMs without reasoning violate Bayes’ rule at substantial task-performance cost. Native reasoning largely closes the gap for closed-form posteriors, while code execution extends belief updating to analytically intractable cases.

7 CONCLUSION

  • Without reasoning, models violate Bayes’ rule in both 20Q and ProteinWorld, while native reasoning yields near-exact closed-form posteriors and code execution extends this to analytically intractable posteriors.The conclusion also reports internal consistency over ten pieces of evidence with native reasoning.Source: §7 · pp. 9–10

A.1 EXAMPLE INSTANCE FOR SIMPLE VERSION

  • In the simple ProteinWorld instance, proteins have three conditionally independent continuous latent attributes, and experiments reveal exact values for individual attributes.The example specifies three experiments, each revealing one attribute.Source: §A.1 · p. 14

A.2 EXAMPLE INSTANCE FOR HARD VERSION

  • The hard ProteinWorld instance uses autoregressively conditioned latent attributes and noisy observations whose dependencies are nonlinear in the attributes.Its example includes experiments with nonlinear observation distributions, including a Bernoulli outcome.Source: §A.2 · pp. 14–15

B QUALITATIVE ANALYSIS OF AGENT BEHAVIOR IN PROTEINWORLD

The qualitative analysis examines how reasoning and tools affect externally determined posterior calculations in two ProteinWorld instances, using Gemma-4-31B-it and Gemini-2.5-Pro as illustrative cases. Reasoning recovers the analytical posterior in the closed-form case, though manual arithmetic and extended rechecking remain costly.

B QUALITATIVE ANALYSIS OF AGENT BEHAVIOR IN PROTEINWORLD

  • The analysis is illustrative rather than exhaustive, covering externally determined posterior calculations in two instances with Gemma-4-31B-it and Gemini-2.5-Pro.Gemini’s underlying reasoning tokens are not exposed; its traces are API summaries.Source: B QUALITATIVE ANALYSIS OF AGENT… · p. 16

B.1 CLOSED-FORM POSTERIORS WITHOUT REASONING

  • Without reasoning, Gemma-4-31B-it identifies the correct posterior mode and excludes impossible hypotheses, but overstates the probability mass on the remaining alternative.The passage characterizes this as a stronger-model failure pattern; weaker models show no consistent correspondence to the oracle.Source: §B.1 · p. 16

B.2 CLOSED-FORM POSTERIORS WITH REASONING

  • With reasoning enabled, Gemma-4-31B-it recovers the analytical posterior exactly in the simple ProteinWorld case, extracting the model, evaluating each hypothesis’s likelihood, and applying the posterior update with normalization.Across the inspected models and simple instances, remaining errors are arithmetic slips or reasoning-budget truncation before normalization.Source: §B.2 · pp. 16–17
  • After reaching the correct posterior, inspected models continue re-deriving and checking arithmetic for thousands of additional thinking tokens, raising compute costs in repeated discovery-loop updates.The analysis notes this cost is far above what the task requires.Source: §B.2 · p. 17

B.3 INTRACTABLE POSTERIORS WITHOUT CODE EXECUTION

On ProteinWorld-hard, reasoning-only models recognize the intractable likelihood but fail to recover the oracle posterior; with code execution, Gemini-2.5-Pro closely approximates it. The appendix also defines Dutch-Book tests for belief coherence and the scenarios used to detect violations.

B.3 INTRACTABLE POSTERIORS WITHOUT CODE EXECUTION

  • The model identifies the inference structure and Monte Carlo integration as appropriate for intractable marginalization, but then describes an unexecutable sampling procedure and reports its putative output.This failure is not universal: Gemma-4-31B-it recognizes that no code execution environment is available and proceeds directly to numerical approximation.Source: §B.3 · pp. 17, 19
  • 85% of Gemini-2.5-Pro’s reasoning-only posterior mass falls on Somatoplexin, whose oracle mass is 0.261; the oracle mode receives 0.102.The table compares the posterior after two experiments on ProteinWorld-hard under a uniform prior; neither reasoning-only variant approaches the oracle.Source: §B.3 · pp. 17–18, Table 5 · p. 18
  • Both hypotheses retain oracle mass of 0.162 and 0.261 but receive zero mass from the model, preventing their recovery by later evidence.The reasoning substitutes attribute means for distributions, ignoring variance that permits sign changes; forward KL penalizes this behavior.Source: §B.3 · p. 18

B.4 INTRACTABLE POSTERIORS WITH CODE EXECUTION

  • With native Python, Gemini-2.5-Pro runs a nested Monte Carlo estimator and reaches a posterior close to the oracle, with residual KL divergence consistent with finite-sample noise.The estimator samples attributes in dependency order, averages the product of experiment likelihoods, and normalizes.Source: §B.4 · p. 19

C.1 PAYOFF FORMULA

  • A Dutch Book is a set of bets based on stated probabilities that guarantees the agent a loss regardless of the outcome.The payoff formula uses stakes for events; positive and negative stakes correspond to buying and selling positions.Source: §C.1 · p. 21

C.2 ANALYZED SCENARIOS FOR DUTCH BOOKS

  • The Dutch-Book scenarios test extensionality with equivalent event descriptions, monotonicity with nested events, and normalization with an event and its complement.Source: §C.2 · p. 21
  • Coherence assessment is binary: any nonzero Dutch-Book payoff counts as a violation, so results are reported as violation rates rather than average deviations.The payoff can be scaled arbitrarily by multiplying all stakes by the same constant.Source: §C.2 · p. 21

D REVISITING HARNESSED BELIEF UPDATING IN 20Q

On 20Q, the naive baseline with reasoning matches BED-LLM and sometimes exceeds it on current open-source reasoning models. BED-LLM constrains the model to candidate hypotheses and questions while handling belief aggregation externally.

D REVISITING HARNESSED BELIEF UPDATING IN 20Q

E.1 EXAMPLE DUTCH BOOKS ELICITED FROM LLMS IN CROSS-CHAT REGIME

This section documents example probability judgments and the methods for evaluating Bayesian inference, from expanding and filtering candidate beliefs to scoring them against the history and defining an oracle posterior from answer probabilities.

E.1 EXAMPLE DUTCH BOOKS ELICITED FROM LLMS IN CROSS-CHAT REGIME

  • For the 2028 US presidential election, passages report 50% and 60% for a non-Democrat candidate winning; for Russia controlling at least 75% of Donbass at the end of October 2026, passages report 60% and 70%.Source: §E.1 · p. 24

E.2 20 QUESTIONS

  • Figure 8 presents detailed posterior-fidelity and downstream-task results for 20Q across all analyzed LLMs.The caption gives no numerical results or model ranking.Source: Figure 8 · p. 25

E.3 PROTEINWORLD

  • Figure 9 presents detailed posterior-fidelity and downstream-task results for all analyzed LLMs on ProteinWorld-simple and for Gemini-2.5-Pro on ProteinWorld-hard.The caption gives no numerical results or model ranking.Source: Figure 9 · p. 26

F.1.1 CONSIDERED PROMPTS

  • The considered prompts test normalization with positive and negated questions, monotonicity across six thresholds, and extensionality across semantically equivalent formulations.The prompt sets include ten question pairs for normalization and extensionality and ten threshold series for monotonicity.Source: §F.1.1 · pp. 27–28

F.1.2 PROMPT SETUP

  • Probability judgments use a common prompt, with base events drawn verbatim or near-verbatim from ForecastBench and related variants written to match its style.Models return a scalar probability from 0 to 1 at temperature τ = 0.7; earlier question-response turns remain in context when eliciting another judgment in the same chat.Source: §F.1.2 · p. 28

F.2 20 QUESTIONS

  • The 20 Questions implementation is adapted from Choudhury et al. (2026) and adjusted for the paper’s Bayesian-inference evaluation framework..Source: §F.2 · p. 28

F.2.1 DYNAMIC HYPOTHESIS SET Θk

  • The method dynamically adds novel candidate beliefs to the hypothesis set, breaking strict Bayesian shrinking so beliefs absent from the initial subset can be evaluated.Strict Bayesian updating would assign zero prior and posterior probability to newly introduced hypotheses; initializing with every conceivable target is described as impossible for the LLM.Source: §F.2.1 · p. 29
  • Candidate animals are generated at temperature τ = 1.3, with long beliefs filtered out and only the 100 highest-probability beliefs retained.The passages state these filters are rarely needed and do not affect final performance.Source: §F.2.1 · p. 29

F.2.2 LLM POSTERIOR

  • The LLM posterior scores each candidate by its probability of fitting the full question-answer history, then filters out beliefs with compatibility scores below 0.2.Probabilities are extracted from token log-probabilities for non-reasoning models or verbalized probabilities for reasoning models.Source: §F.2.2 · p. 30

F.2.3 ORACLE POSTERIOR

  • The oracle posterior uses answer probabilities p(y | θ, ξ), filtering out any belief assigned probability ≤0.2 for any observed answer.The passages specify different prompts for token log-probability and verbalized-probability extraction, with generation and normalization settings matching the LLM-posterior procedure.Source: §F.2.3 · pp. 30–31

F.2.4 EXPERIMENT SELECTION

The experiment-selection procedure generates candidate questions from the active hypotheses, estimates information gain through outcome entropy, and uses a direct identification question when few hypotheses remain.

F.2.4 EXPERIMENT SELECTION

  • The procedure evaluates up to 15 candidate Yes/No questions, generated to split the active hypothesis list into roughly equal parts, at temperature 1.3.Source: §F.2.4 · pp. 31–32
  • Because the hypothesis space is too large for direct entropy calculation, expected information gain is computed from the reduction in entropy over outcomes using a simulation of the current distribution.Source: §F.2.4 · p. 32
  • When at most two hypotheses remain, the model instead asks directly whether the target is the first hypothesis, since that question maximizes expected information gain.Source: §F.2.4 · p. 32

F.2.5 EXPERIMENT EXECUTION

The appendix describes how experiments are executed and how simple and hard protein worlds are generated, then details LLM and oracle posterior estimation and experiment selection. The 20 Questions setup uses 40 animals, while the protein experiments use exact attribute observations in the simple world and nonlinear dependencies in the hard world.

F.2.5 EXPERIMENT EXECUTION

  • In 20 Questions, the answerer replies exactly “Yes” or “No” depending on whether the chosen entity fulfills the question, or “Correct!” for a correct guess; then p(θ⋆)j becomes 1 for all j ≥ k + 1 and the run ends.The experimental outcome is generated using a prompt at temperature τ = 0.7.Source: §F.2.5 · p. 32

F.2.6 LIST OF ANIMALS USED

  • The 20 Questions evaluation uses 40 animals drawn as a subset of the 100 animals used in Choudhury et al. (2026).The listed instances range from coyotes and rhinoceros beetles to platypuses and brown bears.Source: §F.2.6 · p. 32

F.3.1 WORLD GENERATION

  • Simple protein worlds assign independent Gaussian attributes by class and experiments reveal one attribute exactly; hard worlds add parent-dependent nonlinear attribute means and randomized multi-attribute experiments.Hard-world parent contributions use a square or tanh transform and addition or multiplication; outcomes use Gaussian, uniform, or Bernoulli(sigmoid) families with σ ∼ Uniform(0.3, 0.7).Source: §F.3.1 · p. 33

F.3.2 LLM POSTERIOR

  • In the protein task, the LLM receives the full world distributions, parameters, prior, and experiment history, then reports probabilities across protein classes.Malformed outputs are retried twice; for Phi-4, the conversation is extended with “Assistant: θ1:” to encourage the required format.Source: §F.3.2 · pp. 33–34

F.3.3 ORACLE POSTERIOR

  • The hard protein-world likelihood is estimated by Monte Carlo with N = 30000, while the simple world permits a closed-form likelihood for exact attribute observations.The hard setup combines at least two Gaussian attributes through nonlinear transforms and uses Gaussian, uniform, or Bernoulli(sigmoid) outcomes.Source: §F.3.3 · p. 34

F.3.4 EXPERIMENT SELECTION

  • Experiment selection maximizes expected information gain over all experiments, estimating the outcome integral with M = 500 forward samples and measuring entropy reduction over hypotheses.Samples are drawn first from the current hypothesis distribution and then from the outcome distribution conditional on each sampled hypothesis.Source: §F.3.4 · p. 34

Related papers

Loading 2610.29331v1…