PaperlayerView exact revision on arXiv

Source-linked AI summary

Diagnosing and Improving Probabilistic Reasoning in Large Language Models

Huaman Sun, Dingcheng Wang, Jason Hartline, Jessica Hullman

arXiv:2609.38005v2cs.AI

TL;DR

LLMs used as decision assistants need to reason from evidence and choose actions under explicit costs, but belief formation and decision quality are distinct abilities to measure. The paper decomposes decision regret and evaluates targeted RL interventions. Targeting one component does not necessarily improve the other, while jointly targeting beliefs and actions improves both when training and evaluation formats match.

  • Problem

    LLMs are increasingly proposed as decision assistants, raising questions about whether they can infer and decide well from evidence, how to measure those abilities, and whether training can improve them.

    Source: §1 · p. 1

  • Method

    The paper decomposes regret into belief and action-optimization components, applying the framework to controlled tasks and RL interventions targeting beliefs, decisions, or both.

    Source: §1 · p. 2, §8 · p. 8

  • Results

    Targeting one component improves it without necessarily transferring to the other, while joint belief-and-action training improves both when elicitation formats match.

    Source: §1 · p. 2, §8 · p. 8

  • Takeaways & Limitations

    Belief formation and decision-making should be evaluated as distinct components because action performance can improve without comparable improvement in beliefs.

    Source: §1 · p. 2, §7 · p. 8

    Whether intervention gains transfer beyond training settings remains for future work, including across domains for belief interventions and different decision problems for decision interventions.

    Source: §8 · p. 9

Abstract

from arXiv · show

Large language models (LLMs) are increasingly proposed as decision assistants who must reason probabilistically from available evidence under explicit decision costs. We propose a decision-theoretic framework that decomposes LLMs' decision loss into two components: forming accurate beliefs from provided evidence and translating those beliefs into actions that optimize a provided utility function. Using a synthetic benchmark with known ground truth, we apply the decomposition to characterize probabilistic reasoning in frontier and open-sourced models. We further evaluate whether RL interventions targeting beliefs, decisions, or both improve these components across three domains, whether improvements transfer across components and elicitation formats, and whether decision performance can improve without improvement in belief formation. We find that targeting one component of probabilistic reasoning redistributes decision loss, improving the target without necessarily transferring to others, and that jointly targeting belief formation and decision-making improves both but hinges on matched formats between training and evaluation.

1 INTRODUCTION

The paper frames probabilistic reasoning as both forming beliefs from evidence and choosing actions under explicit costs. It introduces a decision-theoretic decomposition and uses it to study current models and targeted RL interventions.

1 INTRODUCTION

  • The paper addresses whether LLMs can infer and decide well from evidence, how to measure those abilities, and whether training can improve them for decision-assistant settings.It distinguishes belief formation from using beliefs to act under specified utilities.Source: §1 · pp. 1–2
  • Controlled information structures and explicit utility scenarios let researchers study belief formation and decision optimization while abstracting from ambiguity about preferences or priors.Source: §1 · p. 2
  • The framework decomposes total decision regret into belief loss from inaccurate posterior beliefs and optimization residual from failing to choose the best action under the stated utility.This separates belief formation from decision optimization.Source: §1 · p. 2
  • Across five state-of-the-art model families, the balance of belief-formation and decision-making failures shifts with task difficulty, reasoning effort, and model architecture.Source: §1 · p. 2
  • Belief-only training improves beliefs without necessarily improving decisions; decision-only training improves actions without comparable belief gains, while jointly targeting both improves both conditionally on elicitation format.The introduction also reports that these intervention findings generalize to unseen loss functions.Source: §1 · p. 2

2 RELATED WORK

Prior work studies confidence calibration, belief formation, and consistency between elicited beliefs and actions. This paper adds controlled measurements by specifying utilities and reference posteriors.

2 RELATED WORK

  • Prior studies examine confidence reports, Bayesian belief updating from evidence, and targeted fine-tuning for belief updating and transfer in user-assistant interactions.The cited works include Kadavath et al. (2022), Tian et al. (2023), Xiong et al. (2024), Schrader et al. (2024), Falck et al. (2024), Gupta et al. (2025), and Qiu et al. (2026).Source: §2 · p. 2
  • Related work finds that calibrated uncertainty may not yield consistent actions; other studies test belief–decision coherence, recover preferences, or predict decisions from latent belief-like variables.The passage cites Pal et al. (2025), Yamin et al. (2026a), Yamin et al. (2026b), and Smolin & Wilder (2026), respectively.Source: §2 · p. 2
  • Unlike this literature, the framework specifies utility functions and uses tasks with known reference posteriors to control ambiguity about intended preferences and assumed data-generating models.Source: §2 · pp. 2–3

3 A DIAGNOSTIC FRAMEWORK FOR LLM DECISION-MAKING

The framework elicits a probability report and an action for a binary decision problem with known posterior and explicit losses. It separates decision loss due to beliefs from loss due to action choice, then normalizes regret for aggregation.

3 A DIAGNOSTIC FRAMEWORK FOR LLM DECISION-MAKING

  • For binary outcomes, the prompt provides observed evidence x, the true posterior p*(x) is defined by the data-generating model, and the LLM reports a probability p̂(x).The evidence may be structured examples or unstructured text; the verbalized probability may not reflect the model’s internal uncertainty.Source: §3 · p. 3
  • The decision problem specifies a finite action space and loss function, which assigns a real-valued loss to each action–state pair and determines expected action loss under a probability.Source: §3 · p. 3
  • The framework distinguishes the reported action, the action minimizing expected loss under the true posterior, and the action optimal under the model’s reported belief.Comparing these actions separates probabilistic-reasoning errors from failures to apply the specified loss function.Source: §3 · p. 3
  • Total regret is the excess expected loss of the reported action over the optimal action and decomposes into belief loss plus optimization residual.Belief loss measures the extra expected loss of the belief-implied action relative to the true-posterior optimum; optimization residual compares the reported action with the belief-implied action.Source: §3 · p. 3
  • Optimization residual can be negative when the reported action outperforms the action implied by the reported belief, for example if the model uses a different belief when deciding.Source: §3 · p. 3
  • Regrets are normalized to avoid overweighting loss functions with larger cost scales, but their magnitudes depend on posterior and loss distributions and are comparable only within fixed tasks.Normalized total regret and belief loss lie in [0, 1]; optimization residual lies in [-1, 1].Source: §3 · pp. 3–4

4 DIAGNOSIS OF STATE-OF-ART LLMS

On simple statistical tasks, several frontier models make optimal decisions; as inference becomes harder, belief loss emerges, while models that fail to follow the loss function show optimization residuals. The benchmark decomposes regret across models, reasoning settings, and feature sizes.

4 DIAGNOSIS OF STATE-OF-ART LLMS

  • The controlled benchmark spans five model families and varies binary-feature count, while its generating process specifies the true posterior and optimal action.Models receive labeled observations and an unlabeled test case; beliefs and decisions are elicited separately, with decisions evaluated across nine threshold losses.Source: §4 · p. 4
  • Several frontier models achieve zero regret at every threshold on tasks with no features or one feature.The reported models include GPT-5.5 with medium reasoning and Gemini 3 Flash, Gemini 3.1 Pro, and Claude Sonnet 4.6 at high reasoning effort.Source: §4 · p. 4
  • As inference difficulty increases, regret for models optimal on simple tasks emerges primarily through belief loss.At feature size d = 3, regret first appears and increases at d = 5; for some models, the d = 5 decomposition is dominated by belief loss.Source: §4 · p. 4, Figure 1 · p. 5
  • When models fail to follow the loss function in low-feature settings, optimization residual dominates total regret.The passage reports large residuals for GPT-5.5 without reasoning at d = 0 and d = 1, and larger residuals than with medium reasoning at d = 5 for thresholds τ = 0.7 and 0.8.Source: §4 · p. 4, Figure 1 · p. 5

5 IMPROVING PROBABILISTIC REASONING WITH TARGETED RL POST-TRAINING

The paper tests four RL interventions that target belief accuracy, decision quality, or both. Their designs distinguish direct belief and action supervision from joint supervision and belief-to-action alignment.

5 IMPROVING PROBABILISTIC REASONING WITH TARGETED RL POST-TRAINING

  • Belief-only training rewards accurate reports of beliefs from evidence, without supervising decisions.Its reward is maximized when the reported belief equals the true posterior; the intervention tests whether belief improvements transfer to decisions.Source: §5 · p. 5
  • Decision-only training rewards the optimal action under the true posterior and explicit loss, with no feedback on belief accuracy.This tests whether action improvements can occur without more accurate beliefs, including through an action-specific policy.Source: §5 · p. 6
  • Separate multitask training randomly allocates examples to belief and decision rewards, eliciting decisions independently of reported beliefs.It tests whether separate supervision improves both components without creating competition between them.Source: §5 · p. 6
  • Sequential alignment training rewards accurate belief reporting, then rewards a decision aligned with that belief in a second turn.Target-specific rewards are assigned to each turn; the intervention tests whether alignment supervision improves decisions and transfers across formats.Source: §5 · p. 6

6 EXPERIMENTS

The experiments compare four post-training interventions across synthetic inference, weather forecasting, and clinical reasoning. They use shared loss functions and evaluate beliefs and decisions in independent or two-turn formats.

6 EXPERIMENTS

  • The evaluation spans controlled synthetic inference, structured-text weather forecasting, and clinical reasoning from unstructured notes.Synthetic posteriors are known from tabular observations; weather probabilities come from HailFinder, and clinical posteriors are derived from known causal models.Source: §6 · pp. 6–7
  • Across tasks, the model estimates a probability and selects an action under one of nine binary loss functions with thresholds from 0.1 to 0.9.False-positive and false-negative costs vary while their sum remains fixed at 10.Source: §6 · p. 7
  • All interventions use Qwen3-8B adapters initialized from the same base checkpoint and matched training controls within each task.Training uses GRPO; the schedule is five epochs for synthetic data and two for weather and clinical tasks.Source: §6 · p. 7
  • The evaluation reports average differences from the base model in normalized regret, belief loss, and optimization residual, with bootstrap 95% confidence intervals.These metrics use the regret decomposition defined earlier in the paper.Source: §6 · p. 7
  • Independent evaluation elicits beliefs and actions in separate conversations, whereas two-turn evaluation keeps the reported belief in context for action selection.These formats assess whether separate readouts share a belief representation and how access to a reported belief affects alignment.Source: §6 · p. 7

7 RESULTS

Belief-only training improves belief loss, while decision-only training reduces regret without comparable belief gains. Joint interventions improve both components, but their decision benefits vary with whether evaluation matches the trained format.

7 RESULTS

  • Joint interventions improve belief formation, but decision outcomes differ by format: separate multitask gains are robust, while sequential alignment is strongest in two-turn evaluation.The figure organizes average differences from the base model by intervention and evaluation format; values below zero indicate reductions in the corresponding regret term.Source: Figure 2 · p. 7, §7 · p. 8
  • Belief-only training consistently reduces belief loss, but its belief gains do not necessarily transfer to comparable reductions in decision loss.Under independent evaluation, total-regret reductions are smaller than belief-loss improvements; under two-turn evaluation, total regret falls in all three tasks.Source: §7 · pp. 7–8, Figure 2 · p. 7
  • Decision-only training consistently reduces total regret across tasks, while belief loss decreases only slightly from the base model.Its benefits are much smaller in two-turn evaluation, where the model reports beliefs before choosing actions.Source: Figure 2 · p. 7, §7 · p. 8
  • Separate multitask training consistently reduces total regret with gains balanced across belief loss and optimization residual.The gains remain accessible in both independent and two-turn evaluation.Source: Figure 2 · p. 7, §7 · p. 8
  • Sequential alignment training is among the strongest interventions in two-turn evaluation, but its gains are smaller under independent evaluation.In independent evaluation, the model cannot access its reported beliefs when selecting actions.Source: Figure 2 · p. 7, §7 · p. 8
  • Interventions trained on thresholds 0.2, 0.4, 0.6, and 0.8 generalize well to held-out thresholds 0.1, 0.3, 0.5, 0.7, and 0.9.The reported main findings continue to hold on the held-out thresholds.Source: §7 · p. 8

8 DISCUSSION

The paper’s framework separates belief-formation errors from failures to choose actions under specified costs, and its interventions can improve targeted components without reliable transfer. The discussion also identifies limits in model ambiguity, task scope, and transfer beyond training settings.

8 DISCUSSION

  • Targeting one component improves that component without necessarily transferring, while jointly targeting belief formation and decisions improves both when training and evaluation formats match.The framework separates errors in forming beliefs from errors in translating beliefs into actions under specified costs.Source: §8 · p. 8
  • The framework cannot fully remove ambiguity about the data-generating model, and reported beliefs may not faithfully reflect internal representations.The experiments also focus on binary states and decision problems, and RL experiments use a single base model.Source: §8 · pp. 8–9
  • Whether intervention gains transfer beyond training settings remains an open question, including across domains, loss functions, and decision problems.The discussion proposes testing belief interventions on structurally similar datasets in different domains and decision interventions on unseen losses and other decision problems.Source: §8 · p. 9

A PROMPTS FOR SYNTHETIC TASK EVALUATION

  • Synthetic-task evaluation elicits beliefs under a quadratic scoring rule and decisions under nine cost thresholds, using observed samples as in-context evidence.Beliefs are averaged across five shuffled versions of the same true probability to reduce sensitivity to sample order.Source: A PROMPTS FOR SYNTHETIC TASK EVALUATION · pp. 10–11

B SYNTHETIC TASK

The paper evaluates probabilistic reasoning on synthetic, weather-forecasting, and clinical tasks using known reference probabilities and explicit decision losses. Each task pairs evidence with separate belief and decision prompts.

B SYNTHETIC TASK

  • The synthetic post-training task uses 1,000 evidence cases with five binary features, split into 700 training, 100 validation, and 200 evaluation cases.The prompts ask models to estimate a probability or choose a label under false-positive and false-negative costs.Source: B SYNTHETIC TASK · pp. 11–12

C WEATHER FORECASTING TASK

  • The weather task derives reference probabilities from the HailFinder Bayesian network for seven atmospheric variables and splits distinct contexts into 3,000 training, 300 validation, and 500 evaluation examples.The model estimates hail probability or chooses whether to issue a warning under specified losses.Source: C WEATHER FORECASTING TASK · pp. 12–13

D CLINICAL TASK

  • The clinical task uses filtered SimSUM synthetic patient records, computes reference posteriors from a Bayesian network, and splits retained cases into 2,211 training, 300 validation, and 500 evaluation examples.The outcome is pneumonia or common cold, and the decision is whether to escalate for infection-focused evaluation under specified losses.Source: D CLINICAL TASK · p. 13

E FULL INTERVENTION RESULTS

The intervention-results tables report changes in regret relative to the base model for three tasks, with mean differences and 95% confidence intervals. The supplied captions do not give intervention-level values.

E FULL INTERVENTION RESULTS

  • Table 1 reports synthetic-task intervention-minus-Base changes in regret as mean differences with 95% confidence intervals across 200 cases.Negative values indicate lower regret than Base.Source: Table 1 · p. 14
  • Table 2 reports HailFinder intervention-minus-Base changes in regret as mean differences with 95% confidence intervals across 500 cases.Negative values indicate lower regret than Base.Source: Table 2 · p. 14
  • Table 3 reports SimSUM intervention-minus-Base changes in regret as mean differences with 95% confidence intervals across 500 cases.Negative values indicate lower regret than Base.Source: Table 3 · p. 14

F FULL DIAGNOSIS RESULTS

The full diagnosis figure covers normalized regret decomposition across model–reasoning settings and feature sizes. The accompanying belief comparison reports differences in alignment and variation across settings.

F FULL DIAGNOSIS RESULTS

  • Figure 3 presents normalized regret decomposition across model–reasoning settings, with columns for feature sizes d = 0, 1, 3, and 5.“All regrets are zero” identifies settings optimal at every threshold; Qwen 3.6 35B-A3B uses a different y-axis scale because its regret is considerably higher.Source: Figure 3 · pp. 15, 17

G ELICITED VS. REVEALED BELIEFS

  • Claude Sonnet 4.6 (high), Gemini 3.5 Flash (high), and GPT-5.5 (medium) show smoother, more closely aligned elicited and decision-revealed beliefs than Llama 4 with thinking and non-thinking Qwen 3.6.For most settings, both measures fluctuate more as feature size increases; within GPT-5.5, no thinking shows greater misalignment than medium thinking, including at small feature sizes.Source: G ELICITED VS. REVEALED BELIEFS · p. 18, Figure 4 · p. 19

H TRUE POSTERIOR VS. EMPIRICAL FREQUENCY AS REFERENCE

Using the same observations, the analysis replaces the true-posterior reference with exact-pattern empirical frequency at d = 3 and d = 5. Mean normalized total regret decreases under the empirical-frequency reference, though the component changes and model outcomes vary.

H TRUE POSTERIOR VS. EMPIRICAL FREQUENCY AS REFERENCE

  • The alternative reference uses positive-outcome frequency among exact full-vector matches, whereas the true posterior pools vectors with the same number of active features.The references coincide at d = 0 and d = 1; the comparison at d = 3 and d = 5 changes only the reference on the same observations.Source: H TRUE POSTERIOR VS. EMPIRICAL… · p. 20
  • At d = 3, the decrease is primarily in belief loss; at d = 5, belief loss and signed optimization residual both decrease on average.GPT-5.5 with medium thinking shows a particularly large decrease in both terms at d = 5.Source: H TRUE POSTERIOR VS. EMPIRICAL… · p. 20
  • The average pattern has exceptions: Qwen 3.6 35B-A3B without thinking has higher total regret, while Llama 4 Scout has increases in both components.For Qwen, the belief-loss increase exceeds the optimization-residual decrease.Source: H TRUE POSTERIOR VS. EMPIRICAL… · p. 20

I UTILITY GENERALIZATION

  • Table 4 reports changes in synthetic-task total regret at training-seen and held-out thresholds, including their gap.Entries average 200 cases and include 95% paired bootstrap intervals; the supplied passage does not give cell values.Source: Table 4 · p. 20

H TRUE POSTERIOR VS. EMPIRICAL FREQUENCY AS REFERENCE

  • Mean normalized total regret decreases 18% at d = 3 and 25% at d = 5 under the empirical-frequency reference, averaged across model settings and decision thresholds.The figures compare decompositions using the true posterior and exact-pattern empirical frequency.Source: H TRUE POSTERIOR VS. EMPIRICAL… · p. 20, Figure 5 · p. 21, Figure 6 · p. 24
  • Figure 6 presents the d = 5 decomposition across model settings, with columns for each reference and their componentwise difference.Qwen 3.6 35B-A3B uses a different y-axis scale because it has considerably higher regret.Source: Figure 6 · pp. 24, 26

Related papers

Loading 2609.38005v2…