PaperlayerView exact revision on arXiv

Source-linked AI summary

When Should Models Change Their Minds? Contextual Belief Management in Large Language Models

Haoming Xu, Weihong Xu, Zongrui Li, Mengru Wang, Yunzhi Yao, Chiyu Wu, Jin Shang, Yu Gong, Shumin Deng

arXiv:2605.30219v1cs.AIcs.CLcs.LG

TL;DR

Long-horizon interactions require models to distinguish evidence that should revise beliefs from information that should preserve or be ignored. The paper introduces BeliefTrack to measure this capability and finds that verifier-guided reward learning improves belief management across environments, whereas prompting alone is unreliable.

  • Problem

    Models must determine which contextual information warrants belief revision, preservation, or isolation during long-horizon interactions.

  • Method

    The paper introduces BeliefTrack, a closed-world benchmark using finite belief spaces and symbolic verifiers across Rule Discovery and Circuit Diagnosis.

  • Results

    Verifier-guided reinforcement learning substantially reduces contextual belief-management failures and generalizes across environments, while explicit belief-tracking prompts provide limited and inconsistent gains.

  • Takeaways & Limitations

    The findings suggest that contextual belief management is measurable and actionable through reward learning and representation-level analysis.

  • Takeaways & Limitations

    BeliefTrack covers two synthetic environments and does not evaluate more open-ended forms of belief revision.

Abstract

from arXiv · show

Long-horizon interactions require language models to manage accumulating information: when to update their state, when to preserve their state, and what to ignore. We study this challenge as \textbf{Contextual Belief Management (CBM)}: maintaining a predicted belief state aligned with formal evidence while isolating task-irrelevant noise. To make CBM measurable, we introduce BeliefTrack, a closed-world benchmark spanning Rule Discovery and Circuit Diagnosis, where a finite belief space and symbolic verifiers enable exact turn-level evaluation. BeliefTrack diagnoses three failures: Failed Stay, Failed Update, and Failed Isolation. Across multiple LLMs, vanilla models exhibit severe CBM failures, while explicit belief-tracking prompts provide limited gains. In contrast, reinforcement learning with belief-state rewards reduces failure rates by 70.9\% on average. Further probing reveals latent belief-state dynamics behind these failures, and representation-level steering reduces failure rates by 46.1\% across two tasks\footnote{Code is coming soon at https://github.com/zjunlp/CBM.

1 Introduction

The paper frames long-horizon reasoning as Contextual Belief Management: models must update beliefs when formal evidence changes, preserve them otherwise, and filter irrelevant context. It introduces BeliefTrack, a closed-world benchmark with symbolic verification, and diagnoses distinct belief-management failures in frontier models.

  • 1 Introduction: Contextual Belief Management requires models to determine which accumulated information warrants belief revision and which task-irrelevant context should be ignored.The challenge extends beyond absorbing contextual information to identifying formal evidence and maintaining an evidence-aligned belief state.
  • 1 Introduction: BeliefTrack operationalizes CBM in two closed-world environments, Rule Discovery and Circuit Diagnosis, with finite belief spaces and symbolic verifiers.This controlled setup enables exact evaluation of belief states across multi-turn interactions.
  • 1 Introduction: 135 Rule Discovery examples with task-irrelevant noise exposed substantial belief-management errors in Qwen3.5-Plus, DeepSeek-V3.2, and GPT-5.2.The pilot results suggest that CBM failures can occur even when relevant evidence is explicitly specified.
  • 1 Introduction: BeliefTrack diagnoses three failures: Failed Stay, Failed Update, and Failed Isolation.These diagnostics test whether models preserve stable beliefs, revise them when evidence changes, and isolate beliefs from irrelevant context.
  • 1 Introduction: Vanilla models exhibit severe CBM failures, while explicit belief-tracking prompts yield only limited and inconsistent gains.The paper presents this comparison as a central evaluation finding across the BeliefTrack environments.

2 Related Work

Prior work frames belief management around resolving conflicts among parametric memory, context, user claims, and demonstrations. It also documents failures in multi-turn instruction following and contextual revision, including contextual inertia despite contradictory evidence.

  • Knowledge Conflict: Knowledge-conflict research shows that language models struggle to determine which information to trust across parametric memory, passages, user claims, and demonstrations.Recent work also highlights belief dependencies in conflict resolution.
  • Contextual Revision: Studies report that models degrade in multi-turn instruction following and fail under contextual pressure.These findings extend belief-management concerns beyond isolated knowledge conflicts.
  • Contextual Revision: Models also exhibit contextual inertia by failing to revise earlier generations or intermediate inferences after later contradictory evidence.Related work connects these failures with metacognition, memory management, and epistemic-state mechanisms.

3 Preliminary

This section formalizes Contextual Belief Management (CBM) as maintaining an evidence-aligned belief state across multi-turn interactions. It distinguishes the logically correct oracle state from the model’s predicted state, which may also reflect task-irrelevant noise.

  • CBM is a model’s ability to maintain an evidence-aligned belief state throughout a multi-turn interaction.
  • At each turn, the model receives formal evidence and optional task-irrelevant noise, alongside their accumulated observation histories.Observations are defined as o_t = (e_t, n_t), with n_t = ∅ in clean settings.
  • The task-specific belief space contains candidate hypotheses representing all possible task outcomes, while a belief state retains hypotheses supported by observed formal evidence.
  • The oracle belief state is the logically correct evidence-determined subset, whereas the predicted belief state is the subset produced by the model from observation history.The oracle state is S*_t ⊆ B_E, and the predicted state is Ŝ_t = f_θ(o_1:t) ⊆ B_E.

Contextual Belief Management

Contextual Belief Management evaluates whether models maintain evidence-aligned belief states across turns while isolating task-irrelevant noise. BeliefTrack operationalizes this objective through closed-world, finite-belief environments with symbolic verification and three diagnostic failure modes.

  • Contextual Belief Management: CBM aims to align the model’s predicted belief state with the oracle belief state at every turn of a trajectory.The objective is defined as turn-level alignment between predicted and oracle states over the full trajectory.
  • Diagnostic Failure Modes: The benchmark diagnoses Failed Stay, Failed Update, and Failed Isolation, respectively testing state preservation, evidence-driven revision, and rejection of task-irrelevant noise.These diagnostics use fixed multi-turn templates paired with symbolic verifiers that compute oracle states.
  • BeliefTrack Benchmark: BeliefTrack evaluates evidence-conditioned belief-state tracking over a finite belief space, with all task-relevant evidence specified within each episode.Models output a subset of candidate hypotheses supported by accumulated formal evidence.
  • BeliefTrack Benchmark: BeliefTrack includes Rule Discovery and Circuit Diagnosis, which share the CBM formulation but differ in candidate-hypothesis semantics and formal evidence.Rule Discovery uses finite candidate rules, such as ascending_order or sum_greater_than_10.
  • Benchmark Evaluation: BeliefTrack supports automatic evaluation without human annotation because its environments have finite belief spaces and fully specified verification logic.The benchmark also supports scalable trajectory generation through symbolic, automatically verifiable environments.

4 Methods for Improving CBM

Section 4 presents two CBM improvements: the parameter-free BT-Prompt and verifier-guided GRPO trained with belief-state rewards. BT-Prompt explicitly manages evidence and hypotheses, while reinforcement learning optimizes turn-level belief-state predictions against symbolic oracle states.

  • 4 Methods for Improving CBM: The methods are BT-Prompt, a training-free prompt enhancement, and reinforcement learning with belief-state rewards, a verifier-guided approach.BT-Prompt is parameter-free and applied at test time, whereas the second method optimizes the model with GRPO.
  • BT-Prompt: BT-Prompt instructs models to track valid formal evidence, ignore noise, re-evaluate hypotheses, and revise evidence after explicit corrections.It can restore previously eliminated hypotheses when the evidence excluding them is removed, and is used across both environments and diagnostic trajectory types.
  • RL with Belief-State Rewards: GRPO trains on multi-turn trajectories while targeting one evaluated turn using full observation history and the belief space.A symbolic verifier compares the predicted belief state with the oracle state and assigns reward only for the target turn.
  • RL with Belief-State Rewards: The training objective uses a dense Jaccard belief-state reward that measures set-level alignment and gives partial credit for overlap with the oracle state.This contrasts with sparse exact match and is ablated in Appendix C.1.

5 Experiments

Experiments use strict three-repeat evaluation to measure Failed Stay, Failed Update, and Failed Isolation, with lower rates indicating better performance. Vanilla models show severe CBM failures, whereas belief-state-reward RL substantially improves in-domain and cross-task performance without materially harming general capabilities.

  • Evaluation Protocol: Evaluation samples k = 3 assistant trajectories per fixed user-side template, and a sample fails if any trajectory exhibits the target failure.The three failure modes are Failed Stay, Failed Update, and Failed Isolation; lower values indicate better performance.
  • Baseline Results: Qwen2.5-7B-Instruct fails around 97–99% across metrics, while Qwen3.5-9B reaches 95.4% FIR on Circuit Diagnosis.These results indicate severe CBM failures in vanilla models, especially for Failed Isolation.
  • Baseline Results: BT-Prompt increases Qwen3.5-9B’s FUR in RD by 15.0% and FSR in CD by 9.7% in some settings.Its gains vary across models and environments, showing that explicit test-time instructions alone are insufficient for reliable CBM.
  • RL Results: RL reduces Qwen2.5-7B’s in-domain RD FSR/FUR to 0.0%/2.0% and CD FSR/FUR to 0.0%/0.0%.For Qwen3.5-9B, RL lowers in-domain FSR/FUR to 6.0%/8.0% on RD and 12.1%/15.9% on CD.
  • Cross-Environment Generalization: RD-trained Qwen2.5-7B reduces unseen CD FSR and FUR by 93.9% and 71.1%, respectively, while also reducing FIR by 79.4% in-domain and 63.9% out-of-domain.RL improves belief isolation despite excluding Diso from training, supporting generalization beyond the training environment.
  • General Capabilities: MMLU and GSM8K remain largely stable after RL, while most CBM gains emerge early and later checkpoints fluctuate across metrics and transfer settings.The reported capability changes are described as small fluctuations.

6 Robustness and Mechanistic Analysis

Robustness tests show that vanilla and prompt-based models deteriorate under temporally distant evidence and task-irrelevant context, whereas reinforcement learning improves temporal robustness and belief isolation. Mechanistic analyses link failures to belief-state drift, backtracking failure, and contextual hijacking, while representation-level steering reduces all three failure rates.

  • Temporal Robustness: Larger redundant depth d_red and correction delay d_cor increase Vanilla FSR and FUR, while BT-Prompt largely mirrors Vanilla and sometimes worsens failure rates.These effects indicate difficulty preserving unchanged beliefs and revising them after delayed corrections.
  • Temporal Robustness: RL keeps FSR and FUR relatively low as anchoring depth increases, with in-domain RL strongest and out-of-domain RL degrading more gradually than Vanilla and BT-Prompt.The gains are especially pronounced when training and test environments match.
  • Contextual Robustness: Authority causes the largest FIR increase across Rule Discovery and Circuit Diagnosis, followed typically by Sycophancy, while Stress has a smaller but visible effect.Vanilla and BT-Prompt remain highly vulnerable to these task-irrelevant contextual cues.
  • Contextual Robustness: RL improves belief isolation across noise types and environments, including unseen contextual interference despite training without contextual-contamination trajectories.In-domain RL achieves the strongest reductions, while out-of-domain RL remains below Vanilla and BT-Prompt.
  • Mechanistic Analysis: Prompt-based probing identifies belief-state drift in Failed Stay, backtracking failure in Failed Update, and contextual hijacking in Failed Isolation.Vanilla models lose priority for oracle-supported hypotheses even when the oracle belief remains unchanged or is later restored.
  • Mechanistic Analysis: Representation-level steering reduces FSR, FUR, and FIR in both tasks, with Task A reductions of 78.6%, 92.3%, and 48.8%, respectively.Task B also improves, with corresponding reductions of 20.7%, 23.5%, and 12.8%, without parameter updates.

7 Conclusion

The paper introduces CBM and BeliefTrack for studying evidence-aligned belief tracking in long-horizon interactions. It shows that verifier-guided learning and representation-level steering make belief management more effective and actionable than prompting alone.

  • 7 Conclusion: The paper introduces Contextual Belief Management and BeliefTrack to study evidence-aligned belief tracking in long-horizon interactions.BeliefTrack provides a framework for measuring contextual belief management.
  • 7 Conclusion: Current LLMs exhibit substantial CBM failures that prompting does not reliably fix.The conclusion characterizes prompting as insufficient for consistently resolving belief-management failures.
  • 7 Conclusion: Verifier-guided reward learning improves belief management and generalizes across environments.The reported improvement is attributed to reward learning guided by verifiers.
  • 7 Conclusion: Probing and steering analyses associate CBM failures with modifiable representation-level patterns, making the problem measurable and actionable.The conclusion links representation-level analysis to the possibility of intervening on these failures.

Limitations

The study’s scope is limited to two synthetic environments and an explicitly defined relevance boundary. Real-world belief revision and context sensitivity remain open challenges, since over-filtering may suppress legitimate feedback or changing user intent.

  • Scope of BeliefTrack: BeliefTrack evaluates belief-state tracking only in two synthetic environments, Rule Discovery and Circuit Diagnosis, leaving more open-ended belief revision uncovered.These environments isolate belief-state tracking but do not represent broader forms of belief revision.
  • Context Sensitivity and Societal Impact: BeliefTrack explicitly defines relevant evidence and irrelevant noise, unlike real-world interactions that combine corrections, preferences, uncertainty, emotions, and social context.Future work should study calibrated context sensitivity while balancing robustness to interference with responsiveness to legitimate feedback and changing user intent.

Ethics Statement · A Use of Large Language Models · B Main Experiment Implementations

The paper reports no study-specific ethical concerns because all experiments use synthetic closed-world environments without human subjects, personal data, sensitive attributes, or real-world decision-making. Large language models were used only for linguistic enhancement, not for creative or analytical research contributions.

  • Ethics Statement: Synthetic closed-world experiments avoid human subjects, personal data, sensitive attributes, and real-world decision-making.The authors report no ethical concerns specific to the study.
  • Ethics Statement: The study does not involve human subjects or personal data.These exclusions are stated as part of the study’s ethics statement.
  • Ethics Statement: The study does not involve sensitive attributes or real-world decision-making.The experiments are conducted entirely in synthetic closed-world environments.
  • A Use of Large Language Models: The authors used large language models exclusively for linguistic enhancement.Their stated aims were improving readability and ensuring an academic tone.
  • A Use of Large Language Models: Large language models were not used for idea generation, experimental design, or methodological decision-making.The tools were excluded from both creative and analytical aspects of the research.
  • A Use of Large Language Models: The authors state that the work’s intellectual contributions and methodological frameworks are their own original results.This claim follows their restriction of language-model use to linguistic enhancement.

B.1 Training Details. · B.2 Models and Infra · C Steering Experiment Details

The experiments enforce oracle-level generalization, train RL on stay/update diagnostics while reserving isolation for transfer evaluation, and use specified model infrastructure. Steering interventions derive reward-related hidden-state shifts on Rule Discovery and transfer them to Circuit Diagnosis without parameter updates.

  • B.1 Training Details.: Oracle-disjoint train/dev/test splits prevent evaluation trajectories from sharing underlying rules or faults with training trajectories.This tests generalization to unseen evidence-conditioned belief states rather than oracle-specific memorization.
  • B.1 Training Details.: Checkpoints are selected by development-split validation before testing, with model- and task-specific final training steps.The supplied implementation passage specifies 500 and 374 steps for Qwen2.5-7B-Instruct and 520 and 338 for Qwen3.5-9B across RL-RD and RL-CD, respectively.
  • B.1 Training Details.: GRPO trains only on Dstay and Dupdate, while Diso is excluded so FIR measures transfer to unseen task-irrelevant interference.Dstay and Dupdate correspond to FSR and FUR, respectively; Diso is reserved for FIR evaluation.
  • B.2 Models and Infra: Experiments cover Qwen2.5-7B-Instruct, Qwen3.5-9B, DeepSeek-V3.2, and GPT-5.2, with open-source inference and rollouts implemented in vLLM using bfloat16.GRPO reinforcement learning uses Swift on 5 A800-SMX-80G GPUs in one server.
  • B.2 Models and Infra: RL failure rates drop sharply in early checkpoints and then fluctuate mildly, indicating that most CBM gains emerge early during reward training.The dynamics are reported for FSR, FUR, and FIR across Rule Discovery and Circuit Diagnosis for two training runs.
  • C Steering Experiment Details: Steering vectors use hidden-state differences from vanilla-wrong/RL-right Rule Discovery cases and are evaluated on Rule Discovery before transfer to Circuit Diagnosis.The selected Rule Discovery configurations are transferred without re-tuning to test cross-task generalization.
  • C Steering Experiment Details: For FSR and FUR, steering targets the diagnostic failure turn using the prefix truncated at that point and the final prefix-token hidden state.The steering direction is metric-specific and extracted at a selected layer and prefix-token position.
  • C Steering Experiment Details: For FIR, steering uses full trajectories, averages hidden-state differences within each case, carries only belief-state blocks across turns, and injects one vector online at every turn.Model parameters remain unchanged; configurations are selected by Rule Discovery grid search over middle-to-late layers and scaling coefficients.

C.1 Reward Ablation · D Prompt Templates · E Case Study

The paper compares dense Jaccard and sparse exact-match belief-state rewards, defines prompt templates for Stay, Update, and Isolation challenges, and analyzes representative CBM failures. Case studies reveal failures in belief-state maintenance, contextual filtering, and routing latent beliefs to final outputs.

  • C.1 Reward Ablation: Jaccard reward gives partial credit for predictions overlapping the oracle belief state, unlike sparse exact-match reward.Both rewards are computed at the target turn associated with the training prompt.
  • C.1 Reward Ablation: Both reward variants keep training data, trajectories, hyperparameters, steps, and evaluation sets fixed to isolate reward-function effects.Evaluation covers held-out Rule Discovery and Circuit Diagnosis sets, including FSR, FUR, and FIR trajectories.
  • C.1 Reward Ablation: 15.2% average failure rate with Jaccard reward outperforms 24.9% with exact-match reward for Qwen2.5-7B-Instruct.Jaccard improves five of six diagnostic metrics, including RD-FUR (20.0% → 2.0%), RD-FIR (33.0% →20.0%), and CD-FIR (55.0% →35.0%).
  • D.1 Task Templates: Prompt templates instantiate Dstay for preserving beliefs, Dupdate for revising beliefs after corrections, and Diso for resisting irrelevant context and distractors.These templates systematically evaluate CBM under different belief-state management challenges.
  • D.2 BT-Prompt: The fixed BT-Prompt encourages explicit belief-state reasoning and restricts valid evidence to information explicitly provided and directly usable for candidate determination.It excludes descriptive, preference-oriented, biased, suggestive, advisory, and speculative content from the current valid evidence set.
  • D.3 Noise Templates: Diso pairs identical formal-evidence histories with different task-irrelevant noise, requiring models to output the same evidence-aligned belief state as clean trajectories.Sycophancy, authority, and stress noise test susceptibility to praise, confident assertions, and pressure while the symbolic oracle state remains unchanged.
  • E Case Study: Case studies identify belief-state drift and backtracking failure, contextual hijacking, and latent-output gaps as distinct sources of CBM errors.Errors include misremembering formal evidence, abandoning supported candidates, adopting distractor premises, and failing to route latent beliefs into final answers.
  • E Case Study: Models may rank an oracle-supported candidate highly during intermediate reasoning yet produce a contradictory final belief state, indicating a gap between reasoning competence and stable CBM.This failure reflects unsuccessful routing of latent belief states to final generation.
Loading 2605.30219v1…