PaperlayerView exact revision on arXiv

Source-linked AI summary

PHRBench: A Behavioral Evaluation of Post-Hallucination Reasoning in LLMs

Linghao Meng, Feng He, Xuan Yang, Junyuan Mao, Pinze Ren, Deqing Mu, Hesen Yang, Qiankun Li

arXiv:2610.10455v1cs.CLcs.AI

TL;DR

Prior post-hallucination reasoning studies mainly track final outcomes and aggregate reasoning dynamics, leaving individual responses’ resolution of hallucinated premises undercharacterized. PHRBench evaluates response-level behaviors across four domains and 18 models; it finds a vulnerability–correction tension and predictive signal for recovery, which remains relatively rare.

  • Problem

    Prior PHR analyses emphasize changes in final outcomes and aggregate reasoning dynamics rather than distinguishing how individual responses resolve hallucinated premises.

    Source: §1 · pp. 1–2

  • Method

    PHRBench evaluates 4,820 controlled instances across four domains and 18 models, classifying trajectories as compliance, avoidance, or heuristic correction independently of final-answer correctness.

    Source: §1 · pp. 2–3

  • Results

    Hallucinated context reduces accuracy by 7.7 percentage points on average, while successful recovery is associated with more frequent belief updates and prompt features predict it with an AUROC of 0.847.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The benchmark provides a behavioral account of how LLMs resolve erroneous context and when successful recovery is likely to occur.

    Source: §1 · pp. 2–3

Abstract

from arXiv · show

Hallucinated information can propagate through multi-stage LLM systems and become part of the context for subsequent reasoning. Existing studies of post-hallucination reasoning (PHR) mainly characterize changes in final outcomes and aggregate reasoning dynamics, leaving how models resolve hallucinated premises at the response level insufficiently understood. In this work, we introduce PHRBench, a controlled benchmark for behaviorally structured PHR across four domains and 18 large language models. PHRBench characterizes each reasoning trajectory independently of final-answer correctness through Hallucination Compliance, Hallucination Avoidance, and Heuristic Correction, and defines an insightful trajectory as successful correction that ultimately reaches the correct answer. Across 4820 controlled instances, we find that successful recovery remains relatively rare and is associated with more frequent belief updates along the reasoning trajectory. We further find that properties of the hallucinated prompt contain substantial predictive signal for successful recovery, with a lightweight predictor achieving an AUROC of 0.847. These findings provide a behavioral view of post-hallucination reasoning, characterizing how LLMs resolve erroneous context and when successful recovery is likely to occur.

1 INTRODUCTION

PHRBench examines how models resolve hallucinated premises at the response level, beyond final-answer correctness. It finds a scaling tension between vulnerability and correction, while successful recovery remains rare and shows predictive signal in prompt features.

1 INTRODUCTION

  • Hallucinations that evade upstream safeguards can become premises for downstream decisions in multi-stage LLM systems.The introduction frames downstream recovery as important because safeguards cannot reliably prevent every hallucinated statement from entering later processing.Source: §1 · p. 1
  • Existing PHR analyses emphasize final outcomes and aggregate reasoning dynamics, leaving individual responses’ resolution behaviors insufficiently distinguished.Correct answers can result from accepting or bypassing a hallucinated premise, while correction can still end in an incorrect answer.Source: §1 · pp. 1–2
  • PHRBench evaluates 4,820 instances across four domains and 18 models, separating compliance, avoidance, and heuristic correction from final-answer correctness.It defines an insightful trajectory as heuristic correction followed by a correct answer.Source: §1 · pp. 2–3
  • Prompt-level structural and semantic features predict successful recovery with an AUROC of 0.847, while hallucinated context reduces accuracy by 7.7 percentage points on average.The findings also associate successful recovery with more frequent belief updates.Source: §1 · p. 2, Figure 1 · p. 2

2 RELATED WORK

Prior work studies how hallucinations propagate and shape subsequent reasoning, but does not distinguish the resolution behavior of individual responses from their final outcomes.

2 RELATED WORK

  • Prior studies trace hallucination transmission across generations, interactions, and agents, including accumulation in contaminated conversational histories.Examples include MMHalSnowball, MM-Snowball, the Woozle Effect, Hallucination Cascade, and Collective Hallucination.Source: §2 · p. 3
  • Other work analyzes subsequent reasoning through verification, revision, feedback, and trajectory-level auditing, yet does not distinguish response-level resolution behaviors.These approaches include self-feedback, claim verification, explicit assessment, and modeling hallucinations across reasoning steps.Source: §2 · p. 3
  • The paper separates how an individual response resolves hallucinated content from whether its final answer is correct.Different resolution behaviors can lead to the same outcome.Source: §2 · p. 3

3 PHRBENCH

PHRBench pairs reasoning questions with controlled truthful and hallucinated contexts to study effects on both final answers and reasoning trajectories.

3 PHRBENCH

  • Each question receives a truthful context and multiple hallucinated augmentations, with three corruption patterns: Rule Contradiction, State Distortion, and Pseudoscientific Entanglement.The augmentations preserve task relevance and avoid revealing the answer.Source: §3 · p. 4
  • The evaluation combines final-answer Accuracy with four trajectory metrics: Output Length, Uncertainty Index, Belief Update Frequency, and Branching Complexity.Together, these measures describe outcomes and changes in the reasoning process.Source: §3 · p. 4
  • The 460 base questions come from eight established benchmarks, and Figure 2 shows their reasoning-category distribution and augmentation composition by domain.Scientific tasks cover three domains, while Code Generation adds computational reasoning.Source: §3 · p. 4, Figure 2 · p. 4
  • PHRBench comprises 460 base questions and 4,820 total instances across Chemistry, Biomedicine, Physics, and Code Generation.The questions span multiple-choice and open-ended formats.Source: §3 · pp. 3–5

4 EXPERIMENTAL SETUP

The experimental setup evaluates 18 models and measures both final-answer performance and reasoning trajectories.

4.1 MODELS

  • The evaluation covers 18 proprietary and open-source models across parameter scales, with 10 independently sampled responses per model and prompt.Temperature is set to 0.7, and maximum output size is set to each model’s supported maximum.Source: §4.1 · p. 5

4.2 EVALUATION METRICS

  • Accuracy measures whether each response’s final answer matches the ground truth.It is the outcome metric for final-answer performance.Source: §4.2 · p. 5
  • Trajectory metrics comprise Output Length, Uncertainty Index, Belief Update Frequency, and Branching Complexity.They measure response token count, token-level uncertainty, shifts between consecutive reasoning-step distributions, and branching cues normalized by response length.Source: §4.2 · p. 5

5 RESULTS AND ANALYSIS

Hallucinated context degrades accuracy, while models’ behavioral responses range from compliance and avoidance to correction; successful recovery remains uncommon. Prompt properties also predict insightful trajectories before generation.

5.1 MAIN RESULTS

  • 7.7% average accuracy drop under hallucinated context accompanies stronger degradation at larger scales, even as OL, BUF, and BC rise under both settings.Proprietary models average an 11.0% drop versus 6.1% for open-source models.Source: Table 1 · p. 6, §5.1 · pp. 6–7
  • Chemistry has the lowest average accuracy retention, while State Distortion causes the strongest degradation and Pseudoscientific Entanglement retains approximately 98% of baseline accuracy.The task-domain and augmentation-type patterns are shown alongside model-level accuracy retention.Source: Figure 3 · p. 6, §5.1 · pp. 6–7

5.2 HOW DO MODELS RESOLVE HALLUCINATED PREMISES?

  • 11 of 18 models produce insightful trajectories in fewer than 10% of responses, while Qwen3-235B-A22B reaches 24.91%; heuristic correction increases with model scale.Correction remains relatively infrequent, and its rate exceeds the rate of insightful trajectories.Source: §5.2 · p. 7, Table 2 · p. 7, Figure 4 · p. 8
  • Insightful trajectories average a BUF of 0.63 versus 0.18 for non-insightful trajectories; heuristic correction also has a 0.31 higher BUF than compliance.Across four representative models, heuristic correction involves about 30 more reasoning words than compliance, while avoidance has the highest average branching complexity.Source: §5.2 · pp. 7–8, Figure 5 · p. 8
  • Non-insightful trajectories may access correct knowledge but let the injected premise override it, whereas insightful trajectories reject the conflict and redirect reasoning toward the correct answer.The qualitative cases distinguish correction that is sustained through subsequent reasoning from correction that does not lead to the correct answer.Source: §5.2 · p. 8, Figure 6 · p. 9

5.3 CAN WE PREDICT INSIGHTFUL TRAJECTORIES?

  • A predictor based on 24 prompt features achieves an AUROC of 0.847 and accuracy of 0.863 for estimating whether a prompt will induce insightful reasoning.The features are extracted using an external LLM, and an XGBRegressor estimates the prompt’s propensity to induce insightful reasoning.Source: §5.3 · p. 9
  • Total Prompt Length is the strongest individual predictor at 20.7% of global SHAP contribution, while Domain-Rule Violation and Error Locality contribute 7.4% and 7.2%.The predictive signal spans structural and hallucination–task semantic properties.Source: §5.3 · p. 9
  • The largest pairwise interaction is 0.020 between Total Prompt Length and Question Length; nearly all other pairwise interactions are below 0.01.The authors describe the selected features as capturing largely complementary aspects of conditions associated with insightful reasoning.Source: §5.3 · p. 9

6 CONCLUSION

PHRBENCH separates how models resolve hallucinated premises from whether they answer correctly, finding that successful correction is rare and more frequent belief updating is associated with recovery. Prompt properties also provide predictive signal for insightful trajectories.

6 CONCLUSION

  • PHRBENCH evaluates post-hallucination reasoning across four domains and 18 LLMs, separating compliance, avoidance, and correction from final-answer correctness.An insightful trajectory performs heuristic correction and ultimately reaches the correct answer.Source: §6 · p. 9
  • Hallucinated context degrades performance while larger models exhibit stronger corrective behavior, revealing a tension between vulnerability and recovery capability.Successful correction remains relatively rare and is characterized by stronger belief updating.Source: §6 · p. 9
  • Prompt-level features are analyzed for their ability to predict insightful trajectories.Figure 7 presents a SHAP analysis of those features.Source: §6 · p. 9, Figure 7 · p. 10

APPENDIX

  • The appendices provide detailed benchmark, experimental-setup, results, behavioral-resolution, and trajectory-prediction materials.Source: APPENDIX · p. 14

A.1 DATA COLLECTION AND STATISTICS

PHRBENCH draws on reasoning questions from four domains, standardizes them, and adds controlled truthful and hallucinated context. Its construction includes reasoning-category annotation, review of augmentations, and evaluation across 18 models.

A.1 DATA COLLECTION AND STATISTICS

  • The benchmark collects 460 questions from eight established benchmarks across Chemistry, Biomedicine, Physics, and Code Generation.The four domains contribute 120, 120, 120, and 100 questions, respectively.Source: §A.1 · p. 14
  • Questions receive one mutually exclusive reasoning subtype, retaining compatible source labels or mapping others by their primary reasoning requirement.This taxonomy distinguishes within-domain reasoning patterns from domain-level variation.Source: §A.1 · p. 14, Table 3 · p. 15

A.2 AUGMENTATION CONSTRUCTION

  • Each standardized question receives a truthful augmentation and multiple hallucinated augmentations, with hallucinations spanning rule contradictions, state distortions, and pseudoscientific entanglements.The benchmark construction uses 10 hallucinated augmentations per question in three domains and three per Code Generation problem; retained items pass manual review and LLM-based verification.Source: §A.2 · pp. 15–16

A.3 EXAMPLES OF TASKS

  • Domain examples pair a baseline task with truthful and hallucinated context, illustrating semantic errors such as an ether premise or an unsupported input-order assumption.The examples include physics, code-generation, chemistry, and biomedicine tasks.Source: §A.3 · pp. 17–18

B.1 MODEL SETUP

  • The evaluation covers six proprietary and twelve open-source models, with scaling comparisons within Qwen2.5, Qwen3, and Llama-3.1 families.Each model–prompt pair is sampled independently 10 times under a shared decoding protocol.Source: §B.1 · p. 20

B.2 EVALUATION METRICS

The evaluation combines final-answer accuracy with four measures of reasoning trajectories, including output length, uncertainty, belief-update frequency, and branching complexity. The appendix also clarifies how accuracy retention compares hallucinated and truthful settings.

B.2 EVALUATION METRICS

  • Accuracy measures whether the extracted final answer matches the ground truth, while trajectory metrics characterize generated reasoning properties.The trajectory measures include output length, uncertainty index, belief update frequency, and branching complexity.Source: §B.2 · pp. 20–21
  • Output Length measures trajectory length, while Uncertainty Index averages token-level predictive uncertainty using a fixed reference model.The same reference scorer and metric configuration are used across models, domains, and conditions.Source: §B.2 · p. 21
  • Belief Update Frequency is the fraction of consecutive reasoning-step transitions whose reference-scorer distribution shift exceeds a fixed threshold.A higher value indicates more frequent distributional shifts, but does not by itself establish belief revision, error correction, or improved reasoning.Source: §B.2 · pp. 21–22
  • Branching Complexity measures branching-related expressions per 100 generated words, and model-level statistics average metrics across repeated generations and benchmark instances.Source: §B.2 · p. 22

C DETAILED RESULTS OF MAIN EXPERIMENTS

  • Accuracy retention expresses hallucinated-setting accuracy as a proportion of truthful-setting accuracy; values below 100% indicate degradation and values above 100% indicate improvement.Retention is relative preservation, distinct from absolute accuracy differences reported in percentage points.Source: C DETAILED RESULTS OF MAIN EXPERIMENTS · p. 23

C.1 DOMAIN-WISE RESULTS

Hallucinated context degrades accuracy across all four domains, though the size of the drop varies. Scaling-related vulnerability persists across domains, while reasoning-trajectory changes differ by domain.

C.1 DOMAIN-WISE RESULTS

  • Average accuracy drops are 6.57 points in Chemistry, 8.34 in Biomedicine, 7.12 in Physics, and 6.89 in Code Generation; 17 of 18 models degrade in Code Generation.Chemistry’s smaller drop should not imply greater robustness: its average baseline accuracy is 23.0%, versus 57.2% in Biomedicine, 56.0% in Physics, and 48.4% in Code Generation.Source: §C.1 · p. 23, Table 5 · p. 24, Table 6 · p. 24
  • Scaling-related accuracy degradation recurs across domains, including within Qwen2.5, Qwen3, and Llama-3.1 model families.For Qwen3, scaling from 8B to 235B-A22B increases degradation from 1.9 to 9.7 points in Chemistry, 1.5 to 10.2 in Biomedicine, and 0.3 to 9.1 in Physics.Source: §C.1 · p. 23
  • Reasoning-trajectory changes vary by domain: Physics shows increases in OL, UI, BUF, and BC, while Chemistry and Code Generation show shorter trajectories.Biomedicine also shows longer trajectories and increases in UI and BUF; Chemistry has lower average UI, BUF, and BC, while Code Generation has reduced UI and BUF but a modest increase in BC.Source: §C.1 · p. 23

C.2 RESULTS ACROSS HALLUCINATED AUGMENTATION TYPES

Hallucination type relates to both accuracy loss and reasoning-trajectory changes. The section also defines response-level resolution categories independently of answer correctness, distinguishing premise correction from successful recovery.

C.2 RESULTS ACROSS HALLUCINATED AUGMENTATION TYPES

  • Across 18 models, State Distortion causes the largest average accuracy drop (9.4 percentage points), followed by Rule Contradiction (7.4 points) and Pseudoscientific Entanglement (1.8 points).Code Generation is excluded from this comparison because its hallucinated augmentations contain only State Distortion.Source: §C.2 · p. 26
  • Augmentation types also differ in trajectory changes: State Distortion is associated with longer responses and increased branching, while Rule Contradiction raises output length less and lowers branching complexity.Pseudoscientific Entanglement often elicits longer responses without a commensurate accuracy loss.Source: §C.2 · p. 26

D.1 RESOLUTION DEFINITION

  • Behavioral resolution classifies a response by whether it materially relies on a hallucinated premise or explicitly rejects it and uses a corrected fact in subsequent reasoning.Uncertainty or noting a conflict alone does not count as correction; mixed responses are classified by the premise that ultimately governs downstream reasoning.Source: §D.1 · p. 28
  • A human–LLM annotation procedure labels responses, with manual review finding 95% of labels correct in the audited sample.A manually labeled subset establishes criteria for LLM labeling, followed by review of a sample of assigned labels.Source: §D.1 · p. 29
  • An insightful trajectory requires both Heuristic Correction and a correct final answer; correcting the premise but deriving a wrong answer is a correction failure.The categories remain independent of final-answer correctness, so a correct answer alone does not establish successful hallucination resolution.Source: §D.1 · p. 29

D.2 REASONING DYNAMICS ANALYSIS

Across the evaluated models, heuristic correction is associated with longer reasoning and more belief updating, while avoidance tends to involve more branching. Insightful trajectories show stronger belief updating and longer reasoning than non-insightful trajectories.

D.2 REASONING DYNAMICS ANALYSIS

  • Across models, Heuristic Correction is associated with longer trajectories and higher BUF than Compliance or Avoidance, while Avoidance generally exhibits higher BC.UI shows no consistent ordering across the three behavioral modes; the clearest distinction is between belief revision and branching.Source: §D.2 · p. 30, Table 12 · p. 31, Table 13 · p. 31
  • Insightful trajectories average 34.7 more words and a BUF of 0.68, compared with 0.24 for non-insightful trajectories; UI and BC show no consistent cross-model difference.The comparison covers all 18 evaluated models.Source: §D.2 · pp. 30–32, Table 14 · p. 32

D.3 QUALITATIVE ANALYSIS

The qualitative analysis links successful resolution to retaining a corrected belief through subsequent reasoning. The prompt-prediction analysis uses prompt-visible features and reports predictive performance on held-out base questions.

D.3 QUALITATIVE ANALYSIS

  • Successful resolution depends on a detected conflict producing a persistent revision of the reasoning state; unsuccessful trajectories may later restore the hallucinated premise as a constraint.The paired cases hold the model and base question fixed while varying the hallucinated augmentation.Source: §D.3 · p. 32, Table 15 · p. 32, Table 16 · p. 33

E.1 PROMPT-LEVEL FEATURE DEFINITIONS

  • The 24 prompt-level features comprise nine semantic features about hallucination–task relations and fifteen structural features about prompt complexity and surface reasoning burden.Tables 19 and 20 summarize the feature set.Source: §E.1 · p. 35, Table 19 · p. 35, Table 20 · p. 35

E.2 FEATURE EXTRACTION PROTOCOL

  • Feature extraction uses only prompt-visible inputs, excluding model outputs, correctness labels, behavioral labels, trajectory statistics, and prediction targets.GPT-4o extracts the semantic features in two structured calls; the remaining features are computed deterministically.Source: §E.2 · pp. 35–36

E.3 PREDICTOR TRAINING AND EVALUATION

  • The prompt-feature predictor achieves an AUROC of 0.847 for estimating a hallucinated prompt’s propensity to induce an insightful trajectory.The test split contains 647 instances and keeps base questions disjoint from training.Source: §E.3 · p. 36

E.4 PREDICTION ANALYSIS

Mean absolute SHAP attributions show that prompt-scale structure and hallucination–task relations contribute predictive signal for insightful trajectories, with contributions concentrated among a small set of features.

E.4 PREDICTION ANALYSIS

  • Total Prompt Length contributes 20.75% of global SHAP attribution, ahead of Candidate Length at 8.78% and Question Length at 8.76%.Table 21 reports the global feature-attribution ranking.Source: §E.4 · p. 36, Table 21 · p. 37
  • Domain-Rule Violation and Error Locality contribute 7.40% and 7.20%, respectively, ranking among the seven most influential features.Source: §E.4 · p. 36, Table 21 · p. 37
  • The top seven features account for 68.08% of attribution, combining structural and semantic features; SHAP attribution is not interpreted as causal effect.Entity Count and Contextual Dependency Count also contribute strongly, reflecting task-relevant information amount and interdependence.Source: §E.4 · p. 36
Loading 2610.10455v1…