PaperlayerView exact revision on arXiv

Source-linked AI summary

Illusions of Confidence? Diagnosing LLM Truthfulness via Neighborhood Consistency

Haoming Xu, Ningyuan Zhao, Yunzhi Yao, Weihong Xu, Hongru Wang, Xinle Deng, Shumin Deng, Jeff Z. Pan, Huajun Chen, Ningyu Zhang

arXiv:2601.05905v2cs.CLcs.AIcs.HCcs.LGcs.MA

TL;DR

Point-wise confidence can mask brittle beliefs under contextual interference, a concern for LLMs deployed in context-engineered settings. The paper measures coherence across conceptual neighborhoods and stress-tests beliefs; high-NCB knowledge is more resistant to interference, and Structure-Aware Training reduces brittleness.

  • Problem

    Point-wise confidence measures such as self-consistency can miss brittle beliefs, while contextual inputs can mislead LLMs in real-world deployments.

    Source: §1 · p. 1

  • Method

    NCB measures response coherence across conceptual neighborhoods, stress tests probe beliefs under contextual interference, and SAT optimizes context-invariant beliefs.

    Source: §1 · p. 2, §2.1 · p. 2

  • Results

    Across four LLMs, high-NCB knowledge is more stable under interference; SAT reduces average performance degradation by approximately 30% across stress tests.

    Source: §1 · p. 2, §5.2 · p. 8

  • Takeaways & Limitations

    The findings support treating belief robustness as a structural property and using structure-aware evaluation and training for more trustworthy LLMs.

    Source: §1 · p. 2, §7 · p. 9

    The evaluation focuses on time-invariant facts and excludes dynamic facts, multi-hop reasoning, causal chains, and hierarchical taxonomies.

    Source: Limitations · p. 9

Abstract

from arXiv · show

As Large Language Models (LLMs) are increasingly deployed in real-world settings, correctness alone is insufficient. Reliable deployment requires maintaining truthful beliefs under contextual perturbations. Existing evaluations largely rely on point-wise confidence like Self-Consistency, which can mask brittle belief. We show that even facts answered with perfect self-consistency can rapidly collapse under mild contextual interference. To address this gap, we propose Neighbor-Consistency Belief (NCB), a structural measure of belief robustness that evaluates response coherence across a conceptual neighborhood. To validate the efficiency of NCB, we introduce a new cognitive stress-testing protocol that probes outputs stability under contextual interference. Experiments across multiple LLMs show that the performance of high-NCB data is relatively more resistant to interference. Finally, we present Structure-Aware Training (SAT), which optimizes context-invariant belief structure and reduces long-tail knowledge brittleness by approximately 30%. Code is available at https://github.com/zjunlp/belief.

1 Introduction

Point-wise confidence can fail to reflect robust beliefs, even as LLMs face misleading contextual inputs in real-world deployments. The paper introduces neighborhood-based evaluation and training approaches to address this gap.

1 Introduction

  • The paper frames stable, truthful beliefs as essential for deployments where retrieval, collaboration, or prompt engineering can introduce misleading context.It identifies healthcare, law, and science as high-stakes domains where truthfulness failures limit LLM use.Source: §1 · p. 1
  • Among 995 questions answered correctly with perfect self-consistency, contextual interference reduced accuracy from 100.0% to 33.8%.This pilot result illustrates why point-wise confidence may not reflect a model’s true belief state.Source: §1 · p. 1, Figure 1 · p. 1
  • Across four LLMs, high-NCB knowledge is more stable under interference, while Structure-Aware Training reduces brittleness by roughly 30%.The reported training comparison is against baselines.Source: §1 · p. 2
  • Neighbor-Consistency Belief (NCB) estimates belief robustness by measuring response coherence across a conceptual neighborhood.The neighborhood includes entity prerequisites, logical implications, and thematic associations.Source: §1 · p. 2

2 Preliminary

The paper treats robust belief as structured coherence across related facts rather than isolated correct answers. It estimates this belief state with Bayesian-inspired posterior odds and a computable NCB score.

2.1 Robust Knowledge Belief is Structured

  • Among 995 questions answered correctly in all 30 independent samples, contextual interference reduced accuracy from 100% to 33.8%.The pilot motivates distinguishing structured belief from isolated-fact memorization.Source: §2.1 · p. 2
  • Structured belief means coherent, mutually consistent responses across related questions; unstructured belief relies on memorization of isolated facts.The paper treats the former as robust belief and the latter as brittle belief.Source: §2.1 · p. 2

2.2 Bayesian-Inspired Belief Estimation

  • NCB approximates the posterior probability of structured belief using empirical correctness frequencies for neighboring facts, with a 1/m exponent to keep scores comparable as neighborhood size grows.The posterior-odds derivation assumes conditional independence of neighbor responses given a correct target answer and a structured belief state.Source: §2.2 · p. 3

3 Experimental Design and Setup

The experiments build a Neighbor-Enriched Dataset around time-invariant facts and test belief robustness under social-pressure and source-credibility contexts. The design uses automatically generated neighboring facts and misleading, factually correct distractor facts.

3.1 Data Construction

  • The dataset contains 2,000 time-invariant facts across four categories, each paired on average with 7.84 verified Neighbor Facts and 4.88 Misleading Neighbor Facts.The categories are STEM, Arts & Culture, Social Sciences, and Sports; the misleading neighbor facts describe the distractor entity and are factually correct.Source: §3.1 · pp. 3–4
  • Each data point is embedded in a conceptual neighborhood of generated candidate facts, screened, verified, and expert-annotated before inclusion.Neighbor facts cover prerequisites, logical implications, and thematic associations; the retained set combines the target question and answer with its verified neighbors.Source: §3.1 · pp. 3–4
  • The evaluation excludes dynamic facts because knowledge updating would introduce confounding factors.The dataset therefore focuses on time-invariant factual knowledge.Source: §3.1 · p. 3

3.2 Contextual Interference for Stress Tests

  • The stress tests vary peer consensus or source credibility, probing responses to misleading agreement and authoritative contexts.Peer scenarios include explicit conflict and subtler priming with misleading neighbor facts; source scenarios vary credibility and use falsified claims or irrelevant but true distractor facts.Source: §3.2 · pp. 4–5, Figure 3 · p. 4

4 Stress-Testing Internal Beliefs

The section tests whether NCB predicts resistance to contextual interference among high self-consistency answers, using stress tests across four LLMs. High-NCB samples are generally more robust, while inference-time strategies and model scaling do not consistently close the robustness gap.

4 Stress-Testing Internal Beliefs

  • The stress tests stratify the High Self-Consistency dataset by NCB ranking to assess whether NCB distinguishes robustness among initially correct, perfectly consistent answers.The analysis compares top and bottom percentile subsets at 5%, 20%, and 35%.Source: §4 · p. 5, §4.3 · p. 5

4.1 Implementation Details

  • The evaluation uses Qwen2.5, Qwen3, Qwen3-Thinking, and OLMo2, sampling 30 responses per target question and 10 per neighborhood question at temperature 0.7.Stress tests include direct answering, Chain-of-Thought prompting, and a second-turn Reflection.Source: §4.1 · p. 5

4.2 Metrics.

  • Coverage is the fraction of generated samples with a valid, non-refusal, non-empty entity prediction; accuracy is computed over valid predictions using a loose substring match to the gold answer.Table 3 reports Accuracy and Drop Rate across Standard, CoT, and Reflection settings for top and bottom NCB-ranked subsets.Source: §4.2 · p. 5, Table 3 · p. 6

4.3 Experimental Results and Analysis

  • Among the top/bottom 35% NCB-ranked groups, under single-instance interference (N=1), High NCB had lower Quantity Stressing accuracy drops than Low NCB: Qwen2.5 16.0% vs. 25.7%, and Qwen3 17.6% vs. 28.8%.Source: §4.3 · p. 5, Table 3 · p. 6
  • Under Peer Quantity–Conflict, accuracy drops from 97% to 62% for Low NCB and from 98% to 81% for High NCB as distractors rise from none to unanimous error.A single truth-teller improves performance over unanimous error in both groups; higher source credibility also reduces accuracy, with a larger drop for Low NCB.Source: §4.3 · p. 6, Figure 4 · p. 7
  • Reflection reduces accuracy drops in nearly every evaluated setting, whereas CoT is unstable and can worsen degradation, including Qwen2.5 Low NCB-35% from 25.7% to 31.6%.For Qwen3-Thinking Low NCB-35%, Reflection reduces the drop from 22.6% to 15.4%.Source: §4.3 · p. 6, Table 3 · p. 6
  • CoT effects vary nonlinearly with interference: for Qwen3 High NCB-35% under Source Stressing, CoT raises the drop rate from 14.6% to 33.9% at moderate interference, but the pattern is not monotonic.Reflection is described as relatively invariant across interference quantities.Source: §4.3 · p. 7, Figure 4 · p. 7
  • Across Qwen2.5 model sizes from 1.5B to 72B, High NCB remains more robust than Low NCB, with no clear trend in the performance gap.The authors conclude that increasing scale does not by itself resolve truthfulness.Source: §4.3 · p. 7, Figure 4 · p. 7

5 Structure-Aware Training

The section introduces Structure-Aware Training (SAT) to encourage newly learned facts to remain consistent across varied contexts. On a 100-fact evaluation set, SAT attains 93.0% accuracy and reduces average stress-test degradation by about 30% versus the best baseline.

5 Structure-Aware Training

  • The section motivates SAT by asking whether encouraging contextual invariance during learning improves the robustness of newly acquired knowledge.The reported outcome is that neighborhood-level invariance mitigates long-tail brittleness relative to training on isolated facts alone.Source: §5 · p. 7

5.1 Experimental Setup

  • The evaluation uses 100 facts the base model initially answers incorrectly, then applies the stress tests from Section 4 to compare SAT with answer-based and knowledge-based augmentation baselines.The baselines generate synthetic QA pairs from isolated target facts or supporting contextual evidence, respectively.Source: §5.1 · pp. 7–8
  • SAT trains a student to match a frozen teacher’s output distribution across semantically related neighbor contexts and general or noisy contexts, minimizing KL divergence.Both models start from the Answer-Based Augmentation checkpoint.Source: §5.1 · p. 8

5.2 Results

  • SAT reaches 93.0% ACC on newly learned facts and reduces average performance degradation by approximately 30% across stress tests versus the best baseline.The reported comparison is against training strategies that augment isolated answers or supporting contextual evidence.Source: §5.2 · p. 8, Table 4 · p. 8

6 Related Work

The related-work discussion covers confidence and belief estimation alongside contextual interference in LLMs. It describes limitations of existing confidence estimates and studies of how external information can shape model behavior.

Confidence and Belief Estimation in LLMs

Contextual Interference in LLMs

7 Conclusion

The conclusion frames robust belief as structured and presents NCB and SAT as approaches to evaluate and learn that structure.

7 Conclusion

  • The authors propose NCB to evaluate belief robustness and report that high-NCB knowledge is a robust cognitive anchor against social and authoritative interference.They also report that Structure-Aware Training robustly learns new knowledge.Source: §7 · p. 9

Limitations

The framework is limited in the relations, facts, and judgments it covers, and constructing belief neighborhoods adds computational overhead.

Scope of Neighbor Facts

  • The framework covers entity prerequisites, logical implications, and thematic associations, excluding causal chains and hierarchical taxonomies.Those more complex relations are excluded because they require domain-specific resources and could confound measurement under minimal contextual perturbations.Source: Limitations · p. 9

Static Knowledge Focus

  • Evaluation focuses on time-invariant facts and excludes dynamic facts and multi-hop reasoning.This isolates belief stability from temporal change but reduces direct applicability to real-time knowledge updates.Source: Limitations · p. 9

Human Alignment

  • NCB is an operational proxy for belief robustness, not an empirically validated measure of human-like comprehension.The authors identify validation against human judgments of genuine understanding as future work.Source: Limitations · p. 9

Computational Overhead

  • Constructing belief neighborhoods introduces nontrivial training- and inference-time computational overhead and scalability challenges.The authors call for optimization for practical deployment in large language models.Source: Limitations · p. 9

Ethical Statement

The paper reports ethical risks and methodological scope boundaries, while its appendix formalizes NCB, its assumptions, and its computable surrogate. It also outlines the benchmark construction pipeline.

Ethical Statement

  • The stress-testing protocols could be repurposed for adversarial attacks or misinformation, while automated neighborhood generation may inherit biases despite expert verification.The authors also flag risks of marginalizing specialized knowledge and reinforcing incorrect information or biases against corrective evidence.Source: Ethical Statement · p. 9

A Use of Large Language Models

B.1 Problem Definition

  • The formalization estimates whether a model’s belief state is structured or unstructured from its predictions on a target fact and neighborhood facts.It frames this as the posterior probability of structured belief conditioned on the target and neighbor answers.Source: §B.1 · p. 14

B.2 Key Assumptions

  • Under the stated assumptions, high neighbor consistency indicates strong posterior belief in structured semantic knowledge.The assumptions include similar target-answer accuracy across states and conditional independence of neighbor responses under structured belief.Source: §B.2 · pp. 14–15

B.3 Computable Surrogate

  • The computable NCB surrogate combines target correctness with a geometric mean of neighbor correctness, serving as a monotonic proxy for Bayesian odds.The geometric mean makes scores comparable across neighborhood sizes while preserving the multiplicative structure.Source: §B.3 · p. 15

B.4 Discussion

  • The derivation’s equal-baseline-accuracy assumption may fail in low-resource domains, and conditional independence may fail when neighbors overlap in reasoning paths.The authors say their curation minimizes the latter risk and suggest prior elicitation or copula models for more complex dependencies.Source: §B.4 · p. 15

C Data Construction Pipeline

C.1 Seed Data Sourcing

The benchmark pipeline sources and refines seed questions, generates truth-anchored neighbors, verifies them with experts, and creates misleading neighborhoods for stress tests.

C.1 Seed Data Sourcing

  • Seed samples came from SimpleQA, HotpotQA, and SciQ, with 500 distributed across four domains and filtering aimed at atomic, static, unambiguous facts.The pipeline selected easy HotpotQA questions, used high-confidence domain classification, and refined questions for time constraints, explicit disambiguation, and single intent.Source: §C.1 · p. 15

C.2 Neighbor Generation

  • DeepSeek-V3.2 generated truth-anchored neighbors testing entity prerequisites, logical implications, and thematic associations, with explicit entity names and automated checks.A blind solver checked factual retrievability; structural validation checked clarity, self-containment, and distinctness.Source: §C.2 · p. 16

C.3 Human-in-the-loop Verification

  • Candidates underwent web-retrieval filtering and review by three experts; inclusion required approval from at least two.Experts assessed factual unambiguity, logical relevance, and naturalness.Source: §C.3 · pp. 16–17

C.4 Misleading Set Creation

  • Misleading neighborhoods use a semantically close but incorrect entity as the anchor, creating internally coherent facts that are false for the original question.This consistency trap tests whether models distinguish internal consistency from factual truth.Source: §C.4 · p. 17

D.1 Model Specifications & Environment

The experiments specify the evaluated models and metrics, test belief robustness under contextual interference, and examine sensitivity to data properties and peer ordering. In a pilot, perfect self-consistency did not prevent accuracy from collapsing under a unanimous incorrect peer consensus.

D.1 Model Specifications & Environment

  • The experiments evaluated three Qwen models and one OLMo model, using vLLM with bfloat16 precision and 30 responses per original question.Generation used temperature T = 0.7 on a cluster equipped with eight NVIDIA A100 GPUs.Source: §D.1 · p. 17

D.2 Metrics Computation

  • Coverage measures the proportion of valid normalized responses, while accuracy is computed only on valid responses using loose substring matching.Reported accuracy averages sample-level accuracy scores across the evaluation dataset.Source: §D.2 · pp. 17–18

D.3 Contextual Interference Protocols

  • The interference protocols vary peer consensus from unanimous support to dissent and attribute misleading claims to sources with low, medium, or high credibility.The peer-quantity and source-credibility settings simulate social pressure and authority contexts, respectively.Source: §D.3 · p. 18

D.4 Training Experiment Settings

  • SAT used 30,000 training samples, with student inputs combining context and query while the teacher received the isolated augmented query.Contexts came from general C4 entries or synthesized passages built around neighboring questions; SAT was trained for one epoch, versus three for baseline models.Source: §D.4 · p. 18

E.1 Details of the Pilot Experiment

  • Despite perfect internal consistency (SC=1.0) in isolation, the model’s accuracy collapsed to 33.8% when faced with external pressure, demonstrating that point-wise confidence measures fail to capture latent brittleness.Source: §E.1 · p. 18

E.2 Case Study

  • High-confidence but fragile samples had SC > 0.8 and low NCB, a pattern the authors associate with rote memorization.Qualitative examples are reported in Table 5.Source: §E.2 · p. 19

E.3 Data Popularity and Difficulty Analysis

  • High-NCB samples had higher popularity (median ≈7) and lower difficulty, while low-NCB samples had lower popularity (median ≈4) and higher difficulty.The authors associate low NCB with attempts to memorize obscure, long-tail facts rather than structured understanding.Source: §E.3 · p. 19, Figure 7 · p. 19

E.4 Analysis of Positional Bias in Peer Contexts

  • Accuracy and coverage remained virtually invariant as the lone correct peer moved across all six context positions for both NCB groups.The analysis used the Peer Quantity cfg5 setting with five distractors and one truth-teller.Source: §E.4 · p. 19, Figure 8 · p. 19

E.5 Sensitivity Analysis of NFs’ Quantity and Weighting

This section examines whether NCB’s High–Low distinction persists under reduced neighbor quantities and alternative component weightings; the supplied material also includes synthetic-document generation instructions rather than additional findings.

E.5 Sensitivity Analysis of NFs’ Quantity and Weighting

  • Even with only 20% of Neighbor Facts, the High NCB group maintains a larger radar-chart coverage area than the Low NCB group.The study tested subsampling ratios of 20%, 40%, 80%, and 100%.Source: §E.5 · p. 19, Figure 10 · p. 20
  • Across balanced (1 : 1 : 1) and biased (e.g., 2 : 1 : 1) component weights, the High NCB group remains dominant.The results indicate insensitivity to the tested weighting schemes and support the authors’ view of robust belief as a holistic structural property.Source: §E.5 · p. 19, Figure 11 · p. 20

F.3.4 Synthetic Document Generation with Fact Embedding

  • Synthetic documents should feel real and organic, fit the supplied universe, avoid copied wording and filler, and preserve the target answer’s logic.If the premise cannot be rendered as a realistic document, the instruction is to respond with UNSUITABLE.Source: §F.3.4 · pp. 25–26
  • The output instructions require briefly planning in <scratchpad> tags before placing the final document in <content> tags.Source: §F.3.4 · p. 26
Loading 2601.05905v2…