PaperlayerView exact revision on arXiv

Source-linked AI summary

Internal Consistency and Self-Feedback in Large Language Models: A Survey

Xun Liang, Shichao Song, Zifan Zheng, Hanyu Wang, Qingchen Yu, Xunkai Li, Rong-Hua Li, Yi Wang, Zhonghao Wang, Feiyu Xiong, Zhiyu Li

arXiv:2407.14507v3cs.CL

TL;DR

Reasoning deficiencies and hallucinations are difficult to distinguish in some evaluation examples. The paper synthesizes internal consistency through a Self-Feedback framework and finds an hourglass pattern in answer consistency across latent, decoding, and response levels.

  • Problem

    Reasoning deficiencies and hallucinations are difficult to distinguish in financial question-answer examples, including examples from MMLU and TruthfulQA.

    Source: A. Lack Reasoning and Exhibit… · p. 2

  • Method

    The paper synthesizes internal consistency and organizes Self-Feedback studies by tasks and lines of work.

    Source: §XI · p. 17

  • Results

    Answer consistency follows an hourglass pattern from lower to higher latent layers, through decoding, and finally at the response level.

    Source: B. The Hourglass Evolution of Internal… · p. 5

  • Takeaways & Limitations

    The lines of work help researchers locate their work and facilitate reasonable experimental comparisons.

    Source: §XI · p. 17

    Current evaluation methods generally have a singular focus, making comprehensive understanding of model capabilities difficult.

    Source: E. The Comprehensive Evaluation · p. 17

Abstract

from arXiv · show

Large language models (LLMs) often exhibit deficient reasoning or generate hallucinations. To address these, studies prefixed with "Self-" such as Self-Consistency, Self-Improve, and Self-Refine have been initiated. They share a commonality: involving LLMs evaluating and updating themselves. Nonetheless, these efforts lack a unified perspective on summarization, as existing surveys predominantly focus on categorization. In this paper, we use a unified perspective of internal consistency, offering explanations for reasoning deficiencies and hallucinations. Internal consistency refers to the consistency in expressions among LLMs' latent, decoding, or response layers based on sampling methodologies. Then, we introduce an effective theoretical framework capable of mining internal consistency, named Self-Feedback. This framework consists of two modules: Self-Evaluation and Self-Update. The former captures internal consistency signals, while the latter leverages the signals to enhance either the model's response or the model itself. This framework has been employed in numerous studies. We systematically classify these studies by tasks and lines of work; summarize relevant evaluation methods and benchmarks; and delve into the concern, "Does Self-Feedback Really Work?" We also propose several critical viewpoints, including the "Hourglass Evolution of Internal Consistency", "Consistency Is (Almost) Correctness" hypothesis, and "The Paradox of Latent and Explicit Reasoning". The relevant resources are open-sourced at https://github.com/IAAR-Shanghai/ICSFSurvey.

I. INTRODUCTION

The introduction treats inconsistent expression, deficient reasoning, and hallucination as connected challenges, then presents Internal Consistency Mining and Self-Feedback as a unified framing for addressing them. It positions this framework against related surveys through its internal-consistency perspective and taxonomy of lines of work.

I. INTRODUCTION

  • LLMs can produce inconsistent responses, illogical reasoning on out-of-distribution problems, and overconfident claims despite uncertainty about their limits.The paper locates inconsistency at the response, decoding, and latent levels.Source: §I · p. 1

A. Lack Reasoning and Exhibit Hallucination

  • Reasoning deficiencies and hallucinations are difficult to distinguish in examples, motivating Internal Consistency Mining as a unified perspective on both.The paper notes overlap between financial question-answer examples in MMLU and TruthfulQA.Source: A. Lack Reasoning and Exhibit… · pp. 1–2

B. Self-Feedback to Promote Internal Consistency

  • Self-Feedback combines Self-Evaluation of outputs, logits, or latent states with Self-Update of responses or model behavior.Examples include majority voting after generating multiple answers and resolving contradictions among diverse outputs.Source: B. Self-Feedback to Promote Internal… · p. 2

C. Related Surveys

  • The survey distinguishes itself by reviewing internal consistency and organizing methods into lines of work alongside a Self-Feedback framework.It also analyzes whether Self-Feedback works, aiming to position research and support experimental comparisons.Source: C. Related Surveys · pp. 2–3

D. Structure of the Survey

The survey organizes work on low internal consistency and Self-Feedback, then defines internal consistency through sampling-based expressions across model layers. It also sets boundaries around the topics and methods reviewed.

D. Structure of the Survey

  • The survey traces low internal consistency from its manifestations and proposed sources to Internal Consistency Mining and Self-Feedback, then reviews evaluation, effectiveness, and future directions.It covers signal acquisition and seven lines of work on reasoning elevation and hallucination alleviation.Source: D. Structure of the Survey · p. 3

E. Out-of-scope Topics

  • The survey excludes interaction between internal and external consistency, including conflicts between model-parameter knowledge and user context, and does not actively discuss RAG.Its focus is hallucinations arising from internal consistency.Source: E. Out-of-scope Topics · p. 3

II. INTERNAL CONSISTENCY

  • Internal consistency is modeled by sampling LLM expressions, with response, decoding, and latent consistency treated as distinct expression types.The expression types correspond to text, token choices, and activations or latent states across the model architecture.Source: A. Formulation · pp. 4–5

A. Formulation

  • For a model and query, the survey defines a sampling set of expressions from which consistency can be estimated.Samples may repeat; the paper describes negative variance and negative entropy as simple measures.Source: A. Formulation · pp. 4–5
  • The survey includes common notations in Table III.Source: Table III · p. 4
  • Simple variance and entropy may not guide response updates effectively and apply only when expressions are numerical labels.Source: A. Formulation · p. 5

B. The Hourglass Evolution of Internal Consistency

A sampling experiment across latent, decoding, and response layers finds an hourglass evolution in answer consistency. The authors interpret the pattern as convergence during latent processing, high certainty in decoding, and renewed variability in responses.

B. The Hourglass Evolution of Internal Consistency

C. Status Quo of LLM Consistency

The survey describes inconsistency in answers and refusals, and reviews prompt sensitivity, weak latent reasoning, and theoretical accounts as possible sources. It relates internal consistency to several dimensions of AI reliability.

C. Status Quo of LLM Consistency

  • Self-Consistency majority voting increased GSM8K answer accuracy by about 17.9%, while GPT-4 induced self-contradictions at a rate of 15.7% in a reported experiment.Models also show low consistency in refusing unanswerable “I Don’t Know” questions compared with humans.Source: C. Status Quo of LLM Consistency · p. 6

D. Sources of Low Internal Consistency

  • Prompt variation, long-context position effects, and dispersed attention are discussed as sources of inconsistent responses and reasoning paths.The survey also describes weak latent reasoning and a snowball effect in which an initial reasoning error can compound under full attention.Source: D. Sources of Low Internal Consistency · p. 6

E. How to Understand Internal Consistency?

III. SELF-FEEDBACK FRAMEWORK

Self-Feedback combines Self-Evaluation, which obtains feedback, and Self-Update, which uses it to improve responses or models. The survey classifies work by tasks and lines of work, including consistency-signal acquisition.

A. Formulation

  • Self-Feedback comprises Self-Evaluation and Self-Update, and can improve a model’s own consistency through feedback from a model instance or response.In a broader interpretation, feedback among different models can also be used to improve their capabilities.Source: A. Formulation · p. 7
  • Self-Evaluation obtains feedback from a sampling set, and Self-Update can revise the original expression, optimize model parameters, or update another model.Source: A. Formulation · p. 7
  • External signals during Self-Evaluation or updates to other models during Self-Update constitute generalized Self-Feedback.Source: A. Formulation · p. 7

B. Taxonomy

  • The survey classifies reviewed work by tasks and lines of work rather than dividing it by Self-Evaluate, Self-Update, and feedback signal.Source: B. Taxonomy · p. 7
  • Consistency Signal Acquisition is treated as an important task because many Self-Feedback methods overlook how the feedback signal is obtained.The survey classifies Self-Consistency’s feedback as scalar-based Consistency Estimation.Source: B. Taxonomy · p. 7
  • The taxonomy covers reasoning elevation for QA, hallucination alleviation for open-ended generation, and other tasks such as knowledge distillation and embedding generation.Source: B. Taxonomy · p. 7

IV. TASK: CONSISTENCY SIGNAL ACQUISITION

Consistency signal acquisition evaluates sampled expressions and supplies signals for updating responses or model parameters. The survey organizes methods by the model access they require and covers scalar, textual, contrastive, and external feedback.

IV. TASK: CONSISTENCY SIGNAL ACQUISITION

  • Signal acquisition evaluates expression consistency after sampling and supports updates to responses or model parameters; methods are grouped as black-, gray-, or white-box by access to outputs, logits, or latent states.Source: §IV · p. 7
  • Verbal critiques provide suggestions, external feedback draws on sources such as models or compiler errors, and contrastive optimization compares expressions to promote consistency.Source: §IV · p. 8

A. Uncertainty Estimation

  • Uncertainty methods estimate data, model, or distributional uncertainty; sampling-based approaches infer it from variation across model expressions, including variance across outputs from dropout-generated models.Source: A. Uncertainty Estimation · p. 8
  • A simple sampling-based uncertainty signal is the proportion of unique responses in the sample set, used by Active Prompting [60].Source: A. Uncertainty Estimation · p. 8

B. Confidence Estimation

C. Hallucination Detection

The section surveys ways to obtain consistency signals and use them for reasoning elevation or hallucination alleviation. It describes scalar hallucination checks, textual critique, contrastive selection, and feedback from external sources.

C. Hallucination Detection

  • SelfCheckGPT samples alternative responses, checks whether they support the original, and aggregates support into a hallucination score using five proposed metric families.Source: C. Hallucination Detection · p. 9
  • Other detection approaches measure variability in intermediate-layer latent vectors with INSIDE or use two LLMs to cross-examine factual claims.Source: C. Hallucination Detection · p. 9

D. Verbal Critiquing

E. Contrastive Optimization

  • Contrastive optimization scores sampled responses and selects the highest-scoring candidate; examples span truthful attention-head probes, decoding-path evaluation, and response-level consistency, [71].Source: E. Contrastive Optimization · p. 9

F. External Feedback

  • External feedback can include executor results for code, stronger or peer models, [47], and retrieved information used by RAG.Source: F. External Feedback · p. 9

V. TASK: REASONING ELEVATION

  • Reasoning elevation targets logical reasoning during response generation and is commonly evaluated with question-answering benchmarks.Source: §V · p. 9

A. Reasoning Topologically

The section reviews reasoning topologies, response refinement, and multi-agent collaboration as approaches to consistency across reasoning and generation. It also identifies limits involving task simplicity, cost, deployment, and dialogue stopping.

A. Reasoning Topologically

  • X-of-Thought methods range from direct input-output and stepwise CoT to majority-vote Self-Consistency, branching ToT, and thought aggregation in GoT.Source: A. Reasoning Topologically · p. 10
  • MPSC [76], Universal SC [77], and Soft SC [71] adapt aggregation for multiple solution perspectives, model-based response selection, and soft labels, respectively.Source: A. Reasoning Topologically · p. 10
  • Quiet-STaR [39] uses user-invisible sampled rationales to support next-token reasoning, while prompt optimizers extend topology to the input stage, an area most thought-selection methods overlook.Source: A. Reasoning Topologically · p. 10
  • Topology methods are reported to use simple experimental tasks, incur high reasoning costs, and struggle with general tasks and deployment [73].Source: A. Reasoning Topologically · p. 11

B. Refining with Responses

  • Response-refinement work includes fine-tuning on selected self-generated answers or corrected mistakes, and using NLI probabilities with Max-SAT to optimize consistent responses.Source: B. Refining with Responses · p. 11

C. Multi-Agent Collaboration

  • Multi-agent approaches include iterative debate and generator-critic feedback, or a consensus game that uses generator-discriminator agreement rather than direct dialogue, [89].Source: C. Multi-Agent Collaboration · p. 11
  • Multi-agent methods can require multiple LLM instances and extensive context; fixed dialogue rounds may be inflexible, and the survey notes no flexible, efficient stopping criterion.Source: C. Multi-Agent Collaboration · p. 11

VI. TASK: HALLUCINATION ALLEVIATION

  • Hallucination alleviation targets open-ended generation, including story and code generation, with aims such as improving factuality, reducing errors, and enhancing faithfulness.Source: §VI · p. 11

A. Refining the Response Iteratively

The section reviews iterative response refinement, granular hallucination mitigation during generation, and decoding strategies for truthful output. Across these approaches, feedback may come from the model, external verification, or adjusted token probabilities.

A. Refining the Response Iteratively

  • Self-Refine, Reflexion, and Self-Correct iteratively generate textual feedback and update responses until a stopping condition or iteration limit is reached.Source: A. Refining the Response Iteratively · p. 12
  • Reflexion treats refinement as verbal reinforcement learning and separates task-specific feedback signals from textual reflection; Self-Correct trains a dedicated corrector, limiting task generality or flexibility.Source: A. Refining the Response Iteratively · p. 12

B. Mitigating Hallucination while Generating

C. Decoding Truthfully

  • Contrastive decoding [95] contrasts expert and amateur token distributions; DoLa uses earlier-layer information, while SED [11] and DIVER [54] detect token-level “Chaotic Points” and score candidate tokens.Source: C. Decoding Truthfully · p. 13
  • For open-book generation, CAD [53] and ECAD [96] use relevant or irrelevant knowledge snippets to guide decoding through contrastive strategies.Source: C. Decoding Truthfully · p. 13

D. Activating Truthfulness

The survey describes latent truthfulness interventions and applies the Self-Feedback lens to preference learning and other work. It also identifies white-box hallucination alleviation as a research direction.

D. Activating Truthfulness

  • Truthfulness interventions identify relevant latent features, then adjust attention-head weights or feed-forward layers to improve truthful representations while limiting performance impact.ITI identifies truthfulness-related attention heads and increases their weights; TrFr uses multidimensional orthogonal probes, while TruthX also targets feed-forward layers.Source: D. Activating Truthfulness · p. 13
  • White-box hallucination alleviation may benefit from interpretability research; whether enhancing long-context retrieval heads reduces hallucinations remains an open question.Blocking retrieval-focused attention heads reduced Needle-in-a-Haystack performance from 94.7% to 63.6%.Source: D. Activating Truthfulness · p. 13

VII. TASK: OTHERS

  • The survey also includes Self-Feedback-related work that does not necessarily target internal consistency.This statement introduces the section’s broader coverage of other tasks.Source: §VII · p. 13

A. Preference Learning

  • Preference learning fits Self-Feedback: reward models turn human, model-generated, or inductive feedback into signals for updating models through methods such as supervised fine-tuning or reinforcement learning.The survey notes that response-level reward scores can make it difficult to distinguish responses across different questions; another approach directly scores QA pairs.Source: A. Preference Learning · pp. 13–14

B. LLM-Based Knowledge Distillation

The survey treats distillation and data augmentation as Self-Feedback applications, then reviews evaluation methods and benchmarks. It reports generally low consistency and weak self-knowledge in evaluated models.

B. LLM-Based Knowledge Distillation

  • In knowledge distillation, a teacher evaluates student outputs and signals guide student updates; when teacher and student are the same model, the approach is Self-Knowledge Distillation.Signals can be statistical measures such as KL divergence or textual feedback; Self-KD can evaluate paraphrases for semantics, format, and diversity before fine-tuning.Source: B. LLM-Based Knowledge Distillation · p. 14

C. Data Augmentation

  • Data augmentation focuses on refining datasets through response updates, and current methods mainly use prompt-level evaluation, leaving prompt design and signal-to-data-quality relations as bottlenecks.Self-Instruct, for example, uses ROUGE-L between new and existing instructions as feedback before filtering generated instructions.Source: C. Data Augmentation · p. 14

VIII. EVALUATION

  • The evaluation section distinguishes meta evaluations of model abilities from common-task evaluations of Self-Feedback methods.Meta abilities include uncertainty, consistency, and feedback; common tasks include reasoning QA and code generation.Source: §VIII · p. 14

A. Meta Evaluation

  • Evaluations cover uncertainty, semantic consistency, critique, and self-knowledge; studies generally find low response consistency and weak self-knowledge.Benchmarks include synonymous-query and multilingual consistency tests, CriticBench for critique generation, and unanswerable-question sets for self-knowledge.Source: A. Meta Evaluation · p. 15

B. Common Evaluation

  • Self-Feedback methods are commonly evaluated on real-world tasks including reasoning, code generation, and mathematical problem solving.The survey points to a common-evaluation benchmark table and a separate LLM evaluation survey.Source: B. Common Evaluation · p. 15

IX. DOES SELF-FEEDBACK REALLY WORK?

The survey reviews conflicting evidence about self-correction, argues that internal consistency is not identical to correctness, and offers standards for clearer research comparisons.

A. Conflicting Viewpoints

  • Critiques report weak self-assessment, failed verification, and disputed gains from Reflexion, Multi-Agent Debate, and Self-Refine; self-correction may be feasible for decomposable or verifiable tasks.The survey notes concerns about external truth requirements, resource use, and unfair Self-Refine prompts.Source: A. Conflicting Viewpoints · p. 15
  • The survey answers yes to whether Self-Feedback improves internal consistency, while withholding a direct conclusion that consistency means correctness.It characterizes criticisms as limited in light of their feedback signals, tasks, and test models, while noting that self-correction feasibility depends on task properties.Source: A. Conflicting Viewpoints · p. 15

B. Does Internal Consistency Mean Correctness?

  • Because consistency reinforces both correct and erroneous knowledge, the survey’s “Consistency Is (Almost) Correctness” hypothesis depends on pretraining corpora being predominantly aligned with correct world knowledge.It argues that within-distribution Self-Feedback can reinforce corpus priors, but models struggle to self-evaluate and update beyond their generalization capability.Source: B. Does Internal Consistency Mean… · p. 16

C. Appeals

  • The survey appeals for distinct, accurate names; standardized task terms; separate labels for reasoning and hallucination; same-direction baselines; realistic settings; and prompt robustness tests across LLMs.It specifically cautions against setups requiring pre-given gold labels.Source: C. Appeals · p. 16

X. FUTURE DIRECTIONS AND CHALLENGES

The future agenda spans textual uncertainty expression, balancing latent and explicit reasoning, deeper layer analysis, unified cross-layer framing, and broader evaluation. The conclusion positions internal consistency and Self-Feedback as lenses for organizing work on reasoning and hallucinations.

A. Textual Self-Awareness

  • Using internal consistency signals for textual self-awareness is a proposed direction [142], against reported uncertainty-expression and calibration gaps [140] [141].Yona et al. report that models cannot verbally and faithfully express uncertainty; Kapoor et al. report good calibration only after fine-tuning.Source: A. Textual Self-Awareness · p. 16

B. The Reasoning Paradox

  • Single-token latent reasoning can be accurate, while explicit CoT helps complex problem-solving but lengthy noisy chains disrupt latent reasoning; their equilibrium is under-studied [26].The paper calls for studying that equilibrium to use reasoning resources efficiently and improve reasoning efficiency.Source: B. The Reasoning Paradox · pp. 16–17

C. Dive Deeper

  • Many of the seven summarized lines optimize only the response layer, relying on experience and prompt templates; the authors encourage work on decoding and latent layers.The paper also notes that low entry barriers and extensive participation have brought an influx of low-quality papers.Source: C. Dive Deeper · p. 17

D. The Unified Perspective

  • The field lacks a comprehensive, unified account across response, decoding, and latent layers, though emerging work connects layers through CoT paths [29] and response-guided decoding.The paper argues that considering the layers together can better facilitate Internal Consistency Mining.Source: D. The Unified Perspective · p. 17

E. The Comprehensive Evaluation

  • Because LLMs and Self-Feedback strategies yield many combinations, single-focus evaluation methods make capabilities difficult to understand comprehensively and conveniently.The paper identifies a complete evaluation system spanning meta- to common evaluation, latent states to responses, benchmarks to metrics, and uncertainty to feedback as a consideration.Source: E. The Comprehensive Evaluation · p. 17

XI. CONCLUSION

  • The paper organizes reasoning and hallucination work through internal consistency and Self-Feedback, summarizing tasks, lines of work, evaluation, benchmarks, and future directions.Its stated aim is to help researchers locate work within the broader system and facilitate reasonable experimental comparisons.Source: §XI · p. 17

Related papers

Loading 2407.14507v3…