PaperlayerView exact revision on arXiv

Source-linked AI summary

Don't Hallucinate, Abstain: Identifying LLM Knowledge Gaps via Multi-LLM Collaboration

Shangbin Feng, Weijia Shi, Yike Wang, Wenxuan Ding, Vidhisha Balachandran, Yulia Tsvetkov

arXiv:2402.00367v2cs.CL

TL;DR

LLM knowledge gaps may persist as knowledge evolves, creating a need to identify when models should abstain. The paper adapts existing calibration and adaptation methods, then proposes cooperative and competitive multi-LLM collaboration. Across three LLMs and four QA tasks, the proposed approaches improve abstention performance, while the paper limits its scope to knowledge-focused abstention and leaves unified knowledge-and-safety strategies for future work.

  • Problem

    Knowledge gaps may persist as LLM knowledge evolves, motivating methods to identify unreliable answers and abstain.

  • Method

    The paper adapts calibration, fine-tuning, and prompting baselines, then proposes COOPERATE and COMPETE for multi-LLM evaluation of generated answers.

  • Results

    Across three LLMs and four QA tasks, COOPERATE and COMPETE outperform baselines in 9 of 12 settings, achieving up to 19.3% higher abstain accuracy.

  • Takeaways & Limitations

    The collaboration mechanisms could help identify retrieval-augmentation failures and knowledge gaps in multi-hop reasoning.

  • Takeaways & Limitations

    The paper focuses on knowledge-oriented abstention and leaves unified strategies for knowledge and safety scenarios for future work.

Abstract

from arXiv · show

Despite efforts to expand the knowledge of large language models (LLMs), knowledge gaps -- missing or outdated information in LLMs -- might always persist given the evolving nature of knowledge. In this work, we study approaches to identify LLM knowledge gaps and abstain from answering questions when knowledge gaps are present. We first adapt existing approaches to model calibration or adaptation through fine-tuning/prompting and analyze their ability to abstain from generating low-confidence outputs. Motivated by their failures in self-reflection and over-reliance on held-out sets, we propose two novel approaches that are based on model collaboration, i.e., LLMs probing other LLMs for knowledge gaps, either cooperatively or competitively. Extensive experiments with three LLMs on four QA tasks featuring diverse knowledge domains demonstrate that both cooperative and competitive approaches to unveiling LLM knowledge gaps achieve up to 19.3% improvements on abstain accuracy against the strongest baseline. Further analysis reveals that our proposed mechanisms could help identify failure cases in retrieval augmentation and pinpoint knowledge gaps in multi-hop reasoning.

1 Introduction

LLM knowledge gaps can persist as knowledge evolves, motivating abstention when answers are unreliable. The paper evaluates existing abstention methods and proposes cooperative and competitive multi-LLM collaboration, reporting gains across diverse QA settings.

  • Knowledge gaps can cause confident hallucinations and biases, motivating abstention when LLM knowledge is missing or unreliable.
  • The paper frames AbstainQA as learning whether an LLM should abstain on a question based on limitations of its internal knowledge.
  • Existing baselines span calibration, training, prompting, and self-consistency approaches for incorporating abstention into LLMs.
  • Many stronger baselines require held-out data for training or tuning and rely on single-model self-reflection despite hallucination and confirmation-bias concerns.
  • COOPERATE and COMPETE use multi-LLM collaboration to evaluate proposed answers cooperatively or under conflicting evidence, without held-out sets.
  • Across three LLMs and four QA tasks, the collaboration approaches outperform baselines in 9 of 12 settings, with up to 19.3% higher abstain accuracy.

2 Identifying Knowledge Gaps in LLMs

The paper reviews abstention mechanisms based on calibration, model adaptation, prompting, and consistency, then introduces multi-LLM collaboration to address self-evaluation and held-out-set limitations. COOPERATE synthesizes feedback, while COMPETE tests whether conflicting alternatives overturn an original answer.

  • Existing abstention methods are categorized as calibration, training, prompting, and self-consistency approaches.
  • 2.1 Calibration-Based: Token-probability calibration sets an inference threshold from confidence scores, with threshold selection based on held-out abstention errors.
  • 2.1 Calibration-Based: Temperature scaling reshapes probability distributions before optimizing a calibrated threshold on held-out data.
  • 2.3 Prompting-Based: Prompting and tuning methods make abstention depend on verbalized confidence, requested information, answer-option matching, generated abstain messages, or a none-of-the-above choice.
  • 2.4 Consistency-Based: Self-consistency abstains when the plurality of multiple reasoning paths falls below a threshold optimized on a held-out set.
  • 2.5 Proposed: Multi-LLM Collaboration: COOPERATE gathers feedback from expert LLMs and uses a judge LLM to synthesize an accept-or-reject decision.
  • 2.5 Proposed: Multi-LLM Collaboration: COMPETE presents alternative answers generated by other LLMs and bases abstention on whether conflicting evidence changes preference for the original answer.

3 Experiment Settings

The experiments evaluate abstention strategies across three LLMs and four diverse QA datasets, using metrics that assess answer reliability, effective reliability, abstention decisions, and abstain F1.

  • Models: Three LLMs—Mistral-7B, LLaMA2-70B, and ChatGPT—are evaluated to test whether abstain mechanisms work across models with varying sizes and openness.The default sampling temperature is 0.1, increased to 0.7 when multiple runs are required.
  • Evaluation Setup: Table 1 compares abstain strategies across four datasets and three LLMs, with categories covering calibration, training, prompting, consistency, and collaboration.Incompatible cases are omitted, and a dash indicates failure to produce meaningful abstain decisions.
  • Tasks and Datasets: Four English datasets cover general knowledge QA, multi-hop reasoning, commonsense inference, and propaganda-tactic identification.The datasets are MMLU, Knowledge Crosswords, Hellaswag, and Propaganda.
  • Evaluation Metrics: The evaluation reports Reliable Accuracy, Effective Reliability, Abstain Accuracy, and Abstain F1 for abstention performance.These metrics respectively assess trustworthiness among answered questions, the balance between correct and incorrect coverage, abstention-decision correctness, and the harmonic mean of abstention precision and recall.

4 Results

The collaboration-based approaches outperform baselines across most model-task settings and improve abstention reliability, while exposing weaknesses in existing methods across domains and reasoning contexts.

  • Overall Results: COOPERATE and COMPETE outperform the strongest baseline in 9 of 12 settings, with an average 10.1% improvement in reliable accuracy.COOPERATE performs better with stronger models, whereas COMPETE emphasizes reliability by avoiding wrong answers.
  • Baseline Comparison: Calibration, training, consistency, and prompting approaches achieve average Abstain Accuracy scores of 0.595, 0.576, 0.553, and 0.475, respectively.Instruction tuning and self-consistency threshold are strong baselines but require held-out data for training or hyperparameter tuning.
  • Model Effects: Abstain Accuracy averages 0.524 for Mistral-7B, 0.537 for LLaMA2-70B, and 0.616 for ChatGPT across approaches and datasets.The collaboration approaches work best with ChatGPT while also improving weaker LLMs over baselines.
  • Robustness Across Tasks: Existing baselines can collapse on long propaganda prompts, hard Knowledge Crosswords, or low-base-performance settings, whereas collaboration approaches work across all four tasks.The results indicate broader compatibility for knowledge-intensive tasks spanning domains and reasoning contexts.

5 Analysis

The analysis examines abstention under absolute-abstention, retrieval, fairness, calibration, and multi-hop settings. COOPERATE and COMPETE generally remain robust with retrieval, while COMPETE is strongest for abstention recall and the proposed methods localize multi-hop knowledge gaps.

  • Abstain Absolute: COMPETE achieves the highest abstention rate on average across LLMs and datasets when all questions require abstention.This absolute-abstention setting includes ambiguous questions and temporally mismatched election questions.
  • Abstain Absolute: LLMs abstain less and assume future election outcomes more often for African and Asian countries, raising fairness concerns for marginalized communities and underdeveloped countries.ElectionQA23 performance is divided by the continent where the election takes place.
  • Abstain and Retrieval: COOPERATE and COMPETE are generally robust to retrieved context, whereas ASK CALIBRATE is negatively impacted by retrieval.The comparison uses MMLU with Mistral-7B, with and without retrieved documents.
  • Abstain and Retrieval: The abstain-retrieve-abstain pipeline reduces the incorrect rate by up to 56.6% by applying abstention before and after retrieval.A second abstention after retrieval indicates that retrieval failed to find relevant or helpful documents.
  • Abstain in Multi-Hop: COOPERATE and COMPETE localize multi-hop knowledge limitations, producing final error rates 67.2% to 81.2% lower than the strongest baselines.The approaches are applied separately to each hop in three-hop K-Crosswords questions.
  • Abstain ECE: COOPERATE and COMPETE provide well-calibrated abstain likelihood scores, while COMPETE emphasizes abstention recall and answer reliability.ASK CALIBRATE, COOPERATE, and COMPETE are top-performing approaches for abstain likelihood calibration; COMPETE has much higher abstain recall than most approaches.

6 Related Work

Prior abstention work uses calibration, prompting, and training, but often depends on held-out data or self-evaluation. The paper instead presents COOPERATE and COMPETE as multi-LLM collaboration mechanisms for robust abstention across direct, retrieval-augmented, and multi-hop QA.

  • Calibration-based approaches: Calibration-based approaches extract confidence scores and use held-out thresholds to decide whether LLMs should abstain.Reliance on held-out sets may jeopardize generalization across knowledge domains and reasoning contexts.
  • Prompting-based approaches: Prompting-based approaches use instructions to induce self-reflection and estimate whether generated answers should be trusted.Representative methods include none-of-the-above options, self-evaluation prompting, and self-correction of reasoning.
  • Training-based approaches: Training-based approaches add uncertainty or abstention abilities through special data, objectives, or external verifiers, but require substantial resources and may generalize poorly.Their limitations include computational cost and dependence on patterns and examples in training subsets.
  • Multi-LLM collaboration: COOPERATE and COMPETE use multi-LLM collaboration to promote mechanistic reflection and enable robust abstention in cooperation or competition.The approaches are reported to improve abstention in direct QA, retrieval-augmented QA, and multi-hop reasoning.

7 Conclusion

The paper studies AbstainQA, where LLMs should abstain from answering incorrectly, and organizes 11 baselines across four categories. It proposes COOPERATE and COMPETE, whose experiments support improved abstention with potential benefits for retrieval-augmented and multi-hop reasoning.

  • 7 Conclusion: AbstainQA evaluates whether LLMs should abstain from answering questions incorrectly.The paper curates a taxonomy of 11 abstain baselines across four categories.
  • 7 Conclusion: COOPERATE and COMPETE promote mechanistic reflection through multi-LLM collaboration in cooperative or competitive settings.The conclusion presents them as novel abstain mechanisms.
  • 7 Conclusion: Experiments on four datasets show that COOPERATE and COMPETE advance the state of the art in AbstainQA, with potential applications to retrieval augmentation and multi-hop reasoning.The stated scope includes direct QA, retrieval-augmented LLMs, and multi-hop knowledge reasoning.

Limitations

The paper’s scope is limited by its conceptual focus on knowledge-based abstention, incomplete coverage of existing approaches, computational overhead, inference variability, and possible test-set pollution.

  • The study primarily addresses knowledge-based abstention, while unified strategies covering both knowledge and safety scenarios remain future work.
  • The evaluated mechanisms and experiments do not cover every existing abstention approach because of computational-budget constraints.
  • Multi-LLM collaboration requires multiple model-inference requests, increasing computational overhead whose exact cost depends on the other LLMs used.
  • Test examples may have been unintentionally included in the training data of the adopted LLMs, so results should be interpreted with this limitation in mind.

Ethics Statement

The paper identifies fairness concerns in abstention, especially because election-related abstention errors vary across demographic regions and may reflect internal social biases.

  • LLMs fail to abstain and assume future election outcomes more frequently for African and Asian countries, creating fairness implications.
  • The authors expect abstention quality to vary across demographics, communities, and perspectives, potentially because of internal social biases.

A Analysis (cont.)

The analysis examines retrieval failure, multi-hop reasoning, held-out-set generalization, feedback aggregation, scaling, domain shifts, prompt robustness, and inference overhead. Collaboration performs strongly in several analyses, but abstention quality varies by model size and knowledge domain.

  • Retrieval Failure: The abstain-retrieve-abstain pipeline reduces incorrect answers across three LLMs and four datasets while accounting for retrieval failure.
  • Abstain and Multi-Hop: COOPERATE and COMPETE consistently better pinpoint knowledge limitations in specific reasoning steps, with abstention decisions matching multi-hop failures 73.4% of the time.
  • Held-Out Sets Harms Generalization: Instruction tuning struggles to generalize across knowledge domains and LLMs, with abstain accuracy drops sometimes reaching 33.8%, whereas collaboration requires no held-out set.
  • Minority Opinion: In 8 of 10 manually examined minority-opinion cases, the minority feedback contained new information that the final judge should consider.
  • Scaling Laws: Abstention performance does not steadily increase with LLAMA2 model size, indicating that scaling up alone does not solve abstention.
  • Shift in Knowledge Domains: Abstain accuracy varies from 25% to 87.5% across MMLU knowledge domains, potentially reflecting differences in underlying domain knowledge.
  • Prompt Robustness: COOPERATE and COMPETE are relatively robust to minor changes in prompt phrasing, while inference-request count is the main computational overhead.

B Experiment Details

The experiments use six QA datasets, including four main-task datasets and two absolute-abstention datasets, and evaluate mechanisms with three LLMs of varying size and openness.

  • The study employs six datasets: four for the main experiments and two for absolute-abstention evaluation.
  • The main datasets include MMLU, K-Crosswords, Hellaswag, and Propaganda, with held-out and test splits constructed from official data.
  • The absolute-abstention datasets are AmbigQA and ElectionQA23, covering ambiguous questions and elections held around the globe in 2023.

B.3 Implementation Details

The implementation evaluates abstention likelihoods using token probabilities, prompting, hidden-layer probing, external verification, and collaborative mechanisms. It also documents the prompts, examples, and evaluation settings used for these approaches.

  • Baseline likelihoods: Abstention likelihoods are derived from answer-token probabilities, calibrated probabilities, verbalized confidence, or reject probabilities depending on the baseline.Token Probability uses 1 − p(a), Temperature Scaling applies the same transformation after tuning τ, Ask for Calibration uses verbalized confidence, and Hidden Layers and External Verifier use reject likelihoods.
  • Training-based methods: Instruction Tuning estimates abstention from the probability of the answer token or the abstention phrase “sorry, I don’t have enough knowledge to answer this question.”ChatGPT is fine-tuned with instruction-tuning triplets that explicitly request abstention when knowledge is insufficient.
  • Prompting-based methods: Prompting-based methods estimate abstention from specific response probabilities, including “yes,” “no,” NOTA, or “B.”More Information uses “yes,” Generate and Match uses “no” or 1 − “yes,” None-of-the-Above uses NOTA, and Self-Reflect and Cooperate use “B.”
  • Self-consistency: Self-Consistency uses k = 5 chain-of-thought paths and computes abstention likelihood as 1 − p_lu(LLM, q, k).The implementation converts the self-consistency likelihood into an abstention score using five reasoning paths by default.
  • Prompts, examples, and evaluation views: The appendix supplies prompts and examples for Generate and Match, Cooperate, Compete, and More Information, alongside figures covering retrieval, multi-hop reasoning, and MMLU subtasks.The listed figures compare without retrieval, with retrieval, and retrieval+abstain settings; examine multi-hop reasoning; and show COMPETE performance across 57 MMLU subtasks.
Loading 2402.00367v2…