PaperlayerView exact revision on arXiv

Source-linked AI summary

Measuring and Enhancing Trustworthiness of LLMs in RAG through Grounded Attributions and Learning to Refuse

Maojia Song, Shang Hong Sim, Rishabh Bhardwaj, Hai Leong Chieu, Navonil Majumder, Soujanya Poria

arXiv:2409.11242v4cs.CL

TL;DR

The paper addresses the challenge of assessing whether LLMs are suitable for RAG, where responses are expected to rely on provided documents. It introduces TRUST-SCORE and TRUST-ALIGN, and reports that TRUST-ALIGN beats competitive baselines on TRUST-SCORE in 26 of 27 model-family and dataset configurations.

  • Problem

    The paper studies whether LLMs ground responses in provided documents, while existing RAG evaluations can conflate LLM performance with retrieval quality.

    Source: §1 · p. 2

  • Method

    The paper introduces TRUST-SCORE for answer, citation, and refusal groundedness, and TRUST-ALIGN, which aligns models using a constructed dataset and direct preference optimization.

    Source: §1 · p. 2

  • Results

    26 of 27 model-family and dataset configurations show TRUST-ALIGN outperforming competitive baselines on TRUST-SCORE across ASQA, QAMPARI, and ELI5.

    Source: §1 · p. 2

  • Takeaways & Limitations

    DPO-trained LLaMA-2-7b and LLaMA-3-8b significantly reduce hallucinations in RAG and perform comparably to major closed-source models such as GPT-4.

    Source: §7 · p. 10

    The study assumes complete groundedness, requiring all claims to be derivable from retrieved documents, and frames the task as information retrieval.

    Source: §2.3 · p. 3

Abstract

from arXiv · show

LLMs are an integral component of retrieval-augmented generation (RAG) systems. While many studies focus on evaluating the overall quality of end-to-end RAG systems, there is a gap in understanding the appropriateness of LLMs for the RAG task. To address this, we introduce Trust-Score, a holistic metric that evaluates the trustworthiness of LLMs within the RAG framework. Our results show that various prompting methods, such as in-context learning, fail to effectively adapt LLMs to the RAG task as measured by Trust-Score. Consequently, we propose Trust-Align, a method to align LLMs for improved Trust-Score performance. 26 out of 27 models aligned using Trust-Align substantially outperform competitive baselines on ASQA, QAMPARI, and ELI5. Specifically, in LLaMA-3-8b, Trust-Align outperforms FRONT on ASQA (up 12.56), QAMPARI (up 36.04), and ELI5 (up 17.69). Trust-Align also significantly enhances models' ability to correctly refuse and provide quality citations. We also demonstrate the effectiveness of Trust-Align across different open-weight models, including the LLaMA series (1b to 8b), Qwen-2.5 series (0.5b to 7b), and Phi3.5 (3.8b). We release our code at https://github.com/declare-lab/trust-align.

1 INTRODUCTION

The paper studies whether LLM responses in RAG are grounded in provided documents and introduces TRUST-SCORE and TRUST-ALIGN to evaluate and improve that behavior. TRUST-ALIGN outperforms competitive baselines on TRUST-SCORE in 26 of 27 model-family and dataset configurations.

1 INTRODUCTION

  • TRUST-SCORE evaluates grounded refusals, answer correctness, claim support by citations, and citation relevance while isolating LLM performance from retrieval quality.Existing end-to-end RAG metrics can conflate retriever quality with LLM performance.Source: §1 · pp. 1–2
  • Prompting approaches intended to improve groundability can leave models overly sensitive to prompts, producing exaggerated refusals or excessive responsiveness.The paper reports that many state-of-the-art systems also rely heavily on parametric knowledge when answering.Source: §1 · p. 2
  • 26 of 27 model-family and dataset configurations show TRUST-ALIGN outperforming competitive baselines on TRUST-SCORE across ASQA, QAMPARI, and ELI5.For LLaMA-3-8b, gains over FRONT are 12.56% on ASQA, 36.04% on QAMPARI, and 17.69% on ELI5.Source: §1 · p. 2
  • The paper frames grounded RAG responses as answers derived from retrieved documents rather than the model’s parametric knowledge.A response is grounded when it correctly answers using only the documents and its inline citations support the response.Source: §1 · pp. 1–2
  • TRUST-ALIGN builds a 19K-example dataset of paired preferred and unpreferred responses, then aligns models with direct preference optimization.The dataset addresses inaccurate answers, over-responsiveness, excessive refusal, over-citation, and improper citation.Source: §1 · p. 2

2 PROBLEM DESCRIPTION

The problem setup represents an LLM response as citation-grounded statements or a refusal when retrieved documents are insufficient. It defines answerability from document entailment and treats unsupported or otherwise ungrounded responses as hallucinations within a complete-groundedness setting.

2.1 TASK SETUP

  • Given a question and retrieved documents, the LLM produces citation-grounded statements; when the documents are insufficient, the gold response is a refusal.Each statement has inline citations referring to documents in the retrieved set.Source: §2.1 · p. 3

2.2 ON ANSWERABILITY OF A QUESTION

  • A question is answerable when a subset of retrieved documents entails at least one gold claim; otherwise, the expected response is a refusal.A refusal contains no claims or citations and conveys inability to answer.Source: §2.2 · p. 3

2.3 HALLUCINATION IN LLM IN RAG

  • The paper defines hallucinations as ungrounded responses, including inaccurate answers, over-responsiveness, excessive refusal, over-citation, and improper citation.Its setting assumes complete groundedness: all claims should be derivable from documents, making the task an information-retrieval task.Source: §2.3 · p. 3

3 METRICS FOR LLM-IN-RAG

The section measures whether LLM responses stay within the evidence in retrieved documents, including appropriate refusals and citation support. It also calibrates response scoring to avoid distortions from claims unavailable in the documents.

Insufficiency of the existing metrics

  • Existing answer-recall scoring can reward parametric claims absent from retrieved documents and gives samples different recall ceilings, hindering consistent dataset-wide comparison.A model relying only on documents is limited to the gold claims obtainable from those documents.Source: §3 · p. 4

Answer Calibration

  • PAC averages calibrated answer-correctness scores over answered samples, while RAC averages them over answerable samples; together they form F1AC.Calibrating gold claims to those obtainable from documents sets the maximum recall to 1 for all models and addresses recall consolidation and gamification.Source: §3 · p. 4

Scoring refusals

  • F1GR averages refusal and answerable-case F1 scores, measuring whether models refuse unanswerable questions and answer answerable ones.The metric uses dataset-wide precision and recall for both ground-truth answerable cases and refusals.Source: §3 · p. 4

Measuring attribution groundedness

  • F1GC is the harmonic mean of citation precision and recall, measuring whether citations support generated statements and are relevant to them.Citation recall checks statement support with an NLI model; citation precision scores citations for each statement.Source: §3 · p. 4

Responsiveness

  • Responsiveness is the fraction of questions answered, with high rates expected for answerable questions and low rates for unanswerable ones.The expected rates should align with the dataset distribution.Source: §3 · p. 5

4 THE TRUST-ALIGN DATASET

TRUST-ALIGN builds paired preferred and unpreferred responses for question-document examples, then uses them to align a model with DPO. Its data construction moves from curated questions and documents through augmented examples to response pairs.

Dataset construction

  • TRUST-ALIGN constructs question-document-response preference pairs, using a document-grounded answer or refusal as preferred and a contrasting response as unpreferred.The resulting 19K-sample dataset is used for DPO alignment.Source: §4 · pp. 5–6

Collecting quality questions

  • About 10K seed questions are selected from ASQA, QAMPARI, and ELI5 training splits after clustering and quality scoring.Clusters with scores of 4 or higher are selected.Source: §4 · p. 5

Collecting D’s

  • Relevant documents are retrieved from Wikipedia and Common Crawl, and five oracle documents are selected to match the full 100-document set on EM recall.Seed questions without retrieved relevant documents are filtered out; gold claims come from the source datasets.Source: §4 · pp. 5–6

Augmenting (q,D) set

  • Varying relevant and irrelevant document combinations produce about 70K question-document pairs, including unanswerable examples with similar but irrelevant documents.Document order is shuffled to avoid citation bias.Source: §4 · p. 6

Obtaining r+ and r−

  • GPT-4 produces preferred answers by stitching together gold claims and citations, while a fine-tuned LLaMA-2-7b model supplies hallucinated negative responses.From 40K identified hallucinated responses, the highest-severity samples are selected for both answerable and unanswerable cases before DPO.Source: §4 · p. 6

5 EXPERIMENTAL SETUP

The experiments compare vanilla and TRUST-ALIGNed open-weight models on several question-answering datasets, using established RAG and attribution approaches as baselines. The setup also reports hallucination-type fractions and severity computations for the negative-response model.

Models studied; Baselines

  • The study compares vanilla and TRUST-ALIGNed LLaMA, Qwen, and Phi3.5 open-weight models against ICL, PostCite, PostAttr, Self-RAG, and FRONT baselines.The baseline approaches include demonstrations, post-hoc citation methods, retrieval with reflection tokens, and fine-grained attribution training.Source: §5 · p. 6

Hallucination analysis

  • Table 1 reports each hallucination type’s fraction among 40,985 observed hallucinations in Msft and its severity computation, noting that types may overlap.The caption points to Figure 5 for a detailed breakdown of the last three error types.Source: Table 1 · p. 6

Evaluation datasets

  • Evaluations use ASQA, QAMPARI, and ELI5 test sets, plus ExpertQA for out-of-distribution testing, with the top five retrieved documents appended to each question.ELI5 and ExpertQA gold answers are decomposed into three claims.Source: §5 · p. 6

6 RESULTS AND ANALYSIS

TRUST-ALIGN improves TRUST-SCORE over competitive baselines across nearly all tested configurations, with gains in grounded refusals and citations. Analyses also show the importance of refusal data and report performance across model families, sizes, and ExpertQA.

6 RESULTS AND ANALYSIS

  • 26 of 27 configurations show higher TRUST-SCORE with TRUST-ALIGN; LLaMA-3-8b gains over FRONT are 12.56% (ASQA), 36.04% (QAMPARI), and 17.69% (ELI5).The results cover the LLaMA family on ASQA, QAMPARI, and ELI5.Source: Table 2 · p. 7, §6 · p. 7
  • LLaMA-3-8b improves over FRONT in F1GR by 23.87% (ASQA), 47.95% (QAMPARI), and 45.72% (ELI5), and in F1GC by 22.12%, 38.35%, and 5.55%, respectively.F1GR improves in all 27 configurations, while F1GC improves in 24 of 27.Source: Table 2 · p. 7, §6 · p. 7
  • F1AC increases with TRUST-ALIGN in 9/9 QAMPARI configurations but is mixed on ELI5 (5/9) and ASQA (2/9), partly because answer recall and RAC penalize refusals.For LLaMA-3.2-3b on ASQA, answerable-question recall is 89.02% for TRUST-ALIGN versus 98.69% for FRONT.Source: §6 · pp. 7–8
  • TRUST-ALIGN generalizes across model sizes and families; DPO exceeds SFT on TRUST-SCORE in 26/27 configurations and on F1GC in 25/27.For example, Qwen-2.5-0.5b rises from 22.83% to 54.76% on ASQA; for LLaMA-3.2-3b, DPO gains 6.70% on ASQA TRUST-SCORE over SFT.Source: Table 3 · p. 8, §6 · p. 8, Table 4 · p. 9

6.1 ANALYSIS

  • Adding samples for each of five hallucination types raises TRUST-SCORE by 1.50% (ASQA), 1.78% (QAMPARI), and 2.23% (ELI5).The ablations group the error types into answer-, citation-, and refusal-related classes.Source: §6.1 · p. 8, Table 5 · p. 9
  • TRUST-ALIGN beats GPT-4-critic training for LLaMA-2-7b by 1.29% (ASQA), 1.77% (QAMPARI), and 4.12% (ELI5); on ExpertQA, it beats FRONT in all 27 open-source configurations.On ExpertQA, TRUST-ALIGNed LLaMA-3-8b reaches 54.85% TRUST-SCORE, versus GPT-4 at 54.69%.Source: §6.1 · p. 9, Table 7 · p. 10
  • Removing refusal samples lowers TRUST-SCORE by 10.2% (LLaMA-3-8b) and 11.41% (LLaMA-2-7b), with F1GR declines of 26.34% and 26.97%, respectively.F1GC also declines by 6.87% for LLaMA-3-8b and 6.57% for LLaMA-2-7b.Source: Table 6 · p. 9, §6.1 · p. 9
  • Higher answer responsiveness is associated with greater parametric-knowledge use, and GPT-4 can have higher answer correctness but lower attribution groundedness than the authors’ models.The authors caution that Sparam only partially captures parametric-knowledge use; for example, GPT-4 may rely on parametric knowledge even when documents contain the answer.Source: §6.1 · p. 10

7 CONCLUSION

The paper introduces TRUST-SCORE to evaluate LLM suitability for document-grounded RAG and TRUST-ALIGN, which uses preference data and DPO to improve that score.

7 CONCLUSION

  • TRUST-SCORE measures answer quality, citation quality, and refusal performance, while TRUST-ALIGN uses a constructed dataset and DPO to improve the score.The authors report training LLaMA-2-7b and LLaMA-3-8b with DPO and reducing hallucinations in RAG.Source: §7 · p. 10

A NUANCES OF ANSWERABILITY

  • For generic comprehension, answerability is judged by exact claim matching and an ML entailment model; the NLI choice depends on whether the application requires mathematical understanding.The paper recommends using the same NLI model for constructing gold claims and evaluating responses to avoid improper evaluations.Source: A NUANCES OF ANSWERABILITY · p. 17

B ANSWERABILITY: A CASE STUDY

  • Substring matching can fail in complex, long contexts, so the TRUE-based method semantically checks matching claims and skips that check when substring matching fails.The latter condition reduces the computational cost of using TRUE for semantic entailment.Source: B ANSWERABILITY: A CASE STUDY · p. 17

C.1 ATTRIBUTABLE RETRIEVAL AUGMENTED GENERATION

  • TRUST-SCORE prioritizes responses grounded strictly in provided documents, aiming to minimize unverifiable content and strengthen output reliability.Source: §C.1 · p. 18

C.2 ENHANCE GROUNDED TEXT GENERATION IN ATTRIBUTED LARGE LANGUAGE MODELS

  • TRUST-ALIGN separates answer generation from citation quality in its multi-step data generation, a design the paper says improves answer quality, citation accuracy, and refusal precision.The approach also addresses inappropriate refusals as a hallucination error.Source: §C.2 · pp. 18–19

D METRICS

The metrics evaluate response truthfulness and citation groundedness, then average component scores into a single TRUST-SCORE.

D.1 RESPONSE TRUTHFULNESS

  • Grounded Refusal F1 macro-averages correct refusal of unanswerable questions and correct answering of answerable ones, penalizing both over- and under-responsiveness.Source: §D.1 · p. 19
  • Answer-calibrated correctness disregards claims that cannot be inferred from the documents and scores exact-match recall on the remaining gold claims.This adjustment avoids rewarding correct answers that were not located in the provided documents.Source: §D.1 · p. 20

D.2 ATTRIBUTION GROUNDEDNESS

  • Grounded Citation F1 assesses statement-wise citation recall and per-citation precision, including whether cited documents support the statements they accompany.Citation precision is zero only when one citation fails to entail the statement while the other citations collectively entail it without that citation.Source: §D.2 · p. 20
  • Grounded Citation F1 aggregates precision and recall over answered questions only, and TRUST-SCORE averages the component metrics into one model-ranking score.Source: §D.2 · p. 21

E THE TRUST-ALIGN DATASET

TRUST-ALIGN builds preference pairs from question-document samples, with positive responses grounded in available evidence and negative responses selected to reflect hallucinations.

E THE TRUST-ALIGN DATASET

  • Each TRUST-ALIGN sample pairs a question and retrieved documents with preferred and unpreferred responses; preferred responses include supported gold claims or refuse when the documents are insufficient.The construction proceeds through question selection, document collection, sample augmentation, positive-response construction, and negative-response construction.Source: E THE TRUST-ALIGN DATASET · p. 21

E.1 COLLECTING QUALITY QUESTIONS

  • The seed set prioritizes diverse, challenging questions by clustering them, scoring cluster difficulty, and sampling from clusters scoring at least 4; the resulting set contains about 10K questions.The target counts are 3K each for ASQA and QAMPARI and 4K for ELI5.Source: §E.1 · p. 21

E.2 COLLECTING D’S

  • For ASQA and QAMPARI, the pipeline retrieves 100 Wikipedia documents per seed question; for ELI5, it uses BM25 with Sphere and filters questions without relevant retrieved documents.Source: §E.2 · pp. 21–22
  • A question is answerable when the union of document entailment patterns contains at least one supported gold claim; five oracle documents can match the 100-document set’s exact-match recall.Oracle documents are selected as being equally effective for exact-match recall.Source: §E.2 · p. 22

E.3 AUGMENTING (q,D) SET

  • About 70K question-document pairs are built by varying relevant-document patterns for answerable questions and using similar but non-entailing documents for unanswerable ones; document order is shuffled.Source: Figure 3 · p. 22, §E.3 · p. 22

E.4 OBTAINING r+

  • For each dataset, GPT-4 labels each gold claim used with its index; those labels map to documents by greedily selecting the smallest combination covering the claims, minimizing over-citation.The fine-tuned LLaMA-2-7b model hallucinated in 40K responses among approximately 70K questions; the top 50% by severity yielded 19K samples used for DPO.Source: §E.4 · p. 23, §E.5 · p. 23

F.1 REVISED METRICS ARE LESS BIASED

Across these analyses, TRUST-ALIGN is more robust than prompting, while calibrated metrics better account for refusals and model comparisons reveal both strengths and remaining limits.

F.1 REVISED METRICS ARE LESS BIASED; F.2 UTILIZATION OF PARAMETRIC KNOWLEDGE

  • Calibrated F1AC reduces answer-at-all-costs bias, narrowing performance gaps or revealing stronger model results; on ASQA, GPT-4’s F1GC is five points below its F1AC advantage over our models.The passage reports GPT-4 has higher F1AC but attribution groundedness F1GC five points lower than the aligned models.Source: Figure 6 · p. 24, §F.1 · pp. 24–25, §F.2 · p. 25

F.3 THE SOURCE OF LLM HALLUCINATIONS

  • Except for non-responsive LLaMA-2-7B, ICL models more often produce erroneous answers from parametric knowledge than the authors’ models; errors present in documents remain indeterminate in source.The analysis also reports baseline models including GPT-4 and GPT-3.5 are more susceptible to parametric-knowledge hallucinations.Source: §F.3 · pp. 25–26

F.4 TRUST-ALIGN ENHANCES TRUSTWORTHINESS MORE ROBUSTLY THAN PROMPTING

  • TRUST-ALIGN improves TRUST-SCORE more robustly than prompting, which can drive refusal rates toward extremes and reduce citation groundedness.Refusal prompts have inconsistent effects; post-hoc methods show little change in TRUST-SCORE or F1GR.Source: §F.4 · p. 26

F.5 COMPARISON WITH CLOSED-SOURCE MODELS

  • Aligned models outperform GPT-3.5 and Claude-3.5 on ASQA and closed-source models on QAMPARI, but trail ICL-GPT-4 by 2.45 TRUST-SCORE points on ELI5.On ELI5, the authors also report lower F1AC and F1GC for their models.Source: §F.5 · p. 27, Table 12 · p. 27
  • The authors hypothesize decomposed claims may impair extraction and precise citations; GPT-3.5 and GPT-4 outperform on long-form QA primarily because of base-model capacity limits.The decomposition concern is framed as a hypothesis about information density, redundancy, or inadequate claim coverage.Source: §F.5 · p. 27

F.6 ADAPTABILITY WITH DIFFERENT ALIGNMENT TECHNIQUES; F.7 EVALUATION DATA CREATION WITHOUT USING TRUE

F.8 EFFECT OF DATA SIZE ON DPO PERFORMANCE

  • 57.73 on QAMPARI and 47.55 on ELI5 were achieved using the top 25% of augmented samples; the authors report no clear “more data is better” trend.They suggest lower-severity samples may add limited complexity and redundancy, and that using the top 50% may cause overfitting.Source: §F.8 · p. 30, Figure 7 · p. 30

F.9 FINE-TUNING GPT-4O

G GPT-4 BASED DATA PIPELINE

The GPT-4 data pipeline revises answer coverage and statement-level citations; experiments use specified retrieval, decoding, and fine-tuning settings.

G GPT-4 BASED DATA PIPELINE

  • A GPT-4 critic first corrects or supplements response content against correct answers, then validates and corrects statement-level citation attributions.The pipeline concatenates citation-revised statements into a final revised response.Source: G GPT-4 BASED DATA PIPELINE · pp. 30–31

H.1 IMPLEMENTATION DETAILS

  • Experiments use the top five retrieved documents, temperatures 0.1 and 0.5, full-parameter tuning, and two training epochs for both SFT and DPO.The reported setup uses DPO beta 0.5 and NVIDIA A40 40G GPUs.Source: §H.1 · p. 31

H.2 DATASET DETAILS

The appendix describes evaluation datasets, baseline procedures, and how refusals are detected.

H.2 DATASET DETAILS

  • The evaluation uses 100-word document passages, allows at most three citations per claim, and excludes empty responses.The datasets include ASQA, QAMPARI, ELI5, and an unseen ExpertQA set used to test generalizability.Source: §H.2 · pp. 31–32, Table 17 · p. 31

H.3.1 IN-CONTEXT LEARNING (ICL)

  • ICL demonstrations pair each query with the top five retrieved passages and an answer containing inline citations.Source: §H.3.1 · p. 32

H.3.2 POST-HOC SEARCH (POSTCITE)

  • PostCite generates a closed-book response, then uses GTR to match each statement to a citation from the top five passages.Source: §H.3.2 · p. 32

H.3.3 POST-HOC ATTRIBUTE (POSTATTR)

  • PostAttr generates a closed-book response, then uses a TRUE NLI model to match citations to statements from the top five passages.Source: §H.3.3 · p. 32

H.3.4 SELF-RAG

  • Self-RAG uses reflection tokens to retrieve documents on demand and self-reflection to enhance generation quality.The comparison uses released 7B and 13B models with default settings.Source: §H.3.4 · p. 32

H.3.5 FRONT

  • FRONT grounds supporting quotes before generating cited responses, using automatically collected ChatGPT data with quality filtering.The authors reproduce its 7B model for evaluation.Source: §H.3.5 · p. 32

H.4 REFUSAL DETECTION

  • Refusal prompts use an explicit refusal phrase detected by fuzzy matching, while default-prompt refusals are classified by GPT-4o after human-checked fuzzy matching performed poorly.The refusal-prompt phrase is “I apologize, but I couldn’t find an answer to your question in the search results.”Source: §H.4 · pp. 32–33, Table 18 · p. 32

I DETAILED RESULTS

The appendix points to detailed experiment-result tables and prompt-template tables.

I DETAILED RESULTS

J PROMPT TEMPLATES

Related papers

Loading 2409.11242v4…