Source-linked AI summary
The Great Nugget Recall: Automating Fact Extraction and RAG Evaluation with Large Language Models
Ronak Pradeep, Nandan Thakur, Shivani Upadhyay, Daniel Campos, Nick Craswell, Jimmy Lin
TL;DR
RAG evaluation remains a barrier to progress, and this paper adapts nugget evaluation for RAG by using LLMs to create nuggets and assign them to system answers. On TREC 2024 RAG Track runs, fully automatic scores show strong agreement with human-based variants, with stronger agreement when assignment is automated independently.
Problem
Evaluation of RAG outputs remains deficient and lacks standardization, despite its importance to progress in information access.
Source: §1 · p. 1Method
AutoNuggetizer uses LLMs to create information nuggets and assign them to RAG system answers, calibrating fully automatic evaluation against human-involved strategies.
Source: §1 · p. 1Results
Takeaways & Limitations
The framework offers evaluation with varying degrees of human involvement, including scoring all TREC 2024 RAG Track runs across 301 topics at the cost of LLM inference.
Source: §6 · p. 9Per-topic correlations are quite low, limiting the framework’s suitability for fine-grained debugging of individual answers.
Source: §5.2 · p. 8
Abstract
from arXiv · showhide
Large Language Models (LLMs) have significantly enhanced the capabilities of information access systems, especially with retrieval-augmented generation (RAG). Nevertheless, the evaluation of RAG systems remains a barrier to continued progress, a challenge we tackle in this work by proposing an automatic evaluation framework that is validated against human annotations. We believe that the nugget evaluation methodology provides a solid foundation for evaluating RAG systems. This approach, originally developed for the TREC Question Answering (QA) Track in 2003, evaluates systems based on atomic facts that should be present in good answers. Our efforts focus on "refactoring" this methodology, where we describe the AutoNuggetizer framework that specifically applies LLMs to both automatically create nuggets and automatically assign nuggets to system answers. In the context of the TREC 2024 RAG Track, we calibrate a fully automatic approach against strategies where nuggets are created manually or semi-manually by human assessors and then assigned manually to system answers. Based on results from a community-wide evaluation, we observe strong agreement at the run level between scores derived from fully automatic nugget evaluation and human-based variants. The agreement is stronger when individual framework components such as nugget assignment are automated independently. This suggests that our evaluation framework provides tradeoffs between effort and quality that can be used to guide the development of future RAG systems. However, further research is necessary to refine our approach, particularly in establishing robust per-topic agreement to diagnose system failures effectively.
1 Introduction
The paper addresses the lack of standardized evaluation for RAG answers by adapting nugget evaluation with LLMs. It reports strong run-level agreement with human-based evaluations, while focusing only on nugget recall.
1 Introduction
- AutoNuggetizer uses LLMs to create nuggets and assign them to system answers, refactoring the two-stage nugget evaluation methodology.The paper frames this as a large-scale study of automating nugget evaluation for RAG systems.Source: §1 · p. 1
- The study asks how fully automatic evaluation compares with human-involved strategies, whether automating assignment alone improves agreement, and how human and LLM assignments differ.These questions are examined using diverse runs from the community-wide TREC 2024 RAG Track.Source: §1 · p. 1
- Fully automatic nugget evaluation shows strong run-level correlations with manual strategies, while automating assignment alone yields stronger agreement than automating both stages.The evaluation compares strategies involving fully manual nugget creation and post-editing of automatically generated nuggets.Source: §1 · pp. 1–2
- The study focuses exclusively on evaluating recall of information nuggets, leaving citation support and answer fluency outside its scope.The paper identifies these as other important aspects of RAG evaluation.Source: §1 · p. 2
2 Background and Related Work
Prior RAG evaluation work covers multiple quality dimensions and tasks, while nugget evaluation supplies a method for assessing atomic facts in answers. This paper applies and validates nugget evaluation for RAG answers using human annotations.
RAG Systems and Benchmarks; LLM-as-a-Judge Evaluation
- RAG benchmarks span factoid QA, long-form generation, and sources including knowledge graphs, multilingual documents, coding, tables, and images.Related work also includes LLM-based pairwise response comparison [33] and rubric-based evaluation [47].Source: §2 · p. 2
- RAG evaluation frameworks assess dimensions including faithfulness, answer relevance, context relevance, and retrieval effectiveness, while LLM judges also evaluate factuality and robustness.Examples include RAGAs, ARES, eRAG [36], and LLM-judge work [19] [36] [38].Source: §2 · p. 2
Nugget Evaluation Methodology
- Nugget evaluation began in the TREC QA Track in 2003, with later work extending it to summarization, question answering, and retrieval.Its focus on essential facts for coherent answers connects definition-question evaluation to RAG.Source: §2 · p. 2
Related Nugget Evaluation Frameworks
- Existing nugget-based frameworks often target retrieval or traditional tasks, whereas this study evaluates nuggets from relevant documents in generated answers against NIST annotations.The study uses the MS MARCO V2.1 segment collection, fresh queries, and high-quality human annotations from TREC 2024.Source: §2 · p. 2
3 The Setup of the TREC 2024 RAG Track
The TREC 2024 RAG Track provides the system-answer setting for the study, with 301 topics and distinct retrieval, augmented-generation, and full RAG tasks. The analysis uses answers from the AG and RAG tasks and a recently assembled query set.
3 The Setup of the TREC 2024 RAG Track
- The track gave participants 301 queries and required well-formed answers of up to 400 words for the AG and RAG tasks.The track also included a retrieval task as an intermediate product in a full RAG pipeline.Source: §3 · p. 2
- Table 1 presents an example answer for topic 2024-35227, with citations omitted because citation-support evaluation is outside the paper’s scope.The displayed answer is elided for length.Source: Table 1 · p. 3, §3 · p. 3
- The analysis focuses on generated answers from AG and RAG, where AG uses 100 organizer-provided retrieved results and RAG permits participant-chosen retrieval.The retrieval collection is the 113-million-passage MS MARCO V2.1 segment collection.Source: §3 · p. 3
- The 301 test queries are curated non-factoid Bing search-log queries selected near the July 2024 deadline to reduce relevance-judgment contamination.The paper notes that the web-based corpus may already appear in LLM pretraining data.Source: §3 · p. 3
4 The Nugget Evaluation Methodology
The methodology adapts nugget evaluation’s creation-and-assignment stages for LLM-based RAG evaluation. AutoNuggetizer generates nuggets from relevant documents and uses an LLM to determine whether system answers contain them.
Nugget Creation
- Nugget creation identifies facts that should appear in a good answer, distinguishing vital nuggets from okay, desirable ones.The approach follows the nugget methodology described by Voorhees.Source: §4 · p. 3
- AutoNuggetizer uses UMBRELA [43] to identify documents rated at least related to a query, then uses LLMs to generate AutoNuggets from them.The source documents are an explicit design choice for nugget creation.Source: §4 · p. 3
Nugget Assignment
- Nugget assignment checks whether each created nugget appears in a system answer, a stage performed automatically by an LLM in AutoNuggetizer.The assignment decision is semantic or conceptual rather than limited to lexical matching.Source: §4 · pp. 3–4
Nugget Creation
- Table 2 compares sample nuggets produced under different relevance judgments and post-editing conditions; the resulting nuggets are described as semantically similar.The comparison includes automatic relevance judgments, manual labels, and human post-editing.Source: Table 2 · p. 4
Nugget Assignment
- After assignment, the framework records which nuggets appear in each system answer, providing the basis for computing response and run scores.Table 3 compares assignment across nugget-creation approaches.Source: §4 · p. 4, Table 3 · p. 4
4.1 Nugget Creation
Nugget creation ranges from iterative LLM generation to human-authored nuggets, with a semi-manual option that uses generated nuggets as a draft. The approaches differ in how relevance is assessed and how much assessor effort they require.
4.1.1 Automatic Nugget Creation.
- Figure 1 depicts the iterative nuggetization prompt, while Figure 2 shows the prompt for assigning nugget importance, with at most 10 nuggets presented per turn.Nuggets are sorted by descending importance before the first 20 are selected.Source: Figure 1 · p. 4, §4.1.1 · p. 4, Figure 2 · p. 5
4.1.2 Semi-Manual Nugget Creation.
- In AutoNuggets+Edits, NIST assessors post-edit automatically proposed nuggets generated from documents humans judged at least related to the query.Assessors may add, eliminate, or combine nuggets; they receive 30 without importance labels, and the task takes roughly an hour per topic on average.Source: §4.1.2 · p. 5
- Different automatic-generation inputs and post-editing approaches produce nugget sets described as semantically quite similar despite some differences.The compared sets use automatically generated or human relevance labels, with the post-edited set based on human relevance judgments.Source: §4.1.2 · p. 5
4.1.3 Manual Nugget Creation.
- ManualNuggets has NIST assessors create and importance-label at most 20 nuggets from documents judged at least related to the query, taking roughly 2.5 hours per topic.The procedure follows the TREC 2003 QA Track approach.Source: §4.1.3 · p. 5
4.2 Nugget Assignment
AutoAssign uses an LLM to label nugget support in system answers, while manual assignment uses a NIST assessor. The section also illustrates these approaches with two assignment-and-qrel combinations.
4.2 Nugget Assignment
- AutoAssign uses GPT-4o to label each nugget as supported, partially supported, or unsupported by an answer, processing at most 10 nuggets per turn.The assignment is performed iteratively using a listwise approach.Source: §4.2.1 · p. 5
4.2.1 Automatic Nugget Assignment.
- Manual assignment has the NIST assessor who created the nuggets judge each answer without seeing LLM output, using the same three support labels.The assessor assigns support, partial_support, or not_support.Source: §4.2.2 · p. 5
4.2.2 Manual Nugget Assignment.
- Table 3 contrasts AutoAssign on AutoNuggets with UMBRELA qrels against ManualAssign on edited AutoNuggets with NIST qrels.The example’s evaluated answer is shown in Table 1.Source: §4.2.2 · p. 5
4.3 Evaluation Strategies
The evaluation strategies combine automatic or human nugget creation with manual or automatic assignment. This design includes fully manual and fully automated conditions and isolates the effect of automating assignment.
4.3 Evaluation Strategies
- AutoNuggets+Edits / ManualAssign uses automatically generated nuggets refined by humans, followed by manual assignment by NIST assessors.The source documents were judged at least related by NIST assessors.Source: §4.3 · p. 6
- ManualNuggets / ManualAssign is the fully manual baseline, with NIST assessors creating nuggets from relevant passages and assigning them to system answers.This mirrors the original TREC QA approach.Source: §4.3 · p. 6
- AutoNuggets / AutoAssign generates nuggets from documents marked relevant by UMBRELA and assigns them automatically, forming the end-to-end automated condition.Source: §4.3 · p. 6
- The semi-manual-creation conditions replace manual nugget assignment with AutoAssign, isolating automation of the assignment step.This applies to both AutoNuggets+Edits and ManualNuggets.Source: §4.3 · p. 6
4.4 Scoring
Scoring starts from nugget-support judgments and aggregates query scores into a run score. The section defines strict scoring variants and limits its evaluation scope to nugget recall.
4.4 Scoring
- Each answer’s nuggets are judged as support, partial_support, or not_support before scoring.Source: §4.4 · p. 6
- A run score is the mean of its per-query scores.Source: §4.4 · p. 6
- All Strict scores an answer using strict matching across all nuggets, while Vital Strict is the primary metric and applies strict matching to vital nuggets.Source: §4.4 · p. 6
- The reported scores evaluate nugget recall only, excluding citation support and fluency.Source: §4.4 · p. 6
5 Results
Manual and automatic nugget evaluations agreed strongly in run-level rankings, but agreement was weaker across topics. Automating assignment alone improved topic-level agreement, while assignment disagreements and implementation-specific limits remained.
5.1 Descriptive Statistics
- AutoNuggets conditions averaged around 18 nuggets per topic versus around 14 for edited or manual nuggets, while nugget lengths stayed near 7–8 tokens.Humans were more conservative with “vital” labels; automatic assignment showed more Partial Support, a shift that may reflect the current prompt.Source: §5.1 · p. 6, Table 4 · p. 7
5.2 Fully Automatic Nugget Evaluation
- Kendall’s τ was 0.887 for Vstrict and 0.901 for Astrict under AutoNuggets+Edits / ManualAssign, versus 0.727 and 0.758 under ManualNuggets / ManualAssign.The ManualNuggets condition covered 20 topics; a 20-topic AutoNuggets+Edits subset yielded 0.826 for Vstrict and 0.838 for Astrict.Source: §5.2 · p. 7, Figure 4 · p. 7
- Kendall’s τ fell to 0.297–0.438 across topic/run combinations and 0.360–0.539 on average per topic, indicating greater topic-level variability.The authors say per-topic correlations are too low for fine-grained debugging of individual answers.Source: §5.2 · pp. 7–8
5.3 Automatic Nugget Assignment
- For Vstrict, per-topic average Kendall’s τ rose from 0.490 with fully automatic AutoNuggets/AutoAssign to 0.662 with only nugget assignment automated, while all-topics/runs τ increased from 0.328 to 0.599.Source: §5.3 · p. 8
- Automating assignment also improved per-topic and all-topics/runs agreement with manually created nuggets, supporting effort–quality choices beyond fully automatic evaluation.The pattern appeared with both manually created and automatically generated, post-edited nuggets.Source: §5.3 · p. 8
5.4 Nugget Assignment Confusion Matrices
- For AutoNuggets+Edits, 47.6% of pairs were jointly labeled “No Support” and 23.8% “Support”; for ManualNuggets, the corresponding shares were 54.0% and 22.1%.Figure 6 presents the assignment confusion matrices.Source: Figure 6 · p. 8, §5.4 · p. 9
- Across both nugget sets, AutoAssign assigned “Partial Support” more often than ManualAssign, while the two sets showed similar agreement patterns.The authors characterize LLM assessors as somewhat stricter than NIST assessors and report that draft nuggets did not noticeably increase alignment.Source: §5.4 · p. 9
- The assignment findings reflect one assessor and one LLM implementation; variation across different assessors and models remains unmeasured.The authors note that another LLM or prompt could behave differently.Source: §5.4 · p. 9
6 Conclusions
AutoNuggetizer builds on the established nugget methodology and achieves strong run-level agreement with manual and semi-manual evaluations. The fully automatic condition scored all TREC 2024 RAG Track runs across 301 topics at the cost of LLM inference, while isolated nugget-assignment automation showed stronger agreement, especially per topic.
6 Conclusions
- AutoNuggetizer achieves strong run-level agreement with manual and semi-manual evaluations, suggesting viability as a scalable alternative for assessing RAG systems.The framework leverages LLMs for automatic nugget creation and assignment.Source: §6 · p. 9
- Building on the nugget evaluation methodology dating back over two decades minimizes reinvention of the wheel.The authors note that evaluation aspects not dependent on LLMs can build on existing findings.Source: §6 · p. 9
- The AutoNuggets / AutoAssign condition scored all TREC 2024 RAG Track runs across 301 topics at only the cost of LLM inference.Human assessment remains prohibitively expensive.Source: §6 · p. 9
- Automating nugget assignment in isolation yields stronger agreement, especially per topic, with manual evaluations than full framework automation.The authors also report that LLMs apply stricter assignment labels than human assessors, indicating a need for calibration.Source: §6 · p. 9