Source-linked AI summary
Overview of the TREC 2025 Retrieval Augmented Generation (RAG) Track
Shivani Upadhyay, Nandan Thakur, Ronak Pradeep, Nick Craswell, Daniel Campos, Jimmy Lin
TL;DR
The TREC 2025 RAG Track addresses evaluation of systems responding to complex, multi-sentence information needs with retrieval and generation. It uses a layered assessment framework, and reports that the best relevance-judgment runs achieve agreement fractions of 0.30–0.34 with human judgments.
Problem
Short keyword-style queries offer a limited fit for deep-search needs involving nuanced interpretation, broader evidence coverage, and reasoning-driven responses.
Source: §1 · p. 1Method
The track evaluates retrieval and generation on MS MARCO V2.1 narratives through relevance, response-completeness, citation-support, and agreement assessments.
Source: §1 · pp. 1–2Results
Best relevance-judgment runs achieved agreement fractions of 0.30–0.34 against human judgments, indicating limited agreement even among top runs.
Source: §4.3 · p. 12Takeaways & Limitations
The framework provides fine-grained assessment of retrieval quality, attribution accuracy, and response completeness for narrative-driven queries and attribution-rich answers.
Source: §1 · p. 2
Abstract
from arXiv · showhide
The second edition of the TREC Retrieval Augmented Generation (RAG) Track advances research on systems that integrate retrieval and generation to address complex, real-world information needs. Building on the foundation of the inaugural 2024 track, this year's challenge introduces long, multi-sentence narrative queries to better reflect the deep search task with the growing demand for reasoning-driven responses. Participants are tasked with designing pipelines that combine retrieval and generation while ensuring transparency and factual grounding. The track leverages the MS MARCO V2.1 corpus and employs a multi-layered evaluation framework encompassing relevance assessment, response completeness, attribution verification, and agreement analysis. By emphasizing multi-faceted narratives and attribution-rich answers from over 150 submissions this year, the TREC 2025 RAG Track aims to foster innovation in creating trustworthy, context-aware systems for retrieval augmented generation.
1 Introduction
The track shifts from short keyword queries to complex narratives and evaluates retrieval-augmented systems for relevance, completeness, factual grounding, and agreement. Its layered manual and automated assessment is intended to support context-rich, trustworthy responses.
1 Introduction
- The track builds on the inaugural TREC 2024 RAG Track to advance research on retrieval and generation for complex, real-world information needs.The introduction places the effort within TREC’s broader role in information retrieval and natural language processing evaluation.Source: §1 · p. 1
- TREC 2025 replaces short keyword-style queries with long, multi-sentence narratives intended to reflect deep-search needs and demand broader evidence coverage and reasoning.The track asks participants to preserve transparency and factual grounding in generated outputs.Source: §1 · p. 1
- The track continues to use the segmented, deduplicated MS MARCO V2.1 corpus for retrieval and generation tasks.The corpus is intended to support evidence retrieval and responses grounded in verifiable sources.Source: §1 · p. 1
- The track combines relevance assessment, response completeness, sentence-level citation support, and agreement analysis to evaluate retrieval-augmented systems.The framework combines manual and automated assessment.Source: §1 · p. 2
2 Task Setup
The track organizes participation around four tasks on the segmented MS MARCO V2.1 collection, with narratives serving as the information need. It also provides a multistage reference retrieval pipeline and semi-automates narrative and sub-narrative creation.
2 Task Setup
- Participants receive 105 narratives and submit answers of up to 400 words for Augmented Generation and end-to-end RAG; the track also includes Retrieval and the new Relevance Judgment task.The tasks use the MS MARCO V2.1 segmented collection, designed with deduplication and fine-grained segmentation.Source: §2 · p. 2
- Retrieval returns ranked segment IDs; Augmented Generation uses a fixed top-k list, while end-to-end RAG systems supply their own retrieval and generation pipelines.Relevance Judgment scores documents from 0 to 4 according to how well they address the narrative’s information need.Source: §2 · p. 3
- The reference lists fuse SPLADE-v3 and Arctic-Embed-L retrieval with reciprocal rank fusion, then apply RankLLM and RankQwen3-32B reranking.The pipeline selects the top 100 documents after reranking the top 1,000 candidates from the preceding stage.Source: §2 · pp. 3–4
- Narratives are generated by transforming clusters of related queries into coherent, multi-sentence descriptions of real-world research needs.The paper illustrates this with a sports query cluster expanded to cover societal impact, compensation, inclusion, culture, and business.Source: §2 · p. 4
- Sub-narratives are generated semi-automatically in two LLM stages, then reviewed and post-edited by human annotators for completeness, coherence, and scope alignment.GPT-4.1 nano drafts candidate sub-narratives from the narrative and its query list; GPT-4.1 refines the list.Source: §2 · p. 5
3 Evaluation
The evaluation decomposes narratives into sub-narratives and measures document relevance, response coverage, sentence-level citation support, and agreement between automated and human judgments. These stages use both manual and automated assessments.
3.1 Relevance Assessment
- Assessors label segments on a five-point relevance scale: 0 means unrelated, 1 general but non-specific, 2 one sub-narrative, 3 two or three, and 4 four or more [7].Decomposing narratives into focused sub-narratives helps assessors evaluate relevance in a structured and consistent manner.Source: §3.1 · p. 5
3.2 Response Evaluation
- Response evaluation generates and assigns nuggets, then maps them to sub-narratives; strict vital recall counts vital nuggets fully supported by the response.Automated and post-edited nugget settings vary who generates, edits, and assigns nuggets or sub-narratives.Source: §3.2 · p. 7
3.3 Support Evaluation
- Support evaluation checks each answer sentence against its cited document, grading support as full, partial, or none and restricting human assessment to the highest-ranked citation.Weighted precision measures the weighted proportion of citations that support sentences; weighted recall measures the weighted proportion of sentences supported by citations.Source: §3.3 · p. 8
3.4 Relevance Judgment Evaluation
- Relevance Judgment evaluation compares participant labels with human judgments and uses Cohen’s kappa to measure agreement beyond chance when labels share the 0–4 scale.The comparison also examines ties and disagreements between human judgments and participant labels.Source: §3.4 · p. 9
4 Results
Automated evaluation generally aligns with manual assessments at the run level, while agreement varies by metric and evaluation component. Relevance-judgment submissions show limited agreement with human judgments even among the best runs.
4.1 Retrieval Task
- Retrieval-task rank agreement is measured with Kendall’s τ by comparing scores from manual and automated assessment artifacts.The retrieval results cover 22 narratives selected by NIST; assessors evaluated documents from only the three highest-priority submissions per group due to time constraints.Source: §4.1 · p. 9
- Automated retrieval assessments approximate manual system rankings at the run level, with stronger correspondence at larger cutoffs, particularly nDCG@100.Agreement is less consistent in narrative-level comparisons.Source: §4.1 · p. 10
4.2 AG and RAG Task
- AG and RAG scores are reported across manual and automated response-evaluation settings, with Figure 7 comparing the fully manual setting against three alternative configurations.The alternatives vary automation of nugget generation, relevance judgments, and nugget assignment.Source: §4.2 · pp. 11–12
- Strict vital scores show the weakest alignment between fully manual and fully automated evaluation, but the strongest correlation when nugget assignment alone is automated.The latter comparison holds other components fixed.Source: §4.2 · p. 12
- Sub-narrative coverage has comparable run-level correlations across the fully automated and NIST-qrels configurations, but automated nugget assignment lowers run-level correlations.That assignment also reduces point-level scatter, while altering run ordering.Source: §4.2 · p. 12
- GPT-OSS 120B assessments show strong run-level alignment with NIST assessors for weighted recall and weighted precision, despite individual-narrative dispersion.The passage characterizes the model as a reliable run-level surrogate for these metrics.Source: §4.2 · p. 12
4.3 Relevance Judgment (RJ) Task
- The best relevance-judgment runs achieve agreement fractions of 0.30–0.34 against human judgments, indicating limited agreement even among top submissions.The task received 36 runs from 5 groups.Source: §4.3 · p. 12
5 Conclusion
The TREC 2025 RAG Track shifts evaluation toward complex, long-context narratives and received over 150 submissions. Its multi-layered framework assesses retrieval quality, factual grounding, and response completeness to support research on trustworthy AI systems.
5 Conclusion
- The track received over 150 submissions focused on deep reasoning, evidence synthesis, and transparent attribution for multi-sentence information needs.Its evaluation framework assesses retrieval quality, factual grounding, and response completeness.Source: §5 · p. 12