PaperlayerView exact revision on arXiv

Source-linked AI summary

Ragnar\"ok: A Reusable RAG Framework and Baselines for TREC 2024 Retrieval-Augmented Generation Track

Ronak Pradeep, Nandan Thakur, Sahel Sharifymoghaddam, Eric Zhang, Ryan Nguyen, Daniel Campos, Nick Craswell, Jimmy Lin

arXiv:2406.16828v1cs.IRcs.AIcs.CL

TL;DR

RAG systems use retrieved context to produce grounded LLM answers, but proprietary offerings and limited customization complicate standardized RAG development. Ragnarök provides a reusable framework, collection and topic resources, and industrial baselines; on development topics, GPT-4o provides more detailed answers than Command R+.

  • Problem

    Proprietary end-to-end RAG systems offer limited customization, and the lack of a standardized framework makes large-scale implementation challenging.

    Source: §1 · p. 1

  • Method

    Ragnarök is an open-source, reusable end-to-end RAG pipeline, accompanied by MS MARCO V2.1 resources, development topics, and industrial baselines.

    Source: §1 · p. 1, §3 · p. 2, §6 · p. 4

  • Results

    On development topics, GPT-4o provides more detailed answers than Command R+ with the BM25 and RankZephyr retrieval setup.

    Source: §1 · p. 1

  • Takeaways & Limitations

    The authors open-source Ragnarök and its baselines with the aim of standardizing RAG systems.

    Source: §8 · p. 4

Abstract

from arXiv · show

Did you try out the new Bing Search? Or maybe you fiddled around with Google AI~Overviews? These might sound familiar because the modern-day search stack has recently evolved to include retrieval-augmented generation (RAG) systems. They allow searching and incorporating real-time data into large language models (LLMs) to provide a well-informed, attributed, concise summary in contrast to the traditional search paradigm that relies on displaying a ranked list of documents. Therefore, given these recent advancements, it is crucial to have an arena to build, test, visualize, and systematically evaluate RAG-based search systems. With this in mind, we propose the TREC 2024 RAG Track to foster innovation in evaluating RAG systems. In our work, we lay out the steps we've made towards making this track a reality -- we describe the details of our reusable framework, Ragnar\"ok, explain the curation of the new MS MARCO V2.1 collection choice, release the development topics for the track, and standardize the I/O definitions which assist the end user. Next, using Ragnar\"ok, we identify and provide key industrial baselines such as OpenAI's GPT-4o or Cohere's Command R+. Further, we introduce a web-based user interface for an interactive arena allowing benchmarking pairwise RAG systems by crowdsourcing. We open-source our Ragnar\"ok framework and baselines to achieve a unified standard for future RAG systems.

1 INTRODUCTION

RAG can ground LLM answers in retrieved context, but proprietary systems and limited standardization make large-scale implementation difficult. Ragnarök provides a reusable framework, TREC development resources, and an arena for comparing industrial baselines.

1 INTRODUCTION

  • RAG uses retrieved segments to ground LLM answer summaries, a format associated with mitigating factual inconsistencies and enhancing interpretability and generalization.The paper cites RAG applications including open-domain question answering and fact verification [44].Source: §1 · p. 1
  • Ragnarök provides customizable retrieval, reranking, and generation code to address the difficulty of implementing RAG at scale.It also offers REST APIs and an integrated WebUI.Source: §1 · p. 1
  • The framework supplies a deduplicated MS MARCO V2 collection and two development-topic sets: TREC-RAGgy 2024 and TREC-Researchy 2024.The topic sets draw on long-form TREC Deep Learning questions and Researchy Questions.Source: §1 · p. 1
  • On development topics, GPT-4o provides more detailed answers than Command R+ in the framework’s head-to-head RAG arena.Both baselines use BM25 and RankZephyr retrieval and are evaluated with human preferences.Source: §1 · p. 1

2 RELATED WORK

Prior RAG frameworks and benchmark resources leave gaps in research-friendly evaluation, collection scale, and topic diversity. Ragnarök is positioned alongside existing tools and datasets while selecting non-Wikipedia collections and TREC-derived topics.

RAG Frameworks

  • Existing RAG frameworks are often closed-source or lack research-focused evaluation; Ragnarök adds a WebUI battle arena, REST APIs, sentence-level citation I/O, and integrations with Pyserini and RankLLM [46, 47].FlashRAG [24] is described as similarly motivated, with a focus on pipeline flexibility.Source: §2 · p. 2

Collection selection

  • English Wikipedia is common in RAG datasets, while ClueWeb22 and MS MARCO V2 offer alternative document collections; ClueWeb22 has 22 billion curated web pages.ClueWeb22 was used in TREC CAsT and the forthcoming iKAT [3], while MS MARCO V2 was used in the TREC Deep Learning track.Source: §2 · p. 2

Topic selection

  • The paper avoids English Wikipedia and reuses TREC Deep Learning topics and Researchy Questions [51] for its topic selection.The cited long-form QA datasets include ASQA [52], ELI5 [18], and QAMPARI [4].Source: §2 · p. 2

3 OUR FRAMEWORK

Ragnarök is a reusable, open-source pipeline that retrieves and reranks segments before generating cited answers. Its input/output definitions specify how user topics, ranked references, and sentence-level answers are represented.

3 OUR FRAMEWORK

  • The retrieval module searches a document collection, retrieves the top 100 segments, and reranks them to select the top 20 for generation.It supports lexical retrieval such as BM25 and reranking such as RankZephyr [47].Source: §3 · p. 2
  • The augmented-generation module combines the user topic, top-20 retrieved segments, and a prompt to produce an answer with sentence-level text grounded in cited segments.The framework describes retrieval and augmented generation as sequential modules.Source: §3 · p. 2

3.1 RAG Input/Output Definitions

  • The JSON output lists ranked segment IDs under references and gives sentence-level answer text with zero-based citations, plus a response-length character count.The user supplies a topic through the Ragnarök REST API.Source: §3.1 · p. 2

4 DOCUMENT COLLECTION

The MS MARCO V2 collection’s near-duplicates can reduce retrieval accuracy and document diversity, while chunking choices relate to RAG answer quality. The paper constructs a deduplicated document collection and an overlapping segment collection.

4 DOCUMENT COLLECTION

  • Near-duplicates may reduce retrieval accuracy and document diversity, and retrieved chunk representations correlate with RAG answer quality.The paper describes near-duplicates as documents with sufficiently similar text information.Source: §4 · p. 3
  • 8.35% fewer duplicates remain after the two-stage MS MARCO V2.1 deduplication process.The process uses LSH with MinHash and 9-gram shingles to form equivalence classes, then selects a representative document from each.Source: §4 · p. 3
  • More than 113 million overlapping segments make up the MS MARCO V2.1 segment collection.A sliding window of 10 sentences with a stride of 5 creates segments averaging roughly 500–1000 characters; segment IDs map back to documents, with start_char and end_char marking positions.Source: §4 · p. 3

5 TOPIC COLLECTION

The paper builds two development topic collections to evaluate RAG systems on complex, long-form questions. TREC-RAGgy filters existing TREC topics, while TREC-Researchy samples diverse Researchy Questions.

Topic selection

  • Both collections target non-factoid, long-form questions, addressing the limits of factoid-heavy benchmarks that can yield short, easily memorized answers.MS MARCO contains up to 55% factoid queries.Source: §5 · p. 3
  • Table 2 summarizes the top five TREC-RAGgy categories, Researchy intrinsic attributes, and first words across topics.The distributions are intended to characterize the collections’ topic mix.Source: Table 2 · p. 3

TREC-RAGgy 2024

  • TREC-RAGgy 2024 filters TREC Deep Learning 2021-2023 topics by category and generated-answer classification, excluding topics with long-form answers or requiring cross-source aggregation; it includes 120 of 210 topics (57.1%).Among included topics, 24.2% were aggregation topics, and 65% began with what or how.Source: §5 · p. 3

TREC-Researchy 2024

  • TREC-Researchy samples 102K Researchy Questions to create a smaller, diverse development set, but the queries lack relevance judgments.Sampling iteratively selects queries with the greatest l1-norm distance in the eight-attribute space from the already-sampled set.Source: §5 · p. 3

6 TREC 2024 RAG BASELINES

The paper describes retrieval and generation baselines and a human-evaluated arena for comparing RAG pipelines. In the baseline comparison, GPT-4o provides longer, more detailed answers, while Command R+ cites more relevant segments.

6 TREC 2024 RAG BASELINES

  • BM25 retrieves the top 100 segments, then RankZephyrρ progressively reranks candidates in three passes before the top 20 documents go to generation.RankGPT is provided as a secondary reranker baseline.Source: §6 · pp. 3–4
  • Command R+ and GPT-4o generate answers with sentence-level citations, using mapped parent-sentence citations for Command R+ and a zero-shot ChatQA prompt for GPT-4o.GPT-4o cites relevant segments inline in IEEE format.Source: §6 · p. 4
  • GPT-4o produces longer, more detailed answers and cites fewer segments than Command R+, which produces shorter answers and cites more relevant segments.The authors leave empirical pairwise win-rate computation for future work.Source: §6 · p. 4

6.1 Ragnarök System Arena

  • The WebUI lets users submit one topic to two configurable RAG pipelines and choose their preferred response, with blinded identities revealed after voting.REST APIs connect pipeline stages, and users can preview cited segments in the example interface.Source: §6.1 · p. 4

7 ONGOING WORK

Ongoing work will expand Ragnarök’s pipelines and finalize evaluation for the TREC 2024 RAG Track.

7 ONGOING WORK

  • Planned pipeline additions include Artic-Embed, more pointwise and pairwise rerankers, and advanced techniques such as Self-RAG and CRAG.These are planned additions, not reported completed components.Source: §7 · p. 4
  • The team plans a fresh scrape of Bing search logs near the test-topic submission period to reduce data-leakage risk and support fair evaluation with commercial LLMs.The next evaluation phase will finalize an automatic nugget-based methodology.Source: §7 · p. 4

8 CONCLUSION

The conclusion presents Ragnarök as a reusable framework, collection, topic set, and baseline package for the TREC 2024 RAG Track.

8 CONCLUSION

  • Ragnarök provides a reusable framework, reproducible baselines, and a WebUI arena for comparing retriever, reranker, and generation models.The framework is open-sourced to support a unified standard for future RAG systems.Source: §8 · p. 4
  • The paper introduces MS MARCO V2.1, curated development topics, and standardized I/O definitions for RAG systems..Source: §8 · p. 4
  • The baselines include Command R+ and GPT-4o, with qualitative analysis on the development topics.The conclusion frames these contributions as preparation for the TREC 2024 RAG challenge.Source: §8 · p. 4

A TREC-RAGGY 2024: ADDITIONAL DETAILS

The authors manually classify TREC Deep Learning Track topics from 2021–2023 into seven categories. The categories distinguish different kinds of information and reasoning demands.

A TREC-RAGGY 2024: ADDITIONAL DETAILS

B TREC-RESEARCHY 2024: ADDITIONAL DETAILS

Researchy Questions are characterized using eight intrinsic attributes scored by GPT-4, with a score of 5 used as the labeling threshold. The questions scored zero for harmfulness, while only a tiny fraction scored highly on ambiguity.

B TREC-RESEARCHY 2024: ADDITIONAL DETAILS

C RAGNARÖK SYSTEM ARENA

The Ragnarök System Arena presents RAG outputs for comparison in a web interface, with human-readable answers and a tab that reformats responses as JSON for the TREC 2024 RAG Track. Its examples show prompts, pipeline outputs, and a deduplicated segment pair.

C RAGNARÖK SYSTEM ARENA

  • Figure 2 shows the ChatQA prompt template used for GPT-4o RAG generation with in-text citations.Source: Figure 2 · p. 7
  • Table 3 illustrates deduplication by retaining one MS MARCO V2.1 segment and discarding a near-duplicate segment.The retained segment is msmarco_doc_00_995170174#0; the discarded segment is msmarco_doc_00_995171191#0.Source: Table 3 · p. 8
  • The arena presents two pipelines’ answers side by side for the Pink Floyd query, while a used-car query example shows two blinded systems and human-readable outputs.The blind setup is intended to support fair leaderboards and let users assess response quality without bias.Source: C RAGNARÖK SYSTEM ARENA · p. 7, Figure 3 · p. 8, Figure 4 · p. 9
  • The responses tab reformats final answers as JSON expected by the TREC 2024 RAG Track I/O definitions, complementing the human-readable output view.The interface can toggle between the views to support analysis of different RAG systems.Source: C RAGNARÖK SYSTEM ARENA · p. 7, Figure 5 · p. 9
  • The end-to-end example uses the potty-training question and shows Command R+ and GPT-4o answers from the augmented-generation stage with top-10 retrieved segments.Source: Table 4 · p. 10, C RAGNARÖK SYSTEM ARENA · p. 10
Loading 2406.16828v1…