Source-linked AI summary
RAG-QA Arena: Evaluating Domain Robustness for Long-form Retrieval Augmented Question Answering
Rujun Han, Yuhao Zhang, Peng Qi, Yumo Xu, Jenyuan Wang, Lan Liu, William Yang Wang, Bonan Min, Vittorio Castelli
TL;DR
Existing datasets provide limited cross-domain evaluation for LLM-based RAG-QA and often use short answers unsuited to long-form generation. The paper builds LFRQA’s coherent long-form references and RAG-QA Arena’s model-based comparisons; the authors report high correlation with human judgments and greater preference for LFRQA answers over those of the best LLMs.
Problem
Popular RAG-QA benchmarks use single-source corpora, and short extractive references mismatch long-form LLM answers, limiting cross-domain evaluation.
Source: §1 · pp. 1–2Method
LFRQA integrates short answers from multiple documents into coherent long-form references, while RAG-QA Arena uses model-based evaluators to compare LLM answers directly with them.
Source: §1 · p. 2Results
RAG-QA Arena evaluations highly correlate with human judgments, and LFRQA answers are preferred significantly more than the best LLMs’ answers.
Source: §1 · p. 2Takeaways & Limitations
The authors present LFRQA and RAG-QA Arena as a challenging, robust benchmark for cross-domain RAG-QA evaluation.
Source: §1 · p. 2, §9 · p. 9The work focuses primarily on LLMs in the QA component; studying different retrievers or joint retriever-and-LLM training is left for future research.
Source: Limitations · p. 10
Abstract
from arXiv · showhide
Question answering based on retrieval augmented generation (RAG-QA) is an important research topic in NLP and has a wide range of real-world applications. However, most existing datasets for this task are either constructed using a single source corpus or consist of short extractive answers, which fall short of evaluating large language model (LLM) based RAG-QA systems on cross-domain generalization. To address these limitations, we create Long-form RobustQA (LFRQA), a new dataset comprising human-written long-form answers that integrate short extractive answers from multiple documents into a single, coherent narrative, covering 26K queries and large corpora across seven different domains. We further propose RAG-QA Arena by directly comparing model-generated answers against LFRQA's answers using LLMs as evaluators. We show via extensive experiments that RAG-QA Arena and human judgments on answer quality are highly correlated. Moreover, only 41.3% of the most competitive LLM's answers are preferred to LFRQA's answers, demonstrating RAG-QA Arena as a challenging evaluation platform for future research.
1 Introduction
Existing RAG-QA datasets offer limited evidence about out-of-domain performance and do not provide coherent long-form references suited to LLM answers. The paper introduces LFRQA and RAG-QA Arena to address these evaluation gaps.
1 Introduction
- Single-corpus benchmarks fall short on out-of-domain evaluation, while short extractive answers mismatch LLMs’ long-form responses and concatenating them produces incoherent references.Token-overlap metrics can also penalize supporting information added to long-form answers.Source: §1 · pp. 1–2
- LFRQA integrates multiple short extractive answers into coherent long-form references, and distinguishes itself from similar datasets through corpus grounding and multi-domain coverage.ASQA and ELI5 are described as either not annotated against the underlying corpus or single-domain.Source: §1 · p. 2, Table 1 · p. 2
- RAG-QA Arena uses model-based evaluators to compare LLM answers directly with LFRQA without examining long, potentially noisy retrieved passages.The authors report high correlation between Arena evaluations and human judges following the same instructions and rubrics.Source: §1 · p. 2
- The authors present LFRQA as a challenging, robust benchmark and say Arena can be extended to study retrieval’s impact on answer-generation quality.Their stated experimental focus is the LLMs used for the QA component.Source: §1 · p. 2
2 RAG-QA Task Formulation
RAG-QA retrieves relevant passages and generates free-form answers, while LFRQA targets cross-domain robustness with coherent long-form references assembled from multiple answer sources. Its statistics show broad multi-document synthesis, alongside adaptations and exclusions in its dataset construction.
2 RAG-QA Task Formulation
- RAG-QA retrieves K relevant passages from a passage collection, then a QA model reads those passages and generates a free-form answer.The paper fixes the retriever and focuses on variations of the language model.Source: §2 · pp. 2–3
3.1 Annotated Data
- Short extractive references can omit useful information, penalize long LLM answers under token-overlap metrics, and become incoherent when concatenated.The evaluation goal is to gauge domain robustness where real-world systems encounter unfamiliar data.Source: §3.1 · p. 3, §2 · p. 3
- Annotators integrate all highlighted answers into one coherent answer; omitting any highlighted answer fails quality control.Citations after each answer sentence support data-quality checks, and linguists audit 10% of each batch, returning batches with a valid-answer ratio below 90%.Source: §3.1 · pp. 3–4
3.2 Adapted Data
- For biomedical data, LFRQA uses ROBUSTQA test queries with BioASQ ideal answers rather than short answer spans, without further annotation.BioASQ answers are shorter because factoid queries dominate.Source: §3.2 · p. 4
- SearchQA is excluded because it has only short-form answers and document omissions that prevent reconstructing long-form answers.Source: §3.2 · p. 4
3.3 Data Statistics and Analysis
- 16K test queries span 7 domains; filtering queries with more than 80 ground-truth documents leaves 73 fewer queries than ROBUSTQA.LFRQA has one answer per query and a higher word-per-answer ratio; a 10K-query dev set is also annotated.Source: §3.3 · p. 4
- Around 65% of LFRQA answers use information from ≥2 documents, while 4.9% use information from ≥10 documents (maximum = 80).Figure 3 also reports the distribution of documents used per answer sentence.Source: §3.3 · p. 4, Figure 3 · p. 5
- The reported LFRQA features support its role as a challenging, high-quality dataset for RAG-QA evaluation.Source: §3.3 · p. 5
4 RAG-QA Arena
RAG-QA Arena compares an LLM-generated answer with an LFRQA answer using pairwise preferences. Human and model judges apply the same core rubric, allowing model-based judgments to be compared with human evaluations.
4 RAG-QA Arena
- RAG-QA Arena measures answer quality using win-rate and win-plus-tie rate against LFRQA as the reference.The framework presents generated answers alongside LFRQA answers for pairwise judgments.Source: §4 · p. 5, Figure 4 · p. 5
4.1 Human Evaluation
- Human annotators compare an LLM answer with an LFRQA answer for helpfulness, truthfulness, and completeness, prioritizing truthfulness.Truthfulness is defined to include helpfulness; helpfulness is used when truthfulness alone does not decide the preference.Source: §4.1 · p. 5
4.2 Model-Based Evaluation
- LLM evaluators receive the query and answer pair and use the same three evaluation aspects and mostly the same rubric as human judges.Model-based evaluation is used to scale assessment across the LFRQA test set.Source: §4.2 · pp. 5–6
- Both evaluation modes allow ties; human labels use majority votes from three annotators, defaulting to a tie without a majority.Source: §4.2 · p. 6
5 Experimental Setup
The experiments use a fixed retriever with several proprietary and public LLMs for answer generation and pairwise evaluation. Prompts include chain-of-thought and in-context examples, and the evaluator is selected by its correlation with human judgments.
Experimental setup
- COLBERTV2 retrieves the top 5 passages for main results, with top 10 used in further analysis.Documents are split into chunks of 100 consecutive words.Source: §5 · p. 6
- Answer generation tests GPT-4-TURBO, GPT-4O, GPT-4-0125-PREVIEW, and public Mixtral, Llama, Command R, and Qwen models, including smaller versions.The models were selected from leading Chatbot Arena rankings at the time of writing.Source: §5 · p. 6
- Pairwise evaluation uses several large-context models, and the evaluator with the highest correlation with human judgments is selected.Answer-pair order is shuffled to limit position bias.Source: §5 · p. 6
- Prompts use chain-of-thought, in-context learning, and HTML delimiters, with the model's thinking process removed from final answers.Source: §5 · p. 6
6 Results and Analysis
The results favor LFRQA answers over extractive references and GPT-4, and model-based evaluator judgments align strongly with human judgments. RAG-QA Arena ranks GPT-4O first overall, while Elo analyses support the ranking and show the cost of broader pairwise comparisons.
6.1 LFRQA v.s. RobustQA
- LFRQA significantly outperforms GPT-4 in pairwise comparisons, while ROBUSTQA significantly underperforms, supporting LFRQA as the better ground truth.The comparison used a 700-query sample across seven domains.Source: §6.1 · p. 6
6.2 Quality of Model-based Evaluator
- All evaluator Pearson correlations exceed 0.52 and Cohen’s Kappa values exceed 0.43, indicating strong agreement with human judgments.Evaluators tended to predict fewer ties; GPT-4-0125-PREVIEW was selected as the best-quality evaluator among those tested.Source: §6.2 · p. 7, Table 4 · p. 7
6.3 RAG-QA Arena
- GPT-4O leads the overall dashboard, with GPT-4-TURBO and MIXTRAL-8X22B-INSTRUCT close behind; domain leaders vary across models.GPT-4O leads BI, LI, RE, and WR; MIXTRAL leads FI and SC; GPT-4-TURBO leads TE.Source: §6.3 · p. 7, Table 3 · p. 7
- GPT-4O’s chain-of-thought prompt produced “I couldn’t find an answer” responses for 48.3% of queries; removing chain of thought reduced this rate to levels similar to other competitive models.In 16 of 20 sampled cases, GPT-4O had generated an answer in its thinking process but then declined to answer.Source: §6.3 · p. 7
- 41.3% is GPT-4O’s best win rate against LFRQA, 13.7 percentage points below LFRQA’s win rate against GPT-4O.The comparison uses top-5 retrieved passages; the reported rates are against LFRQA answers.Source: Table 3 · p. 7, §6.3 · p. 8
- With a 43.6% increase in total pairs, the final Elo ranking matches RAG-QA Arena’s win-ratio ranking and differs only slightly on win-plus-tie ranking.Using LFRQA as the comparison target reduces the required comparisons from O(K^2) to O(K).Source: §6.3 · p. 8, Table 5 · p. 8
7 Discussion
The discussion finds human annotations outperform GPT-4 annotations on the reported dimensions. It also weighs alternative evaluation designs against Arena’s efficiency and leaves further evaluator-alignment work for future research.
7.1 LLM as Annotators
- Human annotations outperform GPT-4 annotations by 15.5% on completeness, 23.4% on citation accuracy, and 12.9% on helpfulness.The comparison was made by data linguists on a 100-query sample.Source: §7.1 · p. 9
7.2 Alternative Evaluation Approaches
- Comparing two model responses against LFRQA references in one trial would still require O(K^2) pairs, making it less efficient than Arena.Arena directly evaluates each LLM response against the LFRQA answer.Source: §7.2 · p. 9
- Prompt engineering, in-context example selection, and task-specific evaluator training could further improve alignment with human judges, but are left for future work.Source: §7.2 · p. 9
8 Related Work
Related datasets address parts of RAG-QA evaluation, but differ from LFRQA in answer format, domain coverage, or corpus-grounded human annotation. RAG-QA Arena distinguishes itself by always including a human-annotated LFRQA answer in pairwise evaluation.
8 Related Work
- ROBUSTQA and MULTIHOP-RAG address single-domain coverage, but retain short extractive answers that are less suited to evaluating modern long-form LLM responses.Source: §8 · p. 9
- ELI5 and LONGFACT lack either direct corpus annotation or human-created answers, while ASQA uses a single Wikipedia domain.The paper also notes leakage between ELI5’s validation and training sets.Source: §8 · p. 9
- RAG-QA Arena always includes a high-quality, human-annotated LFRQA answer in its pairwise comparisons.Source: §8 · p. 9
9 Conclusion
The paper presents LFRQA and RAG-QA ARENA as resources for evaluating cross-domain RAG-QA, while identifying limits in model coverage, evaluation cost, and research scope. It also describes potential answer-faithfulness risks and ethical data-collection practices.
9 Conclusion
- LFRQA provides coherent long-form answers for multi-domain RAG-QA benchmarking, and RAG-QA ARENA directly compares LLM answers with them.The authors say the evaluator framework is intended to facilitate RAG-QA evaluation in the LLM era.Source: §9 · p. 9
Limitations
- The evaluation omits Claude and Gemini models because of legal and resource constraints, and GPT-4-0125-PREVIEW costs an average of $300 per model on the full test set.The authors plan to add the omitted models and subsample 10–20% of queries for the public leaderboard.Source: Limitations · p. 10
- Future work could examine smaller evaluators, different retrievers, or joint retriever and LLM training using RAG-QA ARENA.The paper primarily focuses on different LLMs for RAG-QA.Source: Limitations · p. 10
Ethics Statement
- LLM-generated answers may be incorrect or unfaithful when retrieval returns irrelevant passages or an LLM hallucinates.The authors created LFRQA to better evaluate RAG-QA systems and note that these risks are known in the AI research community.Source: Ethics Statement · p. 10
- The work uses public-source data, contracted data professionals for LFRQA annotations, and Appen for human preference annotations, with hourly rates above local minimum-wage standards and $15.The paper states that LFRQA's intended usage is compatible with the underlying data's access conditions.Source: Ethics Statement · p. 10
A Appendix
The appendix describes the LFRQA development set and data licenses, then documents the annotation interface, answer-generation prompt, and human preference evaluation materials.
A.1 Dev Set of LFRQA
- The appendix provides 10K additional development queries for future prompting and training, collected with the same process and quality control as the main data.The authors say they did not use the dev set to fit the paper's scope.Source: §A.1 · p. 12
A.2 Data License
- LFRQA draws on several datasets and follows their data-distribution licenses; the listed licenses include MIT for LoTTE, CC BY 2.5 for BioQAS, and Apache-2.0 for ROBUSTQA.FiQA has no license provided, with data information available on its official website.Source: §A.2 · p. 12
A.3 Annotation Interface
- The annotation interface is used by annotators to write long-form answers, with blue highlights marking original answers from ROBUSTQA.Source: §A.3 · p. 12
A.4 Answer Generation Prompts
- The answer-generation prompt's final two lines request step-by-step thinking; the authors call this the CoT prompt and remove it for GPT-4O.Source: §A.4 · p. 12
A.5 Evaluation Interface
- Human pairwise preference data are collected to benchmark LLM-based evaluators, with the evaluation interface details shown in Figures 6–7.Source: §A.5 · p. 12
- The dev-set summary has no development split for BioASQ or FiQA, following ROBUSTQA.Source: Table 7 · p. 13
A.6 Human Evaluation Instructions
The appendix specifies human preference criteria and decision rules, explains how ratings are grouped for evaluator-correlation analysis, and reports evaluator-selection and no-answer-ratio analyses.
A.6 Human Evaluation Instructions
- Human judgments assess helpfulness, truthfulness, and completeness, with truthfulness defined to require helpfulness as well as correctness.When truthfulness is hard to determine because context is insufficient or conflicting information is unreconciled, the information is considered untruthful.Source: §A.6 · p. 13
- Annotators prefer the fully truthful answer over one containing untruthful information, and among fully truthful answers prefer the one with more truthful or helpful information.They choose “Not sure” when answers seem equally good or are too difficult to differentiate.Source: §A.6 · p. 13
- The appendix reports merging “Better” with “Slightly Better” and “Slightly Worse” with “Worse” when computing correlation with model-based evaluators.Source: §A.6 · p. 13
A.7 Evaluation Model Selection
- GPT-4-0125-PREVIEW alone achieves the best outcome in the reported human–LLM evaluator correlation comparison and is selected for RAG-QA ARENA.Table 8 compares human judges with leading LLM evaluators; its columns are answer pairs.Source: §A.7 · p. 13, Table 8 · p. 13
A.8 No Answer Ratio
- The appendix reports the ratio of “I couldn’t find an answer” responses for each LLM on the full LFRQA test set.Source: §A.8 · p. 13
A.9 Pairwise Evaluation Prompts
- The pairwise evaluation prompt materials include the shared instruction and rubrics, an example template, and in-context examples.The example template is used for both in-context examples and the final test example; the examples label which answer is better or whether there is a tie.Source: §A.9 · p. 13
A.10 Error Analysis
The error-analysis appendix describes a GPT-4O response issue associated with the CoT prompt and presents comparisons involving other model answers and LFRQA, alongside prompt and evaluation materials.
A.10 Error Analysis
- Removing the CoT prompt fixes GPT-4O's case where it puts an answer in <thinking> but generates “I couldn’t find an answer.”The appendix compares this case with GPT-4-TURBO, MIXTRAL-8X22B, and LFRQA answers.Source: §A.10 · p. 13
- Table 9 reports each LLM's no-answer ratio on the entire LFRQA test set, with a marker distinguishing answers generated without CoT.Source: Table 9 · p. 14
- Table 10 compares evaluation results using the top 5 or top 10 passages and reports win and win-plus-tie rates against LFRQA.An asterisk denotes use of the answer-generation prompt without CoT.Source: Table 10 · p. 14
- The answer-generation prompt template allows replaceable component inputs, and most prompt instructions remain the same across model formats.Its final step-by-step section is the CoT prompt, removed for GPT-4O only.Source: Table 11 · p. 14
- Figures 5–7 show the annotation interface and pairwise-preference materials, while Tables 13–15 describe evaluator instructions, an example template, and in-context examples.Table 15 labels choices as answer 1 better, answer 2 better, or tie.Source: Figure 5 · p. 15, Figure 6 · p. 16, Figure 7 · p. 17
- Figures 8–9 show GPT-4O’s original “I couldn’t find an answer” response compared with GPT-4-TURBO and MIXTRAL-8X22B, plus GPT-4O’s answer without CoT.Blue highlights indicate helpful information for answering the query; <answer> tags distinguish answers from <thinking>.Source: Figure 8 · p. 20, Figure 9 · p. 21