Source-linked AI summary
A Critical Evaluation of Evaluations for Long-form Question Answering
Fangyuan Xu, Yixiao Song, Mohit Iyyer, Eunsol Choi
TL;DR
LFQA evaluation lacks reliable evidence for judging long, open-ended answers, with prior practice relying on crowdsourced judgments and simple metrics. The paper studies expert preference judgments and automatic metrics, finding that existing metrics do not predict overall human preferences. It therefore supports multifaceted evaluation focused on attributes such as factuality and completeness, while noting limits in answer scope and stationary evaluation.
Problem
LFQA's flexibility makes evaluation difficult, while prior work largely relies on crowdsourced annotations and simple metrics that are inadequate for long-form answers.
Method
The study combines expert preference judgments with justifications across seven fields and evaluates reference-based, answer-only, and question-answer automatic metrics.
Results
No existing automatic metric predicts human overall preference reliably; fine-grained evaluation is more feasible, with QAFactEval best on factuality and self-BLEU competitive on coherence.
Takeaways & Limitations
LFQA evaluation should move from a single overall score toward targeted, multifaceted assessment of attributes such as completeness, factuality, and ease of understanding.
Takeaways & Limitations
The study covers English questions drawn from search queries or community forums and uses stationary evaluation of pre-generated model outputs.
Abstract
from arXiv · showhide
Long-form question answering (LFQA) enables answering a wide range of questions, but its flexibility poses enormous challenges for evaluation. We perform the first targeted study of the evaluation of long-form answers, covering both human and automatic evaluation practices. We hire domain experts in seven areas to provide preference judgments over pairs of answers, along with free-form justifications for their choices. We present a careful analysis of experts' evaluation, which focuses on new aspects such as the comprehensiveness of the answer. Next, we examine automatic text generation metrics, finding that no existing metrics are predictive of human preference judgments. However, some metrics correlate with fine-grained aspects of answers (e.g., coherence). We encourage future work to move away from a single "overall score" of the answer and adopt a multi-faceted evaluation, targeting aspects such as factuality and completeness. We publicly release all of our annotations and code to spur future work into LFQA evaluation.
1 Introduction
LFQA evaluation remains underdeveloped despite advances in model development, relying heavily on crowdsourced judgments and simple string-matching metrics. This study examines expert human evaluation and automatic metrics to clarify how long-form answers should be assessed.
- Evaluation gap: LFQA evaluation has largely combined crowdsourced human annotations with simple string-matching metrics such as ROUGE.The paper frames this practice as inadequate for evaluating long and complex answers.
- Study scope: The study examines both human and automatic evaluation protocols for long-form answers.It is presented as the first study specifically targeting LFQA evaluation.
- Human evaluation: Domain experts in seven fields perform answer-preference judgments and provide detailed justifications.This approach addresses concerns that crowdworkers may lack the expertise to judge properties such as factuality.
- Automatic evaluation: The automatic evaluation study compares existing text-generation metrics against human judgments of overall quality and fine-grained answer aspects.The paper asks whether metrics beyond ROUGE can track human judgments and whether metrics can mimic preference judgments.
- Implications: The paper releases expert evaluations and recommends replacing poorly defined overall preference with multifaceted assessment of completeness, factuality, and ease of understanding.The released dataset contains 260 ratings and justifications over 140 answer pairs.
2 Background and related work
Prior LFQA research has relied on ROUGE, specialized retrieval-style protocols, and non-expert human evaluation. These approaches have limited applicability to general LFQA because long-form answers are open-ended and often require information unavailable in predefined resources.
- Automatic evaluation: Early LFQA work used ROUGE to measure similarity between human reference answers and model-generated answers.Later work found ROUGE unmeaningful for open-ended long-form answers.
- Specialized protocols: Some recent studies narrow evaluation to subproblems such as exemplification retrieval or ambiguous factoid questions.These protocols target constrained answer settings rather than general LFQA.
- Scope limitations: Specialized protocols cannot be easily adapted to general LFQA when required disambiguated questions, answers, or other structured resources are unavailable.The paper gives Stelmakh et al.'s protocol as an example requiring lists of disambiguated questions and corresponding answers.
3 How do domain experts evaluate long-form answers?
The paper evaluates long-form answers with domain experts across seven fields, finding substantial variation in preferences and agreement across domains. Experts identify factuality, completeness, and ease of understanding as important dimensions, while model performance varies notably by domain.
- 3.1 Collecting expert judgments: Domain experts evaluated paired answers from seven fields using overall A/B preferences, difficulty judgments, and free-form justifications.The study compares highly upvoted human answers with either model-generated answers or lower-upvoted human answers.
- 3.1 Collecting expert judgments: Experts spent 15 to 30 minutes per question, underscoring the demanding nature of evaluating long-form answers.The annotation effort cost $845 for 260 expert judgments.
- 3.2 Quantitative results: Experts slightly preferred GPT-3 answers over human answers overall, choosing model-generated answers in 61.8% of comparisons.As a sanity check, highly upvoted human answers beat lower-upvoted human answers in 62.4% of comparisons.
- 3.1 Collecting expert judgments: History was the hardest domain for models, with human answers averaging 356 words in the H/M setting and substantially exceeding answers in other domains.The study attributes this difficulty to history questions and human answers being longer and more complex.
- 3.2 Quantitative results: Preferences differed sharply by domain: model answers were preferred in economics and law at 90%, whereas human answers were preferred in history at 75.6%.Expert agreement was fair in economics, moderate in biology and physics, and substantial in history.
- 3.3 What makes one answer better than another?: Experts mentioned completeness twice as often as WEBGPT annotators, 12 versus 6 times, because it requires judging whether all necessary information is covered.Experts could identify omitted causes or explanations that shorter answers failed to include.
- 3.3 What makes one answer better than another?: Both experts and crowdworkers treated ease of understanding as a decisive criterion, while crowdworkers more often favored conciseness and specificity.The paper argues that future evaluation should account for these fine-grained aspects and that models can learn to interpret preference justifications.
4 Do automatic metrics correlate with human judgments?
The study tests whether existing and trained automatic metrics reproduce human judgments of long-form answer quality. Metrics perform poorly and inconsistently overall, although some align with particular aspects such as factuality or coherence.
- Evaluation setup: The authors compare existing reference-based, answer-only, question-answer, answer-evidence, and trained preference metrics against human judgments.The evaluation uses prior comparison data and expert annotations, measuring overall preference and fine-grained judgments of coherence and factuality.
- Overall performance: All metrics achieve relatively low accuracy and none remains robust across different types of answer pairs.BERTScore and BLEURT perform poorly on HURDLES human-versus-model data, while supervised metrics perform better on overall ratings in other settings.
- Overall performance: The length baseline often rivals or exceeds unsupervised metrics, demonstrating the brittleness of automatic LFQA evaluation.Choosing longer answers performs strongly for several comparisons, whereas choosing shorter answers is beneficial for WEBGPT coherence judgments 62% of the time.
- Fine-grained evaluation: Fine-grained evaluation is more feasible than modeling overall quality: QAFactEval performs best on factuality, while Self-BLEU is competitive for coherence.QAFactEval requires evidence documents, which may be unavailable or unreliable.
- Metric correlation: Reference-based metrics correlate with one another, with ROUGE and BERTScore producing the same ranking for 63% of answer pairs.Question-answer metrics such as question-generation likelihood and BARTScore also show positive correlation.
5 Conclusion & Future Work
The paper presents LFQA evaluation as inherently multi-faceted and argues that overall answer scores are not meaningful. It recommends expert evaluation and targeted modeling of attributes such as completeness and ease of understanding.
- Conclusion: Expert justifications indicate that answer quality involves multiple aspects, including completeness and conciseness, that can conflict.The authors therefore argue that computing a single overall quality score is not meaningful.
- Future work: The study recommends replacing poorly defined overall-preference judgments with multi-faceted evaluation of attributes such as completeness and factuality.The benchmark combines expert evaluations with existing annotations for overall preference, factuality, and coherence.
- Future work: Future work should model frequent expert-identified aspects such as completeness and ease of understanding, potentially using methods that localize and categorize errors.The recommendation follows the study's finding that metrics trained on overall preferences have important limitations.
Limitations
The study covers a limited range of long-form question-answering settings. Its questions come from search queries or community forums and are restricted to English-speaking cultural contexts.
- Limitations: The questions are drawn from search queries or community forums, leaving other real-world settings outside the study’s scope.Examples of omitted settings include education and commercial question answering.
- Limitations: The study covers only English, so its questions are topically limited to English-speaking culture.
Ethics Statement
The expert annotation protocol was exempt from IRB review, and the collected data will be publicly released under the MIT license. The process avoided collecting uniquely identifying worker information and screened the data for offensive content.
- Ethics Statement: The expert annotation data collection protocol was determined to be exempt from review by an IRB board.
- Ethics Statement: All collected data will be made publicly available under the MIT license.
- Ethics Statement: The data collection process did not require information that could uniquely identify individual workers.
- Ethics Statement: The annotation data were checked for identifying information and offensive content in the questions or answers.
A.1 Related work on text generation evaluation
Text-generation evaluation spans reference-based, distributional, and reference-less approaches, with different metrics targeting different properties of generated text. The passage contrasts closed-ended tasks with open-ended generation, where a single reference may be inadequate.
- A.1 Related work on text generation evaluation: Machine translation and summarization commonly use reference-based metrics such as BLEU, ROUGE, and BERTScore.These metrics measure similarity between generated and reference text.
- A.1 Related work on text generation evaluation: For open-ended generation, comparing generated text with a single reference is not meaningful.The passage motivates distributional metrics as an alternative for such settings.
- A.1 Related work on text generation evaluation: Distributional metrics compare model-generated and human-written text distributions instead of relying on one reference.
- A.1 Related work on text generation evaluation: Reference-less metrics often target specific properties, such as factuality through the relationship between a source document and its summary.
A.2 Expert Annotation
Expert annotation combines domain-based question grouping with qualification-focused recruitment. The study identifies six domains through flairs or clustering and retains experts who meet substantive expertise requirements and agree to continue.
- Question clustering: Four domains—biology, physics, chemistry, and economics—are identified from ELI5 post flairs.
- Question clustering: Tech/cs and law are identified by encoding questions with DPR and clustering them with KMeans.
- Annotators: Experts are hired based on academic background and English proficiency, without additional demographic or geographic restrictions.
- Annotators: 35 potential experts joined a paid pilot, with compensation of $3 per question-answer set.
- Annotators: 13 experts met the qualification requirements and agreed to continue because the task required substantive expertise, time, and attention.
A.2.1 Justification Analysis
The analysis reports statistics for collected explanations and describes preprocessing that standardizes answer mentions before masking candidate labels.
- Analysis materials: Explanation statistics are reported in Table 6, while examples of explanations and manually extracted aspects appear in Table 7.
- Preprocessing: 259 of 260 expert comments remained after preprocessing because comments without answer entities were removed.The same procedure retained 292 of 305 WEBGPT comments.
- Preprocessing: Answer mentions were standardized by prepending “Answer” before candidate labels, then mentions of “A” and “B” were masked.
A.3 Previously Collected Human Evaluation Data
This section describes prior LFQA systems and human-evaluation protocols, then details automatic metrics and exploratory GPT-3 evaluation of answer preferences.
- LFQA systems: HURDLES combines a passage retriever with an answer-generation model, whereas WEBGPT fine-tunes GPT-3 to search and compose answers with reference documents.
- Evaluation aspects: Prior overall evaluations ask which answer is better or more useful, while coherence evaluations emphasize sense, followability, logical order, and repetition.
- Evaluation aspects: Prior factuality protocols differ: one assesses correctness without evidence documents, while another checks entailment by provided evidence and unsupported facts.Faithfulness to evidence is distinct from answer correctness because evidence may be inaccurate or outdated.
- Previously collected data: The expert dataset contains 140 questions, 205 human-written answers, and 75 model-generated answers across human/human and human/model comparisons.
- Automatic metrics: The study compares reference-based metrics including ROUGE-L, BERTScore, and BLEURT with coherence, length, relevance, factuality, and likelihood-based metrics.QAFactEval uses a pipeline combining noun-phrase extraction, question generation, question answering, and learned metrics.
- GPT-3 evaluation: GPT-3 was prompted with two in-context examples, sampled three times per question-answer set, and showed mostly self-consistent judgments that aligned with unanimous human labels.The authors leave broader investigation of language models for automatic LFQA evaluation to future work.