PaperlayerView exact revision on arXiv

Source-linked AI summary

LongEval: Guidelines for Human Evaluation of Faithfulness in Long-form Summarization

Kalpesh Krishna, Erin Bransom, Bailey Kuehl, Mohit Iyyer, Pradeep Dasigi, Arman Cohan, Kyle Lo

arXiv:2301.13298v1cs.CL

TL;DR

Human evaluation of long-form summaries is laborious, and the paper asks how faithfulness judgments can be made more consistent and efficient. LongEval studies evaluation guidelines and annotation choices, finding that source hints have little overall effect but can help annotators at the start.

  • Problem

    Human evaluation of long-form summaries is labor-intensive, expensive to scale, and difficult to design, motivating questions about agreement, workload, and source alignment.

    Source: §1 · pp. 1–2

  • Method

    LongEval evaluates faithfulness guidelines through human studies on SQuALITY and PubMed, comparing whole-summary Likert ratings with averaged binary judgments of summary units.

    Source: §1 · p. 1, §3.1 · p. 4

  • Results

    93% vs 92% accuracy and 0.71 vs 0.64 Fleiss κ favored no-hint over SuperPAL, which also took 41.4 vs 48.2 seconds per unit overall.

    Source: §3.3 · p. 8

  • Takeaways & Limitations

    Hints helped annotate the first few units faster, while annotators became more familiar with the source and summary and needed fewer hints over time.

    Source: §3.3 · p. 8

    The reported trends may change with annotation count, interface, granularity, rating scale, evaluated systems, or absolute versus relative evaluation.

    Source: Limitations · p. 10

Abstract

from arXiv · show

While human evaluation remains best practice for accurately judging the faithfulness of automatically-generated summaries, few solutions exist to address the increased difficulty and workload when evaluating long-form summaries. Through a survey of 162 papers on long-form summarization, we first shed light on current human evaluation practices surrounding long-form summaries. We find that 73% of these papers do not perform any human evaluation on model-generated summaries, while other works face new difficulties that manifest when dealing with long documents (e.g., low inter-annotator agreement). Motivated by our survey, we present LongEval, a set of guidelines for human evaluation of faithfulness in long-form summaries that addresses the following challenges: (1) How can we achieve high inter-annotator agreement on faithfulness scores? (2) How can we minimize annotator workload while maintaining accurate faithfulness scores? and (3) Do humans benefit from automated alignment between summary and source snippets? We deploy LongEval in annotation studies on two long-form summarization datasets in different domains (SQuALITY and PubMed), and we find that switching to a finer granularity of judgment (e.g., clause-level) reduces inter-annotator variance in faithfulness scores (e.g., std-dev from 18.5 to 6.8). We also show that scores from a partial annotation of fine-grained units highly correlates with scores from a full annotation workload (0.89 Kendall's tau using 50% judgments). We release our human judgments, annotation templates, and our software as a Python library for future research.

1 Introduction

LongEval addresses the workload and consistency challenges of judging faithfulness in long-form summaries, combining a survey with guidelines evaluated in human studies. Its findings cover finer-grained judgments, partial annotation, and when source alignment helps.

1 Introduction

  • Human evaluation is labor-intensive, costly to scale, and difficult to design, especially for long generated sequences.The paper notes that large numbers of judged examples are needed for statistical conclusions and correlations with automatic metrics.Source: §1 · p. 1
  • A survey of 162 publications and preprints on long-form summarization found that 119 papers (73%) did not perform human evaluation on summaries of 150 words or longer.Source: §1 · p. 1
  • LongEval presents faithfulness-evaluation guidelines and reports studies on SQuALITY and PubMed, with three-way fine-grained judgments for 120 summaries.The released judgments can be used to benchmark automatic metrics.Source: §1 · pp. 1–2
  • Clause-level judgments aggregated across a summary reduced SQuALITY faithfulness-score standard deviation from 18.5 to 6.8 versus whole-summary Likert ratings.The paper reports significantly higher inter-annotator agreement for the fine-grained approach.Source: §1 · p. 2
  • At 50% workload, partial clause annotation retained high correlation with full-workload faithfulness scores (0.89 Kendall’s τ).The partial-annotation approach addresses the workload of evaluating long summaries.Source: §1 · p. 2
  • Source alignment helps only when summaries are highly extractive or mostly correct.This differs from suggestions for short-form summarization cited in the paper.Source: §1 · p. 2

2 Survey of human evaluation practices

The survey finds that human evaluation of long-form summaries is uncommon and often lacks reproducible, standardized practices. It also documents the substantial workload associated with evaluating long summaries and source documents.

2 Survey of human evaluation practices

  • The survey covered 162 model-evaluation papers published from June 2018 to September 2022 on datasets averaging at least 150 summary words, excluding five pretraining-only studies.The survey considered papers evaluating summarization models on at least one such dataset.Source: §2 · p. 2
  • Among 44 papers evaluating long-form summaries, only 2 released raw annotations, 9 described instructions or interfaces, and 12 reported inter-annotator agreement.Only 12 performed statistical analysis, and 14 evaluated more than one dataset.Source: §2 · p. 3
  • Evaluation setups varied: 37 papers used whole-summary COARSE judgments, six used FINE-grained units, and protocols included Likert scales, A/B tests, and binary labels.The paper reports that COARSE annotations have much lower inter-annotator agreement than FINE annotations.Source: §2 · p. 3
  • Table 2 reports that 73% of papers lacked human evaluation of long-form summaries; within that group, 101 did no human evaluation and 17 evaluated only short summaries.The survey covered 162 papers.Source: Table 2 · p. 3, §2 · p. 3
  • Twenty papers described long-form evaluation as expensive, difficult, or time-consuming because of lengthy summaries and sources; 13 said expert annotators were necessary, especially in technical domains.PubMed is given as an example of a technical domain.Source: §2 · p. 3

3 The LONGEVAL guidelines for faithfulness human evaluation

LONGEVAL develops faithfulness-evaluation guidelines and tests them with human studies on SQuALITY and PubMed. The studies examine fine-grained judgments, partial annotation, and source-document hints, with findings scoped primarily to long-form summarization.

3 The LONGEVAL guidelines for faithfulness human evaluation

  • LONGEVAL evaluates faithfulness by scoring individual summary units and aggregating judgments, comparing FINE unit-level judgments with COARSE whole-summary ratings.The studies use clause-like units formed by segmenting sentences at conjunctions and punctuation; units are equally weighted when aggregated.Source: §3 · p. 3, §3.1 · p. 4

3.4 To what extent do our findings generalize to short-form summarization?

  • The study covers long-form SQuALITY literary summaries and PubMed biomedical summaries, while its findings are more relevant to long-form than short-form summarization.SQuALITY summaries average 227 words and are highly abstractive; PubMed summaries average 205 words and are more extractive, with n-gram blocking used to reduce extractiveness in some generations.Source: §3 · p. 4, §3.4 · p. 9

3.1 RQ1: Does inter-annotator agreement improve using fine-grained annotations?

  • Clause-level FINE judgments reduce faithfulness-score variation versus COARSE ratings, from 14.0 to 7.8 average standard deviation on a 100-point scale.FINE judgments also narrow uncertainty in downstream estimates; on PubMed, average Pearson-correlation uncertainty is 0.15 versus 0.35 for COARSE annotations.Source: §3.1 · pp. 5–6, Table 3 · p. 5

3.2 RQ2: Can we reduce annotator workload by partially annotating a long summary?

  • Annotating 50% of summary units yields 0.78–0.89 Kendall’s τ correlation with full annotation, but partial annotation increases inter-annotator variance.The variance trade-off differs by dataset: in PubMed, COARSE variance is lower than FINE variance at sampling fractions of 0.3 or less.Source: §3.2 · p. 7

3.3 RQ3: Is it useful to align summary units to sentences in the source document?

  • Alignment hints have little overall effect on accuracy, agreement, or annotation time, and are most useful for mostly correct or nearly verbatim summary units.SuperPAL had 0.61 recall@3 versus 0.47 for the next-best method; annotators still reported reading the full source and searching for keywords.Source: §3.3 · pp. 7–9

4 Related Work

The related work contrasts automatic summarization metrics with human evaluation, then distinguishes summary faithfulness evaluation from fact verification. This paper focuses on reference-free faithfulness evaluation for challenging long-form summaries.

4 Related Work

  • Prior work develops NLI- and QA-based automatic metrics, while this paper focuses on human evaluation as a gold standard for developing automatic metrics.The cited work also includes Pyramid protocols and efforts to improve their efficiency, often for salience evaluation with multiple references.Source: §4 · p. 9
  • Unlike cited Pyramid-like protocols for short-form news salience evaluation that assume multiple references, this work addresses reference-free faithfulness in long-form summarization.The paper characterizes long-form tasks such as SQuALITY and PubMed as more challenging and expensive to evaluate.Source: §4 · p. 9
  • Faithfulness evaluation differs from fact verification because summaries are longer and contextual, grounded in a source document, and model-generated.Fact verification instead checks standalone claims against a large knowledge source.Source: §4 · p. 9

5 Conclusion

The conclusion presents LONGEVAL guidelines for more standardized human evaluation of long-form summarization and reports their empirical analysis on two datasets. Fine-grained judgments, partial annotation, and source highlighting yield distinct findings, with future work planned on other evaluation aspects.

5 Conclusion

  • LONGEVAL provides recommendations for standardized human evaluation of long-form summarization, empirically analyzed on two datasets.The guidelines address evaluation of faithfulness.Source: §5 · p. 9
  • Fine-grained annotations have lower inter-annotator variance than coarse-grained annotations, while partial annotation reduces workload and maintains accuracy, and highlighting hints have limited usefulness.These findings summarize the guideline analyses across two datasets.Source: §5 · pp. 9–10
  • The authors plan future experiments on salience and coherence evaluation.Source: §5 · p. 10

Limitations

The reported human-evaluation findings have methodological and scope boundaries: results may shift with experimental choices, the study covers faithfulness rather than other summary qualities, and its fine-grained scoring weights units equally.

Limitations

  • Changing annotation counts, interfaces, fine-grained unit size, Likert scale points, evaluated systems, or relative versus absolute evaluation could change observed trends.The authors describe human evaluation as noisy and affected by confounding variables.Source: Limitations · p. 10
  • The study evaluates faithfulness, and its findings may not generalize to coherence or salience evaluation.The authors identify coherence and salience as important aspects of long-form summarization.Source: Limitations · p. 10
  • The experiments weight fine-grained units equally, although non-uniform weighting may better reflect unit criticality where such importance can be established.The authors note that consistent importance is difficult to establish across domains and depends on consumer preferences.Source: Limitations · p. 10

Ethical Considerations

The paper reports institutional review exemption and describes annotator compensation and time estimates for the human evaluation studies.

Ethical Considerations

  • All human-evaluation experiments were exempt under institutional IRB review.Source: Ethical Considerations · p. 10
  • Upwork annotators received $15–20 per hour, respecting their suggested hourly wage.Source: Ethical Considerations · p. 10
  • Annotators received estimated task times based on pilots, and compensation reflected actual time; some exceeded estimates by 0.5–1 hour.Most freelancers finished within the estimated time window.Source: Ethical Considerations · p. 10

A Bootstrap analysis of inter-annotator variance

The analysis uses bootstrap resampling to estimate confidence intervals for human-evaluation statistics. It resamples annotators for each summary to capture inter-annotator variance.

A Bootstrap analysis of inter-annotator variance

  • Bootstrap resampling estimates uncertainty for average system performance and correlations between human judgments and automatic metrics.The test statistics are recalculated across repeated samples drawn with replacement.Source: A Bootstrap analysis of… · p. 15
  • Unlike Deutsch et al. (2021), the procedure samples annotators with replacement for each summary to capture inter-annotator variance.The input is an N × M matrix, where N is the number of summaries and M the number of annotators.Source: A Bootstrap analysis of… · p. 15
  • For each resample, the algorithm draws M annotators with replacement for every summary, computes a test statistic, and returns percentile bounds as the confidence interval.The bounds are the α/2 and 1−α/2 percentiles of the sampled statistics.Source: A Bootstrap analysis of… · p. 15

B.1 FINE-grained evaluations of SQuALITY and PubMed summaries

The appendices document FINE and COARSE annotation procedures, annotator recruitment, and additional survey statistics. They report agreement above a random baseline and a qualification-task performance difference between Upwork freelancers and MTurk workers.

B.1 FINE-grained evaluations of SQuALITY and PubMed summaries

  • Of nine Upwork freelancers screened, six scoring at least 85% on a perturbed-clause qualification task were recruited.Half of the qualification clauses were synthetically perturbed and manually checked as unsupported.Source: §B.1 · p. 15
  • The six FINE annotators were native or bilingual English speakers with undergraduate degrees; three with biomedical-reading experience were rehired for PubMed.Annotators received detailed faithfulness guidelines, documented in Table 10.Source: §B.1 · p. 15
  • FINE unit-level agreement statistics were well above a uniform-random annotation baseline.The reported measures include Fleiss κ, Randolph κ, and the fraction of sentence pairs with total agreement.Source: §B.1 · pp. 15–16, Table 7 · p. 15

B.2 COARSE-grained evaluation of PubMed summaries

  • After finding surveyed PubMed evaluations had released no human judgment data, the authors collected COARSE ratings from three new annotators in a between-subject FINE comparison using a 5-point Likert scale.The COARSE annotators were experienced readers of research papers; the instructions and examples are documented in Table 11.Source: §B.2 · p. 16

C Additional Survey Statistics

B.3 Crowdworkers or expert annotators?

  • On perturbed SQuALITY qualification, Upwork freelancers reached 90% accuracy and Fleiss κ=0.71, versus MTurk’s 62% accuracy and κ=0.15.The MTurk group had a Master’s qualification and 90%+ approval rating; the Upwork freelancers had professional writing experience.Source: §B.3 · p. 16

D Automatic summarization metrics used for evaluation

The appendix lists automatic metrics used for Pearson correlation analyses and describes a learning effect during FINE annotation. Annotators became faster as they progressed through summary units.

D Automatic summarization metrics used for evaluation

E Learning effect while annotating long-form summaries

  • Annotators became significantly faster after the first 20% of FINE units; the authors hypothesize that early exposure builds familiarity with source topics.Figure 5 describes faster annotation over time irrespective of whether hints are shown.Source: E Learning effect while annotating… · p. 17, Figure 5 · p. 17

G Metric correlations using Kendall’s Tau

The figure captions distinguish Pearson- and Kendall’s Tau-based versions of earlier correlation figures.

G Metric correlations using Kendall’s Tau

  • Figure 6 recasts Figure 4 with Pearson correlation, while Figure 7 recasts Figure 2 with Kendall’s Tau.The supplied captions identify the correlation measures, not the resulting comparisons.Source: Figure 6 · p. 17, Figure 7 · p. 18

Related papers

Loading 2301.13298v1…