PaperlayerView exact revision on arXiv

Source-linked AI summary

Information Satisfaction: A Reader-Centered Axis for Summarization Evaluation

Isabel Cachola, William Walden, Reno Kriz, Mark Dredze

arXiv:2608.14457v1cs.CL

TL;DR

Existing summarization metrics do not adequately measure whether a summary satisfies an individual reader’s informational needs. This paper introduces information satisfaction, tests metrics with informational and audience perturbations, and conducts expert human evaluation, finding that traditional and LLM-based metrics poorly capture it and agree poorly with reader judgments.

  • Problem

    Existing summarization metrics largely assess general quality or isolated properties, providing limited evidence about whether summaries satisfy readers with different roles, expertise, and informational needs.

  • Method

    The paper introduces information satisfaction and evaluates metrics through perturbation tests across four scientific summarization corpora plus expert judgments conditioned on queries and personas.

  • Results

    Traditional, embedding-based, and LLM-as-judge metrics fail robustness checks, remain insensitive or anticorrelated to audience shifts, and do not exceed chance agreement with reader preferences.

  • Takeaways & Limitations

    Evaluating user-centered summarization requires frameworks that explicitly model who the summary is for and what they seek to learn.

  • Takeaways & Limitations

    The persona representation assumes access to the reader’s role, domain, and information needs.

Abstract

from arXiv · show

The majority of work on summarization evaluation focuses on general summary quality (e.g., ROUGE, BERTScore) or specific desired properties (e.g., readability, factuality). However, these metrics fail to measure the utility of a summary to an individual user. For example, a biomedical researcher learning about the latest vaccine research will have different informational needs from a family doctor. Query-focused summarization captures part of this need, but in practice, users rarely state everything relevant in a query: a single short query is likely inadequate to distinguish the needs of a researcher from those of a physician. By contrast, a reader's background or persona (their role and expertise) is comparatively stable across queries and recovers much of this missing context, which makes it a practical signal for assessing whether a summary satisfies that reader's needs. In this work, we assess how sensitive popular summarization metrics are to both informational and persona differences, and find that many popular metrics, including strong LLM-as-judge metrics, fail basic perturbation tests of informational content. We additionally conduct an expert human evaluation, measuring summary preferences based on information satisfaction given a specific person's background and use case. We find that both traditional and LLM-based metrics are insufficient measures of information satisfaction and agree poorly with human judgment.

1 Introduction

Summary utility depends on whether it serves a particular reader’s informational needs, which general-purpose metrics do not capture. The paper introduces information satisfaction using query–persona pairs and shows that popular metrics fail perturbation tests and agree poorly with expert judgments.

  • Motivation: Summary value is reader- and goal-dependent, whereas ROUGE, BERTScore, and targeted metrics primarily evaluate properties of the summary itself.A researcher and physician may need different information from the same biomedical paper.
  • Information satisfaction: Information satisfaction measures how well a summary resolves a reader’s specific informational need, operationalized as a query paired with a persona.The persona describes the reader’s role and expertise.
  • Information satisfaction: Personas recover background, expertise, and purpose that readers often leave implicit, while remaining stable enough to reuse across queries.A sufficiently detailed query could encode this context, but readers rarely provide one.
  • Research questions: The study tests whether metrics respond to changes in informational content and reader persona, and whether their scores correlate with expert human judgments.These questions motivate systematic perturbation experiments and an expert evaluation using specified queries and personas.
  • Findings: Popular metrics, including strong LLM-as-judge metrics, fail simple informational-content perturbation tests and agree poorly with human judgments of information satisfaction.The results indicate that neither traditional nor LLM-based metrics adequately capture information satisfaction.

2 Related Works

Prior summarization evaluation emphasizes reference overlap, semantic similarity, learned quality judgments, and targeted factual-consistency metrics, while related work also develops query-focused summarization and human-annotated benchmarks. These approaches provide important evaluation foundations but do not by themselves capture the full reader-centered perspective motivating this work.

  • Automatic Evaluation: ROUGE measures n-gram overlap with reference summaries, but rewards surface matching and correlates weakly with human quality judgments.Its simplicity and reproducibility have made it a historically central evaluation metric.
  • Automatic Evaluation: BERTScore and MoverScore compare contextualized representations, while BLEURT learns summary-quality judgments directly from human ratings.These metrics address limitations of surface overlap through embeddings or learned evaluation.
  • Automatic Evaluation: Factual-consistency evaluation includes FactCC, QAGS, QuestEval, and SummaC, targeting specific aspects of summary quality.This represents a parallel line of work focused particularly on factual consistency.
  • Query-Focused Summarization: Query-focused summarization tailors summaries to user-provided queries, spanning query-relevance extractive methods and neural query-conditioned abstractive systems.QFS research is supported by datasets including QMSum and AQua-MuSe.
  • Human Evaluation: Human-annotated benchmarks support meta-evaluation through multidimensional quality ratings and fine-grained factual-error annotations.SummEval evaluates coherence, consistency, fluency, and relevance; FRANK and Tang et al. provide factual-error annotations across systems and datasets.

3 Experimental Setup RQ1

The study evaluates summarization metrics across four scientific datasets using controlled perturbations designed to test sensitivity to informational content, summary completeness, length, and audience. It compares lexical, surface-level, embedding-based, reference-free, and LLM-based metrics, including persona-aware measures of information utility.

  • Datasets: Experiments use test-set subsamples of N=50 documents from four scientific summarization corpora spanning domains, document lengths, and intended audiences.The corpora are arXiv, PubMed, SciTLDR, and eLife, covering abstracts, single-sentence TL;DRs, and lay summaries.
  • Metrics: The evaluation covers lexical-overlap, surface-statistics, syntactic-complexity, embedding-based, reference-free, factuality, and LLM-as-judge metrics.Examples include ROUGE, BLEU, chrF, METEOR, BERTScore, SUPERT, SummaQA, BLANC, FActScore, and dimension-specific LLM judging.
  • Persona-aware metrics: Persona Recall and Persona Precision judge extracted information nuggets against a target persona’s role, domain, information needs, and query.Both metrics decompose summaries into subclaims using an LLM extraction step; Persona Precision measures the fraction of summary nuggets meeting the persona’s needs.
  • Perturbation tests: Five perturbation tests create ordered variants with expected metric directions: distractor injection and different-audience rewriting should decrease scores, incremental addition should increase them, and lengthening or shortening should remain stable.Distractors append up to five sentences from other corpus documents, while incremental addition reconstructs the original summary sentence by sentence.
  • Evaluation criteria: Robustness is measured with Spearman rank correlation ρ and Monotonicity, the fraction of consecutive transitions matching the expected direction.For stable tests, transitions are correct when the absolute score change is below 5% of the baseline score.

4 Human Evaluation Collection RQ2

The human evaluation tests whether summaries conditioned on a reader’s self-reported background are preferred by that reader. Annotators provide persona and query information, compare four summaries in randomized tournaments, and may reject summaries that fail their information needs.

  • Evaluation objective: The evaluation measures each annotator’s preference relative to their own background and query rather than aggregating judgments across annotators.The authors argue that disagreement is expected when readers have different backgrounds, so standard inter-annotator agreement is not meaningful.
  • Annotation protocol: Annotators report expertise, research area, use case, and a genuine information need before submitting a query.The platform then retrieves the top 10 matching paper abstracts from OpenAlex and conditions summary generation on this context.
  • Ranking protocol: Annotators rank the four summaries in a randomized single-elimination tournament, comparing two pairs before a final between the winners.Random bracket assignment prevents annotators from inferring the model or prompting condition.
  • Ranking protocol: Annotators can select “neither” when neither summary meets their information needs, which skips the final if chosen for a first-round pair.Each annotator repeats the evaluation loop for up to eight distinct queries.

5 Perturbation Testing Results RQ1

Perturbation tests show that summarization metrics often respond more reliably to surface-level changes than to meaningful audience shifts. No evaluated metric passes all appropriate tests, and audience-shift rewrites are frequently ignored or scored in the wrong direction.

  • Distractor sentences: For distractor sentences, ROUGE-1/L, BLEU, chrF, METEOR, and compression all reached ρ= −1.0 with M= 1.0.BERTScore, BLANC, and SummaQA produced undefined correlations, while LLM-based results varied by judge.
  • Incremental addition: Incremental addition produced negative correlations between −0.56 and −0.65 for ROUGE-1/2/L, BLEU, chrF, METEOR, and compression.These metrics scored partial summaries higher than complete ones; the Llama-3.3-70B judge was the lone bright spot.
  • Lengthen and shorten prose: On lengthen_prose, overlap metrics, extractive coverage and density, compression, and BERTScore reached ρ= −1.0, with mirrored behavior on shorten_prose.Most LLM Judge dimensions also reached |ρ| = 1.0, indicating strong length sensitivity.
  • Different audience: Under Prometheus-7B, fluency, informativeness, overall, and FActScore showed positive correlations of 0.80–1.00, rating audience-shifted summaries higher than originals.The findings indicate that several LLM-based metrics move opposite to the expected quality drop for different audiences.
  • Summary of robustness findings: No single metric passed all appropriate tests, and every evaluated metric was approximately insensitive to or anticorrelated with audience-shift rewrites.Most metrics passed only distractor injection, while LLM-based metrics partially recovered sensitivity on incremental addition.

6 Human Eval Results RQ2

Human evaluation found that annotators generally preferred DeepSeek-generated and persona-conditioned summaries, especially in final-round comparisons. Existing automatic and LLM-based metrics nevertheless agreed with human preferences at or near chance, including persona-aware variants.

  • Model preferences: 72.8% of final-round matchups favored DeepSeek-V3.1 over Llama-3.3-70B-Instruct, with DeepSeek winning 73.1% of direct first-round comparisons.Both DeepSeek variants outperformed both Llama variants across both tournament rounds.
  • Personalization preferences: 63.2% of final-round comparisons favored personalized over generic summaries, while first-round preferences varied by bracket.Personalized summaries won 46.5% in the A-vs.-B bracket; the C-vs.-D bracket showed a different pattern.
  • Personalization preferences: 1.86 was the odds ratio for personalized DeepSeek beating generic DeepSeek, whereas putting Llama on the personalized side reduced the odds ratio to 0.43.The DeepSeek personalization effect was significant (p = 0.046), as was the Llama penalty (p = 0.016), in 248 comparisons.
  • Metric agreement: α values across reference-based, reference-free, and LLM-judge metric families remained at or near chance agreement with annotator choices.Automatic metrics ranged from −0.029 to 0.173, Prometheus judgments from −0.150 to 0.158, and Llama judgments from −0.179 to 0.053.
  • Metric agreement: 0.173 was the highest reported agreement, achieved by SummaQA F1; persona_precision and persona_recall both reached α = −0.150 under the Prometheus judge.The strongest metrics still showed only slight agreement, while the persona-aware variants designed for informational satisfaction were among the worst performers.

7 Conclusion

The paper introduces information satisfaction as a user-centered axis for evaluating summarization and finds that existing metrics cannot measure it. Traditional, embedding-based, and LLM-as-judge metrics fail robustness checks and do not exceed chance agreement with reader preferences tied to queries and personas.

  • Information satisfaction is introduced as a user-centered axis of summarization evaluation.The paper asks whether existing metrics can measure this axis.
  • Traditional, embedding-based, and strong LLM-as-judge metrics fail basic robustness checks.These findings come from perturbation tests of existing evaluation metrics.
  • Nearly every evaluated metric is insensitive or anticorrelated to audience-shift rewrites, where informational satisfaction varies between users.Audience-shift rewrites target differences in the intended audience and therefore test user sensitivity directly.
  • No metric family exceeds chance agreement with reader preferences anchored to a specific query and persona.An expert human evaluation confirms this limitation.

Metric Thr. Range

The agreement analysis uses metric-specific similarity thresholds calibrated to each metric’s scale and variance, with distinct thresholds spanning lexical, structural, and embedding/model-based metrics.

  • Summary-property metrics: Extractive coverage, extractive density, and compression ratio use thresholds of 0.05, 0.1, and 0.1, respectively.Their listed ranges are [0, 1], [0, ∼∞), and [0, ∼∞).
  • Summary-property metrics: Summary length uses a 5.0 threshold, while novelty rates use 0.05 and repetition rates use 0.01 across 1-, 2-, and 3-grams.Summary length ranges over [0, ∞); novelty and repetition rates range over [0, 1].
  • Non-lexical metrics: BERTScore uses a 0.02 threshold, while SUPERT and both SummaQA variants use 0.05.BERTScore, SUPERT, and SummaQA have ranges [0, 1]; BLANC uses a 0.02 threshold over [−1, 1].
  • Agreement analysis: A metric decision is counted as “Neither” when the absolute difference between two metric scores falls below that metric’s listed threshold.The thresholds reflect each metric’s typical scale and variance.
Loading 2608.14457v1…