Information Satisfaction: A Reader-Centered Axis for Summarization Evaluation

1 Introduction

• Existing evaluation treats quality as a property of the summary, although usefulness depends on the reader; standard metrics may miss audience-specific informational needs, while some approaches model style rather than information [39].For example, researchers may need methods and statistics, while physicians may need clinical implications and contraindications from the same source.Source: §1 · p. 1
• Information satisfaction evaluates how well a summary resolves the need that motivated a reader to consult its source, using a query paired with a persona describing the reader’s role and expertise.The query expresses the immediate question, while the comparatively stable persona supplies background and purpose that readers often leave implicit.Source: §1 · pp. 1–2
• The first research question tests whether traditional reference-based and LLM-as-judge metrics respond when summary information or reader persona changes.The study systematically varies both axes to probe sensitivity to information satisfaction.Source: §1 · p. 2
• The second research question tests whether automatic metric scores correlate with expert judgments of summaries rated for a specified query and persona.The human evaluation measures agreement between those judgments and metric scores.Source: §1 · p. 2
• Many popular metrics, including strong LLM-as-judge metrics, fail informational-content perturbation tests and agree poorly with human judgments of information satisfaction.The findings suggest that neither traditional nor LLM-based metrics adequately capture information satisfaction.Source: §1 · p. 2

2 Related Works

• Automatic metrics range from ROUGE and contextual or learned metrics to factual-consistency measures and LLM evaluators, yet assume summary quality is reader-independent.ROUGE rewards surface overlap rather than semantic equivalence and correlates only weakly with human judgments of quality [4].Source: §2 · p. 2
• Query-focused and persona-conditioned summarization exist, but evaluation largely compares query-specific summaries with references rather than testing whether metrics detect persona-relevant informational differences.This work pairs each query with an explicit reader persona.Source: §2 · p. 2

Human Evaluation

• Human-evaluation datasets use expert quality ratings, factual-error annotations, or pairwise preferences without reader goals; this work anchors judgments to a persona and query to measure information satisfaction.Examples include SummEval [13], FRANK [45], Tang et al. [53], and pairwise-preference benchmarks [21] [38].Source: §2 · p. 2

3 Experimental Setup RQ1

• Experiments use 50 test documents from each of four corpora spanning scientific domains, document lengths, and audiences: arXiv, PubMed [8], SciTLDR [5], and eLife [20].The corpora pair scientific articles with abstracts, single-sentence TL;DRs, or editor-written lay summaries.Source: §3 · p. 2
• The metric suite includes ROUGE-1/2/L, BLEU, chrF [47], METEOR, DataStats measures [23], and a spaCy-based syntactic-complexity baseline.DataStats measures extractive coverage and density, compression, novel n-grams, and summary length; syntactic complexity is measured in words and sentences.Source: §3 · pp. 2–3
• Non-lexical evaluation includes BERTScore and the reference-free, source-based metrics SUPERT [19], SummaQA [49], and BLANC [54].BERTScore computes token-level cosine similarity using contextual embeddings.Source: §3 · p. 3
• LLM evaluation compares Llama-3.3-70B and Prometheus-7B [22] [28] judges using FActScore [43], multidimensional LLM-as-judge scoring, and annotator-mirroring prompts.The multidimensional judge scores relevance, consistency, fluency, coherence, informativeness, and an overall score.Source: §3 · p. 3
• Persona Recall and Persona Precision judge extracted summary nuggets against a reader’s role, domain, information needs, and query.Precision penalizes persona-irrelevant nuggets; Recall measures coverage of persona- and source-implied information requirements, penalizing omissions.Source: §3 · p. 3

3.1 Perturbation tests

• Five perturbations create progressively changed summaries: random distractor sentences, incremental reconstruction from empty text, lengthening, shortening, and rewriting for a different audience.The expected metric response is decrease for distractors and audience shifts, increase for incremental additions, and stability for length changes.Source: §3.1 · p. 3
• Metrics are assessed using Spearman rank correlation and monotonicity against perturbation level; stable-test transitions count as correct when score changes remain below 5% of baseline.Monotonicity is the fraction of consecutive transitions matching the expected direction.Source: §3.1 · p. 3

Human evaluation objective

• The evaluation measures summary preference relative to the specific annotator who supplied the query and persona, rather than aggregating judgments across readers.The study asks whether conditioning generation on a reader’s self-reported background yields summaries that reader prefers.Source: §4 · p. 3

Annotation protocol

• Each session elicits a persona, collects a free-form query, retrieves 10 OpenAlex abstracts, and has the annotator rank four summaries; annotators complete up to eight queries.The persona profile combines self-reported expertise, research area, and use case.Source: §4 · pp. 3–4

Summary generation

• A 2×2 design crosses Llama-3.3-70B [22] and DeepSeek-V3.16 [11] with persona-conditioned and unconditioned prompting, holding retrieved context constant.A shared prompt asks models to ground answers in retrieved abstracts and use bracketed in-text citations [18] [35].Source: §4 · p. 4

Ranking protocol

• A randomized single-elimination pairwise tournament reduces ranking effort from six judgments to three, with a “neither” option when neither summary meets the annotator’s needs.The final round is skipped if “neither” is selected in one or both first-round pairs; pairwise comparisons are described as more reliable than absolute Likert ratings [29] [40].Source: §4 · p. 4

Annotator recruitment

• Twenty recruited volunteers with varied domains and roles provide judgments based on genuine information needs, rather than fabricated crowdworker queries.Annotators could skip assigned queries that did not reflect a genuine need; summaries were generated live for each annotator’s query.Source: §4 · p. 4

Summary of robustness findings

• No metric passes all expected tests; most pass only distractor injection, and every metric is insensitive or anticorrelated to audience-shift rewrites.These results motivate the paper’s human evaluation of information satisfaction.Source: Table 2 · p. 6, §5 · p. 6

Distractor sentences

• Lexical overlap metrics behave as expected on distractor sentences, but BERTScore, BLANC, and SummaQA yield undefined correlations and LLM judges disagree.Under Llama-3.3-70B, only FActScore and coherence pass; under Prometheus-7B, a different, largely non-overlapping subset passes.Source: §5 · p. 4, Table 2 · p. 6

Incremental addition

• ROUGE-1/2/L, BLEU, chrF, METEOR, and compression correlate negatively with summary completion (ρ = −0.56 to −0.65), while Prometheus-7B fails every dimension.Llama-3.3-70B dimensions correlate in the expected direction (ρ = 0.42–0.58), but have low monotonicity (M ≤ 0.13); no reference-based metric reliably distinguishes partial from complete summaries.Source: §5 · p. 4, Table 2 · p. 6

Lengthen and shorten prose

• Nearly all metrics are length-sensitive: overlap metrics, extractive coverage and density, compression, and BERTScore reach ρ = −1.0 when prose is lengthened, with mirrored behavior when shortened.SummaQA, SUPERT, and a few LLM dimensions are exceptions; this extends the documented ROUGE failure to BERTScore and most LLM-judge dimensions [52].Source: §5 · p. 4, Table 2 · p. 6

Different audience

• Audience-shift rewrites often go undetected or are rated higher: Prometheus-7B gives fluency, informativeness, overall, and FActScore strong positive correlations (0.80–1.00).Only the Llama-3.3-70B annotator-style judge passes; under Prometheus-7B, only consistency and persona recall pass, while several other metrics fail outright.Source: §5 · pp. 4, 6, Table 2 · p. 6

6 Human Eval Results RQ2

• Across automatic, Prometheus, and Llama judge metrics, Krippendorff’s α means were 0.049, −0.022, and −0.087, respectively, remaining near chance.Persona-aware persona_precision and persona_recall scored α = −0.150 under Prometheus and at or below chance under Llama; even the strongest metric, SummaQA F1, reached only α = 0.173.Source: Table 3 · p. 7, §6 · pp. 7–8
• DeepSeek-V3.1 won 72.8% of final-round matchups against 27.2% for Llama-3.3-70B-Instruct, and 73.1% of direct first-round cross-model comparisons.Both DeepSeek variants outperformed both Llama variants in both rounds.Source: §6 · pp. 7–8, Table 4 · p. 8
• Personalized summaries won 63.2% of final-round comparisons, but first-round win rates differed by bracket: 46.5% in A vs. B and 62.2% in C vs. D.The authors hypothesized that including weaker Llama outputs diluted first-round personalization preferences.Source: §6 · pp. 7–8, Table 4 · p. 8
• With model identity controlled, personalized DeepSeek summaries beat generic DeepSeek summaries (OR = 1.86), while personalized Llama summaries had lower odds of winning (OR = 0.43).The round-stage effect was not significant (OR = 0.82, p = 0.53), supporting the interpretation that raw round differences reflected model composition.Source: §6 · pp. 7–8, Table 4 · p. 8

• Traditional, embedding-based, and strong LLM-as-judge metrics fail robustness checks; nearly all are insensitive or anticorrelated to audience shifts, and none exceed chance agreement with reader preferences.The preferences are anchored to a specific query and persona.Source: §7 · p. 8
• Evaluation must model who a summary serves and what they seek, but persona-conditioned variants still fail because nugget relevance does not reliably measure reader satisfaction.The variants incorporate the reader’s role, domain, and information needs, suggesting the bottleneck is how satisfaction is measured rather than missing persona information.Source: §7 · p. 8
• A stronger information-satisfaction metric should weight information by reader-specific importance, account for prior knowledge, and capture comparative judgments between adequate summaries.The paper argues against treating every information requirement as equally and independently satisfiable.Source: §7 · p. 8
• The analysis and human-annotated dataset provide a foundation for future research on evaluating information satisfaction.Source: §7 · p. 8

Metric Thr. Range

• Thresholds were 10.0 for Words, 1.0 for Sentences, 2.0 for Verb phrases, 2.0 for Clauses, and 1.0 for T-units (all [0, ∞)).A metric decision is counted as “Neither” when the metric-score difference falls below its threshold.Source: Metric Thr. Range · p. 11, Table 5 · p. 11
• Thresholds were 1.0 for Dependent clauses, 1.0 for Complex T-units, 1.0 for Coordinate phrases, and 2.0 for Complex nominals (all [0, ∞)).A metric decision is counted as “Neither” when the metric-score difference falls below its threshold.Source: Metric Thr. Range · p. 11, Table 5 · p. 11
• Ratio thresholds were 3.0 Words / sentence, 3.0 Words / T-unit, 2.0 Words / clause, 0.5 Clauses / sentence, 0.5 Verb phrases / T-unit, 0.5 Clauses / T-unit, and 0.1 Dep. clauses / clause.A metric decision is counted as “Neither” when the metric-score difference falls below its threshold.Source: Metric Thr. Range · p. 11, Table 5 · p. 11
• Other ratio thresholds were 0.2 Dep. clauses / T-unit, 0.3 T-units / sentence, 0.2 Complex T-units / T-unit, 0.2 Coord. phrases / T-unit, 0.1 Coord. phrases / clause, 0.5 Complex nominals / T-unit, and 0.3 Complex nominals / clause.A metric decision is counted as “Neither” when the metric-score difference falls below its threshold.Source: Metric Thr. Range · p. 11, Table 5 · p. 11
• LLM Judge (per-dimension) thresholds were 0.5 for Relevance, Coherence, Consistency, Fluency, Informativeness, and Overall (all [5]).A metric decision is counted as “Neither” when the metric-score difference falls below its threshold.Source: Metric Thr. Range · p. 11, Table 5 · p. 11