Crowdsourcing Lightweight Pyramids for Manual Summary Evaluation

• Pyramid evaluation is considered reliable but requires laborious expert work, while simpler crowdsourced judgments are more subjective and inconsistent across experiments.Source: §1 · p. 1
• The method uses statistical sampling instead of exhaustive SCU extraction and testing to lower Pyramid evaluation cost.Source: §1 · p. 1
• The proposed method correlates with original Pyramid scores better than Responsiveness and shows better stability.Source: §1 · pp. 1–2

• Pyramid creation exhaustively extracts SCU mentions from reference summaries, merges coreferring mentions, and weights each SCU by the number of references containing it.Source: §2 · p. 2
• Pyramid scores normalize the weights of SCUs found in a system summary and average scores across summaries, yielding an essentially recall-style measure.Source: §2 · p. 2
• Automated Pyramid evaluations had not entered mainstream practice, where ROUGE variants still prevailed; this paper therefore compares manual evaluations.Source: §2 · p. 2
• Responsiveness assigns a single Likert-scale quality judgment after evaluators read the source and summary, then ranks systems by average score.Source: §2 · p. 2
• Pairwise comparison selects the preferred summary from two systems at a time, making comparisons across many systems difficult without absolute scores.Source: §2 · p. 2
• The lightweight Pyramid aims to retain interpretable, relatively objective scores while enabling absolute comparisons among multiple systems.Source: §2 · p. 2

Pyramid creation

• Two crowd workers extract eight brief, single-fact SCU-like statements from each reference summary, using only the text and spreading selections across it.Source: §3 · p. 2
• The method samples 32 SCUs per topic rather than exhaustively merging coreferential SCUs across references, allowing repeated facts to affect evaluation more often.Source: §3 · p. 3

System evaluation

• For system evaluation, workers judge whether each of 16 SCUs can be inferred from a summary, with five workers assigned to each SCU in the experiments.Source: §3 · p. 3

Scoring

• Workers with pairwise agreement below 0.5 are filtered, remaining votes are decided by majority, and ties default to “not present.”Source: §3 · p. 3
• Scores are the percentage of judged SCUs matched, averaged over topics; using average answers instead of majority votes performed significantly worse in the experiments.Source: §3 · p. 3

Experimental setup and cost analysis

• The evaluation used 32 sampled crowdsourced unweighted SCUs per topic, versus an average of 105 expert-weighted SCUs; cost reductions were found harmful to correlation.The study used all 20 DUC 2006 topics and 10 randomly selected DUC 2005 topics; system evaluation cost $5.4 per system per topic.Source: §4 · pp. 3–4

Resources

• The released DUC 2005/2006 SCUs, crowdsourcing task pages, and processing and evaluation scripts support future crowdsourced Pyramid evaluations.The authors say the materials can support evaluations of new systems on these datasets and development of datasets with crowdsourced pyramids.Source: §4 · p. 4

Correlations with original Pyramid

• The crowdsourced method correlates better with original Pyramid scores than expert Responsiveness, substantially so on DUC 2006.Table 1 reports the comparison; the Responsiveness scores were expert judgments, and the authors expect the gap to be larger against crowdsourced Responsiveness.Source: §4 · p. 4, Table 1 · p. 4

Stability

• 10% average relative change for lightweight Pyramid versus 24% for crowdsourced Responsiveness in reruns of eight randomly chosen 2006 systems.The authors use relative change in overall system scores as their measure of reproducibility.Source: §4 · p. 4

Number of SCUs

• Correlation increases with judged SCUs per topic, with improvement seemingly stabilizing around 32 SCUs.Figure 1 plots average Pearson and Spearman correlations on DUC 2005 and 2006.Source: §4 · p. 4, Figure 1 · p. 4

Number of topics

• Correlation rises steadily as more topics are evaluated and does not necessarily saturate at the 20 topics available.Figure 2 plots average Pearson and Spearman correlations on DUC 2005 and 2006.Source: §4 · p. 4, Figure 2 · p. 4

Qualitative analysis

• Some topics are harder for crowd workers; a preliminary filter of the 20% lowest-agreement topics raised Spearman correlation by 6 points.The increase was relative to Table 1 and used the original Pyramid ranking over the full topic set; the authors call for further analysis.Source: §4 · p. 5
• On some difficult topics, the 32 sampled SCUs miss important information, causing unjustified degradation in system scores.The authors suggest investigating sample sizes that vary with automatically detected topic complexity.Source: §4 · p. 5

• The lightweight Pyramid is more reliable than common Responsiveness and allows absolute scores for comparing multiple systems, unlike pairwise comparison.Source: §5 · p. 5
• Future work could improve correlation with the original Pyramid, reduce annotation cost, and test datasets lacking original Pyramid evaluations.Suggested approaches include reducing crowdsourcing noise; CNN/DailyMail is one example for further investigation.Source: §5 · p. 5
• The lightweight Pyramid primarily measures recall; precision, nonredundancy, and grammaticality are complementary summary-quality measures.Source: §5 · p. 5