Source-linked AI summary
Summary-Source Proposition-level Alignment: Task, Datasets and Supervised Baseline
Ori Ernst, Ori Shapira, Ramakanth Pasunuru, Michael Lepioshkin, Jacob Goldberger, Mohit Bansal, Ido Dagan
TL;DR
Summary-source alignments were commonly generated with unsupervised methods and not evaluated independently, despite their use in creating summarization training data. This paper introduces proposition-level alignment datasets and a supervised baseline, which outperforms lexical and semantic similarity methods.
Problem
Method
The paper aligns proposition spans called information units and trains a supervised binary classifier using a 23K-instance dataset, alongside crowdsourced development and test datasets.
Source: §1 · p. 2, §3.1 · p. 3, §5.3 · p. 7Results
The supervised SuperPAL model outperforms lexical and semantic similarity methods; Sim-Ensemble scores 16 F1 points lower than SuperPAL, whose average Jaccard score is 0.67.
Source: §5.4 · p. 7, §7 · p. 9Takeaways & Limitations
The dataset suite can support improved proposition-level aligners and proposition-based summarization models, including models for summarization components.
Source: §7 · p. 9
Abstract
from arXiv · showhide
Aligning sentences in a reference summary with their counterparts in source documents was shown as a useful auxiliary summarization task, notably for generating training data for salience detection. Despite its assessed utility, the alignment step was mostly approached with heuristic unsupervised methods, typically ROUGE-based, and was never independently optimized or evaluated. In this paper, we propose establishing summary-source alignment as an explicit task, while introducing two major novelties: (1) applying it at the more accurate proposition span level, and (2) approaching it as a supervised classification task. To that end, we created a novel training dataset for proposition-level alignment, derived automatically from available summarization evaluation data. In addition, we crowdsourced dev and test datasets, enabling model development and proper evaluation. Utilizing these data, we present a supervised proposition alignment baseline model, showing improved alignment-quality over the unsupervised approach.
1 Introduction
The paper establishes summary-source alignment as a supervised, proposition-level task, addressing prior alignments that were used to generate training data but not independently optimized or evaluated. It introduces new datasets and a baseline that outperforms traditional unsupervised alignment approaches.
1 Introduction
- Prior summary-source alignments generated training data for summarization subtasks, but their quality was neither explicitly optimized nor evaluated.These alignments supported salience detection, redundancy recognition, and text rephrasing and fusion.Source: §1 · p. 1
- The paper establishes proposition-level summary-source alignment as a stand-alone auxiliary task with new datasets, a model, and intrinsic evaluation.It uses information units (IUs), individual proposition spans, to produce tighter alignments than full sentences, which can contain matching and non-matching propositions.Source: §1 · p. 1
- The authors derive a 23K-instance training dataset from Multi-Document Summarization evaluation data and create crowdsourced development and test datasets.The training data comes from Pyramid annotations.Source: §1 · pp. 1–2
- The supervised baseline trained on the new data outperforms traditional unsupervised alignment approaches.It can produce abstractive alignments with almost no lexical overlap, where traditional aligners fail.Source: §1 · p. 2
- The proposition-level aligner extracts more accurate alignments than sentence-level methods, and its resources may support proposition-based summarization research.The conclusion suggests using aligned information units to derive proposition-level salience data and train summarization components.Source: §1 · p. 2
2 Background and Related Work
Earlier work used automatically generated sentence alignments to derive training data for summarization subtasks, generally with unsupervised similarity measures. The paper situates proposition-level alignment in Pyramid evaluation and prior efforts to extract proposition-like units.
2 Background and Related Work
- Prior work used reference-source sentence alignments to derive training data for salience detection and sentence fusion.Fusion instances paired source sentences aligned to the same summary sentence, with that summary sentence as the output.Source: §2 · p. 2
- Automatically detected reference-source sentence alignments, typically based on ROUGE, supplied training data for summarization subtasks, but alignment quality was not explicitly assessed.Earlier evaluations generally focused on the final summary rather than the underlying alignments.Source: §2 · p. 2
- The Pyramid method evaluates information overlap between system and reference summaries by aligning proposition-level Summary Content Units.The passage notes that a system-summary sentence may contain both propositions that match the reference and ones that do not.Source: §2 · p. 2
- Automated Pyramid-related work extracted proposition-level units using Open IE or Elementary Discourse Units, while this paper aligns proposition spans extracted with Open IE.The passage describes proposition units as potentially sharing arguments or being discontiguous, and says Open IE output best suits those requirements.Source: §2 · pp. 2–3
3 Dev and Test Alignment Datasets
The section introduces manually annotated development and test datasets for aligning reference-summary information units with corresponding source-document propositions, enabling direct tuning and evaluation of alignment algorithms.
3 Dev and Test Alignment Datasets
- The datasets support direct tuning and evaluation of reference-source alignment algorithms, which prior work lacked.The section presents manually annotated development and test datasets.Source: §3 · p. 3
3.1 Dataset Structure
- Each reference-summary information unit is aligned with source propositions conveying the same information, using proposition spans rather than whole sentences.The paper adopts a somewhat looser information-unit definition to facilitate crowdsourcing.Source: §3.1 · p. 3
3.2 Source Data
- The data cover 21 topics from Multi-News, DUC 2004, and DUC 2007, with one reference summary per topic.Multi-News topics have 3–4 documents each; DUC topics have seven documents each.Source: §3.2 · p. 3
3.3 Annotation
- Annotation proceeds from extracting summary IUs to filtering unlikely source sentences, then verifying and marking aligned source IU spans.Automatic filtering combines BERTscore, an MNLI-based entailment score, and ROUGE-1 precision; crowdsourcing filters remaining candidates before span annotation.Source: §3.3 · p. 4
- Expert review produced 345 development and 312 test reference-source IU alignments with expert-level precision.Workers annotated source IU spans or indicated that no alignment was present.Source: §3.3 · pp. 3–4
- 76% recall against expert alignments and about 0.85 average Jaccard similarity indicate substantial coverage with mostly one- or two-word boundary differences.Crowdsourced and expert annotations were compared using a Jaccard match threshold of 0.25 on both summary and source spans.Source: §3.3 · p. 5
4 Pyramid-Based Training Dataset
The paper derives proposition-level training alignments from Pyramid evaluation data, then creates automatically extracted IU versions and negative examples for supervised alignment training.
4 Pyramid-Based Training Dataset
- The derivation extends extractive system-summary links and Pyramid mappings between reference and system summary spans to align reference summaries with source documents.The resulting dataset is less exhaustive than the manually annotated datasets but was sufficient for training neural alignment models.Source: §4 · p. 5
- 18,505 proposition-level alignments are derived by linking reference-summary IU spans to corresponding document IU spans through Pyramid mappings.The alignments are not exhaustive because Pyramid mappings cover only sentences included in evaluated system summaries.Source: §4 · p. 5
- Matching automatically extracted IUs against Pyramid-based alignments yields 23,492 OIE-based training alignments.A candidate pair is retained when each OIE IU matches its corresponding Pyramid-aligned IU above a Jaccard threshold of 0.25.Source: §4 · p. 5
- The training set adds 219,772 negative examples, with sampled error rates of 11% for positives and 5% for each negative type.Negative examples include high-BERTscore non-alignments and naturally distributed non-aligned IU combinations; 22% of negatives were selected for high BERTscore.Source: §4 · p. 5
5 IU Alignment Methods and Evaluation
The supervised SuperPAL aligner outperforms the ROUGE-based and semantic-ensemble baselines on proposition-level alignment. The section compares IU-level methods and evaluates their performance across datasets.
5 IU Alignment Methods and Evaluation
- Open IE extracts candidate IU spans, but missed extractions lose about 10% of gold alignments and cap automatic alignment recall at 90%.Source: §5 · p. 6
5.1 ROUGE-based Lexical Model
- ROUGEIU matches each summary IU to its two highest-ROUGE document IUs, using the top-k setting that performed best on the development set.Its ROUGE similarity averages R-1, R-2, and R-L recall, treating the summary IU as the reference.Source: §5.1 · p. 6
5.2 Semantic Similarity Ensemble Model
- Sim-Ensemble first retrieves the three most similar source sentences using tuned ROUGE, RoBERTa-MNLI, and BERTScore, then uses word alignment to identify IU spans.Source: §5.2 · p. 6
5.3 Supervised Model
- SuperPAL is a binary IU-alignment classifier built from an MNLI-fine-tuned RoBERTa encoder and a new classification layer, further fine-tuned on Pyramid-based data.Source: §5.3 · p. 7
5.4 IU-level Results
- SuperPAL scores 16 F1 points above Sim-Ensemble, which scores 6 F1 points above ROUGEIU; SuperPAL’s average Jaccard score is 0.67.The comparisons are reported against the gold development and test datasets.Source: Table 3 · p. 6, §5.4 · p. 7
- On DUC, ROUGEIU collapses while SuperPAL degrades more gracefully than on MultiNews, a pattern the authors associate with DUC’s more abstractive content.The authors suggest that ROUGEIU’s lexical basis is inadequate for abstractive alignments with low lexical overlap.Source: §5.4 · p. 7, Table 4 · p. 7
- The authors release SuperPAL-produced proposition alignments for MultiNews and CNN/DailyMail, covering multi-document and single-document summarization.Source: §5.4 · p. 7
6 IU-enhanced Sentence Alignment
The section tests whether proposition-level alignments can support sentence-level salience training and summarization. SuperPALsent improves salient-sentence detection and performs better for longer summaries, while two-sentence summaries slightly favor ROUGE-based training data.
6.1 Salient-sentence Extraction
- The method scores document sentences by length-weighted IU alignment scores, then selects the highest-scoring sentences until all aligned summary IUs are covered.Source: §6.1 · p. 8
- SuperPAL-derived salient sentences match the gold set better than ROUGE-based alternatives in both recall and precision.Recall measures covered summary IUs, while precision measures the share of extracted-sentence tokens belonging to aligned IUs.Source: §6.1 · p. 8, Table 8 · p. 8
6.2 Extrinsic Summarization Evaluation
- At a two-sentence summary length, ROUGE-based training data has a small advantage; for longer summaries, SuperPALsent is not harmful and can improve summarization performance.Source: §6.2 · pp. 8–9, §6 · p. 8
- For summaries longer than two sentences, SuperPALsent outperforms ROUGEsent on all reported measures, significantly by 95% confidence intervals, and achieves the highest global result across lengths.The advantage grows with summary length; CNN/DailyMail summaries average 3.8 sentences.Source: §6.2 · pp. 8–9, Table 7 · p. 8
7 Conclusion and Discussion
The paper presents proposition-level summary-source alignment as a route to cleaner alignments and proposes released resources for developing alignment and summarization methods. It outlines several potential uses for these resources, while noting a coherence trade-off for one future summarization approach.
7 Conclusion and Discussion
- The supervised proposition-level aligner outperforms existing lexical and semantic similarity methods and can yield abstract alignments.The paper advocates proposition-level alignment as a way to obtain cleaner, more accurate alignments than the common sentence-level approach.Source: §7 · p. 9
- Aligned information units can be marked salient to derive proposition-level salience data for training summarization components.The released resources are proposed as a basis for developing improved proposition-based alignment methods.Source: §7 · p. 9
- Proposition-based extractive summaries could offer optimized content in bullet form, albeit with somewhat less coherence.The paper illustrates this potential using an oracle-system summary derived from supervised aligner predictions on CNN/DailyMail.Source: §7 · p. 9
- The data may also contribute to highlighting salient fragments or to selecting and fusing propositions into coherent abstractive summaries.The paper points to a cascaded fragment-selection and generation approach as one related direction.Source: §7 · p. 9
- The authors suggest the released resources could encourage research on proposition-based summarization and improved alignment models for a challenging semantic-matching task.Source: §7 · p. 9
A Annotation Guidelines
The annotation guidelines define information units as standalone facts and describe how annotators extract them, filter candidate alignments, and mark shared information across sentences. Alignment targets the information in a bold span and excludes non-shared content.
A Annotation Guidelines
- An information unit is a standalone fact, and units must cover the whole sentence.Annotators are instructed to find verbs and split around them, keeping each verb with its arguments; a unit may contain multiple verbs when one serves as an argument of another.Source: A Annotation Guidelines · p. 11
- The guidelines illustrate unit extraction by splitting one sentence into separate facts about people being injured and receiving treatment.Source: A Annotation Guidelines · p. 11
- Units should include the subject when possible, may be discontinuous, and may reuse words, but should not repeat a whole fact.The guidelines say a unit need not be grammatically valid or continuous.Source: A Annotation Guidelines · p. 11
- Non-alignment filtering asks whether each secondary sentence contains the significant information in the primary sentence’s bold span, then marks the aligned span.The full primary sentence is context; the questions include whether information repeats, can replace a span without losing meaning or information beyond side details, or is implicated.Source: A Annotation Guidelines · p. 11
- Alignment must concern the same specific information as the bold span; a broader related topic is insufficient when substituting the span loses significant information.The example contrasts a span about a political crisis with sentences about Hun Sen rejecting demands for talks outside the country.Source: A Annotation Guidelines · pp. 11–12
- For information-unit alignment, annotators highlight maximal spans in both sentences that contain only shared information and form a standalone fact.Add words needed to make a standalone fact even if they are not exactly aligned; omit non-shared information, and choose “Not aligned” when there is no joint information.Source: A Annotation Guidelines · p. 12
B A Full Alignment Example
The example depicts summary-source alignment pairs and explains that pairs can be grouped into clusters when they share a summary information unit. Its displayed example omits intersecting information units for clarity, although such intersections can occur.
B A Full Alignment Example
- Alignment pairs are marked in the same color, and pairs sharing a summary information unit can form an alignment cluster.The dataset itself is pairwise; the red spans in the document illustrate the shared-unit cluster possibility.Source: B A Full Alignment Example · p. 12
- The example omits intersecting information units for ease of presentation, though intersections are possible.Source: Figure 3 · p. 13