Source-linked AI summary
ASPECTNEWS: Aspect-Oriented Summarization of News Documents
Ojas Ahuja, Jiacheng Xu, Akshay Gupta, Kevin Horecka, Greg Durrett
TL;DR
Generic and query-based summaries do not capture users’ interest in broad subtopics shared across similar news articles. The authors collect ASPECTNEWS and train a keyword-conditioned system on synthetic data without in-domain training examples. The system produces summaries that better match aspect annotations than generic and keyword-matching baselines, and responds to keyword choice.
Problem
Users may seek broad aspects shared across similar articles, but generic summaries and document-specific queries do not directly match that need.
Source: §1 · p. 1Method
The authors collect aspect-oriented extractive summaries for earthquake and fraud news and train a keyword-conditioned system using synthetic data derived from generic summaries.
Source: §1 · p. 1Results
The system scores higher in agreement with human aspect annotations than generic models, previous aspect-oriented models, and keyword matching, and its summaries respond to keyword choice.
Source: §1 · p. 2Takeaways & Limitations
The results support keyword-controllable summaries for broad subtopics in the evaluated news domains, without training on those target domains.
Source: §1 · p. 2, §7 · p. 8Scaling the dataset is difficult because identifying suitable domains and user intents requires human effort, while collecting real test cases at scale would require a more involved user study.
Source: §4 · p. 4
Abstract
from arXiv · showhide
Generic summaries try to cover an entire document and query-based summaries try to answer document-specific questions. But real users' needs often fall in between these extremes and correspond to aspects, high-level topics discussed among similar types of documents. In this paper, we collect a dataset of realistic aspect-oriented summaries, AspectNews, which covers different subtopics about articles in news sub-domains. We annotate data across two domains of articles, earthquakes and fraud investigations, where each article is annotated with two distinct summaries focusing on different aspects for each domain. A system producing a single generic summary cannot concisely satisfy both aspects. Our focus in evaluation is how well existing techniques can generalize to these domains without seeing in-domain training data, so we turn to techniques to construct synthetic training data that have been used in query-focused summarization work. We compare several training schemes that differ in how strongly keywords are used and how oracle summaries are extracted. Our evaluation shows that our final approach yields (a) focused summaries, better than those from a generic summarization system or from keyword matching; (b) a system sensitive to the choice of keywords.
1 Introduction
Generic summaries cover documents without tailoring to users' varied information needs, while aspect-oriented summaries target broad, recurring subtopics. ASPECTNEWS evaluates keyword-conditioned systems without in-domain training data and finds them more focused than generic and keyword-matching baselines.
1 Introduction
- Aspect-conditioned summaries address users’ broad information needs, which fall between generic summaries and highly focused document-specific queries.For example, an earthquake researcher may seek information about recovery efforts and social impacts rather than a general account of what happened.Source: §1 · p. 1
- ASPECTNEWS provides evaluation data for single-document aspect-oriented extractive summarization in earthquake and fraud news, using soft labels from annotators rather than consensus summaries.The dataset focuses on imagined user needs and two distinct aspects for each domain.Source: §1 · p. 1
- The authors generate synthetic aspect-oriented training data from generic summaries to train a system conditioned on aspect-level keywords, without annotated training data for those aspects.They compare training schemes and prior approaches for adapting to keyword inputs.Source: §1 · p. 1
- The model scores higher in agreement with human aspect annotations than generic models, previous aspect-oriented models, and keyword matching, and its summaries respond to keyword choice.The experiments also report competitive performance on SPACE and significant hallucination by abstractive query-focused systems in this setting.Source: §1 · p. 2
2 Related Work
The paper distinguishes recurring, broad aspects from document-specific queries and prior multi-document aspect settings. ASPECTNEWS emphasizes single-document evaluation and generalization to aspects unseen in training.
2 Related Work
- Query-focused summarization addresses document-specific questions [Baumel et al., 2014; Krishna and Srinivasan, 2018; Frermann and Klementiev, 2019; He et al., 2020; Xu and Lapata, 2020a], unlike recurring aspects such as geography or recovery across earthquake stories.The paper relates aspect-oriented summarization more closely to attribute extraction from reviews and opinions [Angelidis and Lapata, 2018].Source: §2 · p. 2
- ASPECTNEWS evaluates single-document summaries and generalization to new aspect types, whereas SPACE and related attribute-extraction datasets are multi-document settings [Angelidis et al., 2021; Angelidis and Lapata, 2018].The paper frames broad subtopics as closer to users’ information needs than specific queries.Source: §2 · p. 2
- Earlier guided-summary tasks used fixed aspects that often resembled generic summaries, while DUC offered summary granularity rather than aspects.The TAC tasks have fixed aspect sets, and Christensen et al. (2014) produce hierarchies of short summaries across documents.Source: §2 · p. 3
- Other approaches construct document-specific keyword sets or condition on metadata topic tokens, while this work emphasizes evaluation of aspects rather than a purely keyword- or query-driven view.The cited approaches include He et al. (2020), Xu and Lapata (2020a), Tan et al. (2020), Krishna and Srinivasan (2018), and Frermann and Klementiev (2019).Source: §2 · p. 3
3 Aspect-Oriented Data Collection
The authors collect aspect-oriented extractive summaries for earthquake and fraud articles by defining user intents and crowdsourcing sentence selections. Their analysis finds annotator disagreement and substantial differences from generic extractive oracles.
3 Aspect-Oriented Data Collection
- The collection process identifies article clusters, specifies multiple intents per domain, and crowdsources extractive summaries; it targets users’ specific information needs.The dataset is intended to simulate realistic information needs.Source: §3 · p. 3
3.1 Target Domains
- The earthquake and fraud domains have recurring aspects: geography and recovery for earthquakes, and penalty and fraud nature for fraud reports.The authors selected these domains because each had over 200 on-topic articles after filtering and articles were reasonably homogeneous; legislation was too heterogeneous for a unified information need.Source: §3.1 · p. 3
- The authors encode CNN/Daily Mail articles and retrieve those similar to a domain exemplar, finding this more robust than keyword matching for coherent aspects.Keyword matching for “earthquakes” returned primarily tsunami articles due to an imbalanced distribution.Source: §3.1 · p. 3
3.2 Specifying User Intents
- For each domain, the authors define two well-represented aspects and their prompts and keywords based on article inspection and imagined user scenarios.One penalty scenario came from colleagues’ use case of triaging news reports of wrongdoing.Source: §3.2 · p. 4
3.3 Crowdsourcing
- Crowdworkers select and rate article sentences against an intent prompt, and all sentences rated at least 1 form the submitted summary.Articles were truncated to 10 sentences; annotations without a selected sentence in common with another annotator were discarded.Source: §3.3 · p. 4
3.4 Data Analysis & Annotator Agreement
- Annotators’ disagreement is treated as inherent to the task, and aspect-driven summaries have low similarity and exact match with generic extractive oracles.The authors preserve disagreement in evaluation rather than computing consensus summaries because annotators can be unsure which information merits inclusion.Source: §3.4 · p. 4
4 Building an Aspect-Oriented System
The system is built without gold-labeled aspect-oriented training data by synthesizing keyword-conditioned examples from CNN/DM. Keywords are selected from TF-IDF-ranked document tokens that also appear in reference summaries.
4 Building an Aspect-Oriented System
- Scaling the data collection is difficult because identifying suitable domains and specifying user intents requires significant human effort.Collecting real test cases at scale would require a more involved user study.Source: §4 · p. 4
- The system learns keyword-to-summary relationships from CNN/DM without gold-labeled aspect-oriented training data.It generates keywords for each article and trains on the relationship between those keywords and a summary.Source: §4 · p. 4
- Table 4 compares top-three sentence sets using Jaccard similarity and exact-match percentage; 100 examples per aspect make EM integer-valued.Source: §4 · p. 4, Table 4 · p. 5
- The Table 5 example includes both highly specific concepts and broad-topic keywords, which a model can learn to leverage with appropriate reference summaries.Source: Table 5 · p. 5
4.1 Keyword-controlled Data
- Mixed augmentation trains on both the original document-summary pair and a keyword-conditioned pair with a possibly modified summary.Both forms are used for a given document.Source: §4.1 · p. 5
- TF-IDF ranks document tokens, and the keyword extraction process keeps tokens that appear in the reference summary.The selected tokens are intended to be more consequential in affecting the output summary.Source: §4.1 · p. 5
4.2 Aspect-Oriented Model
AOSUMM predicts a summary from a document-keyword pair using a BERT-based extractive summarization model. Keywords are user-defined at inference, while modified oracle summaries provide training targets.
4.2 Aspect-Oriented Model
- AOSUMM fine-tunes BERT for extractive summarization to predict a summary from a document-keyword pair.The training scheme follows BERTSUM and uses modified oracle extractive summaries as gold outputs.Source: §4.2 · p. 6
- During training, a special token followed by keywords is prepended to the document, and user-defined keywords are supplied at inference.Source: §4.2 · p. 6
5 Experiments
Experiments evaluate AOSUMM on ASPECTNEWS and SPACE, with baselines and analyses of metrics, keyword sensitivity, and keyword intensity. On ASPECTNEWS, AOSUMM outperforms baselines across four reported metrics; on SPACE, it is competitive with strong aspect-oriented and query-focused systems.
5 Experiments
- The experiments compare ASPECTNEWS performance with baselines and use SPACE as a point of comparison with a prior multi-document dataset.The authors report AOSUMM as competitive with other systems on SPACE.Source: §5 · p. 6
5.1 Metrics
- ASPECTNEWS evaluation uses F1 and ROUGE against annotations, while SPACE uses ROUGE because its gold summaries are abstractive.Annotator disagreement makes 100 F1 unattainable, and ROUGE complements F1 when predicted sentences resemble but do not match annotations exactly.Source: §5.1 · p. 6
5.2 Baselines & Competitor Models
- Baselines include keyword matching, an extractive oracle, QA-based sentence selection, generic BERTSUM, BERT-FK, and query-conditioned CTRLSUM.AOSUMM is based on BERTSUM and trained using the paper’s synthetic-data techniques.Source: §5.2 · p. 6
5.3 Results
- AOSUMM outperforms ASPECTNEWS baselines across F1, ROUGE-1, ROUGE-2, and ROUGE-L, generally surpassing keyword matching without ASPECTNEWS training.Table 6 compares AOSUMM and baselines across earthquake and fraud aspects; its final row gives the maximum possible F1 given annotator disagreement.Source: §5.3 · p. 6, Table 6 · p. 7
- On SPACE, aspect-oriented models exceed generic BERTSUM, while AOSUMM performs competitively with QT and CTRLSUM.AOSUMM is extractive and trained only on out-of-domain synthetic data, without access to aspects before test-time keywords.Source: §5.3 · p. 7, Table 7 · p. 7
5.4 Ablations and Analysis
- AOSUMM outputs vary with keyword choice, and summaries from fraud keyword sets are more similar than those from earthquake sets.Some sentences suit multiple aspects, while matched keyword sets outperform other sets within the domain.Source: §5.4 · p. 7, Table 9 · p. 7, Table 8 · p. 7
- For most datasets and oracle settings, keyword intensity r = 1 works well.Larger r values concatenate more keywords when generating the oracle extractive summary.Source: §5.4 · p. 7, Table 10 · p. 8
6 Qualitative Evaluation & Comparison
The qualitative comparison examines keyword responsiveness and output quality, noting that CTRLSUM can rephrase prompts or hallucinate, while AOSUMM extracts relevant document passages. It also identifies source copying in abstractive SPACE summaries and anaphora as a challenge for reading extracted sentences in isolation.
Extractive vs. Abstractive Comparison
- The comparison uses qualitative analysis because F1 cannot be directly compared for abstractive outputs, and ROUGE favors extractive systems against extractive ground truths.The analysis considers models’ relative keyword responsiveness and advantages and disadvantages.Source: §6 · p. 8
Keyword Sensitivity Comparison
- CTRLSUM often hallucinates or rephrases GEO prompts instead of synthesizing relevant document information into a cogent summary.Across 100 earthquake articles, “not known” appeared 27 times and “not immediately known” appeared 24 times under GEO keywords.Source: §6 · p. 8
Extractive vs. Abstractive Comparison
- For the SPACE dataset considered, abstractive models heavily copy source text, and their performance difference from AOSUMM is small.The authors suggest extractive models may be almost as well suited for this task.Source: §6 · p. 8
Extractive summaries
- AOSUMM extracts relevant passages, but anaphora can make some extracted sentences difficult to understand in isolation.The paper suggests anaphora resolution as a possible extension.Source: §6 · p. 8
7 Conclusion
ASPECTNEWS introduces aspect-oriented news summarization for domain-specific user intents and evaluates a keyword-controllable system trained on synthetic data without target-domain training. The system performs better than a range of strong baseline methods.
7 Conclusion
- ASPECTNEWS is a dataset for aspect-oriented summarization of news articles, designed to mimic common user intents in domain-specific settings.Unlike query-focused datasets, its task is not driven by document-specific facts or knowledge.Source: §7 · p. 8
- A keyword-controllable system trained on synthetic data performs well on ASPECTNEWS without training on the target domains.The conclusion describes its approach as performing better than a range of strong baseline methods.Source: §7 · pp. 8–9
- The system performs better than a range of strong baseline methods.Source: §7 · p. 9
A.1 Training Details
Training details specify the CNN/Daily Mail data split, BERTSUM-based modifications, learning-rate schedule, and AOSUMM fine-tuning runtime.
A.1 Training Details
- CNN/Daily Mail is split into 287,226 training pairs, 13,368 validation pairs, and 11,490 test pairs.Source: §A.1 · p. 12
- The models follow BERTSUM training with cased bert-base-cased and no lowercasing during dataset preparation.Source: §A.1 · p. 12
- The learning-rate schedule follows Vaswani et al. (2017), with warmup set to 10000.Source: §A.1 · p. 12
- Fine-tuning AOSUMM on the modified CNN/DM dataset takes 8 hours on a single NVIDIA Quadro RTX 8000.Source: §A.1 · p. 12
A.2 Exemplar Sentences
The appendix describes domain filtering and annotation checks, then compares oracle-label and mixed-training choices and details the SPACE evaluation setup.
A.2 Exemplar Sentences
- Domain articles are filtered from CNN/Daily Mail using Universal Sentence Encoder similarity to “An earthquake occurred” or “A fraud occured,” followed by manual exclusion of out-of-domain matches.Source: §A.2 · p. 12
A.3 Crowdsourcing
- Data collection quality checks include detailed annotator instructions, a user-friendly interface, and manual sampling and review.Source: §A.3 · p. 12
A.4 Oracle Derivation: BERTScore vs. ROUGE
- Models trained with BERTScore-derived oracle summaries are significantly stronger than models trained with ROUGE-maximizing sentence selections.The authors hypothesize that ROUGE maximization limits learning to lexical matching, whereas BERTScore can capture more abstract semantic criteria.Source: §A.4 · p. 12, Table 12 · p. 12
A.5 Mixed vs. Non-Mixed
- Mixed augmentation generally provides some benefit, inconsistently improving F1 on GEO, PEN, and NATURE but not RECV.The authors suggest mixed examples may help the model learn the relationship between keywords and aspect-oriented summaries; one model can produce both generic and aspect-oriented summaries.Source: §A.5 · pp. 12–13, Table 13 · p. 12
A.6 SPACE Evaluation Details
- For SPACE evaluation, multiple input documents are concatenated and processed in 512-token BERT chunks before sentence selection.The concatenated representations allow selection of any sentence from any part of the input.Source: §A.6 · p. 13