Source-linked AI summary
Neural Network-Based Abstract Generation for Opinions and Arguments
Lu Wang, Wang Ling
TL;DR
The paper addresses how to generate concise, fluent abstractive summaries from multiple units of opinionated text. It uses an attention-based neural model with importance-based sampling, outperforming extractive and abstractive systems on movie-review and argument datasets in automatic evaluation and receiving stronger human ratings for informativeness and grammaticality.
Problem
The paper studies how to generate concise, fluent abstractive summaries of opinionated text while integrating information from multiple reviews or arguments.
Method
An attention-based neural encoder-decoder integrates multiple opinion-bearing text units, using importance-based sampling to focus training on salient inputs.
Results
The system outperformed state-of-the-art extractive and abstractive methods on movie-review and argument datasets, with higher human informativeness and grammaticality ratings.
Takeaways & Limitations
Attention and importance-based sampling can produce abstractive opinion-consensus summaries that perform strongly in automatic and human evaluations.
Takeaways & Limitations
Some summaries are too short to cover sufficient information, while others contain incorrect domain-dependent facts such as names or numbers.
Abstract
from arXiv · showhide
We study the problem of generating abstractive summaries for opinionated text. We propose an attention-based neural network model that is able to absorb information from multiple text units to construct informative, concise, and fluent summaries. An importance-based sampling method is designed to allow the encoder to integrate information from an important subset of input. Automatic evaluation indicates that our system outperforms state-of-the-art abstractive and extractive summarization systems on two newly collected datasets of movie reviews and arguments. Our system summaries are also rated as more informative and grammatical in human evaluation.
1 Introduction
The paper presents an attention-based neural model that generates one-sentence abstractive opinion-consensus summaries from multiple text units. Importance-based sampling focuses training on salient inputs, and experiments on movie reviews and arguments show stronger automatic and human evaluation results than existing systems.
- 1 Introduction: Abstractive summarization addresses extractive methods’ tendency to include secondary or redundant information and can produce more coherent, concise text.Prior abstract-generation systems often relied on salient-phrase extraction and merging, limiting new-word generation and potentially producing ungrammatical output.
- 1 Introduction: The system generates one-sentence abstractive summaries describing the opinion consensus across multiple reviews or arguments.It is designed for opinionated text about a shared topic, including movies and controversial issues.
- 1 Introduction: The attention-based encoder integrates information from multiple input units to produce informative, concise, and fluent summaries.Importance-based sampling lets the model learn from an important subset while retaining diversified input.
- 1 Introduction: BLEU 24.88 on Rotten Tomatoes exceeded 19.72 from Ganesan et al.’s abstractive system, while the proposed system outperformed extractive and abstractive baselines on both tasks.ROUGE indicated reasonable information coverage, and human judges rated the summaries more informative and grammatical.
2 Data Collection
The paper introduces two newly collected opinion-summarization datasets with editor-written gold abstracts: movie-review consensuses and claims associated with debate arguments.
- 2 Data Collection: The RottenTomatoes dataset contains 246,164 critic reviews and editor-written one-sentence consensuses for 3,731 movies.The splits include 2,458 training movies, 536 validation movies, and 737 test movies.
- 2 Data Collection: Both datasets provide gold-standard abstracts for evaluating summaries of opinionated text.RottenTomatoes uses critic consensuses, while Idebate uses editor-written central claims.
- 2 Data Collection: The Idebate dataset contains 676 debates, 2,259 claims, and 17,359 arguments organized around pro and con points.Each editor-constructed central claim serves as the gold-standard summary for its corresponding arguments.
3 The Neural Network-Based Abstract Generation Model
The model generates abstracts from multiple reviews or arguments using an attention-based encoder and LSTM decoder. Importance-based sampling selects a smaller, more relevant subset of input units to reduce order sensitivity and computational cost.
- 3.3 Encoder: The system represents multiple text units with an attention-based encoder that searches for relevant information to inform abstract generation.Unlike prior single-input encoder-decoder models, the architecture is designed for an arbitrary number of reviews or arguments.
- 3.2 Decoder: An LSTM decoder generates each summary word from the previous word, recurrent state, and attention-derived input representation.The recurrent update function uses the previous output representation, previous state, and input text representation.
- 3.3 Encoder: The encoder computes context-dependent word representations with a bidirectional LSTM and attention coefficients reflecting each word’s relevance to the next summary word.Attention produces a weighted sum of input representations, while the affinity function compares each word with the current decoder context.
- 3.4 Attention Over Multiple Inputs: Importance-based sampling addresses concatenation’s order sensitivity and O(N|z|) attention cost by selecting likely summary-worthy input units.The method assigns each text unit an importance score and samples K candidates from the normalized scores during training.
- 3.5 Importance Estimation: The importance estimator combines ridge regression with pairwise preference regularization to favor text units likely to contribute to the reference summary.Training labels are normalized overlap counts between each text unit and the gold-standard summary.
- 3.5 Importance Estimation: At test time, the system reranks n-best summaries by cosine similarity with the input text units and selects the highest-scoring candidate.The paper identifies more sophisticated reranking methods as future work.
4 Experimental Setup
The experiments specify preprocessing, feature representations, optimization, evaluation, and decoding settings for abstract generation.
- Preprocessing uses Stanford CoreNLP for tokenization, part-of-speech tagging, and dependency-relation extraction.
- Word representations have 300 dimensions, with optional Google News initialization and additional token-level features.
- Encoder and decoder LSTM states and cells use 150 dimensions, while attention projections use 100 dimensions.
- BLEU evaluates generated summaries by measuring up-to-4-gram precision against gold-standard abstracts.
- Training uses Adagrad and terminates when development-set performance no longer improves.
- Beam search retains 20 most probable output sequences at each decoding step and stops after all beams generate end-of-sentence tokens.
5 Results
The model's importance estimator outperforms baselines, while the full abstractive system achieves strong automatic and human-evaluated summary quality across movie reviews and arguments. Importance-based sampling also outperforms uniform sampling, and qualitative analysis identifies benefits and remaining failure cases.
- 5.1 Importance Estimation Evaluation: Our importance estimation model uniformly outperforms baseline systems on both datasets according to MRR, NDCG@3, and NDCG@5.Baselines include SVR, length-based ranking, and centroid-based ranking.
- 5.2 Automatic Summary Evaluation: Our system with pre-trained word embeddings and additional features achieves the best BLEU scores on both datasets, with statistical significance.It also achieves the best METEOR and ROUGE scores on RottenTomatoes among learning-based systems.
- 5.3 Human Evaluation on Summary Quality: Human judges rate our summaries higher than OPINOSIS on all evaluated aspects and higher than LEXRANK in informativeness and grammaticality.Our summaries are ranked best in 18% of evaluations, with an average ranking of 2.3.
- 5.4 Sampling Effect: Importance-based sampling produces BLEU scores comparable to Top K sampling, while both outperform Uniform Sampling; it uniformly leads on METEOR.The same pattern is observed on the Idebate dataset for METEOR.
- 5.5 Further Discussion: Re-ranking n-best summaries improves performance over simply selecting the top-1 generation, indicating that post-processing addresses task-specific issues such as informativeness.The authors suggest studying more sophisticated re-ranking algorithms.
- 5.5 Further Discussion: Lower-informativeness cases often involve summaries shorter than human abstracts or generations containing incorrect domain-dependent facts such as named entities and numbers.These observations identify information coverage and factual accuracy as remaining difficulties.
6 Related Work
Opinion summarization has largely used extraction or phrase assembly, while this work extends abstractive generation to opinion consensus and argument claims.
- Earlier opinion summarization mainly identified summary-worthy sentences or extracted frequent features and opinion phrases.
- The paper is the first, to its knowledge, to study claim generation for arguments.
- Recent abstractive systems commonly concatenate extracted phrases, while manual templates were introduced to improve grammaticality.
- The paper relates its task to attention-based neural machine translation and sequence-to-sequence models for compression and summarization.
7 Conclusion
The paper presents a neural approach for abstractive opinion summarization that attends to salient information across inputs and samples important text units during training. Experiments report state-of-the-art results under automatic and human evaluation.
- The system obtained state-of-the-art results using both automatic evaluation and human evaluation.
- The attention-based method finds salient information from different input text units to generate informative and concise summaries.
- Importance-based sampling addresses the large number of input texts during model training.