Source-linked AI summary
Diversity driven Attention Model for Query-based Abstractive Summarization
Preksha Nema, Mitesh Khapra, Anirban Laha, Balaraman Ravindran
TL;DR
Query-based abstractive summarization must produce concise summaries focused on a query, but encode-attend-decode models often repeat phrases. The paper adds dynamic query attention and diversity-based attention, evaluates them on a new Debatepedia dataset, and reports a 28% (absolute) ROUGE-L gain over vanilla models.
Problem
Query-based abstractive summarization requires summaries focused on query-relevant document information, while encode-attend-decode models often generate repeated phrases.
Method
The paper adds dynamic query attention and a diversity-based attention mechanism to encode-attend-decode summarization, and introduces a Debatepedia-based dataset.
Results
28% (absolute) in ROUGE-L score is reported over a plain encode-attend-decode mechanism.
Takeaways & Limitations
The proposed model produces better query-based abstractive summaries with fewer repeated phrases and outperforms a state-of-the-art diversity model by 7% (absolute) in ROUGE-L score.
Takeaways & Limitations
The plain encode-attend-decode instantiation suffers from repeated words or phrases in its outputs.
Abstract
from arXiv · showhide
Abstractive summarization aims to generate a shorter version of the document covering all the salient points in a compact and coherent fashion. On the other hand, query-based summarization highlights those points that are relevant in the context of a given query. The encode-attend-decode paradigm has achieved notable success in machine translation, extractive summarization, dialog systems, etc. But it suffers from the drawback of generation of repeated phrases. In this work we propose a model for the query-based summarization task based on the encode-attend-decode paradigm with two key additions (i) a query attention model (in addition to document attention model) which learns to focus on different portions of the query at different time steps (instead of using a static representation for the query) and (ii) a new diversity based attention model which aims to alleviate the problem of repeating phrases in the summary. In order to enable the testing of this model we introduce a new query-based summarization dataset building on debatepedia. Our experiments show that with these two additions the proposed model clearly outperforms vanilla encode-attend-decode models with a gain of 28% (absolute) in ROUGE-L scores.
1 Introduction
The paper targets query-based abstractive summarization and addresses repeated phrases in encode-attend-decode outputs with a new dataset and attention mechanisms. Its model dynamically attends to the query and diversifies successive context vectors, outperforming baseline models.
- Query-based abstractive summarization generates document summaries focused on information relevant to a given query.
- The authors create a Debatepedia-based dataset of query, document, and abstractive-summary triplets.
- Encode-attend-decode models can generate repeated words or phrases because successive decoder context vectors may be very similar.
- The proposed diversity mechanism makes successive context vectors orthogonal, while permitting attention to previously used words when needed later.
- The model outperforms a vanilla encoder-decoder model by 28% (absolute) in ROUGE-L score and produces outputs with fewer repetitions.
2 Related Work
Prior work largely focused on extractive or general abstractive summarization, while repeated phrases remained a recurring encoder-decoder problem. This paper evaluates query-based abstractive summarization and reduces such repetitions with a new method.
- Most earlier summarization research constructed summaries by selecting salient sentences from documents.
- Neural encode-attend-decode models achieved strong results in abstractive summarization across datasets including GigaWord and DUC.
- Encoder-decoder models often repeat words or phrases, prompting temporal-attention and distraction-based approaches that use attention history.
- The authors evaluate query-based abstractive summarization, create a dataset, and report fewer repeated phrases in their outputs.
3 Dataset
The authors build a query-based abstractive summarization dataset from Debatepedia, an encyclopedia of debate arguments and quotes. It contains thousands of query-document-summary examples across diverse debate topics.
- The dataset is derived from Debatepedia and contains 663 debates across 53 overlapping categories.
- Each dataset example pairs a query and document with an abstractive summary that is not necessarily copied from the document.
- The corpus contains 12695 query-document-summary triples crawled from Debatepedia.
- Experiments use 10-fold cross-validation, with each fold assigning 80% of documents to training and 10% each to validation and testing.
4 Proposed model
The proposed query-based abstractive summarization model encodes queries and documents, attends dynamically to both, and uses diversity-based document attention to reduce repeated output phrases.
- The model uses separate GRU encoders for the query and document, followed by query attention, document attention, and a decoder.
- At each decoding step, query attention assigns weights to query words so the decoder can focus on different query portions over time.
- Document attention produces a context vector containing relevant document and query information for the decoder at each timestep.
- The baseline encode-attend-decode model can repeat phrases because successive decoder context vectors may be very similar.
- Diversity based attention model: D1 and SD1 diversify each context vector relative to its predecessor, using either hard orthogonality or a learned gating parameter.
- Diversity based attention model: D2 and SD2 incorporate prior context history through a modified LSTM, with SD2 relaxing the orthogonality constraint using a gate.
5 Baseline Methods
The paper compares two adapted diversity baselines that modify context vectors, attention weights, or both to reduce repetition in query-based abstractive summarization.
- Both baselines adapt diversity methods originally proposed for abstractive summarization to the query-based task.
- M1 accumulates previous context vectors and subtracts this history when computing the current diverse context.
- M2 additionally forces each time step’s attention weights to differ from those at the previous time step.
6 Experimental Setup
The experiments evaluate query and diversity components through multiple encoder-attention-decoder variants on a Debatepedia-derived dataset using averaged ROUGE scores across ten folds.
- Compared models: The study compares vanilla, query-encoding, query-attention, diversity, soft-diversity, LSTM-history, and baseline diversity variants.
- Compared models: B1 replaces the diversity cell with a basic LSTM to test whether tracking context-vector history alone is sufficient.
- Compared models: M1 operates on context vectors, whereas M2 operates on attention weights in addition to context vectors.
- Evaluation: The evaluation uses 80% training, 10% validation, and 10% testing data, with ten folds and average Rouge-1, Rouge-2, and Rouge-L scores.
- Evaluation: Table 3 reports performance across models using full-length ROUGE metrics.
7 Discussions
The experiments find benefits from query encoding, dynamic query attention, and diversity mechanisms, while qualitative and quantitative analyses examine repetition in generated summaries.
- Discussion: Adding a query encoder improves performance because query keywords can sharpen the summary’s focus.
- Discussion: Dynamic query attention further improves results by focusing on relevant query portions at different time steps.
- Discussion: All introduced diversity models significantly outperform nondiversity models, with the modified LSTM-based diversity model performing best.
- Repetition analysis: Qualitative examples show that nondiversity baselines produce more repetitions, while the proposed model retains valid repetitions when needed.
- Repetition analysis: The quantitative analysis counts test-set sentences containing repeated words across different models.
8 Conclusion
The paper concludes that successive orthogonalization diversifies context vectors while permitting necessary repeated attention, and reports substantial ROUGE-L gains over plain and state-of-the-art diversity baselines.
- Conclusion: The paper also introduces a new dataset for query-based summarization.
- Conclusion: The model’s novel diversification mechanism uses successive orthogonalization to provide diverse successive context vectors.
- Conclusion: The mechanism can attend to words repeatedly later in a summary instead of aggressively deleting attention history.
- Conclusion: 28% (absolute) in ROUGE-L score is the reported gain over a plain encode-attend-decode mechanism.
- Conclusion: 7% (absolute) in ROUGE-L score is the reported gain over a state-of-the-art diversity model.