PaperlayerView exact revision on arXiv

Source-linked AI summary

Diversity driven Attention Model for Query-based Abstractive Summarization

Preksha Nema, Mitesh Khapra, Anirban Laha, Balaraman Ravindran

arXiv:1704.08300v2cs.CL

TL;DR

Query-based abstractive summarization must produce concise summaries focused on a query, but encode-attend-decode models often repeat phrases. The paper adds dynamic query attention and diversity-based attention, evaluates them on a new Debatepedia dataset, and reports a 28% (absolute) ROUGE-L gain over vanilla models.

  • Problem

    Query-based abstractive summarization requires summaries focused on query-relevant document information, while encode-attend-decode models often generate repeated phrases.

    Source: §1 · pp. 1–2

  • Method

    The paper adds dynamic query attention and a diversity-based attention mechanism to encode-attend-decode summarization, and introduces a Debatepedia-based dataset.

    Source: §1 · p. 2, §3 · p. 3

  • Results

    28% (absolute) in ROUGE-L score is reported over a plain encode-attend-decode mechanism.

    Source: §8 · p. 9

  • Takeaways & Limitations

    The proposed model produces better query-based abstractive summaries with fewer repeated phrases and outperforms a state-of-the-art diversity model by 7% (absolute) in ROUGE-L score.

    Source: §1 · p. 2, §8 · p. 9

    The plain encode-attend-decode instantiation suffers from repeated words or phrases in its outputs.

    Source: §4 · p. 5

Abstract

from arXiv · show

Abstractive summarization aims to generate a shorter version of the document covering all the salient points in a compact and coherent fashion. On the other hand, query-based summarization highlights those points that are relevant in the context of a given query. The encode-attend-decode paradigm has achieved notable success in machine translation, extractive summarization, dialog systems, etc. But it suffers from the drawback of generation of repeated phrases. In this work we propose a model for the query-based summarization task based on the encode-attend-decode paradigm with two key additions (i) a query attention model (in addition to document attention model) which learns to focus on different portions of the query at different time steps (instead of using a static representation for the query) and (ii) a new diversity based attention model which aims to alleviate the problem of repeating phrases in the summary. In order to enable the testing of this model we introduce a new query-based summarization dataset building on debatepedia. Our experiments show that with these two additions the proposed model clearly outperforms vanilla encode-attend-decode models with a gain of 28% (absolute) in ROUGE-L scores.

1 Introduction

The paper targets query-based abstractive summarization and addresses repeated phrases in encode-attend-decode outputs with a new dataset and attention mechanisms. Its model dynamically attends to the query and diversifies successive context vectors, outperforming baseline models.

  • Query-based abstractive summarization generates document summaries focused on information relevant to a given query.Source: §1 · p. 1
  • The authors create a Debatepedia-based dataset of query, document, and abstractive-summary triplets.Source: §1 · p. 1
  • Encode-attend-decode models can generate repeated words or phrases because successive decoder context vectors may be very similar.Source: §1 · pp. 1–2
  • The proposed diversity mechanism makes successive context vectors orthogonal, while permitting attention to previously used words when needed later.Source: §1 · p. 2
  • The model outperforms a vanilla encoder-decoder model by 28% (absolute) in ROUGE-L score and produces outputs with fewer repetitions.Source: §1 · p. 2

2 Related Work

Prior work largely focused on extractive or general abstractive summarization, while repeated phrases remained a recurring encoder-decoder problem. This paper evaluates query-based abstractive summarization and reduces such repetitions with a new method.

  • Most earlier summarization research constructed summaries by selecting salient sentences from documents.Source: §2 · p. 2
  • Neural encode-attend-decode models achieved strong results in abstractive summarization across datasets including GigaWord and DUC.Source: §2 · p. 3
  • Encoder-decoder models often repeat words or phrases, prompting temporal-attention and distraction-based approaches that use attention history.Source: §2 · p. 3
  • The authors evaluate query-based abstractive summarization, create a dataset, and report fewer repeated phrases in their outputs.Source: §2 · p. 3

3 Dataset

The authors build a query-based abstractive summarization dataset from Debatepedia, an encyclopedia of debate arguments and quotes. It contains thousands of query-document-summary examples across diverse debate topics.

  • The dataset is derived from Debatepedia and contains 663 debates across 53 overlapping categories.Source: §3 · p. 3
  • Each dataset example pairs a query and document with an abstractive summary that is not necessarily copied from the document.Source: §3 · p. 3
  • The corpus contains 12695 query-document-summary triples crawled from Debatepedia.Source: §3 · p. 3
  • Experiments use 10-fold cross-validation, with each fold assigning 80% of documents to training and 10% each to validation and testing.Source: §3 · p. 3

4 Proposed model

The proposed query-based abstractive summarization model encodes queries and documents, attends dynamically to both, and uses diversity-based document attention to reduce repeated output phrases.

  • The model uses separate GRU encoders for the query and document, followed by query attention, document attention, and a decoder.Source: §4 · p. 4
  • At each decoding step, query attention assigns weights to query words so the decoder can focus on different query portions over time.Source: §4 · p. 4
  • Document attention produces a context vector containing relevant document and query information for the decoder at each timestep.Source: §4 · pp. 4–5
  • The baseline encode-attend-decode model can repeat phrases because successive decoder context vectors may be very similar.Source: §4 · p. 5, §4.1 · p. 5

Diversity based attention model

  • D1 and SD1 diversify each context vector relative to its predecessor, using either hard orthogonality or a learned gating parameter.Source: §4.1 · p. 5
  • D2 and SD2 incorporate prior context history through a modified LSTM, with SD2 relaxing the orthogonality constraint using a gate.Source: §4.1 · pp. 5–6

5 Baseline Methods

The paper compares two adapted diversity baselines that modify context vectors, attention weights, or both to reduce repetition in query-based abstractive summarization.

  • Both baselines adapt diversity methods originally proposed for abstractive summarization to the query-based task.Source: §5 · p. 6
  • M1 accumulates previous context vectors and subtracts this history when computing the current diverse context.Source: §5 · p. 6
  • M2 additionally forces each time step’s attention weights to differ from those at the previous time step.Source: §5 · p. 6

6 Experimental Setup

The experiments evaluate query and diversity components through multiple encoder-attention-decoder variants on a Debatepedia-derived dataset using averaged ROUGE scores across ten folds.

Compared models

  • The study compares vanilla, query-encoding, query-attention, diversity, soft-diversity, LSTM-history, and baseline diversity variants.Source: §6 · pp. 6–7
  • B1 replaces the diversity cell with a basic LSTM to test whether tracking context-vector history alone is sufficient.Source: §6 · p. 7
  • M1 operates on context vectors, whereas M2 operates on attention weights in addition to context vectors.Source: §6 · p. 7

Evaluation

  • The evaluation uses 80% training, 10% validation, and 10% testing data, with ten folds and average Rouge-1, Rouge-2, and Rouge-L scores.Source: §6 · p. 7
  • Table 3 reports performance across models using full-length ROUGE metrics.Source: §6 · p. 7, Table 3 · p. 7

7 Discussions

The experiments find benefits from query encoding, dynamic query attention, and diversity mechanisms, while qualitative and quantitative analyses examine repetition in generated summaries.

Discussion

  • Adding a query encoder improves performance because query keywords can sharpen the summary’s focus.Source: §7 · p. 7
  • Dynamic query attention further improves results by focusing on relevant query portions at different time steps.Source: §7 · p. 7
  • All introduced diversity models significantly outperform nondiversity models, with the modified LSTM-based diversity model performing best.Source: §7 · p. 7

Repetition analysis

  • Qualitative examples show that nond​​iversity baselines produce more repetitions, while the proposed model retains valid repetitions when needed.Source: §7 · p. 7, Table 4 · p. 8
  • The quantitative analysis counts test-set sentences containing repeated words across different models.Source: §7 · p. 8

8 Conclusion

The paper concludes that successive orthogonalization diversifies context vectors while permitting necessary repeated attention, and reports substantial ROUGE-L gains over plain and state-of-the-art diversity baselines.

Conclusion

  • The paper also introduces a new dataset for query-based summarization.Source: §8 · pp. 8–9
  • The model’s novel diversification mechanism uses successive orthogonalization to provide diverse successive context vectors.Source: §8 · p. 9
  • The mechanism can attend to words repeatedly later in a summary instead of aggressively deleting attention history.Source: §8 · p. 9
  • 28% (absolute) in ROUGE-L score is the reported gain over a plain encode-attend-decode mechanism.Source: §8 · p. 9
  • 7% (absolute) in ROUGE-L score is the reported gain over a state-of-the-art diversity model.Source: §8 · p. 9

Related papers

Loading 1704.08300v2…