PaperlayerView exact revision on arXiv

Source-linked AI summary

A Hierarchical Structured Self-Attentive Model for Extractive Document Summarization (HSSAS)

Kamal Al-Sabahi, Zhang Zuping, Mohammed Nadher

arXiv:1805.07799v1cs.CL

TL;DR

Neural summarization methods may miss document topic structure and struggle to retain information in recurrent states. HSSAS applies hierarchical structured self-attention and classifies sentence-summary membership. The paper reports that it outperforms current extractive state-of-the-art models on CNN/Daily Mail and DUC 2002.

  • Problem

    Neural summarization models may lack latent topic structure, while recurrent models can struggle to retain information in fixed-size states.

    Source: §I · p. 2

  • Method

    HSSAS mirrors document structure with word- and sentence-level self-attention and classifies sentences by summary membership.

    Source: §I · p. 2, §II · p. 3, E. CLASSIFICATION LAYER · p. 4

  • Results

    HSSAS outperforms current extractive state-of-the-art models on CNN/Daily Mail and DUC 2002.

    Source: §I · p. 2, E. EXPERIMENTAL RESULTS · p. 5

  • Takeaways & Limitations

    The attention mechanism provides insight into which words and sentences contribute to document representation and summary-sentence selection.

    Source: §I · p. 2

Abstract

from arXiv · show

The recent advance in neural network architecture and training algorithms have shown the effectiveness of representation learning. The neural network-based models generate better representation than the traditional ones. They have the ability to automatically learn the distributed representation for sentences and documents. To this end, we proposed a novel model that addresses several issues that are not adequately modeled by the previously proposed models, such as the memory problem and incorporating the knowledge of document structure. Our model uses a hierarchical structured self-attention mechanism to create the sentence and document embeddings. This architecture mirrors the hierarchical structure of the document and in turn enables us to obtain better feature representation. The attention mechanism provides extra source of information to guide the summary extraction. The new model treated the summarization task as a classification problem in which the model computes the respective probabilities of sentence-summary membership. The model predictions are broken up by several features such as information content, salience, novelty and positional representation. The proposed model was evaluated on two well-known datasets, the CNN / Daily Mail, and DUC 2002. The experimental results show that our model outperforms the current extractive state-of-the-art by a considerable margin.

I. INTRODUCTION

The paper addresses limitations in document representation and memory in neural summarization by proposing hierarchical structured self-attention for extractive sentence selection. Its architecture follows document structure, uses attention at word and sentence levels, and treats summary membership as classification.

I. INTRODUCTION

  • Neural summarization can lack latent topic structure, and recurrent models may struggle to retain information in fixed-size states.The paper motivates attention as a way to access earlier states without requiring the latest state to store all information.Source: §I · p. 2
  • HSSAS extracts sentences by classifying their summary membership while modeling content richness, salience, redundancy, and position.The model computes a score for each sentence and uses word- and sentence-level self-attention to weight content differentially.Source: §I · p. 2, Figure 1 · p. 3
  • The paper evaluates HSSAS on CNN/Daily Mail and DUC 2002 and reports that it outperforms current extractive state-of-the-art models.The introduction attributes both better performance and insight into contributing words and sentences to its attention mechanism.Source: §I · p. 2

II. THE PROPOSED MODEL

  • The proposed framework is based on recurrent neural networks but is applicable to sentence-extraction models that use distributed representations as input.The paper distinguishes its approach from prior work through hierarchical structured self-attention for sentence and document embeddings.Source: §II · p. 2, §I · p. 2
  • The model’s two attention levels mirror document structure: words form sentences, and sentences form a document.The architecture applies attention after the word-level and sentence-level layers.Source: §II · pp. 2–3, Figure 1 · p. 3

A. WORD ENCODER

The word encoder uses a bidirectional LSTM to represent each sentence from both reading directions. It concatenates the directional states into contextual hidden states that self-attention can weight.

A. WORD ENCODER

  • A bidirectional LSTM reads each sentence in forward and backward directions to encode its words.The forward pass reads from the first word to the last, while the backward pass reads in reverse.Source: A. WORD ENCODER · p. 3
  • Concatenating the forward and backward states produces a hidden state summarizing the sentence around each word.Each unidirectional LSTM has u hidden units; the combined states have dimension 2u.Source: A. WORD ENCODER · p. 3
  • Self-attention takes the whole set of LSTM hidden states as input and produces attention weights.The weights are used to form a sentence vector as a weighted sum of the hidden states.Source: A. WORD ENCODER · p. 3, Figure 2 · p. 3

B. WORD ATTENTION

Word attention converts a variable-length sentence into a fixed-size embedding by weighting its LSTM hidden states. The weights reflect each word’s contribution to the sentence representation.

B. WORD ATTENTION

  • Self-attention takes the sentence’s LSTM hidden states as input and produces a vector of attention weights.Source: B. WORD ATTENTION · p. 3
  • The softmax normalizes the attention weights to sum to 1; its parameters are learnable, and k is a hyperparameter.Source: B. WORD ATTENTION · p. 3
  • The sentence vector is the weighted sum of the LSTM hidden states, using the attention weights.Source: B. WORD ATTENTION · p. 3

C. SENTENCE ENCODER

A bidirectional LSTM encodes sentences in document context, and sentence attention weights their hidden states to form a document representation.

C. SENTENCE ENCODER

  • A bidirectional LSTM encodes the sentence vectors, and concatenated forward and backward states summarize adjacent sentences while focusing on sentence i.Source: C. SENTENCE ENCODER · pp. 3–4
  • The document’s sentence states have dimension N × 2u, where N is the number of sentences and u is the hidden-unit count per unidirectional LSTM.Source: C. SENTENCE ENCODER · p. 4

D. SENTENCE ATTENTION

  • Sentence attention takes all document-level LSTM hidden states as input and yields weights that form the document vector as a weighted sum.Source: D. SENTENCE ATTENTION · p. 4

E. CLASSIFICATION LAYER

The classification layer estimates each sentence’s summary-membership probability from content, document salience, novelty, and position. Its features are combined in a sigmoid scoring function, and training minimizes negative log-likelihood.

E. CLASSIFICATION LAYER

  • Content richness represents each sentence’s information content, while salience measures its relevance to the document.Source: E. CLASSIFICATION LAYER · p. 4
  • Novelty measures a sentence against the current summary, whose representation at sentence j is denoted o_j.Source: E. CLASSIFICATION LAYER · p. 4
  • The positional feature models each sentence’s position in the document using a positional embedding formed from forward and backward position indices.Source: E. CLASSIFICATION LAYER · p. 4
  • Automatically learned scalar weights model the relative importance of the abstract features, and the model minimizes the observed labels’ negative log-likelihood during training.Source: E. CLASSIFICATION LAYER · p. 4
  • The sentence label is binary, indicating whether sentence j is included in the summary.Source: E. CLASSIFICATION LAYER · p. 4
  • Sentence-summary membership is estimated with a sigmoid over content richness, document salience, novelty relative to the accumulated summary, position, and a bias.Source: E. CLASSIFICATION LAYER · p. 4

A. DATASETS

The evaluation uses CNN/Daily Mail and DUC 2002, with DUC 2002 serving as an out-of-domain test set for single-document summarization.

A. DATASETS

  • The joint CNN/Daily Mail dataset has 286,722 training, 13,362 validation, and 11,480 test documents, averaging 28 sentences per document.Source: A. DATASETS · p. 4
  • The Daily Mail dataset split contains 196,557 training, 12,147 validation, and 10,396 test documents.Source: A. DATASETS · p. 4
  • DUC 2002 is an out-of-domain test set of 567 news articles in 59 topic clusters; this study uses its single-document summarization task.Source: A. DATASETS · p. 5

B. BASELINES

The paper compares HSSAS with extractive and abstractive baselines on CNN/Daily Mail and DUC 2002, then reports its training settings and ROUGE evaluation procedure.

B. BASELINES

  • Baselines include Lead-3, SummaRuNNer, Cheng and Lapata’s extractive model, and dataset-specific abstractive or graph-based systems.Paulus et al. [13] and See et al. [15] are abstractive baselines for CNN/Daily Mail; TGRAPH and URANK are used for DUC 2002.Source: B. BASELINES · p. 5

C. SETTINGS

  • The model uses 100-dimensional word embeddings, 200-dimensional hidden states, and limits inputs to 50 words per sentence and 100 sentences per document.The vocabulary is capped at 150k; training uses batches of 64, Adadelta, and gradient clipping, while test-time sentences are selected by membership probability until the compression rate is exceeded.Source: C. SETTINGS · p. 5

D. EVALUATION

  • The automatic evaluation of generated summaries uses ROUGE metrics, which compare n-grams from the evaluated summary with n-grams from one or several human-written reference summaries.For DUC 2002, longer summaries are truncated with ROUGE’s “-l 75” option for recall-only evaluation. For CNN/DailyMail, full-length F1 is used to ensure a fair comparison with the baselines.Source: D. EVALUATION · p. 5

E. EXPERIMENTAL RESULTS

HSSAS performs best on all reported ROUGE metrics on DUC 2002 and outperforms nearly all baselines on CNN/Daily Mail. The authors also note that ROUGE scores do not necessarily reflect readability.

E. EXPERIMENTAL RESULTS

  • HSSAS performs best on ROUGE-1, ROUGE-2, and ROUGE-L on DUC 2002.The comparison is shown in Figure 3 and Table 1.Source: E. EXPERIMENTAL RESULTS · p. 5, Figure 3 · p. 6
  • Although Lead-3 is hard to beat on DUC 2002 because news articles often place important information first, HSSAS performs better.The authors attribute Lead-3’s strong ROUGE results to this article structure.Source: E. EXPERIMENTAL RESULTS · p. 6
  • High ROUGE scores for abstractive summaries do not necessarily indicate greater readability.The authors caution that optimizing a discrete metric such as ROUGE does not guarantee improved summary quality or readability [13],.Source: E. EXPERIMENTAL RESULTS · p. 6
  • ROUGE reliability increases with the number of reference summaries, and scores are lower on datasets with one reference per document than on multi-reference datasets.The authors describe this as an inflexibility of ROUGE [15],.Source: E. EXPERIMENTAL RESULTS · p. 6
  • HSSAS outperforms all baselines on almost all ROUGE metrics on CNN/Daily Mail.The comparison uses the full-length F1 variant of ROUGE, as shown in Figure 4 and Table 2.Source: E. EXPERIMENTAL RESULTS · p. 5, Figure 4 · p. 6, Table 2

IV. RELATED WORK

Prior work spans extractive systems that select source sentences and abstractive systems that generate new wording. Neural approaches include recurrent classifiers and attention-based models; this work distinguishes itself through hierarchical structured self-attention.

IV. RELATED WORK

  • Extractive models select key source content, while abstractive models create representations using different wording [13],.Extractive approaches remain attractive for their lower cost and complexity, while long-input abstractive models can struggle with repetition, unreadability, and incoherence.Source: §IV · pp. 6–7
  • Yousefi-Azar and Hamey build sentence inputs from local word representations of term-frequency features and add random noise to the autoencoder representation.Their approach targets query-oriented single-document extractive summarization.Source: §IV · pp. 6–7
  • Cao et al. use query-attention-weighted CNNs for query-focused multi-document summaries, while Cheng and Lapata apply attention to extract sentences and words.See et al. [15] combine a pointer-generator with coverage in an abstractive model.Source: §IV · pp. 6–7
  • Nallapati et al. classify sentences sequentially as binary summary-membership decisions, without attention; this model uses structured self-attention to guide sentence and document representations.Other cited extractive work includes a recursive autoencoder, CNN-based methods,, and attention-based sequence labeling.Source: §IV · pp. 6–7
  • Rush et al. [28] use a convolutional encoder and feedforward language-model decoder for titles, while Paulus et al. [13] combine supervised word prediction with reinforcement learning.Rush et al. use the first sentence of each article and a bag-of-n-grams representation.Source: §IV · p. 7

V. CONCLUSION AND FUTURE WORK

The conclusion presents hierarchical attention and structured self-attention as routes to sentence and document embeddings, with abstract features learned during training. The authors report improved summarization results and identify reinforcement learning with sequence-to-sequence training, and evaluation beyond ROUGE, as future directions.

V. CONCLUSION AND FUTURE WORK

  • The model uses hierarchical attention that mirrors document structure, structured self-attention for embeddings, and automatically learned abstract features weighted during training.The features take previously classified sentences into consideration.Source: §V · p. 7
  • The authors report that the resulting embeddings provide better representations and that the model outperforms state-of-the-art models on the same datasets.The passage does not provide numerical comparisons.Source: §V · p. 7
  • The authors identify combining reinforcement learning with sequence-to-sequence training and developing evaluation metrics beyond ROUGE for long sequences as future research directions.Source: §V · p. 7

Related papers

Loading 1805.07799v1…