PaperlayerView exact revision on arXiv

Source-linked AI summary

HIBERT: Document Level Pre-training of Hierarchical Bidirectional Transformers for Document Summarization

Xingxing Zhang, Furu Wei, Ming Zhou

arXiv:1905.06566v1cs.CLcs.LG

TL;DR

Neural extractive summarization relies on heuristically created sentence labels, making hierarchical encoder training challenging. HIBERT pre-trains hierarchical bidirectional Transformer encoders on unlabeled data, improving summarization performance over randomly initialized counterparts and achieving state-of-the-art results on CNN/Dailymail and New York Times datasets.

  • Problem

    Heuristically generated sentence-level labels for neural extractive summarization are often inaccurate, making hierarchical encoder training challenging.

    Source: §1 · pp. 1–2

  • Method

    HIBERT pre-trains hierarchical bidirectional Transformer document encoders on unlabeled data before applying them to extractive summarization.

    Source: §1 · p. 2, §3.1 · p. 4, §3.2 · p. 4

  • Results

    HIBERT improves summarization performance over randomly initialized counterparts and achieves state-of-the-art performance on CNN/Dailymail and New York Times datasets.

    Source: §1 · p. 2, §5 · p. 8

  • Takeaways & Limitations

    Adding a large open-domain dataset to pre-training leads to better performance, while the two-stage open-domain and in-domain process is necessary on CNN/Dailymail.

    Source: §5 · p. 8, §4.4 · p. 8

    Deeper and larger models produced inferior results, suggesting that training large or deep models on this dataset without good initialization is challenging.

    Source: §4.4 · p. 7

Abstract

from arXiv · show

Neural extractive summarization models usually employ a hierarchical encoder for document encoding and they are trained using sentence-level labels, which are created heuristically using rule-based methods. Training the hierarchical encoder with these \emph{inaccurate} labels is challenging. Inspired by the recent work on pre-training transformer sentence encoders \cite{devlin:2018:arxiv}, we propose {\sc Hibert} (as shorthand for {\bf HI}erachical {\bf B}idirectional {\bf E}ncoder {\bf R}epresentations from {\bf T}ransformers) for document encoding and a method to pre-train it using unlabeled data. We apply the pre-trained {\sc Hibert} to our summarization model and it outperforms its randomly initialized counterpart by 1.25 ROUGE on the CNN/Dailymail dataset and by 2.0 ROUGE on a version of New York Times dataset. We also achieve the state-of-the-art performance on these two datasets.

1 Introduction

Extractive summarization selects important document sentences but depends on sentence-level labels that are often heuristically generated and inaccurate. HIBERT addresses this training challenge by pre-training its hierarchical encoder on unlabeled data before sentence classification.

  • Extractive summarization ranks sentences under length constraints and selects the highest-ranked sentences as the summary.Source: §1 · p. 1
  • Neural extractive models commonly use hierarchical encoders to represent documents and predict binary labels for each sentence.Source: §1 · p. 1
  • Sentence-level labels are usually created by rule-based methods, such as maximizing ROUGE against reference summaries, and may be inaccurate.Source: §1 · p. 1
  • Training complex hierarchical models with inaccurate binary labels is challenging, and an initial extractive model overfit the training set quickly after the second epoch.Source: §1 · p. 2
  • HIBERT pre-trains the hierarchical encoder on unlabeled data before learning sentence classification from the pre-trained initialization.Source: §1 · p. 2

2 Related Work

Related work spans extractive and abstractive summarization, neural and transformer-based pre-training, and HIBERT’s document-level hierarchical encoder. HIBERT extends bidirectional pre-training from sentence encoders to hierarchical document encoders for summarization.

Extractive Summarization

  • Extractive summarization selects important sentences or other textual units, typically using ranking, classification, sequence labeling, or optimization models.Source: §2 · p. 2
  • Neural extractive models replace manually engineered document features with hierarchical LSTMs or CNNs and use neural decoders for sequence labeling or classification.Source: §2 · p. 2

Abstractive Summarization

  • Abstractive summarization rewrites documents into summaries, but generated content may differ in meaning or repeat content despite copy, coverage, and reinforcement-learning mechanisms.Source: §2 · p. 2

Pre-trained NLP Models

  • Language-model pre-training evolved from predicting surrounding or sequential words to bidirectional masked language modeling with masked tokens.Source: §2 · pp. 2–3
  • During training, HIBERT masks a sentence and uses a decoder to predict the original sentence.Source: Figure 1 · p. 3
  • HIBERT applies pre-training to hierarchical document encoders rather than only word embeddings or sentence encoders, targeting document representations important for summarization.Source: §2 · p. 3

3 Model

HIBERT represents documents hierarchically with nested Transformer encoders: one encodes words within sentences, and another contextualizes sentence representations across the document. It is pretrained by masking or corrupting sentences and predicting their original content.

Document representation

  • HIBERT uses a sentence-level Transformer to encode each sentence and a document-level Transformer to contextualize sentence representations.The two Transformer encoders are nested hierarchically, producing context-sensitive sentence representations for the document.Source: §3.1 · p. 3
  • The sentence representation is the final hidden state at the artificial EOS token, augmented with the sentence's positional embedding.Words and sentences share the same positional embedding matrix.Source: §3.1 · p. 3
  • The document-level Transformer applies self-attention to sentence representations and produces context-sensitive representations for every sentence.This hierarchical bidirectional encoder replaces earlier hierarchical designs based on recurrent or convolutional networks.Source: §3.1 · pp. 3–4

Pre-training

  • HIBERT pretraining randomly selects 15% of document sentences, transforms them through masking, retention, or random replacement, and predicts the selected sentences.The three transformations use 80% masking, 10% unchanged sentences, and 10% random-sentence replacement.Source: §3.2 · p. 4
  • The pretraining objective minimizes the negative log-likelihood of the masked sentences conditioned on their paired documents, enabling automatic generation of unlabeled training data.The pretrained encoder is then applied to downstream document summarization.Source: §3.2 · p. 5

4 Experiments

The experiments evaluate HIBERT-based extractive summarization on CNNDM and NYT50 using automatic ROUGE and human judgments. They compare pre-training strategies, baselines, model sizes, and human rankings, finding benefits from pre-training and a limitation for larger or deeper randomly initialized models.

Experimental setup

  • HIBERTS and HIBERTM are evaluated with full-length F1 ROUGE, while Table 1 reports ROUGE-1, ROUGE-2, and ROUGE-L on CNNDM.The reported evaluation uses the ROUGE-1.5.5.pl script.Source: §4.3 · pp. 6–7, Table 1 · p. 7

Results

  • HIBERTS outperforms BERT by 0.4 to 0.5 ROUGE while using 54.6M rather than 110M parameters.The comparison uses blocks of 10 sentences for the BERT baseline.Source: §4.4 · p. 8
  • 2.0 ROUGE is the improvement of HIBERTM over the HeriTransformer baseline on NYT50.The passage also reports that HIBERT variants outperform BERT significantly according to the ROUGE script.Source: §4.4 · p. 8, Table 2 · p. 8

Human evaluation

  • HIBERTM is selected as best in 30% of human-evaluation cases and is better than all compared systems except Human.Its ratings differ significantly from all comparison systems at p < 0.05, while it still trails Human.Source: §4.4 · p. 8, Table 4 · p. 8

Pre-training strategies

  • The two-stage open-domain and in-domain pre-training process is necessary according to CNNDM validation results.The compared strategies are open-domain only, in-domain only, and Open+In-Domain pre-training.Source: §4.4 · p. 8, Table 3 · p. 8

5 Conclusions

The paper pre-trains hierarchical bidirectional transformer encoders on unlabeled data for extractive summarization. Pre-training improves performance, with further gains from adding a large open-domain dataset, and may extend to other hierarchical encoding tasks.

  • Pre-training hierarchical bidirectional transformer encoders on unlabeled data improves extractive summarization performance.Source: §5 · p. 8
  • Adding a large open-domain dataset to pre-training leads to even better performance.Source: §5 · p. 8
  • The pre-trained encoders may also support tasks requiring hierarchical document encodings, such as document question answering.Source: §5 · p. 8
  • Future work includes improving hierarchical document encoder architectures and designing other objectives for training hierarchical transformers.Source: §5 · p. 9

Related papers

Loading 1905.06566v1…