Source-linked AI summary
HIBERT: Document Level Pre-training of Hierarchical Bidirectional Transformers for Document Summarization
Xingxing Zhang, Furu Wei, Ming Zhou
TL;DR
Neural extractive summarization relies on heuristically created sentence labels, making hierarchical encoder training challenging. HIBERT pre-trains hierarchical bidirectional Transformer encoders on unlabeled data, improving summarization performance over randomly initialized counterparts and achieving state-of-the-art results on CNN/Dailymail and New York Times datasets.
Problem
Heuristically generated sentence-level labels for neural extractive summarization are often inaccurate, making hierarchical encoder training challenging.
Method
HIBERT pre-trains hierarchical bidirectional Transformer document encoders on unlabeled data before applying them to extractive summarization.
Results
HIBERT improves summarization performance over randomly initialized counterparts and achieves state-of-the-art performance on CNN/Dailymail and New York Times datasets.
Takeaways & Limitations
Adding a large open-domain dataset to pre-training leads to better performance, while the two-stage open-domain and in-domain process is necessary on CNN/Dailymail.
Takeaways & Limitations
Deeper and larger models produced inferior results, suggesting that training large or deep models on this dataset without good initialization is challenging.
Abstract
from arXiv · showhide
Neural extractive summarization models usually employ a hierarchical encoder for document encoding and they are trained using sentence-level labels, which are created heuristically using rule-based methods. Training the hierarchical encoder with these \emph{inaccurate} labels is challenging. Inspired by the recent work on pre-training transformer sentence encoders \cite{devlin:2018:arxiv}, we propose {\sc Hibert} (as shorthand for {\bf HI}erachical {\bf B}idirectional {\bf E}ncoder {\bf R}epresentations from {\bf T}ransformers) for document encoding and a method to pre-train it using unlabeled data. We apply the pre-trained {\sc Hibert} to our summarization model and it outperforms its randomly initialized counterpart by 1.25 ROUGE on the CNN/Dailymail dataset and by 2.0 ROUGE on a version of New York Times dataset. We also achieve the state-of-the-art performance on these two datasets.
1 Introduction
Extractive summarization selects important document sentences but depends on sentence-level labels that are often heuristically generated and inaccurate. HIBERT addresses this training challenge by pre-training its hierarchical encoder on unlabeled data before sentence classification.
- Extractive summarization ranks sentences under length constraints and selects the highest-ranked sentences as the summary.
- Neural extractive models commonly use hierarchical encoders to represent documents and predict binary labels for each sentence.
- Sentence-level labels are usually created by rule-based methods, such as maximizing ROUGE against reference summaries, and may be inaccurate.
- Training complex hierarchical models with inaccurate binary labels is challenging, and an initial extractive model overfit the training set quickly after the second epoch.
- HIBERT pre-trains the hierarchical encoder on unlabeled data before learning sentence classification from the pre-trained initialization.
2 Related Work
Related work spans extractive and abstractive summarization, neural and transformer-based pre-training, and HIBERT’s document-level hierarchical encoder. HIBERT extends bidirectional pre-training from sentence encoders to hierarchical document encoders for summarization.
- Extractive Summarization: Extractive summarization selects important sentences or other textual units, typically using ranking, classification, sequence labeling, or optimization models.
- Extractive Summarization: Neural extractive models replace manually engineered document features with hierarchical LSTMs or CNNs and use neural decoders for sequence labeling or classification.
- Abstractive Summarization: Abstractive summarization rewrites documents into summaries, but generated content may differ in meaning or repeat content despite copy, coverage, and reinforcement-learning mechanisms.
- Pre-trained NLP Models: Language-model pre-training evolved from predicting surrounding or sequential words to bidirectional masked language modeling with masked tokens.
- Pre-trained NLP Models: During training, HIBERT masks a sentence and uses a decoder to predict the original sentence.
- Pre-trained NLP Models: HIBERT applies pre-training to hierarchical document encoders rather than only word embeddings or sentence encoders, targeting document representations important for summarization.
3 Model
HIBERT represents documents hierarchically with nested Transformer encoders: one encodes words within sentences, and another contextualizes sentence representations across the document. It is pretrained by masking or corrupting sentences and predicting their original content.
- Document representation: HIBERT uses a sentence-level Transformer to encode each sentence and a document-level Transformer to contextualize sentence representations.The two Transformer encoders are nested hierarchically, producing context-sensitive sentence representations for the document.
- Document representation: The sentence representation is the final hidden state at the artificial EOS token, augmented with the sentence's positional embedding.Words and sentences share the same positional embedding matrix.
- Document representation: The document-level Transformer applies self-attention to sentence representations and produces context-sensitive representations for every sentence.This hierarchical bidirectional encoder replaces earlier hierarchical designs based on recurrent or convolutional networks.
- Pre-training: HIBERT pretraining randomly selects 15% of document sentences, transforms them through masking, retention, or random replacement, and predicts the selected sentences.The three transformations use 80% masking, 10% unchanged sentences, and 10% random-sentence replacement.
- Pre-training: The pretraining objective minimizes the negative log-likelihood of the masked sentences conditioned on their paired documents, enabling automatic generation of unlabeled training data.The pretrained encoder is then applied to downstream document summarization.
4 Experiments
The experiments evaluate HIBERT-based extractive summarization on CNNDM and NYT50 using automatic ROUGE and human judgments. They compare pre-training strategies, baselines, model sizes, and human rankings, finding benefits from pre-training and a limitation for larger or deeper randomly initialized models.
- Experimental setup: HIBERTS and HIBERTM are evaluated with full-length F1 ROUGE, while Table 1 reports ROUGE-1, ROUGE-2, and ROUGE-L on CNNDM.The reported evaluation uses the ROUGE-1.5.5.pl script.
- Results: HIBERTS outperforms BERT by 0.4 to 0.5 ROUGE while using 54.6M rather than 110M parameters.The comparison uses blocks of 10 sentences for the BERT baseline.
- Results: 2.0 ROUGE is the improvement of HIBERTM over the HeriTransformer baseline on NYT50.The passage also reports that HIBERT variants outperform BERT significantly according to the ROUGE script.
- Human evaluation: HIBERTM is selected as best in 30% of human-evaluation cases and is better than all compared systems except Human.Its ratings differ significantly from all comparison systems at p < 0.05, while it still trails Human.
- Pre-training strategies: The two-stage open-domain and in-domain pre-training process is necessary according to CNNDM validation results.The compared strategies are open-domain only, in-domain only, and Open+In-Domain pre-training.
5 Conclusions
The paper pre-trains hierarchical bidirectional transformer encoders on unlabeled data for extractive summarization. Pre-training improves performance, with further gains from adding a large open-domain dataset, and may extend to other hierarchical encoding tasks.
- Pre-training hierarchical bidirectional transformer encoders on unlabeled data improves extractive summarization performance.
- Adding a large open-domain dataset to pre-training leads to even better performance.
- The pre-trained encoders may also support tasks requiring hierarchical document encodings, such as document question answering.
- Future work includes improving hierarchical document encoder architectures and designing other objectives for training hierarchical transformers.