Source-linked AI summary
Controlling Output Length in Neural Encoder-Decoders
Yuta Kikuchi, Graham Neubig, Ryohei Sasano, Hiroya Takamura, Manabu Okumura
TL;DR
The paper addresses the lack of methods for controlling encoder-decoder output length, a capability needed when summaries must meet desired lengths. It examines two decoding-based and two learning-based methods, finding that learning-based methods better handle long summaries and control length without losing summary quality.
Problem
Methods for controlling encoder-decoder output length had not been investigated, despite the importance of desired summary lengths in text summarization.
Method
The paper investigates four length-control methods: two decoding-based methods and two learning-based methods that incorporate desired length into the model.
Results
Learning-based methods generally outperform decoding-based methods for longer summaries and control output length without decreasing ROUGE scores compared with existing summarization methods.
Takeaways & Limitations
Length-controllable learning-based methods can provide desired output lengths while retaining competitive summarization quality.
Abstract
from arXiv · showhide
Neural encoder-decoder models have shown great success in many sequence generation tasks. However, previous work has not investigated situations in which we would like to control the length of encoder-decoder outputs. This capability is crucial for applications such as text summarization, in which we have to generate concise summaries with a desired length. In this paper, we propose methods for controlling the output sequence length for neural encoder-decoder models: two decoding-based methods and two learning-based methods. Results show that our learning-based methods have the capability to control length without degrading summary quality in a summarization task.
1 Introduction
Neural encoder-decoder models have succeeded across sequence-generation tasks, but controlling their output length remains an uninvestigated need for summarization. The paper proposes four length-control methods and finds learning-based methods effective for longer summaries while remaining competitive on standard evaluation.
- Encoder-decoder models have succeeded in machine translation, image captioning, parsing, dialogue response generation, and sentence summarization.
- Summarization systems need to generate summaries at desired lengths because requirements vary with user needs, device size, and source-document information.
- The paper proposes four methods for controlling encoder-decoder output length: two decoding-based and two learning-based.Decoding-based methods receive desired length during decoding, whereas learning-based methods modify the architecture to receive it as input.
- Learning-based methods outperform decoding-based methods for long summaries of 50 or 75 bytes.
- The proposed methods remain competitive with existing methods on standard DUC2004 shared task-1 settings.
2 Background
Summarization systems must handle requested lengths, but standard neural encoder-decoders produce lengths shaped by training data and recent systems use ad-hoc controls. The paper frames explicit length control as an important capability for settings such as DUC2004.
- Traditional summarization research includes rule-based and statistical word deletion, as well as abstractive methods using syntactic transduction.
- Explicitly controlling output sequence length in neural language models or encoder-decoders had not been investigated to the authors’ knowledge.
- Summarization systems must generate various lengths because standard settings provide both the source document and desired summary length as inputs.
- A standard encoder-decoder trained on Gigaword produced an average output length of 38.02 bytes, versus 51.38 bytes for human-written summaries.
- DUC2004 sets a maximum summary length of 75 bytes, motivating systems that can generate sentences up to that limit.
- Recent neural summarization systems inhibit EOS generation, generate a fixed number of words, and truncate outputs to 75 bytes.
3 Network Architecture: Encoder-Decoder with Attention
The base model is an attention-based encoder-decoder using bidirectional RNNs with LSTM units. It encodes source sentences, computes attention-based context during decoding, and trains by maximizing sentence-summary likelihood before beam-search decoding.
- The base architecture consists of a bidirectional RNN encoder, an RNN decoder, and an attention mechanism.
- Encoder: The bidirectional encoder processes the source sentence in forward and backward directions using separate recurrent networks.The paper uses LSTM networks with memory cells in both directions.
- Decoder: The decoder is an LSTM-based RNN whose initial hidden state and memory cell are set from the encoded representations.
- Decoder and Attender: Attention summarizes forward and backward encoder states and computes a weighted context vector for each decoding step.
- Decoder and Attender: The model updates the next-word distribution from the decoder state and uses input feeding to provide the resulting state at the next step.
- Training and Decoding: Training maximizes sentence-summary log likelihood, and beam search finds the output with maximum conditional probability after training.
4 Controlling Length in Encoder-decoders
The section presents four methods for controlling encoder-decoder output length: two decoding-based methods and two learning-based methods. The learning-based methods incorporate length information into the model, either throughout decoding or through decoder initialization.
- Four methods control output length: fixLen and fixRng modify decoding, while LenEmb and LenInit modify the model and training.
- Decoding-based methods: fixLen suppresses EOS generation and stops when the desired length is reached, replacing an overlength sequence’s last word with EOS.
- Decoding-based methods: fixRng allows EOS generation but discards sequences shorter than a minimum or longer than a maximum length.
- Learning-based methods: LenEmb feeds an embedding of the remaining length into the decoder LSTM at every decoding step.The remaining length is initialized after encoding and updated as words are generated.
- Learning-based methods: LenInit inputs the desired length once by initializing the decoder states, using the LSTM memory cell to manage length internally.The method initializes the memory cell as m0 = bc ∗length, where length is the desired length.
5 Experiment
The experiments train the models on approximately 3.6 million Gigaword sentence-headline pairs and evaluate them on DUC2004 short-document summarization. Evaluation uses ROUGE metrics with permutation-based significance testing.
- Approximately 3.6 million Annotated English Gigaword sentence-headline pairs were used for training.The pairs contain each source document’s first sentence and its headline.
- DUC2004 task-1 provides 500 evaluation documents with four human-written summaries per document.Systems generate very short single-document summaries, and outputs exceeding 75 bytes are truncated.
- ROUGE-1, ROUGE-2, and ROUGE-L were used, with a two-sided permutation test for significance at p ≤0.05.
- Models were optimized with Adam using mini-batches of 80, 100-dimensional word embeddings, and 200-dimensional hidden states.
- Beam size 10 was sufficient for all methods except fixRng, which used beam size 30 because it more aggressively discards beam candidates.
6 Result
The experiments evaluate ROUGE across length limits, generated-summary examples, output-length distributions, and comparison with existing methods. Learning-based methods generally perform better for longer summaries and control length without reducing ROUGE on standard settings.
- ROUGE Evaluation: Learning-based methods tend to outperform decoding-based methods for 50- and 75-byte summaries, but show no significant difference at 30 bytes.The authors hypothesize that the 30-byte setting requires a 15.38% average compression rate, below the 30% average in training data.
- Generated Summaries: All models can often generate well-formed sentences, with some learning-based outputs forming single noun phrases suitable for short length settings.The examples include paraphrases and outputs from LenEmb(30) and LenInit(30).
- Length Control Capability: Learning-based model output lengths concentrate near the desired length, unlike the widely dispersed lengths of the standard encoder-decoder.Figure 6 presents histograms for the standard model, LenEmb, and LenInit.
- Length Control Capability: For a 30-byte target, LenInit produces beam candidates with lengths close to the target, whereas standard beam search produces much more varied lengths.The compared candidates remain well-formed in both settings.
- Comparison with Existing Methods: On DUC2004 shared task-1, fixLen is comparable to existing methods, while LenEmb and LenInit control length without decreasing ROUGE.The comparison uses the methods' scores copied from Table 1.
7 Conclusion
The paper examines output-length control in neural encoder-decoder models through decoding-based and learning-based methods. Learning-based methods generally outperform decoding-based methods and control output length without losing ROUGE relative to existing summarization methods.
- The paper examines four output-length control methods: two decoding-based methods, fixLen and fixRng, and two learning-based methods, LenEmb and LenInit.
- Learning-based methods generally outperform decoding-based methods in the summarization experiments.
- LenEmb and LenInit control output length without losing ROUGE score compared with existing summarization methods.