Source-linked AI summary
A Hierarchical Neural Autoencoder for Paragraphs and Documents
Jiwei Li, Minh-Thang Luong, Dan Jurafsky
TL;DR
Generating coherent paragraphs and documents remains difficult for recurrent neural models because coherence extends beyond individual sentences. The paper trains a hierarchical LSTM autoencoder to reconstruct multi-sentence texts and finds that it partially preserves semantic, syntactic, and discourse integrity while outperforming standard sequence-to-sequence models.
Problem
Generating coherent paragraphs or documents requires modeling relationships among text units beyond the sentence level, but neural approaches for broader coherence remain unclear.
Method
The paper trains a paragraph-to-paragraph autoencoder whose hierarchical LSTMs represent tokens, sentences, and paragraphs before reconstructing the input text.
Results
The hierarchical models partially preserve semantic and syntactic integrity and generate meaningful, grammatical sentences in coherent order, outperforming standard sequence-to-sequence models.
Takeaways & Limitations
The results suggest that neural models can encode complex clues about how coherent multi-sentence texts are connected.
Takeaways & Limitations
The proposed coherence metric primarily measures semantic text-order preservation and barely considers syntactic issues.
Abstract
from arXiv · showhide
Natural language generation of coherent long texts like paragraphs or longer documents is a challenging problem for recurrent networks models. In this paper, we explore an important step toward this generation task: training an LSTM (Long-short term memory) auto-encoder to preserve and reconstruct multi-sentence paragraphs. We introduce an LSTM model that hierarchically builds an embedding for a paragraph from embeddings for sentences and words, then decodes this embedding to reconstruct the original paragraph. We evaluate the reconstructed paragraph using standard metrics like ROUGE and Entity Grid, showing that neural models are able to encode texts in a way that preserve syntactic, semantic, and discourse coherence. While only a first step toward generating coherent text units from neural models, our work has the potential to significantly impact natural language generation and summarization\footnote{Code for the three models described in this paper can be found at www.stanford.edu/~jiweil/ .
1 Introduction
The paper addresses the difficulty of generating coherent multi-sentence text by extending neural sequence models from local word composition to hierarchical paragraph representations and reconstruction.
- 1 Introduction: Generating coherent text remains difficult because discourse relations require understanding each unit’s communicative role within recursively larger contexts.The paper argues that handcrafted features may be insufficient and that sentence representations do not yet clearly support broader text coherence.
- 1 Introduction: LSTM models capture local semantic and syntactic composition in neighboring words, motivating their extension to paragraphs and documents.The paper asks whether higher-level discourse relations can be represented compositionally in neural models.
- 1 Introduction: The proposed autoencoder reconstructs paragraph or document sequences from compressed representations built hierarchically over tokens, sentences, and paragraphs.Different LSTM levels capture token-to-token and sentence-to-sentence compositionality.
- 1 Introduction: The work is presented as a first step toward neural generation of larger coherent text structures, with later sections covering the model, experiments, and conclusion.The paper frames paragraph-to-paragraph reconstruction as a component task for broader multi-text generation.
2 Long-Short Term Memory (LSTM)
The section introduces LSTMs as recurrent models that update representations through gated inputs, memory, and outputs, then use softmax-based sequential decoding for sequence generation.
- 2 Long-Short Term Memory (LSTM): At each timestep, an LSTM combines the current text-unit embedding with the previous hidden representation while using input, memory, and output gates.The notation distinguishes e_t for a word or sentence embedding from h_t for the recurrently computed representation.
- 2 Long-Short Term Memory (LSTM): Sequence-to-sequence LSTMs map input sequences to output sequences and define a distribution over tokens predicted sequentially with softmax.The model uses separate input and output LSTMs to capture different compositional patterns.
- 2 Long-Short Term Memory (LSTM): Decoding ends when an <end> token is predicted, using either greedy selection or beam search for each next-token decision.Greedy search selects the token with the largest conditional probability, while beam search retains multiple hypotheses.
3 Paragraph Autoencoder
The paragraph autoencoder hierarchically represents documents through word-, sentence-, and document-level LSTMs, then decodes the compressed representation to reconstruct the original text. An attention variant additionally links decoding to input sentences.
- Paragraph Autoencoder: The proposed autoencoder compresses a paragraph or document into a vector and reconstructs the same document from that representation.The input and output are identical, making this a paragraph-to-paragraph or document-to-document reconstruction task.
- Hierarchical Structure: The hierarchical design treats sentence juxtaposition as contributing joint paragraph meaning, extending compositional modeling beyond neighboring words.This structure is intended to capture interactions among higher-level text units as well as word-level composition.
- Hierarchical Encoder: The hierarchical encoder first derives sentence representations from words, then combines sentence representations into a document representation.A word-level LSTM produces sentence vectors, while a sentence-level LSTM processes those vectors sequentially to obtain e_D.
- Hierarchical Decoder: During decoding, a sentence-level LSTM determines sentence representations and a word-level LSTM generates each sentence token by token until end markers.Document decoding stops when the end_D token is predicted, while sentence decoding stops at the ends token.
- Attention Model: Attention links the current decoding state to encoded input sentences and averages sentence-level weights before word prediction.The attention model uses sentence-level hidden vectors from the encoder and combines the resulting attention vector with the decoder state.
- Training and Testing: Training maximizes output likelihood with softmax token prediction and minibatch stochastic gradient descent, while testing uses greedy decoding without beam search.The training error is backpropagated through word- and sentence-level decoders and then through the encoder.
4 Experiments
The experiments evaluate hierarchical LSTM autoencoders on hotel reviews and Wikipedia paragraphs using lexical-overlap, precision, and coherence metrics. Hierarchical sentence-level modeling outperforms standard sequence-to-sequence models, while sentence-level attention further improves performance and preserves input order.
- 4.1 Dataset: The autoencoder is evaluated on roughly 340,000 hotel-review training documents and 500,000 Wikipedia training paragraphs, with 40,000 and 50,000 test documents respectively.Hotel reviews use a 25,000-word vocabulary; Wikipedia uses a 120,000-word vocabulary, with documents filtered by unknown-word rates.
- 4.3 Evaluations: The evaluation uses ROUGE and BLEU to measure candidate-output closeness to the input reference, alongside a tailored coherence metric.ROUGE measures n-gram recall, while BLEU emphasizes precision and penalizes overly long outputs.
- 4.3 Evaluations: The tailored coherence metric compares output and input sentence order using aligned sentence-position distances and penalizes their absolute differences.The metric also heavily penalizes irrelevant outputs aligned to random inputs, but considers semantic coherence more than syntactic issues.
- 4.4 Results: The hierarchical model that considers sentence-level structure outperforms standard sequence-to-sequence models, and sentence-level attention improves over vanilla hierarchical models.The results summary reports better performance for hotel reviews than for Wikipedia, attributed to the reviews’ more fixed and predictable format.
- 4.4 Results: L = 1.57 for the attention-based hierarchical model on hotel reviews, corresponding to an average relative permutation degree of 1.57 between input sentences.On Wikipedia, the original order remains adequately maintained with L = 2.04; smaller coherence-score values indicate better performance.
5 Discussion and Future Work
The hierarchical LSTM partially preserves semantic and syntactic integrity while generating meaningful, grammatical sentences in coherent order, outperforming standard sequence-to-sequence models. The autoencoder is a preliminary step toward more sophisticated document-generation applications.
- Hierarchical LSTM models partially preserve the semantic and syntactic integrity of multi-sentence texts while generating meaningful and grammatical sentences in coherent order.
- The model performs better than standard sequence-to-sequence models that omit texts’ intrinsic hierarchical discourse structure.
- The results suggest neural models can encode complex clues about how coherent texts are connected.
- Autoencoding larger texts is presented as a preliminary effort toward enabling neural models to handle discourse and more sophisticated generation tasks.
- Performance on the autoencoder task could benefit from more sophisticated neural models, but perfecting reconstruction is not the authors’ long-term goal.
- The paper identifies summarization and question answering as examples of document-to-document generation tasks that could extend this paradigm.