PaperlayerView exact revision on arXiv

Source-linked AI summary

Contextual LSTM (CLSTM) models for Large scale NLP tasks

Shalini Ghosh, Oriol Vinyals, Brian Strope, Scott Roy, Tom Dean, Larry Heck

arXiv:1602.06291v2cs.CL

TL;DR

Language models need context across document hierarchies, but typical LSTMs primarily use words as features. This paper introduces CLSTM, which incorporates topic context into LSTMs and evaluates it across three NLP tasks, finding improvements over baseline LSTMs across Wikipedia and Google News. The study also explores unsupervised thought vectors, which improve over word-only models but remain below supervised-topic CLSTM performance.

  • Problem

    Typical LSTM language models primarily use words as features, motivating evaluation of whether hierarchical contextual features improve multiple NLP tasks.

  • Method

    CLSTM extends LSTM language models by incorporating topic features from different text segments and is evaluated on word prediction, next sentence selection, and sentence topic prediction.

  • Results

    CLSTM improves over LSTM by 2–3% in word-prediction perplexity, approximately 20% in next-sentence-selection accuracy, and approximately 10% in topic prediction across reported experiments.

  • Takeaways & Limitations

    Using contextual features in CLSTM benefits several NLP tasks and has implications for sentence completion, question answering, paraphrase generation, and dialog systems.

  • Takeaways & Limitations

    The study could not compare CLSTM with other topic-integrating LSTM techniques because scalable implementations were unavailable for its large datasets and vocabularies.

Abstract

from arXiv · show

Documents exhibit sequential structure at multiple levels of abstraction (e.g., sentences, paragraphs, sections). These abstractions constitute a natural hierarchy for representing the context in which to infer the meaning of words and larger fragments of text. In this paper, we present CLSTM (Contextual LSTM), an extension of the recurrent neural network LSTM (Long-Short Term Memory) model, where we incorporate contextual features (e.g., topics) into the model. We evaluate CLSTM on three specific NLP tasks: word prediction, next sentence selection, and sentence topic prediction. Results from experiments run on two corpora, English documents in Wikipedia and a subset of articles from a recent snapshot of English Google News, indicate that using both words and topics as features improves performance of the CLSTM models over baseline LSTM models for these tasks. For example on the next sentence selection task, we get relative accuracy improvements of 21% for the Wikipedia dataset and 18% for the Google News dataset. This clearly demonstrates the significant benefit of using context appropriately in natural language (NL) tasks. This has implications for a wide variety of NL applications like question answering, sentence completion, paraphrase generation, and next utterance prediction in dialog systems.

1. INTRODUCTION

The paper argues that hierarchical, long-range context can improve language modeling and introduces CLSTM, which adds topic features to LSTM models for three NLP tasks.

  • Motivation: Documents contain sequential structure across sections, paragraphs, sentences, phrases, and words, motivating hierarchical context in language models.Such structure can potentially improve predictive accuracy.
  • Approach: CLSTM extends LSTM language models by incorporating topics derived from different text segments as contextual features.The approach extends prior contextual RNN language modeling to LSTMs.
  • Tasks: The evaluation covers word prediction, next sentence selection, and sentence topic prediction.These tasks target sentence completion, candidate response selection, and prediction of subsequent conversational topics.
  • Evaluation: CLSTM combines word-level features with topic features and is compared with a baseline LSTM using only word-level features.Experiments use large-scale Wikipedia and Google News document corpora.
  • Additional direction: The paper also reports initial promising results from unsupervised thought embeddings generated through the model structure instead of supervised external topics.This explores whether long-range context can be summarized without extraneous topic annotations.

2. RELATED WORK

Prior work models hierarchy and context in text with probabilistic, recurrent, convolutional, and representation-learning methods. This paper positions CLSTM as a scalable two-level contextual LSTM for large-scale NLP tasks.

  • Hierarchical document models: Earlier document models used HMMs, semantic parsing, and hierarchical probabilistic, Bayesian, HMM, and CRF approaches to represent document structure.These methods target content structure and hierarchical dependencies.
  • Positioning: The paper extends contextual RNN language modeling to LSTMs and evaluates tasks beyond next-word prediction.The authors work with LSTM-based language models.
  • Hierarchical recurrent models: Hierarchical recurrent approaches process inputs at different temporal granularities or model document- and sentence-level structure.Examples include Clockwork RNNs and hierarchical RNN language models.
  • Neural context representations: Other neural methods represent context with paragraph vectors, inferred LDA topics, tree-structured LSTMs, convolutional networks, or skip-thought vectors.These approaches use latent context, structured interactions, multi-level text representations, or surrounding-sentence reconstruction.
  • Positioning: CLSTM models long-range hierarchical signals with a two-level LSTM architecture and had not previously been trained at this scale for the listed NLP tasks, according to the paper.The signals include sentence-level and paragraph-level topics.

3. WORD PREDICTION

For word prediction, CLSTM feeds LSTM cells both word inputs and hierarchical topic features derived from sentence and paragraph context. The best Wikipedia configuration uses sentence- and paragraph-level topics, while larger models eventually show diminishing returns.

  • Task definition: The word prediction task predicts the next word from preceding words and a topic computed from a relevant text subsequence.Topics may derive from the current sentence prefix, previous sentence, or paragraph prefix.
  • Model: CLSTM modifies the standard LSTM gates, cell, and output computations by adding a topic vector T to each operation.The topic signal is incorporated into the input, forget, cell, and output equations.
  • Model: Concatenating word and topic embeddings as a composite input worked better in practice than other topic-word combination strategies.Each LSTM cell receives topic input because cells can have different topics; previous-sentence topics may be shared across cells.
  • Features: The evaluated hierarchical topic features are previous-sentence, current-sentence-segment, and paragraph-segment topic identifiers.Topics are obtained with HTM, a supervised hierarchical topic model with approximately 750 leaf categories.
  • Dataset: The Wikipedia corpus contains 4.7 million documents, and preprocessing produced a 129K-term vocabulary with a 3% validation out-of-vocabulary rate.Models were trained on randomly divided train, validation, and test data, with approximately 1600 topic categories in the dataset.
  • Results: The Word + SentSegTopic + ParaSegTopic CLSTM achieved the best perplexity, indicating value from both local sentence and longer-range paragraph context.When the current segment topic is present, the previous-sentence topic does not matter.
  • Results: Beyond 1024 hidden units, performance gains diminished while computational overhead increased substantially.Increasing hidden units improved performance up to that point.
  • Evaluation boundary: CLSTM models could not be compared with other topic-integrating LSTM techniques because scalable implementations were unavailable for the datasets and vocabularies used.The study therefore used a finely tuned, large-scale LSTM baseline.

4. NEXT SENTENCE SELECTION

Next sentence selection asks the model to choose the most likely continuation from candidate sentences, using preceding sentences and their topics as context. CLSTM combines word and topic features and improves selection accuracy over a baseline LSTM, including under more difficult candidate sets.

  • Problem Instantiation: The task selects the most likely next sentence from candidates given a preceding sentence sequence and corresponding topics.The model evaluates candidate continuations for each context prefix.
  • Approach: CLSTM predicts candidate next sentences using words and topics from the preceding context and the candidate sentence.The baseline LSTM uses words from the context, while CLSTM additionally uses sentence-level topic features.
  • Experimental Results: 10.3% lower perplexity: CLSTM reached 24.81 versus 27.66 for the baseline LSTM with 1024 hidden units.This intrinsic result supported expectations for next sentence scoring performance.
  • Experimental Results: 21% relative accuracy improvement: CLSTM achieved 63% average accuracy versus 52% for the baseline LSTM.The evaluation used 5,000 sentence sequences, with each context choosing among candidate next sentences.
  • Experimental Results: Random selection would achieve 2% accuracy, far below the reported CLSTM average accuracy.The random predictor selects from the candidate sentence set without modeling the context.
  • Experimental Results: CLSTM's improvement was larger when the competing negative sentences shared the same HTM topic, making selection more difficult.The authors report this result as an effect of using non-random, same-topic negatives.

5. SENTENCE TOPIC PREDICTION

The paper evaluates predicting the next sentence’s topic from the current sentence’s words and topic, using hierarchical CLSTM features and alternative unsupervised thought vectors. CLSTM improves over baseline approaches, while thought vectors remain below supervised HTM topics.

  • Task definition: The task predicts the topic of the next sentence from the current sentence’s words and topic, including settings with and without the next sentence’s words.The formulation models P(T_i+1|s_i, T_i, Θ), and the authors use a topic target rather than a joint word/topic target.
  • Model architecture: Ablation experiments found that an unrolled architecture, with a separate LSTM for each sentence, outperformed the rolled-up architecture used for word prediction.The unrolled model treats sentences in a paragraph with separate LSTMs, whereas the rolled-up model feeds them into one LSTM.
  • Topic signals: The experiments use HTM-generated topics because obtaining supervised human topic ratings at scale was difficult.HTM itself was trained on human ratings, but its sentence topics serve as an available substitute for direct sentence-level supervision.
  • Supervised topic features: The CLSTM combines words and sentence topics to predict the next sentence topic, and beats the SentTopic baseline by more than 12% in perplexity.Models were trained with 256, 512, and 1024 hidden units until convergence.
  • Unsupervised topic signals: Thought vectors use the previous sentence’s LSTM representation as a recurrent topic feature for the current sentence.The approach connects sentence-level LSTMs without resetting hidden state, allowing the current sentence to access the previous sentence’s state indirectly.
  • Unsupervised topic signals: The thought-vector CLSTM improved perplexity by 3% over a word-only LSTM but remained below the HTM-based CLSTM.The authors identify better low-dimensional projections as a future direction for closing this gap.

6. RESULTS ON GOOGLE NEWS DATA

On the Google News corpus, CLSTM improves over word-only or topic-only LSTM baselines across word prediction, next sentence selection, and next sentence topic prediction. The reported gains are consistent with those obtained on Wikipedia.

  • Dataset: The Google News subset contains 4.3 million documents and uses a 100K-word vocabulary after filtering infrequent words.The corpus was divided into training, test, and validation datasets, with models using 1024 hidden units.
  • Word prediction: CLSTM improves word-prediction perplexity by approximately 2% over an LSTM with perplexity approximately 37, using word and hierarchical topic features.The features include sentence-segment topics and paragraph-sentence topics.
  • Topic prediction: CLSTM improves next sentence topic-prediction performance by approximately 9% over an LSTM using only the current sentence topic, with baseline perplexity approximately 5.The CLSTM uses both the current sentence’s words and topic as features.
  • Cross-dataset comparison: The CLSTM shows similar improvements over LSTM on both Wikipedia and Google News for each selected NLP task.The conclusion treats this cross-dataset pattern as evidence of comparable gains across the two corpora.

7. CONCLUSIONS

The paper concludes that contextual features improve CLSTM performance across word prediction, next sentence selection, and topic prediction. It reports similar gains on Google News and identifies implications for several NLP applications.

  • Main findings: CLSTM improves performance across word prediction, next sentence selection, and sentence topic prediction relative to LSTM-based baselines.The reported average gains are 2–3% in perplexity, approximately 20% in next sentence selection accuracy, and approximately 10% in topic prediction.
  • Main findings: The reported gains are similar on the Google News dataset, supporting the approach’s generalizability across the two evaluated corpora.The paper also reports an approximately 7% improvement over BOW-DNN for topic prediction.
  • Applications: The authors connect contextual-feature gains to applications including sentence completion, question answering, paraphrase generation, and dialog systems.These applications correspond to the evaluated prediction and selection tasks.

8. FUTURE WORK

Future work extends unsupervised thought-vector modeling toward hierarchical architectures that propagate context across sentences. The authors also propose applying contextual features to paraphrase generation and conversation modeling.

  • Hierarchical thought vectors: A proposed hierarchical LSTM uses sentence-level and word-level LSTMs to model continuity of thought across a paragraph.Connections between hidden layers propagate a previous sentence’s thought vector to influence the words of the next sentence.
  • Applications: The authors plan to explore contextual word and topic features for better paraphrase generation and topic-level signals such as Dialog Acts for next-utterance prediction.These extensions target language modeling and conversation applications.
Loading 1602.06291v2…