Source-linked AI summary
Online Representation Learning in Recurrent Neural Language Models
Marek Rei
TL;DR
Neural language models benefit from learned representations, but this paper investigates whether an RNNLM can continue adapting during testing. It adds a sentence- or document-level vector updated after predictions, and reports lower perplexity with fewer parameters and operations.
Problem
The paper investigates extending RNNLMs with continuous adaptation during testing to improve language modeling while controlling model size and computation.
Source: §1 · p. 1Method
The model represents the current text unit with a document vector and updates it from output error derivatives after each prediction while other parameters remain fixed.
Source: §3 · p. 2Results
Perplexity 90.29 with a 35-dimensional document vector versus 95.71 when increasing M by 35, with fewer total parameters and operations reported in the experiments.
Source: §4 · p. 4Takeaways & Limitations
Continuous online learning can make the language model's text representation more informative while improving accuracy and reducing required parameters and computation.
Source: §5 · p. 5The experiments count only matrix multiplication operations because they account for most neural language-model computation.
Source: §4 · p. 4
Abstract
from arXiv · showhide
We investigate an extension of continuous online learning in recurrent neural network language models. The model keeps a separate vector representation of the current unit of text being processed and adaptively adjusts it after each prediction. The initial experiments give promising results, indicating that the method is able to increase language modelling accuracy, while also decreasing the parameters needed to store the model along with the computation required at each step.
1 Introduction
The paper extends recurrent neural network language models with continuous online adaptation during testing. It represents the current sentence with a vector updated by gradient descent, aiming to improve perplexity while reducing model computation and parameters.
1 Introduction
- Neural network models learn smooth word representations that reduce sparsity in natural-language data.Source: §1 · p. 1
- RNNLMs have achieved strong language-modeling results, including the best results and lowest perplexity when combined with a 9-gram MaxEnt model.Source: §1 · p. 1
- The extension continuously adapts an RNNLM during testing using a sentence-level vector updated by gradient descent.The vector is modified from prediction error signals while the sentence is processed.Source: §1 · p. 1
- The approach exchanges some existing model parameters for online learning while achieving lower perplexity and reducing necessary computation.Source: §1 · p. 1
2 RNNLM
The baseline RNNLM predicts each next word from the previous word and recurrent hidden state. It uses class-based output probabilities, negative log-probability training, backpropagation through time, and a learned sentence-start state.
2 RNNLM
- The RNNLM computes the current hidden state from the previous word and previous hidden vector through weight matrices and an activation function.Source: §2 · pp. 1–2
- Class-based output avoids a full-vocabulary softmax by computing probabilities over classes and words with corresponding weight matrices.Source: §2 · p. 2
- Negative log-probability trains the network with gradient descent and backpropagation through time, while sentence-level unrolling supports processing individual sentences.Source: §2 · p. 2
- A learned sentence-start hidden vector supplies an optimized initial state when no preceding information is available.The paper reports slight performance improvements from this starting-state parameterization.Source: §2 · p. 2
3 RNNLM with online learning
The online-learning RNNLM adds a document vector representing the current text unit and updates it after each prediction. The vector is attached to the output layer so sentence-level backpropagation can still update it at every time step.
3 RNNLM with online learning
- The model adds a document vector that represents the current sentence, paragraph, or larger text and supplies additional evidence for output probabilities.Source: §3 · p. 2
- The document vector is updated during testing from output error derivatives, while the model's other parameters remain fixed.It is reset to a learned starting state at the beginning of each sentence.Source: §3 · p. 2
- For larger texts and domain-specific corpora, testing examples could later be incorporated as additional training data for iterative adaptation.Source: §3 · p. 3
- The approach learns a distributed document vector because individual sentences provide little reusable evidence for updating separate word representations.Source: §3 · p. 3
- The method extends Le and Mikolov (2014)'s document-vector idea from feedforward language models to recurrent language models, while directly evaluating language modeling.Source: §3 · p. 3
- Attaching the vector to the output layer enables updates at every time step, unlike input-layer attachment under sentence-end unrolling.Source: §3 · p. 3
4 Experiments
Experiments compare increasing the standard RNNLM capacity with adding an online-learned document vector, measuring perplexity, resource costs, and sentence similarity. The document-vector approach achieves lower perplexity with fewer parameters and operations, with best performance when the vector remains relatively small.
Data and evaluation
- The evaluation dataset was sampled from English Wikipedia after tokenisation, sentence splitting, lowercasing, shuffling, and random sampling into training, development, and test sets.Dataset sizes are reported in Table 2.Source: §4 · p. 3, Table 2 · p. 3
Language-model performance
- Perplexity fell from 95.71 with a 35-dimensional increase in M to 90.29 with a 35-dimensional document vector, and from 105.50 to 98.23 using half the training data.These comparisons use the same number of added elements in M or D.Source: §4 · p. 4
Resource comparison
- Replacing hidden-vector capacity with an actively learned document vector produced fewer total parameters and operations while also lowering perplexity.The operation accounting includes error backpropagation at each time step; the experiments use V = 16,514, C = 100, and E[O] ≈50.Source: §4 · p. 4
Parameter and operation scaling
- A small document vector achieved lower perplexity with fewer additional parameters than simply increasing the main network components.The same relationship held when perplexity was plotted against additional operations, and Figure 3 averages results over 10 random initialisations.Source: §4 · p. 4, Figure 3 · p. 4
Document-vector size
- Under the constraint D + M = 100, the lowest perplexity occurred near D = 23 and M = 77, while much smaller or larger document vectors reduced performance.Only half of the training data was used for these smaller-model experiments.Source: §4 · p. 5
Qualitative evaluation
- Learned sentence vectors retrieved development sentences with high semantic similarity despite almost no word overlap.The last document-vector state for each sentence was compared using cosine similarity, with examples reported in Table 3.Source: §4 · p. 5, Table 3 · p. 5
5 Conclusion
The proposed continuous-online-learning RNNLM maintains and updates a text-unit vector during processing, achieving lower perplexity with more efficient parameter use. Experiments further indicate higher accuracy alongside reduced storage and computation requirements.
- The modified language model achieves lower perplexity through more optimal use of parameters.Source: §5 · p. 5
- Continuous online learning represents the current text unit with a separate vector that starts in a default state and is updated using backpropagation.Source: §5 · p. 5
- Higher accuracy accompanies reduced model storage and computation when the active learning component is included.The experiments indicate these gains together, without specifying quantitative values.Source: §5 · p. 5