PaperlayerView exact revision on arXiv

Source-linked AI summary

Adaptive Input Representations for Neural Language Modeling

Alexei Baevski, Michael Auli

arXiv:1809.10853v3cs.CL

TL;DR

Neural language modeling must balance representation capacity, parameter count, and computation across different input units and output factorizations. The paper introduces adaptive input embeddings, compares word, character, and sub-word approaches in a self-attentional architecture, and reports faster training with improved benchmark perplexity.

  • Problem

    Neural language models face choices among word, character, and sub-word units, while word-based models incur substantial computation over vocabularies such as BILLION WORD's 800K words.

  • Method

    The paper extends adaptive softmax to input embeddings by assigning more capacity to frequent words and less to rare words, then compares input and output factorizations across word, character, and sub-word models.

  • Results

    Adaptive embeddings train more than twice as fast as character input CNNs while achieving higher accuracy and fewer parameters; WikiText-103 reaches 18.7 perplexity.

  • Takeaways & Limitations

    Adaptive input representations reduce model parameters and can further improve training speed when sharing parameters with an adaptive softmax.

  • Takeaways & Limitations

    Additional dropout is not helpful on larger datasets such as BILLION WORD, and a standard softmax over 800K words is not tractable for comparison.

Abstract

from arXiv · show

We introduce adaptive input representations for neural language modeling which extend the adaptive softmax of Grave et al. (2017) to input representations of variable capacity. There are several choices on how to factorize the input and output layers, and whether to model words, characters or sub-word units. We perform a systematic comparison of popular choices for a self-attentional architecture. Our experiments show that models equipped with adaptive embeddings are more than twice as fast to train than the popular character input CNN while having a lower number of parameters. On the WikiText-103 benchmark we achieve 18.7 perplexity, an improvement of 10.5 perplexity compared to the previously best published result and on the Billion Word benchmark, we achieve 23.02 perplexity.

1 INTRODUCTION

The paper introduces adaptive input embeddings for neural language modeling and compares word, character, and sub-word representations in self-attentional models. Adaptive representations improve efficiency and benchmark performance while reducing parameter counts.

  • Adaptive input embeddings extend adaptive softmax to input word representations by assigning more capacity to frequent words and less to rare words.
  • 23% fewer input and output parameters are achieved on BILLION WORD while attaining higher accuracy than fixed-size embeddings.
  • 61% fewer parameters are achieved when adaptive input representations share parameters with an adaptive softmax output.
  • Adaptive word representations outperform strong character-based models while training more than twice as fast.
  • 23.02 perplexity is achieved on BILLION WORD, nearly 5 perplexity lower than the next best previously published result.

2 RELATED WORK

Prior work reduces language-modeling cost through adaptive output factorization and explores word, character, and sub-word units. The paper directly compares these alternatives and extends variable-capacity modeling to inputs.

  • Adaptive softmax assigns more output parameters to frequent words and fewer parameters to rare words to reduce computational burden.
  • Variable-sized input embeddings are presented as enabling weight sharing with an adaptive softmax output layer.
  • Earlier work evaluated character-based and word-based factorizations but did not directly compare them to each other.
  • This paper compares word-based, character-based, and sub-word input vocabularies for both input and output layers.

3 ADAPTIVE INPUT REPRESENTATIONS

Adaptive input representations partition the frequency-ordered vocabulary into clusters with progressively smaller embedding capacities. Cluster-specific embeddings are projected to a common dimension, and compatible output layers can share weights.

  • The frequency-ordered vocabulary is partitioned into disjoint clusters, with the most frequent words in the head and the least frequent words in the tail.
  • Embedding capacity decreases by a factor of k across clusters, so words in V1 have dimension d while words in Vn have dimension d/k^(n−1).The typical setting is k = 4.
  • Separate cluster embedding tables are looked up, projected to dimension d, and concatenated in the original word order.
  • An adaptive input layer can share word embeddings and projections with an adaptive softmax using the same vocabulary partition, d, and k.
  • The head requires special handling because adaptive softmax adds n−1 cluster embeddings that are not shared with the input.
  • On BILLION WORD, the authors share only word embeddings because sharing projections performed better on WIKITEXT-103 but not on BILLION WORD validation data.

4 EXPERIMENTAL SETUP

Experiments use decoder-only self-attentional models to compare word, character, and sub-word input/output factorizations on BILLION WORD and WIKITEXT-103. The setup standardizes architecture and dataset-specific training conditions across models.

  • 4.1 MODEL: The decoder-only architecture stacks 16 blocks, each combining 16-head self-attention, a feed-forward module, residual connections, and pre-normalization.
  • 4.1 MODEL: All models trained on the same dataset use identical hyperparameters to enable like-for-like comparison.
  • 4.2 DATASETS: BILLION WORD contains 768M tokens and about 800K word types, while WIKITEXT-103 contains about 100M tokens and around 260K types.
  • 4.3 BATCHING: BILLION WORD is batched by sentence, whereas WIKITEXT-103 is partitioned into contiguous token blocks while evaluation requires complete sentences.
  • 4.3 BATCHING: Respecting document boundaries during batching may improve results and is left for future work.
  • 4.4 INPUT AND OUTPUT LAYER HYPERPARAMETERS: Character inputs use convolutional filters, max pooling, highway layers, and a projection to represent each word from its characters.
  • 4.4 INPUT AND OUTPUT LAYER HYPERPARAMETERS: Adaptive word inputs and adaptive softmax use head dimension d = 1024 and reduce subsequent cluster dimensions by k = 4.
  • 4.4 INPUT AND OUTPUT LAYER HYPERPARAMETERS: BPE models learn 32K byte-pair codes and evaluate word-level perplexity by multiplying the probabilities of a word's sub-word units.

5 EXPERIMENTS AND RESULTS

Experiments compare word, character, and sub-word factorizations under a shared self-attentional architecture. Adaptive inputs achieve strong accuracy and speed, including 18.7 perplexity on WIKITEXT-103 and 23.02 on BILLION WORD, while performance varies with word frequency and context.

  • 5.1 MAIN RESULTS: 23.02 perplexity matches an ensemble of over ten models on BILLION WORD, while the large model reaches 24.14 perplexity.
  • 5.2 COMPARISON OF INPUT AND OUTPUT LAYER FACTORIZATIONS: ADP-T achieves the highest accuracy on WIKITEXT-103 at the same speed as BPE, while CNN is 1 perplexity worse and takes well over twice as long to train.Fixed word embeddings perform least well; adaptive softmax improves over standard softmax and greatly speeds training.
  • 5.2 COMPARISON OF INPUT AND OUTPUT LAYER FACTORIZATIONS: ADP-T is 34% faster than ADP on BILLION WORD, and adaptive inputs perform equally well compared with other factorizations.The speed difference follows from having fewer parameters to update in the tied model.
  • 5.3 ANALYSIS: ADP-T performs best across WIKITEXT-103 word-frequency ranges, while CNN performs well with rare words in context but not across all bins.On BILLION WORD, CNN performs best on very rare words but ADP outperforms it in other settings; BPE models perform poorly on rare words.
  • 5.3 ANALYSIS: Additional dropout in adaptive-softmax tail projections helps WIKITEXT-103 but is not helpful on larger BILLION WORD models.A standard softmax over 800K words was not tractable for comparison, and tuning dropout separately by tail band remains future work.

6 CONCLUSION

The paper concludes that adaptive input embeddings provide variable-capacity word representations, reducing parameters while improving accuracy and training speed through adaptive input-output sharing.

  • Sharing adaptive input embeddings with adaptive softmax further reduces parameters and improves training speed.
  • Adaptive input models train faster than character-input CNNs while achieving higher accuracy and new state-of-the-art results on WIKITEXT-103 and BILLION WORD.

A ADDITIONAL EXPERIMENTS ON WIKITEXT-103

Additional WIKITEXT-103 experiments examine fixed-size embedding capacity, weight sharing, and band configurations for adaptive inputs and outputs.

  • Reducing the capacity of fixed-size word input embeddings is beneficial on WIKITEXT-103.
  • Sharing the head projection performs less well than not sharing it.
  • Table 7 reports validation perplexity for the paper’s WIKITEXT-103 models.
  • Table 8 evaluates validation perplexity for tied adaptive inputs and outputs across different band sizes.
  • The adaptive bands use dimensions 1024, 256, 64, and optionally 16 for successive frequency bands.

B ANALYSIS

The analysis examines next-word loss as a function of the previous word’s frequency on BILLION WORD.

  • The appendix extends the analysis by binning test loss according to the frequency of the previous word.
  • The analyzed quantity is loss on the next word.
  • Figure 5 presents model loss across previous-word frequency bins on BILLION WORD.
Loading 1809.10853v3…