PaperlayerView exact revision on arXiv

Source-linked AI summary

Compressing Word Embeddings via Deep Compositional Code Learning

Raphael Shu, Hideki Nakayama

arXiv:1711.01068v2cs.CL

TL;DR

Word embeddings can account for a large share of NLP model parameters and storage, creating pressure to compress them without sacrificing performance. This paper builds embeddings by composing code-selected basis vectors and learns the discrete codes with Gumbel-softmax. It reports loss-free compression across sentiment analysis and machine translation, with rates of 98% and 94%–99%, respectively.

  • Problem

    Word embeddings can dominate model parameters and storage, creating a need to compress them without hurting model performance.

    Source: §1 · p. 1

  • Method

    The method composes each word’s embedding from code-selected basis vectors and learns discrete codes end to end with Gumbel-softmax.

    Source: §1 · pp. 1–2

  • Results

    98% embedding-layer compression in sentiment analysis and 94%–99% in machine translation were reported without performance loss.

    Source: §1 · p. 3, §7 · p. 9

  • Takeaways & Limitations

    Similar words’ learned codes were close in Hamming space, and the dense basis matrix could potentially be further compressed with pruning.

    Source: §7 · p. 9

    Some codewords may be unused or “dead” after training, with none of the words assigned to them.

    Source: §6.2 · p. 9

Abstract

from arXiv · show

Natural language processing (NLP) models often require a massive number of parameters for word embeddings, resulting in a large storage or memory footprint. Deploying neural NLP models to mobile devices requires compressing the word embeddings without any significant sacrifices in performance. For this purpose, we propose to construct the embeddings with few basis vectors. For each word, the composition of basis vectors is determined by a hash code. To maximize the compression rate, we adopt the multi-codebook quantization approach instead of binary coding scheme. Each code is composed of multiple discrete numbers, such as (3, 2, 1, 8), where the value of each component is limited to a fixed range. We propose to directly learn the discrete codes in an end-to-end neural network by applying the Gumbel-softmax trick. Experiments show the compression rate achieves 98% in a sentiment analysis task and 94% ~ 99% in machine translation tasks without performance loss. In both tasks, the proposed method can improve the model performance by slightly lowering the compression rate. Compared to other approaches such as character-level segmentation, the proposed method is language-independent and does not require modifications to the network architecture.

1 INTRODUCTION

Word embeddings can dominate NLP model parameters and storage, motivating a representation that shares information among similar words. The paper composes embeddings from code-selected basis vectors and learns the discrete codes end to end.

1 INTRODUCTION

  • In a simple sentiment model, embeddings account for 98.8% of parameters; sharing information among semantically similar words is proposed to reduce that burden.The introduction gives “dog” and “dogs” as an example of similar words that still need some independent capacity for syntactic differences.Source: §1 · p. 1
  • Each word receives a compact code selecting codewords from multiple codebooks, whose vectors are summed to form its embedding.This reduces the embedding matrix to M × K vectors, usually fewer than the vocabulary size.Source: §1 · p. 2
  • The method learns discrete codes end to end with the Gumbel-softmax trick, allowing arbitrary differentiable losses such as cosine similarity.Source: §1 · p. 2
  • In experiments, over 98% of embedding parameters can be eliminated in sentiment analysis without affecting performance; in machine translation tasks, loss-free compression reached 94%–99%.Source: §1 · p. 3

2 RELATED WORK

The paper situates compositional coding among neural-network compression, hashing, and quantization methods. It distinguishes its goal of compact word embeddings and simultaneous code learning from prior approaches aimed at other tasks or using different optimization procedures.

2 RELATED WORK

  • Prior network-compression methods include low-precision computation, quantization, pruning, and distillation, while pruning has also been applied to neural machine translation.The paper compares its approach with iterative pruning.Source: §2 · p. 3
  • Product and additive quantization use multiple codebooks for source coding, but prior compositional-code work mainly targeted efficient similarity search.This paper instead uses such codes to reduce word-embedding parameters and avoid performance loss.Source: §2 · p. 3
  • Unlike additive quantization’s alternating codebook and code assignment optimization with beam search, this work learns both simultaneously in an end-to-end neural network.Source: §2 · p. 3
  • For producing compositional rather than binary codes, the paper encourages discreteness with the Gumbel-Softmax trick.Source: §2 · p. 3
  • Character-based approaches can require architecture changes and may be difficult for languages such as Japanese and Chinese with large character vocabularies.The paper says its approach does not share these limitations.Source: §2 · pp. 3–4

3 ADVANTAGE OF COMPOSITIONAL CODES

Compositional codes represent each word with M discrete selections from codebooks, reducing code storage relative to binary codes for a given number of basis vectors. The approach also requires fewer basis-vector additions to construct an embedding than binary coding.

3 ADVANTAGE OF COMPOSITIONAL CODES

  • A code has M components, each in [1, K], and requires M log2 K bits; K is chosen as a multiple of 2 for efficient storage.The code length is proportional to vocabulary size, while the number of basis vectors is M × K regardless of vocabulary size.Source: §3 · p. 4
  • For 512 basis vectors, a 32 × 16 compositional scheme uses 128-bit codes, versus 256 bits for binary coding.Source: §3 · p. 4
  • Compositional embeddings sum M basis vectors, versus N/2 for binary codes; the paper describes this as a better balance of shorter codes and lower computation.Table 1 summarizes the code lengths and computation costs for the approaches.Source: §3 · p. 4, Table 1 · p. 4

4 CODE LEARNING WITH GUMBEL-SOFTMAX

The section formulates embedding reconstruction as approximate matrix factorization over codebook basis matrices, then uses Gumbel-softmax to learn discrete one-hot code choices end-to-end.

4 CODE LEARNING WITH GUMBEL-SOFTMAX

  • Each word’s embedding is reconstructed from codebook basis matrices selected by the word’s one-hot code components.The basis matrices form the approximate factorization of the original embedding matrix.Source: §4 · pp. 4–5
  • Gumbel-softmax parameterizes discrete one-hot code choices while the dictionaries and codes are optimized to minimize reconstruction loss.The trick makes discrete code learning part of the end-to-end objective.Source: §4 · p. 5
  • The coding learner is an autoencoder with a Gumbel-softmax middle layer that reconstructs embeddings and computes reconstruction loss.Its hidden layer has size MK/2, and experiments use a fixed temperature of τ = 1.Source: §4 · p. 5
  • After training, argmax on the one-hot vectors yields each word’s code, and the rows of A provide the composing basis vectors.Source: §4 · p. 5
  • For machine translation, codes are learned from a trained baseline embedding matrix, then the other model parameters are retrained because reconstructed embeddings differ.The coding learner is not jointly trained with the translation model because its coding layer takes far more iterations to converge to one-hot vectors.Source: §4 · p. 5

5 EXPERIMENTS

Experiments evaluate maximum loss-free word-embedding compression on sentiment analysis and machine translation, comparing model performance and embedding-layer size with baseline and iterative-pruning models.

5 EXPERIMENTS

  • The experiments assess maximum loss-free embedding compression on sentiment analysis and machine translation, comparing performance and embedding-layer size with baseline and iterative-pruning models.Reported sizes exclude other neural-network components.Source: §5 · pp. 5–6

5.1 CODE LEARNING

  • Code learning uses public GloVe vectors for sentiment analysis and pretrained task-specific embeddings for machine translation, with codes learned by minimizing sampled-embedding reconstruction loss.The code-learning network is trained for 200K iterations with batches of 128.Source: §5.1 · p. 6

5.2 SENTIMENT ANALYSIS

  • Increasing M or K lowers reconstruction loss, but larger M lengthens codes and increases embedding-layer size, requiring a trade-off.Table 2 reports reconstruction loss and embedding-layer size across settings.Source: §5.2 · p. 6, Table 2 · p. 7
  • The baseline using 75K GloVe embeddings reaches accuracy 87.18 with 78 MB of storage, while high-rate iterative pruning degrades accuracy.Table 3 compares model performance and embedding-layer size on IMDB.Source: §5.2 · pp. 6–7, Table 3 · p. 7

5.2 SENTIMENT ANALYSIS; 5.3 MACHINE TRANSLATION

  • 98.4% compression on IMDB, 94% loss-free compression on IWSLT14, and 99% on ASPEC are reported for compositional coding.The IMDB maximum loss-free rate uses a 16 × 32 coding scheme.Source: Table 3 · p. 7, §5.2 · p. 7, Table 4 · p. 8

5.3 MACHINE TRANSLATION

  • For machine translation, the authors train a baseline NMT model, learn codes from its embeddings, then retrain with reconstructed embeddings fixed and other parameters initialized randomly.Source: §5.3 · p. 8
  • At 90% connection pruning, compression is loss-free at 92% on ASPEC, but IWSLT14 shows modest performance loss.Table 4 summarizes the machine-translation performance and embedding-layer-size trade-off.Source: §5.3 · p. 8, Table 4 · p. 8

6 QUALITATIVE ANALYSIS

Semantically similar words receive similar learned codes, and subcode assignments remain used across the coding schemes examined.

6.1 EXAMPLES OF LEARNED CODES

  • Examples from sentiment analysis show that words with similar meanings receive similar codes, a sharing pattern that helps reduce embedding redundancy.The examples use 300-dimensional uncased GloVe embeddings.Source: §6.1 · p. 8, Table 5 · p. 9

6.2 ANALYSIS OF CODE EFFICIENCY

  • In 8 × 8 coding, 31% of words use subcode “0” in the first component, versus 5% using subcode “1”; even the least-used codeword serves about 1000 words.Larger coding schemes show more balanced assignments.Source: §6.2 · p. 9, Figure 3 · p. 9
  • Although learned codewords could go unused, the reported analysis finds even the least-used codeword is assigned to about 1000 words.The analysis considers the number of words containing each subcode in each component.Source: §6.2 · p. 9

7 CONCLUSION

The conclusion presents compositional coding as a way to reduce word-embedding parameters by representing words with basis vectors selected by learned codes. It reports high loss-free compression and notes potential further compression of the dense basis matrix.

7 CONCLUSION

  • 98% of embedding parameters were removed in IMDB sentiment analysis and 94% ∼99% in machine translation without affecting performance.The reported compression rates are loss-free.Source: §7 · p. 9
  • Compositional coding uses a small set of basis vectors selected by each word’s hash code to represent embeddings.The conclusion describes the approach as eliminating redundancy in representing similar words independently.Source: §7 · p. 9
  • The dense basis matrix could be further compressed with pruning, and compositional coding’s advantage is greater when hash codes dominate embedding-layer size.The conclusion links high loss-free compression to semantic inter-similarity among words.Source: §7 · p. 9

A APPENDIX: SHARED CODES

The appendix reports that small code decompositions can assign one code to multiple words, with shared-code examples from sentiment analysis.

A APPENDIX: SHARED CODES

Loading 1711.01068v2…