Source-linked AI summary
word2ket: Space-efficient Word Embeddings inspired by Quantum Entanglement
Aliakbar Panahi, Seyran Saeedi, Tom Arodz
TL;DR
Storing a full word-embedding matrix can require substantial memory as vocabularies and embedding dimensions grow. The paper proposes word2ket and word2ketXS, quantum-computing-inspired methods for compact embedding storage. Across three NLP tasks, the methods achieve high space savings at little accuracy cost, including over 100-fold space reduction for about a 0.5-point Rouge-score drop in one evaluation.
Problem
A d × p word-embedding matrix can occupy a substantial, often dominating, share of model parameters as vocabulary size and embedding dimensions grow.
Source: §1 · p. 1Method
The paper proposes word2ket, which represents each word independently, and word2ketXS, which represents all embeddings jointly for greater storage efficiency but more complex processing.
Source: §1.1 · p. 2Results
Across three NLP tasks, the methods achieved high space savings at little accuracy cost; in one evaluation, over 100-fold space reduction incurred about a 0.5-point drop in Rouge scores.
Source: §1.1 · p. 2, §4 · p. 6Takeaways & Limitations
The reported memory reductions concern the word-embedding component, rather than all parameters in the downstream models.
Source: §4 · p. 7The method does not compress other model parameters, including intermediate-layer weights and the word-probability matrix before the final softmax.
Source: §4 · p. 7
Abstract
from arXiv · showhide
Deep learning natural language processing models often use vector word embeddings, such as word2vec or GloVe, to represent words. A discrete sequence of words can be much more easily integrated with downstream neural layers if it is represented as a sequence of continuous vectors. Also, semantic relationships between words, learned from a text corpus, can be encoded in the relative configurations of the embedding vectors. However, storing and accessing embedding vectors for all words in a dictionary requires large amount of space, and may stain systems with limited GPU memory. Here, we used approaches inspired by quantum computing to propose two related methods, {\em word2ket} and {\em word2ketXS}, for storing word embedding matrix during training and inference in a highly efficient way. Our approach achieves a hundred-fold or more reduction in the space required to store the embeddings with almost no relative drop in accuracy in practical natural language processing tasks.
1 INTRODUCTION
Word embeddings make language inputs suitable for neural networks and encode semantic relationships, but the full vocabulary-by-dimension matrix can dominate model parameters. The paper introduces word2ket and word2ketXS to store embeddings more efficiently, using approaches inspired by quantum computing.
1 INTRODUCTION
- For vocabulary size d and embedding dimension p, the d × p matrix can become a substantial, often dominating, part of a model’s parameter space.The passage gives vocabulary sizes of 10^5 or 10^6 and embedding dimensions from 300 to 1024 as examples.Source: §1 · p. 1
- Classical storage can approximate quantum-register representations in O(log m) space, but cannot represent every m-dimensional vector available to the quantum register.The paper says this loss of representational power does not significantly affect the NLP algorithms it studies.Source: §1 · p. 2
1.1 OUR CONTRIBUTION
- word2ket represents each word independently for more efficient processing, while word2ketXS jointly represents all word embeddings for higher storage efficiency at the cost of more complex processing.Empirical evidence from three NLP tasks shows high space savings at little downstream-accuracy cost.Source: §1.1 · p. 2
2 FROM TENSOR PRODUCT SPACES TO word2ket EMBEDDINGS
Tensor products combine component spaces into a larger space whose vectors can include sums that are not expressible as a single product. word2ket uses such entangled tensors to represent embeddings compactly, with procedures for computing selected embeddings and training the representation.
2.1 TENSOR PRODUCT SPACE
- With orthonormal bases for the component spaces, their pairwise tensor products form an orthonormal basis, and the tensor-product space has dimension equal to the product of the component dimensions.Tensor products can be applied repeatedly, with arbitrary bracketing because the operation is associative.Source: §2.1 · pp. 2–3
2.2 ENTANGLED TENSORS
- A tensor-product space contains linear combinations that cannot be expressed as one product of component vectors; tensors requiring more than one simple tensor are called entangled.The paper notes that the maximum tensor rank in order greater than two is not generally known.Source: §2.2 · p. 3
2.3 THE word2ket EMBEDDINGS
- word2ket represents each p-dimensional word embedding as a rank-r, order-n entangled tensor built from vectors in R^q, where p = q^n and storage is O(rq log(p/q)).The paper uses q ≥ 4 because a tensor product of two R^2 vectors takes the same space as an R^4 vector, while not every R^4 vector has that rank-one form.Source: §2.3 · pp. 3–4
- For downstream computations limited to embedding inner products, word2ket can avoid explicitly calculating the q^n-dimensional vectors and use O(1) additional space for the inner-product calculation.The inner product of tensor products is the product of the component inner products.Source: §2.1 · p. 2, §2.3 · p. 4
- For a batch of b words, storing the reconstructed embeddings and compressed representation requires O(bp + rq log(p/q)) space instead of O(dp); a balanced tensor-product tree enables parallel reconstruction with sequential depth O(log n).Reconstructing the batch takes O(brpn) arithmetic operations.Source: §2.3 · p. 4
- Because gradients through the tensor-product representation involve products of underlying vectors that may yield a high gradient Lipschitz constant, the method applies LayerNorm at each node of its balanced tree.The representation is differentiable and is described as a sequence of O(log n) linear layers with linear activations.Source: §2.3 · p. 4
3 LINEAR OPERATORS IN TENSOR PRODUCT SPACES AND word2ketXS
This section establishes how tensor-product operators act on product spaces, then applies that structure to compress an entire word-embedding matrix in word2ketXS. The factorization supports compact storage and lazy row reconstruction.
3.1 LINEAR OPERATORS IN TENSOR PRODUCT SPACES
- Tensor products of linear operators map V⊗W into U⊗Y, and finite-dimensional A⊗B is represented by a block matrix of B-scaled blocks.Source: §3.1 · p. 4
3.2 THE word2ketXS EMBEDDINGS
- The word-embedding operator F maps each word's one-hot vector to its corresponding embedding vector, with the embedding matrix transpose representing F.Source: §3.2 · p. 5
- word2ketXS represents the full embedding matrix as a tensor product of linear operators, compressing across both its rows and columns.Its storage is O(rqt max(log p/q, log d/t)).Source: §3.2 · p. 5
- Lazy tensors avoid reconstructing the full embedding matrix when downstream computation needs only a few rows, which can be reconstructed efficiently from underlying rows.Source: §3.2 · p. 5
- Figure 1 contrasts a single 256-dimensional word2ket vector using a rank-5, order-4 tensor with a word2ketXS example using twenty 3×2 trainable matrices.The word2ketXS example represents an 81-word, 16-dimensional embedding matrix.Source: Figure 1 · p. 5, §3.2 · p. 5
4 EXPERIMENTAL EVALUATION OF word2ket AND word2ketXS IN DOWNSTREAM NLP TASKS
Across text summarization, translation, and question answering, the experiments evaluate space-efficient embeddings against regular embeddings. word2ketXS substantially reduces embedding parameters with modest task-score drops, while adding training time and leaving other model parameters uncompressed.
4 EXPERIMENTAL EVALUATION OF word2ket AND word2ketXS IN DOWNSTREAM NLP TASKS
- The evaluation compares space-efficient and regular embeddings across text summarization, machine translation, and question answering.The tasks use GIGAWORD with a sequence-to-sequence model, IWSLT2014 German-to-English translation with the same sequence-to-sequence model, and SQuAD with DrQA.Source: §4 · pp. 6–7
- About 100-fold fewer embedding parameters cost roughly 0.5 Rouge points, while word2ketXS also reaches 34,000-fold reduction with scores matching word2ket on GIGAWORD.The reported comparison also notes that word2ket itself achieves a 16-fold parameter reduction with about a 2-point Rouge-score drop.Source: §4 · p. 6, Table 1 · p. 6
- About a 1-point BLEU drop accompanies 100-fold parameter-space reduction on IWSLT2014 German-to-English translation.The reported BLEU drops are 0.5 and 1.5 points at lower and higher space-saving rates, respectively.Source: Table 2 · p. 6, §4 · p. 7
- A 0.5-point F1 drop accompanies 1000-fold parameter-space savings on SQuAD; order-4 word2ketXS saves almost 105-fold with under a 2-point, under-3% relative F1 drop.Training took 5.8 hours with regular embeddings, 7.4 hours with order-2 word2ketXS, and 9 hours with order-4 word2ketXS; training dynamics remained largely unchanged.Source: §4 · p. 7, Table 3 · p. 7, Figure 2 · p. 7
- Embedding compression reduces memory for the embedding component, but does not compress intermediate-layer weights or the pre-softmax word-probability matrix.During training, storing layer activations for backward gradient calculation also requires memory and can dominate the footprint.Source: §4 · pp. 7–8
4.1 RELATED WORK
- Related approaches include dictionary learning, clustering, bit encoding, quantization, pruning, sparsity, low precision, and matrix approximation; the authors state none match word2ketXS space-saving rates.The passage notes that bit-encoding methods are limited to at most 32-fold savings on 32-bit architectures, while parameter sharing and PCA have storage requirements limited by vocabulary size and embedding dimensionality.Source: §4.1 · p. 8