Source-linked AI summary
TensorGPT: Efficient Compression of Large Language Models based on Tensor-Train Decomposition
Mingxue Xu, Yao Lei Xu, Danilo P. Mandic
TL;DR
Embedding layers create substantial storage and memory demands for low-end LLM deployments, while prior low-rank compression approaches require extra training. TensorGPT applies Tensor-Train Decomposition to individual token embeddings; at 0.5×–2.0× embedding compression, it achieves comparable task performance, with larger models showing better accuracy-compression trade-offs.
Problem
How much can LLM embedding layers be compressed without extra training or sacrificing language-task performance, given their substantial parameter share and the resource limits of low-end devices?
Source: §1 · pp. 1–2Method
TensorGPT reshapes each pretrained token embedding into a high-dimensional tensor and uses Tensor-Train Decomposition to represent it as a lower-rank Matrix Product State.
Source: §1 · p. 2, §3.1 · p. 4Results
At compression ratios of 0.5×–2.0×, TensorGPT achieves comparable language-task performance, and compressed models can even outperform originals in language modelling and sentiment classification.
Source: §1 · p. 2Takeaways & Limitations
The results indicate that larger-scale models have better accuracy-compression trade-offs.
Source: §1 · p. 2The study focuses on GPT-series models; extending TensorGPT to other model families such as Llama is left for future work.
Source: §6 · p. 9
Abstract
from arXiv · showhide
High-dimensional token embeddings underpin Large Language Models (LLMs), as they can capture subtle semantic information and significantly enhance the modelling of complex language patterns. However, this high dimensionality also introduces considerable model parameters and prohibitively high model storage and memory requirements, which is particularly unaffordable for low-end devices. Targeting no extra training data and insufficient computation cases, we propose a training-free model compression approach based on the Tensor-Train Decomposition (TTD), whereby each pre-trained token embedding is converted into a lower-dimensional Matrix Product State (MPS). We then comprehensively investigate the low-rank structures extracted by this approach, in terms of the compression ratio, the language task performance, and latency on a typical low-end device (i.e. Raspberry Pi). Taking GPT family models (i.e. GPT-2 and CerebrasGPT) as case studies, our approach theoretically results in $46.89\%$ fewer parameters of the entire model, with a compression ratio $39.38\times$ - $65.64\times$ for the embedding layers. With different hyperparameter choices, the model compressed with our approach can achieve a comparable language task performance to the original model with around $2.0\times$ embedding layer compression. This empirically proves the existence of low-rank structure in GPT family models, and demonstrates that about half of the parameters in the embedding layers are redundant.
1 INTRODUCTION
TensorGPT addresses the challenge of compressing LLM embedding layers without extra training by applying Tensor-Train Decomposition to individual token embeddings. The introduction reports comparable task performance at embedding compression ratios of 0.5×–2.0× and stronger accuracy-compression trade-offs for larger models.
1 INTRODUCTION
- Embedding layers account for a large share of sub-billion LLM parameters, reaching 48.08% in smaller models.Around half of common sub-billion LLMs devote one-third of their total parameters to embeddings.Source: §1 · p. 1, Figure 1 · p. 2
- Prior low-rank compression approaches require extra training, which can demand unavailable data and computation or discard pretrained knowledge when training from scratch.The introduction frames the central question as how much embeddings can be compressed without extra training or sacrificing task performance.Source: §1 · pp. 1–2
- When tensor order rises, DistilGPT2 performance on 1BW tends to improve before declining after order 3; larger models show better compression trade-offs.The trade-off compares perplexity with embedding-layer compression ratio; lower ratios are better.Source: Figure 1 · p. 2
- TensorGPT reshapes each token embedding into a high-dimensional tensor and uses Tensor-Train Decomposition to represent it as a lower-rank Matrix Product State.Processing embeddings individually is intended to preserve each token's information and accommodate changing vocabularies.Source: §1 · p. 2
- At compression ratios of 0.5×–2.0×, TensorGPT achieves comparable language-task performance, and compressed models can even outperform originals in language modelling and sentiment classification.The introduction also reports better accuracy-compression trade-offs for larger-scale models.Source: §1 · p. 2
- The work adapts Tensor-Train Decomposition for non-parallel embedding operations and measures compressed-model latency on Raspberry Pi and higher-end servers.Its evaluation covers compression, task performance, hyperparameters, model size, and edge-device execution.Source: §1 · pp. 2–3
2 PRELIMINARIES
The preliminaries define tensors and tensor operations, then introduce Tensor-Train Decomposition as a way to represent a high-order tensor using smaller tensor cores. They also outline the TT-SVD procedure used for single-token embedding compression.
2 PRELIMINARIES
- An order-N tensor is a multi-way array with N modes, each having a specified dimension; vectors and matrices are order-1 and order-2 cases.The section also describes reshaping a vector into an order-N tensor and vectorizing a tensor back into a vector.Source: §2 · p. 3
- Tensor contraction combines two tensors by summing over matching modes, producing a tensor over their remaining modes.The contracted modes must have equal dimensions.Source: §2 · p. 3
- For single-token embedding compression, Algorithm 1 reshapes the embedding, applies truncated SVD, reshapes each U factor into a TT core, and returns the cores.The truncation step is constrained by an error bound δ.Source: §2 · p. 4
- Tensor-Train Decomposition applies TT-SVD to an order-N tensor, producing N smaller tensor cores whose dimensions are set by the mode sizes and TT ranks.The boundary TT ranks are r0 = rN = 1.Source: §2 · p. 4
3 METHODOLOGY
TensorGPT compresses embeddings by tensorizing and decomposing each token vector into MPS cores, then reconstructs vectors for transformer inference. Vocabulary processing and edge inference are split across devices.
3 METHODOLOGY
- The whole vocabulary is processed on higher-end servers, while inference and vocabulary updates happen on lower-end edge devices.Source: §3 · p. 4
3.1 INDIVIDUAL EMBEDDING VECTOR COMPRESSION
- TensorGPT decomposes each embedding vector separately rather than decomposing the whole embedding matrix.Source: §3.1 · p. 4
- Each token vector is reshaped into an order-N tensor and stored as MPS cores instead of as the full vector.Source: §3.1 · p. 4
- TT SVD decomposes each tensor mode using matrix-level SVD, and its complexity is derived from SVD and its variants.Source: §3.1 · p. 4
- Table 1 summarizes computation and memory complexity for compression and inference; its caption identifies r=1 and I=2 as the highest-compression setting.Source: Table 1 · p. 5
3.2 LANGUAGE MODEL INFERENCE PROCESS WITH THE COMPRESSED EMBEDDINGS
- Compressed embeddings are reconstructed to their original dimension before entering the transformer; because V≫l, memory can still be reduced when embeddings comprise a significant share of model parameters.Source: §3.2 · p. 5
4 EMPIRICAL EVALUATION
The evaluation tests GPT-family models on language modelling and sentiment classification, measuring compression alongside task performance and latency. Significant language-modelling performance loss tends to appear above 2.0× compression, while larger models are more robust to increasing compression and Raspberry Pi latency remains within the reported acceptable range.
4.1 EXPERIMENTAL SETUP
- The evaluation covers GPT-family models on language modelling datasets WikiText2, WikiText103, and 1BW, and sentiment classification on IMDB.Inference latency was measured on Raspberry Pi 5; the main experiments used an RTX A6000 GPU workstation and AMD Ryzen Threadripper PRO 5955WX CPU.Source: §4.1 · p. 5
4.2 EVALUATION METRICS
- Compression ratio is evaluated for model blocks and for token and position embedding layers, alongside perplexity and classification metrics.The classification metrics are accuracy, precision, recall, and F1-score; the language-modelling metrics include perplexity and its logarithmic form.Source: §4.2 · p. 6
4.3 EXPERIMENTAL RESULTS
- Significant language-modelling performance loss tends to appear when compression exceeds 2.0×, and Tensor-Train Decomposition outperforms Tucker decomposition in perplexity.For GPT-2, the best high-compression cases with negligible accuracy drop occur at tensor order N = 3.Source: §4.3 · p. 6, Figure 3 · p. 7
- Larger models are more robust to increasing compression than smaller models in both tasks, especially below 1.0× compression.In sentiment classification, larger models tend to outperform the original model in precision and F1-score, while smaller models tend to worsen as compression increases.Source: §4.3 · pp. 6–7
- A 50-token input text incurs no more than 0.3 seconds of induced latency on Raspberry Pi, which the authors consider acceptable for edge applications.Individual-token embedding compression typically differs by no more than 2 milliseconds between high-end servers and Raspberry Pi.Source: §4.3 · pp. 7–8, Table 2 · p. 8
5 RELATED WORK
Prior compression work divides into matrix- and tensor-based factorization, while tensor-network structure search includes optimization and learning-based approaches. The cited methods differ in their target layers and training requirements.
5.1 MATRIX OR TENSOR FACTORIZATION FOR LANGUAGE MODEL COMPRESSION
- Matrix-based compression includes SVD, weighted factorization, distillation, pruning, and block-wise Kronecker or data-driven decomposition.These methods decompose weight matrices to reduce model size.Source: §5.1 · p. 8
- Block-diagonal methods target FLOPs in feed-forward layers, whereas this work targets embedding parameters and lower-end devices rather than GPU parallelization.The paper reports 0.330–0.364 ms/token additional forwarding latency on Raspberry Pi 5 versus 0.463 ms/token on GPU in its setting.Source: §5.1 · pp. 8–9
- Prior tensor-decomposition approaches described here require extra training, either during GPT fine-tuning or by training the structured model from scratch.The cited examples use Kronecker decomposition or tensor-train structured models.Source: §5.1 · p. 9
5.2 TENSOR NETWORK AND TENSOR NETWORK STRUCTURE SEARCH
- Tensor-network search includes local optimization studied on image tasks and a learning-based approach that trains a meta-model to learn low-rank parameter patterns.The paper notes that the local-search work did not consider language modelling.Source: §5.2 · p. 9
6 CONCLUSION AND FUTURE WORK
TensorGPT uses Tensor-Train Decomposition to identify low-rank embedding structure without extra training, and the tested larger models maintained or improved language-task performance. The work also identifies deployment and model-family boundaries for its conclusions.
6 CONCLUSION AND FUTURE WORK
- Larger tested models maintained or improved language-task performance while reducing model size, with embedding-layer compression ratios of 0.5× to 2.0× reported without performance loss.The tested models include GPT-2-M, GPT-2-L, GPT-2-XL, CerebrasGPT-590M, and CerebrasGPT-1.3B; on Raspberry Pi, compression latency was typically no more than 2 ms per token.Source: §6 · p. 9
- Tensorized embeddings do not natively compile with hidden layers, motivating tensorized hidden layers; CPU tensor operations may also require acceleration.The paper limits its “half parameters redundancy” conclusion to GPT-series models and leaves extensions such as Llama-family models for future work.Source: §6 · p. 9
A PRELIMINARIES
The preliminaries introduce notation and describe TT-SVD as the algorithm for producing core tensors that approximate an input data tensor.
A PRELIMINARIES
- The paper’s notation is summarized in Table 4.The preliminaries introduce the notation before the methodology.Source: A PRELIMINARIES · p. 13, Table 4 · p. 13
- TT-SVD takes a data tensor and approximation accuracy as input and outputs core tensors approximating the tensor.The algorithm initializes the cores and R0 = 1, computes δ-truncated SVD steps, and returns the cores.Source: A PRELIMINARIES · p. 13
- The listed TT-SVD steps initialize cores, perform a δ-truncated SVD, and return the resulting cores.The displayed algorithm also includes an SVT step.Source: A PRELIMINARIES · p. 13
A.1 TENSORS AND TENSOR OPERATIONS
The preliminaries define tensor shapes and entries, then describe tensorization, matricization, and contraction as operations used to express tensor computations.
A.1 TENSORS AND TENSOR OPERATIONS
- An order-N tensor is a multidimensional array with N modes, and vectors and matrices are order-1 and order-2 tensors, respectively.Scalars are order-0 tensors.Source: §A.1 · pp. 13–14
- Tensor entries are indexed across modes; fixing all but one index gives a fiber, while fixing all but two gives a slice..Source: §A.1 · p. 14
- Tensorization folds a vector into a higher-order tensor, while mode-n matricization maps tensor elements into a matrix by stacking mode-n fibers as columns.The passage specifies linear indexing for overlined subscripts.Source: §A.1 · p. 14
- Tensor contraction sums over matching modes; for order-2 tensors it corresponds to matrix multiplication, and for a matrix and vector it corresponds to matrix-by-vector multiplication.The contracted modes must have equal sizes.Source: §A.1 · p. 14
- The preliminaries point to Algorithm 2 for the Tensor-Train Singular Value Decomposition..Source: §A.1 · p. 14
B PROOF OF THE HIGHEST COMPRESSION RATIO IN TAB. 1
The proof identifies the tensor size that maximizes compression in the simplest case. For GPT-series models up to GPT-3, it finds the minimum at I = 2, yielding a best selection of all tensor dimensions equal to 2.
B PROOF OF THE HIGHEST COMPRESSION RATIO IN TAB. 1
- In the simplest case, the proof assumes equal tensor dimensions and equal TT ranks, with N determined by d = I^N.These assumptions reduce the hyperparameter selection to the equal-size, equal-rank setting.Source: B PROOF OF THE HIGHEST COMPRESSION… · p. 15
- For GPT-series models through GPT-3, the relevant range is 2 ≤ I ≤ N ≤ 13, and the objective is monotonic with its minimum at I = 2.The bound uses GPT-3's largest token embedding size, 12,888.Source: B PROOF OF THE HIGHEST COMPRESSION… · p. 15
- I1 = I2 = · · · = IN = 2 and N = ⌊log2 d⌋ give the best hyperparameter selection for the simplest case.The proof concludes this after finding that the objective is minimized at I = 2.Source: B PROOF OF THE HIGHEST COMPRESSION… · p. 15