Source-linked AI summary
TRAWL: Tensor Reduced and Approximated Weights for Large Language Models
Yiran Luo, Het Patel, Yu Fu, Dawon Ahn, Jia Chen, Yue Dong, Evangelos E. Papalexakis
TL;DR
Large language models require substantial computational resources, and matrix-focused compression may miss broader structural patterns. TRAWL applies tensor decomposition across multiple weight matrices, with reported accuracy improvements of up to 16% over single-matrix approaches.
Problem
Large Transformer models require substantial computational resources and energy, while methods operating on individual matrices may miss broader structural patterns.
Source: §1 · pp. 1–2Method
TRAWL stacks multiple weight matrices into higher-order tensors and applies tensor decomposition to exploit model structure and denoise weights.
Source: §1 · p. 2Results
Up to 16% accuracy improvements over single matrix decomposition approaches were reported across RoBERTa and GPT-J on three datasets.
Source: §1 · p. 2Takeaways & Limitations
Layer-by-layer decomposition and selective layer stacking improve performance, with the most notable gains in terminal-layer fully connected layers.
Source: §5 · p. 10The study leaves practical memory and compute savings for inference across different devices for future investigation.
Source: §6 · p. 11
Abstract
from arXiv · showhide
Recent research has shown that pruning large-scale language models for inference is an effective approach to improving model efficiency, significantly reducing model weights with minimal impact on performance. Interestingly, pruning can sometimes even enhance accuracy by removing noise that accumulates during training, particularly through matrix decompositions. However, recent work has primarily focused on single matrix decompositions or lower precision techniques, which may fail to fully capture structural patterns. To address these limitations, we introduce TRAWL (Tensor Reduced and Approximated Weights for Large Language Models), a technique that applies tensor decomposition across multiple weight matrices to effectively denoise LLMs by capturing global structural patterns. Our experiments show that TRAWL improves model performance by up to 16% over baseline models on benchmark datasets, without requiring additional data, training, or fine-tuning.
1 Introduction
Large LLMs impose substantial training and deployment resource demands, while matrix-focused compression may miss broader structural patterns. TRAWL applies tensor decomposition across multiple weight matrices and reports accuracy gains up to 16% over single-matrix approaches.
1 Introduction
- Large Transformer models can capture complex patterns through scale, but their training and deployment require substantial computational resources and energy.Source: §1 · p. 1
- Prior post-training compression includes SVD-based LASER and low-precision CALDERA, but both operate on individual matrices and may miss broader structural patterns.Source: §1 · pp. 1–2
- TRAWL stacks multiple weight matrices into higher-order tensors and applies tensor decomposition to exploit model structure and reduce noise and weight rank.Source: §1 · p. 2
- Up to 16% accuracy improvements over single matrix decomposition approaches were reported across two models and three datasets, without additional training or fine-tuning.The evaluations used RoBERTa and GPT-J on BigBench WikiQA, BiosProfession, and HotpotQA.Source: §1 · p. 2
2 Related Work
Related work spans matrix and tensor compression, while research has also questioned whether Transformer weights themselves are low rank. TRAWL builds on these approaches by exploiting tensor structure in LLMs without additional training or fine-tuning.
2 Related Work
- LASER [15] reduces weight-matrix rank with SVD to remove noise, while CALDERA [13] combines a low-precision backbone with low-rank factors to reduce memory footprint.Source: §2 · p. 2
- A study reports low-rank Transformer features but not weights, challenging weight-based compression assumptions; it proposes few-shot unsupervised feature compression with minimal accuracy loss.Source: §2 · pp. 2–3
- TRAWL acknowledges that weight approximations need not be conventionally low rank, finding that some ranks below original weight dimensionality can improve performance.Source: §2 · p. 3
- TRAWL builds on matrix and tensor decomposition approaches by exploiting tensor structures in LLMs, with enhanced performance and no additional training or fine-tuning.Source: §2 · p. 3
3 Methodology
TRAWL applies tensor decomposition to stacked weight matrices, with formulations that operate within individual layers, across the full model, or on separate layer segments. The methodology considers CP and Tucker decompositions and treats rank as a hyperparameter.
3.1 Tensor Decomposition
- CP represents a tensor as a sum of rank-one tensors, while Tucker approximates it with a core tensor and factor matrices along each mode.The Tucker mode product multiplies tensor slices by the corresponding factor matrix.Source: §3.1 · p. 4
- The method fits decompositions by least squares and treats the number of rank-one components, R, as a hyperparameter whose optimal value is beyond this study's scope.The fitting uses TensorLy.Source: §3.1 · p. 4
3.2 Proposed Approach
- For layer-wise decomposition, selected QKVO matrices are stacked and decomposed separately, then reconstructed matrices are evaluated before the originals are restored.The process is repeated for each layer, as illustrated in Figure 1.Source: Figure 1 · p. 4, §3.2 · p. 5
- Stacking selected QKVO or FC matrices across all layers forms a 3-mode tensor for decomposition that captures interactions across layers.The global tensor is illustrated in Figure 2.Source: §3.2 · p. 5, Figure 2 · p. 6
Segmented Layers (Early, Middle, Last)
- The segmented configuration separately decomposes and evaluates the initial, intermediate, and final thirds of the layers to assess which segments benefit most.Each segment is processed separately.Source: Segmented Layers (Early, Middle, Last) · p. 5
4 Experiments and Results
Across RoBERTa and GPT-J, TRAWL’s layer-by-layer tensor decomposition outperformed LASER most clearly on WikiQA and BiosProfession, while results depended on the layers and matrices decomposed. Decomposing all layers together reduced performance, and the strongest gains came from final FC layers.
4.1 Experimental Setups
- Experiments compared CP and Tucker decomposition with no-decomposition baselines on GPT-J-6B and RoBERTa across WikiQA, BiosProfession, and HotpotQA.When stacking FC matrices, one was transposed to align shapes and transposed back after low-rank approximation; same-shaped QKVO matrices were stacked directly.Source: §4.1 · p. 6
One Layer at a Time
- TRAWL_CP reached 43.46% (RoBERTa) and 68.1% (GPT-J) accuracy on WikiQA, and 73.07% (RoBERTa) and 82.35% (GPT-J) on BiosProfession, outperforming LASER.On HotpotQA, TRAWL_CP was only slightly ahead of LASER for GPT-J, with similar results between methods for RoBERTa.Source: One Layer at a Time · p. 7
Across the Entire Model
- Across all GPT-J FC layers, CP decomposition reduced WikiQA accuracy; layer heterogeneity may exceed the experiment’s rank budget, while layer-by-layer decomposition proved more effective.The authors suggest more complex tensor models might capture heterogeneity better and defer that exploration to future work.Source: Across the Entire Model · pp. 7–8, Figure 4 · p. 8
Segmented Layers
- Only stacking the final third of FC weights improved accuracy; other tested segments declined, while middle-third QKVO weights showed a smaller drop.The greatest gains occurred on WikiQA and BiosProfession when decomposing RoBERTa’s final-third FC weights.Source: Segmented Layers · p. 8, Table 2 · p. 9
4.3 Ablations
- Final FC layers produced the strongest one-at-a-time gains, while QKVO showed no significant improvements; RoBERTa fluctuated more than GPT-J across ranks and decompositions, consistent with [15]’s focus on FC layers.Similar final-layer FC trends were reported for both models; the effects depended on layer, weight matrix, and decomposition rank.Source: Figure 3 · p. 7, §4.3 · pp. 9–10, Figure 5 · p. 9
5 Conclusion
TRAWL denoises LLMs through tensor decomposition, with layer-by-layer decomposition and selective stacking associated with performance gains. The conclusion also identifies further work on decomposition methods, rank selection, and broader applicability.
5 Conclusion
- Adaptive decomposition improves performance, with the largest gains in terminal-layer fully connected layers and sometimes results surpassing the original models through denoising.The reported approaches include layer-by-layer decomposition and selective layer stacking.Source: §5 · p. 10
- TRAWL stacks extracted weight matrices into a tensor and replaces them with low-rank approximations to denoise LLMs.Source: §5 · p. 10
- Future work includes testing other decomposition techniques, refining rank selection, and extending TRAWL to more models and tasks, including computer vision and reinforcement learning.Source: §5 · p. 10
6 Limitations
The study identifies open questions in choosing layers, weights, and decomposition ranks, as well as boundaries in evaluation scope and practical systems evidence.
6 Limitations
- The study focuses on CP and Tucker; other decompositions and joint compression of heterogeneous layers into irregular tensors remain to be explored.Source: §6 · p. 10
- Tensor decomposition rank selection remains challenging because, unlike matrix SVD, it has no universal solution.The study explored ranks using available heuristics but reports no definitive selection method.Source: §6 · p. 10
- The study lacks a systematic, non-brute-force way to identify the best combination of layers and weights to compress.The passage notes this challenge is shared with prior work such as LASER.Source: §6 · p. 10
- The study focuses on denoising rather than model-size reduction, leaving inference memory and compute savings across devices for future investigation.Source: §6 · pp. 10–11
- Findings cover two language models and three benchmark datasets; generalization to other models and datasets requires further validation.Source: §6 · p. 11