PaperlayerView exact revision on arXiv

Source-linked AI summary

Tensor Methods for Language Models: From Token Representation to Training, Adaptation, Inference, Compression, and Interpretability

Matvei Tarasov, Salman Ahmadi-Asl, Andre L. F. de Almeida, Andrzej Cichocki

arXiv:2608.30505v1cs.LGcs.AI

TL;DR

LLM tensor methods are surveyed against a matrix-centric view that underuses the multilinear structure of representations, weights, caches, and activations. The paper organizes existing approaches by lifecycle stage and Transformer component, unifies their theoretical foundations, and concludes that tensorization strategies and decomposition families recur across applications while practical gains remain constrained by the compression-realization gap.

  • Problem

    LLMs contain structured high-dimensional objects whose multilinear structure is underexploited, while tensor methods are often treated as isolated compression mechanisms.

  • Method

    The survey organizes tensor methods across seven LLM lifecycle stages and three Transformer component categories, combining unified notation, theoretical foundations, and comparisons across evaluation protocols and model scales.

  • Results

    Tensorization strategies depend on whether component modes carry semantic meaning, and decomposition families such as TT and TTM recur across embeddings, pre-training, PEFT, and compression.

  • Takeaways & Limitations

    The survey introduces ρ_gap to separate algorithmic compression from device realization, clarifying when parameter savings may translate into memory efficiency, computational efficiency, or interpretability.

  • Takeaways & Limitations

    Training tensorized models can be unstable because dense-matrix optimizers may not suit their altered loss landscapes, and the field lacks sufficient multi-scale evidence on scaling-law interactions.

Abstract

from arXiv · show

Large language models (LLMs) are built from structured high-dimensional objects such as token representations, weights, adaptation updates, caches, and activations, whose multilinear structure is underexploited by the conventional matrix-centric view. Tensor decompositions and tensor networks provide a principled algebraic language for this structure, yet the literature often treats them as isolated compression mechanisms. This survey organizes tensor methods for LLMs through two complementary views: a seven-stage lifecycle taxonomy covering tokenization, embeddings, pre-training, adaptation, compression, inference, and interpretability, and a component view covering embeddings, attention, and feed-forward networks. We provide unified notation and theoretical foundations, analyze tensorization strategies for individual Transformer components, and compare methods at each lifecycle stage while making differences in evaluation protocols and model scales explicit. We further connect tensor methods to neighboring efficiency techniques and probabilistic tensor networks. Finally, we synthesize open challenges and introduce $ρ_{\rm gap}$, a metric for the compression-realization gap between theoretical memory reduction and measured system-level speedup. By treating tensorization as a common structural principle, the survey provides a structured entry point to tensorized language models and clarifies when parameter savings can plausibly translate into memory efficiency, computational efficiency, or interpretability. The GitHub page dedicated to this paper is accessible at \href{https://github.com/ma-tt-a/awesome-tensor-methods-for-llms}{this https URL}.

1 Introduction

This survey addresses the fragmented literature on tensor methods and LLM efficiency by connecting multilinear structure to LLM-specific objects through lifecycle and component views.

  • Modern LLMs face substantial training cost, inference memory-bandwidth demands, KV-cache bottlenecks, and opaque internal representations.
  • The tensor and LLM literature is fragmented across classical decompositions, tensorized neural networks, LLM efficiency, and mechanistic interpretability communities.
  • Classical tensor work provides algebraic and numerical foundations but does not map its concepts onto LLM-specific objects.
  • Neural-network tensorization often predates decoder-only LLMs or focuses on generic, convolutional, and recurrent layers rather than LLM-specific constraints.
  • The survey combines a seven-stage lifecycle taxonomy with a component view covering embeddings, attention, and feed-forward networks.
  • It adds unified notation, protocol-aware comparisons, connections to neighboring efficiency methods, and ρ_gap to distinguish parameter reduction from realized speedup.

2 Preliminaries

Figure 1 provides a reference for the survey’s tensor notation and operations, pairing conventional symbols with tensor-network diagrams.

  • The figure’s left side collects symbols for scalars, vectors, matrices, tensors, and principal tensor-algebraic operations.
  • The right side represents the same objects and operations as tensor-network diagrams.

2.1 Notation

This section introduces tensor objects and core operations, including outer products, Kronecker products, contractions, and mode-n products, with tensor-network and einsum interpretations.

  • Tensors use calligraphic uppercase notation, while matrices and vectors use bold uppercase and lowercase symbols, respectively.
  • An outer product combines vectors into a canonical rank-one tensor; for two vectors, it reduces to a rank-one matrix.
  • The Kronecker product replaces each matrix entry with a scaled copy of another matrix, forming a block matrix.
  • The Kronecker product SVD converts Kronecker approximation into a truncated SVD of a rearranged matrix, then reshapes singular vectors into factor matrices.
  • Contraction sums over paired modes, leaves remaining modes free, and corresponds to einsum operations in array-programming libraries.
  • A matrix product is a contraction over one matched mode pair, with the other modes remaining free.
  • The mode-n product contracts a tensor with a matrix along one tensor mode and can be implemented by unfolding, matrix multiplication, and refolding.

2.2 Tensor Decompositions

The survey reviews CP, Tucker, TT, TTM, and BT decompositions, emphasizing their parameterization choices, structural advantages, uniqueness properties, and trade-offs.

  • CP decomposition: CP represents a tensor as a sum of rank-one outer products and is compact, but rank computation and approximation can be difficult and numerically unstable.
  • Tucker decomposition: Tucker uses a core tensor with mode-specific factors, allowing distinct ranks for different modes but introducing non-uniqueness and a potentially growing core.
  • Tucker variants: Tucker-1 and Tucker-2 compress only selected large modes while retaining other modes in full.
  • TT and TTM: TT uses a chain of cores whose parameter count grows linearly with tensor order under uniform dimensions and ranks, rather than exponentially with the full tensor.
  • TT and TTM: TTM applies a TT-like structure to reshaped matrices and preserves their multilinear operator structure, unlike TT.
  • Block term decomposition: BT sums Tucker terms with potentially different ranks and interpolates between Tucker when K = 1 and CP when every block rank equals one.

2.3 Attention Mechanism

Attention lets each position selectively aggregate information from sequence elements using query–key compatibility and value-weighted sums. Decoder-only LLMs use masked multi-head self-attention, whose inference behavior differs sharply between prompt processing and autoregressive decoding.

  • Scaled Dot-Product Attention: Attention computes context vectors as value-weighted combinations, with weights derived from query–key compatibility.Queries identify relevant context, keys measure compatibility, and values provide the information aggregated.
  • Masked Self-Attention: Masked self-attention restricts each token to itself and preceding tokens, enabling autoregressive decoding.The mask places −∞ above the diagonal, causing future-token weights to vanish after softmax.
  • Multi-Head Attention: Multi-head attention runs several projected attention operations in parallel and concatenates their outputs before applying the output projection.In standard implementations, the head dimension is dh = d/H and key and value dimensions match it.
  • KV-Cache: KV-cache stores prior keys and values for reuse during decoding, reducing redundant computation at the cost of increased memory.MQA and GQA reduce cache footprint by using fewer key-value heads, while MLA caches one low-rank latent vector per token.
  • Computational and Memory Complexity: Training and prefill process full sequences, whereas decoding uses one query against cached history and therefore removes one factor of T from attention cost.Training and prefill must handle a fully materialized attention matrix with O(T^2) memory, while a decode step needs only its last row, using O(T) memory.
  • Computational and Memory Complexity: Decoding is commonly memory-bound because it performs O(Td + d^2) operations while reading the whole KV-cache from HBM.The exact regime depends on batch size, context length, kernels, precision, and hardware.

2.4 Feed-Forward Network

The FFN is a large, position-wise Transformer component built from gated dense projections. Its parameter scale makes it important for compression, while its dense GEMMs and possible role in factual associations create efficiency and interpretability constraints.

  • Architecture: The gated FFN typically accounts for approximately two-thirds of a Transformer model’s parameters.Modern decoder-only LLMs use gate and up projections followed by a nonlinear activation and element-wise multiplication with the gated pathway.
  • Position-Wise Computation: The FFN applies the same weights independently at every sequence position and therefore does not mix information across tokens.Its expansion factor dff/d typically ranges from 2 to 4 in practice.
  • Efficiency Considerations: Because FFNs are dense matrix multiplications, structured replacements must compete with highly optimized GEMM implementations on accelerators.Despite their size, these optimized operations make FFNs fast to evaluate.
  • Interpretability: FFN factorization is relevant to interpretability because FFNs have been argued to function as key-value memories and mediate some factual associations.Thus, their structure matters for more than parameter or compute efficiency.

2.5 Overview of Tensor Decompositions for LLMs

The survey compares five tensor formats by parameterization, structure, and use case. Their trade-offs determine whether a format is best suited to compactness, mode-specific semantics, large matrices, operator preservation, or interpretability.

  • Format Comparison: CP is the most compact format and the only one listed with a uniqueness guarantee, making it suitable when factors are intended to carry meaning.Its approximation problem can nevertheless be difficult and unstable, as discussed elsewhere in the survey.
  • Format Comparison: Tucker assigns separate ranks to modes, fitting tensors with distinct semantic modes such as layers and heads, but its core grows with tensor order.This trades mode-specific structure for potentially larger core storage.
  • Format Comparison: TT and TTM keep parameter counts linear in tensor order and are commonly used for large matrices including embeddings, projections, and adapters.TTM additionally preserves the operator structure of the replaced matrix.
  • Format Comparison: BT occupies an intermediate position, offering more richness than CP while remaining more structured than a single large Tucker core.

3 Lifecycle Overview

The survey organizes tensor methods through complementary lifecycle and component views. The lifecycle taxonomy tracks when tensor structure is introduced or analyzed, while the component view identifies which Transformer objects are tensorized and how their objectives differ.

  • Lifecycle Overview: The lifecycle taxonomy covers tokenization, embeddings, pre-training, adaptation, compression, inference, and interpretability.Tokenization has no published tensorization example in the survey’s walkthrough, so the concrete examples begin with embeddings.
  • Taxonomy Design: The lifecycle is not necessarily linear: adaptation and compression may be repeated or reordered, while interpretability may occur before, during, or after deployment.
  • Taxonomy Design: Lifecycle stages are defined by intervention time, primary tensor object, optimized objective, and evaluation criterion.These distinctions prevent decomposition names alone from being treated as sufficient for comparison.
  • Taxonomy Design: A method spanning multiple stages receives its primary label where the tensor constraint is introduced and its objective is optimized; downstream effects are recorded separately.The lifecycle and component taxonomies are therefore orthogonal and can jointly locate a method.
  • LLaMA-3-8B Examples: For LLaMA-3-8B, the running example uses |V| = 128,000, d = 4096, L = 32, HQ = 32, HKV = 8, dff = 14336, and dh = 128.The model has approximately 8.03B parameters overall.
  • Embeddings: Per-row TT embedding factorization preserves lookup-row independence, while TTM embedding parameterization can violate it because token ordering may affect representation quality.TensorGPT compresses each token embedding independently, whereas the TTM approach trains the model from scratch in that parameterization.
  • Pre-training: Training an FFN down-projection with TTM uses approximately 1M parameters at rank R = 64 instead of about 58.7M dense parameters, a roughly 60× compression ratio.The cited study reports only a small perplexity increase for GPT-2 at sufficiently high rank.

4 Tensorizing the Transformer

The survey analyzes tensorization across embeddings, attention, and feed-forward networks, distinguishing mode-specific structure from imposed reshaping. It emphasizes that compression gains often trade off against expressivity, latency, or implementation constraints.

  • Transformer tensorization is organized component-wise because embeddings, attention, and FFNs impose different structural constraints.
  • Mode-specific tensorization exploits semantically meaningful modes, while imposed tensorization reshapes objects without relying on predefined mode semantics.Mode-specific decompositions can share factors across structurally similar modes; imposed tensorization leaves tensor shape and network topology as design choices.
  • Embeddings: Embedding tensorization includes row-wise or full-table decompositions, n-gram tensors, and morpheme-based tables, but decomposed embeddings require contraction chains during inference.These approaches trade compression or compactness against altered structural assumptions and inference latency.
  • Attention: Attention tensorization targets Q, K, V activations, projection matrices, or joint tensors, with KV-cache methods constrained by efficient kernels, quality, and RoPE compatibility.Cross-layer and cross-head sharing can improve compression, but increases coupling between modes.
  • Projection matrices: TT and TTM differ in mode ordering: TT separates input and output modes, whereas TTM couples corresponding modes within each core.TT bounds reconstructed matrix rank by R_N, while TTM can represent full-rank matrices in principle but may lose expressivity at low ranks.
  • System trade-offs: Tensorized layers can reduce parameters and FLOPs yet run slower than dense layers because sequential small contractions underutilize modern GPUs.Actual speedups require kernel-level optimization and removal of backend overhead.
  • Feed-Forward Networks: Bilinear FFNs provide an exact tensor representation with semantically meaningful input and output modes, but gated variants remain standard and require training from scratch for this formulation.The tensor B captures quadratic interactions, and its slices can be taken as symmetric without changing the quadratic form.

5 Tensorization Across the Lifecycle

Across the LLM lifecycle, tensor methods begin after tokenization and span embeddings, pre-training, adaptation, compression, inference, and interpretability. Evidence quality and deployment constraints vary substantially across stages.

  • The lifecycle view organizes tensor applications by tokenization, embeddings, pre-training, adaptation, compression, inference, and interpretability.
  • Tokenization: No published work applies tensor methods to tokenization itself; existing methods begin with token representations such as morphological or n-gram embeddings.Tokenization remains discrete and non-differentiable, while tensor methods start after the vocabulary has been formed.
  • Embeddings: Modern embedding methods use TTM, CP, or Kronecker-product constructions, achieving compact representations but often requiring training from scratch.TN-gram reduces n-gram embedding parameters from exponential O(|V|^n d) dependence to linear O(Rn|V| + Rd).
  • Pre-training: Tensorized projection methods such as GPT-TTM operate in the low-rank regime where compression is useful but retaining expressivity is difficult.
  • Pre-training: Pre-training comparisons are limited: only multi-linear attention and GPT-TTM report perplexity at two validated scales, while several compression claims lack language-modeling validation.Multi-linear attention is compared with a Transformer-XL baseline of different depth and width, complicating attribution of its compression.
  • Adaptation: Global adaptation methods tensorize aggregated updates, including LoRTA’s fifth-order CP tensor over model, head, layer, and projection modes.
  • Compression: Compression may require healing because reconstruction objectives can misalign with functional preservation, so a Frobenius-optimal approximation need not be operator-optimal.Healing may use short retraining or knowledge distillation.
  • Inference: Inference methods approximate KV-caches with tensor networks while jointly seeking memory reduction, quality preservation, and maintained or improved throughput.The formal joint constrained problem is not explicitly posed or solved in existing work.

6 Relationship with Neighboring Efficiency Methods

Tensor methods can be combined with quantization, pruning, distillation, and efficient attention, but their benefits depend on whether the methods target complementary bottlenecks. Evaluation must therefore separate component gains from end-to-end realization.

  • Tensorization is not mutually exclusive with quantization, pruning, or knowledge distillation, which modify precision, support, or training signals respectively.
  • Methods targeting the same compute or memory bottleneck exhibit diminishing returns, so combined gains are marginal rather than multiplicative.
  • Compositional methods: Tensorization and quantization jointly determine retained factor values and precisions, while metadata can determine whether structural savings survive implementation.Sequentially tuning the techniques searches only a slice of the joint design space.
  • Composition order: The order of decomposition, pruning, quantization, and distillation matters because each operation changes factor distributions, sensitivities, or exploitable structure.
  • Bottleneck-aware combinations: Bottleneck-aware combinations pair weight tensorization with quantization or sparsity, cache factorization with attention methods, and tensorized updates with quantized backbones.The appropriate combination depends on the dominant resource in the operating regime.
  • System evaluation: End-to-end speedup remains limited when untensorized modules, communication, or framework overhead dominate runtime.Even an arbitrarily fast tensorized module can provide only a modest overall gain in that regime.
  • Evaluation: Convincing evaluations should compare dense, tensor-only, neighboring-technique-only, and combined systems at matched quality, hardware, and precision.Memory accounting should include metadata, scales, indices, and temporaries, with ρ_gap reported for tensorized components and the full pipeline.

7 Relationship to Probabilistic Tensor Networks

The section clarifies when tensor decompositions correspond to probabilistic graphical models and when they remain unconstrained algebraic parameterizations. It then discusses implications for interpretability, generative modeling, sequential structure, and inductive bias in language models.

  • Tensor decompositions as probabilistic models: CP decomposition corresponds to a naïve Bayes model when weights and factor entries are non-negative and suitably normalized.The latent class variable selects conditionally independent distributions for the observed variables.
  • Tensor decompositions as probabilistic models: A non-negative, normalized TT representation can be equivalent to a hidden Markov model, with bond dimensions representing hidden-state counts.Each core slice encodes transitions between hidden states and emissions for a token position.
  • Conditional correspondences: Unconstrained real-valued TT and BT decompositions do not inherit probabilistic semantics such as stochastic cores, conserved probability mass, or latent-state interpretations.They should instead be viewed as structured low-rank parameterizations, even when their algebraic forms resemble probabilistic models.
  • Implications for language modeling: Tensor factors applied to LLM embeddings, attention tensors, or activations are not automatically interpretable as topics, syntactic roles, or semantic features.Probabilistic interpretation requires non-negativity and normalization, which most real-valued LLM tensorizations do not use.
  • Implications for language modeling: TT and MPO formats align with token order through chain-like structure, but this alignment does not itself imply probabilistic temporal dependencies.The sequential organization is a property of the parameterization rather than evidence of an underlying latent process.
  • Implications for language modeling: Viewing tensor decompositions as PGM parameterizations suggests conditional-independence structures that may provide inductive biases for analyzing expressivity and generalization.The section emphasizes that this unified perspective requires careful qualification for real-valued tensor networks.

8 Software overview

The software overview covers general tensor toolboxes, framework-integrated tensorized training packages, hardware co-design efforts, and additional language- or decomposition-specific tools. It also introduces practical studies using tensorized layers at research and production scales.

  • Python tensor libraries: TensorLy supports multiple backends, while tntorch targets PyTorch and TensorLy-Torch and T3F integrate tensorization with PyTorch and TensorFlow.These packages provide general-purpose or deep-learning-oriented tensor functionality.
  • Hardware co-design: ETTE and Huang et al. optimize tensorized linear-layer forward passes, whereas FETTA and Tian et al.'s FPGA design address both training and inference.The hardware co-design work targets implementation-level acceleration for tensorized layers.
  • Other tools: Other tools span Julia, C++, MATLAB, single-decomposition implementations, and older packages that have not been updated for five years or more.Examples include ITensor, TenDeC++, Tensor Toolbox, Scikit-TT, TTAX, TorchMPS, and TedNet.
  • Practical studies: Javanmard et al. apply MPO parameterization to every linear layer of a small PicoGPT model and validate it against a dense baseline on Tiny Shakespeare.Their open-source MPOLinear module supports dense-matrix initialization or training from scratch with standard PyTorch autograd.

9 Discussion

The discussion identifies the compression-realization gap as a central systems obstacle and organizes open problems around hardware, rank selection, tensorization design, optimization, scaling, and broader tensor-native directions.

  • Compression-realization gap: Parameter reductions rarely yield proportional training or inference speedups, motivating a protocol that separates nominal compression from realized system performance.The proposed metric compares what a paper claims with what users obtain.
  • Compression-realization gap: ρgap exceeds 1 when tensor decompositions increase arithmetic or reduce device utilization, while memory-bound workloads can produce ρgap below 1.ρgap = 1 means realized speedup matches idealized byte compression.
  • Hardware compatibility: GPU hardware favors dense GEMMs over small, sequential, irregular tensor contractions, making kernel support a key route to closing the compression-realization gap.Fused and structure-aware kernels are identified as practical directions.
  • Rank selection: Adaptive rank allocation remains unresolved because optimal ranks vary across layers and components, and no universal generally applicable allocation method exists.Uniform ranks are convenient but not generally optimal.
  • Tensorization design: Tensorization schemes and network topologies remain important design choices because equivalent parameter counts can differ in approximation quality, contraction cost, expressiveness, and memory efficiency.The survey highlights richer networks beyond shallow TTM as a promising direction.
  • Training and scaling: Tensorized training can be unstable because the parameterization changes the loss landscape, while large-scale from-scratch training remains too expensive for many academic groups.The field still lacks clear evidence for how compression and quality change with model scale.
  • Future directions: Future work extends tensorization beyond isolated weights to tokenization, interpretability, lifecycle co-design, and transient training-state tensors.These directions include jointly forming vocabularies and embeddings, using multilinear structure for interpretability, and reducing communication alongside memory.

10 Conclusion

The survey unifies tensor methods for LLMs across lifecycle stages and Transformer components, showing that shared decompositions serve different objectives and that evidence remains uneven. It identifies the compression-realization gap as central and calls for stage-specific benchmarks reporting ρgap alongside quality.

  • Unified taxonomy: The survey organizes tensor methods through a unified lens spanning seven LLM lifecycle stages and complementary component-level structure.Its component analysis distinguishes embeddings, attention, and feed-forward networks.
  • Component structure: Attention and stacked projections support mode-specific decompositions, whereas embeddings and feed-forward networks typically require imposed reshaping and chain-like networks.The distinction depends on whether tensor modes carry semantic meaning.
  • Cross-stage comparison: TT and TTM recur across embeddings, pre-training, PEFT, and compression, but their objectives and evaluation criteria differ by lifecycle stage.Pre-training optimizes language-modeling loss, while compression targets reconstruction or activation-aware distortion.
  • Uneven coverage: Tokenization remains untouched, interpretability is the newest and most promising stage, and pre-training lacks the multi-scale evidence needed to assess scaling-law interactions.The survey presents these as uneven areas of field coverage.
  • Open directions: The compression-realization gap links algorithmic choices with system behavior, motivating fused kernels, adaptive ranks, improved tensorization schemes, richer networks, and benchmarks reporting ρgap with quality.These directions target both realization and algorithmic sides of the problem.
Loading 2608.30505v1…