PaperlayerView exact revision on arXiv

Source-linked AI summary

Tensor Methods for Language Models: From Token Representation to Training, Adaptation, Inference, Compression, and Interpretability

Matvei Tarasov, Salman Ahmadi-Asl, Andre L. F. de Almeida, Andrzej Cichocki

arXiv:2608.30505v1cs.LGcs.AI

TL;DR

Tensor methods for LLMs are studied across fragmented efficiency and interpretability challenges, including costly training, memory-intensive inference, and opaque internal representations. This survey unifies the field through lifecycle and component taxonomies, finding recurring decomposition families but a persistent gap between parameter savings and realized speedups.

  • Problem

    Tensor methods are often treated as isolated compression mechanisms despite shared challenges in LLM training, inference memory, and interpretability.

    Source: §1 · p. 2, §9.1 · p. 37

  • Method

    The survey organizes tensor methods by a seven-stage LLM lifecycle and by embeddings, attention, and feed-forward components, using unified theory and comparisons.

    Source: §2.5 · p. 12, §10 · p. 42

  • Results

    The synthesis finds that decomposition strategies depend on component semantics, while TT and TTM recur across embeddings, pre-training, adaptation, and compression.

    Source: §10 · p. 42

  • Takeaways & Limitations

    The survey identifies the compression-realization gap as a central obstacle and formalizes ρ_gap to separate algorithmic savings from device-level realization.

    Source: §9.1 · p. 37, §10 · p. 42

    Training tensorized models with standard optimizers can be unstable because tensor parameterizations substantially alter the loss landscape.

    Source: §9.2 · p. 40

Abstract

from arXiv · show

Large language models (LLMs) are built from structured high-dimensional objects such as token representations, weights, adaptation updates, caches, and activations, whose multilinear structure is underexploited by the conventional matrix-centric view. Tensor decompositions and tensor networks provide a principled algebraic language for this structure, yet the literature often treats them as isolated compression mechanisms. This survey organizes tensor methods for LLMs through two complementary views: a seven-stage lifecycle taxonomy covering tokenization, embeddings, pre-training, adaptation, compression, inference, and interpretability, and a component view covering embeddings, attention, and feed-forward networks. We provide unified notation and theoretical foundations, analyze tensorization strategies for individual Transformer components, and compare methods at each lifecycle stage while making differences in evaluation protocols and model scales explicit. We further connect tensor methods to neighboring efficiency techniques and probabilistic tensor networks. Finally, we synthesize open challenges and introduce $ρ_{\rm gap}$, a metric for the compression-realization gap between theoretical memory reduction and measured system-level speedup. By treating tensorization as a common structural principle, the survey provides a structured entry point to tensorized language models and clarifies when parameter savings can plausibly translate into memory efficiency, computational efficiency, or interpretability. The GitHub page dedicated to this paper is accessible at \href{https://github.com/ma-tt-a/awesome-tensor-methods-for-llms}{this https URL}.

1 Introduction

This survey presents tensorization as a common structural principle for addressing LLM training, inference, and interpretability challenges. It unifies fragmented literature through lifecycle and component views, shared foundations, protocol-aware comparisons, and the compression-realization gap ρ_gap.

Motivation

  • Tensor methods provide a common language for LLM training costs, inference memory bottlenecks, and opaque internal representations by exposing underused multilinear structure.Transformer objects such as stacked attention projections, layer collections, activation streams, and KV caches are naturally higher-order.Source: §1 · p. 2

Literature gap

  • Existing literature remains fragmented: classical tensor theory lacks LLM mappings, neural-network tensorization often predates decoder-only LLMs, and efficiency studies treat tensors as one technique among many.Mechanistic-interpretability work using tensor networks is also recent and lacks dedicated survey coverage.Source: §1 · pp. 2–3

Lifecycle view

  • The survey organizes tensor methods with a seven-stage lifecycle spanning tokenization, embeddings, pre-training, adaptation, compression, inference, and interpretability.It frames tensorization across token representations, weights, adaptation updates, caches, and activations.Source: §1 · p. 3

Component view

  • A complementary component view covers embeddings, attention, and feed-forward networks, distinguishing decomposition compatibility with model objects from the training or deployment objective.Together, the two views connect what is tensorized with when and why tensorization is introduced.Source: §1 · p. 3

Foundations and comparisons

  • The survey supplies unified notation and theory, compares methods across model scales, baselines, protocols, and metrics, and connects tensor methods with neighboring efficiency techniques and probabilistic tensor networks.This protocol-aware treatment is intended to make reported comparisons more interpretable.Source: §1 · p. 3

Systems perspective

  • The compression-realization gap ρ_gap separates algorithmic overhead from hardware realization, clarifying why parameter reduction alone does not guarantee end-to-end speedup.The survey also synthesizes open challenges and future research directions around this distinction.Source: §1 · p. 3

2 Preliminaries

This section establishes the survey’s notation for tensor objects, tensor-algebraic operations, and their tensor-network diagram representations.

2 Preliminaries

  • Figure 1 defines symbols for scalars, vectors, matrices, tensors, and core tensor-algebraic operations alongside equivalent tensor-network diagrams.The diagrams include examples of the represented operations.Source: Figure 1 · p. 4

2.1 Notation

This section establishes tensor notation and diagrams, then defines core operations—including outer and Kronecker products, contractions, and mode-n products—used throughout the survey.

Notation

  • The survey denotes tensors with calligraphic uppercase letters, matrices with bold uppercase letters, and vectors with lowercase letters, alongside tensor-network diagrams for objects and operations.Figure 1 collects the notation and provides corresponding graphical illustrations.Source: §2.1 · p. 4
  • An outer product combines N vectors into an Nth-order tensor and yields a canonical rank-one tensor, reducing to the familiar rank-one matrix when N = 2.Tensor-network diagrams represent the outer product by placing tensors next to each other.Source: §2.1 · p. 4
  • The Kronecker decomposition approximates a matrix with a sum of Kronecker products via a rank-R truncated SVD of a rearranged matrix, with the smallest exact R defining Kronecker rank.Kronecker rank depends on the chosen factor shapes, so the decomposition is not intrinsic to the matrix alone.Source: §2.1 · p. 5
  • Contraction sums over paired modes while leaving other modes free, producing an (N + M − 2K)th-order tensor and corresponding to einsum in NumPy [54], PyTorch [55], and JAX.Matrix multiplication is the special case contracting A ∈ R^I×K with B ∈ R^K×J over one mode pair.Source: §2.1 · p. 5
  • The mode-n product contracts a tensor with a matrix along one mode and equivalently applies standard matrix multiplication to the corresponding unfolded tensor before folding it back.Its einsum form is C = einsum("i1 . . . in . . . iN, jin → i1 . . . j . . . iN", A, B).Source: §2.1 · p. 6

2.2 Tensor Decompositions

This section presents CP, Tucker, TT/TTM, and block term decompositions as complementary tensor formats with distinct rank structures, uniqueness properties, scalability, and trade-offs. Together, they span compact rank-one representations, flexible multilinear models, chain-structured high-order compression, and interpolations between CP and Tucker.

2.2.1 Canonical Polyadic Decomposition

  • CP decomposition,, represents tensors as sums of rank-one terms, offering compactness and interpretability but suffering NP-hard rank selection, possible nonexistence of best approximations, and numerical instability.Uniqueness holds under sufficient Kruskal conditions, while non-negativity, orthogonality, or sparsity constraints can improve stability and relax uniqueness requirements.Source: §2.2.1 · p. 6

2.2.2 Tucker Decomposition

  • Tucker decomposition uses a core tensor and mode-specific factors, but its gauge freedom makes the factors and core nonunique and not directly interpretable.Tucker-1 and Tucker-2 compress only selected modes, retaining other modes in full, whereas Tucker-3 compresses all three modes; HOSVD selects orthogonal factors under additional conventions.Source: §2.2.2 · p. 7

2.2.3 Tensor Train Decomposition

  • TT decomposition represents high-order tensors through a chain of third-order cores, with O(NIR^2) parameters under uniform dimensions and ranks instead of exponential full-tensor growth.TT is gauge-nonunique unless canonical conditions restrict the freedom; TTM applies the chain structure directly to reshaped matrices, preserving operator structure and corresponding to matrix product operators, [64].Source: §2.2.3 · p. 8

2.2.4 Block Term Decomposition

  • Block term decomposition sums Tucker terms with independently chosen per-term ranks, interpolating between Tucker when K = 1 and CP when every per-term mode rank equals one.Jointly selecting the number of blocks and ranks trades CP compactness against Tucker flexibility, but introduces more hyperparameters and a more involved fitting procedure.Source: §2.2.4 · pp. 8–9

2.3 Attention Mechanism

Attention lets Transformers form context-aware representations by weighting value vectors according to query-key compatibility, while decoder-only LLMs use masked self-attention and caching to support autoregressive inference. Multi-head projections expose diverse dependencies, but decoding trades computation for cache-memory traffic.

2.3. Attention Mechanism

  • Masked self-attention lets each decoder position access only tokens up to itself, making autoregressive decoding possible while avoiding the fixed-context bottleneck of RNNs [66].Attention forms context vectors as convex combinations of values, with weights derived from query-key compatibility.Source: §2.3.1 · p. 9, §2.3 · p. 9

2.3.2 Interpretation of the Attention Weights

  • Queries represent positions seeking context, keys measure compatibility through QK^T, and values provide information aggregated by attention weights into context-aware representations.In masked attention, weights and summations include only positions j ≤ i.Source: §2.3.2 · p. 10

2.3.3 Multi-Head Attention

  • Multi-head attention runs parallel projected attention operations, concatenates their outputs, and projects them back to the model dimension; standard decoder-only MHA uses dh = d/H.Decoder-only MHA applies masked self-attention to projections from one hidden-state sequence, with query and key-value head counts equal in standard implementations.Source: §2.3.3 · p. 10

2.3.4 KV-cache

  • KV-cache stores prior keys and values for reuse during decoding, reducing redundant computation but increasing memory; MQA, GQA, and MLA reduce this footprint through shared or latent representations.MQA sets HKV = 1, GQA uses 1 < HKV < HQ, and MLA caches one low-rank latent vector per token.Source: §2.3.4 · p. 11

2.3.5 Computational and Memory Complexity

  • During training and prefill, materialized attention can require O(T^2) memory, whereas decoding needs O(T) attention-row memory but must read the KV-cache.The cache occupies MKV = 2LBTHKVdhb bytes for MHA, while MLA instead caches latent vectors; training and prefill are commonly compute-bound, whereas decoding is commonly memory-bound, depending on batch size, context length, kernels, precision, and hardware.Source: §2.3.5 · pp. 11–12, Table 2 · p. 11

2.4 Feed-Forward Network

Feed-forward networks account for roughly two-thirds of Transformer parameters while using position-wise gated projections, making them both attractive compression targets and challenging replacements for highly optimized GEMM. Their potential key-value-memory role also makes FFN factorization relevant to interpretability.

2.4 Feed-Forward Network

  • FFNs typically account for approximately two-thirds of Transformer parameters [70], making them a major target for tensorization and compression.Source: §2.4 · p. 12
  • Modern decoder-only LLMs use gated FFNs with gate and up projections, nonlinear activation, element-wise multiplication, and a down projection [71].Common activations include SiLU/Swish1 [72] and GELU [73].Source: §2.4 · p. 12
  • The expansion factor dff/d typically ranges from 2 to 4, and the same FFN weights are reused independently at every sequence position.Unlike attention, FFNs do not mix information across positions.Source: §2.4 · p. 12
  • Although FFNs are large, they contain only dense matrix multiplications, so highly optimized GEMM makes them fast on modern GPUs.Structured replacements must therefore compete with a strong speed baseline.Source: §2.4 · p. 12
  • FFNs have been argued to function as key-value memories and mediate factual associations [70] [75], making their factorization relevant to interpretability as well as efficiency.Source: §2.4 · p. 12

2.5 Overview of Tensor Decompositions for LLMs

The section compares five tensor formats and links their structural properties to suitable LLM components, highlighting trade-offs among compactness, uniqueness, mode-specific ranks, scalability, and operator preservation.

Format comparison

  • CP is the most compact format and uniquely guarantees factor uniqueness, making it suitable when factors are intended to carry meaning.Its uniqueness property supports interpreting the factors themselves.Source: §2.5 · p. 12
  • Tucker assigns separate ranks to each mode, fitting tensors with distinct mode semantics such as layers and heads, but its core grows with tensor order.Source: §2.5 · p. 12
  • TT and TTM keep parameter counts linear in tensor order and are commonly used for large matrices, including embeddings, projections, and adapters.TTM additionally preserves the operator structure of the replaced matrix.Source: §2.5 · p. 12
  • BT occupies an intermediate position, offering more richness than CP while imposing more structure than a single large Tucker core.Source: §2.5 · p. 12

3 Lifecycle Overview

The survey organizes tensor methods through a seven-stage lifecycle taxonomy and grounds each stage with LLaMA-3-8B examples. The taxonomy distinguishes tensor objects, objectives, and evaluation criteria while allowing stages to be revisited or reordered.

3.1 The Lifecycle Taxonomy

  • The lifecycle taxonomy spans tokenization, embeddings, pre-training, adaptation, compression, inference, and interpretability, distinguishing each by intervention time, tensor object, objective, and evaluation criterion.Methods spanning stages receive a primary label based on where the tensor constraint is introduced and which objective is optimized.Source: §3.1 · pp. 12–13

3.2 Illustrative Examples for LLaMA-3-8B

  • For LLaMA-3-8B [26], TensorGPT compresses each embedding row independently, reducing the embedding table by 128× at TT-rank R = 1 or 12.8× at R = 4.The training-free method applies TT-SVD directly to pre-trained embeddings; the table contains roughly 525M parameters, about 6.5% of the model.Source: §3.2 · pp. 15–16
  • TTM parameterization trains dense weights directly through core contractions; at rank R = 64, one LLaMA-3-8B FFN projection shrinks roughly 60× with approximately 1M parameters and only a small perplexity increase.Chekalina et al. [85] report this result for GPT-2 trained with TTM-parametrized layers at sufficiently high rank.Source: §3.2 · p. 16
  • For a LLaMA-3-8B per-head query projection, a 1,536-parameter KronA adapter can reach rank 128, whereas LoRA costs 540K parameters at the same rank.The comparison concerns attainable rank: Kronecker terms cover only a structured subset of matrices, while LoRA is unconstrained.Source: §3.2 · p. 17
  • TPA reduces LLaMA-3-8B-sized KV-cache storage from 8,192 to 640 values per token at RK = RV = 2, making it 3.2× smaller than grouped-query attention.TPA factorizes query, key, and value matrices so cached factors are never materialized; its baseline is multi-head attention, with MHA, MQA, and GQA as special cases.Source: §3.2 · p. 17
  • Interpretability can analyze bilinear-MLP tensors through feature-basis interactions, output-conditioned quadratic forms, or direct HOSVD, with prerequisites and costs increasing toward the full tensor.The direct tensor has approximately d^3 ≈69B entries per layer, while feature contractions keep analysis at matrix scale.Source: §3.2 · p. 18

4 Tensorizing the Transformer

The section organizes Transformer tensorization component-wise across embeddings, attention, and feed-forward networks, using a mode-specific versus imposed classification. It identifies the structural opportunities and practical tradeoffs governing compatible tensorizations for each component.

4.1 Tensorization Strategies

  • Tensorization strategies are mode-specific when tensor modes have semantic roles, but imposed when modes arise from arbitrary reshaping guided mainly by parameterization inductive bias.Semantic modes can support factor sharing across structurally similar modes, whereas imposed tensorization leaves tensor-network choice unconstrained in principle [112].Source: §4.1 · p. 19

4.2 Embedding Layer

  • Embedding tensorization spans classical, n-gram, and morpheme-based tables, trading structural compatibility and memory savings against expressiveness, rank, or lookup latency.N-gram decompositions are structurally necessary for prohibitively large tables, with higher rank lowering final loss [76]; decomposed classical embeddings require inference-time contractions, while morpheme composition may be less expressive [88].Source: §4.2 · p. 19

4.3 Attention Mechanisms

  • Attention tensorization targets semantic Q, K, V activations or projection tensors, with KV-cache methods constrained by GPU efficiency, non-tensor baselines, and RoPE compatibility.Joint projection tensors can share factors across layers and heads, but this increases mode coupling; output projections require accounting for their greater width.Source: §4.3 · p. 20

4.4 Feed-Forward Network

  • TT and TTM impose tensor structure on FFN projections, but their mode ordering and rank behavior differ: TT bounds reconstructed matrix rank, whereas bounded-rank TTM can represent almost every full-rank matrix.Despite this theoretical expressivity, low-rank TTM parameterizations can struggle in practice.Source: §4.4 · pp. 20–21

4.5 Components Overview

  • Across tensorized Transformer components, parameter and FLOP reductions do not guarantee faster execution because sequential small contractions can underperform dense GEMMs without kernel-level optimization.The dominant system-level constraint is the compression–latency tradeoff, with backend-overhead removal and optimized contractions identified as open requirements [82].Source: §4.4 · p. 21, §4.5 · p. 22, Table 5 · p. 22

4.4 Feed-Forward Network

  • Bilinear FFNs admit an exact mode-specific tensor representation useful for mechanistic interpretability, whereas nonlinear gated FFNs lack this representation and remain difficult to deploy without retraining.Pearce et al. [106] use the bilinear tensor directly to discover circuits, but gated variants remain the deployed default [71].Source: §4.4 · pp. 21–22

5 Tensorization Across the Lifecycle

This section organizes tensor methods across the LLM lifecycle, from tokenization and embeddings through pre-training, adaptation, compression, inference, and interpretability. It surveys tensor-network approaches while emphasizing structural comparisons, stage-specific objectives, and limitations in comparability and system-level realization.

5 Tensorization Across the Lifecycle

  • The lifecycle taxonomy organizes tensor applications by stage, pairing formal problem definitions with literature overviews and summary tables while acknowledging sparse evidence.The stages differ in whether tensorization targets representations, model parameters, adaptation updates, compressed weights, or inference caches.Source: §5 · p. 22

5.1 Tokenization

  • No published work tensorizes tokenization itself: existing methods begin after the fixed, non-differentiable vocabulary map, at token representations.MorphTE [88] and TN-gram [76] take the vocabulary as given, while stochastic [117], [119] and learnable [120] tokenizers remain non-tensorized.Source: §5.1 · p. 23

5.2 Embeddings

  • Embedding tensorization either trains a tensorized table from scratch or compresses a pretrained table, with TT-family methods dominant but structurally distinct alternatives serving different purposes.TTM training from scratch can compress Transformer-XL but violates row independence, whereas TensorGPT [81] is training-free but may reduce expressivity and add latency; TN-gram [76] and MorphTE [88] target n-gram and morpheme tables.Source: §5.2 · pp. 23–24

5.3 Pre-training

  • Tensorized pre-training can target any Transformer component under memory or computation budgets, but attention methods face causal masking while projection tensorization trades low-rank efficiency against expressivity.Multi-linear attention [87] and TensorCoder [128] restructure attention but encounter masking constraints; GPT-TTM [85], Hypoformer, CoMERA [82], and Shapeshifter tensorize projections or ranks, with Table 7 evidence limited to methods reporting validated language-modeling perplexities.Source: §5.3 · pp. 24–25

5.4 Adaptation

  • Local methods tensorize each update independently, whereas global methods aggregate all update weights into one tensor and parameterize them jointly, spanning Kronecker, MPO, TT, quantum-inspired, CP, Tucker, and adaptive-rank TT approaches.Source: §5.4 · pp. 26–27

5.5 Compression

  • Post-training compression first factorizes pretrained weights or activations and may require healing because reconstruction error can misalign with functional preservation.Zagitov et al. [137] show that Frobenius-optimal compression can lose functionally important quality, while lightweight LoRA-style repair recovers only part of the loss; methods use Kronecker, Tucker, MPO, TT, sparsity, and activation-aware objectives, but Table 9 comparisons span different models and protocols.Source: §5.5 · pp. 27, 29

5.6 Inference

  • Inference-stage tensorization compresses KV-caches through low-rank, Tucker, mixed-precision, or permutation-based structures, seeking lower memory without sacrificing quality or throughput.TPA [77] uses (RK + RV)(H + dh) memory per token instead of 2Hdh, Tucker Attention [80] reduces it to 2R, while EinSort [105] must store a permutation per token and no method explicitly solves the full joint constrained problem.Source: §5.6 · p. 31
  • Inference methods are compared by decomposition family and whether they are training-free post-hoc methods or require training the full model from scratch.Reported perplexity, benchmark score, and latency are excluded because results are not directly comparable across methods with different parameter-count settings.Source: §5.6 · p. 31, Table 10 · p. 32

5.7 Interpretability

  • Interpretability is the survey’s least-developed stage, where tensor theory primarily makes computations and circuits structurally visible rather than improving efficiency.Taylor [49] rewrites the framework of in graphical tensor notation up to the construction of the induction head.Source: §5.7 · p. 32
  • Interpretability methods target different objects: Bilinear MLPs [106] impose multilinearity, PolySAE [107] adds low-rank quadratic and cubic decoding, and TensorLens [108] models the full Transformer as a fourth-order operator.PolySAE adds roughly 3% parameters on GPT-2 small, while TensorLens represents attention, FFNs, normalization, and residual connections jointly as T ∈ R^T×d×T×d.Source: §5.7 · p. 32

6 Relationship with Neighboring Efficiency Methods

Tensor methods are complementary to neighboring efficiency techniques because they alter structural factors or operator graphs while quantization, pruning, and distillation act on precision, support, or training signals. Effective combinations must be optimized around the active bottleneck, account for composition order and metadata, and be evaluated with matched baselines and end-to-end measurements.

Complementarity and generalization

  • Tensorization can compose with quantization, pruning, and distillation, while tensorized PEFT generalizes matrix low-rank adaptation and tensorized attention generalizes MQA/GQA.These methods act on different design variables, though efficient-attention methods can overlap with tensorized attention and yield only marginal rather than multiplicative gains.Source: §6 · pp. 32–33

Joint optimization and metadata

  • Factor-wise memory depends jointly on retained scalars, per-factor precision, and metadata, so sequential decomposition and quantization may miss better joint configurations.Irregular structure, scales, zero-points, and sparsity indices can erode value-only compression estimates; mixed-precision factor schemes are one step beyond sequential pipelines [104].Source: §6 · pp. 33–34

Order and error interaction

  • Composition order changes factor sensitivities and approximation errors, which can amplify, cancel, or regularize one another rather than add linearly.Pruning before decomposition can destroy exploitable low-rank structure, whereas pruning factors afterward creates a different structured approximation; distillation may train the tensorized student or heal compression.Source: §6 · p. 34

Bottleneck-aware combinations

  • Combinations should target the active bottleneck: weight-bandwidth regimes favor tensorization with quantization or structured sparsity, whereas long-context decoding favors tensor caches with MQA/GQA and IO-aware kernels.Tensorized updates can reduce adaptation state alongside a quantized backbone, while distillation may recover quality at additional training cost; inactive-resource improvements need not produce end-to-end gains.Source: §6 · p. 34

End-to-end limits

  • End-to-end speedup is bounded by the runtime fraction affected, so untensorized modules, communication, or framework overhead create diminishing returns and regime-dependent benefits.The same combination can therefore help prefill, decoding, fine-tuning, and batch serving by different amounts.Source: §6 · p. 34

Evaluation protocol

  • Combined-method evaluations should compare dense, tensor-only, neighboring-only, and combined systems at matched quality, hardware, and precision.They should also report metadata and temporary-buffer memory, adapter state, prefill versus decoding latency, per-module and end-to-end timing, operation order, tuning budgets, and ρgap for both tensorized components and the full pipeline.Source: §6 · pp. 34–35

7 Relationship to Probabilistic Tensor Networks

Tensor decompositions correspond to probabilistic graphical models only under non-negativity and normalization constraints; unconstrained real-valued tensor networks generally share algebraic structure without probabilistic semantics. For language models, this distinction limits probabilistic interpretability and generative use while still motivating tensorization as a conditional-independence inductive bias.

7.1 Tensor Decompositions as Parameterizations of Graphical Models

  • CP decompositions correspond to naïve Bayes models when factors are non-negative and normalized, while TT decompositions correspond to HMMs under stochastic core constraints.In the CP case, the latent class variable explains conditional independence; in the TT case, bond dimensions represent hidden states and cores encode transitions and emissions.Source: §7.1 · p. 35
  • These probabilistic correspondences are conditional: unconstrained real-valued TT or BT decompositions are structured parameterizations, not HMMs, mixtures, or other PGMs.BT decompositions acquire PGM interpretations only when factors and cores are non-negative and normalized; otherwise, they lack probabilistic semantics.Source: §7.1 · pp. 35–36

7.2 Tensor Networks as Matrix-Valued Stochastic Processes

  • An MPO is not inherently a probabilistic “matrix-valued Markov chain,” because its cores lack transition probabilities and conserved probability mass unless explicitly constrained and normalized.The terminology refers to matrix slices and sequential contraction, not a stochastic state-space process.Source: §7.2 · p. 36

7.3 Implications for Language Models

  • For LLMs, tensor factors are not automatically interpretable latent variables, and generative tensor networks require non-negativity and normalization that can complicate optimization and reduce expressivity.TT and MPO chain structure aligns with token ordering but does not itself imply probabilistic temporal dependencies; the PGM perspective instead suggests conditional-independence structure as an inductive bias.Source: §7.3 · pp. 36–37

8 Software overview

Section 8 surveys software ecosystems for tensorized language models, spanning Python libraries, hardware co-design, cross-language tools, and case studies from research-scale prototypes to production-scale LLM deployment.

8.1 Packages and Frameworks.

  • Python support spans general tensor toolboxes, PyTorch- and TensorFlow-oriented training packages, broader machine-learning libraries, and arbitrary-topology tensor-network tools.TensorLy supports NumPy [54], JAX, and PyTorch [55]; TensorLy-Torch and T3F build on PyTorch and TensorFlow [169], while quimb and cotengra target tensor-network construction and contraction optimization.Source: §8.1 · p. 37
  • Hardware co-design tools target tensorized-layer acceleration, with ETTE and Huang et al. [175] focusing on inference and FETTA and Tian et al. covering training and inference.These efforts co-design software and hardware for tensorized layers.Source: §8.1 · p. 37
  • The remaining ecosystem includes Julia, C++, and MATLAB tools, single-decomposition packages, and projects such as TTAX, TorchMPS, and TedNet that have not been updated for at least five years.Examples include ITensor, TenDeC++ [179], Tensor Toolbox, and Scikit-TT.Source: §8.1 · p. 37

8.2 Case studies

  • Case studies span scales: Javanmard et al. [185] provide an open-source MPO parameterization for PicoGPT, while Kozyrev et al. [187] deploy Minima on Qwen3-32B [9].MPOLinear supports dense initialization or training from scratch with PyTorch autograd and is validated on Tiny Shakespeare; Minima combines sensitivity analysis, Tucker/TT/Tensor Ring compression, healing, custom Triton/CUDA kernels, and speculative decoding.Source: §8.2 · p. 37

9 Discussion

The discussion defines ρ_gap to distinguish nominal tensor compression from realized speedup and identifies hardware, rank, tensorization, optimization, scaling, benchmarking, and compatibility challenges. It also proposes tensor-native future directions spanning tokenization, interpretability, lifecycle co-design, training state, elastic computation, and multimodal architectures.

9.1 Compression-Realization Gap

  • Comparable ρ_gap measurements require matched devices, precision, quality, scope, optimized dense baselines, and separately reported prefill and decoding regimes.The compute-bound interpretation uses MFU, whereas memory-bound runtimes require achieved memory-bandwidth utilization instead.Source: §9.1 · p. 38
  • ρ_gap separates theoretical byte compression from realized speedup, attributing discrepancies to decomposition arithmetic costs or device underutilization and extending comparison beyond tensor methods.ρ_gap = 1 means ideal byte compression is fully realized; values below one can occur in memory-bound settings, while values above one are typical.Source: §9.1 · p. 38

9.2 Gaps, Challenges, and Possible Solutions

  • The main implementation gaps are inefficient GPU execution, unresolved adaptive rank allocation, under-theorized tensorization choices, unstable tensorized optimization, limited scaling evidence, and uncertain interactions with neighboring methods.Fused or structure-aware kernels, adaptive ranks, tensor-structure-aware optimizers, and larger-scale studies are proposed, but no universal rank allocator exists.Source: §9.2 · pp. 39–40
  • Reliable evaluation requires shared baselines, equally tuned methods, stage-specific benchmarks, and quality reported alongside ρ_gap so parameter savings are not confused with system speedups.The review also notes that practical gains from combining tensorization with quantization, pruning, distillation, or other efficiency methods remain unresolved.Source: §9.2 · pp. 40–41

9.3 Future Directions

  • Future tokenization research should jointly structure vocabularies and embeddings, because discrete non-differentiable tokenizers currently prevent direct factorization despite lost subword relatedness.MorphTE [88] imposes embedding structure using an external analyzer but leaves the vocabulary unchanged.Source: §9.3 · p. 41
  • Tensor methods offer emerging interpretability routes through analysis tools, multilinear reformulations, inductive architectural biases, and graphical notation that exposes contractions, factorizations, and shared factors.PolySAE [107], TensorLens [108], and bilinear layers [106] exemplify these routes; graphical notation [49] is proposed as both an expository and reasoning tool.Source: §9.3 · p. 41
  • Tensor-native lifecycle design should jointly optimize embeddings, ranks, adaptation, compression, contraction schedules, training state, and distributed communication under explicit multi-objective resource budgets.Training-state tensorization must address changing ranks, accumulated approximation errors, convergence, stability, communicated bytes, synchronization, and throughput, with results reported as Pareto frontiers.Source: §9.3 · pp. 41–42
  • Conditional tensorization and multimodal or modular decompositions could make rank a runtime compute budget and separate shared linguistic factors from modality- or expert-specific structure, but batching, tail latency, routing, capacity, and negative-transfer risks require evaluation.Elastic methods should report worst-case memory and tail latency, while multimodal systems must preserve rare-expert behavior and modality-specific capacity.Source: §9.3 · p. 42

10 Conclusion

The survey unifies tensor methods for LLMs across the seven-stage lifecycle, showing that tensorization depends on component structure while decomposition families recur under different objectives. It identifies uneven field coverage and the compression-realization gap as central challenges, formalized by ρ_gap and addressed through algorithmic and systems-level directions.

Unified findings

  • Tensorization depends on component structure: attention and stacked projections support mode-specific decompositions, whereas feed-forward networks and embeddings typically require reshaping and chain-like TT or TTM networks.TT and TTM recur across embeddings, pre-training, PEFT, and compression, but objectives differ between language-modeling loss under parameter budgets and reconstruction or activation-aware distortion.Source: §10 · p. 42

Open challenges

  • The compression-realization gap is the central obstacle because parameter savings rarely yield proportional speedups; ρ_gap separates decomposition and contraction effects from device utilization.Fused and structure-aware kernels target realization, while adaptive rank allocation, better tensorization schemes, and richer tensor-network topologies target algorithmic efficiency.Source: §10 · pp. 42–43

Field coverage

  • Coverage remains uneven: tokenization is untouched, interpretability is the newest and most promising stage, and pre-training lacks multi-scale evidence about interactions with scaling laws.The survey calls for per-stage benchmarks that report ρ_gap alongside quality.Source: §10 · pp. 42–43

Conclusion

  • The survey frames tensor methods as a coherent LLM subfield and an entry point for future tensorized language-model research.It is intended both as a reference for existing work and as a guide to the next generation of methods.Source: §10 · p. 43

Related papers

Loading 2608.30505v1…