A Practical Tensor-Network Compression Pipeline for Production-Scale Large Language Models

Sergii Kozyrev; Davyd Maiboroda

Source-linked AI summary

A Practical Tensor-Network Compression Pipeline for Production-Scale Large Language Models

Sergii Kozyrev, Davyd Maiboroda

arXiv:2602.01613v1Feb 2, 2026cs.LGcs.AI

TL;DR

Transformer LLM deployment is constrained by VRAM and latency; Minima uses learned sensitivity to guide structural compression and healing, reducing peak VRAM from 64 GiB to 40 GiB while improving throughput.

Problem
Transformer LLM deployment is constrained by VRAM footprint and tokens-per-second latency, while production compression scheduling after sensitivity profiling remains human-in-the-loop.
Source: §1 · p. 1, §3.2 · p. 5
Method
Minima is an end-to-end compression pipeline whose CNN predicts layer- and patch-level tensor-network sensitivity and whose short fine-tuning stage recovers most pre-compression quality.
Source: §1 · p. 2
Results
64 GiB →40 GiB peak VRAM; throughput rises from ∼40 TPS to ∼50 TPS, or ∼75 TPS with speculative decoding, while benchmark accuracy stays within ±1 percentage point of baseline.
Source: §1 · pp. 2–3
Takeaways & Limitations
Minima is a practical stepping stone toward shared-tensor-backbone architectures, with future work targeting quantization and sparse-dictionary integration and multimodal and multitask models.
Source: §7 · p. 12, §6 · p. 12
Cores remain layer-local; Minima does not yet share cores across layers, though its partitioning and parameterization are compatible with future global sharing.
Source: §6 · p. 12

Abstract

from arXiv · show

Large language models are limited in deployment by GPU memory and inference latency. We present Minima, a production compression pipeline that learns where and how to structurally compress a Transformer and turns that compression into real serving gains. Minima trains a lightweight convolutional predictor to estimate layer- and patch-level sensitivity, applies a mixture of Tucker, tensor-train, and tensor-ring decompositions to low-sensitivity regions, performs a short healing fine-tune, and executes the resulting operators with custom Triton and CUDA kernels. The reduced memory footprint enables speculative decoding with a small draft model and a larger verifier. On Qwen3-32B at an 8k-token context window, Minima reduces peak VRAM from 64 GiB to 40 GiB. For a single active request, throughput increases from 40 tokens per second (baseline) to 50 tokens per second (Minima) and 75 tokens per second (Minima with speculative decoding). Under 50 parallel requests, throughput is 34, 44, and 53 tokens per second respectively, showing that Minima remains effective under high concurrency even when speculative decoding gains compress. We position Minima relative to recent tensor-network, low-rank plus quantization, and cross-layer sharing methods, and argue that it is a practical step toward more aggressive structural compression via shared tensor backbones with tiny per-layer adapters.

1 Introduction

Minima targets the VRAM and latency constraints that complicate long-context LLM deployment, using a learned, multi-stage compression pipeline. On Qwen3-32B, it reduces memory use and raises throughput while keeping quality close to baseline.

Motivation

Existing tensor-network methods often target smaller models or non-production tasks, require extensive retraining, or lack engineering for latency-critical serving.Source: §1 · p. 1

Design target

The design target is roughly 2× lower memory use and 2× higher speed at the same quality budget on production LLM workloads.Source: §1 · pp. 1–2

Pipeline

Minima predicts patch sensitivity, mixes Tucker, TT, and TR for low-sensitivity regions, heals with short fine-tuning, then optimizes kernels and uses freed VRAM for speculative decoding.The small draft model proposes tokens for a larger verifier.Source: §1 · p. 2

Main findings

Minima reduces peak VRAM by ≈37% at long context (64 GiB →40 GiB) and increases throughput from ∼40 TPS to ∼50 TPS via compression and kernel optimization alone, reaching ∼75 TPS with speculative decoding.Source: §1 · pp. 2–3
Perplexity increases by at most ≈3% (relative), while accuracy changes on common language benchmarks remain within ±1 percentage point of baseline.Source: §1 · p. 3

2 Background and Related Work

The section reviews tensor decompositions, Transformer tensor networks, low-rank and quantization methods, and cross-layer sharing. It positions Minima as sensitivity-guided and largely layer-local, while describing a path toward shared-tensor designs.

2.1 Tensor decompositions for neural network compression

Classical low-rank methods evolved from SVD and matrix factorization to CP, Tucker, TT, TR, and block-term decompositions; reviewed work often targets modest-scale or non-LLM tasks and needs manual rank tuning or costly search.Source: §2.1 · p. 3

2.2 Tensor networks in Transformers and LLMs

For Transformer LLMs, prior work uses MPOs or attention-focused Tucker, while comparisons find no single tensor network dominates across layer types and compression ratios; Minima instead combines Tucker, TT, and TR with sensitivity guidance and serving-oriented kernel and decoding work.Source: §2.1 · p. 3, §2.2 · pp. 3–4

2.3 Low-rank, quantization, and adaptive rank selection

Low-rank and quantization methods include factorization-aware training, learned tensor ranks, joint rank and bit-width optimization [13], and outlier-aware 3–4-bit quantization [14]; Minima complements them in the structural domain and is compatible with quantizing TN factors.Source: §2.3 · p. 4

2.4 Cross-layer parameter sharing and dictionary learning

Cross-layer methods share bases, dictionaries, or TT cores; Minima remains largely layer-local, with learned sensitivity deciding where compression is safe, and identifies shared tensors with small per-layer controllers as a future direction.Source: §2.4 · pp. 4–5

3 The Minima Compression Pipeline

Minima applies learned sensitivity estimates to choose where and how to compress a decoder-only Transformer, then heals and optimizes the resulting operators. Freed VRAM supports speculative decoding alongside the compressed model.

3.1 Problem setting and design goals

The assumed deployment setting is autoregressive generation with decoder-only Transformers, context lengths up to 8k tokens, and latency targets of tens of tokens per second per GPU.Design goals include reducing memory, improving throughput, avoiding large-scale retraining, and retaining modularity for future structural advances.Source: §3.1 · p. 5

3.2 Stage 1: Analyze — CNN-based sensitivity prediction

Brute-force search across decompositions and ranks is prohibitively expensive; Minima trains a lightweight CNN on sampled patch-level compression effects to predict sensitivity and recommend a compression budget.Its inputs include local spectra, condition-number estimates, magnitude and sparsity statistics, and positional information.Source: §3.2 · pp. 5–6

3.3 Stage 2: Compress — Mixed Tucker, TT, and TR decompositions

For each patch, Minima selects a tensor-network family and ranks to meet a predicted compression target, using Tucker for suitable reshaped matrices, TT for long or skinny matrices, and TR when cyclic structure fits.The selection reflects that decomposition performance varies by regime.Source: §3.3 · pp. 6–7
Across the reference model, parameters fall by ∼35–40% and peak VRAM at 8K context drops from 64 GiB to 40 GiB.Source: §3.3 · p. 7

3.4 Stage 3: Heal — Fast accuracy recovery

Healing freezes the architecture and ranks while updating TN cores and factors, training on next-token prediction and downstream tasks, with optional distillation; a few thousand steps recover most performance loss.Source: §3.4 · p. 7

3.5 Stage 4: Optimize kernels — Triton/CUDA implementation

Because TN indirection can make naïve layers slower despite fewer FLOPs, custom Triton kernels fuse operations, tile for GPU occupancy, specialize common shapes, and organize storage for coalesced access.On the reference model, compressed kernels match or slightly exceed baseline throughput at 8K context before speculative decoding.Source: §3.5 · p. 7

3.6 Stage 5: Speculative decoding

At 8K context on one A100 80 GB GPU, speculative decoding moves throughput from ∼50 TPS to ∼75 TPS while retaining the verifier’s original KV-cache budget.A smaller draft model proposes multiple tokens for batch verification by the compressed main model.Source: §3.6 · p. 7

4 Experimental Setup

The experiments use Qwen3-32B and assess language-modeling quality, benchmark accuracy, memory use, and throughput. They compare dense, compressed, and speculative-decoding variants on A100 GPUs.

4.1 Models and datasets

Evaluation covers held-out validation perplexity, standard reasoning and understanding benchmarks, peak VRAM at 8K context, and tokens-per-second across context lengths.Source: §4.1 · p. 8

4.2 Baselines

The variants are an uncompressed dense baseline, healed Minima without speculative decoding, and Minima with speculative decoding using a 1–2B draft model.Source: §4.2 · p. 8

4.3 Implementation details

Experiments use PyTorch and custom Triton kernels on NVIDIA A100 GPUs with 80 GB VRAM and bf16 precision; latency evaluation uses batch size 1 unless stated otherwise.Source: §4.3 · p. 8

5 Results

At 8K, Minima reduces VRAM and raises throughput over baseline; reported quality changes are modest, and speculative decoding adds throughput gains.

5.2 Throughput under concurrent request load

At 8K, single-request throughput is ∼40 TPS baseline, ∼50 TPS Minima, and ∼75 TPS Minima + Spec; at 50 requests, values are ∼34, ∼44, and ∼53 TPS, respectively.The incremental speculative-decoding benefit is smaller at high concurrency.Source: Table 2 · p. 9, §5.2 · p. 9

5.3 Accuracy and perplexity

Minima shows modest perplexity degradation and sub-point benchmark-accuracy drops or minor fluctuations.The reported quality changes are described as comparable to those in other substantially compressed models.Source: Table 3 · p. 10, §5.3 · p. 10

5.4 Ablations

Against uniform TN ranks, CNN-predicted patch-wise ranks yield a small TPS gain at similar perplexity, while mixed TN types deliver the bulk of compression and TPS improvement.The mixed choices are Tucker, TT, and TR.Source: §5.4 · pp. 10–11
Adding speculative decoding to the ablation sequence provides the largest throughput gain, especially at long contexts.The sequence adds draft/verify decoding after sensitivity-based ranks and mixed TN types.Source: §5.4 · pp. 10–11

6 Discussion

Minima combines Tucker, TT, and TR choices guided by predicted sensitivity, taking a post-training structural-compression path that can be layered with quantization. Its cores remain layer-local, though the design is described as compatible with future cross-layer sharing.

Comparison to tensor-network LLMs

Minima lets Tucker, TT, and TR coexist under predicted sensitivity, targeting a balanced 2× reduction rather than extreme compression ratios.The discussion contrasts this with CompactifAI and TensorLLM and their more specific TN designs.Source: §6 · p. 11

Comparison to low-rank and quantization methods

Unlike training-time factorization and joint rank/bit-width optimization, Minima uses post-training sensitivity guidance and can pair TN cores with 4-bit quantization.The discussion also notes possible combinations with SpQR.Source: §6 · p. 11

Comparison to basis/dictionary sharing approaches

Minima's cores are layer-local today, but the authors describe shared dictionaries or TT cores with per-layer adapters as a compatible future direction.The discussion connects this direction to Basis Sharing, CoSpaDi, and MetaTT-style designs.Source: §6 · pp. 11–12

Hardware and systems aspects

Minima follows algorithm-architecture co-design on commodity GPUs with Triton/CUDA kernels that respect tensor-network structure.The discussion says this is believed essential for practical competitiveness with pure quantization or pruning.Source: §6 · p. 12

7 Future Directions and Conclusion

Minima combines learned sensitivity with mixed tensor-network compression and serving optimizations to reduce memory and improve throughput. The authors present shared tensor backbones with small per-layer adapters as a next step toward more global compression.

Minima combines CNN-based sensitivity mapping, mixed Tucker/TT/TR decompositions, a short healing fine-tune, custom kernels, and speculative decoding enabled by freed VRAM.Source: §7 · p. 12
On a 32B-parameter model, Minima achieves approximately 2× compression in VRAM and up to ∼2× speed-up in TPS at long contexts, while keeping quality close to the original model.Source: §7 · p. 12
A proposed shared tensor backbone would span Transformer layers and submodules, with tiny per-layer adapters selecting or modulating its slices.Minima currently operates mostly at the layer level.Source: §7 · p. 12
Future work will explore tighter integration with quantization and sparse-dictionary methods and extend the pipeline to multimodal and multi-task foundation models.Source: §7 · p. 12