Source-linked AI summary
Sequential Functional Structured Tucker Compression for Large Language Model Attentions
Jiangfeng Chen, Xinyu Wang, Tianshuo Yan, Hanwei Wu, Xiao-Wen Chang, Yang Zhang, Lei Ding
arXiv:2610.00717v1cs.CLcs.AIstat.ML
TL;DR
Independent attention-projection compression can miss shared Q/K/V structure and the representation shifts caused by earlier compression. FTC adapts joint Q/K/V compression to the current student, treats the output projection separately, and performs strongly across evaluated models and tasks.
Problem
Independent matrix approximation can overlook shared Q/K/V structure and may not account for representation changes as earlier layers are compressed.
Source: §1 · p. 1Method
FTC sequentially fits shared Q/K/V Tucker factors using an activation-induced metric from the current compressed model, allocates ranks and sparse-core capacity under a storage budget, and compresses the output projection separately.
Source: §1 · pp. 1–2Results
Takeaways & Limitations
FTC's reported improvements span downstream tasks and are achieved without fine-tuning or gradient-based recovery.
Source: §1 · p. 2
Abstract
from arXiv · showhide
Post-training compression of LLM attention is often formulated as independent matrix approximation, ignoring both the shared structure among attention projections and the representation shift introduced by earlier compression. We propose FTC, a sequential structured compression framework that adapts the approximation to the current compressed model while jointly exploiting the native Q/K/V head structure under a fixed storage budget. The output projection is handled separately to account for the changed post-attention representation. FTC requires neither fine-tuning nor gradient-based recovery. Across seven decoder-only LLMs from 6B to 32B parameters, FTC achieves the lowest WikiText-2 perplexity among the compared methods at every tested keep ratio on five modern GQA models, with the largest gains under aggressive compression. The improvements transfer to downstream tasks and remain substantial at the 32B scale.
1 INTRODUCTION
FTC addresses independent attention-projection compression by adapting to the evolving compressed model and exploiting shared Q/K/V structure. It couples rank and sparse-core allocation under a storage budget and handles the output projection separately.
1 INTRODUCTION
- Independent low-rank projection approximations can repeatedly spend budget on feature directions shared across Q/K/V, while joint factors can represent common directions and leave budget for projection- and head-specific differences.The remaining challenge is to share useful structure without overly restricting individual heads.Source: §1 · p. 1
- FTC uses an activation-induced metric from the current compressed model to fit shared Q/K/V Tucker factors while retaining head-specific core coefficients.The head-slice mode remains uncompressed, supporting MHA and GQA; slice balancing prevents projection-magnitude differences from dominating the shared fit.Source: §1 · pp. 1–2
- FTC treats the output projection separately because it acts on the post-attention representation, calibrating its target with updated attention features before compression.The calibration matches original attention-branch outputs while regularizing toward pretrained weights and has a closed-form linear solution.Source: §1 · pp. 1–2
- FTC jointly allocates Tucker ranks and sparse-core capacity under a storage budget that counts factors, retained coefficients, and support metadata, then gradually prunes and refits the core.This balances subspace and sparsification errors.Source: §1 · p. 2
- Across seven decoder-only LLMs from 6B to 32B, FTC delivers lower WikiText-2 perplexity and higher aggregate downstream accuracy under aggressive compression on all five GQA models.The comparison is reported without fine-tuning or gradient-based recovery; a matched per-matrix control supports the benefit of the structured representation.Source: §1 · p. 2
2 RELATED WORK
Related work addresses activation-aware low-rank approximation, shared or tensorized representations, and sequential correction and sparsification. FTC combines these directions through sequentially updated activation metrics, joint head-wise Q/K/V structure, and within-layer rank–sparsity allocation.
2 RELATED WORK
- Activation-aware low-rank methods use input activations or parameter sensitivity, whereas FTC builds its metric from the current compressed model and fits joint head-wise Q/K/V rather than independent matrices.The cited examples include Fisher-weighted SVD [Hsu et al., 2022], ASVD and SVD-LLM [Yuan et al., 2023; Wang et al., 2025c].Source: §2 · p. 2
- Shared and tensorized methods exploit common low-dimensional structure, but FTC retains head-specific core coefficients alongside Tucker factors shared across Q/K/V.Prior examples include Basis Sharing, TensorLLM and LeSTD [Gu et al., 2025], and joint Q/K/V factorization [Wang et al., 2025d].Source: §2 · pp. 2–3
- FTC uses current-student activations for joint Q/K/V approximation and compresses the output projection separately using the updated attention representation.This differs from prior sequential correction or refinement approaches such as SparseGPT [Frantar & Alistarh, 2023], MPIFA and AA-SVD [Zhao et al., 2025; Sinha & Fleuret, 2026].Source: §2 · p. 3
- Unlike methods allocating low-rank approximation and pruning across layers or sublayers, FTC couples Tucker ranks and retained latent-core coefficients within each attention layer.Core sparsity selects rank-one interactions in the latent representation, not zeros in reconstructed weight matrices; practical acceleration requires a kernel that exploits that representation.Source: §2 · p. 3
3 FUNCTION-DRIVEN STRUCTURED COMPRESSION
FTC compresses attention layers sequentially using the current student representations, jointly modeling Q/K/V with shared Tucker factors and a budgeted sparse core. It compresses the output projection separately using the updated attention representation.
3 FUNCTION-DRIVEN STRUCTURED COMPRESSION
- FTC proceeds layer by layer, updating the student representation before using it to guide compression of the next attention layer.At each layer it builds an activation-induced metric, jointly compresses Q/K/V, and processes the output projection separately after updating attention features.Source: Figure 1 · p. 3, §3 · p. 3
3.1 PRELIMINARIES AND PROBLEM FORMULATION
- The Q/K/V objective uses the current student stream and a regularized activation-induced metric, with storage constrained by a layerwise budget that includes factors, core values, and support metadata.The joint tensor stacks head-wise projections while preserving native head organization; GQA and MQA shared K/V heads are included once rather than replicated.Source: §3.1 · pp. 3–4
3.2 SEQUENTIAL ACTIVATION-INDUCED TENSOR APPROXIMATION
- Metric whitening converts the activation-induced approximation objective exactly into a Euclidean tensor reconstruction problem.The whitened mode-1 factor is mapped back after compression.Source: §3.2 · p. 4
3.3 SHARED TUCKER REPRESENTATION WITH SLICE BALANCING
- FTC shares the first two Tucker factors across Q/K/V slices while each head slice retains its own core coefficients, and balances slices by relative rather than absolute reconstruction error.This prevents high-energy slices from dominating shared-factor estimation.Source: §3.3 · pp. 4–5
3.4 BUDGET-COUPLED RANK–SPARSITY ALLOCATION
- FTC jointly selects Tucker ranks and retained core coefficients under the layerwise budget, trading reduced subspace error against storage consumed by factors and support metadata.The sparse core uses a bit-packed support mask and stores only retained coefficients; after allocation, the core is gradually pruned with support-constrained refitting.Source: §3.4 · pp. 5–6
3.5 SEPARATE COMPRESSION OF THE OUTPUT PROJECTION
- After Q/K/V compression, FTC recomputes attention features and uses them to construct a separate activation-induced metric for the output projection.Optional calibration balances matching teacher attention-branch outputs against deviation from pretrained output weights; the updated block supplies the next layer's metric inputs.Source: §3.5 · p. 6
4 EXPERIMENTS
FTC is evaluated across seven decoder-only models spanning MHA and GQA, using perplexity, downstream performance, and controlled ablations. It performs especially well under aggressive compression, while ablations show that several components matter more as the storage budget tightens.
4 EXPERIMENTS
- The evaluation covers seven models—two MHA and five modern GQA—and compares FTC with five published baselines plus a per-matrix control.FTC uses 256 WikiText-2 calibration sequences; all main-table results use one fixed recipe without fine-tuning or gradient-based recovery.Source: §4 · pp. 6–7
4.1 PERPLEXITY ACROSS THREE MODEL GENERATIONS
- At ρ = 0.2, FTC cuts WikiText-2 perplexity by 39–88% versus the best competing baseline across five GQA models, while staying within 1.21–1.83× dense perplexity.SVD-LLM reaches 4.8–37.3× dense perplexity at this ratio.Source: Figure 2 · p. 7, §4.1 · p. 7
- At ρ = 0.2, FTC has 39.5% lower perplexity than the strongest competing baseline on Qwen3-32B, with 9.22 versus 15.23 perplexity.On Qwen3-8B, FTC’s corresponding reduction is 51.9% (10.70 vs. 22.23).Source: §4.1 · p. 8
4.2 DOWNSTREAM TASKS
- At ρ = 0.2, FTC exceeds the best published baseline by 10.7–17.0 percentage points in avg6 across four 7–8B models and by 21.1 points on Qwen3-32B.Figure 3 tracks avg6 and 5-shot GSM8K retention across the compression sweep.Source: Figure 3 · p. 8, §4.2 · p. 9
- At ρ = 0.4, FTC retains 86–93% of dense avg6 on four 7–8B models, but only 12–55% of GSM8K; Qwen3-32B retains 98.7% and 84.7%, respectively.The authors report that reasoning-intensive generation degrades earlier under compression, while increased scale provides greater robustness.Source: §4.2 · p. 9
4.3 ATTRIBUTION AND ABLATIONS
- At ρ = 0.2, removing whitening, sequential calibration, or joint tensorization raises perplexity by 79.8–1662.4%, 70.2–457.7%, or 65.0–1102.8%, respectively.At ρ = 0.6, the corresponding increases are at most 11.7%.Source: Table 3 · p. 9, §4.3 · p. 9
- At ρ = 0.2, removing anchoring changes perplexity by 1.5–14.5%, removing slice-wise normalization by 11.8–492.4%, and removing the WO teacher target by 2.6–182.6%.Replacing head-concat with K/V repeat-expansion increases perplexity by 2.6–64.6% across four GQA models.Source: §4.3 · p. 9
5 CONCLUSION
FTC combines an activation-induced metric, sequential calibration, joint Q/K/V tensorization, and budget-coupled rank–sparsity allocation. Across MHA and GQA models from 6B to 32B, it achieves lower perplexity and stronger downstream retention than compared methods without fine-tuning or gradient-based recovery.
5 CONCLUSION
- FTC combines an activation-induced metric, sequential calibration, joint Q/K/V tensorization, and budget-coupled rank–sparsity allocation.Ablations show metric whitening, sequential calibration, and joint tensorization become increasingly important as the compression budget tightens.Source: §5 · p. 9
- Across MHA and GQA models from 6B to 32B, FTC achieves consistently lower perplexity and stronger downstream retention than compared methods without fine-tuning or gradient-based recovery.The reported gains span models from 6B to 32B parameters.Source: §5 · p. 9
AI USE STATEMENT
The authors used generative AI tools for research feedback, mathematical review, experimental-result interpretation, manuscript drafting and revision, literature identification, and workflow-diagram preparation. They report independently checking retained AI-assisted material and taking responsibility for the final content.
AI USE STATEMENT
- Generative AI tools supported methodology and experimental-design feedback, mathematical-derivation review, and interpretation of experimental results.The tools also assisted with manuscript drafting and revision, literature identification, and preparation of a draft workflow diagram.Source: AI USE STATEMENT · p. 12
- The authors say they checked retained AI-assisted material, verified references and numerical statements, and take responsibility for the final content.They report independently checking mathematical arguments, crosschecking numbers against experimental records, and revising assisted text and figures.Source: AI USE STATEMENT · p. 12
A EXPERIMENTAL DETAILS AND FULL RESULTS
The appendices describe reproduction, storage accounting, calibration and evaluation protocols, compression costs, and the scope of the full perplexity and downstream-results tables.
A.1 REPRODUCTION PROTOCOL AND ACCOUNTING
- Native LeSTD’s per-head factorization is not directly applicable to GQA, so the authors use enh-LeSTD, a GQA-compatible port retaining LeSTD’s data-free objective.The port uses the same native head-concat tensorization as FTC to isolate the compression objective from the tensorization needed for GQA support.Source: §A.1 · p. 12
- The retained storage ratio ρ includes Q/K/V shared factors, nonzero core values, and support-mask metadata, while the output projection is counted separately.With the remaining model unchanged, whole-model storage reduction is (1 − ρ)s_attn.Source: §A.1 · p. 12, Table 4 · p. 12
- FTC uses 256 calibration sequences of 2,048 tokens per model, while data-dependent baselines use 128 sequences from their published configurations.Evaluation uses WikiText-2 test perplexity and downstream tasks with 1,000 examples per task, except MathQA (983) and GSM8K (200); Qwen3-32B uses CPU offloading and layer-wise compression.Source: §A.1 · p. 12
- FTC compression takes 16–39 minutes per setting for 7–8B models, 21–38 minutes for GPT-J-6B, 39–166 minutes for Llama-2-13B, and 96–170 minutes for Qwen3-32B.The timings are for the complete pipeline; Qwen3-32B uses CPU offloading and layer-wise compression.Source: §A.1 · p. 13, Table 5 · p. 13
A.2 FULL PERPLEXITY TABLES
- Table 6 covers WikiText-2 perplexity for all models at ρ ∈ {0.8, 0.6, 0.4, 0.2}; Table 7 reports PTB and C4 perplexity.Mistral’s high PTB values are attributed to tokenizer/domain mismatch, which affects all methods equally.Source: Table 6 · p. 13, Table 7 · p. 14
A.3 FULL DOWNSTREAM GRID
- The downstream grid reports per-task accuracy for six base tasks, MathQA, GSM8K, and avg6 across model–method–ratio configurations.The tables cover MHA, GQA, and hybrid models; “–” marks configurations that are not applicable, including diverged native LeSTD at ρ = 0.2.Source: §A.3 · p. 14, Table 8 · p. 15, Table 9 · pp. 16–17
A.4 HYPERPARAMETER ABLATIONS
The ablations vary components of the final recipe one at a time at ρ ∈ {0.6, 0.2} on four GQA models. They report that several design choices matter more under aggressive compression.
Ablation protocol and additional design studies
- The one-at-a-time ablations use four GQA models at ρ ∈ {0.6, 0.2}, changing one component while holding the remaining recipe and compression budget fixed.The fixed recipe includes head-concat tensorization, budget-coupled allocation, slice-wise normalization, and a separate teacher-calibrated WO target.Source: §A.4 · p. 17
QKV teacher-map regularization
- At ρ = 0.2, disabling the QKV teacher map gives the lowest perplexity; mapped variants remain 2.1–11.3% higher even at their best settings.Among mapped variants, μ = 3 performs best on Mistral-7B, Llama-3-8B, and Qwen3-8B, while Qwen2.5-7B favors μ = 10.Source: §A.4 · p. 17, Table 10 · p. 18
Rank allocation
- Replacing budget-coupled allocation with fixed Tucker ranks increases perplexity by 3.7–6.2% at ρ = 0.2.The allocator’s gain is described as moderate compared with the larger benefits from the activation-induced metric, joint tensorization, and sequential calibration.Source: §A.4 · p. 18, Table 11 · p. 18
Output-projection metric
- Reusing the QKV activation metric for the output projection increases perplexity by 55–708% at ρ = 0.6 and by more than 4000% at ρ = 0.2.The ablation supports constructing the output-projection metric from its own attention-output representations.Source: §A.4 · p. 18, Table 11 · p. 18
Refit iterations
- A second refit changes perplexity only marginally in the evaluated settings, and the final recipe retains two refits.The passage reports that the first refit captures nearly all of the improvement.Source: §A.4 · p. 18, Table 12 · p. 18
Anchoring
- Removing the WO anchor changes perplexity by −0.6% to +0.8% at ρ = 0.6, but increases it by 1.5–14.5% across the four GQA models at ρ = 0.2.At ρ = 0.2, Qwen2.5-7B shows the largest sensitivity, with a 14.5% increase.Source: §A.4 · p. 18
Slice-wise normalization
- Removing slice-wise normalization changes perplexity by −0.3% to +5.5% at ρ = 0.6, but increases it by 11.8–492.4% across the four GQA models at ρ = 0.2.The largest increase is 492.4% on Qwen2.5-7B; the text attributes the greater sensitivity to energy imbalance across head slices under a tighter core budget.Source: §A.4 · p. 18
A.5 UNIFORM RECIPE, CALIBRATION SOURCE, SEEDS, STRUCTURED CORE
The appendix tests FTC’s fixed recipe and calibration choices across models and compression settings. Sensitivity is often modest, but calibration size and 2:4 sparsity can sharply affect Qwen2.5-7B at ρ = 0.2.
A.5 UNIFORM RECIPE, CALIBRATION SOURCE, SEEDS, STRUCTURED CORE
- The same fixed recipe is used across models and all four compression ratios without model-specific tuning.The recipe includes the allocator, Q/K/V and output-projection settings, two refits, 256 calibration windows, and slice-wise normalization.Source: §A.5 · pp. 18–19, Table 13 · p. 19
- At ρ = 0.2, increasing damping to 10^-2 raises perplexity by 1.4%–2.7%, while damping at 1 raises it by 11.7%–24.8% across three models.At ρ = 0.6, changes range from −1.5% to +3.0%; the recipe fixes ϵ = 10^-4.Source: Table 14 · p. 19, §A.5 · p. 19
- The whitening guard escalates damping only for some layers at ρ = 0.2; Qwen2.5-7B requires escalation in 9 of 28 layers, with 3 reaching ϵ = 1.The guard is never activated at ρ ≥ 0.4 and serves as a numerical safeguard rather than a compression-quality improvement.Source: Table 15 · p. 19, §A.5 · p. 19
- Reducing calibration from 256 to 128 windows raises Qwen2.5-7B perplexity from 12.55 to 43.90 at ρ = 0.2, while effects are modest in most other settings.The passage reports +2.3% for Mistral-7B, +3.7% for Llama-3-8B, and −0.2% for Qwen3-8B at this ratio.Source: Table 16 · p. 20, §A.5 · p. 20
- At ρ = 0.2, a 2:4 core raises perplexity by 11.3%–366.2% across four models, with the largest penalty on Qwen2.5-7B.Qwen2.5-7B rises from 12.55 to 58.50 perplexity; at ρ = 0.6 and 0.4, the increases are only 0.3%–1.3% and 0.8%–2.0%, respectively.Source: Table 17 · p. 20, §A.5 · p. 20
- Switching calibration from WikiText-2 to C4 consistently improves C4 perplexity but generally worsens WikiText-2 perplexity, with larger effects at ρ = 0.2.PTB trends are less uniform, and Qwen2.5-7B’s calibration source also affects numerical stability at ρ = 0.2.Source: §A.5 · p. 20, Table 18 · p. 21
A.6 WHERE DRIFT ACCUMULATES ALONG DEPTH
- FTC generally maintains smaller residual-stream drift across depth, especially under aggressive compression on Mistral-7B and Llama-3-8B.On Qwen models at ρ = 0.2, drift is less discriminative through middle layers and separates more clearly near the output.Source: Figure 4 · p. 22
A.7 ROBUSTNESS AND SENSITIVITY
Calibration sampling produces stable results across the tested GQA models and settings.
A.7 ROBUSTNESS AND SENSITIVITY
- Across three independently sampled calibration sets, perplexity standard deviation is at most 0.03 in displayed settings, except Qwen2.5-7B at the most aggressive budget, where it is 0.07.At ρ = 0.2, downstream avg6 standard deviations range from 0.002 to 0.008.Source: Table 19 · p. 22, §A.7 · p. 22
B IMPLEMENTATION AND OPTIMIZATION DETAILS
This section describes FTC’s sequential, activation-aware attention compression using current student representations, preserving native Q/K/V head structure and handling the output projection separately. It also covers implementation refinements and safeguards, including optional 2:4 alignment and maintaining the prescribed compression budget.
B.1 OVERALL COMPRESSION ALGORITHM
- Each layer compresses Q/K/V using current student activations, updates the student, then separately compresses WO,l before propagating representations to the next layer.The Q/K/V tensor is jointly approximated with shared Tucker factors; the output projection is processed using representations from the updated attention branch.Source: §B.1 · pp. 22–23
B.2 GRADUAL PRUNING AND SPARSE-CORE REFINEMENT
- Progressive prune–refit lets shared Tucker subspaces and surviving core coefficients adapt as sparsity tightens, rather than fixing final support from the initial dense core.HOSVD initializes candidate subspaces for rank allocation, while the allocator’s one-step top-K projection is only a surrogate; the selected core is pruned progressively.Source: §B.2 · pp. 23–24
B.3 CALIBRATION AND COMPRESSION OF THE OUTPUT PROJECTION
- Output-projection calibration targets teacher-branch outputs on the current student representation, regularizes toward pretrained WO,l, and compresses the target with its own activation-induced metric.The metric is induced by current student-branch representations, and the approximation is an independent metric-whitened low-rank compression.Source: §B.3 · pp. 25–26
B.4 OPTIONAL PATTERN-ALIGNED REFINEMENT FOR 2:4 SPARSITY
- For 2:4 sparsity, an optional orthogonal gauge rotation is accepted only when it sufficiently increases retained core energy, aligning coordinates without changing Tucker ranks or the dense subspace.The accepted coordinates initialize the subsequent 2:4-constrained sparse-core refinement.Source: §B.4 · p. 26
B.5 NUMERICAL STABILITY AND PRACTICAL DETAILS
- Damping, WO-only teacher-Gram anchoring, and fallback metric handling address instability while keeping the original Tucker ranks, sparse-core budget, and WO,l low-rank budget unchanged.The final recipe uses anchoring coefficient a = 0.5; if metric-based approximation remains unstable, the identity metric is used.Source: §B.5 · p. 27
C ADDITIONAL DETAILS FOR SEQUENTIAL ACTIVATION-INDUCED TENSOR APPROXIMATION
- At each layer, the metric is built from current student activations after earlier layers have been compressed, incorporating the representation shift into the approximation.The activation-induced metric measures the effect of projection-weight perturbations on outputs over calibration representations.Source: C ADDITIONAL DETAILS FOR SEQUENTIAL… · p. 27
- Q, K, and V share an input-side activation metric, which the joint tensor objective applies consistently across all Q/K/V slices.Source: C ADDITIONAL DETAILS FOR SEQUENTIAL… · p. 28
- The joint tensor stacks native Q/K/V head blocks as frontal slices, retaining shared GQA/MQA key/value heads without replication and leaving the slice-enumeration mode uncompressed.Query slices precede key and value slices.Source: C ADDITIONAL DETAILS FOR SEQUENTIAL… · p. 28
- Metric whitening converts the activation-weighted tensor approximation into an ordinary Frobenius-norm approximation in whitened coordinates.The shared-input metric acts only on mode 1; the metric is damped for numerical stability.Source: C ADDITIONAL DETAILS FOR SEQUENTIAL… · p. 28
- Each compressed layer replaces its original Q/K/V projections before the updated student produces representations for subsequent compression, with the activation-induced metric recomputed at every layer.This makes each approximation account for representation changes caused by earlier compression.Source: C ADDITIONAL DETAILS FOR SEQUENTIAL… · pp. 28–29
D PROOF OF THE WEIGHTED SHARED-SUBSPACE CHARACTERIZATION
The proof characterizes the optimal shared mode-1 subspace through weighted concatenation, and the following section outlines derivations for budget-coupled rank–sparsity allocation.
D PROOF OF THE WEIGHTED SHARED-SUBSPACE CHARACTERIZATION
- For fixed shared mode-1 factors, the optimal coefficients follow projection, and the optimal subspace is the leading left singular subspace of the weighted concatenation.This formally justifies weighted concatenation; the full Tucker representation extends sharing to both compressed modes while retaining the third mode in full.Source: D PROOF OF THE WEIGHTED… · pp. 29–30
E ADDITIONAL DETAILS FOR BUDGET-COUPLED RANK–SPARSITY ALLOCATION
- The allocation derivations establish fixed-subspace error decomposition and optimal top-K_l core projection, then use these results for budget-coupled rank search.Gradual pruning and support-constrained refinement are described separately.Source: E ADDITIONAL DETAILS FOR… · p. 30
E.1 FIXED-SUBSPACE ERROR DECOMPOSITION
The appendix decomposes approximation error into subspace and latent-core terms, then uses that structure to score rank pairs and interpret core sparsity. Candidate scoring reuses common singular subspaces and projection cores before the selected representation undergoes gradual refinement.
E.1 FIXED-SUBSPACE ERROR DECOMPOSITION
- The total approximation error separates into selected Tucker-subspace error and projection-core approximation error, with the latter evaluable directly in latent core space.Source: §E.1 · p. 30
E.2 OPTIMAL CARDINALITY PROJECTION OF THE CORE
- For fixed orthonormal factors and ranks, retaining the Kl largest-magnitude projection-core entries gives the optimal cardinality-constrained core approximation.This result is exact for the selected subspace; later refinement may adjust core values, support, and Tucker factors.Source: §E.2 · p. 31
E.3 INTERPRETATION OF LATENT-CORE SPARSITY
- Sparse latent-core coefficients select rank-one interactions between shared factor directions, while reconstructed Q/K/V matrices are generally dense.Each Q/K/V slice has its own core coefficients and may activate a different subset; Kl counts active interactions, not nonzero reconstructed-matrix entries.Source: §E.3 · pp. 31–32
- Core sparsity yields computational benefits only when implementations exploit the sparse latent interaction structure instead of first materializing dense Q/K/V matrices.Source: §E.3 · p. 32
E.4 EFFICIENT EVALUATION OF CANDIDATE RANK PAIRS
- A common HOSVD initialization reuses nested singular subspaces and projection cores to evaluate feasible rank pairs without reconstructing each Tucker approximation.The implementation uses one pair of mode-wise decompositions and one maximal core projection per layer, followed by energy and top-Kl evaluations over the candidate grid.Source: §E.4 · pp. 32–33
- The top-Kl projection is only a rank-configuration proxy; after selection, gradual sparse-core refinement progressively sparsifies and refits the core.Source: §E.4 · p. 33