PaperlayerView exact revision on arXiv

Source-linked AI summary

CHASE: Channel-Aligned Structure Exploitation for Geometry-Aware Model Engineering

Wei Wang, Wei Jiang, Ziran Liu

arXiv:2610.09476v1cs.LGcs.CL

TL;DR

Trained networks exhibit useful structural regularities that are not directly specified by their architectures or training objectives. CHASE applies GSA measurements to six model-design applications, and the reported results show improvements across its new attention, cache-compression, and pruning methods.

  • Problem

    Trained networks develop structural regularities that are not directly specified by their architectures or training objectives.

    Source: §1 · p. 1

  • Method

    CHASE uses GSA measurements of spectral concentration, physical-channel alignment, support structure, and singular-basis changes to guide six applications across model modification, reconfiguration, and compression.

    Source: §5 · p. 16

  • Results

    CORA, COEC, and CORAM have shown effectiveness for their applications, while CAGA substantially improves MHA-to-GQA conversion and SAKV and CAPS improve over their respective baselines.

    Source: §5 · p. 16

  • Takeaways & Limitations

    GSA-identified structures can be used directly to design methods for a range of model operations.

    Source: §5 · p. 16

    The support-stability certificate is derived for fixed-dimensional perturbations, while CHASE uses measured structure to guide dimension-changing operations.

    Source: §2.3 · p. 4

Abstract

from arXiv · show

Geometric and Spectral Alignment (GSA) characterizes trained networks through spectral concentration, physical-channel alignment, support structure, and changes in singular bases. In this paper, we propose CHASE (Channel-Aligned Structure Exploitation) to use these structures in practical model design. CHASE covers six applications across model modification, reconfiguration, and compression. CORA, COEC, and CORAM apply GSA to parameter-efficient finetuning, structured-pruning compensation, and model merging. We further develop three new methods. CAGA uses GSA to identify multi-head attention heads that can share a KV representation and constructs the shared key and value heads through geometric alignment and low-rank subspace extraction. SAKV uses GSA to determine which adjacent layers can share a low-rank KV-cache representation and the retained rank for each layer group. CAPS uses GSA spectral structure to group output neurons and selects retained input channels separately for each group. Results from CORA, COEC, and CORAM establish the effectiveness of GSA for adaptation, pruning compensation, and model merging. Experiments on CAGA show that geometric shared-head construction substantially improves MHA-to-GQA conversion, and SAKV and CAPS improve over representative baselines for KV-cache compression and structured pruning. These results show that the structures identified by GSA can be used directly to design methods for a range of model operations.

1 INTRODUCTION

Trained networks exhibit structural regularities that concentrate useful computation in a relatively small set of directions and interactions. CHASE uses GSA measurements to apply those structures across model modification, reconfiguration, and compression.

1 INTRODUCTION

  • Trained networks can concentrate activation magnitude, knowledge, behavior, and removable or shareable computation in a small number of channels, parameters, or directions.These observations motivate a shared-structure view of trained networks.Source: §1 · p. 1
  • GSA characterizes spectral concentration and alignment through physical channels, and studies how these structures change when weights are modified.Effective-rank windows identify low-dimensional subspaces; GSA-II relates adjacent-layer subspaces through physical-channel supports and distinguishes core, overlap, and noise.Source: §1 · pp. 1–2
  • CORA, COEC, and CORAM apply GSA to parameter-efficient finetuning, structured-pruning compensation, and model merging.They use coherent singular-basis changes, compensate for pruning-induced basis and spectral changes, and separately represent rotational, spectral, and right-subspace factors.Source: §1 · p. 2
  • CHASE presents six applications, including CAGA for attention-head sharing, SAKV for shared KV-cache representations, and CAPS for structured-pruning selection.CAGA constructs shared key and value heads through geometric alignment and low-rank subspace extraction; SAKV selects layer groups and retained ranks, while CAPS groups output neurons and selects input channels per group.Source: §1 · pp. 2–3

2 GEOMETRIC AND SPECTRAL STRUCTURES USED IN CHASE

CHASE uses spectral concentration, physical-channel alignment, support structure, and singular-basis changes to guide model operations. The section explains how these measurements characterize layers, channel relationships, support reliability, and structured weight changes.

2 GEOMETRIC AND SPECTRAL STRUCTURES USED IN CHASE

  • CHASE maps effective rank to SAKV and CAPS, channel-support overlap and spectral or subspace similarity to CAGA, and singular-frame changes to CORA, COEC, and CORAM.The complete theoretical development and proofs for these GSA structures are given in GSA-I and GSA-II Liu et al. (2026a;b).Source: §2 · p. 3, §2.4 · p. 5

2.1 SPECTRAL CONCENTRATION AND EFFECTIVE RANK

  • The effective-rank window measures how many singular directions retain a chosen fraction of spectral energy, providing a low-dimensionality signal for SAKV rank allocation and CAPS neuron grouping.Spectral energy alone does not measure task importance, so applications requiring task-dependent selection add calibration information.Source: §2.1 · p. 3

2.2 INTER-LAYER ALIGNMENT IN PHYSICAL CHANNELS

  • GSA-II compares adjacent layers’ retained output and input subspaces in physical-channel coordinates, identifying active supports and separating group-specific core connections, shared overlap, and weakly structured noise.Channels present in multiple active supports are hub channels; CAGA combines support overlap with spectral and right-subspace similarity when selecting heads to share.Source: §2.2 · pp. 3–4

2.3 STABILITY OF THE CHANNEL STRUCTURE

  • Active-column gaps, overlap margins, and residual bounds form a certificate for support stability under specified perturbations, but the certificate is derived for fixed-dimensional perturbations.CHASE uses measured structure to guide dimension-changing operations.Source: §2.3 · p. 4

2.4 COHERENT SINGULAR-FRAME MOTION

  • Under the theory’s low-disruption conditions, small model changes favor coherent rather than independent motion of the left and right singular frames.Representing changes as singular-basis rotations and spectral-value changes provides a structured parameterization for modifying, compensating, or combining pretrained weights.Source: §2.4 · p. 5

3 CHANNEL-ALIGNED STRUCTURE EXPLOITATION

This section describes how Geometric and Spectral Alignment structures inform six model operations spanning modification, reconfiguration, and compression, through CORA, COEC, CORAM, CAGA, SAKV, and CAPS. These methods use structure for tasks including adaptation, model merging, KV-cache sharing, and grouped input-channel selection.

3.1 CORA: COHERENT SINGULAR-BASIS ROTATION FOR PEFT

  • CORA combines coherent rotations of both singular bases with separate singular-value changes, matching or exceeding evaluated LoRA and SVD-based PEFT baselines with fewer trainable parameters.It achieves 82.16% average accuracy on LLaMA-2-7B with 6.6M trainable parameters and 86.65% on LLaMA-3-8B with 10.3M; limiting adaptation to the leading singular subspace is not equally effective across tasks, and using the full pretrained weight recovers some performance loss.Source: §3.1 · p. 6

3.2 COEC: TWO-SIDED GEOMETRIC COMPENSATION FOR STRUCTURED PRUNING

  • At 30% structured sparsity, COEC reduces WikiText-2 perplexity for every evaluated model under Wanda-sp, FLAP, and RCPU selections; under Wanda-sp on Qwen2.5-32B, perplexity falls from 13.90 to 9.64.COEC corrects both singular bases and singular values; it modifies only retained weights, preserving pruning’s parameter, memory, and FLOP reductions without added inference computation.Source: §3.2 · pp. 6–7

3.3 CORAM: GEOMETRY-COMPATIBLE AGGREGATION FOR MODEL MERGING

  • CORAM separates and merges singular-basis and singular-value changes, improving over OrthoMerge by 0.25–1.35 points and matching or exceeding Task Arithmetic, TIES, and DARE in reported experiments.Its update-derived λ is within 0.72 points of the best value found by evaluating multiple λ choices.Source: §3.3 · pp. 7–8

3.4 CAGA: COHERENT ALIGNMENT FOR GROUPED ATTENTION

  • CAGA groups heads using physical-channel support overlap, spectral similarity, and right-subspace similarity, then constructs shared KV projections through RoPE-compatible key alignment and low-rank value-subspace extraction.Head grouping has a relatively small effect once shared heads are constructed properly; CAGA retains standard GQA structure without additional inference operations.Source: §3.4 · pp. 9–10
  • At G = 16, CAGA’s aligned-key and shared-value construction improves WikiText-2 perplexity from 926 to 25.1 and seven-task zero-shot average from 32.6 to 44.3 before continued training.The same construction yields usable models at G = 8 after continued training, while all tested methods fail at G = 4.Source: §3.4 · p. 10

3.5 SAKV: SPECTRALLY ALIGNED KV-CACHE COMPRESSION

  • SAKV uses measured spectral structure to choose adjacent-layer groups and retained key and value ranks, addressing fixed grouping and rank allocation that can mismatch variation across depth.It uses a small calibration set only to select groups and ranks, without training or modifying model parameters; for RoPE models, it factorizes keys before rotation and values directly.Source: §3.5 · pp. 10–11
  • SAKV partitions contiguous layers into groups and selects the smallest key and value ranks that meet per-layer reconstruction-error thresholds, adjusting thresholds to reach the target compression ratio.For keys, it measures error after RoPE and removes the component shared across token positions, normalizing by the variation visible to attention; the calibrated groups and ranks are fixed for inference.Source: §3.5 · pp. 11–12
  • At 64K on Llama-3.1-8B, SAKV scored 85.48 at 12× and 79.66 at 16× compression; across 20 matched RULER settings, it was significantly better than xKV in 14 and tied in 6.At 64K on Qwen2.5-14B, SAKV improved over xKV by 4.64 points at 12× and 18.92 points at 16× compression.Source: §3.5 · p. 12

3.6 CAPS: CHANNEL-ALIGNED PRUNING SELECTION

  • CAPS groups output neurons around distinct SVD-derived anchors, then retains input channels separately per group using weight-strength and activation scores, enabling smaller dense operations without deleting a channel layerwide.The same input channel can be kept for some groups and pruned for others; hardware reduction depends on how groups with different channel sets are tiled.Source: §3.6 · pp. 12–13
  • CAPS had the lowest WikiText-2 perplexity at all 12 model–budget settings, highest ZS-7 accuracy at 11, and highest MMLU accuracy at 10; at 20–30% MAC reduction, it beat the strongest competitor by 0.50–1.01 perplexity, 2.94–7.29 ZS-7 points, and 7.59–14.60 MMLU points.Gains were largest on the three GQA models; on Llama-2-7B, CAPS still had lower perplexity at all three pruning levels.Source: §3.6 · p. 14

4 RELATED WORK

The related work spans adaptation, pruning, merging, attention reconfiguration, and KV-cache compression. CHASE's applications use geometric and spectral structure to address limitations in these areas.

Parameter-efficient finetuning

  • CORA couples left- and right-singular-basis rotations while changing singular values, following coherent basis motion for parameter-efficient finetuning.This differs from unrestricted low-rank updates and one-sided orthogonal transformations.Source: §4 · p. 14

Structured pruning

  • CAPS groups output neurons by leading singular directions and selects retained input channels separately for each group, keeping computation dense within groups.MAC reduction depends on how the groups map to implementation row tiles.Source: §4 · p. 15

Post-pruning compensation

  • COEC jointly corrects both singular bases and adjusts singular values after pruning, without retraining or changing the reduced model dimensions.Existing compensation approaches described here leave weights unchanged, add a bias, or apply a one-sided rotation.Source: §4 · p. 15

Model merging

  • CORAM represents finetuned weights through rotational, spectral, and right-subspace changes, then merges those components separately.This treats singular-value and subspace-orientation changes according to their own geometry rather than as one weight-space update.Source: §4 · p. 15

MHA-to-GQA conversion

  • CAGA groups attention heads using physical-channel support overlap and retained spectral and subspace similarities, then constructs shared KV projections through alignment and low-rank extraction.The resulting layer retains standard GQA form without additional inference-time transformations.Source: §4 · pp. 15–16

KV-cache compression

  • SAKV selects contiguous layer groups and separate key and value ranks using reconstruction errors, rather than fixed sharing patterns and ranks.This extends prior KV-cache compression work that primarily exploits within-layer or cross-layer redundancy.Source: §4 · pp. 14, 16

5 CONCLUSION

CHASE uses GSA-identified structures across six applications in model modification, reconfiguration, and compression. The reported results support applying these structures directly to practical model operations.

Geometric and spectral structure

  • GSA characterizes trained networks through spectral concentration, physical-channel alignment, support structure, and singular-basis changes.CHASE applies these identified structures in model modification, reconfiguration, and compression.Source: §5 · p. 16

Applications

  • CHASE presents six applications: CORA, COEC, CORAM, CAGA, SAKV, and CAPS, each using GSA structures for a different model operation.Their methods address finetuning, pruning compensation, merging, attention conversion, KV-cache compression, and structured pruning.Source: §5 · p. 16

Results

  • CORA, COEC, and CORAM report effectiveness for finetuning, pruning compensation, and model merging, while CAGA, SAKV, and CAPS improve over respective baselines.CAGA improves MHA-to-GQA conversion through geometric shared-head construction.Source: §5 · p. 16

A DETAILED EXPERIMENTS FOR CAGA

The CAGA experiments examine Llama-2-7B conversion at G=16 and G=8, with shared-head construction compared before continued training and grouping methods compared after training.

A DETAILED EXPERIMENTS FOR CAGA

  • Experiments use Llama-2-7B with 32 query heads and evaluate conversion to 16 or 8 KV heads; calibration uses 128 C4 sequences of 2,048 tokens.The calibration samples determine head statistics, grouping, and key-alignment transformations; evaluation sets are not used during conversion.Source: A DETAILED EXPERIMENTS FOR CAGA · pp. 19–20

A.1 EFFECT OF SHARED-HEAD CONSTRUCTION

  • 926 to 25.1 WikiText-2 perplexity and 32.6 to 44.3 seven-task zero-shot average: CAGA outperforms direct averaging at G=16 before continued training.The table covers Llama-2-7B at G=16 using CAGA head grouping.Source: Table 2 · p. 20, §A.1 · p. 20

A DETAILED EXPERIMENTS FOR CAGA

  • The experiments compare neighboring-head grouping, Jin et al. (2025)'s cosine/value-based grouping, and CAGA grouping, alongside direct averaging and Procrustes-based shared-head construction.CAGA aligns keys with RoPE-compatible transformations and builds shared values from a rank-d_h approximation of grouped value matrices.Source: A DETAILED EXPERIMENTS FOR CAGA · p. 20

A.2 GROUPING COMPARISON AND CONTINUED TRAINING

At G=8, the grouping methods perform similarly after continued training, while the experiment holds shared-head construction fixed to compare grouping choices.

Grouping comparison and continued training

  • The comparison changes only head grouping: neighboring-head, Jin et al. (2025)'s cosine/value-based method, and CAGA grouping all use CAGA shared-head construction.Evaluation reports WikiText-2 and C4 perplexity and a seven-task zero-shot average after 100M and 500M training tokens.Source: §A.2 · p. 20
  • For G=8, continued training uses SlimPajama data, full-parameter AdamW updates, and knowledge distillation from the original Llama-2-7B model.The setup uses a 2 × 10^-5 learning rate, weight decay 0.1, 5% warmup, cosine decay, and 128 sequences of 2,048 tokens per update.Source: §A.2 · p. 20
  • After 500M tokens, grouping methods yield WikiText-2 perplexity from 6.64 to 6.75 and zero-shot average from 53.49 to 54.36; construction has the larger effect.At both training budgets, the three grouping methods perform similarly.Source: §A.2 · p. 21

B.1 EXPERIMENTAL SETUP

SAKV is evaluated on two instruction-tuned models using long-context benchmarks at matched cache-compression ratios. Calibration selects layer partitions and key/value ranks without changing model parameters.

B.1 EXPERIMENTAL SETUP

  • SAKV is evaluated on Llama-3.1-8B-Instruct and Qwen2.5-14B-Instruct-1M using RULER at 32K, 64K, and 128K contexts.SAKV and xKV use the same prompts and are compared at matched compression ratios, approximately 4×, 8×, 12×, and 16× when supported.Source: §B.1 · p. 21
  • LongBench evaluation at 4× and 8× compression tests whether RULER behavior transfers to a broader set of long-context tasks.The passage does not report LongBench outcomes.Source: §B.1 · p. 21
  • Calibration uses four PG-19 sequences to select each contiguous layer group and its key/value ranks, with gmax = 8 and separate key and value thresholds.The resulting groups and ranks are fixed for evaluation; calibration trains or modifies no model parameters.Source: §B.1 · p. 21

B.2 COMPARISON WITH XKV

Compared with xKV, SAKV’s advantage on RULER grows at higher compression, while the reported LongBench settings show no clear separation. Paired analysis finds SAKV significantly better in most matched RULER settings and worse in none.

B.2 COMPARISON WITH XKV

  • At 64K on Llama-3.1-8B, SAKV scores 85.48 at 12× and 79.66 at 16× compression, while xKV falls to 63.17 and 35.10, respectively.SAKV at 12× is 3.02 points below xKV at 8×; xKV loses 25.33 points moving from 8× to 12×.Source: §B.2 · p. 21
  • On Qwen2.5-14B at 64K, the SAKV–xKV difference grows from 2.77 points at 8× compression to 4.64 at 12× and 18.92 at 16×.The largest advantage occurs at high compression, where fixed layer grouping and rank allocation begin to lose substantial long-context accuracy.Source: §B.2 · p. 21
  • SAKV is significantly better in 14 of 20 matched RULER settings, statistically indistinguishable in 6, and worse in none.The indistinguishable cases occur primarily at 4× compression, where both methods remain close to the uncompressed model.Source: §B.2 · p. 21

B.3 LONGBENCH PERFORMANCE

  • At 4× and 8× compression on LongBench, both methods remain close to the uncompressed model, with no clear separation between them.The LongBench results contrast with the clearer SAKV advantage on RULER at higher compression.Source: §B.3 · p. 21

B.4 ADDITIONAL ANALYSIS OF SAKV

SAKV’s analysis separates the roles of layer grouping and adaptive rank allocation and examines key/value tolerances, key reconstruction criteria, and calibration choices. Its design choices matter differently across compression levels and model settings.

Layer grouping versus rank allocation

  • At 64K and 8× compression, SAKV groups paired with xKV ranks score 89.81 on Llama-3.1-8B and 90.18 on Qwen2.5-14B, versus xKV’s 88.50 and 87.49.Much of the improvement at this compression ratio comes from replacing fixed four-layer grouping; rank allocation matters increasingly at higher compression as rank must be distributed more selectively across depth.Source: §B.4 · pp. 21–22

Separate key and value tolerances

  • At 64K and 8× compression on Llama-3.1-8B, separate key and value tolerances score 89.62, versus 10.31 with one shared tolerance.Keys and values exhibit different spectral decay, making one reconstruction threshold unsuitable for both cache types.Source: §B.4 · p. 22

RoPE-aware key criterion

  • On Qwen2.5-14B at 64K, the RoPE-aware and Frobenius key criteria differ little at 8× compression but substantially at 12× and 16×.Early-layer token-independent key components can dominate Frobenius energy even though remaining variation becomes important after RoPE.Source: §B.4 · p. 22, Table 6 · p. 23
  • The RoPE-aware criterion is not uniformly better: at 32K and 16× compression on Llama-3.1-8B, Frobenius scores 89.36 versus 79.89.The authors use RoPE-aware reconstruction as a single criterion intended to remain reliable across both model families, particularly when offset-dominated keys challenge Frobenius.Source: §B.4 · p. 22

Maximum group size

  • Increasing gmax from 2 to 4 and 8 raises the 64K, 8× Llama-3.1-8B RULER score from 84.37 to 87.56 and 89.62.The main experiments use gmax = 8.Source: §B.4 · pp. 22–23

Shared and separate key/value partitions

  • At 64K and 8× compression, shared versus separate key/value layer partitions score 89.71 versus 89.75 on Llama-3.1-8B and 90.26 versus 90.30 on Qwen2.5-14B.Separate partitions provide little additional benefit.Source: §B.4 · p. 23

Calibration length

  • For Llama-3.1-8B evaluated at 64K and 8× compression, calibration at 32K, 64K, and 128K yields RULER scores of 89.57, 89.62, and 89.00.The grouping and rank allocation are not sensitive to matching calibration length exactly to evaluation length.Source: §B.4 · p. 23

C ADDITIONAL EXPERIMENTS FOR CAPS

CAPS experiments evaluate pruning across four language models and seven transformer projections per block. The setup specifies spectral grouping, calibration, target-MAC search, recovery training, baselines, and evaluation measures.

Models

  • CAPS is evaluated on Llama-2-7B, Llama-3-8B, Mistral-7B, and Qwen2.5-7B, pruning seven linear projections in every transformer block.Embeddings, normalization layers, and the language-model head remain unchanged.Source: C ADDITIONAL EXPERIMENTS FOR CAPS · p. 23

CAPS configuration

  • CAPS sets ε = 0.4, groups output neurons using singular directions containing 60% of spectral energy, and removes redundant anchors above 0.85 absolute cosine similarity.For each direction, four output neurons with the largest absolute coordinates are anchor candidates; remaining neurons are assigned to the most similar non-redundant anchor.Source: C ADDITIONAL EXPERIMENTS FOR CAPS · p. 23

Recovery training

  • CAPS uses bisection on ρ to reach target MAC reduction within 0.2 percentage points, then runs one recovery-training phase with removed connections fixed at zero.Recovery budgets are 34.1M, 66.6M, and 99.9M tokens at 10%, 20%, and 30% MAC reduction, respectively; the Llama-2-7B baselines at 30% use about 1% more tokens.Source: C ADDITIONAL EXPERIMENTS FOR CAPS · p. 23

Calibration

  • All methods use WikiText-2 calibration from 128 windows of 4096 tokens, totaling 524,288 tokens, with seed 0.For CAPS, the activation term is computed from input-channel activations.Source: C ADDITIONAL EXPERIMENTS FOR CAPS · p. 23

Baselines

  • Whole-unit baselines remove all weights associated with selected units, whereas CAPS removes an input channel only for selected output-neuron groups and retains it for others.For GQA models, baseline attention pruning uses KV-group granularity and may miss nominal MAC targets; CAPS adjusts ρ to reach the target directly.Source: C ADDITIONAL EXPERIMENTS FOR CAPS · p. 24

Evaluation

  • Evaluation reports WikiText-2 test perplexity, ZS-7 accuracy, and 5-shot MMLU macro accuracy, with comparisons made within the same model and evaluation set.Each model–method–budget configuration is run once; calibration uses seed 0 and recovery training uses seed 1234.Source: C ADDITIONAL EXPERIMENTS FOR CAPS · p. 24

C.1 RESULTS

CAPS leads on perplexity across all tested model–budget settings and on most accuracy settings, with its largest advantages on GQA models. Its comparison with whole-unit baselines is made separately for each metric.

C.1 RESULTS

  • CAPS has the lowest WikiText-2 perplexity at all 12 model–budget settings, the highest ZS-7 accuracy at 11, and the highest MMLU accuracy at 10.The differences are particularly large on the three GQA models.Source: §C.1 · p. 24
  • At 20–30% MAC reduction on Llama-3-8B, Mistral-7B, and Qwen2.5-7B, CAPS improves over the strongest baseline by 0.50–1.01 perplexity, 2.94–7.29 ZS-7 points, and 7.59–14.60 MMLU points.On Llama-2-7B, CAPS retains lower perplexity at all three pruning levels, while ZS-7 and MMLU differences are smaller.Source: §C.1 · p. 24

C.2 COMPARISON WITH THE BEST WHOLE-UNIT BASELINE

  • Table 8 compares CAPS with the strongest whole-unit baseline separately for each metric; lower ΔPPL and higher ΔZS-7 and ΔMMLU are better.The strongest baseline can differ across metrics.Source: §C.2 · p. 24, Table 8 · p. 25

C.3 COMPUTATION ACCOUNTING

CAPS’s reported MAC reduction is based on selected input channels, but accelerator row tiling can reduce realizable savings when groups use different channels.

C.3 COMPUTATION ACCOUNTING

  • CAPS computes MAC reduction from selected input channels relative to dense versions of the seven target linear layers.Whole-unit baselines compute MAC counts from the dimensions of their reduced matrices.Source: §C.3 · p. 25
  • When accelerator row tiles combine groups with different selected inputs, computation uses their channels’ union, so realizable MAC reduction depends on supported row granularity.Reordering rows does not change model output.Source: §C.3 · p. 25
  • At the nominal 30% setting, T=2 retains 24.2–24.5% MAC reduction across the four models, and the reduction shrinks as tile size increases.Rows are reordered by overlap in selected input channels before tiling.Source: §C.3 · p. 25, Table 9 · p. 26
Loading 2610.09476v1…