PaperlayerView exact revision on arXiv

Source-linked AI summary

LaMoC: Loss-Aware Modular Compression for LLMs

Mohanad Odema, Jacob Song

arXiv:2608.30226v1cs.AIcs.CLcs.PF

TL;DR

Existing modular compression methods mainly use activation statistics, leaving module-level loss sensitivity underexplored. LaMoC blends activation and Empirical Fisher statistics through gradient-error alignment and solves the resulting blending problem empirically. Across four model families and eight models, it improves language modeling and downstream task accuracy over state-of-the-art modular compression methods.

  • Problem

    Existing joint modular compression primarily relies on activation statistics, leaving module-level loss-sensitivity characterization underexplored.

  • Method

    LaMoC blends activation and gradient-weighted Empirical Fisher statistics through a two-tiered optimization that selects per-module blending rates.

  • Results

    LaMoC improves language modeling and downstream task accuracy across four model families and eight models, with averages of 2.5% and 1%, respectively.

  • Takeaways & Limitations

    Gradient-error alignment provides a loss-aware modular compression methodology that improves over state-of-the-art modular compression pipelines across benchmarks and compression rates.

  • Takeaways & Limitations

    The methodology relies on empirical observations, a small subset of candidate γ values, and heuristic γ selection, while analytical optimization and broader validation remain future work.

Abstract

from arXiv · show

Modular compression has enabled considerable parameter reduction in LLMs while preserving strong language understanding and downstream task accuracy. However, existing joint modular compression methods primarily rely on activation statistics, leaving loss-sensitivity information and its module-level characterization underexplored. We investigate addressing this gap with LaMoC, a loss-aware modular compression methodology that blends activation and Empirical Fisher statistics through gradient-error alignment. LaMoC improves joint compression by selecting compression statistics that better align local module reconstruction error with the downstream loss. Our contributions are three-fold: (1) We characterize the Empirical Fisher as a module-level loss-aware proxy that can be blended with the activation statistics required for compression. (2) We reformulate joint modular compression as a two-tiered optimization problem that minimizes module reconstruction error while tuning the activation and gradient information blending rate. (3) We implement an empirically driven methodology with statistical validation to solve the resulting compression problem. We evaluate LaMoC across four model families spanning eight models. On the 4-8B models, LaMoC achieves an average 2.5% reduction in perplexity and a 1% relative improvement in task accuracy over state-of-the-art modular compression methods.

1 Introduction

LLM parameter reduction supports efficient deployment, but modular compression remains predominantly activation-driven and underexplores module-level loss sensitivity. LaMoC addresses this gap by blending gradient-weighted statistics with activation statistics through a practical optimization methodology, improving compression outcomes across models.

  • Motivation: Parameter reduction improves LLM efficiency and memory footprint through pruning and low-rank approximation.Pruning removes weights or structured components, while low-rank approximation replaces dense matrices with smaller factors.
  • Motivation: Modular compression jointly compresses related weight groups, but existing methods primarily rely on activation-driven reconstruction objectives.This approach uses module-level reconstruction while tailoring compression operators to coupled matrices and their functional roles.
  • Research gap: Loss-sensitivity information remains underexplored in modular compression, motivating questions about its systematic incorporation, module-level characterization, and practical optimization.The introduction frames these issues as three research questions concerning performance, blended statistics, and loss-aware optimization.
  • Contributions: LaMoC characterizes module-level loss sensitivity with gradient-weighted second-order activation statistics and blends them with activation statistics.The method treats this blending as a per-module design choice within modular compression.
  • Contributions: LaMoC formulates compression as a two-tiered optimization that minimizes modular reconstruction loss while selecting per-module blending rates.An empirically driven, statistically validated methodology uses fixed and adaptive strategies to control gradient-information blending.
  • Results: 2.5% average improvement in language-modeling perplexity and 1% average improvement in downstream task accuracy are reported across 4 model families and 8 models.The reported improvements are relative to state-of-the-art modular compression pipelines.

2 Background

Prior work combines parameter reduction with modular, activation-aware reconstruction, while loss-aware methods generally operate on individual parameters or matrices. Joint decomposition applies module-specific low-rank or pruning operations to coupled transformer weights, using activation statistics to rank dimensions and construct reduced matrices.

  • Related works: Parameter reduction commonly uses pruning or low-rank approximation to remove or approximate less important parameters.Pruning can target layers, modules, channels, heads, or hidden dimensions, while low-rank approximation replaces dense matrices with factors.
  • Related works: Modular compression applies pruning or low-rank approximation to structured groups of weight matrices according to functional and architectural properties.It introduces a module-level reconstruction objective for coupled matrices whose outputs interact through nonlinear operations and downstream transformations.
  • Related works: Activation-aware compression reconstructs output feature representations using calibration-derived second-order activation statistics.These Gram autocorrelations guide weight projection or dimension-importance selection, but may not capture downstream-loss effects.
  • Related works: Loss-aware compression uses loss-derivative information, including Hessian- or Fisher-based estimates, to guide pruning and low-rank reconstruction.The supplied motivation identifies the combination of such information with modular compression as underexplored.
  • Preliminaries: Joint decomposition defines transformer dimensions, projection matrices, and activation Gram matrices C = X^⊤X for module-specific compression.The notation specifies hidden, head, and intermediate dimensions along with RoPE-transformed attention activations.
  • MLP module: The MLP module prunes shared intermediate dimensions by ranking activation-Gram-based scores and selecting the top-r dimensions.The resulting selection matrix is applied to the up and gate projections, while the corresponding transpose is used for the down projection.
  • QK module: The QK module truncates query and key matrices along the head dimension using per-head channel-importance scores derived from activation correlations.The scores yield selection matrices that retain the most important channels for each head.
  • VO module: The VO module jointly compresses value and output projections through successive SVD operations using activation Gram statistics collected at the value-projection input.The procedure produces low-rank approximations for both projection matrices.

3 LaMoC: Loss-Aware Modular Compression for LLMs

LaMoC adds module-level loss sensitivity to modular compression by blending canonical activation and Fisher-weighted statistics, then selecting module-specific blending rates through a two-tiered objective. Its reference-gradient proxy supports practical candidate evaluation while preserving strong correlation with measured loss changes.

  • Gradient-Error Alignment: LaMoC blends canonical activation statistics with Fisher-weighted activation statistics to align module reconstruction with downstream loss sensitivity.The blending coefficient γ controls the degree of gradient information, with γ = 1 recovering canonical activation statistics and γ < 1 emphasizing loss-sensitive channels or energy directions.
  • Gradient-Error Alignment: The Fisher weighting uses a diagonal empirical Fisher approximation that treats per-channel gradient magnitudes as importance scores while ignoring cross-channel gradient correlations.These scores define the Fisher-weighted activation Gram used in the effective statistic.
  • Gradient-Error Alignment: The effective Gram matrix rescales the Fisher-weighted statistic to match the canonical statistic’s total energy before blending.This trace-ratio rescaling keeps the canonical and Fisher-weighted components on a matched energy scale.
  • Problem Formulation: LaMoC formulates compression as a two-tiered optimization that minimizes module reconstruction loss while selecting a module-specific γ that minimizes predicted cross-entropy degradation.The search spans layers, modules, and viable γ candidates, creating a combinatorial optimization problem.
  • Statistical Validation: Pearson correlation reaches 0.91±0.28 for layerwise proxy estimates and remains 0.913 after compounded predictions across modules at 15% compression.The reported correlation varies by layer because the proxy depends on calibration settings and a first-order reference-gradient approximation.
  • Practical Method: The heuristic selects γ only when the proxy predicts lower cross-entropy than canonical compression; otherwise, it assigns γ = 1.Selected module-specific effective Grams are then used to assemble the gradient-aligned compressed model.

4 Experiments

LaMoC is evaluated as a training-free, loss-aware extension of modular compression across multiple models, compression rates, and downstream benchmarks. It improves language modeling and task accuracy over modular-compression baselines, with stronger gains generally appearing for smaller models and more aggressive compression.

  • Setup: LaMoC builds on MoDeGPT using MLP, QK, and VO module groupings, with fixed or adaptive γ selection for gradient-information blending.The adaptive strategy selects γ⋆ from candidate values by minimizing expected cross-entropy loss.
  • Language Modeling: LaMoC improves perplexity across all models and compression settings, averaging a 2.46% relative reduction versus MoDeGPT.Absolute perplexity reduction averages 0.8, with a 0.07–6.11 range.
  • Language Modeling: 3.39% average relative perplexity reduction is achieved by Qwen3-4B-It-2507, while smaller Llama models reach up to 3.20% average reduction.The Qwen3 model also has an average absolute perplexity reduction of 1.83.
  • Compression Rate and γ Strategy: 2.73 absolute perplexity reduction occurs at 50% compression, compared with 0.23 at 20%, while adaptive γ reaches 2.42% relative reduction.The adaptive strategy exceeds the two fixed strategies, which reach 1.42% and 1.52%, respectively.
  • Task Proficiency: +0.99% relative task-accuracy improvement is obtained across benchmarks, including +3.89% on 5-shot MMLU and +0.71% on 0-shot accuracy.The corresponding absolute gains are +0.53, +1.65, and +0.40 percentage points, respectively.
  • Per-Module Contribution: MLP + VO produces the largest positive per-module performance change on both evaluation suites for Qwen3-4B-It-2507.This comparison covers 20% and 40% compression.
  • Ablations: Trace rescaling raises average perplexity improvements from 0.19% to 3.16% and task-accuracy gains from 0.02 to 0.40 percentage points.The ablation evaluates Llama-3.2-1B and Qwen3-4B-It under adaptive γ⋆.
  • Robustness: Repeated sampling preserves a 0-shot accuracy improvement of +0.76 ± 0.26 percentage points across three calibration seeds.The evaluation uses Qwen3-4B-Instruct-2507 at 20% and 40% compression.

5 Discussion

The discussion argues that combining activation awareness, loss information, non-uniform truncation, and modularity yields additional compression gains. It also presents LaMoC as complementary to activation-statistics-based frameworks while identifying analytical γ selection as future work.

  • Performance Gains: Stacking activation awareness, loss curvature information, non-uniform truncation, and modularity unlocks further performance gains.LaMoC demonstrates this combination within training-free modular compression.
  • Generality: LaMoC is complementary to compression techniques because it modifies activation statistics and can scale to frameworks relying on them.The discussion states that the proposal can be incorporated into any such framework.
  • Future Directions: An analytical method for selecting optimal γ per module remains a future direction because the current approach relies on empirical observations and heuristic selection.The present exploration covers only a small subset of candidate γ values.

6 Conclusion

LaMoC characterizes loss-aware modular compression and injects loss sensitivity through gradient-weighted Gram matrices. Its experiments show improvements over state-of-the-art modular compression methods across benchmarks and compression rates.

  • Method: LaMoC injects loss sensitivity through gradient-weighted Gram matrices within a two-tiered optimization over module reconstruction and gradient blending.The method is presented as a gradient-error-aligned methodology for modular LLM compression.
  • Results: LaMoC improves over state-of-the-art modular compression methods across benchmarks and compression rates.This is the conclusion’s reported experimental finding.

Limitations

The paper’s empirical validation spans models from 1B to 33B parameters, but broader scale and architectural coverage remain open. Future evaluation is needed for models at or beyond 30B parameters, emerging architectures, and additional task types.

  • Scale: The method has been tested on models ranging from 1B to 33B parameters, leaving broader validation at or beyond the ≥30B tier needed.The stated scope boundary concerns the extent of experimental validation.
  • Architectures and Tasks: Evaluation on Hybrid, State-space, linear-attention, and Mixture-of-Experts architectures remains future work.The paper also identifies long-context reasoning and agentic tool calling as unevaluated task areas.
  • Methodological Scope: The methodology is derived from empirical observations, and its γ selection is heuristic rather than theoretically analytical.The paper identifies deriving a theoretically proven analytical solution for γ as important future exploration.

C.3 Adaptive γ Selection

LaMoC evaluates adaptive γ selection by comparing proxy-based choices with oracle layer-wise configurations and by testing its behavior across models, calibrations, and compression settings.

  • Selection policy: Adaptive γ⋆ selects the candidate with the lowest predicted cross-entropy change among options passing significance and magnitude thresholds.Blending requires a predicted reduction at least k standard errors below zero and larger than ε; ε = 0 and k = 0 recover the special case.
  • Selection policy: The proxy can choose more aggressive gradient blending than the oracle, while constrained selection can better match oracle choices.This comparison is reported for Llama-3.2-1B MLP modules at r = 0.85.
  • Model behavior: Per-layer optimal γ selection does not directly translate to pure downstream gains because of compounded selection effects.The section also compares adaptive selection approaches against an oracle aware of the best γ configuration per layer.
  • Model behavior: 70%-77% of selections use blending for EXAONE 4.5-33B and Qwen3-4B-Instruct across reported calibration and compression settings.Both models outperform their respective baselines on the target tasks.

C.4 Free-form Generation Evaluation

The free-form generation evaluation compares Adaptive γ⋆ with UniQL on EXAONE 4.5-33B at 20% compression across GSM8K, HumanEval, and MT-Bench.

  • Evaluation: Adaptive γ⋆ outperforms UniQL on GSM8K, HumanEval, and MT-Bench at 20% compression.The evaluation uses EXAONE 4.5-33B and compares against the UniQL baseline.
  • Evaluation: Tables 18 and 19 report γ selection counts for EXAONE 4.5-33B and Qwen3-4B-Instruct across calibration and compression settings.The entries encode γ < 1.0, γ = 0.75, and γ = 0.5 choices.

C.5 Candidate Selection Set Granularity

LaMoC tests whether the granularity of the candidate γ set affects compression outcomes using Qwen3-4B-Instruct at 20% compression with WikiText-2 and Alpaca calibration.

  • Candidate grid: The adaptive γ grid is expanded to {0.125, 0.25, 0.375, 0.5, 0.625, 0.75, 0.875, 1.0}.The expanded grid is evaluated with WikiText-2 and Alpaca calibration.
  • Candidate grid: Finer and coarser candidate grids can both improve accuracy over the baseline MoDeGPT.The finer grid can be affected by predictor noise, whereas the coarser grid introduces generalization.

C.6 Reference Gradients Ablation

The reference-gradient ablation compares gradients reused from a canonical compressed model with exact candidate-specific gradients and measures the associated computational and wall-time costs.

  • Gradient approximation: Reference gradients maintain performance close to exact gradients while requiring 37.9× less backward-pass compute.Exact gradients are recomputed for each candidate, whereas reference gradients are computed once on the canonical γ = 1.0 compressed model.
  • Tensor structures: The gradient Gram G, Fisher diagonal w, and activation and Fisher-weighted activation Grams C0 and CF are reported for MLP, VO, and QK modules.For QK, w is the diagonal of G; CF is constructed as diag(√w) C0 diag(√w).
  • Computational cost: LaMoC’s additional compression time mainly comes from caching reference gradients and evaluating every candidate.The comparison is reported against canonical MoDeGPT on an NVIDIA RTX Ada 6000.
  • Computational cost: For Llama-3.1-8B, LaMoC takes 2.5 h versus 3 h 10 min for the stated MoDeGPT comparison.The passage describes this increase as manageable with potential future optimization.
Loading 2608.30226v1…