PaperlayerView exact revision on arXiv

Source-linked AI summary

LowRankArena: A Standardized Evaluation Platform for SVD-Based LLM Compression

Zishan Shao, Lixun Zhang, Kangning Cui, Wenhao Wu, Jinhee Kim, Yixiao Wang, Ting Jiang, Hancheng Ye, Qinsi Wang, Fan Yang, Danyang Zhuo, Yiran Chen, Hai Li

arXiv:2608.26389v1cs.CLcs.LG

TL;DR

Prior SVD-compression studies are hard to compare because benchmarks, budgets, precision, and inference setups differ, leaving method-level progress uncertain. LowRankArena standardizes these dimensions and audits representative methods, finding that rankings shift across backbones and keep ratios while speedups remain workload-dependent and often limited end to end.

  • Problem

    Existing SVD evaluations use inconsistent benchmarks, compression ratios, and setups, often mixing auxiliary techniques with low-rank effects and limiting fair comparison.

  • Method

    LowRankArena enforces aligned tasks, uniform-precision budgets, comparison regimes, inference measurements, and reproducible checkpoints for standardized SVD evaluation.

  • Results

    Under aligned protocols, no method remains the clear leader across backbones: rankings depend on backbone and budget, while efficiency gains concentrate in compute-bound prefill rather than the full generation lifecycle.

  • Takeaways & Limitations

    LowRankArena provides a protocol-aligned foundation for reproducible comparison of future SVD compression methods and their deployable efficiency.

  • Takeaways & Limitations

    The 70B feasibility audit measures engineering readiness under one H200 recipe rather than intrinsic algorithmic potential, with failures caused by implementation bottlenecks.

Abstract

from arXiv · show

SVD-based low-rank compression has become a fast-growing direction for reducing the memory and computational cost of large language models (LLMs). However, meaningful comparison across existing studies remains difficult as prior evaluations use varied benchmarks, inconsistent ratios, and diverse setups, often failing to isolate low-rank effects from auxiliary techniques. As a result, it remains unclear whether reported gains reflect method-level improvements or differences in evaluation protocol. This lack of comparability highlights the need for a unified, reproducible evaluation platform. To address this problem, we present LowRankArena, a standardized evaluation platform for SVD-based LLM compression. LowRankArena unifies task versions, uniform-precision compression budgets, comparison regimes, and inference measurements, and provides a reproducible pipeline with over 3 TiB released compressed checkpoints. Using LowRankArena, our aligned audit of five representative SVD methods reveals that prior findings are highly conditional under standardized protocols: clear leaders and performance tiers shift across backbones and keep ratios, multiple-choice accuracy can hide large perplexity degradation, and nominal low-rank savings yield workload-dependent and often limited end-to-end speedups. Our code is available at: https://github.com/Zishan-Shao/lowrankarena.git.

1 Introduction

Prior SVD-compression results are difficult to compare because evaluation protocols, budgets, and inference measurements vary. LOWRANKARENA standardizes these factors and finds that rankings and efficiency conclusions remain conditional on architecture, metric, and workload.

  • Motivation: SVD evaluations vary in benchmarks, task versions, backbones, compression-ratio definitions, precision, and inference backends, obscuring method-level effects.Auxiliary quantization, remapping, and isolated latency measurements can be bundled with or substituted for low-rank effects.
  • Motivation: Researchers often rely on reported figures because reproducing baselines across model families, keep ratios, tasks, and inference configurations requires substantial GPU resources.This makes direct re-evaluation under a shared pipeline difficult.
  • Contribution: LOWRANKARENA standardizes task versions, model settings, compression budgets, and end-to-end inference measurements while releasing more than 3 TiB of reproducible artifacts.The released artifacts include the full evaluation pipeline and an extensive collection of compressed checkpoints.
  • Findings: Under aligned settings, no stable architecture-invariant method ordering emerges; relative strengths depend on backbone, keep ratio, and metric group.Efficiency gains are workload-dependent, concentrating in compute-bound prefill and diminishing when generation is decoding-heavy.

2 The LOWRANKARENA Platform

LOWRANKARENA converts heterogeneous SVD claims into matched, reproducible comparisons by controlling models, tasks, budgets, precision, and inference conditions. Its primary regime isolates uniform-precision low-rank effects while separate audits examine deployment and feasibility boundaries.

  • Design: LOWRANKARENA fixes assumptions that vary across prior work, separates pure low-rank compression from auxiliary techniques, and releases common reference artifacts.This is intended to make comparisons easier to interpret as method-level differences rather than protocol artifacts.
  • Design: The platform standardizes benchmarks, compression budgets, and inference environments through a shared artifact interface, task definitions, budget axis, and inference stack.Model quality, compression budget, and deployment efficiency remain separately controlled dimensions.
  • Comparison regimes: Uniform-precision SVD is the primary comparison regime, while mixed-precision, remapping-based, and runtime-adaptive approaches receive auxiliary audits.This avoids confounding low-rank quality with implementation-specific deployment advantages.
  • Compression budget: Fixed-r comparisons preserve layer precision, so they isolate retained low-rank subspace selection rather than bit-width reduction or dense fallback.For each compressed layer, the retained rank is compared at matching precision.
  • Evaluation setup: The evaluation covers LLaMA-1/2, Llama-3.1, and Qwen3 with five SVD-style methods, aligned keep ratios, multiple-choice accuracy, perplexity, and serving metrics.Inference measurements use a single vLLM path with matched TTFT, inter-token latency, E2E latency, and throughput conditions.
  • Research questions: The empirical study asks whether standardized SVD progress survives protocol alignment, remains competitive with structured pruning, and translates nominal savings into deployable efficiency.These questions connect accuracy claims to end-to-end inference outcomes.

3 Q1: Re-Assessing Existing SVD Rankings

Standardized evaluation reshapes the ranking landscape: no SVD method is universally strongest, and apparent superiority depends on backbone, keep ratio, and metric group. Detailed results show large perplexity differences alongside shifting architecture-specific rankings.

  • Overall ranking: Standardization reshapes rankings by revealing that perceived superiority is confounded with experimental setup rather than algorithmic merit alone.The unified framework re-evaluates existing methods under controlled conditions.
  • Leaderboard interpretation: The main leaderboard evaluates methods across Llama and Qwen models under matched task versions, scripts, and budget definitions, yet the strongest recipe still changes across conditions.Rankings are not intrinsic to a method independent of evaluation context.
  • Context-dependent rankings: MoDeGPT leads aggregate retention on Llama backbones at moderate ratios, while SVD-LLM and Basis Sharing reach C4 perplexities of 1187.78 and 461.21 versus MoDeGPT’s 51.82 on Llama-3.1-8B at 60% keep.The aggregate result reflects a trade-off between generative stability and downstream accuracy.
  • Architectural sensitivity: ASVD drops from #2 to #5 at 80% keep when moving from Llama-1 to Llama-3.1, while Basis Sharing rises from #4 to #2.These shifts demonstrate that ranking changes across model families.
  • Architectural sensitivity: On Qwen3-8B-Base, ASVD and Basis Sharing take the top spots at 80% and 60% keep, respectively, pushing MoDeGPT to second.The strongest recipe therefore varies with backbone, budget, and metric mix.

4 Q2: Comparison with Structured Pruning

Under matched budgets, SVD methods remain competitive with structured pruning on multiple-choice accuracy, but aggressive compression can expose severe language-modeling degradation and no universal winner.

  • SVD methods remain competitive with structured pruning on MCQ benchmarks, without absolute dominance across all regimes.
  • ASVD’s 0.353 MCQ average sits at the 0.357 macro choice floor, whereas SVD-LLM and Basis Sharing remain above chance across all seven tasks despite elevated C4 perplexities.
  • At aggressive keep ratios, several SVD methods encounter a capability cliff, while the strongest pruning baseline degrades more gracefully.
  • SVD-LLM and Basis Sharing reach 1187.78 and 461.21 C4 PPL at a 60% keep ratio, compared with MoDeGPT at 51.82.
  • Even MoDeGPT fails to close the gap to LLM-Pruner, which achieves 34.85 C4 PPL under the same budget.

5 Q3: End-to-End Inference Gains

Nominal low-rank savings translate unevenly into end-to-end inference gains: compute-bound prefill can benefit substantially, while decode-heavy workloads often see marginal or negative improvements.

  • Factorized weights mainly benefit compute-bound stages such as prefill, while decode-heavy workloads face memory-bandwidth, kernel-launch, and dual-GEMM bottlenecks.
  • Prefill-heavy and balanced workloads achieve TTFT speedups up to 4.20x, but gains diminish sharply during decoding.
  • In decode-heavy scenarios, most methods provide marginal or sub-1× throughput improvements, with SVD-LLM dropping to 0.80×.
  • DoBi-SVD can fall back to dense execution for high-rank layers, so its measured speedup is not attributable to low-rank factorization alone.

6 Secondary Audits, Scope, and Limitations

Secondary audits show that rankings, speedups, and implementation readiness depend strongly on calibration, workload, hardware, and software conditions. The 70B audit measures engineering readiness rather than intrinsic algorithmic potential.

  • Calibration Sensitivity and Ranking Uncertainty: Calibration changes absolute retention and can reverse close method orderings, while MoDeGPT remains the clear leader across tested conditions.The largest within-WikiText-2 range is 0.0284, below the 80%-keep leader margins, but this is not a universal uncertainty bound.
  • Instruction-Tuning and Extended Backbones: Instruction-tuned audits reinforce that rankings remain conditional on keep ratios, while downstream MCQ accuracy can persist despite sharply degraded perplexity.Instruction-tuned results are treated as auxiliary evidence because prompt templates and decoding configurations introduce heightened sensitivity.
  • Inference on More Devices: Large TTFT gains in prefill-heavy settings translate into marginal end-to-end improvements and disappear when decoding dominates.Speedups remain sensitive to workload phase, GPU type, and kernel execution paths.
  • Implementation Readiness under a Fixed Budget: The fixed one-H200 feasibility audit measures whether released code produces standardized artifacts, separating implementation readiness from theoretical scalability.The released implementations remain coupled to dependency versions, numerical kernels, offload assumptions, and architecture patterns.
  • Implementation Readiness under a Fixed Budget: At 70B, failures arise from code bottlenecks including dependency, memory, CUDA eigensolver, and GPU-offloading issues rather than demonstrated algorithmic limits.The audit therefore records engineering maturity and identifies where repairs, fallbacks, or sharding are needed.

7 Related Works

Related work spans decoder-oriented SVD methods, attention-side compression, runtime systems, structured pruning, hybrid schemes, and general evaluation infrastructure. LowRankArena focuses on static checkpoint compression and isolates uniform-precision low-rank effects.

  • SVD Compression for LLMs: Recent decoder-oriented SVD methods improve recovery through activation-aware rescaling, loss-sensitive truncation, parameter sharing, or adaptive rank allocation.Examples include ASVD, SVD-LLM, DoBi-SVD, ZSSVD, Basis Sharing, D-Rank, and adaptive rank methods.
  • Attention-Side Low-Rank and Runtime Systems: Attention-side methods target long-context serving memory and are treated as complementary because LowRankArena focuses on static checkpoint compression.Runtime systems motivate end-to-end serving audits because practical speed depends on memory movement and backend execution, not FLOP reduction alone.
  • Structured Pruning and Hybrid Schemes: Structured pruning removes architectural components, whereas SVD factorization preserves original module interfaces; pruning is used as a matched deployment-level baseline.The comparison matches parameter budgets and precision.
  • Structured Pruning and Hybrid Schemes: LowRankArena prioritizes uniform-precision SVD and treats quantization and hybrid variants as auxiliary analysis to avoid confounding low-rank effects with precision changes.Mixed-precision and remapping-based advantages rely on different deployment assumptions.
  • LLM Evaluation and Compression Benchmarks: Existing evaluation tools provide broad task coverage or compression benchmarking, but are not specialized for SVD checkpoint compression and its calibration, budget, artifact, and backend requirements.LowRankArena addresses these specialized comparability needs.

8 Conclusion

LowRankArena concludes that standardized protocols make SVD compression findings more conditional: no method leads across all backbones, budgets, and metrics, while practical speedups concentrate in compute-bound prefill. The released platform and artifacts support reproducible future comparisons.

  • 8 Conclusion: No SVD method remains the clear leader across all evaluated backbones; leaders and performance tiers depend on backbone and budget.Close orderings can also be calibration-sensitive.
  • 8 Conclusion: End-to-end efficiency is workload-dependent, with speedups concentrated in compute-bound prefill rather than the full generation lifecycle.This reflects the platform’s emphasis on deployment-level inference measurements.
  • 8 Conclusion: LowRankArena provides a foundation for reproducible, protocol-aligned comparison of future SVD compression methods.The appendix documents experimental suites, inference systems, robustness audits, artifact management, standardization protocols, limitations, and maintenance plans.
  • 8 Conclusion: The appendix extends evaluation with task configurations, cross-device serving, large-model feasibility, instruction-tuning sweeps, and BoolQ label-bias analysis.These audits delineate operational scope rather than introducing additional method rankings.

A.1 Task and Metric Configuration

LowRankArena separates base and instruction-tuned evaluation suites, combines accuracy with perplexity and reasoning metrics, and measures serving through a controlled end-to-end harness. Its audits show conditional rankings, perplexity degradation hidden by MCQ stability, and workload-sensitive serving outcomes.

  • A.1 Task and Metric Configuration: Base-model evaluation uses separate perplexity, seven-task zero-shot MCQ, and math-retention suites with explicitly defined task versions and macro averaging.MCQ uses acc_norm when available, otherwise acc; math combines 0-shot MathQA with 5-shot MMLU-Math.
  • A.1 Task and Metric Configuration: Instruction-tuned evaluation covers MMLU-Pro, GSM8K, and auxiliary IFEval with specified shot counts, scoring metrics, and tokenizer chat-template settings.GSM8K uses 8-shot chain-of-thought prompting and exact_match; IFEval’s headline metric is prompt_level_strict_acc.
  • A.1 Task and Metric Configuration: Calibration-based methods use WikiText-2 sequences under method-specific sample counts and native variants, while pruning uses prune_only without calibration and added recovery is excluded.This preserves released calibration recipes for the primary setup.
  • A.1 Task and Metric Configuration: Accuracy suites use LM-Eval-Harness with the vLLM backend, while perplexity uses a contiguous-text runner on non-overlapping 2048-token blocks from WikiText-2 and C4.The benchmark reports macro means of configured headline metrics.
  • A.1 Task and Metric Configuration: Inference measurements use fresh vLLM servers and matched request streams, schedulers, tokenizers, tensor parallelism, dtypes, and KV-cache policies across methods.The primary benchmark uses one A100 80GB GPU, with RTX A5000 and L40S reruns for cross-device validation.
  • A.1 Task and Metric Configuration: Cross-device audits find that low-rank checkpoints can improve some prefill-heavy measurements, while end-to-end latency and generation throughput remain sensitive to workload, hardware, and execution paths.The serving tables are not a separate leaderboard because method coverage differs by device and some artifacts are not loadable through the shared vLLM path.
  • A.1 Task and Metric Configuration: At 70B, the fixed-budget audit records source-code readiness and feasibility rather than method quality or algorithmic impossibility.The audited paths do not directly produce standardized artifacts out of the box under the stated recipe.
  • A.1 Task and Metric Configuration: Across standardized leaderboards, method ordering shifts by backbone, keep ratio, and metric, while MCQ accuracy can remain comparable as WikiText-2 and C4 perplexity degrade.Instruction-tuned sweeps also show substantial downstream degradation, especially on GSM8K under stronger compression.

B.6 BoolQ Label-Bias Audit

The BoolQ audit identifies a strong output-label bias in the anomalous compressed checkpoint and shows that calibration sampling can substantially affect BoolQ while leaving other multiple-choice tasks stable.

  • Reproducibility audit: 0.4116 BoolQ accuracy is reproduced by two complete reruns and an independent offline recomputation, ruling out evaluation nondeterminism.The reruns concern the MoDeGPT-compressed Llama-3.1-8B checkpoint at an 80% keep ratio.
  • Label bias: 62.2% of BoolQ labels are “yes,” but the checkpoint predicts “yes” for only 3.58% of examples, indicating strong output-label bias.Class-conditional accuracy is 99.68% on “no” examples and 5.56% on “yes” examples.
  • Calibration sensitivity: Calibration-draw sensitivity is assessed by regenerating checkpoints with different WikiText-2 samples while fixing the model, compression ratio, and evaluation procedure.This separates checkpoint-level reproducibility from robustness to calibration sampling.

C.2 Adapting LOWRANKARENA to New Methods

LOWRANKARENA adapts to new methods through registered, metadata-rich checkpoint artifacts that reuse the released evaluation, memory, speed, and reporting paths.

  • Registration: New artifacts can be registered through Arena.register or Arena.register_manifest as local development overlays or persisted registry entries.The Arena interface wraps the checkpoint registry, manifests, loaders, runners, and reporting utilities.
  • Shared execution: Registered artifacts use the same Arena.evaluate, Arena.memory, and Arena.speed calls as released baselines.This provides a common execution surface for evaluation, memory measurement, and speed measurement.
  • Artifact-centered extension: The benchmark-facing object is a registered checkpoint with explicit metadata, while method-specific compression code may remain separate.This artifact-centered design separates method implementation from benchmark integration.
  • Artifact-centered extension: New methods reuse the same task suites, budget axis, precision policy, loading path, serving adapters, and normalized JSON result schema.The design avoids bespoke evaluation scripts for each method.
  • Artifact documentation: The checkpoint-to-result map records each released checkpoint’s method, source model, keep ratio, precision, calibration metadata, and normalized evaluation records.The accompanying metadata describes the artifact catalog, audit index, intended use, and known limitations.
  • Artifact documentation: Main artifacts determine the primary leaderboard, while auxiliary artifacts support sensitivity, instruction-tuned, mixed-precision, remapping, cross-device, and feasibility analyses without determining rankings.Main artifacts are uniform-precision checkpoints and results underlying the central ranking and efficiency analyses.

D Discussion & Limitations

LOWRANKARENA contributes a large standardized artifact and evaluation infrastructure, while its conclusions remain bounded by uniform-precision posttraining SVD, current serving systems, and deployment-safety concerns.

  • Contributions: Over 3 TiB of released artifacts accompany LOWRANKARENA’s standardized platform and checkpoint zoo.The contribution includes a reproducibility workflow and a matched re-evaluation audit of five representative SVD methods.
  • Methodological scope: The primary evaluation focuses on uniform-precision, posttraining SVD compression to isolate low-rank factorization effects.Hybrid SVD with INT4/INT8 quantization and unconstrained sparse masking are outside the current evaluation scope.
  • Hardware and software dependence: End-to-end serving measurements are tied to A100, L40S, RTX A5000, and H200 hardware and to the engineering maturity of vLLM.The paper frames these measurements as empirical system results rather than hardware-independent properties.
  • Broader impacts: Compressed checkpoints may unpredictably affect downstream safety guardrails, alignment protocols, or inherent biases, so safety-critical capabilities require independent re-auditing before deployment.This warning applies broadly to model compression, including SVD-based truncation.
Loading 2608.26389v1…