PaperlayerView exact revision on arXiv

Source-linked AI summary

UniQL: Unified Quantization and Low-rank Compression for Adaptive Edge LLMs

Hung-Yueh Chiang, Chi-Chih Chang, Yu-Chen Lu, Chien-Yu Lin, Kai-Chiang Wu, Mohamed S. Abdelfattah, Diana Marculescu

arXiv:2512.03383v3cs.LGcs.AI

TL;DR

Edge devices have shared, workload-dependent resources, while recompression and existing elastic-training approaches carry substantial costs or constraints. UniQL combines quantization and structured pruning in a one-shot cloud workflow and allows pruning to be configured on-device across Transformers, SSMs, and hybrid models. The compressed models show memory and throughput gains across these architectures, with reported latency speedups for adaptive edge pruning.

  • Problem

    Shared, workload-dependent edge resources make fixed-size compressed models uncertain to deploy, while recompression, replicas, and elastic training have substantial costs or applicability limits.

    Source: §1 · pp. 1–2

  • Method

    UniQL combines quantization and structured pruning in a one-shot post-training cloud pipeline and supports configurable on-device pruning across Transformers, SSMs, and hybrid models.

    Source: §1 · p. 2

  • Results

    Compressed models show substantial memory and throughput gains across Transformers, SSMs, and hybrids; adaptive pruning yields 2.7×–3.4× latency speedups on edge devices.

    Source: §1 · p. 2, §6 · p. 12

  • Takeaways & Limitations

    The compressed models can elastically adapt to runtime constraints across Transformers, SSMs, and hybrid models.

    Source: §6 · p. 12

Abstract

from arXiv · show

Deploying large language models (LLMs) on mobile platforms faces significant challenges due to the limited memory and shared computational resources of the device. Resource availability may be an issue as it is directly impacted by the current device workload, adding to the uncertainty of model deployment. We introduce UniQL, a unified post-training quantization and low-rank compression framework with on-device configurable pruning rates for edge LLMs. UniQL is a general framework that integrates quantization and low-rank compression for Transformers, State Space Models (SSMs), and hybrid models to support diverse edge applications. In our proposed joint framework, we introduce an efficient structured weight-sorting method that speeds up computation by 20x, quantization-aware singular value decomposition (SVD) to minimize quantization errors, state-aware weight sorting for SSMs, and a fused rotary positional embedding (RoPE) kernel for pruned models. Our framework performs weight-sorting, fine-tuning, and quantization in the cloud in a single-pass workflow, while enabling on-device configurable pruning rates up to 35%. Our experiments show that quantized and pruned models achieve a memory reduction of 4x-5.7x and a token-throughput improvement of 2.7x-3.4x, maintaining accuracy within 5% of the original models at 15% pruning across Transformers (Llama3 and Qwen2.5), SSMs (Mamba2), and hybrid models (Nemotron-H and Bamba-v2). The code and quantized models are available at: https://github.com/enyac-group/UniQL.

1 Introduction

UniQL targets edge deployment where shared, workload-dependent resources can make fixed-size compressed models unusable, while recompression and existing elastic-training approaches carry substantial costs or scope constraints. It combines one-shot cloud compression with configurable on-device pruning across model architectures.

1 Introduction

  • Workload-dependent resource availability can prevent fixed-size pre-compressed or pre-quantized models from running, while recompression, replicas, and elastic training have practical costs or limits.Re-compression may take several hours on cloud GPUs; elastic training requires GPU resources and curated data for a particular model type.Source: §1 · pp. 1–2
  • UniQL sorts grouped MLP weights without gradients or full pseudo-inverses (20× faster than prior art), uses quantization-aware SVD and state-aware sorting, then fine-tunes and quantizes.For attention weights Wv and Wo, the quantization-aware SVD is designed to minimize quantization errors; state-aware sorting addresses SSM sensitivity to state matrices.Source: §1 · p. 2
  • The framework targets Transformers, SSMs, and hybrid models with pruning and quantization-friendly algorithms for their blocks.The stated objective is broad architectural support under post-training conditions with limited access to server-class GPUs and curated datasets.Source: §1 · p. 2
  • The authors describe UniQL as the first post-training framework to combine quantization and structured pruning for LLMs in a one-shot fashion.This is stated as a “to the best of our knowledge” claim.Source: §1 · p. 2
  • The integrated pruned-RoPE kernel yields 2.7×–3.4× latency speedups for adaptive pruning on edge devices.The contribution statement says this is supported by comprehensive profiling.Source: §1 · p. 2
  • The cloud pipeline sorts weights, fine-tunes across sampled global pruning rates, fuses refined patches, and quantizes once; devices then adapt pruning to system utilization.The framework supports configurable on-device pruning, including a 35% rate example.Source: §1 · p. 2, Figure 1 · p. 2, Figure 2 · p. 3

2 Related work

Prior work reduces model size through quantization and pruning, but the approaches differ in hardware requirements, supported architectures, and adaptability. UniQL is positioned against these limits as a post-training framework for multiple model families with on-device adaptation.

Transformer compression

  • Transformer compression uses low-bit quantization and unstructured, semi-structured, or structured pruning; structured pruning supports standard hardware but can reduce performance, while other methods may require specialized hardware.The cited work also includes one-shot compression, flexible bit-width quantization, and quantization with semi-structured sparsity.Source: §2 · p. 3

SSM compression

  • SSM work includes low-bit quantization and structured or unstructured pruning, with examples spanning head removal, block- or module-wise pruning, token pruning, and dimension reduction.The cited studies include SSM and vision-SSM approaches; UniQL focuses on a broader framework spanning Transformers, SSMs, and hybrids.Source: §2 · p. 3

Elastic training for LLMs

  • Elastic-training methods adapt LLMs to deployment constraints, but require GPU resources and curated datasets and are tailored to particular model types and sizes.The passage contrasts this with UniQL’s post-training on one server GPU across architectures and its support for on-device adaptation.Source: §2 · p. 3

3 Proposed Framework: UniQL

UniQL jointly sorts and decomposes weights for Transformers, SSMs, and hybrid models, then fine-tunes and quantizes a shared model to support on-device pruning. Its pipeline uses architecture-specific sorting and quantization-aware decomposition to prepare configurable pruned models.

3.1 Notations

  • The notation defines the post-pruning dimension D′ and the calibration correlation matrix C used in the framework’s sorting and decomposition procedures.It also specifies dimensions for Transformer and Mamba blocks and names the SVD factors.Source: §3.1 · p. 4

3.2 Structured Weight Sorting

  • UniQL sorts grouped weights by importance so devices can prune the least significant columns, co-designing the sorting with quantization and fused kernels across Transformers, SSMs, and hybrids.The framework uses calibration activations to guide sorting and supports joint decomposition of grouped weights.Source: §3.2 · p. 4
  • For MLPs, UniQL sorts intermediate channels using ridge-leverage scores and avoids pseudo-inverse computation, which is costly for large matrices and must be recomputed for different pruning rates.The sorting matrix reorders the up and gate projection columns and the down projection rows.Source: §3.2 · p. 4
  • For attention, symmetric Q/K sorting supports fused RoPE indexing, while quantization-aware SVD merges eigenvalue scaling into weight groups to reduce quantization errors.The joint decomposition also supports grouped-query attention.Source: §3.2 · pp. 5–6, Figure 4 · p. 6
  • For Mamba, state-aware sorting uses correlations from SSM states to order output columns of the relevant weights.The method forms state correlations over calibration samples and computes ridge-leverage scores for sorting.Source: §3.2 · p. 7

3.3 Masked LoRA Fine-tuning

  • One-shot masked LoRA fine-tuning trains the unpruned sorted model across sampled pruning rates, yielding configurable on-device pruning rates after training.The described instruction tuning uses Alpaca for five epochs on a single cloud GPU.Source: §3.3 · p. 8

3.4 Quantization and On-device Adaptive Pruning

  • UniQL applies group-wise uniform symmetric quantization to the fine-tuned model, then prunes intermediate, head, or state dimensions on-device while keeping the hidden dimension fixed.The described setup uses 4-bit weights and group size 128; Mamba pruning reduces both state and head dimensions.Source: §3.4 · p. 8

4 Experimental Results

UniQL evaluates one-pass structured pruning and post-training quantization across Transformers, SSMs, and hybrid models. Results cover downstream accuracy, compression time, model size, and generation speed.

4.1 Setups

  • Experiments span Transformer, hybrid, and SSM models, with zero-shot evaluation averaging accuracy and length-normalized accuracy across five tasks.The model set includes Llama-2-7B, Llama-3.1-8B, Qwen-2.5-7B, Nemotron-H-8B, Bamba-9B-v2, and Mamba-2-8B.Source: §4.1 · pp. 8–9

4.2 Zero-shot Downstream Tasks

  • With fine-tuning, UniQL reaches 67.2% on Llama-2-7B and 70.0% on Nemotron-H-8B, surpassing SVD-LLM consistently in the structured-pruning comparison.Without fine-tuning, it outperforms MoDeGPT and SVD-LLM in most cases, reaching 66.7% on Llama-2-7B and 69.1% on Qwen-2.5-7B; Table 2 compares models in FP16.Source: Table 2 · p. 9, §4.2 · p. 10
  • At 15% pruning, UniQL achieves 71.4% on Llama-3.1-8B and 70.5% on Nemotron-H-8B, outperforming SVD-LLM across comparable settings.At 35% pruning, it achieves 62.7% on Llama-3.1-8B and 57.7% on Mamba2-8B; Table 4 evaluates multiple pruning rates in one run.Source: §4.2 · p. 10, Table 4 · p. 10

4.3 Compression Time

  • UniQL's matrix decomposition takes 0h19m, 22× faster than MoDeGPT and 1.8× faster than SVD-LLM, and its compression algorithm is O(1) in the number of compression rates.With masked fine-tuning, UniQL takes 6h59m versus 7h03m for MoDeGPT and 15h57m for SVD-LLM; PTQ takes an extra forty minutes.Source: §4.3 · p. 11, Table 5 · p. 11

4.4 Model Size and Latency Profiling

  • 3.9×–5.7× memory reductions and 2.7×–3.4× throughput improvements in generation are reported for UniQL, which also outperforms TRT-AWQ and TAO-HQQ in profiling.On Nano 8G, UniQL is 1.7× faster than TAO-HQQ in TPOT; at 35% pruning, its models generate 2.1× faster than TAO-HQQ.Source: §4.4 · p. 11, Table 6 · p. 11, Table 7 · p. 11

5 Ablation Study

The ablations examine fused RoPE latency and masked LoRA fine-tuning for pruned models. Reported accuracy gains vary by model and quantization setting.

5 Ablation Study

  • The ablation study evaluates the effectiveness of individual framework components.Table 8 reports TPOT and TTLT on Nano 8G in milliseconds.Source: §5 · p. 11, Table 8 · p. 11

Fused rotary positional embedding

  • A fused RoPE kernel with index gathering reduces latency by 10% (1.1× speedup) for 4-bit Llama-3.1-8B at 0% and 25% compression.The index gathering addresses positions broken by structured sorting; Table 9 reports TPOT on A6000.Source: Table 8 · p. 11, §5 · p. 12

Masked LoRA fine-tuning

  • Masked LoRA fine-tuning improves FP16 accuracy by 2.6% on Llama-3.1-8B and 3.7% on Qwen-2.5-7B at 25% compression.The reported changes are from 67.0% to 69.6% and from 62.1% to 65.8%, respectively.Source: Masked LoRA fine-tuning · p. 12
  • At 25% compression, masked LoRA fine-tuning improves 4-bit accuracy by 2.7% on Llama-3.1-8B and 3.3% on Qwen-2.5-7B.The reported changes are from 65.0% to 67.7% and from 60.7% to 64.0%, respectively.Source: Masked LoRA fine-tuning · p. 12

Masked LoRA fine-tuning.

The quantization-aware SVD decomposition incorporates eigenvalues into weight factors to improve 4-bit accuracy at 25% pruning. The reported gains differ across Llama-3.1-8B and Qwen-2.5-7B.

Quantization-aware decomposition

  • At 25% pruning, quantization-aware SVD improves 4-bit accuracy by 7.5% on Llama-3.1-8B and 3% on Qwen-2.5-7B.The respective changes are from 60.2% to 67.7% and from 61.0% to 64.0%.Source: Masked LoRA fine-tuning · p. 12
  • The decomposition combines long-tailed eigenvalues with U, making each eigenvalue σ_i the quantization scaling factor for its group.The weight matrix is written W = (UΣ)V.Source: Masked LoRA fine-tuning · p. 12

Masked LoRA fine-tuning

  • Table 10 reports average accuracy for models under different settings.Table 9 reports TPOT in milliseconds for fused RoPE profiling on A6000.Source: Table 10 · p. 12, Table 9 · p. 12

6 Conclusion

UniQL combines post-training quantization and low-rank pruning to support adaptive edge deployment. Its structured sorting, quantization-aware decompositions, and fused rotary kernels deliver memory and throughput gains across Transformers, SSMs, and hybrid models.

6 Conclusion

  • UniQL combines post-training quantization and low-rank pruning to support adaptive deployment of LLMs on edge devices.The framework uses on-device configurable pruning and a one-shot cloud compression pipeline.Source: §6 · p. 12
  • Structured weight-sorting, quantization-aware decompositions, and fused rotary kernels are the framework's mechanisms for compression and acceleration.Source: §6 · p. 12
  • UniQL achieves memory and throughput gains across Transformers, SSMs, and hybrid models, with compressed models adapting to runtime constraints.Source: §6 · p. 12

Impact Statement

The impact statement notes potential benefits and risks of lightweight edge deployment. The supplied appendix and baseline passages describe GQA sorting methods and compare pruning methods in FP16.

Impact Statement

  • Lightweight edge deployment may broaden LLM access, but UniQL does not mitigate output-related societal risks and responsible deployment remains necessary.The statement also notes potential privacy and harmful-content risks, while environmental impact depends on adoption and usage.Source: Impact Statement · p. 13

A.1 Group-query Attention: Query-Key

  • For GQA query-key sorting, the method aggregates channel-correlation norm scores across grouped self-attention heads and uses symmetric sorted indices to enable RoPE.Source: §A.1 · p. 17

A.2 Group-query Attention: Value-Output

  • For GQA value-output sorting, one SVD decomposition is performed per key-value head, with quantization-aware SVD incorporated into the procedure.Source: §A.2 · p. 17

B.1 Comparison with Additional Baselines

  • At 25% compression, UniQL consistently outperforms prior pruning-based methods, and UniQL-ft achieves the strongest performance across evaluated tasks.The comparison evaluates all methods in FP16 for a fair and controlled setting.Source: §B.1 · p. 18, Table 11 · p. 18

B.2 Evaluation on the MMLU Dataset

On MMLU, the evaluated pruned models maintain competitive accuracy, with masked fine-tuning recovering a large drop for Nemotron-H. On MBPP+, UniQL maintains competitive performance against pruning baselines while reducing model size.

B.2 Evaluation on the MMLU Dataset

  • Pruned Llama-3.1-8B, Qwen-2.5-7B, and Nemotron-H-8B maintain competitive five-shot MMLU accuracy, and outperform MoDeGPT and SVD-LLM.Nemotron-H has a large accuracy drop that is recovered with low-cost masked fine-tuning; UniQL's 4-bit models perform comparably to TRT-AWQ and TAO-HQQ.Source: §B.2 · p. 18, Table 12 · p. 18
  • MMLU evaluation uses five-shot accuracy with batch size eight, and Table 12 compares UniQL with baselines under different settings.Source: §B.2 · p. 18, Table 12 · p. 18

B.3 Evaluation on Coding Tasks

  • On MBPP+, UniQL maintains competitive performance against SVD-LLM and MoDeGPT while reducing model size across pruning ratios and bit-width configurations.The results use Llama-3.1-8B-Instruct with batch size 1 and zero-shot evaluation.Source: §B.3 · p. 19, Table 13 · p. 19

C.1 Ablation study on calibration sets

The calibration-set ablation finds the best average accuracy when Alpaca is used for all stages at a fixed 25% pruning rate. Additional experiments report stable performance trends for 3-bit UniQL, which is simulated in FP16 as a proof of concept.

C.1 Ablation study on calibration sets

  • At 25% pruning, using Alpaca for weight-sorting, masked fine-tuning, and PTQ yields the best average accuracy across five zero-shot downstream tasks.The comparison controls hyperparameters and calibration-sample counts across configurations.Source: §C.1 · p. 19, Table 14 · p. 19

C.2 Ablation study on 3-bit UniQL

  • UniQL supports 8-, 6-, 4-, and 3-bit settings, with the 3-bit precision simulated in FP16 as a proof of concept.Source: §C.2 · p. 19, Table 15 · p. 20
  • The 3-bit variant retains competitive accuracy across multiple models, while performance trends remain stable as precision decreases.The reported interpretation highlights weight-sorting and recovery fine-tuning as effective across the tested bit-widths.Source: §C.2 · p. 19, Table 15 · p. 20

D Pareto-front Analysis

UniQL is compared with quantized and FP16 baselines on accuracy–latency trade-offs and energy efficiency across edge and cloud platforms. The analysis also shows how layer-wise pruning rates vary in hybrid models.

D Pareto-front Analysis

  • Across architectures, UniQL yields better latency–accuracy trade-offs under the same memory budget, with Pareto-dominant points for Llama-3.1-8B.The comparisons cover A6000 and Nano 8G; the 2–4GB regime is highlighted as critical for edge deployment.Source: Figure 5 · p. 20, D Pareto-front Analysis · p. 20, Figure 6 · p. 21

E Energy Profiling

  • At 35% pruning, Orin Nano energy falls to 143.12 J/request for Qwen-2.5-7B and 153.64 J/request for Mamba2-8B.Without pruning, the corresponding values are 208.23 J and 224.56 J; FP16 models exceed the device’s 8 GB memory limit.Source: E Energy Profiling · p. 21, Table 16 · p. 21
  • UniQL achieves higher throughput-per-energy across Transformer and SSM-based architectures on an A6000 GPU.The analysis profiles Llama-3.1-8B and Nemotron-H-8B across batch sizes, with 1024 prefilling and 1024 generated tokens per request.Source: Figure 7 · p. 21, E Energy Profiling · p. 22

F Layer-wise Pruning Rates

  • Hybrid-model self-attention layers receive lower pruning rates, and pruning them leads to significant accuracy drops.The pattern is reported for Bamba-9B-v2 layers 9, 18, and 27, and Nemotron-H-8B layers 7, 18, 29, and 40.Source: F Layer-wise Pruning Rates · p. 22, Figure 8 · p. 22
  • Layer-wise rates are allocated as P = L Pavg × Softmax(−s/ε), where s_i is layer importance and Pavg is target global sparsity.The experiments set ε to 0.1.Source: F Layer-wise Pruning Rates · p. 22

G Implementation Details

Implementation details specify the datasets and sample settings used for calibration and channel correlations, the shared masked LoRA fine-tuning configuration, and Hadamard fusion choices for pruning.

G.1 Calibration sets

  • Calibration uses wikitext-2 for BI scores and post-training quantization, and Alpaca for channel correlations and masked LoRA fine-tuning.BI scores and layer-specific pruning rates use 128 wikitext-2 samples of length 2048; quantization calibration uses 256 samples of length 2048.Source: §G.1 · p. 23, Table 17 · p. 23

G.2 Hyper-parameters of Masked LoRA fine-tuning

  • Masked LoRA fine-tuning uses five Alpaca epochs, rank 8, scaling factor 16, and sequence length 256 across all models.The setup also uses 100 warmup steps, AdamW, and microbatching; hyperparameters are not tuned per model or experiment.Source: §G.2 · p. 23, Table 18 · p. 23

G.3 Hadamard Transform Fusion

  • Pruned channels omit Hadamard rotations while the hidden dimension remains unchanged, enabling on-device pruning without mismatched pre-fused matrix shapes.For Qwen-2.5-7B, the configuration removes all Hadamard matrices because their use empirically degrades accuracy.Source: §G.3 · p. 23
Loading 2512.03383v3…