PaperlayerView exact revision on arXiv

Source-linked AI summary

Budget-Aware Compression Pipeline for Single-GPU LLM Inference: Methods, Trade-offs, and Coupling Effects

Hongyu Yu, Yifei Shen

arXiv:2608.30076v1cs.CL

TL;DR

Single-GPU inference of 70B language models is constrained by memory, long-context KV-cache growth, throughput, and integration costs. This paper formulates compression as a joint budget problem and combines quantization, pruning, and KV-cache compression, producing a 33 GB deployment that supports 10k-token inputs with at least 10 tokens/s and at most 5% absolute accuracy loss.

  • Problem

    70B models exceed single-GPU memory budgets, while long-context KV-cache growth and compute, latency, and integration constraints complicate practical deployment.

  • Method

    The paper jointly budgets memory, throughput or latency, and integration cost while combining post-training quantization, structured layer pruning, and KV-cache compression.

  • Results

    The resulting 70B deployment fits within 33 GB of VRAM, supports at least 10k-token inputs and at least 10 tokens/s, and maintains accuracy loss within 5% absolute.

  • Takeaways & Limitations

    Compression should be treated as a coupled, budget-driven system-design problem, supporting pipeline search that jointly optimizes bit-width, pruning ratios, and KV retention.

  • Takeaways & Limitations

    The evaluation is limited to single-GPU inference for a 70B dense checkpoint, English QA and reasoning benchmarks, and the serving stack used for integration-cost measurement.

Abstract

from arXiv · show

Single-GPU deployment of 70B-parameter language models on an NVIDIA GPU is constrained by device memory, long-context throughput, and engineering integration cost. We cast single-GPU inference as a budget-aware design problem over these three axes and study how pruning, quantization, and KV-cache compression interact under realistic execution. Controlled ablations show that layer-wise pruning makes weight quantization more robust. KV-cache sparsification complements INT8 KV quantization by reducing memory without hurting decoding speed, while static vector quantizers often conflict with dynamic caching. Guided by these coupling results and explicit budget tracking, we assembled a practical pipeline and compressed a 70B model to about 33 GB, sustained about 57 tokens/s on 10k token prompts on a single A40, and kept absolute accuracy within 5% on common and reasoning benchmarks. We contribute design rules and a reproducible evaluation protocol that jointly report quality, memory, and end-to-end speed, and we provide a foundation for automated pipeline search under realistic single-GPU constraints.

1 Introduction

Single-GPU 70B inference is constrained by memory, throughput, and integration costs, motivating a unified budget-aware compression pipeline. The proposed workflow combines quantization, pruning, and KV-cache compression, with ablations used to identify compatible combinations.

  • A 70B model in FP16 exceeds 100 GB, while an A40 provides 48 GB and long-context KV-cache memory grows linearly with sequence length.
  • Prior work addresses quantization, pruning, and KV-cache compression separately, but lacks a unified pipeline evaluated under realistic single-GPU constraints.
  • The design treats memory footprint, throughput/latency, and integration cost as a three-dimensional deployment budget.
  • The pipeline combines 4-bit weight quantization, layer-wise pruning, and hierarchical KV-cache sparsification with 8…
  • Targeted ablations isolate positive and negative interactions among compression techniques and support actionable rules for avoiding failure modes.
  • The end-to-end system fits within 48 GB, supports ≥10k-token inputs, achieves ≥10 tokens/s, and limits absolute accuracy loss to ≤5%.
  • The study outlines automated pipeline search over quantization bit widths, pruning ratios, and KV retention rates under explicit budgets.

2 Related Work

Related work spans post-training quantization, pruning and sparsity, KV-cache compression, and serving-system constraints. The paper addresses the limited study of interactions among these techniques through controlled comparative ablations.

  • Post-training quantization reduces weight or activation precision, with methods ranging from 4-bit weight-only schemes to low-bit vector and rotation-based approaches.
  • Unstructured pruning removes low-importance weights, while SliceGPT changes width and ShortGPT removes transformer blocks using layer-importance scores.
  • KV-cache compression uses low-bit quantization, dynamic eviction or streaming, and sparse retrieval or compression to control long-context runtime memory.
  • Serving systems and compiler toolchains constrain feasible combinations through memory layouts, scheduling, static shapes, and kernel-fusion compatibility.
  • Prior work usually evaluates compression techniques in isolation, leaving interaction effects across quantization, pruning, and KV-cache management insufficiently characterized.
  • The paper uses systematic comparative study to quantify synergies and interference and identify mutually compatible combinations for single-GPU inference.

3 Problem Formulation and Budget-Aware Analysis Framework

The framework formulates long-context 70B inference on a 48 GB A40 as a constrained design problem over memory, throughput, and integration cost. It defines a modular search space and evaluates instantiated pipelines using end-to-end resource, speed, integration, and quality measurements.

  • 3 Problem Formulation and Budget-Aware Analysis Framework: 70B deployment on a 48 GB A40 is challenging because FP16 parameters exceed 140 GB and KV-cache memory grows linearly with sequence length.
  • 3 Problem Formulation and Budget-Aware Analysis Framework: The deployment objective is to satisfy memory and latency constraints while maintaining generation quality for long-context inputs such as ≥10,000 tokens.
  • 3.1 Deployment Budget and Constraints: The deployment budget is formalized as B = (Bmem, Bthr, Bint), covering memory footprint, throughput/latency, and integration cost.
  • 3.1 Deployment Budget and Constraints: Bmem caps weights, activations, and KV caches below 48 GB, while Bthr requires at least 10 tokens/sec under 10k-token contexts.
  • 3.1 Deployment Budget and Constraints: Bint measures engineering effort through kernel availability, compiler assumptions, and compatibility with dynamic batching.
  • 3.2 Compression Modules and Decision Variables: The decision space includes PTQ methods, layer or width pruning, weight sparsity, retrieval-aware caching, hierarchical eviction, and quantized KV storage.
  • 3.2 Compression Modules and Decision Variables: Module interactions matter because pruning alters activation distributio
  • 3.3 Budget-Aware Evaluation Protocol: Evaluation records peak device memory, generated tokens per second, integration effort, and quality metrics on specified benchmarks.

4 Single-Method Effects under Single-GPU Budgets

Controlled ablations compare quantization, pruning, and KV-cache compression under single-GPU budgets, revealing distinct quality, throughput, memory, and integration trade-offs. Layer selection and module ordering strongly affect whether compression remains viable at more aggressive budgets.

  • Post-Training Quantization: AWQ and GPTQ deliver the strongest weight-only quantization results across the screened datasets, while vector quantizers reach 2-bit regimes at higher integration overhead.AWQ is selected for the 70B backbone because it achieves higher task accuracy than GPTQ with identical 38 GB artifacts and clean mainstream-engine integration.
  • Layer Pruning: 25% depth pruning causes steep reasoning and knowledge-task losses for ShortGPT, showing that depth-only pruning saturates at higher ratios.The saturation is architecture dependent: Llama-70B has smooth, banded importance profiles, whereas Qwen-32B has pronounced middle-layer peaks.
  • Continuity-Aware Planning: The continuity-aware planner selects candidates through BI-based scores, local continuity windows, and greedy interval selection with continuity bonuses.BI is estimated from consecutive-block hidden-state changes, with larger BI indicating larger representational change and higher importance.
  • Continuity-Aware Planning: Continuity-aware reordering improves aggressive pruning outcomes by preferentially selecting contiguous low-importance bands instead of scattered BI-ranked layers.At 20 of 80 pruned layers, the reordered schedule reaches 0.7243 on ARC_E and 0.4091 on GPQA-Diamond, versus 0.7012 and 0.2879 for ShortGPT.
  • KV-Cache Compression: PyramidKV combined with KV quantization lowers peak usage from 40 690 MB to 39 029 MB while sustaining 57 tok/s; without KV optimization, 10k-token decoding exhausts memory.On 8B screening runs, ShadowKV has the highest throughput but conflicts with subsequent quantization, whereas PyramidKV and SnapKV provide consistent memory savings with moderate latency overhead.

5 Coupling Effects in Method Stacking

Method interactions determine whether compression remains usable under deployment budgets. The study finds compatible combinations that preserve efficiency, alongside quantization, sparsity, and caching conflicts that increase instability or integration cost.

  • Positive coupling: Applying ShortGPT depth pruning after AWQ produces the strongest positive coupling observed.Applying pruning first reshapes activation distributions and sensitivity, making quantization calibration harder and increasing quantization noise in critical directions.
  • Positive coupling: KV-cache sparsification combined with quantization reduces peak KV memory by over 1.6 GB while preserving 57 tokens/s on 10k-token prompts.The combination reuses existing Transformers hooks and AutoAWQ tooling, requiring only modest engineering effort.
  • Negative coupling: Static-cache vector quantizers conflict with dynamic KV caching, and their throughput falls below a denser AWQ baseline despite INT2 weight compression.These methods assume statically allocated caches, so enabling them disables dynamic KV caching.
  • Negative coupling: AWQ followed by unstructured Wanda sparsity destabilizes decoding on mathematical prompts and raises perplexity to the thousands.The combination undermines AWQ’s outlier-aware weight reconstruction.
  • Pipeline-level trade-offs: Negative coupling increases integration cost through kernel recompilation, graph-boundary changes, and loss of dynamic KV handling.The study therefore prioritizes compatibility and end-to-end efficiency over isolated per-method benchmark gains.
  • Pipeline-level trade-offs: The AWQ+ShortGPT+PyramidKV pipeline is selected over QTIP-centric alternatives because it satisfies memory, throughput, and integration budgets together.The QTIP-centric configuration failed to coexist with PyramidKV, causing repeated compilation failures and unacceptable integration overhead.

6 Unified Pipeline and End-to-End Results

The unified deployment combines AWQ quantization, ShortGPT depth pruning, and PyramidKV with INT8 KV compression for long-context inference on one A40. The resulting system fits within memory and throughput targets while preserving accuracy within the stated margin.

  • Pipeline construction: The instantiated pipeline targets DeepSeek-R1-Distill-Llama-70B on a single NVIDIA A40.Its design is guided by the coupling study and a three-dimensional budget covering memory, throughput, and integration cost.
  • Backbone compression: AWQ W4A16 reduces the model to 38 GB, after which ShortGPT prunes 10 of 80 transformer blocks.Activations remain in FP16 to preserve FlashAttention compatibility and simplify Transformers deployment.
  • Backbone compression: The resulting checkpoint occupies ∼33 GB, a 75% reduction relative to the FP16 baseline, while accuracy remains within a 5% margin.The reported accuracy condition applies to broad QA and commonsense tasks.
  • KV-cache optimization: 57.21 tokens/s is sustained on 10k-token prompts with peak device memory of 39 029 MB.The configuration combines PyramidKV with window size 1024 and INT8 KV-cache quantization.
  • KV-cache optimization: The FP16 baseline exceeds 48 GB and triggers an out-of-memory failure on the long-context workload.This comparison identifies KV-cache compression as necessary for running the 10k-token setting fully on the A40.
  • Budget summary: The final deployment satisfies Bmem ≤48 GB, exceeds the Bthr = 57 tokens/s target, and uses existing open-source integration pathways.The integration budget limits engineering effort to kernel adaptation rather than bespoke compiler work.
  • Incremental budget satisfaction: Adding PyramidKV and KV quantization addresses runtime cache growth after AWQ plus depth pruning has reduced the weight artifact.The remaining runtime-state pressure determines whether 10k-token decoding can run fully on device.

7 Conclusion

The paper frames 70B-model compression as coupled, budget-driven system design for single-GPU long-context inference. Its pipeline and evaluation approach support practical deployment analysis and future automated search over compression choices.

  • Conclusion: The unified pipeline deploys a 70B-class model on one A40 with 10k-token contexts under explicit memory, throughput, and integration constraints.The instantiated system uses AWQ, ShortGPT-style depth pruning, and PyramidKV-based KV compression.
  • Conclusion: The resulting deployment fits within a 33 GB VRAM envelope while maintaining competitive long-context reasoning accuracy.This outcome is reported for DeepSeek-R1-Distill-Llama-70B.
  • Conclusion: Naive composition is insufficient because Wanda destabilizes quantized activations and aggressive pruning with SliceGPT or ShortGPT causes substantial accuracy degradation.Continuity-aware layer reordering and layer-fusion distillation are introduced to extend depth-only pruning more safely.
  • Conclusion: The study provides a blueprint for automated pipeline search jointly optimizing bit-width, pruning ratios, and KV retention under explicit deployment targets.Its framing treats compression as a coupled system-design problem rather than independent method selection.

Limitations

The evaluation is scoped to single-GPU inference for a 70B dense checkpoint, with English QA and reasoning benchmarks on the studied A40 deployment setting. The authors caution that the recipe is not universal across hardware, model families, tasks, or serving engines.

  • Scope: The study is limited to single-GPU inference for a 70B dense checkpoint and the hardware, model family, and 10k-token setting evaluated.Larger mixture-of-experts systems and GPUs with ≤24 GB VRAM may require additional engineering and have different bottlenecks.
  • Evaluation boundary: Evaluation focuses on English-language QA and reasoning benchmarks, excluding multilingual generation, safety-critical domains, and calibrated factuality under distribution shift.These exclusions define the reported quality scope.
  • Systems boundary: The integration-cost trade-off may change on engines with native sparse-KV support or specialized vector-quantization kernels.The measured integration budget applies to the serving stack used in the experiments.

Ethical Considerations

The paper highlights improved accessibility from lower deployment costs while warning that easier access can also increase harmful uses. Its methods do not address bias, privacy leakage, or unsafe content generation.

  • Reduced memory and compute requirements can lower deployment cost and energy use, benefiting practitioners with limited hardware.The paper also identifies on-device and private deployment as potential benefits.
  • Easier deployment may increase availability for harmful uses, including large-scale misleading or spam content generation.
  • The compression methods do not mitigate model bias, privacy leakage, or unsafe content generation.The paper recommends access control, rate limiting, logging, content filtering, and compliance with licensing and data-governance requirements.

A.1 Calibration Corpora and Tasks

The appendix evaluates calibration, implementation, and deployment behavior under a fully on-device 10k-token A40 setting. It emphasizes comparable end-to-end measurements and shows that pruning plus PyramidKV is needed for the tested 70B configuration to fit.

  • A.1 Calibration Corpora and Tasks: 512 prompts from the checkpoint’s training mixture calibrate the 70B model, while 8B screening uses 128 WikiText2 and C4 sequences of 256 or 512 tokens.Calibration sequences for the 70B model are disjoint from downstream benchmarks.
  • A.1 Calibration Corpora and Tasks: Experiments use NVIDIA A40 GPUs with 48 GB VRAM, CUDA 12.2, and PyTorch 2.3 with compute capability 8.6 support.AutoAWQ supplies W4A16 quantization, ShortGPT uses its official implementation, and PyramidKV uses a 1,024-token hierarchical cache.
  • B Additional Pipeline Validation: Dense and AWQ-only 70B runs fail at 10,240-token prompts, whereas AWQ plus pruning plus PyramidKV fits within the A40 device budget.Adding PyramidKV reduces peak memory from 44.17 GB to about 42.56 GB with an unquantized cache; KV quantization reduces memory further but trades off decode speed.
  • B Additional Pipeline Validation: The validation target is fully on-device 10k-token inference on one A40, where batch size 1 is already memory constrained.The integration checklist tracks kernel availability, dynamic KV-cache compatibility, compiler assumptions, and batch/context regime.
  • B Additional Pipeline Validation: Serving-system comparisons are scope-limited because systems target distinct operating points and may alter budgets through offload, PCIe bandwidth, or specialized kernels.In the fixed 10,240-token A40 target, AWQ-only remains out of memory without PyramidKV.

C Reproducibility Notes

The reproducibility notes define controlled measurement procedures for allocation, throughput, and integration assumptions. They also document end-to-end results and sensitivity checks under the fixed A40 evaluation setting.

  • C Reproducibility Notes: Seeds are fixed to 42 across PyTorch, NumPy, and the evaluation harness.
  • C Reproducibility Notes: Measurements discard the first 32 generated tokens and profile each configuration for at least three full decoding passes.Calibration preprocessing scripts and pipeline YAML files are available upon request and depend only on public checkpoints.
  • C Reproducibility Notes: OOM results count only after model, prompt-tensor, and cache allocation reach the same target sequence length as successful runs.This prevents partially loaded models from being treated as comparable deployment points.
  • C Reproducibility Notes: The reported artifact uses public checkpoints and implementations, while specialized runtime or fixed-shape compiler requirements count toward integration budget.
  • C Reproducibility Notes: Prefill and decode speeds are logged separately before total throughput is reported, distinguishing phase-specific acceleration from sustained decoding performance.
  • C Reproducibility Notes: Table 10 compares dense, AWQ-only, AWQ-plus-pruning, and PyramidKV variants using 10,240-token inputs and 256 generated tokens on one A40.Dense and AWQ-only runs fail because the full prompt and cache exceed device capacity.
  • C Reproducibility Notes: Table 11 reports nearby continuity settings and finds that the best pruning setting remains model dependent.Nearby settings generally improve GPQA-Diamond over the corresponding BI-only baseline.
  • C Reproducibility Notes: Table 12 treats integration as a checklist requiring compatibility with the same fully on-device 10k-token A40 setting.
Loading 2608.30076v1…