PaperlayerView exact revision on arXiv

Source-linked AI summary

RAMP: Reinforcement Adaptive Mixed Precision Quantization for Efficient On Device LLM Inference

Arpit Singh Gautam, Saurabh Jha

arXiv:2603.17891v1cs.LGcs.AI

TL;DR

Large language models face a memory bottleneck, while uniform bit widths ignore substantial differences in layer sensitivity and existing mixed-precision methods lack demonstrated cross-model transferability. RAMP learns transferable mixed-precision policies and achieves Pareto-optimal accuracy-efficiency trade-offs while supporting standardized GGUF deployment.

  • Problem

    Large language models create a memory bottleneck, and uniform bit-width allocation ignores layer-sensitivity differences while prior mixed-precision methods lack demonstrated cross-model transferability.

  • Method

    RAMP learns transferable mixed-precision quantization policies using reinforcement learning, layer embeddings, a global bit budget, and quality- and budget-aware rewards.

  • Results

    5.54 PPL at 3.68 GB (3.65 effective bits) on Llama-2-7B, with RAMP outperforming AWQ by 6% in size and 1% in quality.

  • Takeaways & Limitations

    RAMP enables Pareto-optimal accuracy-efficiency trade-offs and deployment through standardized GGUF formats with portable kernels across CPU, GPU, and Apple Silicon.

  • Takeaways & Limitations

    RAMP has been evaluated only on decoder-only Llama and Mistral architectures, supports discrete layer-level bit widths, and remains limited to post-training quantization.

Abstract

from arXiv · show

Post training quantization is essential for deploying large language models (LLMs) on resource constrained hardware, yet state of the art methods enforce uniform bit widths across layers, yielding suboptimal accuracy efficiency trade offs. We present RAMP (Reinforcement Adaptive Mixed Precision), an off policy Soft Actor Critic framework that learns per layer bit width assignments to minimize perplexity under a global bit budget. The policy conditions on an 11 dimensional embedding of activation statistics, weight properties, and structural descriptors, enabling zero shot transfer across model families and scales. To enable stable sub 4 bit quantization, we introduce Scale Folding, a preconditioning technique that migrates activation outliers into weights via per channel scaling and normalization layer compensation. A quality prioritized reward with asymmetric penalties and budget cliffs drives rapid convergence. On Llama 2 7B, RAMP achieves 5.54 perplexity at 3.68GB (3.65 effective bits), outperforming uniform 4 bit AWQ (5.60 at 3.90 GB) and GPTQ by 6% in size and 1% to3% in quality. Critically, a policy trained only on Llama 2 7B generalizes zero shot to Llama 2 13B and Mistral 7B, often surpassing target specific training, supporting the hypothesis that quantization sensitivity is primarily architectural. The HALO pipeline exports allocations to GGUF format for kernel free inference on CPUs, GPUs, and edge devices, retaining 99.5% of FP16 commonsense reasoning performance.

1 INTRODUCTION

Modern LLMs face a memory bottleneck that limits deployment on consumer, edge, and cost-sensitive hardware. RAMP addresses the accuracy–efficiency trade-off by learning transferable mixed-precision allocations and supporting portable sub-4-bit inference.

  • The Memory Wall in Large Language Models: Llama-2-13B requires approximately 26 GB in FP16, while Llama-2-7B requires about 13.5 GB, creating a deployment bottleneck on many consumer GPUs.The constraint also affects edge devices, cost-sensitive cloud environments, and privacy-sensitive on-device applications.
  • Limitations of Existing Quantization Methods: Uniform bit-width allocation over-allocates precision to robust layers and under-allocates it to sensitive layers, producing a suboptimal accuracy–efficiency trade-off.Embedding layers, attention output projections, and final language-modeling heads are described as particularly sensitive, while many intermediate MLP layers tolerate lower precision.
  • Limitations of Existing Quantization Methods: Existing quantization strategies require costly per-model optimization and generally do not transfer across architectures or model variants.The text identifies GPTQ’s O(d^2) layer complexity and AWQ’s need for recalibration as examples of deployment costs.
  • Hardware and Deployment Challenges for Mixed Precision: Mixed-precision inference can be 1.2–1.5× slower than uniform quantization because varying bit-widths introduce kernel fragmentation and switching overhead.GGUF also supports only predefined static patterns, limiting arbitrary learned mixed-precision formats.
  • RAMP: RAMP frames layer-wise bit allocation as constrained sequential decision-making and uses Soft Actor-Critic with normalized 11-dimensional layer features to learn transferable policies.The reward prioritizes quality while treating bit efficiency as a flexible constraint, and feature normalization is intended to support transfer across models.
  • Contributions: 5.54 PPL at 3.68 GB (3.65 effective bits) lets RAMP outperform AWQ by 6% in size and 1% in quality on Llama-2-7B.Scale Folding supports practical 3-bit quantization, while HALO exports allocations for consumer hardware and reports 3.05× speedup over FP16 with 98–99% baseline reasoning performance.

2 BACKGROUND & RELATED WORK

LLM compression methods trade accuracy, memory, compute, and retraining cost, with quantization favored for portable post-training deployment. RAMP addresses the field’s uniform-allocation and non-transferability limitations using transferable mixed-precision policies and Scale Folding.

  • Quantization Fundamentals: Quantization reduces precision to shrink memory and compute while usually requiring only post-training calibration, making it the dominant portable LLM compression paradigm.Unlike pruning and distillation, it typically avoids retraining and supports diverse deployment platforms.
  • Post-Training Quantization: PTQ is standard for modern LLMs because it calibrates lower-bit representations without retraining, whereas QAT offers adaptation at prohibitive 7B-plus training cost.Weight-only quantization is prevalent because activation memory is rarely the inference bottleneck.
  • Quantization Challenges: Activation outliers dominate quantization scales, forcing coarse representations and potentially driving perplexity above 10.Information-bottleneck layers can contain rare but critical signals whose loss severely harms quality.
  • Research Gap: No prior mixed-precision method had demonstrated cross-model transferability, requiring repeated optimization for each architecture, scale, or seed.SAC had also not previously been applied to layer-wise LLM quantization.
  • RAMP Positioning: RAMP combines SAC policy learning, 11-dimensional layer embeddings, tiered rewards, and Scale Folding to address uniform allocation, non-transferability, and hardware fragmentation.Its positioning includes GGUF export and reliable sub-4-bit inference without custom kernels.

3 THE RAMP METHODOLOGY

RAMP formulates mixed-precision quantization as constrained sequential decision-making, assigning layer bit widths to minimize quantized-model perplexity under a memory budget. Its SAC policy uses normalized 11-dimensional layer embeddings intended to support transfer across model scales.

  • MDP Formulation: RAMP casts mixed-precision quantization as a Markov Decision Process in which a policy assigns bit widths layer by layer under an average bit budget.The objective minimizes quantized-model perplexity subject to the memory constraint.
  • MDP Formulation: The action space contains discrete bit widths {3, 4, 5, 6}, while transitions append each previous action to the next layer’s context.The reward combines perplexity quality and average-bit-budget terms.
  • RAMP Pipeline: Figure 1’s pipeline uses SAC to discover a mixed-precision strategy, then applies scale folding, layer-wise quantization, and GGUF export.The stages target kernel-free deployment of the resulting quantized model.
  • Layer Embeddings: Each layer is represented by an 11-dimensional embedding combining activation features, weight statistics, structural descriptors, and contextual features.The representation includes activation magnitude and importance, weight mean and standard deviation, dimensions, layer type, depth, prior bit width, and running average bit width.
  • Layer Embeddings: The embedding’s structural descriptors encode normalized depth, input and output dimensions, and attention-versus-MLP layer type.Contextual features add coarse depth, the previous-layer bit width, and the running average bit width.
  • Embedding Normalization: Normalization by layer width or depth and standardization to zero mean and unit variance are intended to make embeddings approximately invariant across model scales.This scale invariance holds empirically across Llama-2-7B, Llama-2-13B, and Llama-3-8B.

3.3 Soft Actor-Critic Agent Architecture

RAMP uses Soft Actor-Critic to learn layer-wise bit-width actions from compact states, with entropy regularization and twin critics supporting exploration and value estimation.

  • RAMP uses Soft Actor-Critic, an off-policy algorithm that jointly learns a policy and value function.
  • The actor maps each state to a continuous action that is squashed and converted to the nearest discrete bit-width.
  • Twin Q-networks reduce overestimation by estimating action values with separate critics.
  • Entropy maximization and automatic temperature tuning encourage exploration during policy learning.
  • Replay-buffer minibatches update the critics, actor, and temperature parameter, with target networks updated by Polyak averaging.

3.4 Quality-First Reward Design with Cliff Penalty

RAMP combines quality and budget signals into a terminal reward that prioritizes perplexity while using progressively stronger penalties for excessive average bit-width.

  • The terminal reward combines quantized-model quality with budget adherence after layer-wise actions are completed.
  • The quality term uses a 2:1 asymmetry, penalizing perplexity degradation more heavily than rewarding improvement relative to the FP16 baseline.
  • 0 reward penalty applies when average bit-width is at most 4.0, while moderate overruns receive a linear penalty through 4.25.
  • Average bit-width above 4.25 incurs quadratic punishment, creating a cliff that strongly discourages budget violations.
  • Training rapidly corrects overly aggressive assignments, explores near the budget boundary, and typically reaches near-final performance within 120 episodes.

4 HARDWARE-AWARE DEPLOYMENT (HALO)

HALO converts RAMP’s mixed-precision allocations into portable GGUF models, addressing kernel fragmentation and stabilizing sub-4-bit inference through Scale Folding.

  • 4.1 The Kernel Fragmentation Problem: Naive mixed-precision inference is typically 1.2–1.5× slower than uniform 4-bit inference despite using fewer total bits.
  • 4.2 HALO Deployment Pipeline: HALO maps learned allocations to standardized GGUF types supported natively by llama.cpp, avoiding custom mixed-precision kernel development.
  • 4.2 HALO Deployment Pipeline: RAMP assigns each quantizable layer a bit-width in {3, 4, 5, 6}, then quantizes it after Scale Folding preconditioning.
  • 4.3 Scale Folding: Activation Outlier Migration: Scale Folding migrates activation outliers into weights and compensates normalization layers, producing smoother activations for stable 3-bit quantization.
  • 4.5.1 Scale Folding Ablation: Without Scale Folding, Llama-2-7B reaches 5.58 perplexity; with folding, it reaches 5.54 PPL after 150 episodes with stable 3-bit assignments.
  • 4.5.3 Portability Ablation: A single GGUF file runs unmodified across platforms with stable perplexity, while latency varies with hardware capability.

5 EVALUATION

RAMP is evaluated across model families and scales using calibration-based layer embeddings, with results showing Pareto improvements, transferable policies, correlated allocations, and retained downstream accuracy.

  • 5 EVALUATION: Llama-2-7B trains the policy, while Llama-2-13B, Llama-3-8B, and Mistral-7B test scale, parameterization, vocabulary, and architectural transfer.
  • 5 EVALUATION: Calibration extracts 11-dimensional layer embeddings from activations collected on 128 WikiText-2 sequences, requiring approximately 2 minutes per model.
  • 5.6 Zero-Shot Policy Transfer: Zero-shot policies consistently outperform or match target-specific training, while layer-wise bit patterns show depth-wise correlation of ρ ≈0.9 across models.
  • 5.7 Downstream Task Performance: RAMP retains 99.5% of FP16 downstream accuracy on average while achieving substantially higher compression than uniform 4-bit baselines.

6 ANALYSIS & DISCUSSION

RAMP’s learned policies transfer across model scales while revealing consistent structural patterns in layer precision. Ablations support stable convergence and practical deployment, but evaluation remains bounded by architecture, bit-width, and granularity constraints.

  • Transferability: Zero-shot policies trained on Llama-2-7B match or outperform direct training on Llama-2-13B and Mistral-7B, suggesting architecture-level sensitivity.
  • Embedding stability: Pearson correlations above 0.93 show that normalized 11-dimensional embeddings encode structural roles independently of model scale or exact parameter values.
  • Allocation patterns: Precision rises with depth: early layers average ∼3.77 bits, middle layers ∼4.01 bits, and late layers ∼4.52 bits.Late layers are enriched for 5-bit assignments, while middle layers are mostly 3–4 bits.
  • Embedding expressivity: Individual outlier proxies correlate weakly with assigned bits (|r| ≈0.03), whereas the full compositional 11D embedding captures sensitivity beyond single features.
  • Allocation patterns: Output projections and final layers receive the highest average precision, while intermediate MLP down-projections are compressed most aggressively.
  • Ablations: Asymmetric quality prioritization with a cliff penalty yields 33% faster convergence and 7× lower variance, while SAC is 8× more sample-efficient than PPO.
  • Deployment: HALO trades marginal peak throughput for portable GGUF deployment using mature kernels across CPUs, GPUs, and Apple Silicon without custom development.
  • Limitations: RAMP was evaluated only on decoder-only Llama and Mistral architectures, with discrete layer-level bit-widths {3, 4, 5, 6}; other architectures and finer allocations remain untested.

7 CONCLUSION

RAMP combines transferable reinforcement-learned mixed precision, Scale Folding, quality-prioritized rewards, and HALO export. It reports improved Pareto efficiency and argues that quantization sensitivity is largely structural within Transformer architectures.

  • RAMP learns transferable mixed-precision policies via reinforcement learning, achieving Pareto-optimal accuracy-efficiency trade-offs with standardized GGUF deployment.
  • A policy trained on Llama-2-7B generalizes zero-shot to Mistral-7B and Llama-2-13B, often yielding lower perplexity than target-specific policies.
  • Scale Folding stabilizes activation distributions and enables reliable sub-4-bit quantization.
  • A quality-prioritized reward with asymmetric penalties and cliff constraints drives rapid, stable SAC convergence.
  • 5.54 PPL at 3.68 GB versus AWQ’s 5.60 PPL at 3.90 GB gives RAMP a strictly superior Llama-2-7B Pareto point.The stated comparison corresponds to 6% smaller size and 1.1% higher quality.
  • HALO exports GGUF models runnable on CPUs, GPUs, Apple Silicon, and select edge devices without custom kernels.
  • RAMP argues that quantization sensitivity is largely a structural property of Transformer architectures rather than model-instance specific.

8 BROADER CONTEXT AND IMPACT

HALO lowers deployment barriers by making one GGUF artifact portable across major hardware platforms, while the paper frames potential energy benefits as illustrative estimates. The broader agenda includes dynamic, hardware-aware, and multi-objective extensions.

  • Deployment impact: A single GGUF artifact runs unmodified across NVIDIA/AMD GPUs, x86 CPUs, Apple Silicon, and some mobile platforms.
  • Energy and carbon: Illustrative 8 h/day estimates for a 7B-class model indicate 66–75% lower annual carbon emissions than cloud A100 baselines.
  • Energy and carbon: The carbon estimates use stated grid-intensity, pricing, TDP, and benchmark-power assumptions and are order-of-magnitude calculations rather than direct measurements.
  • Future directions: Future directions include hybrid sparsity, input-adaptive policies, hardware co-design, and a general agent mapping models and efficiency targets to compressed artifacts.

A 11-DIMENSIONAL EMBEDDING SPECIFICATION

RAMP represents each quantizable layer with a normalized, fixed-dimensional embedding intended to abstract sensitivity, structural role, and sequential context. Activation statistics are extracted from forward passes and transformed to promote cross-model size invariance.

  • Embedding design: The SAC policy uses an 11-dimensional layer state that abstracts sensitivity, structural role, and sequential context while remaining approximately invariant to model scale.
  • Feature extraction: Activation scales are extracted from linear-layer forward hooks over 128 WikiText-2 training sequences and aggregated as per-channel maxima.
  • Normalization: Logarithmic scaling, clipping to [0,1], and per-model normalization promote size invariance for cross-model generalization.

B SAC HYPERPARAMETER CONFIGURATION

The experiments use one fixed Soft Actor-Critic configuration for sequential bit allocation, with optimization, replay-buffer, schedule, and network details documented in referenced tables. These settings were empirically stable across evaluated models without per-model retuning.

  • A single fixed SAC configuration balances sample efficiency, training stability, and policy expressivity for sequential bit allocation.
  • Optimization hyperparameters, replay-buffer details, training schedule, and actor-critic architectures are documented in Tables 25–27.
  • The configuration remained empirically stable across all evaluated models without per-model retuning.
Loading 2603.17891v1…