Quantized Evolution Strategies: High-precision Fine-tuning of Quantized LLMs at Low-precision Cost

Yinggan Xu; Kajetan Schweighofer; Risto Miikkulainen; Xin Qiu

Source-linked AI summary

Quantized Evolution Strategies: High-precision Fine-tuning of Quantized LLMs at Low-precision Cost

Yinggan Xu, Kajetan Schweighofer, Risto Miikkulainen, Xin Qiu

arXiv:2602.03120v2Feb 3, 2026cs.LGcs.AI

TL;DR

Quantized models are difficult to fine-tune because their parameters are discrete and standard gradient-based methods require backpropagation and continuous, high-precision state. QES performs full-parameter optimization directly in quantized space using accumulated error feedback and stateless seed replay. It improves quantized baselines across reasoning and SFT tasks and outperforms QuZO.

Problem
Quantized optimization can stagnate or produce inaccurate updates because discrete parameters cause gradient signals to vanish or become distorted, while standard first-order methods require non-differentiability-violating operations and high-precision state.
Method
QES combines accumulated error feedback for preserving high-precision update trajectories with stateless seed replay for reconstructing optimization states without storing them.
Results
QES significantly improves quantized baselines and outperforms QuZO across two reasoning tasks and four SFT benchmarks.
Takeaways & Limitations
QES makes full-parameter fine-tuning of quantized models possible at inference-level memory, supporting adaptation in low-resource environments.
Takeaways & Limitations
The current work focuses on standard linear integer quantization, such as INT4, INT8, and W8A8, and uses a fixed replay window K.

Abstract

from arXiv · show

Post-Training Quantization (PTQ) is essential for deploying Large Language Models (LLMs) on memory-constrained devices, yet it renders models static and difficult to fine-tune. Standard fine-tuning paradigms, including Reinforcement Learning (RL), fundamentally rely on backpropagation and continuous weights to compute gradients. Thus they cannot be used on quantized models, where the parameter space is discrete and non-differentiable. While Evolution Strategies (ES) offer a backpropagation-free alternative, optimization of the quantized parameters can still fail due to vanishing or inaccurate gradient estimation. This paper introduces Quantized Evolution Strategies (QES), an optimization paradigm that performs full-parameter fine-tuning directly in the quantized space. QES is based on two innovations: (1) it integrates accumulated error feedback to preserve high-precision weight updating signals, and (2) it utilizes a stateless seed replay to reduce memory usage to low-precision inference levels. QES significantly outperforms the state-of-the-art zeroth-order fine-tuning methods on a variety of tasks, making direct fine-tuning for quantized models possible. It therefore opens up the possibility for scaling up LLMs entirely in the quantized space. The source code is available at https://github.com/dibbla/Quantized-Evolution-Strategies .

1 Introduction

QES enables full-parameter fine-tuning directly in quantized space by preserving high-precision update signals and replaying optimizer states with inference-level memory. Across reasoning and SFT tasks, it improves quantized baselines and outperforms QuZO.

PTQ reduces deployment memory but leaves models essentially static, while direct fine-tuning methods for quantized models remain limited.
Quantized optimization suffers from gradient stagnation and inaccurate updates because discrete parameters can make gradient signals vanish or distort.
QES performs full-parameter fine-tuning directly on low-precision quantized weights while maintaining high-precision learning dynamics through accumulated error feedback.
Stateless seed replay replaces high-precision optimizer-state storage, reducing QES memory consumption to the order of low-precision inference.
QES significantly improves quantized baselines and outperforms QuZO across two reasoning tasks and four SFT benchmarks.

2 Related Work

Prior approaches reduce memory or avoid backpropagation, but quantized fine-tuning remains constrained by latent full-precision state, restricted update spaces, and poor reasoning-task convergence. QES connects quantized evolution with residual-accumulation principles from signal processing.

PTQ enables 3-4 bit inference with negligible performance degradation, motivating memory-efficient deployment of LLMs.
STE-based quantized fine-tuning is biased and requires full-precision latent weights and optimizer states, while QLoRA updates only small high-precision adapters.
RLHF has broad fine-tuning applications but fundamentally relies on backpropagation, limiting its use for non-differentiable quantized parameters.
ZO and ES provide backpropagation-free alternatives for large models, but prior methods were not designed for quantized optimization.
QuZO and QZO extend zeroth-order optimization to quantized SFT but have limited success on reasoning tasks and often fail to converge meaningfully.
Delta-Sigma modulation and error feedback accumulate discretization errors so time-averaged behavior tracks a continuous trajectory more accurately.

3 Method

QES estimates update directions through discrete perturbations, preserves sub-threshold updates with accumulated error feedback, and reconstructs residual states through seed replay. This trades computation for memory while addressing stagnation and discretization errors.

3.1 The Optimization Challenge: Standard first-order methods fail for quantized models because quantization is non-differentiable and high-precision gradients or optimizer states may exceed available memory.
3.1 The Optimization Challenge: ES estimates update directions by perturbing parameters, evaluating reward-based fitness, and aggregating perturbations weighted by their rewards.
3.1 The Optimization Challenge: Standard ES updates can vanish after discretization when their magnitude is smaller than the parameter lattice gap, causing stagnation or inaccurate weight updates.
3.2 Accumulated Error Feedback: QES applies stochastic discrete perturbations directly within the integer quantized parameter space and enforces codebook limits through boundary gating.
3.2 Accumulated Error Feedback: Accumulated error feedback carries fractional update components across iterations until the effective update crosses the rounding threshold, |u| ≥0.5.
3.3 Stateless Seed Replay: Stateless seed replay reconstructs residual states from recent seeds and rewards, reducing optimizer-state memory from O(d) to O(K) while trading additional computation for memory.
3.3 Stateless Seed Replay: Replay assumes historical and current boundary masks are close because discrete updates are sparse, making Wτ approximately equal to Wt.

4 Experiments

QES is evaluated across supervised fine-tuning, arithmetic reasoning, and grade-school mathematics using quantized backbones and comparisons against quantized and full-precision baselines. It consistently improves over QuZO, with gains remaining strong across model sizes, formats, and replay settings.

Experimental setup: QES is evaluated on classification SFT, Countdown arithmetic reasoning, and GSM8K grade-school mathematics across quantized model settings.The setup targets both differentiable and non-differentiable objectives without full-precision gradients or auxiliary optimizer states.
Reasoning: On Qwen2.5-3B W8A8 GSM8K, QES reaches 80.82% while QuZO collapses to 4.40%.Across Countdown and GSM8K, QuZO is especially brittle on coarse INT4 settings and smaller models, whereas QES produces consistent gains.
Reasoning: 82.64% GSM8K accuracy on Llama-3.1-8B INT4 improves over the 64.14% base by 18.5 points.This appendix case study extends the reported gains to a larger model and different architecture family.
Replay ablation: Stateless Seed Replay matches or exceeds the Full Residual oracle in three of six Countdown configurations.The largest gap is Qwen2.5-3B W8A8, while the corresponding GSM8K gap is within 0.76 points.
Replay ablation: With fixed decay, reducing replay window K from 50 to 20 cuts reconstruction cost by 60% while losing only 1.25 accuracy points.Replay compute scales linearly with K; wall-clock training increases by approximately 16.7% for 1.5B and 12.5% for 3B models.

5 Temporal Equivalence to Continuous Optimization

QES maintains a virtual continuous trajectory while storing only discrete weights and accumulated residual error. Its final deviation from the ideal trajectory is bounded by half a quantization step, avoiding the cancellation or random-walk errors of stateless updates.

Failure of stateless updates: Stateless quantized updates can stagnate when update magnitudes fall below half the grid spacing, because quantization cancels the ideal continuous update.Stochastic quantization can instead produce errors that accumulate as a random walk.
Residual feedback: QES introduces residual state e_t and defines virtual continuous parameters Θ_t as the physical quantized weights plus accumulated residual error.The residual preserves update information that would otherwise be discarded by quantization.
Temporal equivalence: QES update rules make Θ_t evolve according to unconstrained, high-precision gradient-ascent dynamics despite discrete physical weights.The derivation rewrites the quantized update and residual correction as Θ_t + αĝ_t.
Error bound: The physical quantized weights deviate from the ideal trajectory only through the final residual e_T, with ||e_T||∞ ≤ Δ/2.Because Θ_T rounds to the nearest grid point, QES remains within half a grid step of the ideal trajectory.

6 Limitation and Future Work

The current study is limited to standard linear integer quantization and fixed-window replay. Future work proposes broader quantization formats and adaptive replay control.

Scope boundary: QES is currently demonstrated only with standard linear integer quantization, specifically INT4, INT8, and W8A8.The authors identify binary networks and floating-point formats such as FP4 as possible extensions.
Replay control: Stateless replay currently uses a fixed lookback window K and decay rate γ, requiring manual tuning of the compute-memory trade-off.An adaptive algorithm could tune K and γ using convergence stability or available hardware resources.
Future scaling: QES may support training more parameters within the same hardware memory by trading precision for parameter count and using low-precision inference during updates.The paper presents this as a future scaling direction rather than a demonstrated general capability.

7 Conclusion

QES addresses gradient stagnation, discretization error, and memory barriers by combining accumulated error feedback with stateless seed replay for direct quantized-space optimization.

QES targets gradient stagnation, inaccurate discretized updates, and prohibitive memory requirements in quantized LLM fine-tuning.
Accumulated error feedback preserves minuscule learning signals so QES can progress on low-precision, non-differentiable landscapes.
Stateless Seed Replay replaces stored high-precision error states, bringing full-parameter fine-tuning within standard quantized-inference memory constraints.
QES was evaluated empirically on challenging reasoning and SFT settings using the stated optimization defaults.

A.2 SFT Tasks

The SFT experiments use prompt-based k-shot protocols and compare QES with first-order, MeZO, and QuZO baselines under specified RoBERTa-large settings.

RoBERTa-large is evaluated on SNLI, MNLI, RTE, and SST-5 using prompt-based k-shot fine-tuning and mean accuracy across three seeds.
The first-order FP32 and W8 baselines use AdamW at a 1×10^-5 learning rate, with W8 weights snapped to the quantization grid after optimizer steps.
QES uses population N = 8, decay γ = 0.9, and perturbation scale σ = 1×10^-3 across the SFT experiments.
Reasoning experiments use Qwen2.5-1.5B and -3B quantized to INT4, INT8, or W8A8, with binary correctness evaluated on held-out problems.
For reasoning configurations, Seed Replay and Full Residual share σ and α, while differing in residual storage versus seed replay rematerialization.

C Scaling Case Study: Llama-3.1-8B

The Llama-3.1-8B INT4 case study reuses Qwen2.5-3B hyperparameters and reports strong QES gains while retaining a small optimizer-state footprint.

QES on Llama-3.1-8B uses the fixed Qwen2.5-3B INT4 configuration without per-model hyperparameter search.
In Countdown curves, QES tracks the Full-Residual oracle closely while QuZO frequently collapses on INT4 and the 1.5B model.
18.5 points: QES improves GSM8K accuracy from 64.14% to 82.64% on Llama-3.1-8B INT4.
The result extends across architecture family and model scale, from Qwen2.5 to Llama-3.1 and from 1.5B/3B to 8B.
Larger models are described as offering more redundant parameters and smoother loss landscapes, matching larger QES gains on GSM8K.
QES holds optimizer state to approximately 29.7 KB, whereas FP16 residuals for an 8B model would require roughly 16 GB of additional VRAM.

D Detailed Ablation Results

The ablation appendix compares Stateless Seed Replay with Full Residual and reports replay-window, decay, fidelity, memory, and runtime analyses summarized in the main text.

Table 6 compares Stateless Seed Replay with the Full Residual oracle across six Countdown model-format configurations.
Table 7 reports replay-window K and decay γ effects alongside update and boundary-hit ratios for fidelity analysis.
The appendix also details peak VRAM and wall-clock measurements summarized in Section 4.

E.1 Memory

QES matches inference-level memory footprints by replacing per-parameter residual storage with a small stateless replay buffer, while trading additional computation for memory.

Memory footprint: QES holds optimizer state at approximately 29.7 KB across configurations, matching QuZO’s inference-only total footprint.The buffer stores seed-and-reward history rather than full-precision optimizer states.
Memory footprint: Full Residual adds 2.44 GB for 1.5B models and 5.17 GB for 3B models, largely undoing quantization’s memory benefit.It materializes FP16 residuals for every parameter.
Memory footprint: Standard QAT requires over 24 GB of VRAM for a 3B model, roughly 13× the QES INT4 footprint.The QAT footprint includes full-precision weights, gradients, activations, and optimizer states.
Memory–computation trade-off: Replay adds approximately 16.7% overhead for 1.5B models and 12.5% for 3B models under equal hardware budgets, with parallelization available during updates.The practical penalty depends on the inference engine and target hardware rather than being a strict linear multiplier.
Memory–computation trade-off: Using K=20 retains most of K=50’s accuracy at 40% of reconstruction cost, allowing a tunable throughput–accuracy trade-off.Performance degrades gracefully as K decreases when decay γ is held constant.