Training Dynamics Impact Post-Training Quantization Robustness

Motivation

• Post-training quantization maps high-precision models to lower-precision representations and supports efficient low-bit inference.The introduction frames PTQ as a widely adopted deployment workflow enabled by specialized hardware and quantization algorithms.Source: §1 · p. 1

Scope and contributions

• The study analyzes quantization error across open-source LLM trajectories up to 32B parameters and 15T tokens, then verifies training-dynamics effects in controlled experiments.It measures hundreds of intermediate checkpoints and studies learning-rate scheduling and weight averaging as interventions.Source: §1 · p. 2

Controlled interventions

• Maintaining larger learning rates reduces quantization error when other hyperparameters remain fixed.The paper also reports that optimized learning-rate schedules and weight averaging can improve quantization performance.Source: §1 · p. 2

Mechanistic analysis

• Quantization robustness is linked to the geometry of the loss, with the proposed interventions associated with flatter minima.The paper analyzes loss geometry to connect learning-rate and weight-averaging interventions with quantization performance.Source: §1 · p. 2

Scope and contributions

• Training hyperparameters and resulting dynamics significantly change how easily modern LLMs can be quantized.The authors recommend monitoring PTQ throughout pretraining and during hyperparameter selection because seemingly similar learning-rate choices can diverge in later quantization error.Source: §1 · p. 2

2.1 POST-TRAINING QUANTIZATION

• Post-training quantization reduces numerical precision to lower memory use and can improve speed in memory-bandwidth-limited LLM inference.Modern methods may optimize weight, activation, and KV-cache quantization using calibration data and reconstruction-error objectives.Source: §2.1 · pp. 2–3
• Quantization methods use auxiliary scaling, grouping, and outlier-handling strategies to represent tensors in low precision.These approaches divide the problem into smaller groups and preserve dynamic-range information through high-precision states.Source: §2.1 · p. 3

2.2 LLM TRAINING HYPERPARAMETERS

• Learning-rate schedules are central pretraining choices, including cosine decay and Warmup–Stable–Decay schedules with constant and linear-decay phases.Peak and post-annealing learning rates can also affect performance, scaling-law derivation, and adaptability.Source: §2.2 · p. 3

2.3 MODEL BRITTLENESS TO POST-TRAINING QUANTIZATION

• Under OLMo2 trajectories, quantization error grows gradually during cosine decay but rises sharply under steep linear annealing, while model souping lowers PTQ error below individual runs.The figure compares 3-bit quantization error across training trajectories and highlights weight averaging as a robustness intervention.Source: Figure 2 · p. 4
• Earlier scaling studies associated more training data with higher quantization degradation, but they overlooked training dynamics.The paper finds that degradation abruptly increases as learning rates decay regardless of training-data size, and that prior results are largely confounded by learning-rate hyperparameters.Source: §2.3 · pp. 3–4

• Quantization error rises sharply during steep linear annealing after increasing more gradually during slow cosine decay, linking degradation to optimization dynamics across OLMo2 models.Source: §3.1 · pp. 4–5
• Across 12 downstream benchmarks, 3-bit accuracy degradation follows a pattern similar to validation-loss degradation and generally increases as the learning rate decays.Source: §3.2 · p. 5
• In SmolLM3, context extension reduces PTQ degradation, mid-training amplifies it, and subsequent SFT and APO decrease it.Source: §3.2 · p. 5
• The released SmolLM3 model, averaging mid-training and APO weights, has lower 3-bit quantization degradation than either component.Source: §3.2 · p. 5, Figure 3 · p. 5

4.1 REPLICATING THE OBSERVED PHENOMENA

• Under WSD, models trained for 10B–100B tokens reach comparable post-decay quantization error, showing that error spikes track learning-rate decay rather than token count.Source: §4.1 · p. 6

• Quantization error is shaped by training dynamics, motivating continuous PTQ monitoring and hyperparameter evaluation before large-scale runs.Source: §4.1 · p. 6

4.2 SCALING TRENDS IN PRIOR WORK ARE DOMINATED BY LEARNING RATE SCHEDULES

• Cosine decay reproduces increasing δP T Q with larger token budgets, whereas WSD slows growth at 70M and eliminates it at 160M.Source: §4.2 · p. 6, Figure 5 · p. 7
• The results challenge attributing quantization degradation to dataset size alone because comparable WSD schedules produce lower validation loss with slower or stable degradation.Source: §4.2 · p. 6
• Checkpoint comparisons across token counts require caution because learning-rate schedules affect observed robustness and scaling-law estimates.Source: §4.2 · pp. 6–7

5.1 LEARNING RATE

• At similar validation loss, larger learning rates achieve better 4-bit and 3-bit quantization with no apparent cost.Comparing learning rates of 1e−3 and 3e−3 shows the larger rate has lower quantization error at comparable full-precision performance.Source: §5.1 · p. 7, Figure 6 · p. 7
• Cosine schedules achieve lower quantization error and better validation loss than WSD schedules despite the cosine schedule’s rapid decay.The comparison indicates that the schedule’s learning-rate trajectory, not only its peak rate, is associated with quantization performance.Source: §5.1 · p. 7
• LAWA produces lower 3-bit quantized validation loss than all tested cooldowns, although it remains worse than cooldowns for full-precision validation.The experiment averages checkpoints from the stable phase of a 160M-parameter model trained on 100B tokens at constant learning rate.Source: Figure 7 · p. 8, §5.2 · p. 8

5.2 WEIGHT AVERAGING

• Averaging checkpoints from open-source OLMo-1B trajectories improves both full-precision and quantized performance over the final checkpoint.This result holds despite limited control over checkpoint-saving frequency, supporting weight averaging as a practical direction for improving PTQ robustness.Source: §5.2 · p. 8

5.3 WEIGHT DECAY

• Among models with comparable full-precision performance, larger decoupled weight decay yields lower 4-bit and 3-bit quantization error.The analysis uses a training implementation that decouples learning rate from weight decay.Source: §5.3 · p. 8

6.1 LOSS LANDSCAPE

• Loss-landscape flatness is proportional to learning-rate magnitude, and the quantized model’s distance from the trained model grows with larger learning rates.The 2D slices use the final checkpoint, a previous checkpoint, and its 4-bit GPTQ-quantized version; all slices contain a local minimum around the final model.Source: §6.1 · p. 9, Figure 8 · p. 9
• The quantization direction in the loss landscape is structured similarly to the direction of the previous optimization step.This links the geometry around the quantized model to geometry induced by training, while smaller learning rates place successive checkpoints closer together.Source: §6.1 · p. 9

6.2 CURVATURE

• Learning-rate decay consistently increases sharpness, while constant learning rates largely preserve a low Hessian spectrum after an initial top-eigenvalue rise.Sharpness, Hessian trace, and leading eigenvalues are estimated along a 160M-transformer trajectory during stable and decay phases.Source: §6.2 · p. 9, Figure 9 · p. 10
• The authors hypothesize that learning-rate decay moves models through sharper regions, increasing sensitivity to perturbations such as quantization.This is presented as a hypothesis based on the observed relationship between learning-rate decay, curvature, and quantization robustness.Source: §6.2 · p. 10
• Weight averaging leads to wider minima and improved Hessian conditioning, which may explain LAWA’s superior quantization robustness.The analysis distinguishes averaging from learning-rate decay: both can improve performance, but they produce solutions with substantially different curvature properties.Source: §6.2 · p. 10

7 DISCUSSION

• Learning rate magnitude, checkpoint averaging, and model souping all modulate quantization robustness under controlled experiments.The study associates these interventions with convergence to flatter minima that may benefit quantization performance.Source: §7 · p. 10
• Quantization degradation is not presented as an unavoidable consequence of training-data scale, because existing training tools can improve low-bit quantization quality.The conclusion emphasizes an intricate relationship between training dynamics and general model robustness.Source: §7 · p. 10

A QUANTIZATION PROTOCOL

• The study replicates its learning-rate-driven quantization trends with LLM.int8() and AWQ in addition to GPTQ.GPTQ is the central method because it is popular and accessible, while the alternatives test whether the phenomena are broader PTQ trends.Source: A QUANTIZATION PROTOCOL · p. 18, Figure 10 · p. 18
• GPTQ quantizes linear layers with default GPTQModel and HuggingFace settings, using C4 calibration data and group size 128.AWQ uses Kwon et al. (2023), while LLM.int8() follows the stated HuggingFace implementation.Source: A QUANTIZATION PROTOCOL · p. 18

B PTQ ROBUSTNESS ON ADDITIONAL MODELS IN THE WILD

• Some model families exhibit unpredictable quantization behavior: Amber’s 4-bit degradation rises sharply after a transient validation-loss spike, despite full-precision recovery.The cause of this training-dynamics change is unidentified.Source: B PTQ ROBUSTNESS ON ADDITIONAL MODELS… · p. 18
• Figures 11–15 report quantization degradation for OpenSci-1.3B, Amber-7B, Apertus-8B, OLMo-1 1B, and OLMo-1 7B using 3- and 4-bit GPTQ where specified.OpenSci-1.3B was trained on 1T Nemotron-cc tokens.Source: Figure 11 · p. 18, Figure 12 · p. 19, Figure 13 · p. 19
• The analysis supplements these model-family comparisons with open-source training configurations and a weight-averaging procedure using checkpoints collected throughout training.LAWA uses a rolling window of five checkpoints, while released-checkpoint analyses average consecutive checkpoints over varying window lengths.Source: C PRETRAINING HYPERPARAMETERS AND SETUP · p. 20

D EVALUATION

• Evaluation uses held-out RefinedWeb validation loss and downstream accuracy across twelve tasks, with full-precision, 3-bit GPTQ, and 4-bit GPTQ SmolLM3 measurements.The task suite includes ARC, OpenbookQA, PIQA, HellaSwag, WinoGrande, MathQA, PubMedQA, SciQ, Social IQa, CommonsenseQA, and MMLU.Source: D EVALUATION · p. 20
• Weight decay improves PTQ robustness: with fixed learning rate and WSD, larger decay coefficients yield more robust models.The comparison varies only the weight-decay parameter λ, using a fixed learning rate of 3e−3.Source: Figure 19 · p. 22
• AdamW shows a late-training loss-gradient spike, while AdamC reduces that spike and changes weight norms without directly modulating quantization error.These optimizer comparisons hold other hyperparameters fixed.Source: Figure 20 · p. 22, §E.2 · p. 22
• Cosine-schedule PTQ error varies with training horizon, whereas peak learning rate and scheduler shape have larger effects than duration alone.The comparison uses 160M models trained with different token budgets under cosine decay and WSD schedules.Source: §E.3 · pp. 22–23, Figure 21 · p. 23
• Larger learning rates produce lower quantization degradation even at similar validation loss in OLMo2-7B during a 50B-token annealing run.The experiment follows 250B-token pretraining and evaluates four learning-rate values.Source: §E.4 · p. 23

F ADDITIONAL DETAILS AND RESULTS FOR LOSS LANDSCAPES

• The loss landscape is constructed by interpolating among the final, previous, and quantized checkpoints in a two-dimensional plane and evaluating validation loss on sampled grid points.The directions span the previous-training-step and quantization perturbations from the final model.Source: F ADDITIONAL DETAILS AND RESULTS FOR… · p. 23
• The 3-bit GPTQ loss landscape shows the same pattern as the main analysis, with flatness of the basin becoming more relevant under larger weight perturbations.The landscape is visualized using checkpoints from the training trajectory and a dequantized approximation of the quantized model.Source: F ADDITIONAL DETAILS AND RESULTS FOR… · p. 23
• Hessian trace is estimated with Rademacher-sampled Hessian-vector products, while sharpness is estimated as the largest Hessian eigenvalue using power iterations.Both statistics are measured on held-out validation sequences from in-house Pythia-160M models.Source: G SECOND ORDER STATISTICS · pp. 23–25
• Cosine schedules achieve lower quantization error and better validation loss than WSD in the compared setting, while their loss trade-off from full precision to quantized weights is sharper.The comparison indicates that schedule shape matters beyond learning-rate magnitude at any single point.Source: Figure 22 · p. 24
• LAWA lowers validation loss before and after 4-bit quantization compared with individual checkpoints, with larger averaging windows performing best.The analysis averages weights along the OLMo-1B trajectory.Source: Figure 24 · p. 24
• Larger learning rates yield flatter loss basins, lower sharpness, smaller Hessian-trace estimates, and lower quantization error.The results suggest convergence to wider minima when maximum learning rates are higher, even when schedules decay to zero.Source: Figure 26 · p. 25, Figure 25 · p. 25

H LIMITATIONS

• The analysis leaves optimizer choice and schedule-free methods largely unexplored, so its conclusions do not cover the full optimization pipeline.The authors identify these areas as follow-up work.Source: H LIMITATIONS · p. 25
• Experiments focus on dense quadratic models, while conclusions about sparse and sub-quadratic architectures are expectations rather than directly tested results.The authors expect similar conclusions for those architectures but do not establish them here.Source: H LIMITATIONS · p. 25