PaperlayerView exact revision on arXiv

Source-linked AI summary

Replay the Curvature: Accurate and Scalable NVFP4 Quantization for Large Language Model Inference

Ruiyi Ding, Jie Li, Kang He, Ziyan Liu, Chengru Song, Yuedong Xu, Yuan Cheng

arXiv:2609.36654v1cs.LGcs.CLcs.DC

TL;DR

GPTQ’s sequential updates complicate block-scale selection, and large-model reconstruction also faces accelerator-memory and within-layer execution bottlenecks. The paper combines Schur Replay, which evaluates scales in their induced GPTQ state, with memory-bounded output-row-parallel execution. Across seven benchmarks, the resulting models retain 99.35% and 100.84% question-weighted BF16 recovery.

  • Problem

    Block scales are difficult to select because GPTQ updates later columns after each quantization, and large-model weights, activations, and second-order state cannot all remain on one accelerator.

    Source: §1 · pp. 1–2

  • Method

    Schur Replay replays GPTQ updates for candidate scales and scores resulting block error, while the execution infrastructure tiers activations and distributes independent output rows across tensor-parallel ranks.

    Source: §1 · p. 2

  • Results

    99.35% and 100.84% question-weighted BF16 recovery across seven benchmarks on Qwen3.5-397B-A17B and Llama-3.3-70B-Instruct, respectively.

    Source: §1 · p. 2, §6 · p. 9

  • Takeaways & Limitations

    The results show accurate and scalable hardware-valid NVFP4 conversion without retraining in the evaluated models and benchmarks.

    Source: §6 · p. 9

    Evidence is empirical: the exhaustive scale audit covers sampled groups, and downstream evaluation spans two architectures and seven tasks.

    Source: §6 · p. 9

Abstract

from arXiv · show

Large language models make weight storage and memory traffic major inference costs, motivating low-precision formats that represent each weight with only a few bits. Such formats use a scale to map floating-point values into a small codebook; NVFP4 improves local range utilization by letting every 16 E2M1 weights share an E4M3 block scale. Choosing that scale is difficult in GPTQ because quantizing one column updates those that follow, so evaluating a block independently can misestimate its final reconstruction error. Large models pose a second challenge: full-precision weights, calibration activations, and second-order state cannot all remain on one accelerator, while assigning complete layers to devices leaves each time-consuming layer solve serial. We introduce \emph{Schur Replay}, a scale-selection algorithm that reproduces the GPTQ updates caused by each block scale and scores the resulting block error after accounting for compensation from unquantized columns. Separately, our execution infrastructure keeps only the active layer resident, tiers activations across device, host, and disk, retires full-precision layers after export, and distributes independent output rows across tensor-parallel ranks. Together, the algorithm and infrastructure attain $99.35\%$ and $100.84\%$ question-weighted recovery from BF16 across seven benchmarks on Qwen3.5-397B-A17B and Llama-3.3-70B-Instruct. On the 397B model, the infrastructure reduces measured per-layer time by $15.17\times$ over ModelOpt and $23.14\times$ over LLM Compressor, with lower memory used per GPU.

1 Introduction

NVFP4’s shared block scales make scale choice consequential, while GPTQ’s sequential updates can change later reconstruction errors. The paper introduces Schur Replay and memory-bounded, output-row-parallel execution to address scale selection and large-model reconstruction.

1 Introduction

  • NVFP4 uses a tensor-wide FP32 scale and one E4M3 block scale per 16 E2M1 weights, for approximately 4.5 bits per value.PTQ selects scales from calibration data without retraining, and GPTQ uses activation-derived second-order information to compensate rounding error.Source: §1 · p. 1
  • GPTQ’s sequential updates mean initially similar block scales can lead to different final reconstruction errors.Existing NVFP4 methods alter representation or scale initialization; this work instead evaluates scales in their induced GPTQ state.Source: §1 · pp. 1–2
  • Large-model reconstruction is constrained because weights, calibration activations, and second-order state cannot all remain in accelerator memory, while assigning a whole layer to one device leaves its solve serial.The infrastructure targets both memory residency and parallelism within a layer.Source: §1 · p. 2
  • Schur Replay evaluates each exportable scale in the GPTQ state it creates, replaying its updates and scoring block error after accounting for compensation from unquantized columns.The algorithmic contribution is described as recurrence-aware NVFP4 reconstruction that captures feedback within each 16-value block.Source: §1 · p. 2
  • The execution infrastructure keeps only the active layer resident, tiers activations across device, host, and disk, and distributes independent output rows across tensor-parallel ranks.Full-precision layers are retired after recoverable NVFP4 export.Source: §1 · p. 2
  • 99.35% and 100.84% question-weighted BF16 recovery across seven benchmarks, with 15.17× and 23.14× faster measured per-layer compute than ModelOpt and LLM Compressor on the 397B model.The recovery results cover Qwen3.5-397B-A17B and Llama-3.3-70B-Instruct, respectively.Source: §1 · p. 2

2 Related Work

Related work spans PTQ methods based on rounding, second-order compensation, calibration, and rotations, alongside training-based approaches. Existing NVFP4 methods change representation or initialization, while Schur Replay scores scales in the GPTQ recurrence; existing infrastructure supports offload and execution but not the paper’s stated combination of layer-wise parallel solving and active-layer residency.

2 Related Work

3 Preliminaries

The preliminaries frame reconstruction-based PTQ around a linear layer’s weights and calibration inputs. GPTQ uses a shared Hessian factor while processing input columns sequentially, with output rows remaining independent.

3 Preliminaries

  • For a linear layer, reconstruction-based PTQ is formulated using weight matrix W and calibration inputs X, with M, N, and T denoting output width, input width, and calibration positions.The passage defines H = XX^T and the quantization error Δ = Wq − W.Source: §3 · p. 3
  • OBS and OBC/OBQ compensate forced quantization errors through remaining unconstrained coordinates, using the shared input Hessian structure [Hassibi et al., 1993, Frantar et al., 2022].The output rows share one input Hessian but contribute independent quadratic objectives.Source: §3 · p. 3
  • GPTQ quantizes output rows in a common input-column order and reuses one Hessian factor.Quantization is sequential across columns because each error changes later values, while output rows remain independent; NVFP4’s shared scale across consecutive columns makes initial-block-only scale selection ignore recurrence-produced values.Source: §3 · p. 3

4 Method

The method pairs Schur Replay’s recurrence-aware group-scale selection with a separate execution infrastructure that preserves the row-wise GPTQ solve. It uses conditional reconstruction costs to compare scales and distributes large-model reconstruction across storage tiers and ranks.

4.1.1 NVFP4 Group Quantization

  • Each NVFP4 group shares one positive scale across 16 consecutive columns, so its scale must account for GPTQ’s sequential updates within the group.The group quantizer rounds normalized values to the E2M1 levels.Source: §4.1.1 · p. 4

4.1.2 Conditional Group Metric

  • The conditional Schur score measures group error after fixing the committed prefix and allowing the unquantized future to compensate optimally.This is a candidate-ranking surrogate, not the final discrete layer loss.Source: §4.1.2 · p. 4

4.1.3 Scale Proposal by Conditional Quadratic Minimization; 4.1.4 Recurrence-Exact Replay on the Exportable Grid

  • A fixed-code quadratic update proposes a scale neighborhood, then recurrence-exact replay evaluates exportable candidates from the same block-entry state before committing the winner.This separates an inexpensive proposal from the final decision, which accounts for scale-dependent codes and intra-group GPTQ updates.Source: §4.1.3 · p. 5, §4.1.4 · p. 5, Figure 1 · p. 6

4.1.4 Recurrence-Exact Replay on the Exportable Grid

  • With h = 4, the local window contains a global surrogate optimum for 83.28% of Qwen rows and 83.53% of Llama rows; h = 8 exceeds 99.96% for both.The audit covers sampled groups; retaining the status-quo scale also guarantees the local search does not increase the calibration-defined surrogate.Source: §4.1.4 · p. 6

4.2 Faster PTQ Framework

  • The infrastructure changes placement and scheduling, not the quantization objective or update order within a row.It combines layer-wise memory virtualization with same-layer output-row-parallel GPTQ.Source: §4 · p. 3, §4.2 · p. 6

4.2.1 Batched Candidate Replay and Parallel Commit; 4.2.4 Same-Layer Output-Row-Parallel GPTQ

  • Batched replay processes independent row–candidate pairs while retaining the original column order, and row-parallel ranks solve disjoint output rows using a shared Hessian.The batching exposes independent work without replacing the sequential GPTQ recurrence with an independent-block approximation.Source: §4.2.1 · p. 6, §4.2 · p. 6, §4.2.4 · p. 7

4.2.2 Aggressive Layer-Wise Memory Virtualization

  • GPU memory prioritizes the active layer and solver-critical state, while lower-priority activations and reconstruction state spill through CPU memory, SSD, and HDFS.Only the active layer is installed on device during calibration; the next layer can be prefetched while the current one is solved.Source: §4.2.2 · p. 7

4.2.3 Recoverable Streaming Export

  • Each completed layer is packed into an NVFP4 fragment, and an atomic manifest advances only after the fragment and next-layer inputs are durable.This supports layer-grained recovery and full-precision retirement.Source: §4.2.3 · p. 7

5 Experiments

Across seven benchmarks, Schur Replay achieves the highest question-weighted recovery among the compared quantized methods on both evaluated models. The experiments also report ablations and infrastructure measurements, with projected rather than measured full-model runtimes for 397B.

5.1 Setup and Baselines

  • Accuracy evaluation uses 15,461 questions across seven datasets, while comparisons include GPTQ, rescaling, scale-selection, residual-channel, and micro-rotation methods.Both models share a decontaminated calibration set and deterministic single-sample greedy evaluation protocol.Source: §5.1 · p. 8

5.2 W4A4 Accuracy

  • 99.35% and 100.84% question-weighted recovery on Qwen3.5-397B-A17B and Llama-3.3-70B-Instruct exceed ScaleSweep’s 97.28%/99.03% and MR-GPTQ’s 93.86%/91.81%.Tables 1 and 2 report all seven task scores; BF16 is the full-precision reference.Source: Table 1 · p. 8, Table 2 · p. 8, §5.2 · p. 8

5.3 Ablations

  • Removing Schur scoring, recurrence replay, or both loses 90, 15, or 115 answers, while increasing h from 4 to 8 raises surrogate-optimum coverage but does not improve recovery.With ten candidates for both models, Qwen macro recovery is 98.92% at h = 4 and 98.62% at h = 8.Source: §5.3 · p. 9

5.4 Infrastructure Efficiency

  • 120.25 seconds per steady layer for our 397B pipeline compares with 1,824.29 for ModelOpt and 2,782.89 for LLM Compressor; peak memory is 35.0, 218.6, and 86.6 GB/GPU, respectively.The corresponding 397B full-model totals are projections; Schur Replay calibration measurements are excluded from infrastructure speedup ratios.Source: §5.4 · p. 9, Figure 3 · p. 9

6 Conclusion

Schur Replay and separate memory-bounded, output-row-parallel GPTQ infrastructure produce accurate NVFP4 models without retraining in the reported experiments. The conclusion bounds the evidence to two architectures and seven tasks, and the appendices and planned artifacts support reproduction.

6 Conclusion

  • The combined method retains 99.35% and 100.84% question-weighted BF16 recovery across seven benchmarks, while 397B full-model runtimes remain projections.The infrastructure reaches 35.0 GB peak memory per GPU and reduces measured per-layer and 70B full-model quantization compute against evaluated public toolchains.Source: §6 · p. 9
  • The evidence is empirical, covering sampled groups and downstream evaluation on two architectures and seven tasks; transfer to other formats, calibration regimes, and model families remains future work.Complete end-to-end measurements at the largest deployments are also identified as future work.Source: §6 · p. 9

AI use statement / Ethics statement

  • The authors report AI assistance for code drafting, debugging, and language polishing, with author review and responsibility for the final work.The quantitative experiments use public models and benchmarks, while a proprietary production integration is not used to support quantitative claims.Source: AI use statement · p. 10, Ethics statement · p. 10

Reproducibility statement / A Code & Artifact Availability

  • The appendix supplies pseudocode, derivations, calibration details, ablation definitions, and additional measurements; planned releases include code, calibration data, and quantized artifacts subject to license terms.The proprietary Prompt Enhancer, internal evaluation assets, and production-specific integration are outside the release scope.Source: Reproducibility statement · p. 10, A Code & Artifact Availability · p. 14

B Complete Ablation Tables

The ablation separates recurrence-aware residual construction from conditional Schur scoring, showing that both improve the controlled aggregate. Schur scoring has the larger main effect, and removing both components adds a negative interaction.

B Complete Ablation Tables

  • The four controlled cells cross replay versus static residuals with conditional Schur versus Euclidean scoring; none reproduces a published method.The two table panels together report the per-dataset absolute correct counts for all cells.Source: B Complete Ablation Tables · p. 15, Table 4 · p. 16
  • Schur scoring evaluates a shared block scale under the curvature remaining after still-unquantized future columns are eliminated.This extends the conditional metric from single-weight decisions to shared-scale selection.Source: B Complete Ablation Tables · p. 15
  • Replacing Schur scoring with Euclidean scoring loses 90 questions, while replacing replay with static residuals loses 15; removing both yields 11,853 correct, 10 below the additive prediction.The joint loss is 0.871 recovery points by the additive prediction, with an additional 10-question (0.083-point) interaction.Source: B Complete Ablation Tables · p. 15, Table 4 · p. 16

B.1 Weight-Only NVFP4 (W4A16)

The W4A16 comparison isolates weight quantization by retaining BF16 activations, while the appendix pseudocode records the replay and execution procedures. In this setting, Schur Replay leads weight-reconstruction and recovery comparisons, although reconstruction error does not always predict end-to-end accuracy.

B.1 Weight-Only NVFP4 (W4A16)

  • Schur Replay has the lowest weight-reconstruction error and highest recovery in both act-order settings, with a larger advantage over GPTQ than the gap between GPTQ variants.Across the five arms, reconstruction-error ordering matches recovery ordering, unlike some W4A4 results where activation quantization adds another error source.Source: §B.1 · p. 16, Table 6 · p. 17
  • Act-order helps Schur Replay on both metrics but lowers GPTQ recovery from 98.84% to 96.28% despite improving its reconstruction error.Thus weight-reconstruction error alone does not determine end-to-end accuracy.Source: §B.1 · p. 16
  • The single arm above 100% weighted recovery offsets losses on LiveCodeBench and GPQA-Diamond with small gains on GSM8K and HumanEval.Single-run greedy decoding does not resolve differences of this size.Source: §B.1 · p. 16

C Pseudocode

  • The appendix pseudocode proposes a local exportable-scale window, replays the sequential group recurrence for its candidates, scores conditional Schur losses, and commits the selected scale.The candidate replay updates later columns using each normalized quantization error before evaluating the resulting residual.Source: C Pseudocode · p. 17
  • The execution pseudocode bounds accelerator residency to one active layer and tiers activations across GPU, host, and disk storage.It also describes recoverable layer export and a row-parallel GPTQ schedule that assigns independent output rows across ranks.Source: C Pseudocode · p. 18

D.1 Partitioned Proof of the Conditional Metric

The partitioned proof derives the conditional group curvature by eliminating already committed and still-free coordinates through Schur complements. The accompanying appendix passages specify calibration and quantizer settings and identify the scope of the benchmark-count tables.

D.1 Partitioned Proof of the Conditional Metric

  • The partitioned derivation removes the committed prefix contribution and eliminates the still-free future coordinates to obtain the prefix-conditional group curvature.The derivation uses the damped triangular factor and does not assume that inversion commutes with taking principal submatrices.Source: §D.1 · p. 19

D.2 Calibration and Quantizer Configuration

  • Both model families use the same 255-document calibration set, while the scale proposal uses three iterations and both headline arms use a fixed replay half-window h = 4.The incoming exportable scale is retained, ties are deterministic, and the production Qwen Hessian is damped at 1% of its mean diagonal.Source: §D.2 · pp. 19–20

D.3 Quantized Module Coverage

  • The appendix identifies exact correct-question-count tables for Qwen3.5-397B-A17B and Llama-3.3-70B-Instruct.The Qwen table caption identifies SOAR as the strongest external quantized baseline by question-weighted recovery.Source: Table 7 · p. 21, Table 8 · p. 21

D.4 Benchmark Evaluation, Provenance, and Uncertainty

The appendix documents benchmark scoring and paired uncertainty alongside an exhaustive audit of the scale proposal’s local search window. The audit cautions that surrogate-optimum coverage does not establish downstream accuracy ordering.

D.4 Benchmark Evaluation, Provenance, and Uncertainty

  • Evaluation uses seven datasets totaling 15,461 questions, with one deterministic greedy completion per model–dataset pair.The scoring pipeline reproduces reported dataset percentages, and a second execution changes none of 6,768 repeated per-question verdicts.Source: §D.4 · p. 21
  • Schur Replay leads question-weighted recovery at 99.35% on Qwen and 100.84% on Llama, while the Llama macro scores span just 0.22 points across ARCQuant, AWQ+GPTQ, and Schur Replay.The close macro ranking is descriptive, not a monotone scaling result, because architecture and task mix differ between the models.Source: §D.4 · p. 22, Table 9 · p. 22
  • On Qwen, Schur Replay exceeds SOAR by 0.750 percentage points with a 95% paired-bootstrap interval of [0.220, 1.281] and McNemar p = 0.0051.On Llama, the 0.388-point advantage has an 80% paired-bootstrap interval of [0.013, 0.763].Source: §D.4 · p. 22

D.5 Exhaustive Exportable-Scale Audit

  • At h = 4, the proposal-centered window contains the exhaustive surrogate optimum for 83.28% of Qwen rows and 83.53% of Llama rows.Layer-wise coverage ranges from 82.79–85.83% on Qwen and 83.10–85.06% on Llama.Source: §D.5 · p. 23, Figure 4 · p. 23
  • Widening h from 4 to 8 raises coverage of the surrogate optimum from 83.28% to 99.965% and doubles the replayed neighborhood, while macro recovery changes from 98.92% to 98.62%.Coverage concerns the calibration-defined surrogate, whereas recovery is measured on downstream tasks; the sweep does not establish a monotone relation between search width and downstream quality.Source: §D.5 · p. 23

D.6 Layer-Wise Recovery and Numerical Validation

The scaling test measures calibration-state execution on a Qwen3.5-397B-width mock layer, while the numerical-validation and layer-wise mechanisms support reliable, memory-bounded recovery. The optimized path alone completes the 8,192-block test among the evaluated paths.

D.6 Layer-Wise Recovery and Numerical Validation

  • The stress test fixes sequence length at 16,384 and scales calibration blocks from 32 to 8,192 on an eight-GPU, 72 GB-per-GPU node.The node also has 3.5 TB of local SSD capacity and mounted HDFS storage.Source: §D.6 · p. 23, Figure 5 · p. 24
  • At 8,192 blocks, only the optimized path completes, in 1,080.9 seconds; both public baselines fail under host-memory pressure.The measurements use a Qwen3.5-397B-width mock layer, not an end-to-end model run.Source: §D.6 · p. 23, Figure 5 · p. 24
  • The forward pass keeps the current layer active, uses cached inputs for its predecessor, and replaces older layers with metadata-based skip modules.The current layer captures its inputs and terminates the forward pass.Source: §D.6 · pp. 23–24
  • After packing, retired parameters become zero-element tensors, making accidental reuse fail rather than propagate meta-valued outputs.This preserves the active-layer residency bound despite framework hooks that might otherwise move parameters automatically.Source: §D.6 · pp. 23–24
  • Layer outputs are checkpointed with an atomic manifest only after next-layer inputs are written, and resume checks model depth and checkpoint interval.Validated fragments are retained for resumed runs; finalization checks digests and rejects duplicate module keys or conflicting absorbed tensors.Source: §D.6 · p. 24
  • Validation checks relocation, batching, graph replay, spill, checkpoint, and distributed-solve paths against reference implementations.Hessian inversion increases diagonal damping after Cholesky failure; unresolved failure invalidates the solve rather than substituting an identity Hessian.Source: §D.6 · p. 24

E Sample Outputs

These sample disagreements illustrate how outputs can diverge through incomplete generations, task-semantic mistakes, or incorrect reasoning, while Schur Replay sometimes reaches the benchmark-expected answer. The examples span code, arithmetic, probability, science, and multiple-choice questions, including both qualified and benchmark-tested code-generation outcomes.

E.1 MMLU-Pro disagreements

  • Across three Llama MMLU-Pro categories, both baselines choose the same distractor while Schur Replay emits the keyed option; on Qwen, it distinguishes the good/bad percentages that the baselines reverse.For the Qwen question, the keyed answer is A (34%); both baselines emit C (64%). The three displayed Llama generations are complete archived answers with no reasoning text.Source: §E.1 · pp. 25–26

E.2 GSM8K disagreements

  • On the hotel problem, Four Over Six and Schur Replay account for late-checkout guests’ departure and recover keyed 98, while SOAR treats them as present and obtains 113; shoe baselines double results by changing units.Schur Replay preserves the pair-level unit through both arithmetic steps.Source: §E.2 · pp. 26–27

E.3 CMath disagreements

  • SOAR and Schur Replay finish reasoning and emit the gold-equivalent value, while Four Over Six enters a repetitive loop and exhausts the archived generation budget before producing an answer.In another comparison, all three generations set up the same remaining area and division; both baselines make a factor-of-ten error in the final quotient, while Schur Replay evaluates it consistently.Source: §E.3 · pp. 27–28

E.4 LiveCodeBench disagreements

  • For Qwen LiveCodeBench, Schur Replay tests transitive closure rather than direct indegree, with no per-question correctness verdict; on Llama, it passes while SOAR and Four Over Six fail.The Llama failures arise from mutating indegrees during a topological pass and treating revisited nodes as failure, respectively.Source: §E.4 · pp. 28–29
  • Schur Replay passes the LiveCodeBench reachability task by testing whether a vertex reaches every other vertex, matching the task definition.The observed disagreement contrasts this transitive-reachability check with the other implementation’s failure to match the task definition.Source: §E.4 · p. 30

E.6 HumanEval disagreements

  • On HumanEval, SOAR excludes booleans despite the canonical isinstance rule, Four Over Six emits that rule, and Schur Replay’s trace is incomplete; on same_chars, Schur Replay passes while both baselines fail multiplicity tests.The same_chars task requires comparing distinct-character sets rather than character multiplicities.Source: §E.6 · pp. 31–32

E.7 GPQA-Diamond disagreements

  • On GPQA-Diamond, Schur Replay identifies the keyed reading-frame defect and selects A, while SOAR selects D and Four Over Six does not finish; on the qubit question, it derives the zero Bloch vector and selects A while both baselines select D.The first answer follows from a 34-base-pair lox site causing a frameshift; the second places the mixed state at the Bloch-sphere center.Source: §E.7 · pp. 32–33

Related papers

Loading 2609.36654v1…