PaperlayerView exact revision on arXiv

Source-linked AI summary

QuantForge: Discovering Residual Decompositions for MXFP4 Post-Training Quantization

Qiulin Shang, Zhoutong Wu, Jie Hu, Kun Yuan

arXiv:2609.34680v2cs.LG

TL;DR

Strict MXFP4 PTQ requires coordinating coordinate design, discrete encoding, and path-level correction, while performance scores alone do not identify the next design change. QuantForge uses controlled experiments and verified code revisions to discover HiRes; across tasks, HiRes achieves the lowest seven-model Robust Fit and Qwen3-32B Fit-7.

  • Problem

    Strict MXFP4 PTQ must coordinate coordinate design, discrete encoding, and path-level correction, while performance scores alone cannot identify which design should change next.

    Source: §1 · pp. 1–2

  • Method

    QuantForge records competing explanations, selects distinguishing controls, compiles outcomes into checked code revisions, and remeasures revised programs.

    Source: §1 · p. 2, §4.1 · p. 4

  • Results

    0.09300 seven-model Robust Fit is the lowest, and HiRes also achieves the lowest Fit-7 on Qwen3-32B (0.027369).

    Source: §1 · p. 3, §6.1 · p. 8

  • Takeaways & Limitations

    Remeasured residuals allow the quantization decomposition to emerge across stages that adapt to the model state produced by preceding stages.

    Source: §1 · p. 2

    Realized latency, throughput, and memory on native MXFP4 kernels remain untested.

    Source: §7 · p. 10

Abstract

from arXiv · show

Four-bit post-training quantization can reduce the memory demands of large language models, but preserving accuracy under strict MXFP4 W4A4 requires coordinating several design choices. Coordinate transforms change block-encoding errors, which in turn affect the residuals propagated through the network. The useful algorithmic decomposition is therefore not fully known before search. LLM-driven program evolution offers a way to explore these choices, but performance scores alone do not explain which design should change next. We introduce QuantForge, a PTQ discovery system that records competing explanations, selects controls that distinguish them, and checks that successor code implements the resulting conclusions. This residual compilation guides program revisions while retaining useful programs even when their original explanations are rejected. Remeasuring the revised program reveals the next error to address. This process discovers HiRes, a fixed MXFP4 quantizer that shapes coordinates, refines legal code assignments, and recovers errors along attention and MLP paths. Each stage acts on residuals measured after the preceding stage has executed. Across seven tasks, HiRes achieves the lowest seven-model Robust Fit (0.09300) and the lowest quantized Fit-7 at 32B. In matched-budget comparisons of LLM-driven program evolution, each with 240 evaluator calls, QuantForge reaches a held-out transfer target in six of eight runs, compared with three each for textual memory and reflection memory, and one for score-only evolution, despite evaluating fewer new programs. These results show that QuantForge improves the discovery of transferable PTQ algorithms by turning controlled evidence into subsequent program changes. We have open-sourced the QuantForge search framework.

1 INTRODUCTION

Strict MXFP4 PTQ couples coordinate choice, legal code assignment, and propagated path errors, while performance scores alone cannot identify the next design change. QuantForge uses controlled evidence to guide successive revisions, discovering HiRes.

1 INTRODUCTION

  • MXFP4 coordinate choices affect shared-scale encoding and reconstruction, while quantized errors propagate through attention and gated MLP paths, making manual design difficult.A change to one step can alter what subsequent steps need to correct.Source: §1 · pp. 1–2
  • Program scores select candidates but cannot distinguish whether a transform helps by spreading outliers or improving shared-exponent selection, which imply different successors.Those explanations point respectively toward stronger coordinate transforms or revised block encoding.Source: §1 · p. 2
  • QuantForge retains useful programs by performance while using controlled experiments to turn conclusions into required, execution-probed code changes.Remeasuring revised programs reveals the next error to address.Source: §1 · p. 2
  • HiRes shapes coordinates, refines legal code assignments, and recovers path errors using residuals measured after preceding stages execute.Its structural recovery remeasures the MLP residual after attention correction.Source: §1 · p. 2
  • Across seven models and seven tasks, HiRes achieves the lowest Robust Fit (0.09300) and the lowest quantized Fit-7 on Qwen3-32B.The Robust Fit balances average and worst-model damage.Source: §1 · p. 3

2 RELATED WORK

Prior PTQ work addresses reconstruction, magnitude redistribution, coordinate transforms, and microscaling; QuantForge focuses on turning experimental resolution into subsequent quantizer changes.

2 RELATED WORK

  • Prior PTQ approaches use second-order reconstruction, magnitude redistribution, coordinate transforms, and microscaling methods to improve low-bit representations.HiRes combines error-supported geometric construction, legal discrete refinement, and recovery fitted to the realized network.Source: §2 · p. 3
  • HiRes is compared with related methods under one MXFP4 contract, combining geometric construction, legal discrete refinement, and recovery fitted to the realized network.Source: §2 · p. 3
  • Unlike program-evolution and scientific-agent work focused on performance, broader research cycles, or hypothesis testing, QuantForge records competing predictions and checks that evidence changes later code.Its focus is the step from experimental resolution to a subsequent quantizer.Source: §2 · p. 3

3 PRELIMINARIES: MXFP4 POST-TRAINING QUANTIZATION

The preliminaries define the strict MXFP4 W4A4 contract and the reconstruction and evaluation settings used by HiRes.

3 PRELIMINARIES: MXFP4 POST-TRAINING QUANTIZATION

  • Strict W4A4 uses legal MXFP4 operands and stored payloads, with contiguous blocks of 32 E2M1 values sharing one E8M0 power-of-two scale.Weights are quantized once, activations dynamically, and low-dimensional recovery runs outside GEMM without higher-precision weight matrices.Source: §3 · p. 3
  • GPTQ compensates for weight-rounding error across remaining input-feature columns using a stabilized factorization of H−1; HiRes applies reconstruction in transformed coordinates.The row-wise curvature is determined by H.Source: §3 · p. 3
  • Candidates and controls share the format, target modules, calibration data, evaluator, and resource limits, with numerical quality measured by simulated quantization in PyTorch.The complete execution contract appears in Appendix B.Source: §3 · p. 4

4 QuantForge: PROGRESSIVE RESIDUAL FACTORIZATION

QuantForge tracks useful programs separately from unresolved design questions, uses discriminating controls to resolve them, and compiles evidence into verified revisions before remeasuring residuals.

4 QuantForge: PROGRESSIVE RESIDUAL FACTORIZATION

  • QuantForge’s search state pairs an executable frontier of useful programs with a residual frontier of unresolved design questions.Controlled experiments resolve questions into requirements on later code, and remeasurement exposes the next question.Source: §4 · p. 4

4.1 FROM PROGRAM GAIN TO ALGORITHMIC RESIDUAL

  • A random 1% allocation distinguishes whether targeted-weight gains reflect weight identity or added capacity; that unresolved distinction is an algorithmic residual.A useful program may remain retained even when its original explanation is rejected.Source: §4.1 · p. 4

4.2 RESOLVING RESIDUALS WITH COUNTERFACTUALS

  • Before evaluation, proposals register competing explanations and predictions, then choose the least costly controls that distinguish every explanation pair.Controls share the evaluation contract and differ only in the specified intervention; an additional control is useful when it separates accounts the first leaves indistinguishable.Source: §4.2 · p. 4

4.3 COMPILING RESOLUTION INTO PROGRAM STRUCTURE

  • Experimental verdicts extend supported interventions, revise contradicted constructions, or leave inconclusive residuals open, with code changes checked by a targeted execution probe.Compilation preserves a measured gain while revising its interpretation into a requirement on the next relevant computation.Source: §4.3 · pp. 4–5

4.4 PROGRESSIVE RESIDUAL FACTORIZATION

  • Remeasurement after each revision identifies the next intervention, allowing useful operations and their relevant program states to determine the decomposition progressively.In MXFP4, the resulting hierarchy proceeds from geometric shaping to discrete realization and network-structured recovery.Source: §4.4 · p. 5

5 HiRes: HIERARCHICAL RESIDUAL RESOLUTION FOR MXFP4

HiRes develops a staged MXFP4 quantizer in which coordinate shaping, legal code realization, and structural recovery respond to residuals measured after preceding operations. Its stages refine the representation and correct remaining errors along attention and MLP paths.

5 HiRes: HIERARCHICAL RESIDUAL RESOLUTION FOR MXFP4

  • The discovered construction extends transform-based PTQ by addressing residuals measured after quantization.Each stage acts on the model state produced by its predecessor.Source: §5 · p. 5, §5.3 · p. 7

5.1 STAGE I: GEOMETRIC SHAPING

  • Reference coordinates balance activation statistics against inverse weight geometry, then interaction correction uses actual W4A4 encoding errors to refine them.Diagonal rescaling balances weight and activation quantization risk; accepted changes are selected using strict reconstruction loss, not risk ratios alone.Source: §5.1 · pp. 5–6

5.2 STAGE II: DISCRETE REALIZATION

  • GPTQ realizes legal Block32 W4 weights using transformed-input curvature, then activation refinement selects legal codes against the installed quantized weights.Dynamic activation refinement keeps the exponent fixed and changes at most one legal code.Source: §5.2 · p. 6

5.3 STAGE III: STRUCTURAL RECOVERY

  • After W4/A4 installation, HiRes fits attention corrections, then remeasures the MLP residual before fitting its correction.The MLP fit uses fresh gate and up outputs from the attention-corrected model; its shift and scales adjust the SiLU-gated product.Source: §5.3 · pp. 6–7

6 EXPERIMENTS

HiRes leads the reported cross-model MXFP4 comparisons, while QuantForge reaches a held-out transfer target more often than the matched-budget alternatives. The experiments also examine stage contributions and how controlled evidence is compiled into program changes.

6 EXPERIMENTS

  • The evaluation uses matched BF16 checkpoints, strict MXFP4 W4A4, and 128 ordered C4-train sequences of 2,048 tokens, with seven benchmark tasks.Discovery uses Qwen3-4B feedback and Llama-3.1-8B transfer validation; other listed models are evaluated after HiRes is frozen.Source: §6.1 · p. 7

6.1 HiRes: ROBUST MXFP4 QUANTIZATION

  • 0.09300 is HiRes’s lowest seven-model Robust Fit; at Qwen3-32B, its Fit-7 is 0.027369, ahead of WUSH and BRQ.On OLMo-2, the frozen algorithm also achieves the lowest Fit-7 (0.077301), ahead of BRQ and WUSH, without model-specific tuning.Source: Table 1 · p. 8, §6.1 · p. 8
  • Each successive stage improves three-task fitness on both 8B models; removing K, D, or one-code refinement increases Fit-3 on both.Fitting MLP recovery before attention correction also increases damage at the same deployment order and solver budget.Source: Table 2 · p. 8, §6.1 · p. 8

6.2 QuantForge: EVIDENCE-GUIDED ALGORITHM DISCOVERY

  • 6/8 QuantForge runs reach the held-out PPL ≤8.00 target, versus 3/8 each for ReflectMem and TextMem and 1/8 for Score-only.At 240 calls, median held-out PPL is 7.80 for QuantForge, versus 8.09 for ReflectMem, 9.20 for TextMem, and 10.03 for Score-only; QuantForge evaluates 141 new programs.Source: Figure 2 · p. 9, §6.2 · p. 9
  • Removing mandatory implementation and compliance checks raises median held-out PPL from 7.80 to 8.06, although it frees 30 calls for candidate evaluations.The comparison holds competing explanations, compiled controls, and memory fixed.Source: §6.2 · p. 9

7 CONCLUSION

The conclusion presents QuantForge’s controlled-experiment and residual-remeasurement process as a way to discover PTQ decompositions, producing the staged HiRes quantizer. The appendix describes search and transfer protocols that separate proposal feedback from held-out evaluation.

7 CONCLUSION

  • QuantForge uses controlled experiments and successive residual measurements to guide code changes, producing HiRes’s staged quantization procedure.The conclusion reports cross-model aggregate and 32B transfer results as support for the construction, while noting that native MXFP4-kernel latency, throughput, and memory remain untested.Source: §7 · p. 10

A QuantForge: SEARCH PROCEDURE

  • The search retains successful GPTQ descendants while allowing revisions to transforms, discrete reconstruction, and residual recovery under a shared strict-MXFP4 protocol.The final three-stage quantizer was not part of the initial round-to-nearest program.Source: §A.1 · p. 14

A.1 DISCOVERY PROTOCOLS

  • Qwen3-4B guides proposals, while Llama-3.1-8B independently validates transfer; held-out results do not return to search or select descendants.The algorithm-development search uses Fit-3, whereas the quantizer benchmark uses a seven-task panel.Source: §A.1 · p. 14

A.2 CONTROLLED EXPERIMENTS AND PROGRAM UPDATES

QuantForge uses controlled experiments to turn competing explanations into checked program changes, then remeasures revised programs to expose the next residual. HiRes applies coordinate shaping, code refinement, and structural recovery conditionally to the states produced by preceding stages.

A.2 CONTROLLED EXPERIMENTS AND PROGRAM UPDATES

  • Competing explanations are tested with controls chosen to distinguish them; observed outcomes determine required code responses, while useful programs can survive rejected explanations.Descendants are checked against both their declared response and an execution probe scoped to the targeted residual.Source: §A.2 · p. 15

A.3 SEARCH CYCLE

  • The search keeps executable programs and unresolved residuals on separate frontiers, then remeasures each revised program to open its next intervention space.History records completed experiments, Pareto donor programs, duplicate proposals, and stagnation information.Source: §A.3 · p. 15

B.1 QUANTIZATION SETUP

  • HiRes stages are measured conditionally: shaping uses BF16 weights and calibration inputs, discrete realization uses installed coordinates, and structural recovery uses installed W4/A4 operators.Attention recovery precedes collection of new calibration outputs for the gated-MLP fit.Source: B HiRes: ALGORITHM DETAILS · p. 16, §B.1 · p. 16, §B.4 · p. 18

B.2 GEOMETRIC SHAPING

  • Geometric shaping estimates bounded coordinate interactions from anchor-quantization errors, then balances activation and weight quantization risks using strict-encoding errors.Each block tests the proposed balance, a half step, and identity, accepting only a trial whose reconstruction loss does not exceed identity.Source: §B.2 · pp. 17–18

B.3 DISCRETE REALIZATION

  • Discrete realization scores legal code changes by their exact change in block-local output-error objective and accepts only a negative-score change.At most one element code changes per physical block; zero-score ties retain nearest rounding.Source: §B.3 · p. 18

B.4 STRUCTURAL RECOVERY

  • Structural recovery fits residuals of the installed W4/A4 model along attention and then the gated MLP, remeasuring the state after attention correction.The MLP fit uses new calibration outputs and aggregates cellwise weighted-ridge estimates by coordinatewise medians.Source: §B.4 · pp. 18–19

C HiRes: ADDITIONAL RESULTS

  • The additional-results section examines the frozen algorithm’s accuracy, evaluation sensitivity, stage contributions, remeasurement, and computational and storage costs.Its listed comparisons include geometry and code-refinement ablations and attention/MLP recovery paths.Source: C HiRes: ADDITIONAL RESULTS · p. 19

C.1 COMPLETE QUANTIZATION RESULTS

Across the supplied comparisons, HiRes achieves the lowest quantized Fit-7 on Qwen3-32B and leads the reported quantized outcomes on a new OLMo-2-13B model family. Its seven-model Robust Fit is also lower than GPTQ and FOCUS.

Seven-model comparison

  • 0.09300 Robust Fit for HiRes is below FOCUS (0.14718) and GPTQ (0.18624) across the seven-model comparison.FOCUS improves on GPTQ’s Fit-7 across all seven models.Source: §C.1 · p. 20

Model-level endpoints

  • On Llama-3-8B, HiRes lowers WikiText-2/C4 perplexity to 7.111/11.026 from WUSH’s 7.133/11.117; on Mistral-7B, it lowers them to 5.664/8.810 from 5.689/8.834.The comparisons use the shared seven-task MXFP4 operand and encoding contract.Source: §C.1 · pp. 19–20, Table 5 · p. 21, Table 7 · p. 22

OLMo-2-13B transfer

  • HiRes achieves the lowest quantized Fit-7 and C4 perplexity, and the highest accuracy on ARC-Challenge, ARC-Easy, HellaSwag, and PIQA on OLMo-2-1124-13B Base.This evaluation uses a frozen algorithm on a new architecture and model family; its results do not enter algorithm design or hyperparameter selection.Source: Table 8 · p. 22, §C.1 · p. 22

Qwen3-32B scale transfer

  • 0.027369 quantized Fit-7 for HiRes is the lowest on Qwen3-32B, below WUSH (0.036489) and BRQ (0.042486).HiRes leads among quantized methods on five of seven tasks, including WikiText-2/C4 perplexity (7.926348/13.044736), ARC-E, HellaSwag, and PIQA; its Fit-7 is 25.0% lower than WUSH’s.Source: §C.1 · p. 22, Table 9 · p. 23

C.2 CALIBRATION AND EVALUATION SENSITIVITY

Calibration-seed tests compare HiRes with WUSH on Qwen3-8B and Llama-3-8B, while alternative aggregates summarize the frozen seven-task evaluations. HiRes improves Fit-7 across all tested seeds and lowers MeanDamage on all seven models.

Evaluation sensitivity

  • The seed analysis uses calibration seeds 0, 3, and 5 on Qwen3-8B and Meta-Llama-3-8B, with each configuration collecting its own recovery statistics and rerunning downstream evaluation.The comparisons prespecified minimum differences of 0.002 for Fit-7 and 0.003 for Fit-3.Source: §C.2 · p. 23

Calibration seeds

  • HiRes improves on WUSH in 3/3 calibration seeds on both Qwen3-8B and Llama-3-8B.On Qwen3-8B, mean Fit-7 is 0.032692 ± 0.000298 for HiRes versus 0.049776 ± 0.002902 for WUSH; the mean paired difference is −0.017085 (95% CI [−0.023948, −0.010221]). On Llama-3-8B, the means are 0.116158 ± 0.004006 and 0.121639 ± 0.003560, respectively.Source: §C.2 · p. 23, Table 11 · p. 24

Alternative summaries

  • HiRes lowers MeanDamage on all seven models; on Qwen3-32B, MeanDamage is 0.013294 versus 0.022416 for WUSH.On Qwen3-32B, worst-task damage is 0.04144 for HiRes versus 0.05056 for WUSH; on Qwen3-8B, WT2, C4, and Fit-7 improve while Acc5 is slightly lower.Source: §C.2 · p. 24, Table 12 · p. 25

C.3 STAGE AND GEOMETRY ABLATIONS

Stage and geometry ablations show that successive HiRes components reduce quantization damage, while activation-code refinement improves Fit-7 across paired model–seed comparisons. The results also indicate that the refinement balances sensitivity and realized-weight mismatch rather than optimizing sensitivity alone.

C.3 STAGE AND GEOMETRY ABLATIONS

  • Each successive stage improves three-task fitness on both 8B models, with geometry providing the largest initial gain and later stages reducing remaining damage.The stage comparison adds the stages in deployment order; each row retains preceding stages.Source: §C.3 · p. 25, Table 13 · p. 25
  • 0.00787 on Qwen3-8B and 0.00699 on Llama-3-8B are the fitness gains from local interaction K.Removing diagonal balance D degrades fitness by 0.01067 on Qwen3-8B and 0.01091 on Llama-3-8B; removing both corrections is worse than removing either alone.Source: §C.3 · p. 25, Table 14 · p. 25
  • The full geometry method is best in all 6/6 model–seed pairs, with Fit-7 reductions over removing K of 0.005259–0.006581 on Qwen and 0.005361–0.006121 on Llama.Removing both geometric corrections is worse than removing either alone, and each correction lowers Fit-7 when used without the other.Source: §C.3 · p. 26, Table 15 · pp. 26–27
  • Adding refinement and recovery to WUSH lowers mean Fit-7 by 0.010784 on Qwen and 0.002878 on Llama; HiRes geometry lowers it by a further 0.006301 and 0.002603.Both steps improve Fit-7 in all three paired seeds on each model; WT2 and C4 perplexity decrease at both steps, while mean accuracy changes slightly in opposite directions across the models.Source: §C.3 · p. 27, Table 16 · p. 28, Table 17 · p. 28

C.4 ACTIVATION-CODE REFINEMENT

  • The full mismatch-aware activation-code rule has lower Fit-7 than nearest rounding and sensitivity-only refinement in all 6/6 model–seed pairs.Its mean gains over nearest rounding and sensitivity-only are 0.006632 and 0.010177 on Qwen, and 0.005482 and 0.007635 on Llama.Source: §C.4 · p. 28, Table 19 · p. 29
  • Sensitivity-only minimizes its own objective at 0.912/0.921 on Qwen/Llama but raises mismatch to 1.043/1.036; the full rule lowers mismatch to 0.918/0.926 and obtains the best endpoints.The full rule accepts sensitivity terms of 0.947/0.953; local errors are measured on layers using refinement, with measurement segments independent of rule-fitting data.Source: §C.4 · p. 29, Table 20 · p. 30
  • 0.61 codes per Block32 on Qwen and 0.68 on Llama are changed on average, affecting 1.91% and 2.12% of A4 elements, respectively.The observed edit rate is below the 1/32 = 3.125% element-level ceiling.Source: §C.4 · p. 30, Table 21 · p. 30

C.5 RECOVERY PATHS AND REMEASUREMENT

Recovery improves when both attention and MLP paths are used and statistics are measured after the preceding quantized operations. The paper also reports HiRes latency and storage overhead, matched-budget transfer results for QuantForge, and evidence that verdict-guided updates improve successor programs.

C.5 RECOVERY PATHS AND REMEASUREMENT

  • The full attention-and-MLP recovery pair is best on both models, improving over the stronger single path by 0.00652 on Qwen3-8B and 0.00543 on Llama-3-8B.Across seeds, damage increases from both paths to MLP only, attention only, and neither; the single-path effects are much closer on Llama than Qwen.Source: §C.5 · p. 31, Table 24 · p. 32
  • Remeasuring MLP statistics after attention improves WT2, C4, and ARC-C in all 6/6 model–seed pairs, reducing Fit-3 by 0.006444–0.009132 on Qwen and 0.009304–0.012954 on Llama.Both conditions retain attention-to-MLP deployment; the control changes whether MLP statistics are measured before or after attention correction.Source: §C.5 · p. 33, Table 25 · p. 33
  • Fitting recovery after actual MXFP4 encoding reduces deployed MSE by 9.52% on Qwen and 8.49% on Llama, versus 0.019% and 0.012% for pre-encoding fitting.Post-encoding fitting also lowers Fit-3 by 0.016127–0.016479 on Qwen and 0.008312–0.009614 on Llama across calibration seeds.Source: §C.5 · pp. 33–34, Table 26 · p. 33, Table 27 · p. 34

C.6 COMPUTATIONAL AND STORAGE COSTS

  • 1.87%–2.61% is the reported increase in persistent payload over BF16 parameters for the measured HiRes models.The accounting includes retained auxiliary tensors but excludes activations, temporary workspaces, and allocator reserves.Source: §C.6 · p. 34, Table 28 · p. 34
  • 3.97%–4.49% is HiRes's added full-forward latency over BF16 for batch size 1 and 2,048 input tokens with caching disabled.The measurement includes transforms, dynamic A4, candidate scoring and selection, recovery, GEMM, attention, and final logits.Source: §C.6 · p. 34, Table 29 · p. 35

D.1 MATCHED-BUDGET COMPARISONS

  • 6/8 QuantForge runs reach the PPL ≤8.00 held-out transfer gate, compared with 3/8 ReflectMem runs after 240 search calls.QuantForge's median held-out PPL is 7.80 versus ReflectMem's 8.09; QuantForge also has lower final perplexity than Score-only in eight paired seeds and TextMem in seven.Source: §D.1 · p. 36, Table 30 · p. 36, Table 32 · p. 37
  • Four QuantForge runs contain the full ordered hierarchy of geometry, discrete realization, and structural recovery, compared with none in either baseline.The audit classifies final champions by executable behavior, not textual mentions.Source: §D.1 · p. 37, Table 34 · p. 37

D.2 VERDICT-GUIDED PROGRAM UPDATES

  • QuantForge has lower held-out perplexity in 8/8 paired seeds than the arm without residual compilation, with mean paired difference −0.275 and 95% CI [−0.368, −0.182].The comparison also reports 2.67× more verified residual closures and fewer invalid successors, 0.13 versus 0.38.Source: §D.2 · p. 38, Table 36 · p. 38, Table 37 · p. 38

D.3 CONTROL SELECTION AND IMPLEMENTATION CHECKS

Under the 240-call budget, minimal discriminating control selection produces the strongest held-out result among the compared policies. Implementation checking also involves a trade-off: execution probes yield the lowest median held-out perplexity, while stricter layer-output checks reject some genuine implementations.

D.3 CONTROL SELECTION AND IMPLEMENTATION CHECKS

  • Minimal discriminating control selection improves held-out PPL over both alternatives in 8/8 seeds and gives the best budget-limited result among the three policies.At 69 control calls, random selection yields 4.6 explicit verdicts versus 11.3 for minimal selection; running all controls yields 12.1 verdicts but leaves only 92 new candidates and median held-out PPL worsens to 8.02.Source: §D.3 · p. 40, Table 41 · p. 40, Table 42 · p. 40
  • At the same 69 control calls, minimal selection costs 6.1 calls per explicit verdict versus 15.0 for random selection.Implementation rates are similar across the policies, at 0.91–0.92, separating control discrimination from implementation success.Source: §D.3 · p. 40
  • The execution-probe configuration has the lowest median held-out PPL among the three implementation checkers.The comparison uses 40 injected cases: 20 genuine implementations and 20 apparent edits that leave the target computation unchanged.Source: §D.3 · p. 40, Table 43 · p. 40
  • Replacing source diffs with execution probes lowers median held-out PPL from 7.96 to 7.80; source diffs accept 0.45 of apparent implementations.Stricter layer-output comparison accepts none of 20 apparent edits but rejects 4/20 genuine implementations and leaves fewer new candidates, with median PPL 7.91.Source: §D.3 · p. 41

Related papers

Loading 2609.34680v2…