PaperlayerView exact revision on arXiv

Source-linked AI summary

ThinkBit: Search Codewords, Recalibrate Scales, Select by Reasoning

Rhea Sanjay Sukthanker, Pashmina Cameron

arXiv:2610.34886v1cs.AIcs.CL

TL;DR

ThinkBit addresses whether scalar two-bit quantization can preserve difficult reasoning, which likelihood and multiple-choice metrics may not reveal. It combines layerwise bit allocation, codeword search, scale recalibration, and QAT, then selects between checkpoints by reasoning tests; at matched size, Rec-h2 scores 46.7 on MATH-500 versus 30.0 for EfficientQAT.

  • Problem

    Perplexity and multiple-choice accuracy can barely distinguish two-bit configurations whose sampled reasoning accuracy differs by more than ten points.

    Source: §1 · p. 1

  • Method

    ThinkBit allocates layer bitwidths by sensitivity, searches codewords block by block, recalibrates scales with codewords frozen, applies QAT, and selects a checkpoint using a paired reasoning test.

    Source: §1 · p. 1, §9 · p. 18

  • Results

    46.7 on MATH-500 for Rec-h2 versus 30.0 for same-data-and-token EfficientQAT at matched size; the selected 0.52 GB checkpoint is smaller than every published 2-bit GGUF and more accurate on MATH-500.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The evidence supports selecting two-bit models on reasoning tasks, where most of the degradations are seen.

    Source: §1 · p. 3

    Files of four or more bits per parameter remain stronger, and some two-bit reasoning traces still do not finish.

    Source: §9 · p. 19

Abstract

from arXiv · show

Reasoning language models generate thousands of tokens per answer, and generation speed is limited by memory bandwidth, so storing weights in two bits instead of bfloat16 can potentially make inference 8$\times$ cheaper. Two-bit methods are usually judged by perplexity and multiple-choice question-answering accuracy, both of which say little about whether a model can still reason through more difficult problems. We present ThinkBit, a strictly scalar two-bit framework built to preserve reasoning, in which every weight is an integer codeword times a bfloat16 scale shared by a group of weights. We first use a sensitivity metric to assign each layer a ternary, 2-bit, or 3-bit tier to allocate bitwidths optimally among layers. This is followed by a layerwise Gumbel-softmax-based scalar quantization algorithm that searches for optimal codewords block by block. Since a layerwise scheme does not capture how quantized layers interact with each other, we recalibrate scales, with the codewords frozen, using a knowledge-distillation-based loss against the teacher model. We then perform a short quantization-aware training (QAT) stage that lets the model mimic the base model's distribution by reassigning codewords. Finally, a paired reasoning test on training problems selects the final checkpoint between the recalibration stage and the QAT stage. On Qwen3-1.7B at matched size, our recalibrated checkpoint scores 46.7 on MATH-500 against 30.0 for EfficientQAT trained on the same data, and 73.0 after QAT on reasoning traces; at 0.52 GB it is smaller than every published 2-bit GGUF of the model yet more accurate on MATH-500. We show that this framework scales and up to 14B, a 3.74 GB model scores 94.8 on MATH-500, against 97.5 for the 29.5 GB bfloat16 model.

1 Introduction

ThinkBit asks whether scalar two-bit quantization can preserve difficult reasoning, a question obscured by likelihood and multiple-choice metrics. It combines layerwise allocation and codeword search with scale recalibration, QAT, and reasoning-based checkpoint selection.

1 Introduction

  • Perplexity and multiple-choice accuracy barely distinguish configurations whose sampled reasoning accuracy differs by more than ten points, motivating reasoning-task evaluation.Source: §1 · p. 1
  • ThinkBit uses five stages: sensitivity-based layer precision allocation, blockwise codeword search, frozen-codeword scale recalibration, QAT codeword reassignment, and paired reasoning-test checkpoint selection.Source: §1 · pp. 1–2
  • Published 2-bit labels can mask storage: Unsloth Qwen3-1.7B releases use 2.82-3.62 bits per parameter, so ThinkBit compares whole-model bytes.Source: §1 · p. 2
  • Removing codeword search, scale recalibration, both, or budget allocation leaves the 7-task QA average at 53.7-54.8 but lowers MATH-500 accuracy from 43.0 after QAT to 28.4–37.0.Source: §1 · p. 3
  • On Qwen3-1.7B, ThinkBit reaches 47.5 on sampled MATH-500 before QAT and 43.0 after, versus 18.5–24.1 and 32.6–37.8 from tuned GPTQ initializations.Source: §1 · p. 3
  • At matched size, Rec-h2 scores 46.7 on MATH-500 versus 30.0 for EfficientQAT trained on the same data and tokens.Source: §1 · p. 3

2 Methodology

ThinkBit converts linear-layer weights to integer codewords with shared bf16 scales, then allocates precision, searches codewords, recalibrates scales and norms, and uses QAT plus a reasoning test to select a checkpoint. Its stages move from layerwise calibration toward whole-model matching to the bf16 model.

2 Methodology

  • ThinkBit converts bf16 models into small-integer linear-layer weights, using layerwise calibration for allocation and search, whole-model recovery for recalibration and QAT, and evaluation for selection.The five stages are Allocate, Search, Recalibrate, QAT, and Selection.Source: §2 · p. 3

2.1 Weight format and bit accounting

  • Each weight is an integer codeword times a bf16 scale shared across 128 weights; decoder cost includes codeword bits plus each weight’s share of the 16-bit scale.The framework uses ternary, 2-bit, or 3-bit codeword grids, and distinguishes decoder bits per weight from total bits per parameter.Source: §2.1 · p. 3

2.2 Allocate: How do we allocate precision appropriately?

  • Layer precision is allocated by ranking sensitivity times tier rounding noise and minimizing expected loss under a decoder-bit budget, with Lagrangian relaxation solving the resulting knapsack problem.The ranking drops the per-layer weight-variance factor and treats layers as having equal weight variance.Source: §2.2 · p. 4
  • The allocation rule does not select 2-bit tiers on its own, so the Qwen3-1.7B run uses 75.3% ternary, 6.0% 2-bit, and 18.7% 3-bit weights.Query and key projections are held at ≥2 bits in that run; the text notes this constraint is unnecessary on this model.Source: §2.2 · p. 4

2.3 Search: How to best inititalize codewords with PTQ?

  • Codeword search optimizes distributions blockwise, using Gumbel-softmax for ternary and 3-bit layers and ReinMax for 2-bit layers, then fits quantized block outputs to the bf16 block on identical inputs.Query and key projections refine their GPTQ codewords separately, while ternary layers begin refinement from uninformative logits.Source: §2.3 · pp. 5–6, Table 1 · p. 6

2.4 Recalibrate: Is tuning scales and norms enough?

  • Recalibration trains only bounded scale and RMSNorm-gain offsets against whole-model forward KL, keeping codewords frozen and the stored format and size unchanged.Training retains the step with the lowest held-out KL.Source: §2.4 · p. 7

2.5 QAT: Does moving codewords help?

  • QAT trains codeword latents with a straight-through estimator and adds hidden-state distillation to forward KL, while scales and gains continue training.Latents start partway toward bf16 weights, leaving codewords unchanged initially while allowing reassignment during training.Source: §2.5 · p. 7

2.6 Selection: Which checkpoint reasons better, recalibrated or QAT?

  • A paired sign-flip test compares recalibrated and QAT checkpoints on held-out level-3–5 MATH training problems, retaining the higher-accuracy candidate when p ≤ 0.05 and otherwise preferring the smaller checkpoint.Each candidate samples four answers per problem, and scoring requires both a closed reasoning trace and a correct final answer.Source: §2.6 · pp. 7–8

3 Experimental setup

Experiments use Qwen3-1.7B, 8B, and 14B with the same ThinkBit hyperparameters, and compare checkpoints across stages and model sizes. Selected checkpoints use longer Stage 4 training on a mixture containing complete reasoning traces.

Models

  • The study evaluates Qwen3-1.7B, Qwen3-8B, and Qwen3-14B with the same ThinkBit hyperparameters.All models are in thinking mode; the 8B and 14B models have untied heads.Source: §3 · p. 8

Data

  • The base mixture combines model-generated replies, model continuations of web and Wikipedia text, and raw text from multiple sources.Replies are capped at 4,096 tokens, so training does not see a complete long reasoning trace.Source: §3 · p. 8
  • Selected checkpoints receive longer Stage 4 training on a mixture with complete reasoning traces, which make up about 60% of the data.The mixture also includes long-context books and papers; all sources undergo the base mixture’s benchmark decontamination.Source: §3 · p. 8

Checkpoint names

  • Checkpoint names encode the stages completed, head-and-embedding precision, and whether reasoning-mixture training was used.“Selected” identifies the checkpoint retained by Stage 5.Source: §3 · p. 8

Evaluation

  • Evaluation reports a 7-task mean and generation benchmarks spanning math, code, instruction following, and tool use.Sampled MATH-500 is the primary reasoning metric: four answers per problem, with credit only for a closed </think> trace and correct answer.Source: §3 · p. 8

Checkpoint names

  • At 1.7B, Rec-h2 uses a second recalibration pass that quantizes the head and embedding to 2 bits; Rec-h2 (1 pass) uses one.Table 3 summarizes the data used by each stage, with an asterisk marking longer QAT on the reasoning mixture for selected checkpoints.Source: Table 2 · p. 9, Table 3 · p. 9

4 Where does the quality come from?

On Qwen3-1.7B, scale recalibration with codewords frozen improves likelihood and sampled math accuracy, while subsequent QAT improves some metrics but not reasoning uniformly. Reasoning scores also reflect whether traces finish, not only whether their answers are correct.

Recalibration and QAT

  • Recalibration improves output-distribution matching without improving representation fidelity: next-token KL falls from 1.30 to 0.57 while hidden-state distance rises from 0.21 to 0.28.The output objective weights errors by activations and allows scales in one layer to offset rounding errors in others.Source: §4 · p. 9
  • Long QAT reassigns 6.4% of codewords, raises the 7-task mean from 51.1 to 54.6, but lowers sampled MATH-500 from 47.5 to 43.0.WikiText-2 perplexity improves partly through Wikipedia in the training mix, while GSM8K, HumanEval, and MATH-500 do not improve.Source: §4 · pp. 9–10
  • Recalibration lowers held-out KL to bf16 from 0.641 to 0.282 with codewords frozen, while long QAT lowers KL from 0.311 to 0.199 on its own following text slice.The QAT starting KL is not directly comparable with the end of recalibration because the scored text differs.Source: Figure 2 · p. 10
  • Sampled MATH-500 rises from 14.8 to 47.5 after recalibration, then falls to 43.0 after QAT; WikiText-2 perplexity falls from 46.91 to 19.46 and then 16.54.The table compares Qwen3-1.7B checkpoints with bf16 head and embedding; recalibration raises the 7-task mean from 43.8 to 51.5.Source: §4 · p. 9, Table 4 · p. 10

Reasoning outcomes

  • On broader evaluation, reading comprehension and commonsense retain at least 80% of bf16 scores, while code and non-chat math do not recover.This section also separates trace completion from accuracy on completed traces because strict scoring requires a closed </think> tag.Source: §4 · p. 10
  • At 1.7B, two-bit checkpoints close 55.4–88.0% of reasoning traces, versus 99.2% for bf16 and 47–56% for 2-bit GGUFs.Codeword search primarily improves accuracy on closed traces, while recalibration and QAT mainly affect closing rates.Source: §4 · p. 10, Table 5 · p. 11

5 Scaling across model sizes

ThinkBit is applied unchanged across Qwen3 sizes, with reasoning-mixture QAT improving sampled math and AIME performance relative to recalibrated checkpoints. At 14B, the two checkpoints are not significantly different on the training-problem selector, while comparisons with released low-bit models vary by task and size.

Reasoning-mixture QAT

  • Against Rec-h2, reasoning-mixture QAT raises sampled MATH-500 from 46.7 to 73.0 at 1.7B, 86.6 to 92.2 at 8B, and 92.7 to 94.8 at 14B.At 1.7B, the gain is mainly accuracy on finished traces; the 7-task mean moves by at most 2.7 points, and HumanEval falls at 8B and 14B.Source: §5 · p. 11

Published low-bit comparisons

  • At every size, selected checkpoints are smaller than the smallest published 2-bit GGUF and more accurate on every task reported for that comparison.On sampled MATH-500, QAT+R-h2 scores 73.0 versus 1.0 at 1.7B, 92.2 versus 35.2 at 8B, and 94.8 versus 85.2 at 14B.Source: §5 · p. 11

Scaling across model sizes

  • At 1.7B, 8B, and 14B, QAT+R-h2 scores 73.0, 92.2, and 94.8 on sampled MATH-500, versus bf16 scores of 91.2, 96.9, and 97.5.The selected checkpoints occupy 0.52, 2.08, and 3.74 GB, compared with 3.44, 16.4, and 29.5 GB for bf16.Source: §5 · p. 11, Table 6 · p. 12

Two-bit QAT comparisons

  • Rec-h2 exceeds EfficientQAT and BitDistiller on sampled MATH-500 at each size, although those methods match or exceed Rec-h2 on the 7-task mean.At 1.7B, Rec-h2 scores 46.7 versus 34.2 and 27.8; quantizing the competitors’ head and embedding to 2 bits yields 30.0 and 25.3.Source: §5 · p. 14

Scaling and checkpoint comparisons

  • At 14B, Rec-h2 scores 92.7 on sampled MATH-500 versus 90.5 for QAT-h2, while GSQ-started recalibration scores 92.8 versus 89.7 for layer-wise-start codewords.On the training-problem selector, QAT+R-h2 and Rec+R-h2 differ nonsignificantly at 14B (92.7 versus 91.9, p=0.16).Source: §5 · pp. 11, 14

6 Model size and inference

The paper verifies packed checkpoint sizes and reports weight-memory and decoding measurements for serving. At matched size, reasoning performance varies across quantized checkpoints and competing low-bit files.

Model size and inference

  • The Qwen3-1.7B QAT and recalibrated checkpoints each measure 0.520 GB with a 2-bit head and embedding, and decode bit-identically after repacking.On one H100, 8B and 14B checkpoints use 5.68 and 9.31 GiB of weight memory and decode 212 and 150 tokens/s at batch 1.Source: §6 · p. 14

Equal bytes, same harness

  • At 0.52 GB, QAT+R-h2 scores 73.0 on sampled MATH-500, versus 41.9 for Q2_K; the latter is 0.78 GB.The 0.52 GB QAT+R-h2 checkpoint also scores 68.6 on greedy MATH-500, versus 27.8 for Q2_K.Source: §6 · p. 14, Table 7 · p. 15
  • Ternary Bonsai leads non-reasoning columns, while QAT+R-h2 leads sampled MATH-500 among files up to 4 bits per parameter at 0.52 GB.The table compares files in the same harness and groups rows by bits per parameter.Source: §6 · p. 14, Table 7 · p. 15

7 Controls behind each stage

The controls show that conventional benchmarks can stay nearly unchanged while reasoning accuracy falls, and that checkpoint selection should use a paired reasoning test. The section also compares ThinkBit with QAT baselines and reports the selector’s decisions.

7 Controls behind each stage

  • The controls remove one stage or design choice under frozen hyperparameters; recalibration plus QAT uses 805M tokens, while no-recalibration QAT arms use 719M.Held-out KL is still falling at the last step.Source: §7 · p. 16
  • Removing codeword search, scale recalibration, both, or mixed bit placement leaves the 7-task mean at 53.7–54.8 but lowers sampled MATH-500 from 43.0 to 28.4–37.0.Table 8 reports a 6.0–14.6-point MATH-500 cost for removing any stage; paired contrasts are reported in Table 9.Source: Table 8 · p. 16, §7 · p. 16, Table 9 · p. 17
  • QAT raises the 7-task mean from 51.5 to 54.6 but lowers sampled MATH-500 from 47.5 to 43.0; HumanEval does not follow the same ordering.The authors characterize the result as concerning long-form reasoning.Source: §7 · p. 17
  • At about the same size and token budget, EfficientQAT scores 28.8 on sampled MATH-500, versus 43.0 for QAT-hbf16 and 47.5 for Rec-hbf16.The passage reports p<10^-4 for the comparisons and an official EfficientQAT score of 34.2 on the same data.Source: §7 · p. 17
  • The selector compares recalibrated and QAT checkpoints on four completions for each of 313 disjoint MATH training problems, using a paired sign-flip test.It keeps the higher mean when p ≤ 0.05 and otherwise the smaller checkpoint; MATH-500 agrees with every reported decision.Source: §7 · p. 17

8 Related work

Related work spans post-training quantization, very-low-bit formats, quantization-aware training, sensitivity-based allocation, and evaluation of quantized models. ThinkBit positions its scalar format and reasoning-based evaluation within these lines of work.

8 Related work

  • Earlier LLM quantizers use layerwise distillation, outlier handling, second-order rounding, and learned rounding or clipping; most approach lossless performance at three to four bits but degrade sharply at two.Examples include ZeroQuant [81], GPTQ, AWQ, and AdaRound [52].Source: §8 · p. 17
  • GSQ jointly learns scalar-grid assignments and group scales through a Gumbel-softmax relaxation; ThinkBit builds on GSQ..Source: §8 · p. 18
  • Strong two-bit methods often leave scalar quantization for lattice, trellis, or additive vector codebooks, while ThinkBit retains one bf16 scale per 128-weight group.The paper notes that rotations are orthogonal to its stages and could be combined with them.Source: §8 · p. 18
  • Prior QAT and distillation approaches vary in training strategy; ThinkBit fits scales and gains to the bf16 output distribution with codewords frozen, then distills using forward KL.Reverse-KL and on-policy distillation are alternatives the paper does not use.Source: §8 · p. 18
  • ThinkBit uses a Kronecker-factored Hessian sensitivity score to assign layers ternary, 2-bit, or 3-bit tiers under a bit budget.The score is based on activation and gradient traces, following KronQ [34] and K-FAC [49].Source: §8 · p. 18
  • Prior evaluations note that perplexity can miss losses on knowledge-intensive or long-context tasks; this paper uses paired tests to compare competing two-bit configurations on reasoning problems.The paper also situates its evaluation among work on reasoning-specific quantization and calibration.Source: §8 · p. 18

9 Conclusion

The conclusion presents ThinkBit as a strictly scalar two-bit framework and reports stronger reasoning than comparison quantizers within the tested scope. The appendix gives its end-to-end procedure and evaluation settings.

9 Conclusion

  • ThinkBit allocates layer precisions, searches codewords block by block, recalibrates scales with codewords frozen, applies QAT, and selects a checkpoint by paired reasoning tests.The conclusion reports that likelihood-based metrics can hide reasoning losses and that the selector uses training problems without touching the test set.Source: §9 · p. 18
  • The reported scope remains bounded: files of four or more bits per parameter are stronger, and some two-bit reasoning traces do not finish.The conclusion also reports that at 14B the model comes within a few points of bf16 on MATH-500.Source: §9 · p. 19

A Full algorithmic procedure and notation

  • Algorithm 2 maps the bf16 model, calibration and training data, reasoning problems, and bit budget through sensitivity scoring, allocation, search, recalibration, QAT, and selection.Its selector uses the paired sign-flip result to choose the higher-accuracy candidate or, absent significance, the smaller one.Source: A Full algorithmic procedure and… · p. 25
  • Codeword search proceeds block by block; recalibration freezes codewords, while QAT trains codewords through the straight-through estimator along with scales and gains.The procedure uses held-out KL to retain the best recalibration and QAT states.Source: A Full algorithmic procedure and… · p. 25

B Evaluation settings

  • Checkpoints, including GGUF baselines, are dequantized to dense bf16 and evaluated with the same lm-eval and vLLM stack and settings.Thinking-mode results use the models’ chat template; other suites do not.Source: B Evaluation settings · p. 26, Table 11 · p. 27

C Per-task likelihood scores

Across the seven tasks, each stage raises performance, while quantizing blocks produces a depth-dependent likelihood pattern. On Qwen3-1.7B, the query–key bitwidth constraint is unnecessary under the tested setup.

C Per-task likelihood scores

  • Chance-normalized retention across the seven tasks rises from 47.6% after GSQ to 73.1% after recalibration and 84.2% after QAT.Each stage raises all seven tasks; after QAT, the largest remaining gaps to bf16 are 7.8 points on HellaSwag and 5.6 on MMLU.Source: Table 12 · p. 27, C Per-task likelihood scores · p. 27

D Codeword search across depth and the query–key rule

  • On Qwen3-1.7B, WikiText-2 perplexity stays near bf16 while GSQ quantizes the first half of the blocks, then rises over the final third.Figure 5 tracks perplexity relative to the model’s own bf16 as blocks are quantized, before recalibration.Source: D Codeword search across depth and the… · p. 27, Figure 5 · p. 28
  • On Qwen3-1.7B, forcing query and key projections to at least 2 bits is unnecessary: removing the rule leaves subsequent measurements close and reasoning differences nonsignificant.Without the rule, the allocator assigns 15 of 28 query and 3 of 28 key projections to ternary at the same 2.0 decoder bits per weight; the MATH-500 difference is +1.8, p=0.12.Source: D Codeword search across depth and the… · p. 28, Table 13 · p. 28

E Further ablations

The ablations show that likelihood-based equivalence can miss substantial differences in reasoning, while stronger codeword-search starts outperform stronger GPTQ starts after the same later stages. Paired tests also compare checkpoints across sizes and report the selector’s choices.

E Further ablations

  • Moving codewords toward their bf16 values and briefly recalibrating scales performs worse than leaving them unchanged.These short controls do not isolate QAT’s codeword choices from its longer optimization.Source: E Further ablations · p. 28
  • Scale recalibration alone matches scales plus RMSNorm gains, while gains alone recover less; reverting reassignments has task-dependent effects.Reverting MLP reassignments lowers MMLU but raises HumanEval, while reverting attention reassignments lowers GSM8K.Source: E Further ablations · p. 28
  • With α=0.5, QAT reassignments are single grid steps toward bf16 weights, concentrated in value projections and the 3-bit tier.Latents initialized on codewords (α=0) are almost never reassigned.Source: E Further ablations · p. 29
  • After the same recalibration and QAT, stronger GPTQ starts trail codeword search by more than 20 reasoning points, while likelihood barely separates them.QAT narrows but does not close the reasoning gap.Source: E Further ablations · p. 29, Table 14 · p. 29
  • Removing Wikipedia, MMLU-auxiliary data, and then Evol-Instruct code changes the 7-task mean by at most 0.4.These ablations do not cover earlier stages or self-generated replies.Source: E Further ablations · p. 29
  • Every stage removal is equivalent to the full framework on the 7-task mean, but paired tests show that this does not capture the larger MATH-500 differences.Skipping QAT is not equivalent on the 7-task mean.Source: E Further ablations · p. 30, Table 16 · p. 30

F Paired tests by size, head precision and scale

G How checkpoints are selected

  • The selector kept the recalibrated checkpoint in every base-mixture decision, and MATH-500 agrees with those decisions.It uses four sampled answers on each of 313 disjoint level-3–5 MATH training problems and a paired sign-flip test.Source: G How checkpoints are selected · p. 30, Table 18 · p. 31

H What the recalibration changes

The analyses show that recalibration improves the next-token distribution even as blockwise weight and output errors rise. Further tests characterize QAT’s codeword changes, trace completion, and the packed serving format.

H What the recalibration changes

I QAT initialization and learning rate

  • With α=0, codewords almost never cross a rounding boundary; with α=0.5, KL initially rises during reassignment, then falls below its starting value.Starting KL differs across runs because each run is scored on different held-out text, so KL comparisons apply within runs.Source: I QAT initialization and learning rate · p. 32, Figure 6 · p. 32

J Where QAT reassigns codewords

  • QAT reassigns 6.4% of codewords, concentrated in value projections, the 3-bit tier, and the last block; almost all move one step toward bf16 weights.Most reassigned codewords were already more than half a grid step from their bf16 value.Source: J Where QAT reassigns codewords · p. 32, Figure 7 · p. 33

K Forcing an answer at a token budget

L Trace closing and accuracy on closed traces

M Packed files and serving

  • Packed checkpoints decode codewords bit-exactly and stay within 0.6% of grid-computed sizes; vLLM reads 4-bit containers, not packed 2-bit widths.The reported serving setup uses one H100; at 8B and 14B, serving cuts weight memory 1.7–3.0× and speeds batch-1 decoding.Source: Table 24 · p. 34, M Packed files and serving · p. 34, Table 25 · p. 34

N Retention on a broad benchmark suite

Retention varies across benchmark types: most knowledge and commonsense tasks retain at least 80% of bf16 scores, while several other tasks retain about half. Size comparisons report reasoning results for Qwen3-1.7B and Qwen3-14B checkpoints.

N Retention on a broad benchmark suite

  • After QAT, most knowledge and commonsense tasks retain at least 80% of bf16’s score, while MATH-500, TriviaQA, and HumanEval+ retain about half.The 14-benchmark retention comparison omits RACE because its test passages appear in MMLU auxiliary training data.Source: N Retention on a broad benchmark suite · p. 34, Figure 8 · p. 35

O Accuracy by model size in bytes

  • The 0.52 GB recalibrated checkpoint scores BFCL live 59.9 and AIME@16 1.0; it does not solve competition problems.Its BFCL multi-turn score is 2.4, with non-live at 73.0.Source: O Accuracy by model size in bytes · p. 36

P Qwen3-14B compared by size

Related papers

Loading 2610.34886v1…