PaperlayerView exact revision on arXiv

Source-linked AI summary

RoleQ: Role-Aligned Scales for Converting Language Models to Ternary

Johannes Knittel, Alec Helbling, Hanspeter Pfister

arXiv:2610.21808v1cs.AIcs.CL

TL;DR

Sub-four-bit quantization can retain acceptable perplexity while losing factual knowledge and task accuracy, and ternary conversion has faced a trade-off between retention and computational cost. RoleQ aligns scales with Transformer-unit roles, and RoleQ-T converts existing checkpoints to ternary. The paper reports that RoleQ-T outperforms tested post-training ternary baselines at matched or lower memory-resident rates, with the largest gains on knowledge.

  • Problem

    Below four bits, acceptable perplexity can conceal losses in factual knowledge and task accuracy, while ternary methods that preserve knowledge have incurred substantial computational cost.

    Source: §1 · p. 1

  • Method

    RoleQ assigns scales according to whether vectors read from or write to the residual stream, and RoleQ-T combines this format with data-free rotation, ternary GPTQ and brief scale distillation.

    Source: §1 · p. 1

  • Results

    RoleQ-T outperforms Qronos, PT2-LLM and QTEA on perplexity, zero-shot accuracy and MMLU at matched or lower memory-resident rates, with the largest gains on knowledge.

    Source: §1 · p. 2

  • Takeaways & Limitations

    At very low precision, both scale sharing and the placement of additional bits shape knowledge retention, so low-bit formats should be judged by where they spend bits as well as by their rate.

    Source: §5 · p. 9

    The placement-rule advantage is supported in a rotated basis, not native coordinates, and the evaluated models and tasks do not establish a general scaling trend or cover generation and reasoning.

    Source: §5 · p. 9

Abstract

from arXiv · show

Weight quantization below four bits shrinks memory and bandwidth at a perplexity that looks acceptable, but often at the cost of factual knowledge and task accuracy. Ternary weights have mostly been obtained by training from scratch or long fine-tuning. We introduce RoleQ, a quantization framework that places scales by the functional roles of Transformer units rather than by the layout of the weight matrix, as conventional formats do. Vectors that read from the residual stream take one scale per output unit; those that write to it, one per input unit. The rule is structural, needs no search, calibration or stored state, and holds on integer and ternary grids. In a rotated basis it matches a per-layer axis search at half the solver work, and at ternary it cuts perplexity by up to half against layout-based placement in two families at every size from 0.6B to 14B, where per-row scales leave MMLU at chance. Our lightweight post-training pipeline converts an existing checkpoint to 1.59 bits per weight in a few GPU-hours, a fraction of the cost of learned ternarization, and leads every post-training ternary method at matched or lower memory-resident bits on perplexity, zero-shot accuracy and MMLU, most widely on knowledge. A sparse second ternary plane placed by a per-unit Fisher saliency keeps the matmul multiplication-free and recovers what ternary loses first: at 8B, a fraction of a bit returns about half the MMLU that conversion costs, well ahead of random or Hessian-based placement at equal bits.

1 INTRODUCTION

Sub-four-bit quantization can preserve perplexity while losing factual knowledge and task accuracy, and existing ternary conversion faces a trade-off between retention and computational cost. RoleQ assigns scales by Transformer-unit function, while RoleQ-T converts checkpoints efficiently and improves on post-training ternary baselines.

1 INTRODUCTION

  • Below four bits, acceptable perplexity can conceal losses in factual knowledge and task accuracy, motivating evaluation with perplexity, seven-task zero-shot average and five-shot MMLU.The stated conversion challenge is to approach training-based knowledge retention at post-training conversion cost.Source: §1 · p. 1
  • At 8B, RoleQ-T converts Qwen3-8B-Base in a few GPU-hours to 1.59 bpw, with 11.33 perplexity, 60.3 zero-shot accuracy and 47.6 MMLU.The passage contrasts this with tens of A100-hours for CAT-Q’s learned ternarization at this size.Source: §1 · p. 1
  • RoleQ assigns one scale per output unit to vectors reading from the residual stream and one per input unit to vectors writing to it, on integer and ternary grids.This structural rule needs no calibration data or stored axis choices.Source: §1 · p. 1
  • At 14B, ternary RoleQ placement raises MMLU from 25.4 with per-output-channel scales to 38.5 at equal bits.The introduction reports the placement-only comparison for Qwen3-14B.Source: Figure 1 · p. 2
  • At 8B, adding Fisher-selected ternary planes reaches 2.16 bpw with 9.30 perplexity, 65.9 zero-shot accuracy and 62.8 MMLU.The sparse second plane preserves multiplication-free matrix products; at 1.99 bpw on post-trained Qwen3-8B, RoleQ-T surpasses CAT-Q on all three measures for 0.28 additional bits.Source: §1 · p. 2
  • RoleQ-T outperforms Qronos, PT2-LLM and QTEA on perplexity, zero-shot accuracy and MMLU at matched or lower memory-resident rates, with the largest gains on knowledge.At 14B, it surpasses QTEA’s released checkpoint after scale distillation using 0.67 fewer bpw.Source: §1 · p. 2

2 RELATED WORK

Prior work spans ternary training and conversion, low-bit codebooks, scale-axis selection, and mixed-precision allocation. RoleQ is positioned as a structural, role-based scale choice without calibration, alongside a unit-level loss-sensitivity approach to allocating extra precision.

2 RELATED WORK

  • Pretraining, quantization-aware training, code fine-tuning and learned ternarization retain knowledge at computational cost, while direct post-training methods are cheaper but can lose factual knowledge.CAT-Q optimizes codes for 60 epochs using 512 calibration samples; ScaleQ-1.58 uses the model’s own reasoning traces.Source: §2 · p. 2
  • Two-bit lattice and trellis codebooks provide strong quality references, while RoleQ uses scalar, codebook-free ternary codes that support multiplication-free matrix products.QTIP adopts QuIP#-style fine-tuning, reported at 50–100 GPU-hours for a 70B model.Source: §2 · p. 3
  • Prior scale-axis methods select axes empirically or refine scales in a solver using calibration data; RoleQ selects each axis from vector role without calibration or stored axis flags.The described alternatives include per-module reconstruction tests, solver-sweep refitting and folded scales on a second axis.Source: §2 · p. 3
  • Prior mixed-allocation methods use curvature, gradient, Fisher or task-conditioned scores, whereas RoleQ conditions its score on factual recall and reports curvature-based selection can lose to random placement at tight budgets.Source: §2 · p. 3

3 METHOD

RoleQ assigns scales by Transformer units’ functional roles, while RoleQ-T combines role-aligned ternary quantization with rotation, a GPTQ adaptation, and scale distillation. Sparse second ternary planes allocate extra precision to units ranked by Fisher saliency.

3 METHOD

  • RoleQ-T combines role-aligned scales, a data-free learned rotation, ternary GPTQ, and brief scale distillation with frozen codes to convert existing checkpoints.The placement rule is data-free; the solver and distillation use unlabeled data.Source: §3 · p. 3

3.1 ROLEQ: ROLE-ALIGNED SCALES

  • Readers receive one scale per vector that reads from the residual stream, while writers receive one per vector that writes to it.The rule aligns scales with functional units rather than shared residual coordinates.Source: §3.1 · p. 4

3.2 ROTATIONS

  • 1.38× lower 8B perplexity than random Hadamard rotation comes from the recipe s1(l4) + s2(opt) + r4.Rotations add 0 bpw and fold into the weights, apart from r4’s runtime Hadamard.Source: §3.2 · p. 4

3.3 TERNARY GPTQ: THE ROLEQ-T SOLVER

  • RoleQ-T adapts GPTQ to ternary role-aligned scales, using exact vector fits at different sweep stages for readers and writers.Writer scales are fit during the sweep; reader grids are initialized before it and their scales refit at final codes, while the overall sweep remains greedy.Source: §3.3 · p. 5

3.4 SCALE DISTILLATION

  • Scale distillation refits scales and norm gains against the full-precision teacher’s next-token distribution while keeping quantized codes frozen.It uses unlabeled data and takes 2 h on one A100 at 8B.Source: §3.4 · p. 5

3.5 FISHER-PLACED SPARSE SECONDARY PLANES

  • A sparse second ternary plane goes to Fisher-ranked functional units, targeting one-plane quantization residuals while keeping matrix products to two ternary matmuls and an addition.The score uses observed-target gradients as a sensitivity proxy, not an exact estimate of recoverable loss; Fisher rankings vary with the item set and scoring checkpoint, with split-half Spearman correlations of 0.87–0.98.Source: §3.5 · pp. 5–6

4 EXPERIMENTS

Across Qwen and Llama experiments, RoleQ-T converts existing checkpoints to ternary weights and leads tested post-training ternary baselines on complementary quality measures. Role-aligned scale placement improves ternary results, while sparse Fisher-selected planes recover knowledge at added bit budgets.

4.1 TERNARY CONVERSION RECIPE

  • RoleQ-T reaches 1.590 bpw with scale distillation in three to four A100 hours on Qwen3-8B-Base, and 2.157 bpw with probing and planes in under ten hours.The experiments cover Qwen3 Base and post-trained models, a Llama-3.1-8B-Base cross-family check, and WikiText2 perplexity, seven-task zero-shot average and five-shot MMLU.Source: §4.1 · p. 6, §4 · p. 6
  • At CAT-Q’s 1.71 bpw, RoleQ-T-g128 leads on perplexity (17.00 vs 27.68) and zero-shot accuracy (61.7 vs 61.0), and ties on MMLU (53.1 vs 52.9).At 1.99 bpw, RoleQ-T planes reach 61.8 MMLU versus CAT-Q’s 52.9, with 0.28 additional bits; RoleQ-T uses under ten GPU-hours, compared with CAT-Q’s 8–64 A100-hours.Source: §4.1 · p. 7
  • At 14B, RoleQ-T with planes and scale distillation reaches 8.05 perplexity, 69.2 zero-shot and 68.3 MMLU at 2.156 bpw, versus QTEA’s 11.78, 56.6 and 47.8 at 2.26 bpw.The one-plane run also leads after distillation at 1.589 bpw, but trails at the PTQ stage.Source: §4.1 · p. 7

4.2 THE PLACEMENT RULE

  • At ternary, RoleQ has the lowest mean perplexity at every tested size in both model families, improving over the best uniform placement by up to 2×.At 14B, MMLU rises from 25.4 with per-output-channel scales to 38.5; the placement comparison uses shared solver, rotation, calibration draw and paired seeds without distillation.Source: §4.2 · p. 7

4.3 SPARSE SECONDARY PLANES

  • At 8B, +0.57 bpw of Fisher-selected planes raises MMLU from 33.3 to 56.0 at PTQ and from 47.6 to 62.8 after distillation.At 1.99 bpw, knowledge-Fisher selection beats random placement by 6.2 MMLU points and Hessian×weight by 9.0; at 14B, plane gains are smaller in five of six measures.Source: §4.3 · pp. 7–8

4.1 TERNARY CONVERSION RECIPE

  • At 1.590 bpw, RoleQ-T leads Qronos, PT2-LLM and QTEA on perplexity, zero-shot accuracy and MMLU while using matched or fewer bits.On Qwen3-8B-Base, it retains 47.6 MMLU, compared with 25.9–34.8 for un-distilled baselines and 76.8 at full precision.Source: §4.1 · pp. 6–7, Table 1 · p. 8

4.3 SPARSE SECONDARY PLANES

  • At matched bits, ternary planes outperform four-level grids in some sizes and bases, but four-level grids lead at 14B under r1, so neither format dominates across sizes.Ternary planes retain multiplication-free matmul without a codebook or group metadata.Source: §4.3 · p. 8

5 CONCLUSION

RoleQ assigns scales by functional units, and RoleQ-T converts existing checkpoints to ternary; role-aligned placement benefits persist to the largest models tested. RoleQ-T outperforms tested post-training ternary baselines at matched or lower memory-resident rates, with its largest gains in factual knowledge.

5 CONCLUSION

  • Role-aligned placement benefits persist to the largest models tested, using the Transformer’s reader and writer roles as a structural scale-placement rule.Table 2 compares WikiText2 perplexity across four Qwen3-Base sizes and Llama-3.1-8B-Base under matched PTQ settings.Source: §5 · pp. 8–9, Table 2 · p. 9
  • RoleQ-T outperforms tested post-training ternary baselines at matched or lower memory-resident rates, with the largest gains in factual knowledge.Perplexity alone does not capture those knowledge gains.Source: §5 · p. 9
  • Fisher-guided secondary planes allocate extra precision to loss-sensitive units and further recover knowledge.Table 3 reports the selector comparison on post-trained Qwen3-8B after scale distillation at matched bit rates.Source: §5 · p. 9, Table 3 · p. 9
  • Placement-rule support is in a rotated basis; four Qwen3 sizes and one Llama model do not establish a general scaling trend.The advantage does not extend to native coordinates, and the best uniform placement varies by size and family.Source: §5 · p. 9
  • The evaluations are weight-only and omit generation and reasoning, including long chains of thought; they measure storage and quality, not inference speed.At 2 bpw on Llama, QTIP trellis codes, alone or with GuidedQuant’s Hessian, remain ahead.Source: §5 · p. 9
  • At very low precision, scale sharing and the placement of additional bits both shape knowledge retention; under tight budgets, precision allocation can matter more than the overall rate.The paper concludes that low-bit formats should be judged by both their bit count and where bits are spent.Source: §5 · p. 9

A NOMENCLATURE AND BIT ACCOUNTING

This appendix defines the paper’s quantization grids, scale granularity, and memory-resident bit accounting. It also distinguishes conversion stages and describes the hardware rationale for ternary weights while noting that the paper reports no latency or energy measurements.

A NOMENCLATURE AND BIT ACCOUNTING

  • INTb denotes a grid with 2^b levels; ternary refers to the three-level code, while INT2 is reserved for four-level grids.Rates use log2 L information bits per code; weights alone are quantized, with activations, embeddings, and the head remaining in fp16.Source: A NOMENCLATURE AND BIT ACCOUNTING · p. 16
  • Readers take per-output-channel scales on the unit axis, while writers take per-input-channel scales on the unit axis.The five reader matrix types are WQ, WK, WV, Wgate, and Win; the two writers are WO and Wout.Source: A NOMENCLATURE AND BIT ACCOUNTING · p. 16
  • Memory-resident information rate counts every term needed by a decoder, including scales, zero points, indices, group or plane membership, and non-regenerable rotations.Per-vector scales and reintroduced norm gains move RoleQ-T from 1.585 to 1.590 bpw; packing is not charged.Source: A NOMENCLATURE AND BIT ACCOUNTING · p. 16
  • The paper distinguishes five stages of increasing cost, from calibration-free PTQ through calibrated PTQ, scale distillation, fine-tuning, and training from scratch.Scale distillation refits scales and norm gains to the fp model’s next-token distribution with codes frozen.Source: A NOMENCLATURE AND BIT ACCOUNTING · p. 17
  • Ternary weights can replace multiplication with adding, subtracting, or skipping activations, while memory traffic still scales with bits per weight.The paper reports no latency or energy measurements of its own.Source: A NOMENCLATURE AND BIT ACCOUNTING · p. 17

B SETUP DETAILS

Table 6 lists the models, corpora, seeds, solver settings, and evaluation protocol shared across the experiments.

B SETUP DETAILS

  • Table 6 lists the models, corpora, seeds, solver settings, and evaluation protocol shared by every experiment.The passage provides no further setup values.Source: B SETUP DETAILS · p. 17

C THE PLACEMENT RULE: PER-SEED CELLS AND CONTROLS

The controls test whether placement differences reflect scale placement rather than code, fit, rotation, reference choice, or run variation. Their results support a rotated-basis placement rule, with separation varying across model size and family.

C THE PLACEMENT RULE: PER-SEED CELLS AND CONTROLS

  • On the four-level g128 grid, Qwen’s ordering is RoleQ < per-OC < per-IC at every tested size, including 14B.At 14B the values are 10.76, 10.90, and 11.64, respectively.Source: C THE PLACEMENT RULE: PER-SEED CELLS… · p. 17
  • At 1.7B, writer and reader placement margins lie outside the seed spread at 19% and 30%; at 14B, only the reader margin does.The 14B reader margin is 8.2% at 2.9× the seed range, while the writer margin is 1.3% at 0.5×.Source: C THE PLACEMENT RULE: PER-SEED CELLS… · p. 17
  • AdaDim’s per-layer search agrees with the placement rule on 96.4% of cells in the rotated basis, versus 14.3% unrotated.The paper therefore states the placement rule for a rotated basis.Source: C THE PLACEMENT RULE: PER-SEED CELLS… · p. 17
  • The best uniform reference placement varies: per-IC at Qwen-8B and per-OC at 14B and on Llama.On Llama-8B, role and per-OC placements of the same grid tie in the recipe basis.Source: C THE PLACEMENT RULE: PER-SEED CELLS… · p. 17
  • Llama-8B is the only cell where both writer and reader halves separate, at 1.97× and 11.52×, respectively.Those runs are at 9.2× and 18.2× full precision.Source: C THE PLACEMENT RULE: PER-SEED CELLS… · p. 17
  • At 8B, separate PTQ-only placement runs reproduce the corresponding recipe-ladder bodies seed by seed.RoleQ values were 30.7 / 31.5 versus 31.0 / 31.0; per-OC values were 36.0 / 35.0 versus 35.4 / 34.9.Source: C THE PLACEMENT RULE: PER-SEED CELLS… · p. 17

D THE CONVERSION RECIPE: DETAILS

The conversion recipe combines role-aware solver steps with scale distillation and evaluates methods using memory-resident rates. Its reported group-scale comparison finds role-axis placement on par with standard placement after distillation.

Solver settings

  • GPTQ uses activation order, 0.01× mean-diagonal damping, and 128-column lazy batching, capturing each Hessian after preceding layers are quantized.Lazy batching changes speed but not the result.Source: D THE CONVERSION RECIPE: DETAILS · p. 17

Solver steps

  • Writer vectors use an exact unweighted ternary step on compensated vectors, while reader rows receive a scale refit that can only lower error.The writer step is exact per step, but its order remains GPTQ’s greedy activation order.Source: D THE CONVERSION RECIPE: DETAILS · pp. 18–19

Bit accounting

  • Decoder-rate comparisons count every term needed to produce a weight; folded or seed-regenerable rotations cost zero, and packing is excluded.The accounting includes decoder-required metadata such as scales, indices, memberships, and reintroduced norm gains.Source: Table 5 · p. 19

Scale distillation

  • Scale distillation uses 400 steps and batch size 8 on RedPajama-1T-Sample, while calibration remains on C4.The teacher is not run during training; cached final residuals allow the stage to fit beside the student on one GPU.Source: D THE CONVERSION RECIPE: DETAILS · p. 20

Protocol notes

  • The calibration draw differs from the standard protocol, and the authors report that this shape favours the baselines.Competitor numbers were produced in the authors’ harness and validated against method-specific evaluation code.Source: D THE CONVERSION RECIPE: DETAILS · p. 20

Group scales

  • With group scales, role-axis placement is on par with standard placement after scale distillation; at PTQ it leads on all three instruments by less than 3× the seed range.At PTQ, the reported differences are 18.49 versus 19.16 perplexity, +1.2 zero-shot, and +2.6 MMLU.Source: D THE CONVERSION RECIPE: DETAILS · p. 20

E SPARSE PLANES: DETAILS

The sparse-plane analysis compares allocation strategies and controls at matched rates, while examining which saliency signals recover different outcomes. Its scope includes a 1.7B budget sweep and 8B matched-rate comparisons.

Refit by role

  • For readers, two plane scales are jointly refit by an exact 2×2 weighted least-squares system; writer scales instead require a coupled 2n×2n system handled inside GPTQ recursion.The writer shortcut is unavailable because H acts on the axis indexed by the scales.Source: E SPARSE PLANES: DETAILS · p. 21

Axis-search comparison

  • AdaDim’s axis search was not run end to end; its differing choices from the rule occur in few cells and have negligible reported H-metric error relative to seed noise.The reported differences are 4 of 112 cells at 1.7B and 6 of 112 at 8B.Source: Table 8 · p. 22

Matched-bit controls

  • At 8B after scale distillation, planes lead both four-level controls on perplexity by 1.6% and on MMLU by 2.4 points, with the MMLU gap within seed variance.The perplexity lead has the same sign on every seed pairing and is 2.7× the per-output-channel run’s seed range.Source: E SPARSE PLANES: DETAILS · pp. 21–22

Trellis-coded comparison

  • At matched bits, trellis-coded methods lead the ternary recipe on every instrument in the reported Llama-3.1-8B-Instruct comparison.Against QTIP after distillation, the recipe trails by 4.3 zero-shot and 8.9 MMLU points; against GuidedQuant, by 2.5 and 6.6.Source: Table 10 · p. 23

Selector and item effects

  • Text Fisher is best on perplexity and worst on factual recall, while knowledge Fisher shows the reverse; the checkpoint changes effect size, not instrument direction.The authors condition the score on the item set because the item set determines what is recovered.Source: E SPARSE PLANES: DETAILS · p. 23

F ROTATION DETAILS

The rotation details treat basis choices as part of a run’s identity and test how rotation slots affect quantization. They also report a 1.7B budget sweep comparing second-plane placements.

Rotation slots

  • The rotation slots act on distinct weight axes: r1 folds into reader inputs and writer outputs, r2 mixes V→O per kv-head, and r4 rotates Wout’s input space.The learned s1 and s2 refine r1 and r2 at the same endpoints using weights alone and no labels.Source: F ROTATION DETAILS · pp. 23–24

Plane budget sweep

  • At 1.7B, the second-plane Fisher advantage over random placement declines from 1.96× at +0.15 bits to 1.12× at +0.90; Hessian×weight overtakes Fisher at +1.20 bits.The sweep is for Qwen3-1.7B post-trained PTQ WikiText2 perplexity; the decay is not claimed at 8B.Source: Figure 3 · p. 24

Basis labelling

  • The recipe basis is s1(l4)+s2(opt)+r4, and runs from different bases are kept in separate table rows rather than used for cross-basis plane effects.The authors state that basis has a significant impact on quantization results.Source: F ROTATION DETAILS · p. 24

Rotation and axis choice

  • The axis preferred by AdaDim agrees with the role rule on 96.4% of rotated cells but 14.3% of unrotated cells.The proposed structural explanation is that r1 mixes the residual-stream axis while leaving the unit axis untouched.Source: F ROTATION DETAILS · pp. 24–25

G ABLATION DETAILS

The ablations compare rotation objectives, slots, solver controls, and calibration-draw shapes. They show that weight-space fit and model perplexity do not rank rotation choices identically.

Rotation

  • At 1.7B, the full recipe basis scores 73.66 perplexity at PTQ and 20.26 after scale distillation; at 8B it scores 22.42 versus 30.99 under r1.With per-output-channel scales, s1(opt) reaches 585.6 perplexity versus 130.3 with the placement rule.Source: G ABLATION DETAILS · p. 25
  • The lowest weight-space error, 0.65× that of random Hadamard, has the highest perplexity at 130.3, while l4 reaches 89.99.The comparison is among s1 objectives with the recipe’s rotation ablation setup.Source: G ABLATION DETAILS · p. 25, Table 13 · p. 26

Solver and draw

  • Placement runs share 82–86% of their trits with one another and 83.8% with a second-seed run of the same configuration.The authors interpret this as evidence that the margin between placements is a distributional effect rather than a different set of rounding decisions.Source: G ABLATION DETAILS · p. 25
  • The shape of the calibration draw moves every method by more than most format effects, and the paper draw favours baselines as much as it favours RoleQ.The compared draws are 512×512 and the standard 128×2048 tokens.Source: G ABLATION DETAILS · p. 25, Table 14 · p. 27

Related papers

Loading 2610.21808v1…