Source-linked AI summary
Scaling Post-Training Ternarisation to Qwen3-8B Capability Retention, Reproduction, Lossless Packing, and Packed Execution
Anirudh Malik, M Sparsh Mehra, Poojith Devan
TL;DR
Ultra-low-bit labels do not by themselves specify a model’s actual representation, capability retention, or execution cost. This study scales an existing post-training Qwen3 conversion pipeline to 8B and evaluates capability, perplexity, serialization, and packed execution, establishing a validated baseline with open limitations in breadth and kernel efficiency.
Problem
Nominal 1.58-bit labels do not specify deployed storage or execution cost, while PTQ must also establish how much capability survives conversion.
Method
The study scales a Qwen3 post-training weight-only A16 pipeline using existing KOTMS, E2M-ATQ, and GPTQ-style components, with matched evaluation, packing, and direct execution.
Results
The 8B conversion retains 78.5% chance-corrected capability, improves 8.9 points over the matched 4B run, packs to 8.24 GiB without measurable perplexity cost, and executes at 15.52 tokens/s in 7.35 GiB.
Takeaways & Limitations
Aggressive post-training conversion can scale into the 8B regime while retaining substantial capability, producing a smaller artifact, and enabling direct packed execution.
Takeaways & Limitations
Broader seeds and calibration distributions, direct packed-model capability evaluation, and kernel optimization remain next experiments, while the current kernel is a prototype.
Abstract
from arXiv · showhide
Ultra-low-bit language models promise reductions in storage and memory traffic, but a nominal "1.58-bit" label does not specify the deployed representation or its execution cost. We study a scale-up of an aggressive post-training conversion pipeline from Qwen3-4B to Qwen3-8B. The conversion uses KOTMS rotation, E2M-ATQ adaptive ternarisation, and GPTQ-style error compensation in a weight-only A16 configuration. We do not claim these algorithms as new. Our contribution is the end-to-end scale-up characterisation: an external reproduction gate, matched 4B/8B capability analysis, cross-corpus perplexity, effective-bit accounting, lossless lattice-aware packing, and direct packed execution. The 8B model reaches a three-corpus perplexity ratio of 1.361x, with WikiText-2, C4, and PTB ratios of 1.318x, 1.393x, and 1.371x. On eight zero-shot tasks at n = 500, mean accuracy is 64.6% versus 72.4% for FP16, corresponding to 78.5% chance-corrected retention and a 7.8-point absolute cost. The matched 4B run retains 69.6%, yielding an 8.9-point 8B advantage. The packed checkpoint is 8.24 GiB and preserves the recorded perplexity to measurement precision. Direct packed execution reaches 15.52 tokens/s in 7.35 GiB, while a preliminary packed GEMV remains slower than FP16 cuBLAS. The result is a validated scale-up baseline: model size improves robustness to aggressive post-training discretisation, actual serialisation is solved for the measured artefact, and direct execution is feasible, while broader seeds, calibration distributions, and kernel optimisation remain open.
1 Introduction
Ultra-low-bit conversion must be evaluated across information content, retained capability, and actual execution rather than nominal bit width alone. This study frames Qwen3-8B conversion as a coupled system problem spanning representation, capability, and deployment.
- Transformer cost reflects both parameter count and dense numerical representation, motivating post-training quantization as a way to reduce storage and data movement without full retraining.
- 1.58-bit terminology describes one symbol’s information content, whereas complete checkpoints also include residual planes, offsets, scales, masks, rotations, embeddings, heads, and metadata.
- Capability is distinct from representation because PTQ measures how much of an existing pretrained model survives a large numerical perturbation.
- Execution gains depend on hardware-aware packing and kernels, so numerical bit width alone does not determine low-bit inference performance.
- The paper’s contribution is empirical and system-level, covering validated 8B scale-up, de-confounded scaling, packing, direct execution, and reproducibility safeguards.
2 Related Work
The study builds on trainable ternary models and post-training quantization while positioning its contribution as a Qwen3 post-pretraining scale-up using existing weight-side components. Its representation converts selected linear projections while retaining higher precision elsewhere.
- Related work: BitNet and BitNet b1.58 established trainable low-bit and ternary Transformer targets, whereas this study converts Qwen3 after pretraining.
- Related work: GPTQ, AWQ, and OmniQuant motivate calibration data, salient-weight protection, hardware-friendly inference, and non-uniform quantization sensitivity.
- Related work: TWLA combines KOTMS and E2M-ATQ with an activation-aware extension; this study uses its weight-side components in an A16 configuration without claiming a new ternarization algorithm.
- Model and representation: Only 252 linear projections are converted, while embeddings, the language-model head, normalization parameters, activations, and KV cache remain FP16.
- Model and representation: The adaptive representation uses four disjoint salience masks, with a second plane allocated selectively rather than uniformly.
4 Conversion Pipeline
The conversion pipeline combines KOTMS rotation, adaptive ternarisation, GPTQ-style compensation, and provenance checks under a memory-constrained calibration protocol. The evaluated artefact is an adaptive mixed representation rather than a pure single-plane ternary network.
- Conversion pipeline: KOTMS processes 252 projections with a measured 35-minute rotation stage before multi-hour quantization, after metadata checks.
- Conversion pipeline: E2M-ATQ selects a least-squares scale, quantizes the residual into a second support, and alternately refines μ, α0, and α1 with discrete supports fixed.
- Conversion pipeline: GPTQ-style compensation uses calibration activations and approximate inverse-Hessian information; the 8B configuration used 64 samples of 2048 tokens with seed 0 and percdamp=0.01.
- Representation: The evaluated model is an adaptive mixed representation with approximately 1.64 bits/weight, while embeddings, the head, normalization parameters, activations, and KV cache remain higher precision.
- Calibration: 64 calibration samples fit reliably within the approximately 10.98 GiB practical budget, whereas the widest projection requires 12.78 GiB at 128 samples.
- Validation: Every benchmark requires clean state keys, measurable weight changes, and end-to-end perplexity reproduction to catch plausible outputs from incorrect or malformed checkpoints.
6 Reproduction Gate
The external reproduction gate passed, with the converted model closely matching the published TWLA result. However, a 7.9% local FP16 discrepancy makes converted-to-FP16 ratios unreliable, while cross-corpus perplexity remains within the preregistered range.
- 7.9%: the local FP16 baseline differs from the published result, making absolute converted-model scores safer than ratios.
- The gate validates the integrated evaluation pipeline and provenance checks, but it is not an independent TWLA reimplementation.
- 1.361×: mean perplexity cost across three corpora falls inside the preregistered 1.25–1.45 range.WikiText-2 is most favorable at 1.318×, whereas C4 reaches 1.393×, showing that calibration-domain results alone are insufficient.
8 Capability Results
The 8B conversion retains substantial zero-shot capability, with non-uniform losses and stronger performance than the matched 4B run. Mathematical capability remains degraded but measurable above chance, while the two-size comparison does not establish a scaling law.
- Capability Results: 7.8 points: mean accuracy cost is highly non-uniform, with contextual tasks more robust and knowledge-intensive tasks weaker.The pattern is consistent with, but does not prove, a redundancy hypothesis.
- Mathematics: Degraded, Not Destroyed: All six reported mathematics subjects remain above the 25% four-way chance floor, contradicting the preregistered expectation of at-or-below-chance performance.The result preserves some mathematical structure without preserving full mathematical competence.
- Matched 4B–8B Scaling: The study should not be called a scaling law: it uses only two model sizes, one seed, and finite evaluation settings.The narrower conclusion is that increasing size changes some low-bit failures from near-destruction to measurable residual capability.
- Matched 4B–8B Scaling: Six of eight tasks: the 8B conversion outperforms 4B, led by ARC-Easy (+29.2), LAMBADA (+18.2), ARC-Challenge (+16.9), and MMLU (+10.8).HellaSwag and PIQA are the two losses, despite strong 4B retention on those tasks.
11 Representation Accounting
The measured adaptive linear-weight representation is near ternary in information rate but is not equivalent to a 1.64-bit total checkpoint. Lattice-aware serialization preserves the measured model behavior, whereas post-hoc reconstruction fails badly.
- Approximately 1.64 bits/weight: the adaptive representation exceeds the ideal single-plane ternary rate of 1.585 bits/weight.This rate applies to quantized linear weights, not the total checkpoint.
- The reconstructed FP16 tensor can contain many distinct values, so zero-count or distinct-value tests cannot verify the quantized representation.The project instead checks state provenance and end-to-end perplexity.
- 8.24 GiB: lattice-aware packing reduces the recorded checkpoint from 16.9 GB while preserving the discrete supports and associated parameters.
- Approximately 0.0007%: the WikiText-2 difference after packing, with about 0.036% relative RMSE, supports measurement-precision fidelity.
- 50.5% relative error and approximately 18,477 perplexity: fitting a simplified lattice post hoc to the saved FP16 tensor fails catastrophically.The failure shows that serialization must be designed into the quantizer because supports and activation-domain scales are not uniquely recoverable afterward.
13 Direct Packed Execution
Direct packed execution replaces the earlier offloaded regime and makes the 8B artifact runnable within 7.35 GiB. End-to-end feasibility is demonstrated, but the prototype packed GEMV remains slower than FP16 cuBLAS.
- About 1.5 tokens/s: the earlier offloaded path primarily measured PCIe transfer and CPU-offload overhead rather than low-bit arithmetic.
- 15.52 tokens/s in 7.35 GiB: the updated runtime consumes packed weights directly.
- 10.49×: the direct-to-offloaded decode ratio reflects removal of the offload bottleneck, not an FP16 speedup.
- 0.2082 ms versus 0.0451 ms: the packed Triton GEMV is 4.6× slower than tested FP16 cuBLAS at 4096 × 2560.The comparison identifies a kernel maturity gap rather than an intrinsic lower bound on ternary arithmetic.
- The study’s contribution is a validated packed 8B execution path, while broader serving efficiency remains an open engineering problem.The present work changes the strategy to PTQ and adds an external reproduction gate, but does not establish that PTQ universally beats QAT.
15 Failure Modes and Reproducibility
The study treats reproducibility, provenance, evaluation safeguards, and deployment support as necessary to distinguish valid 8B results from plausible but wrong numbers. Its matched evidence supports capability, storage, and direct-execution feasibility, while stopping short of universal speedup claims.
- Failure Modes and Reproducibility: Explicit safeguards were motivated by failures involving incorrect model identity, misleading low-bit artifacts, memory retention, CUDA spill, and inconsistent scoring.The workflow added model and checkpoint checks, state and weight-change validation, end-to-end perplexity reproduction, memory instrumentation, GPU caps, and named scoring protocols.
- Failure Modes and Reproducibility: 78.5% matched retention and above-chance mathematics support the capability result.The evidence is based on the matched capability evaluation rather than raw accuracy ratios.
- Failure Modes and Reproducibility: +8.9 retention points at 8B support the limited two-point scale-up comparison.The study frames this as evidence from the matched comparison, not a universal scaling law.
- Failure Modes and Reproducibility: 16.9 GB to 8.24 GiB with negligible recorded PPL change supports the storage result.The result depends on lossless lattice-aware packing of the measured representation.
- Failure Modes and Reproducibility: 15.52 tok/s direct packed execution in 7.35 GiB supports execution feasibility, but not a universal speedup claim.The isolated packed GEMV is 4.6× slower than FP16 cuBLAS, and FP16 8B did not fit the same card without offload.
- Failure Modes and Reproducibility: The contribution is system-level validation rather than novelty for ternary weights, low-bit algorithms, or native low-bit inference.The listed contributions cover scale-up, reproduction, matched comparison, representation accounting, packing, direct execution, and safeguards.
17 Product and Deployment Implications
The packed 8B artifact aligns capability, serialization, and direct execution sufficiently for technically plausible local deployment, but production value remains bounded by kernel efficiency, scope limitations, and incomplete comparisons.
- Product and Deployment Implications: The artifact fits in 8.24 GiB, preserves measured perplexity, and executes directly at 15.52 tokens/s in 7.35 GiB.These properties make local deployment on a 12 GB-class GPU technically plausible.
- Product and Deployment Implications: Universal serving-cost reduction should not be claimed before end-to-end comparisons across supported hardware and optimized baselines.The current deployment evidence supports potential value in distribution, local storage, single-GPU feasibility, and hardware co-design.
- Product and Deployment Implications: The retained capability profile favors contextual and commonsense behavior over difficult knowledge retrieval.This motivates targeted applications whose workloads match the measured profile rather than replacing a general FP16 model everywhere.
- Limitations and Next Experiments: The study is limited by WikiText-2 calibration and evaluation, only two model sizes, partial parameter quantization, and a prototype kernel.It also reports a 7.9% local FP16 discrepancy and lacks same-protocol comparison against every competing ternary PTQ method.
- Limitations and Next Experiments: The highest-value next experiments include multiple seeds, broader calibration, direct packed-model capability evaluation, kernel optimization, additional model sizes, and identical-protocol comparisons.These experiments address robustness, coverage, execution efficiency, and comparative positioning.
19 Conclusion
The Qwen3-8B scale-up establishes an empirical baseline for aggressive post-training low-bit conversion, combining retained capability, compact lossless serialization, and direct packed execution. The remaining challenge is turning this working path into a broadly efficient serving stack.
- Conclusion: 78.5% chance-corrected capability retention and an 8.9-point improvement over the matched 4B run establish the main scale-up result.The conclusion presents this as a strong empirical baseline rather than a universal scaling law.
- Conclusion: The adaptive representation uses near 1.64 bits/weight, and lattice-aware serialization reduces the checkpoint to 8.24 GiB without measurable perplexity cost.It is not a pure single-plane ternary network.
- Conclusion: 15.52 tokens/s at 7.35 GiB demonstrates direct packed execution on the tested RTX 5070.The prototype GEMV remains 4.6× slower than FP16 cuBLAS on one shape.
- Conclusion: Aggressive post-training conversion can scale into the 8B regime while retaining substantial capability, shrinking the artifact, and enabling direct packed execution.Broadly efficient serving remains the next engineering problem.
A Experimental Ledger
The ledger records the 8B conversion’s main capability, storage, execution, and scaling results, alongside guardrails against overinterpreting effective bit width, speed, scaling, or status claims.
- Approximately 1.64 bits/weight describes the adaptive ternary representation, not the whole model’s bits per parameter.
- 78.5% chance-corrected capability retention is reported for the 8B model under the matched eight-task protocol.
- +8.9 retention points separate the matched 8B result from the 4B result.
- 8.24 GiB is the packed artifact size, with measured perplexity preserved.
- 15.52 tokens/s is reached by direct packed execution in 7.35 GiB on the tested RTX 5070.
- The local FP16/PPL ratio is not directly comparable to the published ratio without the baseline caveat.