PaperlayerView exact revision on arXiv

Source-linked AI summary

Post-Training Ternarization of Qwen3-4B Capability, Effective Bit Budget, Storage Compression, and Deployment

Anirudh Malik, M Sparsh Mehra, Poojith Devan

arXiv:2609.01962v1cs.AIcs.LG

TL;DR

Ultra-low-bit labels do not fully capture a model’s stored representation, retained capability, or runtime behavior. This study applies a weight-only post-training conversion to Qwen3-4B using TWLA components and finds substantial storage reduction with uneven capability retention, while runtime speed remains unestablished.

  • Problem

    Ultra-low-bit labels do not fully describe complete-model storage, retained capability after post-training conversion, or deployment behavior.

  • Method

    The study applies KOTMS rotation, E2M-ATQ ternarization, and GPTQ-style compensation to Qwen3-4B with FP16 activations and ILA-AMP omitted.

  • Results

    The packed artifact shrinks from 8.29 GiB to 3.96 GiB while preserving recorded perplexity, but capability retention is uneven across tasks.

  • Takeaways & Limitations

    The work establishes a materially smaller packed representation and substantial retained capability on some tasks, without establishing a universal inference-speed advantage.

  • Takeaways & Limitations

    The study lacks an independent Qwen3-4B reproduction gate and an end-to-end packed-model task benchmark.

Abstract

from arXiv · show

Ultra-low-bit language models can reduce storage and memory bandwidth, but a nominal "1.58-bit" label does not fully describe the stored representation, retained capability, or runtime behavior. We study an end-to-end post-training conversion of Qwen, an instruction-tuned 4B-parameter model, using KOTMS rotation, E2M-ATQ ternarization, and GPTQ-style error compensation from TWLA. The experiment is weight-only: activations remain at 16-bit precision, so ILA-AMP is omitted. We evaluate effective bit accounting, task capability retention, perplexity, calibration sensitivity, checkpoint composition, and deployment behavior. The final conversion uses 1.641 effective bits per weight for quantized linear weights, with 81.62% of model parameters targeted. Across ten scored capability comparisons, accuracy falls from 64.5% to 54.7%. Degradation is uneven: BoolQ retains 84.6% chance-corrected teacher performance, while ARC-Challenge retains 43.8%. Perplexity rises from 13.639 to 18.748 on WikiText-2, 24.700 to 31.992 on PTB, and 19.831 to 28.966 on C4. A subsequent packing run preserves the ternary planes and scales, reducing reported model size from 8.29 GiB to 3.96 GiB with essentially unchanged perplexity. A separate third-party packing attempt was lossy and is excluded from the primary artifact claim. The packed artifact has not been benchmarked end-to-end for task accuracy or generation throughput. A preliminary Triton GEMV microbenchmark is 4.6x slower than FP16 cuBLAS on one tested shape. We therefore do not claim that compression alone yields faster inference.

CLOE V2.0 – OneBit AI

This study characterizes an end-to-end post-training ternary conversion of Qwen3-4B, separating representation, capability, storage, and execution behavior. It uses prior low-bit components rather than proposing a new ternarization algorithm.

  • Evaluation framing: The study frames low-bit conversion as a four-part empirical problem spanning capability, representation, storage, and deployment behavior.
  • Representation: The nominal 1.58-bit label does not equal complete-model storage cost because offsets, scales, residual planes, metadata, unquantized tensors, packing, and execution buffers also contribute.
  • Capability: Post-training conversion asks how much capability survives in an already-trained model, unlike native low-bit training, which constrains the training formulation.
  • Execution: A smaller representation is not automatically faster because unpacking, dequantization, rotation, auxiliary scales, and kernel overhead can erase arithmetic advantages.
  • Scope and contribution: The study applies prior KOTMS, E2M-ATQ, and GPTQ components to Qwen3-4B for end-to-end empirical characterization.The configuration follows the weight-only W1.58A16 path, with activation quantization disabled.

3 Model and Conversion Target

The conversion targets Qwen3-4B linear projections with adaptive ternary representations while retaining FP16 activations and other non-quantized components. KOTMS, E2M-ATQ, and GPTQ-style compensation produce a measured 1.641-bit/weight targeted representation.

  • Base model: Qwen3-4B is an instruction-tuned, dense 4B-parameter model with 36 transformer layers.
  • Conversion target: 252 linear tensors across all 36 layers are targeted, while embeddings and the untied language-model head remain BF16.
  • Execution precision: Weights are quantized aggressively, but activations, attention operations, and the KV cache remain FP16 in the W1.58A16 configuration.
  • Effective bit budget: 1.641 bits/weight is the measured effective rate for the ternarized weight representation, distinct from ideal single-symbol ternary information and total checkpoint size.
  • Conversion pipeline: The pipeline applies KOTMS rotation, E2M-ATQ weight conversion, and GPTQ-style error compensation; ILA-AMP is omitted because activations remain A16.
  • Salience masks: 96.5% of the measured weights use a single ternary plane, while the two most salient masks receive two planes.
  • Error compensation: GPTQ-style compensation propagates quantization error column-by-column using an inverse Hessian estimated from calibration activations.

5 Experimental Setup

The experiments use a controlled, resource-constrained evaluation setup on one RTX 5070, with WikiText-2 calibration and zero-shot capability testing. Memory behavior required correcting a graph-retention leak before final calibration.

  • Environment: All primary measurements were obtained on one NVIDIA RTX 5070 with 12,227 MiB VRAM under Windows using lm-evaluation-harness 0.4.12 and TWLA commit 805cbb9.
  • Calibration: Calibration used 64 WikiText-2 samples of sequence length 2048, constrained by the widest projection’s tensor size and practical VRAM limits.
  • Memory control: A broken no_grad decorator caused memory to grow from 1.99 GiB after sample one to 10.45 GiB by sample 64; restoring it reduced per-layer requirements below 3 GiB.
  • Perplexity evaluation: WikiText-2 evaluation used 146 sequences of 2048 tokens, while PTB and C4 tested whether WikiText-2 calibration and evaluation overstated the headline result.
  • Perplexity results: C4 perplexity changes from 19.831 to 28.966 in the reported evaluation.
  • Capability evaluation: Capability evaluation used zero-shot lm-evaluation-harness with approximately 1500 examples per task, batch size 2, paired records, and 10,000-resample paired bootstrap intervals.
  • Retention metric: Corrected retention uses student accuracy, teacher accuracy, and the larger chance or majority-class floor rather than raw accuracy ratios.
  • Scoring protocols: MMLU was evaluated with both answer-letter and continuation scoring because the protocols measure different output behaviors.

6 Main Results

Post-training ternarization preserves capability unevenly: contextual and commonsense tasks are more robust than knowledge-heavy tasks, while degradation varies across evaluation corpora and scoring protocols.

  • Language modelling quality: Perplexity degradation depends on the evaluation corpus: WikiText-2 is most favorable, whereas C4 is least favorable, so 1.375× is not invariant.WikiText-2 is the calibration distribution; PTB and C4 serve as out-of-distribution checks.
  • Task-level capability: No reported confidence interval crosses zero under the paired bootstrap protocol, although the study uses one seed and a finite evaluation limit.
  • Task-level capability: Across ten scored comparisons, the descriptive arithmetic mean is 64.5% for FP16 versus 54.7% for W1.58A16, not an all-benchmark or chance-corrected aggregate.LAMBADA is guarded separately because open-vocabulary tasks lack the same simple chance floor as fixed-choice tasks.
  • Capability stratification: BoolQ retains 84.6% of teacher performance, while ARC-Challenge retains 43.8%, with commonsense and contextual tasks generally outperforming knowledge-oriented tasks.HellaSwag, PIQA, and WinoGrande retain 72–81%; MMLU continuation retains 50.0%.
  • Capability stratification: The retention pattern is stratified rather than uniform and is consistent with greater damage to precise stored knowledge or fine-grained factual associations.The small benchmark suite does not establish a causal law; broader task families are needed for confirmation.
  • Task-level capability: MMLU scoring changes both arms substantially: letter scoring gives 68.50% versus 50.96%, while continuation scoring gives 45.10% versus 35.04%.Neither MMLU result should be quoted without its scoring protocol because answer formatting can confound capability with decoding behavior.

7 Representation Analysis

The conversion uses an adaptive ternary representation with 1.641 effective bits per targeted weight, while complete-checkpoint storage remains higher because non-quantized tensors and metadata are retained. Calibration evidence supports the completed 64-sample configuration as a measured result, not a proven optimum.

  • 7.1 From nominal 1.585 bits to 1.641 effective bits: 1.641 bits/weight is the measured effective budget for the targeted adaptive weight representation.The theoretical one-plane ternary lower bound is 1.585 bits/weight; a selectively used second plane raises the measured budget.
  • 7.1 From nominal 1.585 bits to 1.641 effective bits: 3.6% of weights receive a second ternary plane selected by Hessian-based salience.The selection reflects non-uniform parameter importance rather than arbitrary allocation.
  • 7.3 Representation verification: 1,169 distinct reconstructed row values collapsed to 235 after conversion, while WikiText-2 perplexity changed from 13.639 to approximately 18.748.The end-to-end perplexity check also guards against evaluating a rotated or unconverted checkpoint.
  • 7.2 Effective bits versus complete-checkpoint storage: 1.641 bits/weight does not describe the complete checkpoint, which retains BF16 embeddings, the language-model head, rotation matrices, and metadata.Source accounting estimates approximately 745 MiB for packed ternary weights, 1,483 MiB for embedding/head tensors, and 78 MiB for rotation matrices.
  • 8 Calibration and Quality Sweep: 19.087 perplexity resulted from 96 calibration samples versus 18.749 with 64 samples, a 1.8% worsening rather than evidence that 64 is inherently better.The comparison was single-seed and the remaining calibration levers were not completed.
  • 8 Calibration and Quality Sweep: 64 samples is the best completed configuration in the provided sweep, not a proven global optimum.FineWeb-Edu calibration was queued but not started, and group_size=64 was not evaluated.

9 Storage and Packing

The section distinguishes effective quantized representation from serialized checkpoint size and shows that correct packing preserves model behavior, whereas lossy packing is excluded from the primary claim. Deployment measurements further indicate that compression and inference efficiency remain separate questions because the tested software path is not native ternary execution.

  • Storage and Packing: The first quantized checkpoint stored converted values in FP16 tensors, so its lower effective information rate did not reduce disk footprint.This demonstrates why quantization and serialization must be treated as distinct operations.
  • Storage and Packing: A later lattice-aware packing run persisted ternary planes with scales and offsets, and reproduced the unpacked model’s perplexity within reported measurement precision.The report also records 0.037% relative RMSE and a 50-second packing time.
  • Storage and Packing: 1.641 bits/weight describes the adaptive quantized weights, whereas 3.96 GiB describes the full serialized checkpoint including unquantized components and metadata.The BF16 embedding and head tensors dominate the remaining packed footprint.
  • Storage and Packing: The third-party 2.659 GiB artifact discarded the second ternary plane and GPTQ residual, producing a maximum absolute weight error of 0.305.Because it had no perplexity or task benchmark and was judged unsuitable for publication, its size and speed are excluded from the primary model claim.
  • Deployment Behavior: The measured deployment path is a reconstruction-oriented implementation rather than an optimized native ternary accelerator, so its behavior does not establish a general limitation of ternary representations.Weights remain stored in FP16, and KOTMS adds an activation-side matrix operation.
  • Deployment Behavior: The section therefore keeps storage compression separate from inference efficiency, since a compact representation helps deployment only when execution kernels consume it directly and efficiently.A Triton kernel can reproduce its encoded dense operation with approximately 0.00195 maximum absolute discrepancy, but this does not establish end-to-end speedup.

11 Failure Modes as Experimental Boundary Conditions

The study treats infrastructure failures, evaluation safeguards, and incomplete comparisons as boundary conditions for interpreting aggressive PTQ. Within those constraints, it reports a working Qwen3-4B conversion and frames its contribution as empirical characterization rather than a new quantizer.

  • Failure Modes: Memory leaks and misleading CUDA allocation readings could invalidate calibration runs, motivating instrumentation, clean failure limits, and per-layer checkpointing.A checkpointing edit raised memory from 1.99 GiB to 10.45 GiB; another probe showed system-memory spill under Windows.
  • Evaluation Boundary Conditions: Evaluation safeguards rejected chance-uncorrected retention ratios, flagged MMLU letter scoring, required per-item logs, and enforced paired task evaluation.The authors explicitly classify these corrections as experimental method rather than editorial cleanup.
  • Scope of the Result: The PTQ route produces a working 4B W1.58A16 conversion without full retraining, with retained capability on several tasks and verified multi-gigabyte compression.The study does not claim that PTQ universally beats QAT or provide a controlled same-model, same-budget comparison.
  • Capability Boundary: Capability is uneven: contextual and commonsense tasks remain stronger than knowledge-heavy tasks, while the evidence cannot isolate model-scale and conversion-procedure effects.The comparison with the prior Cloe study supports stratified reporting rather than a single average.
  • Contribution Boundary: The contribution is an end-to-end Qwen3-4B characterization covering effective-bit accounting, task degradation, calibration sensitivity, packing, and deployment measurements.The paper explicitly treats KOTMS, E2M-ATQ, and GPTQ as prior methods rather than claiming a new ternarization algorithm.

14 Product and Deployment Implications

The deployment evidence supports a compression-first interpretation: the artifact becomes materially smaller, but current execution measurements do not establish faster or cheaper serving. The practical opportunity therefore depends on native kernels, hardware-aware execution, and workload-specific validation.

  • Compression-First Deployment: 8.29 GiB becomes 3.96 GiB, making the packed artifact materially easier to move and store.The study presents model distribution as the most immediate deployment opportunity.
  • Execution Opportunities: Native kernels could consume stored ternary planes directly and remove reconstruction overhead.The proposed path is a future engineering opportunity, not a demonstrated runtime result.
  • Hardware Co-Design: Specialized low-bit kernels and hardware co-design are identified as routes toward converting representation efficiency into runtime efficiency.The paper connects this opportunity to the low-cardinality arithmetic structure and prior BitNet systems.
  • Workload Scope: The capability profile does not justify broad general-purpose replacement, while targeted workloads may offer a more favorable quality-efficiency trade-off.This implication follows the stronger retention on some contextual and commonsense tasks and weaker retention on knowledge-heavy tasks.
  • Claims Not Supported: The dataset does not support universal inference speedup, serving-cost reduction, production robustness, or superiority over every alternative PTQ method.The authors also reject claims of equivalence to native 1.58-bit models and general scaling laws across model sizes.

15 Limitations and Required Next Experiments

The main limitations concern validation breadth, incomplete tuning, partial quantization, and the gap between perplexity preservation and end-to-end deployment evidence. The next experiments therefore target reproducibility, calibration, packed-model benchmarking, and broader capability validation.

  • Validation: Qwen3-4B lacks an independent reproduction gate because it is absent from the TWLA paper.The authors identify reproduction on the published reference model as the highest-value validation.
  • Validation: Single-seed capability results do not quantify run-to-run variation.Repeating the capability suite across seeds is listed as a required next experiment.
  • Calibration: The percdamp=0.01 run did not complete, leaving alternative calibration corpora and representation parameters unevaluated.A broader calibration corpus such as FineWeb-Edu is proposed for a rerun.
  • Packed Artifact: The packed artifact has perplexity evidence but no end-to-end task benchmark, generation-throughput measurement, or exact packed-model capability evaluation.The next step is to benchmark the exact packed artifact end-to-end.
  • Deployment Scope: The Triton GEMV is preliminary, and embeddings plus the LM head remain BF16, limiting conclusions about native performance and total footprint.The study also notes that the final task suite is smaller than comprehensive LLM evaluations.

16 Conclusion

The study shows that Qwen3-4B can enter an aggressive near-ternary post-training regime with meaningful but uneven capability retention and substantially smaller lossless-packed storage. Runtime efficiency remains unresolved because the current reconstruction and kernel paths are slower than FP16.

  • Conclusion: 1.641 bits/weight and a 9.8 percentage-point mean accuracy cost summarize the representation and capability trade-off.Contextual and commonsense tasks are more robust than knowledge-heavy tasks.
  • Conclusion: 8.29 GiB becomes 3.96 GiB with recorded perplexity preserved to measurement precision in the lattice-aware packing run.This distinguishes the smaller artifact from a merely fake-quantized FP16 checkpoint.
  • Conclusion: 4.6× slower than FP16 cuBLAS was the preliminary packed Triton GEMV result on the tested shape.The measurement reflects the current software path and should not be generalized to ternary arithmetic itself.
  • Conclusion: The strongest interpretation is that near-ternary PTQ can create a smaller 4B artifact while retaining meaningful pretrained capability, but production efficiency requires native packing, kernels, and broader validation.The study explicitly avoids claiming that ternarization alone solves efficient inference or destroys capability.
  • Experimental Scope: The primary completed conversion used Qwen3-4B with 64 calibration samples, sequence length 2048, WikiText-2 calibration, and seed 0.These configuration details define the reported experimental scope.

C Implementation and Reproducibility Checklist

The reproducibility checklist specifies the model, quantization, calibration, evaluation, and software environment used for the conversion. It also records guardrails against overstating the bit budget, capability results, packing speed, hardware efficiency, or comparative claims.

  • Configuration: The experiment uses Qwen3-4B with TWLA E2M-ATQ, KOTMS rotation, and GPTQ compensation.The base model is instruction-tuned Qwen3-4B.
  • Calibration and quantization: Calibration uses WikiText-2 with 64 samples and 2048 tokens, damping 0.1, block/group size 128, order-2 salience grouping, and num_p=1.These settings define the recorded quantization configuration.
  • Evaluation: Evaluation uses the full WikiText-2 test split for perplexity and a zero-shot capability suite, with paired bootstrap analysis over 10,000 resamples.The checklist identifies both evaluation scopes and the statistical procedure.
  • Environment and artifacts: The recorded environment is an NVIDIA RTX 5070 with 12,227 MiB, Windows, and specified PyTorch, Transformers, Datasets, lm-eval, NumPy, and Python versions.Checkpoint and rotated-checkpoint hash prefixes are also recorded for artifact identification.
Loading 2609.01962v1…