RaBiT: Residual-Aware Binarization Training for Accurate and Efficient LLMs

Youngcheon You; Banseok Lee; Minseop Choi; Seonyoung Kim; Hyochan Chong; Changdong Kim; Youngmin Kim; Dongkyu Kim

Source-linked AI summary

RaBiT: Residual-Aware Binarization Training for Accurate and Efficient LLMs

Youngcheon You, Banseok Lee, Minseop Choi, Seonyoung Kim, Hyochan Chong, Changdong Kim, Youngmin Kim, Dongkyu Kim

arXiv:2602.05367v3Feb 5, 2026cs.AI

TL;DR

Extreme 2-bit quantization must balance efficiency against performance, but residual binarization is undermined by inter-path adaptation. RaBiT couples binary paths through a shared full-precision weight and function-aware initialization, achieving state-of-the-art 2-bit results with competitive efficiency. Its scope remains limited by unresolved questions about safety alignment under extreme compression.

Problem
Residual binarization offers matmul-free efficiency but suffers from inter-path adaptation, where shared QAT gradients make parallel paths redundant and weaken error compensation.
Method
RaBiT sequentially derives binary paths from one shared full-precision weight and stabilizes QAT with function-aware initialization.
Results
RaBiT achieves state-of-the-art 2-bit performance, with up to a 4.49× inference speed-up and competitive results against hardware-intensive VQ methods.
Takeaways & Limitations
RaBiT offers a viable pathway for efficient deployment of high-performance low-bit LLMs on consumer hardware.
Takeaways & Limitations
The specific effects of extreme compression on safety alignment remain an important area for future investigation.

Abstract

from arXiv · show

Efficient deployment of large language models (LLMs) requires extreme quantization, forcing a critical trade-off between low-bit efficiency and performance. Residual binarization enables hardware-friendly, matmul-free inference by stacking binary ($\pm$1) layers, but is plagued by pathological feature co-adaptation. We identify a key failure mode, which we term inter-path adaptation: during quantization-aware training (QAT), parallel residual binary paths learn redundant features, degrading the error-compensation structure and limiting the expressive capacity of the model. While prior work relies on heuristic workarounds (e.g., path freezing) that constrain the solution space, we propose RaBiT, a novel quantization framework that resolves co-adaptation by algorithmically enforcing a residual hierarchy. Its core mechanism sequentially derives each binary path from a single shared full-precision weight, which ensures that every path corrects the error of the preceding one. This process is stabilized by a robust initialization that prioritizes functional preservation over mere weight approximation. RaBiT redefines the 2-bit accuracy-efficiency frontier: it achieves state-of-the-art performance, rivals even hardware-intensive Vector Quantization (VQ) methods, and delivers a $4.49\times$ inference speed-up over full-precision models on an RTX 4090. Code is available at https://github.com/SamsungLabs/RaBiT.

1. Introduction

At 2-bit quantization, residual binarization offers matmul-free efficiency but suffers from training instability and redundant path features. RaBiT enforces a residual hierarchy from one shared full-precision weight, achieving strong accuracy and efficiency.

Residual binarization preserves matmul-free efficiency but struggles to preserve performance because of training instabilities.
Inter-path adaptation makes parallel residual paths learn redundant features, breaking error compensation and limiting expressive power.
RaBiT replaces heuristic path constraints with a shared full-precision weight that sequentially derives binary paths to correct predecessor errors.
4.49× inference speed-up and halved training memory footprint accompany RaBiT’s state-of-the-art accuracy.

2. Related Works

Prior work addresses low-bit efficiency and residual-path co-adaptation through varying compromises. RaBiT instead structurally enforces error-compensating joint optimization.

PTQ methods successful at 3–4 bits degrade substantially at 2 bits because coarse weight approximation cannot mitigate information loss.
Residual binarization stacks binary paths to increase effective precision while retaining matmul-free efficiency.
Inter-path adaptation occurs when shared QAT gradients drive parallel paths toward redundant features, compromising the error-compensation hierarchy.
RaBiT resolves this issue structurally, enabling joint optimization while algorithmically enforcing the residual hierarchy.

3. Motivation

The motivation analysis distinguishes useful residual alignment from superficial path anti-correlation. Effective residual binarization requires later paths to represent remaining functional error and reduce total MSE.

QAT trains a quantized student to functionally mimic a full-precision teacher using task and intermediate distillation losses.
Strongly negative path correlation can reduce MSE by turning the interaction term into an error-cancellation bonus.
The second path must positively align with the first path’s residual, because negative path-to-path correlation alone can be mechanically induced.
The Llama2-7B layer analysis evaluates path correlation, residual alignment, and their contributions to total MSE.
RaBiT reaches 0.65 residual alignment in Layer 5, compared with 0.26 for DB-LLM, and converts interaction into a loss-reducing bonus.

4. Method

RaBiT combines residual binarization with coupled training and function-aware initialization. Dynamic path derivation preserves error correction while parallel binary execution maintains efficiency.

4.1. The Residual Binarization Architecture: Dual-scale binary blocks use binary cores with per-channel scaling vectors, enabling matmul-free computation through additions and subtractions.
4.1. The Residual Binarization Architecture: Stacking k = 2 binary paths increases representational capacity while preserving matmul-free execution, with approximately 2.02 effective bits after scale storage.
4.2. Coupled Training for Co-Adaptation Mitigation: RaBiT uses one shared full-precision weight instead of independent latent weights to prevent inter-path co-adaptation.
4.2. Coupled Training for Co-Adaptation Mitigation: During each forward pass, B1 is derived from WFP, the first-path residual is computed, and B2 is derived from that residual.
4.2. Coupled Training for Co-Adaptation Mitigation: Learnable scales remain independent while binary cores are dynamically recomputed, balancing path-specific magnitudes with training stability.
4.2. Coupled Training for Co-Adaptation Mitigation: Inference freezes the final binary cores, executes paths in parallel without matrix multiplications, and reduces optimizer-state memory usage by 50%.
4.3. Stable Initialization for Functional Preservation: Function-aware initialization uses iterative residual SVID and input/output channel preconditioning to preserve function and stabilize QAT.
4.3. Stable Initialization for Functional Preservation: The initialization approach reduces initial task loss and stabilizes the QAT startup phase.

5. Experiments

RaBiT is evaluated across model families, datasets, quantization baselines, ablations, training dynamics, and inference throughput. Results show competitive or state-of-the-art accuracy, improved optimization behavior, and substantial speed and memory advantages.

Evaluation Setup: RaBiT is evaluated on Llama2, Llama3, and Gemma3 models using 200 million calibration tokens from WikiText-2 and C4.Primary assessment uses validation perplexity with context length 4096, alongside zero-shot reasoning benchmarks.
Evaluation Setup: RaBiT is compared against post-training, Vector Quantization, and hardware-efficient binary or residual 2- to 3-bit methods.The baselines include GPTQ, EfficientQAT, AQLM, QuIP#, QTIP, BitStack, DB-LLM, MBOK, and DBF.
Main Results: RaBiT achieves 5.78 WikiText-2 perplexity on Llama2-7B, outperforming MBOK at 6.99 PPL and DBF at 6.10 PPL.On Llama3-8B, BitStack degrades to 2.75e3 PPL while RaBiT maintains robust convergence and high fidelity.
Main Results: RaBiT reaches 61.51% average zero-shot accuracy on Llama2-7B, surpassing QTIP’s 58.97% while retaining a matmul-free architecture.Its 5.78 PPL also edges out QTIP’s 5.86 PPL on Llama2-7B.
Main Results: RaBiT generalizes across architectures, achieving 66.66% average zero-shot accuracy on Qwen3-4B versus 66.01% for DBF and 66.13% for QTIP.On Qwen3-4B, it also improves Wiki2 perplexity to 8.27 versus 8.76 for QTIP; on Llama3-8B, it reports 7.34 PPL versus 8.70 for QuIP#.
Ablation and Training Dynamics: Coupled QAT reduces Llama2-7B perplexity from 6.55 under Standard QAT to 5.84, while initialization methods further improve RaBiT to 5.78 PPL.RaBiT maintains negative inter-path correlation and the lowest, most stable training loss, unlike Standard QAT and constrained variants.
Inference Efficiency: 2-bit RaBiT delivers a 4.49× end-to-end decoding speed-up over FP16 on an NVIDIA RTX 4090.The reported efficiency is attributed to the 8× model-size reduction, matmul-free binary computation, and avoidance of lookup tables or rotations.

6. Conclusion

RaBiT addresses inter-path adaptation in 2-bit LLM quantization through residual coupling and function-aware initialization, achieving state-of-the-art performance against binary and VQ methods.

RaBiT uses on-the-fly residual coupling to prevent structural breakdown during extreme 2-bit quantization.The framework trains each path against the residual left by its predecessor and adds function-aware initialization for stable convergence.
RaBiT achieves state-of-the-art performance at 2-bit precision, outperforming existing binary and hardware-intensive Vector Quantization approaches.
RaBiT offers a viable pathway for efficient deployment of high-performance low-bit LLMs and a scalable foundation for future research.

Impact Statement

RaBiT reduces computational requirements for 2-bit LLM deployment and can support local execution, while extreme compression leaves safety-alignment effects requiring further investigation.

RaBiT’s efficiency enables high-performance models to run on consumer hardware such as an RTX 4090.The paper connects this accessibility with local deployment, privacy, and data sovereignty benefits compared with cloud-based solutions.
Extreme compression may introduce dual-use risks by enabling malicious applications outside controlled environments.
The specific effects of extreme compression on safety alignment remain an important area for future investigation.

A. Mathematical Analysis of Training Dynamics

Standard QAT can make residual binary paths redundant because a shared gradient pushes them in the same direction, whereas RaBiT structurally trains later paths to correct earlier residual errors without freezing parameters.

Standard QAT: Standard QAT applies the same global gradient to both paths, giving them a persistent tendency to become positively correlated and redundant.The shared update ultimately breaks the intended residual hierarchy and compromises expressive capacity.
RaBiT: RaBiT derives the second binary path from the residual R1 = WFP − ˆ W1, making residual error correction its optimization target.A shared full-precision blueprint replaces independent latent weights and defines the second path on the fly.
RaBiT: Residual alignment can make the second path anti-correlated with the first, implementing active cancellation instead of redundancy.Overshoot by the first binary approximation can cause the residual to point in the opposite direction.
Optimization trade-off: Iterative training avoids adaptation by freezing one path, but its axis-aligned update is 45° misaligned with the optimal joint descent direction.This fixed misalignment produces an inefficient zig-zag optimization trajectory.
Optimization trade-off: RaBiT updates one shared weight with the full gradient, allowing coordinated co-adaptation without restricting optimization to an inefficient path.The coupled mechanism resolves adaptation while preserving full parameter co-adaptation and optimization efficiency.
Output-level analysis: For output analysis, the covariance between paths is approximated by alignment with the teacher output minus the first path’s expected squared norm.Because binarization can overshoot in magnitude, the second term may dominate, driving covariance and correlation toward negative values.

B. Extended Analysis: Inter-Path Adaptation under KL Divergence

Under KL divergence, RaBiT’s residual coupling enforces the negative path correlation needed for error correction, whereas Standard QAT produces redundancy and leaves optimization bias unchecked.

KL-divergence analysis: KL divergence can be locally approximated by a Hessian-weighted mean squared error around the teacher output.The student output is modeled as the sum of binary paths, y_s = h1 + h2.
KL-divergence analysis: Minimizing the local objective requires maximizing the negative magnitude of Hessian-weighted path correlation.The interaction term is defined using a generalized cosine similarity in the inner-product space induced by the Hessian.
Failure of Standard QAT: Standard QAT makes parallel paths redundant, producing positive or near-zero path correlation and leaving the destabilizing bias term unchecked.Both paths are updated independently with the same gradient signal, so the second path fails to exploit the interaction bonus.
RaBiT residual coupling: RaBiT derives the second binary path as sign(WFP − Ŵ1), forcing it to be anti-parallel to the first path’s error vector.This residual coupling structurally enforces the intended error-correction hierarchy.
RaBiT residual coupling: The coupled construction drives path correlation toward −1 and locally cancels the KL bias term.RaBiT therefore supplies the negative correlation that Standard QAT fails to learn.

C. Initialization Analysis: Functionality vs. Approximation

RaBiT’s initialization evaluates quantization by both reconstruction error and initial functional loss, showing that function-aware scaling can sacrifice weight approximation while substantially improving functionality.

Evaluation setup: Initialization quality is evaluated using Avg. MAE, Avg. MSE, and initial Knowledge Distillation loss on Llama2-7B’s first q_proj layer.The analysis compares weight reconstruction with task-relevant functionality before QAT begins.
Function-aware scaling: Adding I/O Channel Importance Scaling increases Avg. MSE from 0.122 to 0.302 but reduces KL divergence loss from 13,760 to 2,672, an 81% reduction.The method allocates limited 2-bit capacity toward channels identified as functionally important.
Function-aware scaling: The initialization analysis supports prioritizing functional preservation over minimizing weight reconstruction error under extreme quantization.The reported trade-off is visually illustrated through the weight and difference matrices in Figure 3.

D. Extended Results

Extended experiments cover reasoning accuracy, GPU implementation, throughput, training configuration, scaling calibration, and initialization convergence, with RaBiT maintaining strong efficiency and stable optimization choices.

Zero-shot reasoning accuracy: Tables 9 and 10 detail zero-shot reasoning accuracy for Llama and Gemma models across five common benchmarks.These tables complement the average scores reported in the main text.
GPU implementation: RaBiT’s CUDA implementation uses bit-packed binary GEMV, register-level staging, and a matmul-free pipeline tailored to memory-bound decoding.Weight packing maps groups of 32 columns into uint32_t representations and reduces raw weight footprint by 32×.
End-to-end throughput: Efficient packing and pipelining shift the decoding bottleneck from global memory bandwidth toward register throughput.The implementation reports measurable efficiency gains during LLM decoding.
End-to-end throughput: RaBiT achieves nearly twice the decoding throughput of other 2-bit methods across generated lengths from 64 to 1024 tokens.Its parallel binary paths avoid the sequential bottleneck of DBF, while the relative performance gap remains consistent for longer generations.
Training configuration: RaBiT training uses six epochs, Muon, cosine learning-rate decay, function-aware initialization, and Tmax = 20 SVID iterations.Learning rate and scaling intensities are tuned separately for each model family.
Scaling calibration: The Llama2-7B scaling grid search reaches minimum initial KL divergence loss of 2,672 at αin = 0.80 and αout = 0.65.The search uses 128 randomly selected calibration samples and balances input activation with output gradient statistics.
Initialization convergence: Initialization loss stabilizes around 15 iterations and reaches its optimum at 20, after which further iterations provide no additional benefit.RaBiT therefore selects Tmax = 20 as its default initialization setting.

F.4. Robustness to Random Seed and Optimizers

RaBiT remains stable across random seeds and produces identical reported perplexity with AdamW and Muon, indicating robustness to these experimental choices.

Random-seed stability: Five-seed experiments show negligible standard deviation in RaBiT’s perplexity and zero-shot question-answering accuracy.The reported results are described as robust across random seeds.
Optimizer sensitivity: AdamW and Muon both produce a perplexity of 5.78 for RaBiT on Llama2-7B.The matching result is reported as statistically identical across optimizers.

G. Extended Analysis of Inter-Path Adaptation

RaBiT improves residual binarization by producing negative covariance that reduces layer-wise MSE and suppressing the high error peak in sensitive early layers. Its initialization and sequential residual coupling are designed to preserve function while enforcing error-correction across binary paths.

Layer-wise MSE decomposition: RaBiT consistently generates substantial negative covariance across most layers, reducing total MSE relative to Standard QAT.The covariance term acts as a loss-reducing bonus through effective error cancellation.
Layer-wise MSE decomposition: The results provide visual evidence that RaBiT’s coupled training enforces the intended error-correction hierarchy, unlike Standard QAT.Standard QAT exhibits much smaller covariance and suffers from inter-path adaptation.
Layer-wise MSE decomposition: RaBiT dramatically suppresses the exceptionally high MSE peak observed in Standard QAT’s initial layers.The analysis identifies layer sensitivity as a critical optimization challenge in extreme quantization.
RaBiT training: During training, RaBiT sequentially derives each binary path from a shared full-precision weight, accumulates effective weights, and updates the residual after every path.The forward pass computes binary paths from successive residuals before producing the layer output and loss.
RaBiT initialization: RaBiT initialization uses channel-importance-calibrated preconditioning before iterative residual SVID derives scales for multiple binary paths.The procedure normalizes input and output calibration statistics, preconditions the shared full-precision weight, and maps scales back to the original domain.