Source-linked AI summary
Sparse-BitNet: 1.58-bit LLMs are Naturally Friendly to Semi-Structured Sparsity
Di Zhang, Xun Wu, Shaohan Huang, Yudong Wang, Hanyong Shao, Yingbo Hao, Zewen Chi, Li Dong, Ting Song, Yan Xia, Zhifang Sui, Furu Wei
TL;DR
The paper studies the largely unexplored interaction between 1.58-bit quantization and semi-structured N:M sparsity in LLMs. It introduces Sparse-BitNet, which jointly applies ternary quantization and dynamic sparsification with stable training, and reports lower degradation than full-precision baselines alongside speedups up to 1.30×. The findings support combining extreme low-bit quantization with structured sparsity for efficient LLMs.
Problem
The interaction between extremely low-bit quantization and semi-structured N:M sparsity remains largely unexplored despite both being studied for LLM efficiency.
Method
Sparse-BitNet jointly enforces 1.58-bit BitNet weight quantization and dynamic N:M semi-structured sparsity while ensuring stable LLM training.
Results
1.30× maximum speedups are achieved in both training and inference, while 1.58-bit BitNet shows smaller accuracy degradation than full-precision models under identical sparsity constraints.
Takeaways & Limitations
Combining extreme quantization with semi-structured sparsity is a viable direction for efficient LLM deployment.
Takeaways & Limitations
The comparison keeps architecture, data mixture, token budget, optimizer, and learning-rate schedule fixed across BF16 and BitNet variants.
Abstract
from arXiv · showhide
Semi-structured N:M sparsity and low-bit quantization (e.g., 1.58-bit BitNet) are two promising approaches for improving the efficiency of large language models (LLMs), yet they have largely been studied in isolation. In this work, we investigate their interaction and show that 1.58-bit BitNet is naturally more compatible with N:M sparsity than full-precision models. To study this effect, we propose Sparse-BitNet, a unified framework that jointly applies 1.58-bit quantization and dynamic N:M sparsification while ensuring stable training for the first time. Across multiple model scales and training regimes (sparse pretraining and dense-to-sparse schedules), 1.58-bit BitNet consistently exhibits smaller performance degradation than full-precision baselines at the same sparsity levels and can tolerate higher structured sparsity before accuracy collapse. Moreover, using our custom sparse tensor core, Sparse-BitNet achieves substantial speedups in both training and inference, reaching up to 1.30X. These results highlight that combining extremely low-bit quantization with semi-structured N:M sparsity is a promising direction for efficient LLMs. Code available at https://github.com/AAzdi/Sparse-BitNet
1 Introduction
LLM scaling makes efficiency essential, motivating quantization and semi-structured sparsity. This work shows that 1.58-bit BitNet is more compatible with N:M sparsity than full-precision models and introduces Sparse-BitNet to combine them.
- LLM scale increases training and inference costs, making efficiency a central research challenge.
- Approximately 42.3% of pretrained 1.58-bit BitNet’s quantized states are zero, revealing a naturally sparse representation without explicit pruning.
- Existing research largely studies N:M sparsity in full-precision models and low-bit quantization separately, leaving their interaction underexplored.
- Sparse-BitNet jointly enforces 1.58-bit quantization and N:M sparsity while ensuring stable training.
- Across Qwen-2.5 scales from 0.5B to 3B and two training settings, BitNet consistently degrades less than BF16 and reaches 1.30× speedups in training and inference.The settings are sparse pretraining and dense-to-sparse schedules.
- 1.58-bit BitNet is inherently more compatible with semi-structured N:M sparsity than BF16, showing smaller accuracy degradation under identical constraints.
2 Sparse-BitNet
Sparse-BitNet integrates ternary quantization with dynamic semi-structured N:M sparsity in a jointly trained Sparse-BitLinear architecture. Its training strategy recomputes masks per step and uses dual straight-through estimation to maintain adaptable sparse topologies.
- N:M Sparsity: N:M sparsity keeps at most N non-zero weights in each group of M consecutive weights, supporting hardware acceleration while retaining fine-grained structure.The section focuses on 6:8 sparsity and benchmarks the standard 2:4 pattern.
- Sparse-BitLinear Architecture: Sparse-BitNet replaces standard linear projections with Sparse-BitLinear layers that jointly apply ternary quantization and N:M masking during training.High-precision master weights are retained for optimization while effective weights are sparse and ternary.
- Mask Generation: Magnitude pruning selects the N largest absolute pre-quantized weights within each M-sized group to construct the sparsity mask.Using pre-quantized weights preserves fine-grained magnitude rankings and avoids ties among ternary values.
- Quant-and-Mask Computation: The forward pass quantizes activations and master weights, then applies the N:M mask to the quantized weights before computing the output.This ordering enforces the N:M pattern on the discrete weights used for inference and provides a defined hardware layout.
- Training Strategy: Dynamic mask recomputation evaluates the N:M mask at every training step, allowing network topology to evolve with the master weights.The procedure is described as projected optimization that prevents mask staleness.
- Training Strategy: Dual straight-through estimation passes gradients through both ternary quantization and mask selection, including to weights masked during the forward pass.Dense gradient updates allow pruned weights to receive feedback and potentially re-enter the Top-N set.
3 Experiments
Experiments evaluate Sparse-BitNet across model scales, sparsity settings, efficiency benchmarks, and training-design ablations. The results show greater robustness of ternary BitNet to structured sparsity and identify training choices that preserve convergence and mask exploration.
- Experimental setup: Qwen2.5 models at 0.5B, 1.5B, and 3B use a common 6:8 sparsity pattern, retaining 6 of every 8 weights.Models are trained from scratch under matched data, token budgets, optimization, and learning-rate schedules.
- Main results: Sparse-BitNet consistently incurs smaller perplexity degradation than BF16 under identical 6:8 constraints across all evaluated model scales.Degradation is measured relative to each method’s own dense baseline.
- Main results: At 2:4 sparsity, BF16 incurs a +18.8% normalized PPL increase, whereas BitNet increases by only +5.7%.Using a 10% degradation threshold, BF16 crosses it at 4:8 while BitNet crosses it at 3:8.
- Training design: Stable training requires updating masked master weights, constructing masks from continuous weights, and applying quantization before masking in the forward pass.These choices support mask exploration, avoid tie-driven selection instability, and produce sparse discrete inference weights.
- Training design: Delaying the switch to sparsity worsens convergence: PPL reaches 27.48 or 27.39 when only the final 25% or 50% of training is sparse, versus 26.31 for sparse-from-scratch.Using 75% sparse steps improves PPL to 26.71 but remains worse than training sparsely throughout.
- Training design: Masking from continuous master weights yields validation PPL 26.31, while masking from quantized weights yields 32.23.The reported gap is attributed to reliable continuous magnitude ranking versus tie effects in ternary weights.
- Mechanistic analysis: BitNet’s dense training dynamics become polarized, with decreasing near-zero mass and a higher-magnitude active mode that separates candidate weights from lower-magnitude thresholds.BF16 instead maintains a unimodal concentration near zero, coupling pruning boundaries to the main weight population.
4 Related Works
BitNet and sparsity are presented as separate efficiency strategies, while Sparse-BitNet combines them by introducing sparsity dynamically during training. This enables the method to target both inference deployment and training efficiency.
- Quantization and BitNet: BitNet b1.58 uses ternary {−1, 0, 1} weights, adding zeros while achieving performance comparable to full-precision models.
- Model Pruning: N:M sparsity imposes fine-grained hardware-oriented constraints, unlike unstructured pruning, which often lacks direct hardware acceleration.
- Model Pruning: Structured pruning removes coherent architectural components and typically causes significant accuracy degradation, requiring expensive retraining.
- Model Pruning: Sparse-BitNet introduces sparsity dynamically during training rather than applying it only as a post-hoc optimization to pretrained dense models.
5 Conclusion
Sparse-BitNet combines ternary quantization with dynamic 6:8 masking and is reported to be more robust to semi-structured sparsity than BF16 models. Its ablations identify dense-master-weight masks and gradient flow through masked regions as important, while custom kernels reach 1.30× inference speedups.
- Sparse-BitNet is reported as more robust to semi-structured sparsity than BF16 counterparts across varying scales.
- Dynamic 6:8 masking combined with ternary quantization reduces performance degradation and delays model collapse.
- Dense master weights for mask computation and gradient flow through masked regions are identified as critical optimization choices.
- 1.30× inference speedups are demonstrated by custom 6:8 kernels.
A.1 Training hyperparameters
The appendix points readers to Table 5 for the training hyperparameters used across the experiments.
- Table 5 summarizes the full training hyperparameters used for all runs.
- The supplied appendix passages provide no individual hyperparameter values.
- The table is identified as covering training hyperparameters for all experiments.
A.2 Baselines and fairness controls
The experiments control training conditions across variants and use matched structured-sparse baseline implementations. Architecture, data, optimization, and evaluation settings are held fixed while the studied sparsity and quantization components vary.
- All variants use identical tokens, data mixture, optimizer settings, architecture, initialization, and evaluation protocol.Only the sparsity and quantization components are changed.
- Structured sparse baselines use a custom kernel with the same 6:8 pattern and weight layout as the main results.
B Pseudo torch-style implementation of Sparse-BitLinear.
Algorithm 2 outlines Sparse-BitLinear by masking before quantization, then applying ternary scaling, while an STE preserves dense gradient flow during backpropagation.
- Sparse-BitLinear applies pre-quantization masking followed by ternary scaling.
- The WeightQuantMasked autograd function implements the straight-through estimator during backpropagation.
- Dense gradient flow helps mitigate sparsity-induced degradation.
C Raw PPL for sparsity sweep
The section reports raw validation perplexity statistics for an N:8 sparsity sweep on Qwen2.5-0.5B, summarized in Table 6.
- Table 6 summarizes complete validation perplexity statistics for N:8 sparsity training.
- The reported values are raw validation PPL results.
- The N:8 sparsity sweep is conducted on Qwen2.5-0.5B.