Source-linked AI summary
Fairy2i: Training Complex LLMs from Real LLMs with All Parameters in $\{\pm 1, \pm i\}$
Feiyu Wang, Xinyu Tan, Bokai Huang, Yihao Zhang, Guoan Wang, Peizhuang Cong, Tong Yang
TL;DR
Extremely low-bit quantization can strain accuracy, and complex-valued LLM methods have required impractical from-scratch training rather than reusing pretrained real-valued checkpoints. Fairy2i converts real layers into an equivalent widely-linear complex form and applies low-bit quantization with residual correction. On LLaMA-2 7B, it outperforms real-valued binary and ternary methods and restores performance to near-FP16 levels at effective 2-bit precision.
Problem
Extremely low-bit quantization can suffer severe accuracy drops, while complex-valued LLM methods requiring from-scratch training are impractical for most practitioners at modern model scales.
Source: §1 · p. 1, §2.2 · p. 4Method
Results
On LLaMA-2 7B, Fairy2i outperforms state-of-the-art real-valued binary and ternary methods, restoring performance to near-FP16 levels at effective 2-bit precision.
Source: §5 · p. 12Takeaways & Limitations
Fairy2i bridges complex-valued low-bit representation with practical reuse of pretrained real-valued LLMs.
Source: §5 · p. 12The current implementation was limited to 30 billion tokens due to computational constraints.
Source: §5 · p. 12
Abstract
from arXiv · showhide
Large language models (LLMs) have revolutionized artificial intelligence, yet their massive memory and computational demands necessitate aggressive quantization, increasingly pushing representations toward the theoretical limit of a single bit. While complex-valued LLMs, such as iFairy, offer a superior chance for low-bit representation compared to real-valued counterparts, they require training from scratch, preventing the utilization of the vast ecosystem of pre-trained real-valued foundation models. Here we present Fairy2i, a universal framework that transforms pre-trained real-valued layers into an equivalent widely-linear complex form, enabling extremely low-bit quantization while reusing existing checkpoints. By proving a lossless mathematical equivalence between real and widely-linear maps, we convert standard Transformers into the complex domain and employ a phase-aware quantization scheme with a highly efficient codebook of fourth roots of unity. Furthermore, we introduce a recursive residual quantization mechanism that iteratively minimizes quantization error, allowing inference to proceed via efficient multiplication-free accumulation. We demonstrate that Fairy2i restores the performance of LLaMA-2 7B at an effective 2-bit precision to levels nearly comparable with full-precision baselines, significantly outperforming state-of-the-art real-valued binary and ternary quantization methods. This work bridges the gap between the representational efficiency of complex-valued arithmetic and the practical utility of pre-trained models, paving a new way for efficient inference on commodity hardware. We open-source the Fairy2i model and code at https://huggingface.co/PKU-DS-LAB/Fairy2i-W2 and https://github.com/PKULab1806/Fairy2i-W2.
1 Introduction
Extremely low-bit quantization strains model accuracy, while complex-valued methods face a barrier to using pretrained real-valued checkpoints. Fairy2i converts real layers into an equivalent complex form, then applies low-bit quantization and residual correction.
1 Introduction
- At extremely low bit widths, PTQ typically loses accuracy, while scratch QAT can be prohibitively time-consuming and unstable.The introduction links these limits to restricted codebooks and, for PTQ, factors such as limited calibration data and outlier channels.Source: §1 · p. 1
- Fairy2i re-parameterizes real-valued layers as equivalent widely-linear complex maps, preserving forward computation and allowing pretrained checkpoints to be reused.The transformation applies to Transformer projections, including attention and feed-forward layers.Source: §1 · p. 2
- Fairy2i continues low-bit QAT from real-valued checkpoints using a phase-aware complex quantizer.Its compact codebook is {±1, ±i}.Source: §1 · p. 2
- Recursive residual quantization adds successive low-bit corrections to reduce quantization error at a small extra bit cost.The introduction specifies one more 1–2 bits per weight plus small per-group metadata.Source: §1 · p. 2
- The authors apply Fairy2i to off-the-shelf open-source LLMs, including the LLaMA series, and report that evaluation validates the framework’s efficiency.The introduction does not provide benchmark values here.Source: §1 · p. 2
2 Related Work
Prior work explores low-bit PTQ and QAT, as well as complex-valued neural networks, but reports limits in accuracy, training cost, and compatibility with pretrained real-valued models. Fairy2i addresses the compatibility gap through an equivalent widely-linear representation.
2.1 Quantization for Large Language Models
- At 2 bits or below, real-valued PTQ typically degrades because its limited codebook cannot adequately approximate heavy-tailed LLM weights without adapting network parameters.PTQ approaches have pushed toward 2 bits, following earlier results at 3–4 bits.Source: §2.1 · p. 3
- Extremely low-bit QAT adapts parameters to quantization noise, but scratch training is costly and real-valued binary or ternary topologies can destabilize adaptation to pretrained weights.The passage notes that BitNet and BitNet b1.58 retain performance when trained from scratch, while some distillation work targets full-precision models.Source: §2.1 · p. 3
2.2 Complex-Valued Neural Networks
- Complex-valued neural networks have a history in signal processing and imaging, but their application to NLP and LLMs remains scarce.The passage attributes their theoretical potential to representing weights and activations in the complex domain.Source: §2.2 · p. 3
- iFairy [22] uses the {±1, ±i} codebook and demonstrates complex-valued Transformer performance gains over real-valued binary networks, but requires training from scratch.That requirement is impractical for most practitioners given the scale of modern LLMs.Source: §2.2 · pp. 3–4
- Fairy2i bridges the gap with a widely-linear representation that mathematically equates a real-valued layer to a complex-valued one, enabling the {±1, ±i} codebook and direct initialization from powerful real-valued checkpoints.Source: §2.2 · p. 4
3 Methods
Fairy2i preserves a pretrained real model’s behavior by reparameterizing its linear layers as widely-linear complex maps, then applies low-bit complex quantization and recursively quantizes residual error. Its design combines compact storage with parallel, multiplication-free inference.
3.1 Widely-linear transformation from real domain to complex domain
- The real-to-complex transformation is exact and unique, so the network computes the same outputs before quantization; Hermitian attention scores also match the original real dot products.Odd real dimensions can be padded with a zero channel and the padded output discarded.Source: §3 · p. 4, §3.1 · pp. 5–6
3.3 Recursive residual quantization
- Recursive residual quantization repeatedly projects the remaining error to the same codebook and adds each correction, producing a finite sum of low-bit terms.The procedure applies to both U and W; T=1 is basic PhaseQuant, while T=2 adds one extra term.Source: §3 · p. 4, §3.3 · pp. 7–8
3.2 Complex-valued low-bit quantization
- PhaseQuant deterministically projects each complex weight to the nearest codeword by angle and estimates two axis-wise scaling factors from the projected axes to recover its magnitude.The paper uses per-tensor scaling factors and leaves investigation of group strategies for future work. In large-scale Transformer layers, matrix multiplication dominates; using 3 real multiplications and 5 additions reduces the count from 4 to 3 real multiplications per element, giving a theoretical 25% reduction in GEMM FLOPs for both forward and backward passes.Source: §3.2 · pp. 6–7
3.4 Storage Efficiency and Parallel Multiplication-Free Inference
- At T=1, Fairy2i uses 1 bit per real parameter on average; storage scales linearly with T, so T=2 uses 2 bits per real parameter.Discrete weights reduce matrix-vector products to conditional additions, subtractions, skips, and real/imaginary swaps; stages can run in parallel, making critical-path latency effectively O(1) in T. CPU LUTs can pack four 2-bit complex weights into an 8-bit index.Source: §3.4 · pp. 8–9
4 Evaluation
Fairy2i is evaluated on LLaMA-2 7B for perplexity and zero-shot reasoning, then examined across recursive quantization stages and learning-rate schedules. The results compare its low-bit configurations with full-precision and quantization baselines.
4.1 Experimental Setup
- Experiments use LLaMA-2 7B; Fairy2i continues QAT pre-training on a RedPajama subset for 30 billion tokens and is compared with the original uncompressed FP16 model.Source: §4.1 · pp. 9–10
4.2 Main Results
- 7.85 C4 perplexity for Fairy2i-W2 (2-bit), versus 8.54 for AQLM, 11.01 for QuIP#, 10.61 for 3-bit GPTQ, and 6.63 for FP16.Fairy2i-W1 reaches 11.03, compared with 11.75 for Real-Binary and 11.03 for Real-Ternary (1.58 bit).Source: §4.2 · p. 10, Table 1 · p. 11
- 62.00% average zero-shot accuracy for Fairy2i-W2, compared with 57.28% for AQLM and 64.72% for FP16.Fairy2i-W1 reaches 48.66%, versus 46.21% for Real-Binary and 0.04% below Real-Ternary.Source: §4.2 · p. 10, Table 1 · p. 11
4.3 Ablation Studies
- Increasing recursive stages from 1 to 2 improves C4 perplexity by 20.76% and average zero-shot accuracy by 19.03%; W3 adds only 4.11% and 0.88%, respectively.W3 increases memory footprint by 50% relative to W2, making T=2 a reported accuracy–storage tradeoff.Source: §4.3 · p. 11, Table 2 · p. 11
- Fairy2i-W2 is reported as robust across the tested learning-rate schedules, while scheduled decay enables lower final training loss.The schedules vary decay timing and target learning rates; the figure shows schedules and corresponding loss curves.Source: §4.3 · p. 11, Figure 3 · p. 12, Table 3 · p. 12
5 Conclusion
Fairy2i converts pre-trained real-valued layers into an equivalent widely-linear complex form for low-bit quantization, combining existing checkpoints with complex-valued representations. On LLaMA-2 7B, it reaches near-FP16 performance at effective 2-bit precision, while the current implementation was limited by computational constraints.
5 Conclusion
- At effective 2-bit precision, Fairy2i restores LLaMA-2 7B performance to near-FP16 levels.The conclusion reports this result alongside outperformance of state-of-the-art real-valued binary and ternary methods.Source: §5 · p. 12
- Fairy2i uses an exact widely-linear representation to convert real-valued layers into the complex domain while reusing pre-trained checkpoints.It combines the {±1, ±i} codebook with recursive residual quantization.Source: §5 · p. 12
- The current implementation was limited to 30 billion tokens by computational constraints.Future work includes specialized CUDA kernels and CPU optimization, scaling to larger and multimodal models, and further study of the complex-valued loss landscape.Source: §5 · p. 12