PaperlayerView exact revision on arXiv

Source-linked AI summary

Low-Rank Ternary Adaptation for Fine-Tuning Transformers

Alexandru-Dragos Manolache, Yunqiang Li, Jan van Gemert

arXiv:2608.24469v1cs.CVcs.LG

TL;DR

Ternary transformers are efficient, but existing low-bit adaptation methods either require dequantization or fail to keep merged weights ternary. The paper introduces a low-rank Kronecker-factorized ternary multiplicative adapter for direct keep, zero, and sign-flip updates, and reports improved results across six language and vision models.

  • Problem

    Existing low-bit LoRA-style methods cannot directly adapt ternary weights while keeping the merged model strictly in {−1, 0, 1} without dequantization or requantization.

  • Method

    The method applies a Kronecker product of two small ternary factors multiplicatively to ternary weights, encoding discrete keep, zero, and sign-flip updates.

  • Results

    Across six language and vision models, the method recovers substantial performance lost to ternarization and outperforms requantized QLoRA under the same 1.58-bit quantization level.

  • Takeaways & Limitations

    The adapter provides parameter-efficient fine-tuning with direct merging into a model that remains ternary and requires no dequantization or requantization.

  • Takeaways & Limitations

    Zero-valued base weights remain zero after adaptation, and Kronecker structure may constrain updates for poorly factorable layer shapes.

Abstract

from arXiv · show

Ternary transformers offer extreme memory and compute efficiency, but existing low-bit LoRA-based methods cannot directly fine-tune ternary weights. Current approaches either require dequantization, restoring low-bit base weights to higher precision to merge with adaptation weight, or update only quantization parameters, preventing a merged model that remains ternary. We propose ternary multiplicative adaptation, which represents discrete updates of ternary weights such as sign flips or zeroing through a low-rank Kronecker factorization into two small ternary matrices applied element-wise to ternary weights. This design is parameter-efficient and expressive, preserves the ternary domain, and supports direct merging without dequantization. Experiments on six models across language and vision, including ternarized LLaMA-3 1B and 3B and a ternary ViT-B/16, demonstrate that our method recovers much of the performance lost to quantization and outperforms strong low-bit and ternary baselines. Code is available at https://github.com/alexmanoo/ternary_adaptation.

1 Introduction

Ternary quantization reduces transformer memory and compute costs, but existing low-bit adaptation methods do not directly preserve ternary weights during fine-tuning and merging. The proposed ternary multiplicative adapter uses compact Kronecker-structured ternary updates and improves performance across language and vision models.

  • Motivation: Ternary quantization restricts weights to {−1, 0, 1} and reduces a 16-bit Llama 8B model’s theoretical memory footprint from 16GB to 1.6GB.This corresponds to log2 3 ≈1.58 bits per weight.
  • Limitations of Existing Methods: QLoRA dequantizes base weights before merging, while QA-LoRA avoids dequantization at 2 bits but is incompatible with ternary quantization.Re-quantization after dequantized merging can introduce accuracy degradation, especially below 2 bits.
  • Proposed Method: Ternary multiplicative adaptation represents discrete weight changes as keep, zero, or sign-flip operations applied directly to ternary weights.The update operates element-wise through a ternary mask rather than using continuous additive changes.
  • Proposed Method: The adaptation uses a Kronecker product of two small ternary matrices, combining parameter efficiency with high expressivity while preserving the ternary domain.The merged adapter can be fully merged into the base model without dequantization or requantization.
  • Empirical Evaluation: Across six language and vision models, the method recovers much of the performance lost to quantization and outperforms strong low-bit and ternary baselines.It outperforms requantized QLoRA at the same 1.58-bit level and narrows the gap to full ternary fine-tuning on a ternary ViT-B/16.

2 Related Work

Related work covers transformer quantization, ternary neural networks, parameter-efficient fine-tuning, low-rank decompositions, and quantized-backbone adaptation. The paper positions its method as a native ternary alternative to continuous additive updates that leave the ternary domain.

  • Quantization in Transformers: Quantization reduces memory and computation by representing transformer weights and activations in low-bit formats, with PTQ methods minimizing quantization error through calibration or scaling.Examples include LLM.int8(), GPTQ, AWQ, OmniQuant, SmoothQuant, and SpinQuant.
  • Ternary Neural Networks: Ternary neural networks restrict weights to {−1, 0, 1}, enabling memory and compute reductions by replacing multiplications with sign operations.Prior CNN work used learned scaling factors to improve efficiency while retaining accuracy.
  • PEFT Methods: PEFT methods reduce trainable parameters, while LoRA applies low-rank updates without changing model architecture or input tokens.LoRA merges its low-rank matrices with base weights at inference without increasing inference cost.
  • Low-rank Decomposition: Low-rank decomposition methods trade rank, parameter count, flexibility, and expressivity through simple, SVD-based, or adaptive decompositions.AdaLoRA, SaLoRA, and IncreLoRA dynamically allocate or control rank across layers.
  • PEFT with Quantized Backbones: QLoRA and QA-LoRA adapt quantized backbones, but continuous additive updates or quantization-parameter merging do not ensure a ternary merged model.The proposed method instead applies a Kronecker-factorized multiplicative update directly to ternary weights.

3 Method

The method adapts ternary weights through element-wise multiplicative updates, using ternary Kronecker factors to retain the ternary domain while providing compact, expressive updates. Training uses real-valued proxy factors, while merging produces a deployable ternary layer with no inference overhead.

  • Ternary multiplicative adaptation: The adapted weight W′_tern = W_tern ⊙ Δ_tern remains in {−1, 0, 1}, enabling direct merging without dequantization.The multiplicative update retains, zeros, or sign-flips base weights.
  • Kronecker-structured ternary adaptation: Δ_tern is factorized as A ⊗ B, where smaller ternary matrices satisfy p·r = d_out and q·s = d_in.The Kronecker product constructs an update matching the full weight shape.
  • Kronecker-structured ternary adaptation: The Kronecker product provides high rank because rank(A ⊗ B) = rank(A)·rank(B), while trainable parameters remain below a full weight update.The parameter count is n_params = p·q + r·s, typically smaller than d_out·d_in.
  • Training: Real-valued proxy matrices are projected to ternary factors during the forward pass, with gradients passed through the projection using a straight-through estimator.Thresholds are calculated from each proxy matrix’s absolute mean.
  • Inference and merging: After fine-tuning, W_merged = W_tern ⊙ (A ⊗ B), and the proxy factors can be discarded because the final weights remain ternary.The deployed layer preserves the original layer’s shape, precision, and activations.
  • Efficiency and expressivity: For square layers, the adaptation uses O(d) training memory, whereas matching its expressivity with LoRA requires rank r = d and O(d^2) memory.The method uses 8d Bytes for parameters, 8d Bytes for gradients, and 16d Bytes for the optimizer per layer.

4 Experiments

Experiments evaluate ternary multiplicative adaptation across ternary language and vision models, testing parameter efficiency, downstream adaptation, and performance against quantized baselines. The method improves benchmark results while preserving ternary weights and enabling merged, zero-overhead inference.

  • Experimental setup: The adaptation is applied to self-attention and feed-forward blocks, excluding task-specific output heads.Balanced factorization chooses dimensions satisfying m = p · r and n = q · s under integer divisibility constraints.
  • Parameter efficiency: About 0.06% trainable parameters cover every weight in each Llama-3.2-1B layer through balanced factorization.The factors form balanced submatrices while retaining expressive multiplicative coverage.
  • LLM PTQ backbones: For Llama-3.2-1B, average accuracy rises from 33.2 to 35.3, while PPL improves from 86.5 to 44.6 versus ternary SpinQuant.Performance improves on six of nine benchmarks and is equal or at most 0.8 percentage points worse on the remaining three.
  • LLM PTQ backbones: For Llama-3.2-3B, average accuracy rises from 31.9 to 38.3 and PPL falls from 45.6 to 22.3 versus ternary SpinQuant.The method improves all nine benchmarks and outperforms the 2-bit SpinQuant baseline on every benchmark.
  • Merged deployment: Compared with QLoRA, native ternary adaptation avoids requantization degradation and retains a merged ternary model without additional inference parameters or latency.On Llama-3.2-3B, QLoRA requantization changes average accuracy from 38.3 to 37.5 and PPL from 22.3 to 22.9.
  • Downstream adaptation: Downstream adaptation improves GSM8K exact-match accuracy by +2.9, +3.1, and +1.0 points across three pretrained ternary backbones.The merged models remain strictly ternary without inference overhead.
  • Weight transformations: After adaptation, weights quantized to 0 remain zero, so accuracy gains come from reassigning signs of existing non-zero weights.The multiplicative update cannot reactivate pruned connections.

5 Concluding Remarks

The proposed ternary multiplicative adapter preserves strictly ternary merged models while recovering performance across language and vision models. Its main limitation is that zero weights cannot be reactivated, and Kronecker structure may constrain some layer shapes.

  • Across six language and vision models, the adapter improves over ternary PTQ baselines and requantized QLoRA while narrowing the gap to full ternary fine-tuning.The merged models remain at the 1.58-bit ternary constraint.
  • The adapter applies keep, zero, and sign-flip updates through two small ternary factors combined by a low-rank Kronecker product.This preserves the ternary domain without dequantization or post-hoc requantization.
  • Zero weights cannot be reactivated because ternary weights lack a zero-point to absorb the update.This zero-locking is the trade-off for strict ternary closure.
  • The Kronecker structure may constrain update patterns for poorly factorable layer shapes.

S1 Additional analysis on weight transitions

The adaptations primarily reorganize signs among non-zero ternary weights rather than changing sparsity. Balanced and Normalized initialization produce extensive sign flipping, whereas all-ones initialization mostly preserves the backbone.

  • 99.16% of all weights remain unchanged under all-ones initialization, with 0.84% of non-zero weights pruned to zero and no direct sign flips.
  • Approximately 26% of all weights flip between −1 and 1 under Balanced and Normalized initialization, while less than 1% are pruned to zero.
  • 49.21% of active connections flip under Balanced initialization, compared with 48.85% under Normalized initialization.
  • The adaptations mainly reorganize non-zero weight signs rather than changing sparsity, helping recover accuracy while keeping the final model strictly ternary.

S2 Initialization Ablation

Initialization materially affects adaptation behavior: Balanced initialization gives the strongest reported average accuracy and lowest perplexity on Llama-3.2-1B.

  • Balanced initialization achieves the best average accuracy and lowest perplexity among the three evaluated strategies.

S3 Training Dynamics

On Llama-3.2-1B, the method trains with slightly lower peak VRAM and shorter epoch time than QLoRA, while showing smooth loss convergence and stable gradient norms.

  • 31.0GB peak VRAM and 1.43h per epoch are reported for the method, versus 31.2GB and 1.83h for QLoRA.
  • Figures 2 and 3 show smooth loss convergence and stable gradient norms during training.
Loading 2608.24469v1…