Source-linked AI summary
S2-BNN: Bridging the Gap Between Self-Supervised Real and 1-bit Neural Networks via Guided Distribution Calibration
Zhiqiang Shen, Zechun Liu, Jie Qin, Lei Huang, Kwang-Ting Cheng, Marios Savvides
TL;DR
Self-supervised learning for binary neural networks remains less explored than for real-valued networks, and the paper examines how to train BNNs without human-annotated labels. It guides a binary student with a self-supervised real-valued model’s prediction distribution. The framework improves naïve contrastive learning on ImageNet and is reported as comparable to some supervised BNN methods.
Problem
Self-supervised learning has been studied less for BNNs than for real-valued networks, leaving the mechanisms and training of label-free BNNs underexplored.
Source: §1 · pp. 1–2Method
A self-supervised real-valued branch guides the BNN to mimic the reference model’s final prediction distribution after its projection MLP and softmax.
Source: §1 · p. 2Results
The framework improves naïve contrastive learning by 5.5∼15% on ImageNet and is comparable to some supervised BNN methods in the reported ReActNet results.
Source: §1 · p. 2, §6.3 · p. 7Takeaways & Limitations
In the reported experiments, recovering knowledge from a well-learned real-valued network is more effective and practical for self-supervised BNNs than learning them directly with contrastive methods.
Source: §1 · p. 2Combining contrastive and guided learning may not be optimal because their optimization spaces differ.
Source: §6.3 · p. 8
Abstract
from arXiv · showhide
Previous studies dominantly target at self-supervised learning on real-valued networks and have achieved many promising results. However, on the more challenging binary neural networks (BNNs), this task has not yet been fully explored in the community. In this paper, we focus on this more difficult scenario: learning networks where both weights and activations are binary, meanwhile, without any human annotated labels. We observe that the commonly used contrastive objective is not satisfying on BNNs for competitive accuracy, since the backbone network contains relatively limited capacity and representation ability. Hence instead of directly applying existing self-supervised methods, which cause a severe decline in performance, we present a novel guided learning paradigm from real-valued to distill binary networks on the final prediction distribution, to minimize the loss and obtain desirable accuracy. Our proposed method can boost the simple contrastive learning baseline by an absolute gain of 5.5~15% on BNNs. We further reveal that it is difficult for BNNs to recover the similar predictive distributions as real-valued models when training without labels. Thus, how to calibrate them is key to address the degradation in performance. Extensive experiments are conducted on the large-scale ImageNet and downstream datasets. Our method achieves substantial improvement over the simple contrastive learning baseline, and is even comparable to many mainstream supervised BNN methods. Code is available at https://github.com/szq0214/S2-BNN.
1. Introduction
The paper investigates self-supervised learning for binary neural networks, where conventional contrastive learning may not transfer well from real-valued models. It develops practical training designs and guided distillation from a self-supervised real-valued network, reporting substantial gains over naïve contrastive learning.
1. Introduction
- Self-supervised binary neural networks remain underexplored despite their potential for resource-limited devices, motivating study of their mechanisms and practical training designs.The paper frames whether contrastive learning that works well on real-valued networks is suitable for BNNs as its first question.Source: §1 · p. 1
- The authors explore optimizers, learning-rate schedulers, and data augmentation to establish a base framework that improves naïve contrastive learning by about 5.5%.They note that binary weights and activations produce predictions unlike those of real-valued networks.Source: §1 · pp. 1–2
- Guided distillation adds a self-supervised real-valued branch that trains the BNN to mimic the reference model’s post-MLP, post-softmax predictions.The teacher is self-supervised and its final output classes are agnostic.Source: §1 · p. 2
- Guided learning without contrastive loss adds 5.5% performance, and the paper distinguishes its focus on predictive probabilities from contrastive learning’s focus on latent data patterns.The authors report that recovering knowledge from a well-learned real-valued network is more effective and practical for self-supervised BNNs in their experiments.Source: §1 · p. 2
- The framework improves naïve contrastive learning by 5.5∼15% on ImageNet; Table 1 covers XNOR Net, Bi-Real Net, and ReActNet.The paper also reports downstream transfer effectiveness for its learned models.Source: §1 · p. 2, Table 1 · p. 2
2. Related Work
The related work surveys supervised BNN advances and established self-supervised approaches, while identifying self-supervised BNNs as an unexplored direction. It positions this paper’s real-to-binary distillation as distinct from prior quantization supervision.
Binary Neural Networks
- XNOR Net uses real-valued scaling factors for binary weight kernels, Real-to-binary improves training and activation attention, and ReActNet studies nonlinear BNN activations.The passage describes ReActNet as a strong MobileNet-based baseline with competitive ImageNet performance.Source: §2 · p. 2
Related Work
- Figure 2 compares activation distributions under SGD and Adam against the ±1 bounds; its caption reports that Adam alleviates activation saturation.Source: Figure 2 · p. 3
- Prior BNN advances have mainly targeted supervised learning, while this paper identifies self-supervised BNNs as an unaddressed research direction.It contrasts its self-supervised distillation from a strong teacher with quantization work that supervises intermediate features using valid input samples.Source: §2 · pp. 2–3
Self-supervised Learning
- The paper places contrastive visual representation learning among recent self-supervised approaches, alongside MoCo and SimCLR, and contrasts it with distillation from a strong self-supervised teacher.It notes concurrent work SEED as another use of self-supervised distillation.Source: §2 · p. 3
Related Work
- Figure 3 marks first-layer channelwise weight l1 norms in green against a red reference value of 0.025.Source: Figure 3 · p. 3
3. Optimizer Effects of SSL on BNNs
Activation saturation suppresses gradients in BNNs, while Adam can amplify small gradients and reactivate weights. In the reported linear evaluation comparison, Adam performs best at smaller learning rates, unlike SGD.
Saturation on Activations and Gradients
- When activations exceed ±1, the approximated sign-function derivative suppresses their gradients to zero, leaving more BNN weights inactive.The passage reports this saturation across most layers and its critical effect on gradient magnitudes across channels.Source: §3 · p. 3
Different Optimizer Effects
- Adam’s second-moment adaptation amplifies small historical gradients, helping revive inactive weights and yielding larger first-layer weights than SGD in the reported visualization.The figure’s red reference line marks 0.025, and its green marks represent channelwise l1 norms.Source: §3 · p. 3
Optimizer Effects
- With SGD, accuracy decreases as learning rates shrink, whereas Adam’s accuracy rises dramatically and its best result exceeds SGD’s best.Figure 4 compares the optimizers in linear evaluation while exploring learning-rate configurations.Source: §3 · p. 4, Figure 4 · p. 4
- Adam updates parameters using normalized first- and second-moment estimates, while the passage describes momentum SGD as accumulating its first momentum.Source: §3 · pp. 3–4
4. Data Augmentation Adjustments
The paper reduces two augmentation probabilities relative to MoCo V2, reporting an approximately 1.0% ImageNet improvement with this lighter strategy.
Data Augmentation Adjustments
- On ImageNet, decreasing the probabilities of ColorJitter and GaussianBlur, rather than adding augmentations, yields an additional ∼1.0% improvement; this choice reflects BNNs’ limited ability to distinguish same-class images of different shapes.Source: §4 · p. 4
- The strategy inherits its augmentations from MoCo V2 but lowers the probabilities of ColorJitter and GaussianBlur.The authors apply this lighter augmentation to facilitate classifying two differently shaped images from the same class.Source: §4 · p. 4
- The authors argue that augmentation settings for BNNs require reconsideration because their properties differ from real-valued networks.Source: §4 · p. 4
5. Our Approach
The approach adapts contrastive learning for binary networks, then uses real-valued reference predictions to guide the binary model, with progressive binarization to ease optimization. It also tests guided learning without contrastive loss.
5. Our Approach
- 5.5% over vanilla MoCo V2, ∼4% from adding guided learning, and a further 5.5% after removing contrastive loss describe the three-stage approach.The stages are an enhanced contrastive baseline, contrastive plus guided learning, and guided learning alone.Source: §5 · p. 4
5.1. Preliminaries
- BNNs binarize both weights and activations to {-1, +1}, while training maintains real-valued weights and uses an approximate gradient.The sign function binarizes activations; the gradient approximation can suffer vanishing gradients when activations exceed [-1, 1].Source: §5.1 · p. 4
5.2. Real-Value Guided Distillation
- Guided learning minimizes KL divergence between binary-model outputs and those of a self-supervised real-valued reference model.The method uses the same data augmentation for both models; in practice, it optimizes with cross-entropy, which is equivalent to the KL objective following MEAL [31] [33].Source: §5.2 · p. 5
5.3. Progressive Binarization
- Real-valued activations retain finer details in middle and high-level layers, motivating progressive binarization because direct distribution recovery is challenging.Training starts with a fully real-valued model, then uses a partially binarized stage with real-valued weights and binary activations before the final binary model; the intermediate stage omits or reduces weight decay.Source: §5.3 · pp. 5–6
5.4. What Happens If Removing Contrastive Loss
- The paper compares an enhanced contrastive baseline, contrastive plus guided learning, and guided learning alone; the reference network can be trained online or offline.The experiments use offline pretraining with a MoCo V2 800-epoch pretrained real-valued ResNet-50 teacher, which is reused across experiments.Source: §5.4 · p. 6
- Guided learning alone improves performance by an additional 5.5%, and the authors relate the result to contrastive and guided learning optimizing different directions.They describe guided learning as recovering knowledge from the real-valued representation, while contrastive learning explores patterns such as instance discrimination and colorization.Source: §5.4 · p. 6
6. Experiments
Experiments evaluate the self-supervised BNN framework on ImageNet and transfer benchmarks, with ablations of training choices and architecture. The guided method improves over contrastive learning, while transfer results and visualizations provide further evidence about its performance and learned representations.
6.1. Datasets and Implementation Details
- Experiments use ImageNet 2012 for pretraining and four downstream benchmarks; training generally runs for 200 epochs, followed by linear evaluation or fine-tuning.MoCo V2 is the contrastive baseline, while the distillation setup omits momentum updates, shuffling BN, a memory bank, and contrastive loss.Source: §6.1 · pp. 6–7, §6 · p. 6
6.2. Ablation Studies
- Adam improves results by 2.8%, and a 0.1 learning rate is optimal for Adam-optimized models in linear evaluation.The optimizer comparison is in pretraining; the scheduler comparison varies the linear-evaluation learning rate.Source: §6.2 · p. 7
- ImageNet Top-1/5 results rise from 49.402/73.152 to 50.410/73.968 with lite rather than vanilla MoCo V2 augmentation.The comparison uses the same contrastive loss and Adam optimizer.Source: §6.2 · p. 7
- The proposed multi-step binarization generally improves accuracy, and the method brings substantial improvement across XNOR Net, Bi-Real Net, and ReActNet backbones.The supplied passage gives no numeric values for these ablations.Source: §6.2 · p. 7, Table 3 · p. 7
6.3. Main Results
- 14.6%: With ReActNet on ImageNet, the method outperforms self-supervised MoCo V2 and is comparable to some supervised BNN methods while using about half their OPs.Table 4 reports Top-1 accuracy comparisons; the authors identify Bi-RealNet-18 and CI-BCNN as supervised examples.Source: Table 4 · p. 7, §6.3 · p. 7
- Guided-learning activation maps show more detail than contrastive maps and appear closer to real-valued maps; extending training from 200 to 400 epochs changes accuracy only from 61.5% to 61.8%.The authors attribute the marginal epoch gain to the binary student recovering teacher capability; offline teacher pretraining is the main extra cost, and excluding it, total computation is nearly MoCo V2's.Source: Figure 10 · p. 8, §6.3 · p. 8
- Under conventional self-supervised fine-tuning, transfer results generally match ImageNet linear-evaluation performance; the improvement from y to z is dramatically higher than from x to y across datasets, and the best result is close to the self-supervised real-valued baseline.Source: §6.3 · p. 8
7. Conclusion
The paper studies self-supervised binary networks and proposes guidance from a self-supervised real-valued reference network. It reports better results than contrastive learning and supervised BNN schemes, with learned parameters relevant to downstream tasks.
7. Conclusion
- The paper proposes a guided learning paradigm that uses a real-valued reference network to distill training of a self-supervised binary network.Source: §7 · p. 8
- The authors report better results than contrastive learning and supervised BNN schemes, attributing the method's performance to mimicking the reference network's representation quality.Source: §7 · p. 8
- The learned parameters may support downstream tasks that depend on good representations, including fine-grained recognition.Source: §7 · p. 8