PaperlayerView exact revision on arXiv

Source-linked AI summary

Unsupervised Representation Learning for Binary Networks by Joint Classifier Learning

Dahyun Kim, Jonghyun Choi

arXiv:2110.08851v3cs.LGcs.CV

TL;DR

SSL has succeeded with large floating-point networks, but these models are not readily deployable to edge devices and SSL for binary networks is seldom explored. BURN jointly trains a binary network with an adapting floating-point target and adds losses and training changes; it outperforms SSL baselines across evaluated tasks and sometimes supervised pretraining.

  • Problem

    Self-supervised learning has had limited exploration for binary models, despite the deployment relevance of efficient models in resource-constrained settings.

    Source: §1 · p. 1

  • Method

    BURN jointly trains a binary network with a randomly initialized classifier attached to a pretrained floating-point feature extractor, adding feature similarity loss, dynamic balancing, and modified multi-stage training.

    Source: §1 · p. 2

  • Results

    Across five downstream tasks and seven datasets, BURN consistently outperforms existing SSL baselines by large margins and sometimes supervised pretraining.

    Source: §5 · p. 8

  • Takeaways & Limitations

    BURN’s evaluations support binary-network representation learning across varied downstream tasks, with occasional gains over supervised pretraining.

    Source: §5 · p. 8

    BURN requires a pretrained floating-point feature extractor; a framework that avoids a pretrained floating-point model would be more efficient.

    Source: §5 · p. 8

Abstract

from arXiv · show

Self-supervised learning is a promising unsupervised learning framework that has achieved success with large floating point networks. But such networks are not readily deployable to edge devices. To accelerate deployment of models with the benefit of unsupervised representation learning to such resource limited devices for various downstream tasks, we propose a self-supervised learning method for binary networks that uses a moving target network. In particular, we propose to jointly train a randomly initialized classifier, attached to a pretrained floating point feature extractor, with a binary network. Additionally, we propose a feature similarity loss, a dynamic loss balancing and modified multi-stage training to further improve the accuracy, and call our method BURN. Our empirical validations over five downstream tasks using seven datasets show that BURN outperforms self-supervised baselines for binary networks and sometimes outperforms supervised pretraining. Code is availabe at https://github.com/naver-ai/burn.

1. Introduction

SSL has worked well with large floating-point networks, but quantized and binary models for resource-constrained deployment remain less explored. BURN trains a binary network with an adapting floating-point target and reports strong performance across downstream tasks.

1. Introduction

  • The work addresses limited SSL research on quantized models, despite their relevance to efficient deployment on resource-constrained devices.Binary networks are described as especially efficient in highly resource-constrained settings.Source: §1 · p. 1
  • Across downstream tasks, BURN outperforms SSL baselines by large margins and sometimes supervised pretraining.The evaluations cover multiple tasks and datasets, including detection, linear evaluation, semi-supervised fine-tuning, few-shot learning, and transfer.Source: Figure 1 · p. 1, §1 · p. 2
  • BURN jointly trains a randomly initialized floating-point classifier and binary network, using the classifier’s outputs as targets that adapt during training.The classifier is attached to a fixed pretrained floating-point feature extractor.Source: §1 · p. 2
  • Feature similarity loss, dynamic loss balancing, and modified multi-stage training are proposed to improve accuracy.Feature similarity supplies stable gradients that bypass the randomly initialized classifier; its relative importance declines as the classifier becomes a less-random target.Source: §1 · p. 2

2. Related Work

Related work covers self-supervised representation learning for floating-point models and binary networks, where most prior work focused on supervised training. BURN is positioned against S2-BNN by using a changing floating-point target rather than a fixed one.

2.1. Self-Supervised Representation Learning

  • Self-supervised methods use instance discrimination, feature regression with EMA targets, cluster assignments, or similarity-score distributions, but do not address smaller quantized models.The paper compares with BYOL and SWAV because of their performance and similarities to other SSL methods.Source: §2.1 · p. 2

2.2. Binary Networks

  • Binary-network research includes architecture search, specialized activations, and object detection, but has mostly focused on supervised training.ReActNet learns binarization thresholds; HCEBN uses multiple experts to increase representation capacity without increasing operation count.Source: §2.2 · p. 2
  • S2-BNN applies supervised KL divergence to binary-network SSL with a frozen floating-point target; BURN instead uses a changing target.The paper identifies the fixed target as potentially limiting and notes that SSL research supports changing targets.Source: §2.2 · pp. 2–3

3. Approach

BURN trains a binary network against a moving target built from a pretrained, fixed floating-point feature extractor and a jointly trained classifier. Feature similarity, dynamic loss balancing, and modified multi-stage training address unstable gradients and differing convergence speeds.

3. Approach

  • BURN replaces a fixed floating-point target with a changing target network, jointly training its classifier and the binary network using KL divergence.The target combines a pretrained, fixed floating-point feature extractor with a randomly initialized classifier.Source: §3 · p. 3

3.1. Joint Classifier Training as Moving Targets

  • A randomly initialized classifier provides moving targets for the binary network, but its near-random early outputs can produce noisy gradients.The classifier and binary network are jointly trained so the target can adapt to the current training scenario.Source: §3.1 · p. 3

3.2. Stabilize Gradients by Feature Similarity Across Precision

  • Feature similarity across floating-point and binary features supplements KL loss; with it, gradient magnitudes and surges are reduced during training.The loss uses cosine distance between the feature vectors; the fixed pretrained extractor supplies stable targets while the classifier changes.Source: §3.2 · p. 4, Figure 3 · p. 4

3.3. Dynamic Balancing of λ

  • As the classifier improves, cosine annealing decays the feature-similarity loss weight from λ0 to λTmax, emphasizing it more early and less later.Tmax is the maximum training iteration and t is the current iteration.Source: §3.3 · p. 4

3.4. Modified Multi-Stage Training for BURN

  • Modified multi-stage training initializes the second-stage floating-point classifier with its first-stage weights, helping it converge quickly alongside the binary network.The modification addresses the mismatch between the binary network's faster convergence and the randomly initialized classifier's slower convergence.Source: §3.4 · pp. 4–5

4. Experiments

Experiments compare BURN with SSL and supervised pretraining across downstream tasks, including ImageNet evaluation, Pascal VOC detection, VOC07 SVM classification, and transfer learning.

4.1. Results on Downstream Tasks

  • Across Pascal VOC detection, ImageNet evaluation, VOC07 SVM, and transfer benchmarks, BURN beats SSL baselines, also exceeding supervised pretraining on detection.Source: Table 1 · p. 5, Table 2 · p. 6, Table 3 · p. 6
  • Up to +13.04% top-1 on ImageNet linear evaluation, while fine-tuning gains are at least +3.67% top-1 and +5.30% top-5 (1% labels), and +3.98% top-1 and +4.41% top-5 (10% labels).Source: §4.1 · p. 5, Table 2 · p. 6
  • On VOC07, few-shot SVM gains are roughly +1% to +10% mAP and full-shot gains reach +8.33% mAP over SSL; both settings are near supervised pretraining.Source: §4.1 · p. 6, Table 3 · p. 6
  • BURN wins by large margins on CIFAR10, CUB-200-2011, and Birdsnap; Places205 shows smaller method variation, consistent with the stated object-to-scene domain gap.Source: Table 4 · p. 6, §4.1 · p. 7

4.2. Further Analyses

  • The random-classifier-target baseline reaches 57.33% top-accuracy on ImageNet, and feature similarity, dynamic balancing, and modified multi-stage training each add non-trivial gains.Source: §4.2 · p. 7, Table 5 · p. 7
  • Compared with EMA-target BYOL and tuned MoCov2, jointly trained classifier targets perform better; simulated binary EMA-target distance stays high and does not decrease.Source: §4.2 · pp. 7–8, Figure 5 · p. 7
  • Cosine distance beats L1 and L2; L2 classifier gradients start at 1 × 10^6, while L1 and L2 show potential feature-extractor gradient explosions unlike subdued cosine gradients.Source: §4.2 · p. 8, Figure 6 · p. 8, Table 6 · p. 8
  • Smooth cosine annealing ranks best among balancing choices; without MST, BURN is more accurate than baselines at similar or less time, while with MST it beats S2-BNN at comparable time.Source: §4.2 · pp. 7–8, Figure 4 · p. 7, Table 7 · p. 8

5. Conclusion

BURN jointly trains an FP classifier and binary network, with feature similarity loss, dynamic loss balancing, and modified multi-stage training. Across five downstream tasks and seven datasets, it outperforms SSL baselines and sometimes supervised pretraining; it requires a pretrained FP feature extractor.

5. Conclusion

  • Across five downstream tasks and seven datasets, BURN consistently outperforms existing SSL baselines by large margins and sometimes supervised pretraining.The paper also reports detailed analyses of BURN.Source: §5 · p. 8
  • BURN jointly trains an FP classifier and binary network, adding feature similarity loss, dynamic loss balancing, and modified multi-stage training to improve accuracy.Source: §5 · p. 8
  • BURN requires a pretrained FP feature extractor; the authors identify a framework without a pretrained FP model as a more efficient direction.Source: §5 · p. 8
  • The authors warn that improved binary representations for edge vision could contribute to surveillance systems monitoring populations for private information and personal attributes.They suggest gating code and pretrained models behind security and ethical screenings.Source: §5 · p. 8

A. Details on Downstream Task Configurations and Implementation Details (Section 4)

The appendix describes protocols for five downstream evaluation tasks and the implementation used for the experiments. It follows prior-work protocols where possible and uses ReActNet-A as the binary backbone.

Object Detection

  • Object detection fine-tunes the entire Faster R-CNN framework on Pascal VOC 2007 and 2012, then evaluates on the VOC 2007 test set using the configuration from.Source: A. Details on Downstream Task… · p. 11

Linear Evaluation

  • Linear evaluation freezes the backbone and trains a single-layer softmax classifier for 100 epochs on ImageNet or ImageNet100.The stated learning rate starts at 30 and drops by a factor of 0.1 at epochs 60 and 80; momentum is 0.9 with no weight decay.Source: A. Details on Downstream Task… · p. 11

Semi-Supervised Fine-Tuning

  • Semi-supervised fine-tuning trains the backbone and linear classifier for 20 epochs, selecting the best of nine combinations of backbone and classifier initial learning rates.It uses 1% or 10% of randomly sampled labeled training images, with momentum 0.9 and weight decay 0.0005.Source: A. Details on Downstream Task… · p. 11

SVM Image Classification

  • SVM image classification extracts 4,096-dimensional features from the last ReActNet layer and uses LIBLINEAR; VOC07 trainval and test splits support full-shot and few-shot evaluation.Few-shot mAP is averaged over five samples, with standard deviation reported; k ranges from 1 to 96 shots per class.Source: A. Details on Downstream Task… · p. 11

Transfer Learning

  • Transfer learning uses frozen-backbone linear evaluation on object-centric datasets and Places205, with separate training schedules for the two dataset groups.Object-centric classifiers train for 100 epochs; Places205 classifiers train for 28 epochs.Source: A. Details on Downstream Task… · p. 11

Implementation Details

  • Experiments use ReActNet-A, ImageNet pretraining for 200 epochs, and an ImageNet-pretrained MoCov2 ResNet50 as the FP network.The binary pretraining uses LARS with batch size 2,048; the FP network is pretrained for 800 epochs.Source: A. Details on Downstream Task… · p. 11

B. Additional Comparisons to InfoMin and SimCLRv2 (Section 4)

On ImageNet100 and VOC07 evaluations after ImageNet100 pretraining, BURN exceeds InfoMin and SimCLRv2 across the reported comparisons, while a limited-pretraining-data comparison omits object detection and transfer learning.

Experimental Scope

  • The ImageNet100 comparison omits object detection and transfer learning because the authors believe the pretraining data is insufficient and most methods perform poorly on those tasks.The reported comparisons cover linear evaluation, semi-supervised fine-tuning, and SVM image classification.Source: B. Additional Comparisons to InfoMin… · pp. 11–12

Performance Comparison

  • The authors believe binary-network capacity limits performance on InfoMin and SimCLRv2 instance discrimination, with InfoMin scoring especially poorly across tested tasks.They note a similar issue in lower-capacity FP networks.Source: B. Additional Comparisons to InfoMin… · p. 12

Linear Evaluation

Semi-Supervised Fine-tuning

SVM Image Classification

B.1. Further Investigation on Performance Degradation of InfoMin (Follow-up of Section B)

The analysis tests whether InfoMin's poor ReActNet performance reflects strong augmentation or its momentum encoder and large negative set. Weaker augmentation worsens results, while fewer negatives help the binary backbone but not the FP backbone.

Effect of strong data augmentation

  • Weakening InfoMin's augmentation further degrades ReActNet linear-evaluation performance at epochs 200, 400, and 600, so stronger augmentation may not explain its performance drop.The comparison motivates investigating the momentum encoder.Source: §B.1 · pp. 13–14, Figure 7 · p. 13

Usage of momentum encoder

  • The authors propose that a momentum encoder supplying many negatives may make negative-sample features too different to help binary networks, especially with 65,536 negatives.They vary negative counts to compare ReActNet with ResNet50.Source: §B.1 · p. 14
  • InfoMin with ReActNet performs best with 1/16 as many negatives, whereas ResNet50 performs worse with fewer negatives; VOC07 SVM results show the same trend.The authors caution that this does not fully explain InfoMin's performance drop.Source: Figure 8 · p. 13, §B.1 · p. 14

C. Plots of Various λ(t) (Table 7 in Section 4.2)

Figure 9 visualizes three λ(t) choices compared in Table 7: a constant function, a shifted and horizontally reflected Heaviside step function, and a smooth annealing function.

C. Plots of Various λ(t) (Table 7 in Section 4.2)

  • Figure 9 plots the constant function, shifted and reflected Heaviside step function, and smooth annealing function used as λ(t) choices compared in Table 7.The Heaviside choice is H(−t + Tmax/2), where Tmax is the maximum iterations.Source: C. Plots of Various λ(t) (Table 7 in… · p. 14

Related papers

Loading 2110.08851v3…