PaperlayerView exact revision on arXiv

Source-linked AI summary

Split-Brain Autoencoders: Unsupervised Learning by Cross-Channel Prediction

Richard Zhang, Phillip Isola, Alexei A. Efros

arXiv:1611.09842v3cs.CV

TL;DR

Unsupervised representation learning seeks features that adapt to unknown downstream tasks, but traditional autoencoders have not reliably produced strong transfer representations. Split-brain autoencoders address this by training two disjoint subnetworks on complementary cross-channel predictions, using the full input signal. The method achieves state-of-the-art performance on several semantic representation-learning benchmarks and promising performance on RGB-D images.

  • Problem

    Traditional autoencoders have not been shown to produce strong transfer representations, and their bottleneck creates a trade-off between forced abstraction and retained information.

    Source: §1 · p. 1

  • Method

    Split-brain autoencoders split the network into two disjoint cross-channel encoders that predict complementary subsets of raw input channels and concatenate their representations.

    Source: §1 · p. 2, §3.1 · p. 3

  • Results

    State-of-the-art performance is achieved relative to previous self-supervised methods on Lab images, with promising performance also demonstrated on RGB-D images.

    Source: §5 · p. 8

  • Takeaways & Limitations

    The method uses raw data as supervision and extracts features from the full input while avoiding a representational bottleneck during training.

    Source: §1 · p. 2, §5 · p. 8

    With a fixed architecture size, adding more than two subnetworks makes each subnetwork smaller and less expressive, potentially worsening its original task.

    Source: §5 · p. 8

Abstract

from arXiv · show

We propose split-brain autoencoders, a straightforward modification of the traditional autoencoder architecture, for unsupervised representation learning. The method adds a split to the network, resulting in two disjoint sub-networks. Each sub-network is trained to perform a difficult task -- predicting one subset of the data channels from another. Together, the sub-networks extract features from the entire input signal. By forcing the network to solve cross-channel prediction tasks, we induce a representation within the network which transfers well to other, unseen tasks. This method achieves state-of-the-art performance on several large-scale transfer learning benchmarks.

1. Introduction

The paper addresses weaknesses in reconstruction-based unsupervised representations by using complementary cross-channel prediction tasks. Split-brain autoencoders split the network into disjoint sub-networks whose concatenated features use the full input and transfer strongly.

  • Traditional autoencoders face a bottleneck trade-off: smaller bottlenecks force more abstraction but carry less information, while strong transfer representations remain unestablished in practice.Source: §1 · p. 1
  • Context encoders can underperform because inpainting is difficult to evaluate, creates a training–deployment domain gap, and may be solved without high-level reasoning.Source: §1 · pp. 1–2
  • Colorization is an effective cross-channel pretext task because spatial correspondence supports standard losses, systematic corruption avoids a domain gap, and color prediction may require object-level reasoning.Source: §1 · p. 2
  • Cross-channel encoders predict one subset of channels from another, but single-direction designs leave the target channels unused during feature extraction.Source: §1 · p. 2
  • Split-brain autoencoders add one architectural split, train two disjoint concatenated sub-networks on complementary channel-prediction tasks, and extract features from the full input tensor.Source: §1 · pp. 1–2
  • The method uses raw data as supervision, targets RGB and RGB-D representation learning, and reports state-of-the-art performance on several semantic benchmarks alongside ablation studies.Source: §1 · p. 2

2. Related Work

Related work includes reconstruction, self-supervised pseudo-label prediction, multisensory representation learning, and image channel prediction. This paper uses channel prediction for representation learning rather than treating it solely as a graphics objective.

  • Autoencoders reconstruct images with a bottleneck, while denoising autoencoders undo random iid corruption; related generative approaches include variational autoencoders.Source: §2 · p. 2
  • Self-supervised methods predict pseudo-labels derived from raw data, including colorization, egomotion, video, inpainting, cooccurrence, context, and sound.Source: §2 · pp. 2–3
  • Representation learning from multisensory signals also appears in structure learning, co-training, and multi-view learning, including bidirectional prediction between sensory modalities.Source: §2 · p. 3
  • Computer vision and graphics research studies channel prediction tasks such as colorization, depth prediction, and surface-normal prediction, whereas this work studies their utility for representation learning.Source: §2 · p. 3

3. Methods

The method uses cross-channel prediction and a split architecture to learn representations from complementary views of the full input. It supports alternative losses and aggregation strategies while addressing the information imbalance of single cross-channel encoders.

3.1. Cross-Channel Encoders

  • Cross-channel encoding splits input channels into X1 and X2, training a CNN to predict X2 from X1 so its layers learn high-level representations.The prediction task can use different loss functions, including regression and classification.Source: §3.1 · p. 3
  • Classification loss predicts quantized output distributions with cross-entropy, while regression uses an ℓ2 objective; classification may better represent uncertain predictions.The paper hypothesizes that matching the full distribution can improve representations for inherently uncertain tasks.Source: §3.1 · p. 3
  • Using all input and output channels with ℓ2 regression recovers the traditional autoencoder objective, whereas cross-channel encoders use only channel subsets.This contrast motivates an architecture that preserves cross-channel abstraction without discarding part of the input during feature extraction.Source: §3.1 · p. 3, §3.2 · p. 4

3.2. Split-Brain Autoencoders as Aggregated Cross-Channel Encoders

  • Split-brain autoencoders combine two disjoint cross-channel encoders that predict opposite channel subsets, producing a representation pretrained on the full input tensor.The sub-networks can be formed by splitting each layer along its channel dimension and concatenating their representations.Source: §3.2 · p. 3
  • The proposed split architecture aggregates the two mappings through concatenation and is compared with a shared representation trained to perform both mappings simultaneously.The figure’s Lab-image example uses colorization in one branch and grayscale prediction in the other; alternative aggregation is described with a combined loss.Source: §3.2 · pp. 3–4, Figure 2 · p. 4

4. Experiments

The experiments evaluate split-brain autoencoders on ImageNet, Places, and PASCAL transfer benchmarks, while testing channel splits, objectives, and aggregation methods. The method uses Lab-channel cross-prediction with concatenated subnetworks and achieves strong transfer performance, including the best unsupervised results across evaluated ImageNet layers and Places conv2-5.

Experimental setup

  • The method uses Lab images, splitting inputs into L lightness and ab color channels for cross-channel encoding.The authors found the L and ab split most effective among explored color-space alternatives.Source: §4.1 · p. 4
  • The full method concatenates disjoint subnetworks for colorization and grayscale prediction, training both with classification losses.The study also evaluates regression-trained variants, ensembles, and alternative aggregation schemes.Source: §4.1 · p. 4

Transfer learning tests

  • The split-brain autoencoder achieves the best performance among unsupervised methods on all ImageNet layers and from conv2-5 on Places.Figure 3 reports the same cross-benchmark comparison against previous large-scale unsupervised methods.Source: Table 2 · p. 5, Table 3 · p. 5, Figure 3 · p. 6
  • Evaluation uses frozen representations with linear classifiers for 1000-way ImageNet classification and 205-way Places classification, plus fine-tuning on PASCAL classification, detection, and segmentation.ImageNet and Places assess task and dataset generalization, while PASCAL uses established downstream frameworks.Source: §4.1.1 · pp. 5–6

Experimental considerations

  • Changing the architecture can produce 4× denser feature maps while keeping parameter count fixed but increasing memory use 5.6× and runtime 7.4×.The authors note that the effect of this architectural change on representational quality is unclear.Source: §4.1.1 · p. 5

Ablation studies

  • Ablations compare autoencoding, cross-channel prediction problems and losses, and aggregation methods on the ImageNet classification benchmark.The study includes Lab reconstruction, randomly dropped inputs, colorization, grayscale prediction, and alternative encoder combinations.Source: §4.1 · p. 5, Figure 4 · p. 6

4.1.2 Split-Brain Autoencoder Performance

Split-brain autoencoders perform cross-channel prediction with complementary branches and achieve strong transfer performance on RGB and RGB-D benchmarks. The method reaches state-of-the-art results on most self-supervision benchmarks and substantially improves over autoencoder and initialization baselines.

RGB performance

  • Split-Brain Auto (cl,cl) achieves state-of-the-art performance on almost all established self-supervision benchmarks.The comparison includes previously proposed self-supervision methods and ablation baselines.Source: §4.1.2 · p. 7
  • 5% to 20% higher performance throughout the network is achieved by Split-Brain Auto (reg,reg) over the autoencoder and prior unsupervised baselines.The split architecture improves performance even with the same regression objective.Source: §4.1.2 · p. 7
  • Approximately 1-2% higher performance throughout the network results from using classification rather than regression for Split-Brain Auto.This comparison concerns Split-Brain Auto (cl,cl) versus Split-Brain Auto (reg,reg).Source: §4.1.2 · p. 7

Cross-channel objectives

  • Colorization, L→ab(cl), produces stronger transfer representations than inpainting, relative context, and adversarial feature networks on ImageNet classification.Classification loss is stronger than regression for the colorization objective.Source: §4.1.3 · p. 7
  • Grayscale prediction from color, ab→L(cl) and ab→L(reg), learns representations above the Gaussian baseline and complements the colorization network.Regression performs better than classification for grayscale prediction, while the typical loss difference is small.Source: §4.1.3 · p. 7

RGB-D performance

  • 38.1% RGB-D ensembled performance exceeds the Gaussian and Kr¨ahenb¨uhl et al. baselines on NYU-D.The method reaches 18.9% and 33.2% on the individual RGB and depth domains, respectively.Source: §4.2 · pp. 7–8

5. Discussion

The split-brain autoencoder is an unsupervised pre-training method that uses disjoint cross-channel encoders to predict subsets of raw data. It avoids a representational bottleneck, uses the full input during pre-training, and shows state-of-the-art performance on Lab images plus promising RGB-D performance.

Method

  • Split-brain autoencoders contain two disjoint sub-networks, each predicting one subset of raw data from another.The sub-networks are trained as cross-channel encoders using raw data as supervision.Source: §5 · p. 8
  • The method does not require a representational bottleneck, uses input dropout to force abstraction, and pre-trains on the full input data.These design choices address weaknesses identified in prior self-supervised methods.Source: §5 · p. 8

Results

  • The method achieves state-of-the-art performance relative to previous self-supervised methods on Lab images and promising performance on RGB-D images.Source: §5 · p. 8

Future direction

  • Exploring more than two cross-channel sub-networks remains a future direction because fixed-size networks make each sub-network smaller and less expressive.The paper instead fixes the full AlexNet architecture size for fair comparison with prior methods.Source: §5 · p. 8

Appendix

The appendix provides additional analysis and implementation details for the split-brain autoencoder experiments.

Appendix

  • Section A provides additional analysis, while Section B provides implementation details.Source: Appendix · p. 9

A. Additional analysis

Concatenating complementary cross-channel encoders is more effective than alternative aggregation strategies. Training one network on multiple mappings or adding the autoencoder objective lowers or fails to improve representation performance.

Cross-channel encoder aggregation

  • Concatenating complementary signals improves representation learning beyond an ensembled colorization network.The comparison suggests the gain is not explained solely by an ensembling effect.Source: A. Additional analysis · p. 9
  • Combining cross-channel encoders through concatenation is effective for representation learning.Source: A. Additional analysis · p. 9

Alternative aggregation

  • Training a single network to perform both cross-channel mappings is not effective for representation learning on full Lab images.Source: A. Additional analysis · p. 9
  • Adding the autoencoder objective during training lowers performance in higher layers.The tested mapping is (L,ab,Lab)→(ab,L,Lab).Source: A. Additional analysis · p. 9

Conclusion

B. Implementation Details

Pre-training uses a fully convolutional AlexNet that predicts heavily downsampled outputs, while feature evaluation transfers its parameters into an AlexNet classification architecture. Implementation choices also differ between graphics and classification colorization objectives.

Pre-training architecture

  • Pre-training uses 180×180 crops from 256×256 images and predicts values at a heavily downsampled 12×12 resolution.Full-resolution prediction is possible with upsampling-convolutional or dilated-convolution layers, but requires additional memory and run-time.Source: B. Implementation Details · p. 9

Feature evaluation

  • Feature evaluation copies pre-training parameters into an AlexNet classification architecture for ImageNet, Places, and PASCAL tests.During linear classification, feature maps are spatially downsampled so each layer has approximately the same number of features.Source: B. Implementation Details · p. 9, Table 7 · p. 9

Architecture specification

  • Table 6 specifies the fully convolutional AlexNet pre-training architecture, including spatial resolution, channels, kernels, strides, dilation, and padding.The first and last channel sizes depend on the pre-text task, and the last layer is removed during transfer evaluation.Source: Table 6 · p. 9
  • Table 7 specifies the AlexNet feature-evaluation architecture, including downsampled spatial resolutions and approximately constant feature-map sizes.The first-layer channel size depends on the pre-text task, such as 3 for split-brain autoencoders or 1 for L→ab(cl) colorization.Source: Table 7 · p. 9

Colorization objectives

  • The classification colorization objective omits class rebalancing and uses one-hot class encoding instead of the graphics model’s soft encoding.The passage reports that this simplification achieves higher performance on ImageNet.Source: B. Implementation Details · p. 10

C. Change Log

The change log records versioned updates through the CVPR 2017 camera-ready release, including corrected results, supplemental material, and references to concurrent work.

v2 Paper accepted to CVPR 2017

  • Version 2 records CVPR 2017 acceptance, updated Table 4 results, corrected Table 2 values, and added supplemental material.The Table 4 update covers results for Misra et al. and Donahue et al. with the 112×112-resolution model.Source: C. Change Log · p. 10

v3 CVPR 2017 Camera Ready

  • Version 3 identifies the CVPR 2017 camera-ready release and adds references to concurrent work.It also records various text changes.Source: C. Change Log · p. 10

Related papers

Loading 1611.09842v3…