PaperlayerView exact revision on arXiv

Source-linked AI summary

Split-Brain Autoencoders: Unsupervised Learning by Cross-Channel Prediction

Richard Zhang, Phillip Isola, Alexei A. Efros

arXiv:1611.09842v3cs.CV

TL;DR

Unsupervised representation learning seeks features that adapt to unknown downstream tasks, but traditional autoencoders have not reliably produced strong transfer representations. Split-brain autoencoders address this by training two disjoint subnetworks on complementary cross-channel predictions, using the full input signal. The method achieves state-of-the-art performance on several semantic representation-learning benchmarks and promising performance on RGB-D images.

  • Problem

    Traditional autoencoders have not been shown to produce strong transfer representations, and their bottleneck creates a trade-off between forced abstraction and retained information.

  • Method

    Split-brain autoencoders split the network into two disjoint cross-channel encoders that predict complementary subsets of raw input channels and concatenate their representations.

  • Results

    State-of-the-art performance is achieved relative to previous self-supervised methods on Lab images, with promising performance also demonstrated on RGB-D images.

  • Takeaways & Limitations

    The method uses raw data as supervision and extracts features from the full input while avoiding a representational bottleneck during training.

  • Takeaways & Limitations

    With a fixed architecture size, adding more than two subnetworks makes each subnetwork smaller and less expressive, potentially worsening its original task.

Abstract

from arXiv · show

We propose split-brain autoencoders, a straightforward modification of the traditional autoencoder architecture, for unsupervised representation learning. The method adds a split to the network, resulting in two disjoint sub-networks. Each sub-network is trained to perform a difficult task -- predicting one subset of the data channels from another. Together, the sub-networks extract features from the entire input signal. By forcing the network to solve cross-channel prediction tasks, we induce a representation within the network which transfers well to other, unseen tasks. This method achieves state-of-the-art performance on several large-scale transfer learning benchmarks.

1. Introduction

The paper addresses weaknesses in reconstruction-based unsupervised representations by using complementary cross-channel prediction tasks. Split-brain autoencoders split the network into disjoint sub-networks whose concatenated features use the full input and transfer strongly.

  • Traditional autoencoders face a bottleneck trade-off: smaller bottlenecks force more abstraction but carry less information, while strong transfer representations remain unestablished in practice.
  • Context encoders can underperform because inpainting is difficult to evaluate, creates a training–deployment domain gap, and may be solved without high-level reasoning.
  • Colorization is an effective cross-channel pretext task because spatial correspondence supports standard losses, systematic corruption avoids a domain gap, and color prediction may require object-level reasoning.
  • Cross-channel encoders predict one subset of channels from another, but single-direction designs leave the target channels unused during feature extraction.
  • Split-brain autoencoders add one architectural split, train two disjoint concatenated sub-networks on complementary channel-prediction tasks, and extract features from the full input tensor.
  • The method uses raw data as supervision, targets RGB and RGB-D representation learning, and reports state-of-the-art performance on several semantic benchmarks alongside ablation studies.

2. Related Work

Related work includes reconstruction, self-supervised pseudo-label prediction, multisensory representation learning, and image channel prediction. This paper uses channel prediction for representation learning rather than treating it solely as a graphics objective.

  • Autoencoders reconstruct images with a bottleneck, while denoising autoencoders undo random iid corruption; related generative approaches include variational autoencoders.
  • Self-supervised methods predict pseudo-labels derived from raw data, including colorization, egomotion, video, inpainting, cooccurrence, context, and sound.
  • Representation learning from multisensory signals also appears in structure learning, co-training, and multi-view learning, including bidirectional prediction between sensory modalities.
  • Computer vision and graphics research studies channel prediction tasks such as colorization, depth prediction, and surface-normal prediction, whereas this work studies their utility for representation learning.

3. Methods

The method uses cross-channel prediction and a split architecture to learn representations from complementary views of the full input. It supports alternative losses and aggregation strategies while addressing the information imbalance of single cross-channel encoders.

  • 3.1. Cross-Channel Encoders: Cross-channel encoding splits input channels into X1 and X2, training a CNN to predict X2 from X1 so its layers learn high-level representations.The prediction task can use different loss functions, including regression and classification.
  • 3.1. Cross-Channel Encoders: Classification loss predicts quantized output distributions with cross-entropy, while regression uses an ℓ2 objective; classification may better represent uncertain predictions.The paper hypothesizes that matching the full distribution can improve representations for inherently uncertain tasks.
  • 3.1. Cross-Channel Encoders: Using all input and output channels with ℓ2 regression recovers the traditional autoencoder objective, whereas cross-channel encoders use only channel subsets.This contrast motivates an architecture that preserves cross-channel abstraction without discarding part of the input during feature extraction.
  • 3.2. Split-Brain Autoencoders as Aggregated Cross-Channel Encoders: Split-brain autoencoders combine two disjoint cross-channel encoders that predict opposite channel subsets, producing a representation pretrained on the full input tensor.The sub-networks can be formed by splitting each layer along its channel dimension and concatenating their representations.
  • 3.2. Split-Brain Autoencoders as Aggregated Cross-Channel Encoders: The proposed split architecture aggregates the two mappings through concatenation and is compared with a shared representation trained to perform both mappings simultaneously.The figure’s Lab-image example uses colorization in one branch and grayscale prediction in the other; alternative aggregation is described with a combined loss.

4. Experiments

The experiments evaluate split-brain autoencoders on ImageNet, Places, and PASCAL transfer benchmarks, while testing channel splits, objectives, and aggregation methods. The method uses Lab-channel cross-prediction with concatenated subnetworks and achieves strong transfer performance, including the best unsupervised results across evaluated ImageNet layers and Places conv2-5.

  • Experimental setup: The method uses Lab images, splitting inputs into L lightness and ab color channels for cross-channel encoding.The authors found the L and ab split most effective among explored color-space alternatives.
  • Experimental setup: The full method concatenates disjoint subnetworks for colorization and grayscale prediction, training both with classification losses.The study also evaluates regression-trained variants, ensembles, and alternative aggregation schemes.
  • Transfer learning tests: The split-brain autoencoder achieves the best performance among unsupervised methods on all ImageNet layers and from conv2-5 on Places.Figure 3 reports the same cross-benchmark comparison against previous large-scale unsupervised methods.
  • Transfer learning tests: Evaluation uses frozen representations with linear classifiers for 1000-way ImageNet classification and 205-way Places classification, plus fine-tuning on PASCAL classification, detection, and segmentation.ImageNet and Places assess task and dataset generalization, while PASCAL uses established downstream frameworks.
  • Experimental considerations: Changing the architecture can produce 4× denser feature maps while keeping parameter count fixed but increasing memory use 5.6× and runtime 7.4×.The authors note that the effect of this architectural change on representational quality is unclear.
  • Ablation studies: Ablations compare autoencoding, cross-channel prediction problems and losses, and aggregation methods on the ImageNet classification benchmark.The study includes Lab reconstruction, randomly dropped inputs, colorization, grayscale prediction, and alternative encoder combinations.

4.1.2 Split-Brain Autoencoder Performance

Split-brain autoencoders perform cross-channel prediction with complementary branches and achieve strong transfer performance on RGB and RGB-D benchmarks. The method reaches state-of-the-art results on most self-supervision benchmarks and substantially improves over autoencoder and initialization baselines.

  • RGB performance: Split-Brain Auto (cl,cl) achieves state-of-the-art performance on almost all established self-supervision benchmarks.The comparison includes previously proposed self-supervision methods and ablation baselines.
  • RGB performance: 5% to 20% higher performance throughout the network is achieved by Split-Brain Auto (reg,reg) over the autoencoder and prior unsupervised baselines.The split architecture improves performance even with the same regression objective.
  • RGB performance: Approximately 1-2% higher performance throughout the network results from using classification rather than regression for Split-Brain Auto.This comparison concerns Split-Brain Auto (cl,cl) versus Split-Brain Auto (reg,reg).
  • Cross-channel objectives: Colorization, L→ab(cl), produces stronger transfer representations than inpainting, relative context, and adversarial feature networks on ImageNet classification.Classification loss is stronger than regression for the colorization objective.
  • Cross-channel objectives: Grayscale prediction from color, ab→L(cl) and ab→L(reg), learns representations above the Gaussian baseline and complements the colorization network.Regression performs better than classification for grayscale prediction, while the typical loss difference is small.
  • RGB-D performance: 38.1% RGB-D ensembled performance exceeds the Gaussian and Kr¨ahenb¨uhl et al. baselines on NYU-D.The method reaches 18.9% and 33.2% on the individual RGB and depth domains, respectively.

5. Discussion

The split-brain autoencoder is an unsupervised pre-training method that uses disjoint cross-channel encoders to predict subsets of raw data. It avoids a representational bottleneck, uses the full input during pre-training, and shows state-of-the-art performance on Lab images plus promising RGB-D performance.

  • Method: Split-brain autoencoders contain two disjoint sub-networks, each predicting one subset of raw data from another.The sub-networks are trained as cross-channel encoders using raw data as supervision.
  • Method: The method does not require a representational bottleneck, uses input dropout to force abstraction, and pre-trains on the full input data.These design choices address weaknesses identified in prior self-supervised methods.
  • Results: The method achieves state-of-the-art performance relative to previous self-supervised methods on Lab images and promising performance on RGB-D images.
  • Future direction: Exploring more than two cross-channel sub-networks remains a future direction because fixed-size networks make each sub-network smaller and less expressive.The paper instead fixes the full AlexNet architecture size for fair comparison with prior methods.

Appendix

The appendix provides additional analysis and implementation details for the split-brain autoencoder experiments.

  • Appendix: Section A provides additional analysis, while Section B provides implementation details.

A. Additional analysis

Concatenating complementary cross-channel encoders is more effective than alternative aggregation strategies. Training one network on multiple mappings or adding the autoencoder objective lowers or fails to improve representation performance.

  • Cross-channel encoder aggregation: Concatenating complementary signals improves representation learning beyond an ensembled colorization network.The comparison suggests the gain is not explained solely by an ensembling effect.
  • Cross-channel encoder aggregation: Combining cross-channel encoders through concatenation is effective for representation learning.
  • Alternative aggregation: Training a single network to perform both cross-channel mappings is not effective for representation learning on full Lab images.
  • Alternative aggregation: Adding the autoencoder objective during training lowers performance in higher layers.The tested mapping is (L,ab,Lab)→(ab,L,Lab).
  • Conclusion: The proposed aggregation of complementary signals by concatenation outperforms the tested alternatives.

B. Implementation Details

Pre-training uses a fully convolutional AlexNet that predicts heavily downsampled outputs, while feature evaluation transfers its parameters into an AlexNet classification architecture. Implementation choices also differ between graphics and classification colorization objectives.

  • Pre-training architecture: Pre-training uses 180×180 crops from 256×256 images and predicts values at a heavily downsampled 12×12 resolution.Full-resolution prediction is possible with upsampling-convolutional or dilated-convolution layers, but requires additional memory and run-time.
  • Feature evaluation: Feature evaluation copies pre-training parameters into an AlexNet classification architecture for ImageNet, Places, and PASCAL tests.During linear classification, feature maps are spatially downsampled so each layer has approximately the same number of features.
  • Architecture specification: Table 6 specifies the fully convolutional AlexNet pre-training architecture, including spatial resolution, channels, kernels, strides, dilation, and padding.The first and last channel sizes depend on the pre-text task, and the last layer is removed during transfer evaluation.
  • Architecture specification: Table 7 specifies the AlexNet feature-evaluation architecture, including downsampled spatial resolutions and approximately constant feature-map sizes.The first-layer channel size depends on the pre-text task, such as 3 for split-brain autoencoders or 1 for L→ab(cl) colorization.
  • Colorization objectives: The classification colorization objective omits class rebalancing and uses one-hot class encoding instead of the graphics model’s soft encoding.The passage reports that this simplification achieves higher performance on ImageNet.

C. Change Log

The change log records versioned updates through the CVPR 2017 camera-ready release, including corrected results, supplemental material, and references to concurrent work.

  • v2 Paper accepted to CVPR 2017: Version 2 records CVPR 2017 acceptance, updated Table 4 results, corrected Table 2 values, and added supplemental material.The Table 4 update covers results for Misra et al. and Donahue et al. with the 112×112-resolution model.
  • v3 CVPR 2017 Camera Ready: Version 3 identifies the CVPR 2017 camera-ready release and adds references to concurrent work.It also records various text changes.
Loading 1611.09842v3…