PaperlayerView exact revision on arXiv

Source-linked AI summary

Improvements to context based self-supervised learning

T. Nathan Mundhenk, Daniel Ho, Barry Y. Chen

arXiv:1711.06379v3cs.CVcs.LGcs.NE

TL;DR

Self-supervised methods still trail supervised methods in transfer performance, while patch/context learning can also rely on incidental cues and leave mid-level features underused. This paper combines enhancements to context-based learning with broader datasets and multiple network architectures. The authors report outperforming other methods on standard benchmark tests, while results on other architectures remain close to but below supervised ImageNet training.

  • Problem

    Self-supervised methods remain well short of supervised methods in transfer performance, and patch/context learning raises concerns about incidental boundary cues and underused mid-layers.

    Source: §1 · p. 2, §2 · p. 2

  • Method

    The authors develop transferable self-supervised enhancements, use two additional datasets, and test across multiple neural-network architectures.

    Source: §3 · p. 3

  • Results

    The authors report that their improvements outperform other methods on all standard benchmark tests.

    Source: §4.3 · p. 8

  • Takeaways & Limitations

    The authors present the techniques as potentially applicable to other self-supervised methods and demonstrate results across several architectures.

    Source: §1 · p. 2, §3 · p. 3

    The extent to which trivial patch-boundary completion is a problem remains unclear, since it may also help form low-level features.

    Source: §2 · p. 2

Abstract

from arXiv · show

We develop a set of methods to improve on the results of self-supervised learning using context. We start with a baseline of patch based arrangement context learning and go from there. Our methods address some overt problems such as chromatic aberration as well as other potential problems such as spatial skew and mid-level feature neglect. We prevent problems with testing generalization on common self-supervised benchmark tests by using different datasets during our development. The results of our methods combined yield top scores on all standard self-supervised benchmarks, including classification and detection on PASCAL VOC 2007, segmentation on PASCAL VOC 2012, and "linear tests" on the ImageNet and CSAIL Places datasets. We obtain an improvement over our baseline method of between 4.0 to 7.1 percentage points on transfer learning classification tests. We also show results on different standard network architectures to demonstrate generalization as well as portability. All data, models and programs are available at: https://gdo-datasci.llnl.gov/selfsupervised/.

1. Introduction

Self-supervised learning trains networks on tasks such as predicting image-patch arrangements, with the aim of learning representations that transfer to other tasks. This paper seeks to improve its transfer performance through several techniques and evaluates their generality across architectures, datasets, and tasks.

1. Introduction

  • Patch-arrangement learning trains a network to predict relative patch positions, with the goal of learning image features and semantics useful for transfer tasks.The passage describes transfer classification on PASCAL VOC as an example.Source: §1 · p. 1
  • Self-supervised tasks also include image transformations, completion, colorization, motion, object counting, and multi-task ensembles.Source: §1 · p. 1
  • Self-supervised methods remained well short of supervised methods in transfer-learning performance.Source: §1 · pp. 1–2
  • The authors aim to improve self-supervised learning, with techniques intended to generalize across architectures, datasets, and tasks.They identify CUB as a dataset for probing shortcomings of approaches to chromatic aberration.Source: §1 · p. 2

2. Issues and Related Work

Patch/context methods predict the arrangement of image patches and can perform well on transfer benchmarks, but may exploit incidental cues instead of semantic structure. The section discusses chromatic aberration, patch-boundary matching, and concerns about the usefulness of mid-level features.

2. Issues and Related Work

  • Patch/context learning assigns labels to patch arrangements and trains a network to predict them; implementations can use shared-weight Siamese paths.Source: §2 · p. 2
  • Patch/context methods can perform strongly on transfer benchmarks, including PASCAL VOC detection and classification and VOC 2012 segmentation.The cited examples include a top VOC 2007 detection score for and top detection and second-highest segmentation scores for.Source: §2 · p. 2
  • Chromatic aberration can reveal patch location, while channel removal or grayscale conversion sacrifices joint color information and channel jitter may blur features.Figure 1 illustrates chroma blurring of training patches and shows an example where it blends out aberration.Source: §2 · p. 2, Figure 1 · p. 3
  • Patch/context networks may exploit simple boundary patterns rather than semantic structure, but the extent of this problem is unclear because boundary cues might help form low-level features.Source: §2 · p. 2
  • Evidence from layer selection and CSAIL Places linear tests suggests self-supervised networks may use middle layers less effectively than early or later layers.Many methods perform comparably to or better than supervised ImageNet training in early AlexNet convolutional layers but struggle deeper in the network.Source: §2 · p. 2

3. Approach

The approach combines tools intended to transfer across self-supervised methods with experiments on additional datasets and network architectures. It starts from a two-patch spatial-configuration baseline and tests enhancements empirically.

3.1.1 Chroma Blurring (CB)

  • Chroma blurring smooths the Lab a and b channels while leaving luminance untouched, aiming to reduce aberration cues while retaining intensity detail.The method uses a 13x13 box filter on the chroma channels before converting the image back to RGB.Source: §3.1.1 · p. 3

3.1.2 Yoked Jitter of Patches (YJ)

  • Yoked jitter applies the same random crop offset to every patch, preserving their relative spacing while adding crop variation.The authors note this may remain prone to low-level boundary completion.Source: §3.1.2 · p. 3

3.1.3 Additional Patches and Configurations

  • Three-patch sets and extra configurations expand image coverage and mix scales to make simple pattern matching harder.The method uses three patches rather than two; the authors did not test four-patch sets.Source: §3.1.3 · p. 4

3.1.4 Random Aperture of Patches (RA)

  • Random apertures mask parts of two patches while leaving one intact, aiming to favor learning in fourth-layer features without eliminating fifth-layer learning.The aperture is randomly sized from 64x64 to 96x96; the figure caption describes the intended layer effect as simplified and imprecise.Source: §3.1.4 · pp. 4–5, Figure 3 · p. 5

3.1.5 Rotation with Classification (RWC)

  • Rotation with classification adds orientation-specific arrangement classes by flipping or rotating the whole image, weakening chromatic-aberration cues to patch location.The method considers 180-degree flips and 90- and 270-degree rotations.Source: §3.1.5 · p. 5, Figure 4 · p. 5

3.1.6 Miscellany

  • Miscellaneous experiments include yoked mirroring, zooming, and cropping; independently rescaled patches; and layer-specific learning-rate multipliers.The learning-rate schedule increases toward conv4 and then decreases across the nine Siamese AlexNet layers.Source: §3.1.6 · pp. 5–6

3.2. Verification Datasets

  • CUB birds and CompCars provide fine-grained validation datasets with different object classes, and CUB also tests methods addressing chromatic aberration.The authors motivate using datasets distinct from standard benchmarks during technique development to support confidence in generalization.Source: §3.2 · p. 6

3.3. Alternative Networks

  • The authors demonstrate the approach on four network types, including AlexNet variants, ResCeption, and batch-normalized Inception, to examine generalization and portability.Most ablation and validation experiments use a custom batch-normalized AlexNet; benchmark tests use standard CaffeNet/AlexNet.Source: §3.3 · p. 6, Figure 5 · p. 6

4. Experiments

The experiments ablate context-learning enhancements on CUB and CompCars, then evaluate the resulting methods on standard transfer benchmarks and additional network architectures. The combined improvements outperform other methods on the standard benchmarks, while results on other architectures show portability and a remaining gap to supervised training.

4.2. Validation and Ablation on CUB Birds and CompCars

  • Rotation with classification, chroma blurring, and random aperture produce the largest gains across CUB, CompCars, and post hoc VOC results.Table 1 describes CUB and CompCars scores as single-class accuracy and VOC scores as mAP.Source: Table 1 · p. 7
  • Ablations and validation use a custom batch-normalized AlexNet transferred to CUB and CompCars, with three training runs averaged per condition.The five convolution and batch-normalization layers are transferred, and new fully connected layers are initialized.Source: §4.2 · p. 7

4.3. Standard Transfer Learning Testing Battery

  • For benchmark testing, the authors transfer convolution layers one through five to an off-the-shelf CaffeNet and follow each test's standard protocol, with two noted differences.Detection uses multiscale training and testing; segmentation layers six and seven are initialized randomly.Source: §4.3 · pp. 7–8
  • The combined improvements outperform other methods on standard benchmarks; on VOC, RWC gives the best results, with RRM adding only a slight improvement.The benchmark suite includes VOC classification, detection, and segmentation, plus linear tests on ImageNet and CSAIL Places.Source: §4.3 · pp. 7–8, Table 2 · p. 8

4.4. Portability to Other Networks

  • On additional networks, CompCars results are about one to two percentage points from ImageNet supervised training, while CUB results are three to six points away.The experiments also report linear-test results for ImageNet and CSAIL Places; RRM edges out the other two conditions slightly on ImageNet, while RWC is slightly better on Places.Source: Table 3 · p. 8, Table 4 · p. 8, Table 5 · p. 8

A. Appendix: CompCars Dataset Augmentation

The CompCars training set is augmented with hue-channel swaps and minor perspective jitter, creating 24 variants per image.

A. Appendix: CompCars Dataset Augmentation

  • Each image has 24 augmentations, including hue-rotated versions, perspective-jittered versions, combinations of both, and repeats of the original.Fourteen variants have random perspective jitter, 12 have hue rotation, six combine the two, and four repeat the unaugmented image.Source: A. Appendix: CompCars Dataset… · p. 11
  • Hue rotation swaps color channels, reflecting the hypothesis that car models vary in color but may have characteristic color styles.The authors give conservative, low-saturation family sedans and intensely saturated sports cars as examples.Source: A. Appendix: CompCars Dataset… · p. 11
  • Perspective jitter randomly perturbs three Euler angles by +/- 0.00286 degrees before forming a perspective transformation matrix.The permutations are created before training because perspective transformation is modestly expensive and can be a bottleneck.Source: A. Appendix: CompCars Dataset… · p. 11

B.1. Yoked Jitter Gets Better with Extra Patches

The authors test whether extra patch configurations with yoked or random jitter reduce trivial pattern completion between adjacent patches.

B.1. Yoked Jitter Gets Better with Extra Patches

  • Hybrid patches are intended to reduce the network's ability to learn trivial pattern completion between adjacent patches.Source: §B.1 · p. 11
  • The extra-patch-configuration test compares yoked jitter with random jitter.Source: §B.1 · p. 11
  • The larger gain with yoked jitter provides some evidence that extra patch configurations may reduce trivial pattern completion.Source: §B.1 · p. 11

B.2. Do Rotations Need Classification?

The authors compare rotating patches alone with rotation classification to test whether the classification objective matters.

B.2. Do Rotations Need Classification?

  • Rotating a patch without classifying its rotation makes almost no difference compared with no rotation.Source: §B.2 · p. 11
  • The authors suggest that classification may sharpen object features by requiring the network to recognize rotated objects uniquely.Source: §B.2 · p. 11
  • The authors also suggest that classification may help mitigate chromatic aberration.Source: §B.2 · p. 11

B.3. How much does Chroma Blurring Help?

Chroma blurring removes chromatic-aberration effects while retaining some color processing and can improve classification over no color processing. Its benefit subsides with the full toolset, where rotation with classification may partly mitigate aberration.

B.3. How much does Chroma Blurring Help?

  • Chroma blurring gives a moderate classification boost over no color processing, but its benefit subsides with the full toolset.The table reports that chroma blurring ceases to help once the full toolset is added.Source: §B.3 · p. 11, Table 8 · p. 11
  • Yoked jitter generally improves results over random jitter, with a larger improvement when extra patch configurations are included.The hybrid patches may prevent low-level trivial boundary completion because they use mixed scales.Source: Table 6 · p. 11
  • Rotating patches without assigning rotations their own classes yields almost no gain.The comparison uses the full toolset except RRM and WV.Source: Table 7 · p. 11
  • Rotation with classification may partly mitigate chromatic aberration, helping explain why chroma blurring adds no benefit with the full toolset.The authors present this as a suspected explanation.Source: Table 8 · p. 11, §B.3 · p. 11

B.4. The Benefit of Adding Different Kinds of Patches

Extra patch configurations appear helpful, but their interactions with one another and the other tools are not deterministic; Table 9 separates the contributions of two configuration types.

B.4. The Benefit of Adding Different Kinds of Patches

  • Extra patch configurations appear to help, although their interactions with one another and other tools are not deterministic.Table 9 parses the contributions of the two new configuration types.Source: §B.4 · p. 11
  • The two types of new patch configurations are evaluated separately in Table 9.Source: §B.4 · p. 11

B.5. Two v. Three Apertures

Using two apertures rather than three performs better, while aperture-based randomization boosts AlexNet’s middle layers at the expense of its early layers. The effects of extra patch configurations depend on the combination and toolset.

B.5. Two v. Three Apertures

  • Two apertures perform better than three; applying apertures to all three patches causes a noticeable drop, particularly on CUB Birds.The authors did not test applying an aperture to only one patch.Source: §B.5 · p. 11, Table 10 · p. 14
  • Random Aperture boosts AlexNet layers four and five but degrades layers one and two, consistent with a bias toward middle layers.Figure 8 also describes the tradeoff as improving layers four and five at the expense of layers one, two, and three; this experiment used no padding during self-supervised training.Source: §B.5 · p. 13, Figure 8 · p. 13
  • Using both extra patch configurations is always better than using only 3x3 patches, but hybrid patches alone are worse with the full toolset.The authors hypothesize that 2x2 patches aid rotation classification by including the image’s top and bottom, which can cue upside-down orientation.Source: Table 9 · p. 13

B.6. RRM is a Tiny Bit Better

RRM is slightly better on ImageNet linear and VOC tests, roughly even on the CSAIL Places linear test, and does not seem to hurt. Revision notes also report changes to segmentation evaluation, the Inception model, and CaffeNet pretraining.

B.6. RRM is a Tiny Bit Better

  • RRM yields slightly better ImageNet linear and VOC results, is roughly even on CSAIL Places linear, and does not seem to hurt.Earlier experiments showed a stronger gain pattern, so the amount of benefit remains unclear.Source: §B.6 · p. 14

C. Appendix: Document Revision History

  • The revision corrected the segmentation method to the standard from and fixed a flaw in the self-supervised Inception model, slightly improving results.The earlier segmentation results used a method that deviated slightly from the standard.Source: C. Appendix: Document Revision History · p. 15
  • Adding padding to self-supervised CaffeNet pretraining noticeably improves VOC detection and segmentation, with little effect on classification.This applies only to CaffeNet; newer networks such as Inception use padding by default, while transfer-learning CaffeNet remained vanilla.Source: C. Appendix: Document Revision History · p. 15

Related papers

Loading 1711.06379v3…