Source-linked AI summary
Between-class Learning for Image Classification
Yuji Tokozume, Yoshitaka Ushiku, Tatsuya Harada
TL;DR
Image classification needs further improvement, while mixing images appears visually meaningless. The paper adapts Between-Class learning by mixing different-class images and predicting their mixing ratio, motivated by CNNs’ waveform-like processing. It reports 19.4% top-1 error on ImageNet-1K and 2.26% on CIFAR-10, while noting that BC learning can require a carefully designed training schedule.
Problem
Image classification has reached relatively stable performance, but further improvement is needed, and applying sound-oriented mixing to visually meaningless mixed images appears inappropriate.
Method
BC learning mixes two images from different classes with a random ratio and trains the model to predict the mixing ratio, using internal divisions and a waveform-based BC+ variant.
Results
19.4% top-1 error on ImageNet-1K and 2.26% top-1 error on CIFAR-10 were achieved using BC learning variants.
Takeaways & Limitations
BC learning can impose constraints on feature distributions and is presented as a simple method applicable beyond images and sounds.
Takeaways & Limitations
Training between-class examples among 1,000 classes can require many epochs, so the learning schedule should be carefully designed.
Abstract
from arXiv · showhide
In this paper, we propose a novel learning method for image classification called Between-Class learning (BC learning). We generate between-class images by mixing two images belonging to different classes with a random ratio. We then input the mixed image to the model and train the model to output the mixing ratio. BC learning has the ability to impose constraints on the shape of the feature distributions, and thus the generalization ability is improved. BC learning is originally a method developed for sounds, which can be digitally mixed. Mixing two image data does not appear to make sense; however, we argue that because convolutional neural networks have an aspect of treating input data as waveforms, what works on sounds must also work on images. First, we propose a simple mixing method using internal divisions, which surprisingly proves to significantly improve performance. Second, we propose a mixing method that treats the images as waveforms, which leads to a further improvement in performance. As a result, we achieved 19.4% and 2.26% top-1 errors on ImageNet-1K and CIFAR-10, respectively.
1. Introduction
The paper asks whether Between-Class learning can improve image classification despite mixed images lacking visual objectness. It applies cross-class image mixing, argues CNNs process images partly as waveforms, and reports improved benchmark performance.
- Training deep neural networks has become relatively easy, creating a need for novel approaches to further improve image-classification performance.
- Mixed images lose objectness and do not visually make sense, making the applicability of BC learning to images appear inappropriate.
- The paper argues that CNNs partly treat input data as waveforms, so mixtures of images can make sense to machines even when they do not visually make sense to humans.
- The authors propose internal-division mixing and an improved waveform-based method, with the latter subtracting each image’s mean and accounting for per-image standard deviation.The waveform-based method is called BC+ and is reported to produce further improvement.
- 19.4% top-1 error on ImageNet-1K was achieved by ResNeXt-101 (64 × 4d), improving from 20.4% with the simplest BC learning.
- 2.26% top-1 error on CIFAR-10 was achieved by Shake-Shake Regularization, improving from 2.86% with BC+.
- BC learning mixes two examples from different classes with a random ratio and trains the model to predict each class’s mixing ratio.This transfers the method from sound recognition to image classification.
2. BC learning for sounds
BC learning for sounds trains models on mixtures of different-class sounds and their mixing ratios. The paper describes this as imposing feature-distribution constraints through class separation and positional regularization.
- BC learning selects two sounds from different classes, mixes them using a random ratio, and trains the model to output the mixing ratio of each class.Labels are mixed as r t1 + (1 − r) t2, while the simplest sound mixture is r x1 + (1 − r) x2.
- Sound mixing must account for differing sound pressure levels because simple linear mixing may not match the intended perceptual ratio.The sound method therefore considers the pressure levels G1 and G2.
- BC learning is argued to enlarge Fisher’s criterion, the ratio of between-class distance to within-class variance.
- The feature of a mixed sound is hypothesized to lie near the internal dividing point between the features of the two original sounds as the mixing ratio changes.Figure 2 depicts the class distributions and the trajectory produced by varying the ratio from 0 to 1.
- BC learning is argued to regularize the positional relationship among class feature distributions.If class distributions are irregular, mixtures of two classes may fall across a third class’s decision boundary and be misclassified.
3. From sounds to images
The paper extends BC learning from sounds to images by mixing examples from different classes and training models to predict their mixing ratio. It motivates this transfer by treating image inputs as waveform-like data and develops increasingly specialized mixing schemes.
- BC learning for images: BC learning mixes two examples from different classes with a random ratio and trains the model to output the mixing ratio.The method uses only mixed data and labels, with KL-divergence replacing the usual cross-entropy loss.
- 3.1. Simple mixing: The simplest image mixing method uses internal divisions because image pixels lack the absolute center and energy concept available for sound data.
- 3.2. Why BC learning works on images: CNNs are argued to process images partly as waveforms, since image values contain frequency structure and convolutional filters can act as frequency filters.This motivates transferring a sound-based mixing strategy to image data despite mixed images lacking human-perceived objectness.
- 3.2. Why BC learning works on images: PCA visualization shows mixed-image features distributed between two classes, with trajectories obtained by varying the mixing ratio between particular image pairs.
- 3.3. BC+: Images as waveform data: BC+ subtracts each image's per-image mean and mixes zero-mean waveform components while accounting for image energy through per-image standard deviations.The energy-aware coefficient preserves the intended amplitude ratio between the two images.
4. Experiments
Experiments across ImageNet-1K, CIFAR-10, and CIFAR-100 show that BC learning improves diverse image-classification networks, with stronger results from waveform-based BC+ in several settings. Ablations and feature visualization support the proposed mixing design and its effect on feature distributions, while ImageNet training requires a carefully designed schedule.
- ImageNet-1K: 19.43% single-crop top-1 validation error was achieved by ResNeXt-101 with 150-epoch BC training on ImageNet-1K.The corresponding 10-crop top-1 error was 18.22%, and the single-crop top-1 error improved by around 1% over the default reported performance.
- CIFAR: 2.26% CIFAR-10 error was achieved with Shake-Shake Regularization using BC+, with all five trials ranging from 2.25%–2.28%.Simple BC learning improved all evaluated networks, while BC+ further improved performance; the paper does not claim this result as state-of-the-art.
- CIFAR: On CIFAR-10, BC learning’s testing error decreased at nearly the same early-stage speed as standard learning, while extra epochs lowered error for the 11-layer CNN and DenseNet.The training curves represent averages across all trials.
- CIFAR: BC learning also improved CIFAR-100 performance for the 11-layer CNN, ResNet-29, and DenseNet, but not ResNeXt-29 or Shake-Shake Regularization.The authors note that learning between-class examples among 100 classes may be difficult in settings without improvement.
- Ablation analysis: Per-image mean subtraction and division by sqrt(r^2 + (1 − r)^2) were important ablation components, whereas accounting for image-energy differences added little.These results support treating image data as waveform data as part of the performance improvement.
- Ablation analysis: Ratio labels with KL loss performed best, while single-label training worsened performance and multi-label sigmoid cross-entropy produced only marginal improvement.The authors explain that recognizing a mixed image as one particular class is inappropriate and that ratio labels enable more efficient learning of between-class examples.
- Ablation analysis: Mixing exactly two images from different classes outperformed same-class, randomly paired, and three-class mixtures.The authors attribute this to BC learning’s role in constraining feature distributions across different classes.
- Ablation analysis: Mixing at middle network layers worsened performance, suggesting that features combining spatial and semantic information are unsuitable for this operation.The reported middle layers were pool2 for CIFAR-10 and pool3 for CIFAR-100.
5. Conclusion
The paper applies BC learning to image classification by mixing images and training models to predict mixing ratios. Internal-division mixing improves performance, while waveform-based mixing improves it further and constrains feature distributions.
- BC learning mixes two images from different classes and trains the model to predict their mixing ratio.The method adapts an approach originally developed for sound recognition to image classification.
- Internal-division mixing significantly improves image-classification performance.
- Waveform-based image mixing further improves performance and can impose constraints on feature distributions.
- The authors assume BC learning may extend beyond images and sounds to other modalities.
A. Analysis on learned features
The analysis examines learned feature representations using the same 11-layer CNN as the main paper and provides more detailed feature analysis.
- The analysis visualizes learned features from an 11-layer CNN.
- The detailed analysis uses the same model as the main paper’s Fig. 6.
- The feature analysis is presented as an extension of the main paper’s visualization of BC learning’s feature-distribution constraint.
A.1. Fisher’s criterion
The authors compare Fisher’s criterion across class pairs using tenth-layer activations and report more discriminative features with BC learning than with standard learning.
- Fisher’s criterion is calculated for every combination of two classes.
- The criterion uses class features and their class-wise average vectors.
- The comparison uses activations from the 10-th layer on training data.
- BC learning produces a larger mean Fisher’s criterion than standard learning.
- The result indicates that BC learning learns a discriminative feature space.
A.2. Activation of final layer
The final-layer analysis examines mean pre-softmax neuron activations for training images by class. With BC learning, neurons respond selectively to corresponding classes while responses to other classes are similar and mostly negative.
- The analysis measures final-layer activations against training images.
- Under BC learning, each neuron responds primarily to its corresponding class.
- Responses to other classes are nearly equal and mostly negative.
- Each matrix element is the mean pre-softmax activation of neuron i for class j.
- The feature-distribution analysis also includes training and testing data.
A.3. Training features vs. testing features
BC learning produces training and testing feature distributions with similar shapes, unlike standard learning, where some testing examples lie near the origin.
- BC learning yields training and testing feature distributions with similar shapes.With standard learning, some testing examples are projected onto points near the origin.
B. Details of CIFAR experiments
The CIFAR experiments largely follow prior learning settings, with documented modifications for selected architectures and a cosine schedule for Shake-Shake.
- Most CIFAR learning settings follow the original settings from prior work.
- ResNet-29, ResNeXt-29, and DenseNet use slightly modified learning settings to achieve satisfactory performance.
- Shake-Shake uses cosine learning-rate scheduling, while other models divide the learning rate by 10 at listed schedule epochs.
B.2. Configuration of 11-layer CNN
The 11-layer CNN uses ReLU, batch normalization, dropout, and specified weight initialization; BC learning achieves low CIFAR error rates despite this simple architecture.
- The 11-layer CNN applies ReLU in hidden layers and batch normalization after every convolutional layer.
- The model applies 0.5 dropout after fc4 and fc5 and uses specified initialization schemes for convolutional and fully connected layers.
- 5.2% and 23.7% error rates are achieved on CIFAR-10 and CIFAR-100, respectively, with BC learning despite the simple architecture.