PaperlayerView exact revision on arXiv

Source-linked AI summary

Breaking Inter-Layer Co-Adaptation by Classifier Anonymization

Ikuro Sato, Kohta Ishikawa, Guoqing Liu, Masayuki Tanaka

arXiv:1906.01150v1cs.LGstat.ML

TL;DR

Joint optimization can make a feature extractor and classifier co-adapt, producing complex features tied to one classifier. FOCA instead trains features against randomly generated weak classifiers; theory and experiments support point-like, class-separable features within the stated scope.

  • Problem

    Jointly optimized feature extractors and classifiers can become co-adapted, leaving complex features tied to a specific classifier and motivating training without that dependence.

    Source: §1 · p. 1

  • Method

    FOCA optimizes a feature extractor using independently generated weak classifiers rather than jointly optimizing it with one particular classifier.

    Source: §1 · p. 2

  • Results

    Under special conditions, FOCA features are point-like and class-separable; real-data experiments provide supportive evidence under more general conditions.

    Source: §3 · p. 5, §5 · p. 8

  • Takeaways & Limitations

    With point-like class features, a classifier strong on a partial dataset is also strong on the full dataset, and experiments report strong performance using small subsets.

    Source: §1 · p. 2, §4.1 · p. 6

    The point-like property is not proved for multi-layered classifiers, although the toy experiment exhibits it.

    Source: §3 · p. 5

Abstract

from arXiv · show

This study addresses an issue of co-adaptation between a feature extractor and a classifier in a neural network. A naive joint optimization of a feature extractor and a classifier often brings situations in which an excessively complex feature distribution adapted to a very specific classifier degrades the test performance. We introduce a method called Feature-extractor Optimization through Classifier Anonymization (FOCA), which is designed to avoid an explicit co-adaptation between a feature extractor and a particular classifier by using many randomly-generated, weak classifiers during optimization. We put forth a mathematical proposition that states the FOCA features form a point-like distribution within the same class in a class-separable fashion under special conditions. Real-data experiments under more general conditions provide supportive evidences.

1. Introduction

Joint optimization can create fragile inter-layer co-adaptation, while dropout does not necessarily prevent it. FOCA avoids explicit adaptation to one classifier by optimizing features against independently generated weak classifiers; theory and experiments support simple, point-like class distributions.

1. Introduction

  • Dropout randomly deactivates neurons during training to discourage co-adaptation, although a critique says it does not necessarily prevent it.The introduction frames this as motivation for a more fundamental approach.Source: §1 · p. 1
  • Inter-layer co-adaptation can make secondary optimization degrade test performance despite dropout, and excessively complex feature distributions are identified as a major factor.Prior work also reported negative effects in cross-domain transfer.Source: §1 · pp. 1–2
  • The paper investigates whether a feature extractor can be trained without co-adapting to a particular classifier and what features result.It seeks a fundamental alternative to empirically searching for the best-performing feature layers.Source: §1 · p. 2
  • FOCA marginalizes independently generated weak classifiers instead of jointly optimizing one classifier with the feature extractor, avoiding explicit co-adaptation to a particular classifier.The introduction presents this as a way to break unwanted inter-layer co-adaptation.Source: §1 · p. 2
  • Under special conditions, a strong classifier for a partial dataset must also be strong for the full dataset when class features are point-like and class-separable.The introduction reports experimental support: small-to-large dataset classifier solutions are close, and low-dimensional features appear nearly point-like.Source: §1 · p. 2

2. Optimization Method to Break Inter-Layer Co-Adaptation

Naive joint optimization lets the feature extractor and classifier adapt to each other, potentially producing excessively complex features. FOCA instead optimizes the feature extractor over independently generated weak classifiers, with a later classifier trained on fixed features.

2.1. Joint Optimization: a Review

  • Naive SGD updates the classifier to discriminate presented features and the feature extractor to strengthen the current classifier, regardless of complexity, potentially yielding excessively complex features.The paper uses this as the joint-optimization failure mode FOCA aims to avoid.Source: §2.1 · p. 2

2.2. Feature-extractor Optimization through Classifier Anonymization (FOCA)

  • FOCA minimizes expected loss over a distribution of independently sampled weak classifiers, so the feature extractor does not explicitly co-adapt to one classifier.The method depends on classifiers being distinct; classifiers that are too strong can drive complex boundaries, while classifiers that are too weak or adversarial can prevent convergence.Source: §2.2 · p. 2
  • FOCA defines its weak-classifier distribution using uniformly sampled, batch-specific classifiers trained with norm-regularized batch loss; each batch covers all classes.Random initialization is assumed, leaving almost no continuity between classifiers for nearby feature parameters; a batch classifier need not be weak in the classical sense and can even be adversarial on the full dataset.Source: §2.2 · p. 3
  • After feature extraction, a single classifier is trained on fixed features using the entire dataset, and its architecture may differ from the primary optimization classifier.The approximate procedure samples one weak classifier per feature update and holds its distribution fixed when computing the feature gradient.Source: §2.2 · p. 3

3. Mathematical Property

Under specified architectural, loss, classifier, and data conditions, FOCA’s class-separable optimum gives identical features within each class and distinct features across classes. A toy experiment also shows point-like class clusters, though the property is not proven for multi-layer classifiers.

3. Mathematical Property

  • The proposition assumes a multi-layer feature extractor with the specified activation and fully connected last layer, two target values, sample-wise squared loss, a linear classifier, and batches pairing one sample from each class.Source: §3 · p. 4
  • Under these conditions, a class-separable optimum of the classifier-anonymized sample-wise losses gives identical features within each class and different features across classes.Source: §3 · p. 4
  • The toy experiment shows same-class features confined to a vicinity much smaller than the distance between class centroids, without explicitly maximizing between-class scatter relative to within-class scatter.Source: §3 · p. 4
  • The proposition is not proven for multi-layer classifier cases, although the toy experiment also exhibits point-like features in that setting.Source: §3 · p. 5

4. Experiment

The experiments test whether FOCA features show simple, class-separable structure under realistic conditions, including whether classifiers trained on small subsets retain performance. Across partial-data tests, solution distances, and low-dimensional analyses, the results support that picture.

4. Experiment

  • The experiments used CIFAR-10 and CIFAR-100, comparing FOCA with Plain, Noisy, Dropout, and Batch Normalization using secondary classifiers trained on fixed features and reduced datasets.The smallest tested datasets contained one example per class: 10 samples for CIFAR-10 and 100 for CIFAR-100.Source: §4 · p. 5, §4.1 · p. 6

4.1. Test Performances of Classifiers Trained on Partial Datasets

  • FOCA outperformed the other methods on CIFAR-10 when nD′ ≤ 10^3 and on CIFAR-100 at every tested partial-dataset size.On CIFAR-10, FOCA's average error at nD′ = 10 was 14.45%, versus Dropout's 14.95% at 5 × 10^4, Plain's 14.50% at 10^4, Noisy's 14.75% at 10^3, and Batch Normalization's 16.93% at 10^2; on CIFAR-100, FOCA's error at nD′ = 10^2 was 46.03%, lower than every other method at nD′ = 5 × 10^4.Source: Figure 2 · p. 6, §4.1 · p. 6

4.2. Approximate Geodesic Distances between Solutions

  • FOCA's approximate geodesic distances between large- and small-dataset solutions were 40–180 times smaller on CIFAR-10 and 3–9 times smaller on CIFAR-100 than those of other methods.Its test error also remained almost constant along the interpolated solutions, supporting the interpretation that the two solutions are virtually the same point.Source: Figure 3 · p. 7, §4.2 · p. 7

4.3. Low-Dimensional Properties

  • For CIFAR-10, FOCA features looked nearly one-dimensional per class under PCA, while normalized LDA projections were nearly point-like and linearly separable with a large margin.Other methods' PCA projections spanned two dimensions, and their normalized LDA features were linearly non-separable; FOCA also had orders-of-magnitude larger generalized eigenvalues and the lowest class-1-vs-rest test error in Table 1.Source: §4.3 · pp. 7–8, Figure 4 · p. 8, Table 1 · p. 8

5. Conclusion

Naïve joint optimization can tightly couple a feature extractor and classifier, making the classifier difficult to replace without harming test performance. FOCA avoids explicit adaptation to a particular classifier; under special conditions its features form class-separable point-like distributions, with real-data experiments offering supporting evidence.

5. Conclusion

  • Naïve joint optimization can tie feature-extractor and classifier parameters so tightly that replacing the classifier degrades test performance.Source: §5 · p. 8
  • FOCA is designed to break unwanted inter-layer co-adaptation by producing features that do not explicitly adapt to a particular classifier.Source: §5 · p. 8
  • Under special conditions, a mathematical proposition guarantees class-separable point-like feature distributions, and real-data experiments under more general conditions provide supportive evidence.Source: §5 · p. 8

Related papers

Loading 1906.01150v1…