No One Representation to Rule Them All: Overlapping Features of Training Methods

Raphael Gontijo-Lopes; Yann Dauphin; Ekin D. Cubuk

Source-linked AI summary

No One Representation to Rule Them All: Overlapping Features of Training Methods

Raphael Gontijo-Lopes, Yann Dauphin, Ekin D. Cubuk

arXiv:2110.12899v3Oct 20, 2021cs.LG

TL;DR

High-performing supervised models often make similar predictions, raising the question of whether different training methods learn similar functions and limiting expected ensemble diversity. The paper systematically studies models across training differences and finds that greater methodological divergence produces uncorrelated errors, specialized behavior, and overlapping feature sets, improving ensembles and downstream performance.

Problem
High-performing models often make similar predictions, suggesting shared biases and limiting ensemble benefits and the practical value of lower-accuracy models.
Method
The paper conducts a large-scale empirical study of 82 models spanning hyper-parameters, architectures, frameworks, objectives, and datasets.
Results
Models with more divergent training methodologies produce increasingly uncorrelated errors, improve ensemble accuracy, and learn overlapping but not supersetting representations.
Takeaways & Limitations
Different training methodologies can produce useful specialization, allowing lower-accuracy models to improve high-accuracy models and combined representations to boost downstream performance.
Takeaways & Limitations
Existing training methodologies did not produce a single best or supersetting representation, leaving open whether such an objective exists.

Abstract

from arXiv · show

Despite being able to capture a range of features of the data, high accuracy models trained with supervision tend to make similar predictions. This seemingly implies that high-performing models share similar biases regardless of training methodology, which would limit ensembling benefits and render low-accuracy models as having little practical use. Against this backdrop, recent work has developed quite different training techniques, such as large-scale contrastive learning, yielding competitively high accuracy on generalization and robustness benchmarks. This motivates us to revisit the assumption that models necessarily learn similar functions. We conduct a large-scale empirical study of models across hyper-parameters, architectures, frameworks, and datasets. We find that model pairs that diverge more in training methodology display categorically different generalization behavior, producing increasingly uncorrelated errors. We show these models specialize in subdomains of the data, leading to higher ensemble performance: with just 2 models (each with ImageNet accuracy ~76.5%), we can create ensembles with 83.4% (+7% boost). Surprisingly, we find that even significantly low-accuracy models can be used to improve high-accuracy models. Finally, we show diverging training methodology yield representations that capture overlapping (but not supersetting) feature sets which, when combined, lead to increased downstream performance.

1 INTRODUCTION

The paper revisits whether high-performing models trained with different methodologies necessarily learn similar functions and prediction behaviors. It studies diverse models to test whether training methodology affects representations, errors, ensembling, and downstream features.

High-performing supervised models often make similar predictions, suggesting they share biases despite capturing diverse data features.
Correlated errors limit ensemble gains, while similar features may omit information needed for downstream transfer.
The central question is how training methodology affects learned representations and prediction behavior.
The study evaluates 82 models across hyper-parameters, architectures, objectives, frameworks, and datasets, including contrastively trained and large-scale models.
Models with more divergent training methodologies produce increasingly uncorrelated errors and can specialize in different data subdomains.

2 RELATED WORK

Prior work established the value of accurate models with independent errors, but often found similar behavior among high-performing models. Recent contrastive learning and large-scale datasets expand the set of high-performing training methodologies motivating this study.

Ensemble quality depends on combining accurate models that make independent errors, while diverse solutions can also quantify epistemic uncertainty.
Bootstrapping on data subsets did not produce deep ensembles more accurate than a single model trained on the full dataset.
High-performing techniques often yielded similar prediction behavior, with high-accuracy models tending to dominate lower-accuracy models.
This similarity was associated with the historical concentration of strong image-classification methods around supervised ImageNet training and similar pretraining distributions.
Contrastive learning, new architectures, augmentation, regularization, losses, and pretraining schemes broadened the available high-performing methodologies.

3 METHOD

The study categorizes model pairs by progressively larger training differences, fixes a ResNet-50 base, and uses calibrated, accuracy-matched models to probe whether methodology produces useful prediction diversity. Ensembling serves as an analysis tool rather than a state-of-the-art optimization target.

3.1 MODEL CATEGORIZATION: Model pairs are categorized as reinitializations, hyper-parameters, architectures, frameworks, or datasets, ordered by increasing training-methodology divergence.
3.1 MODEL CATEGORIZATION: The categories can overlap because changing architecture may also change its training hyper-parameters, and each category is defined relative to a fixed base model.
3.2 MODEL SELECTION: The study collects representations and predictions from 82 models, using RandAugment-trained ResNet-50 as the base and covering five model categories.
3.2 MODEL SELECTION: Models are temperature-scaled and generally restricted to 74–78% ImageNet accuracy so observed effects reflect training-methodology divergence rather than intrinsic accuracy differences.
3.3 ENSEMBLING: Ensembling is used to probe when training methodology yields uncorrelated, useful predictions rather than to maximize state-of-the-art accuracy.
3.3 ENSEMBLING: Error inconsistency measures test examples where one model is correct and the other is incorrect, and increases as training methodologies diverge from Reinit toward Dataset.

4 RESULTS

Diverging training methodologies produce increasingly uncorrelated errors and specialized models, improving ensemble accuracy and efficiency. These models also provide complementary representations and downstream benefits, while lower-accuracy models can improve stronger ensembles when sufficiently diverse.

4.1 AS TRAINING METHODOLOGY DIVERGES, ERRORS BECOME MORE UNCORRELATED: As training methodologies diverge, models make increasingly uncorrelated errors, creating more opportunities for ensemble improvements.Error inconsistency measures test examples where one model is correct and the other is incorrect; it increases from reinitialization toward dataset differences.
4.2 AS UNCORRELATED ERRORS INCREASE, SO DOES ENSEMBLE EFFICIENCY: 83.4% top-1 accuracy results from ensembling ResNet-50 (76.5%) with ALIGN-ZeroShot (75.5%), a boost of nearly 7%.The models lie within the restricted 74–78% accuracy range, so the relative improvement translates into absolute accuracy improvement.
4.2 AS UNCORRELATED ERRORS INCREASE, SO DOES ENSEMBLE EFFICIENCY: Ensemble efficiency also increases with error inconsistency because divergently trained models convert inconsistent errors into correct predictions at higher rates.The improvement therefore reflects both more opportunities for correction and more efficient conversion of those opportunities.
4.3 DISSIMILAR TRAINING METHODOLOGIES CREATE SPECIALIZED MODELS: Divergent training makes models specialize in different data subdomains, with greater relative confidence on examples where each model alone is correct.Reinitializations show little confidence asymmetry, whereas architecture and dataset differences produce stronger specialization.
4.4 MODEL SPECIALIZATION DEPENDS ON TRAINING METHODOLOGY: Dataset differences produce class-linked specialization: CLIP-S favors anthropogenic images, whereas ResNet-50 favors nature images.Architecture differences create more specialization than reinitialization, but without clear class-specific correlation.
4.5 DIFFERENT ENOUGH LOWER-ACCURACY MODELS CAN IMPROVE ACCURACY: 86.66% ensemble accuracy is achieved by combining BiT-1k (82.85%) with only lower-accuracy models capped at 77.44%.Across four high-accuracy starting models, the most beneficial additions are trained with different losses and/or datasets.
4.7 DOWNSTREAM TASK PERFORMANCE: Diverse training improves downstream transfer: the highest-accuracy ImageNet model is not best on Pascal VOC, and the strongest combinations use differently trained models.CLIP-S has the greatest prediction diversity among analyzed models, and diversity can better indicate downstream performance than individual accuracy.

5 CONCLUSION

Diverse training methodologies produce models with uncorrelated errors and overlapping but non-supersetting features, enabling stronger ensembles and downstream representations. These findings motivate continued exploration of new objectives, frameworks, and datasets.

Diverse optimization objectives and datasets produce uncorrelated errors, enabling more efficient ensembles because models specialize in different data subdomains.The paper also finds that even individually low-accuracy models can contribute to high-accuracy ensembles.
Diverse models learn overlapping but not supersetting feature sets, and combining their embeddings can improve downstream performance.
Behavior and feature diversity show that high-accuracy representations are not strict supersets of low-accuracy representations.
The results leave open whether an objective could produce a single best or supersetting representation, since existing methodologies have not found one.
The study encourages practitioners to develop new training methodologies, frameworks, and datasets to expand learned features and model behaviors.

A.1 CONCURRENT WORK

Concurrent work likewise suggests that optimizing a single objective can limit downstream feature diversity, while explicitly mixing training sources or optimizing prediction diversity can improve out-of-distribution performance.

Upstream accuracy can saturate downstream performance when models are trained on the same objective, consistent with limited feature diversity from single-objective optimization.
Diverse ensembles mixing losses and datasets, and adversarially diversified predictions, have been associated with improved out-of-distribution performance.

A.2 MODEL CATEGORIZATION

The model categories are hierarchical rather than mutually exclusive: changing a higher-level training choice also changes lower-level choices needed to optimize that setting.

Model categories form supersets because changing architecture also changes its training hyper-parameters, while changing frameworks can alter both hyper-parameters and architecture.
Changing dataset scale or source can therefore accompany changes in other training choices rather than isolating a single factor.

A.3 MODEL LIST PER CATEGORY

The study organizes models by training methodology and records their accuracy, error inconsistency, and ensemble behavior, while separately listing high-accuracy, low-accuracy, and excluded models.

Table 1 lists the main-analysis models with training methodologies, calibration temperatures, ImageNet accuracy, error inconsistency, and ensemble accuracy relative to ResNet-50.
High-accuracy models used for downstream and ensemble analyses are listed separately as a stronger base for ensembling.
Low-accuracy models remain included when their training methodologies differ substantially from the typical model, because they can still be useful.
Other trained models outside the target 74-78% ImageNet accuracy range are excluded from the main analysis.

A.4 ADDITIONAL ERROR ANALYSIS

Different training methodologies increase prediction diversity, but ensemble gains depend on converting examples solved by only one model.

As training methodologies diverge, examples where both models are wrong or correct decrease, so ensembles must convert more one-model-correct examples to benefit.The added diversity is useful only if ensembles efficiently convert the increased one-model-correct cases.

A.5 L-BFGS DETAILS

The appendix details representation-classifier training and provides supplementary analyses of model diversity, interpolation, specialization, and representation similarity.

L-BFGS details: Linear classifiers are trained with L-BFGS on normalized pre-logit representations, using random or diversity-selected dimensions after concatenation.The setup follows Kornblith et al. (2019) and omits augmentation when training linear heads.
Models and analyses: Table 1 covers models spanning supervised and contrastive methods, with individual accuracies and ensemble results reported alongside model categories.The main model set is restricted to a 74–78% accuracy range.
Error analysis: As training methodology diverges, inconsistent errors increase while examples jointly correct or jointly incorrect decrease, indicating greater prediction diversity.The dataset-divergence category has more inconsistent-error examples than jointly correct examples.
Interpolation: Equal-weight logit averaging performs near the optimum when interpolating ensemble members, consistent with calibration making model confidences comparable.The interpolation analysis places the best parameter near 0.5.
Specialization and representations: CLIP-S and SimCLR specialize along different anthropogenic and nature-related class ranges, while CKA and error consistency are not well correlated.The CKA result suggests representation geometry does not directly track prediction-error similarity.