Source-linked AI summary
What makes instance discrimination good for transfer learning?
Nanxuan Zhao, Zhirong Wu, Rynson W. H. Lau, Stephen Lin
TL;DR
The paper asks why instance discrimination transfers better than supervised pretraining and what representations are learned. Through analyses of augmentations, dataset semantics, reconstruction, and task alignment, it finds that low- and mid-level representations matter most, while intra-class invariance can weaken transfer; an exemplar-based supervised scheme improves generalization.
Problem
The work investigates why contrastive instance-discrimination pretraining can outperform supervised pretraining for downstream transfer and what knowledge transfers from these models.
Method
The paper analyzes augmentation effects, pretraining-data semantics, representation reconstruction, task misalignment, and a supervised exemplar-based alternative.
Results
Transfer depends primarily on low- and mid-level representations; supervised intra-class invariance can increase task misalignment, while exemplar-based pretraining improves downstream generalization.
Takeaways & Limitations
Supervised labels can be exploited more effectively by preserving variation among same-category instances instead of explicitly enforcing their invariance.
Abstract
from arXiv · showhide
Contrastive visual pretraining based on the instance discrimination pretext task has made significant progress. Notably, recent work on unsupervised pretraining has shown to surpass the supervised counterpart for finetuning downstream applications such as object detection and segmentation. It comes as a surprise that image annotations would be better left unused for transfer learning. In this work, we investigate the following problems: What makes instance discrimination pretraining good for transfer learning? What knowledge is actually learned and transferred from these models? From this understanding of instance discrimination, how can we better exploit human annotation labels for pretraining? Our findings are threefold. First, what truly matters for the transfer is low-level and mid-level representations, not high-level representations. Second, the intra-category invariance enforced by the traditional supervised model weakens transferability by increasing task misalignment. Finally, supervised pretraining can be strengthened by following an exemplar-based approach without explicit constraints among the instances within the same category.
1 INTRODUCTION
The paper argues that transfer success from instance discrimination depends mainly on low- and mid-level representations, while supervised intra-category invariance can misalign pretraining with downstream tasks. It studies these mechanisms and proposes using labels without forcing same-category instances together.
- Contrastive pretraining can outperform supervised pretraining for downstream transfer, raising questions about why labels may be unhelpful or harmful.MoCo achieved AP 46.6 versus 42.4 on PASCAL VOC object detection.
- Both pretraining approaches benefit from image augmentations, but contrastive models rely on low-level augmentations more strongly.With proper augmentations, supervised pretraining may still prevail on COCO object detection and Cityscapes semantic segmentation.
- Transfer performance is largely unaffected by whether pretraining data share high-level semantics with the target data, whereas synthetic data with mismatched low-level properties reduce transfer.Unsupervised pretraining on a much smaller dataset only marginally degrades transfer performance.
- The evidence indicates that low-level and mid-level representations, rather than high-level representations, are primarily transferred.This conclusion follows from experiments varying pretraining data semantics and image properties.
- Supervised pretraining is more vulnerable to localization error because it models discriminative object parts rather than the holistic information needed for accurate localization and delineation.Contrastive representations pretrained to discriminate instances support more holistic modeling.
- The paper attributes weaker supervised transfer to intra-class invariance, which discards instance-specific variation and increases task misalignment.Its exemplar-based supervised method removes explicit constraints among same-category positives while separating true negatives, improving ImageNet classification and downstream transfer.
- These conclusions also align with results from few-shot image recognition and facial landmark prediction.The paper reports that both additional transfer scenarios support the conclusions from its earlier analyses.
2 AN ANALYSIS FOR VISUAL TRANSFER LEARNING
The analysis finds that transfer performance depends strongly on augmentation and low-level information, while supervised pretraining can lose transferability through semantic and intra-category constraints. Contrastive representations preserve more holistic image information and support stronger transfer in several settings.
- Image augmentations: Image augmentations improve both pretraining approaches, but contrastive models benefit substantially more across ImageNet classification and transfer tasks.Supervised models can perform well with horizontal flipping alone, while Gaussian blurring slightly harms them.
- Dataset semantics: With proper augmentations, contrastive models outperform supervised models on PASCAL VOC but underperform them on COCO and Cityscapes.This pattern suggests that COCO detection and Cityscapes segmentation may benefit more from high-level information than PASCAL VOC detection.
- Dataset semantics: Contrastive transfer is relatively unaffected by pretraining image semantics, whereas supervised transfer depends on semantic alignment between pretraining and target data.Pretraining on faces and scenes achieves almost the same transfer results as ImageNet, while synthetic data with different low-level statistics causes a substantial drop.
- Task misalignment and information loss: Supervised pretraining produces more localization errors than contrastive pretraining in PASCAL VOC detection.The analysis compares false-positive categories using the detection toolbox and identifies poor localization as the more frequent supervised error.
- Task misalignment and information loss: Feature inversion reconstructs contrastive representations more completely, whereas supervised representations lose information across large image regions.Contrastive reconstructions preserve spatially complete content, while supervised features emphasize discriminative object parts rather than whole images.
- Task misalignment and information loss: Contrastive models reconstruct images at the correct scale and location, indicating that instance discrimination preserves information about spatial and scale factors.The models still reconstruct hue inaccurately, consistent with the broad space of color augmentations.
3 A BETTER SUPERVISED PRETRAINING METHOD
The paper proposes exemplar-based supervised pretraining that uses labels to exclude true negatives without imposing explicit similarity constraints on same-category instances.
- Traditional supervised learning minimizes intra-class variation, emphasizing discriminative regions while losing information from other regions.
- The proposed method trains an individual classifier for each instance, separating it from negative examples in an exemplar-SVM-inspired framework.
- The exemplar-based loss uses semantic labels to filter true negatives rather than discriminating against every other instance.
- 68.9% ImageNet classification performance remains below the 77.3% traditional supervised baseline, leaving room to improve both classification and transfer.
- Filtering true negatives consistently improves ImageNet classification and downstream transfer, unlike traditional supervised learning, which improves ImageNet performance while compromising transfer.
4 IMPLICATIONS FOR OTHER TRANSFER LEARNING SCENARIOS
The paper tests exemplar-based pretraining beyond ImageNet transfer in few-shot recognition and facial landmark prediction, where task alignment determines the relative transfer performance of pretraining methods.
- The additional experiments examine few-shot recognition and facial landmark prediction to test whether supervised semantic overfitting affects other transfer scenarios.
- Few-shot recognition: The few-shot setup pretrains on 64 base classes, then evaluates a fixed representation with a linear classifier on five novel classes using 1-shot or 5-shot data.
- Few-shot recognition: In 5-way few-shot recognition, MoCo-v2 is weaker on base and novel classes, while Exemplar-v2 outperforms supervised baselines on novel classes.
- Facial landmark prediction: For facial landmark prediction, the pretext task is face identification on CelebA and the downstream task predicts five landmarks on MAFL.
- Facial landmark prediction: MoCo-v2 outperforms supervised pretraining for landmark transfer, while Exemplar-v2 maintains MoCo-v2's performance because identity provides little pose information.
5 RELATED WORKS
Related work establishes supervised ImageNet pretraining as broadly useful, while newer studies analyze task relations and unsupervised contrastive transfer. This paper focuses on why instance discrimination improves transfer.
- Supervised ImageNet pretraining supports applications ranging from detection and segmentation to texture synthesis and style transfer.
- Taskonomy and related work study task similarities, cooperation, and competition to understand transfer learning and task misalignment.
- This work analyzes unsupervised contrastive pretraining with instance discrimination because prior studies reported superior transfer to detection, segmentation, and pose estimation but left the explanation unclear.
6 CONCLUSION
The study finds that transfer depends mainly on low- and mid-level representations, while instance discrimination preserves broader image information than supervised pretraining. Supervised transfer can improve when labels are used without enforcing invariance among same-category instances.
- Low- and mid-level representations, rather than high-level representations, are mainly transferred when networks are finetuned end-to-end.Contrastive representations learned on different datasets share low- and mid-level features that adapt quickly to target problems.
- Contrastive output features remain overfit to the high-level semantics of their pretraining datasets.
- Instance discrimination learns rich information for pixel reconstruction through a holistic encoding of the entire image.
- Supervised intra-class invariance discards patterns uninformative for classification, potentially losing information useful under task misalignment.An exemplar-SVM-style scheme based on instance discrimination improves downstream generalization.
- Experiments on two additional transfer scenarios support the generalization ability of these findings.
A EFFECTS OF PRETRAINING AND FINETUNING ITERATIONS
Longer pretraining improves detection transfer for both supervised and unsupervised models, while their optimization dynamics differ. Supervised pretraining can continue improving with many epochs but does not uniformly surpass MoCo across detection metrics.
- Longer pretraining consistently improves detection transfer for both supervised and unsupervised models.The study reports no overfitting issue for either method in this setting.
- Unsupervised pretraining converges faster during pretraining, whereas supervised models converge faster initially during detection finetuning but may not converge optimally.
- Supervised pretraining continues improving through 800 epochs but may overfit on ImageNet classification.
- After extended supervised pretraining, MoCo still leads on detection AP and AP75, while supervised pretraining leads on AP50.
- The section evaluates intermediate pretraining and finetuning checkpoints and reports longer supervised pretraining for PASCAL VOC detection transfer.
C EFFECTS OF DATASET SEMANTICS ON PRETRAINING
The study reports transfer experiments using models pretrained on varied datasets and reconstructs network inputs with an encoder-decoder architecture. The supplied passages primarily document the dataset-study setup and reconstruction implementation.
- The reconstruction network uses an encoder-decoder architecture based on the original deep image prior.
- The reconstruction architecture includes convolutional, downsampling, batch-normalization, leaky-ReLU, and upsampling layers, with stride 2 for sampling layers.
- The reconstruction input is fixed uniform noise, and each image is optimized for 3000 Adam iterations at learning rate 0.001.
D.2 EVALUATING RECONSTRUCTIONS BY PERCEPTUAL METRICS
The study evaluates reconstruction quality with perceptual distance on 1000 ImageNet validation images. Reconstructions from MoCo features are generally closer to the original images than those from supervised features.
- Perceptual distance is measured between reconstructions and input images using a deep-learning metric with SqueezeNet.The evaluation samples one image per class from the ImageNet validation set.
- The reconstruction evaluation is illustrated with example dataset images and a perceptual-distance figure.
- 5.59 is the average perceptual distance for MoCo reconstructions, compared with 6.43 for the supervised network.
- MoCo reconstructions are generally closer to the original images than supervised reconstructions.
E MORE RESULTS ON EXEMPLAR-BASED SUPERVISED PRETRAINING
The paper reports full transfer performance for exemplar-based supervised pretraining and studies its parameter sensitivity against MoCo. Table 9 describes consistent improvements over MoCo baselines when labels are used without explicit constraints on positives.
- The paper reports full transfer performance of exemplar-based supervised pretraining in Table 9.
- An ablation study evaluates parameter k and τ for MoCo and Exemplar pretraining using ImageNet linear readout.
- Exemplar-based supervised pretraining shows consistent improvements over MoCo baselines while using labels without explicit constraints on positives.
F ADDITIONAL RESULTS OF DIAGNOSING DETECTION ERROR
The paper analyzes detection errors across 20 VOC07 object categories using pie charts of four false-positive types. False positives are selected from the top-ranked detections for each category.
- The analysis covers 20 object categories on the VOC07 test set.
- For each category, a pie chart shows the distribution of four error types among top-ranked false positives.
- The number of selected false positives equals the number of ground-truth objects in each category.