Source-linked AI summary
ViTAMINS: An Empirical Study of Training Self-Supervised Vision Transformers with Synthetic Hard Negatives
Nikos Giakoumoglou, Andreas Floros, Kleanthis-Marios Papadopoulos, Tania Stathaki
TL;DR
Contrastive learning for vision transformers has received less attention despite its simplicity and efficiency, motivating stronger negative-sampling strategies. ViTAMINS integrates on-the-fly synthetic hard negatives into existing contrastive frameworks and evaluates the resulting representations broadly. The method improves ImageNet accuracy, transfer performance, and emergent semantic properties, including strong k-NN classification and segmentation-related behavior.
Problem
Contrastive learning for vision transformers remains underexplored despite its simplicity and efficiency, prompting the question of whether better negative sampling can strengthen representations and emergent properties.
Method
ViTAMINS integrates synthetic hard negatives generated on-the-fly into existing transformer-based contrastive learning frameworks.
Results
ViTAMINS improves ImageNet linear accuracy, diverse downstream performance, semantic segmentation properties, attention boundary alignment, and k-NN classification across evaluations.
Takeaways & Limitations
Synthetic hard negatives can strengthen contrastive vision-transformer representations and produce emergent semantic properties without DINO’s complex procedures.
Abstract
from arXiv · showhide
We introduce ViTAMINS, a method that integrates synthetic hard negatives into unsupervised vision transformer pretraining to improve representation quality. Our approach is thoroughly benchmarked on ImageNet and transfer learning, image retrieval, copy detection, and image, video segmentation tasks. Notably, our proposed negatives give rise to emergent properties, where learned representations contain explicit information about the semantic content of an image and serve as excellent classifiers (up to +11.3% over baselines). ViTAMINS achieves these benefits through simple modifications to existing contrastive frameworks and outperforms competing methods while being more resource efficient, e.g., our ViT-B surpasses V-JEPA with ViT-L. Our findings motivate reconsidering contrastive learning as a simpler yet powerful alternative to dominant generative and self-distillation approaches.
1. Introduction
Vision-transformer self-supervised learning includes contrastive, self-distillation, and clustering-based approaches, but contrastive learning remains a simple and efficient option. ViTAMINS addresses whether synthetic hard negatives can strengthen contrastive representations and emergent properties through a lightweight modification.
- Self-supervised learning categories: Contrastive, self-distillation, and clustering-based methods use different mechanisms to avoid representational collapse in vision transformers.Contrastive methods compare augmented views and instances, self-distillation matches teacher and student embeddings, and clustering-based methods optimize clustering objectives.
- Emergent properties: Joint-embedding vision transformers can develop explicit semantic segmentation information, including attention aligned with object boundaries.These emergent behaviors are reported more clearly in self-supervised joint-embedding models than in supervised transformers or ConvNets.
- Motivation: Contrastive learning remains competitive and often surpasses generative methods in linear probing while retaining simplicity and efficiency.Unlike self-distillation, contrastive learning explicitly uses negatives to define representation boundaries.
- ViTAMINS: ViTAMINS introduces synthetic hard negatives generated on-the-fly into transformer-based contrastive learning.The method adapts established synthetic negative-generation approaches rather than adding complex architectures or training schemes.
- Empirical improvements: 73.1% and 77.1% top-1 ImageNet linear accuracy are achieved with ViT-S/16 and ViT-B/16, respectively, while Swin-T and Swin-S reach 75.4% and 78.0%.The reported gains are evaluated against training without synthetic hard negatives or without negatives.
- Empirical improvements: ViTAMINS also improves diverse downstream tasks and yields semantic segmentation features, precise object-boundary attention maps, and 73.3% top-1 k-NN accuracy with ViT-B/16.These findings are presented as emergent properties of the learned representations.
2. Related Work
Related work frames joint-embedding methods as approaches that prevent representational collapse through contrastive negatives, teacher–student asymmetry, or clustering objectives. Contrastive learning uses positive and negative relationships, typically optimized with InfoNCE or related losses and supplied by batches or memory banks.
- Joint-embedding methods: Joint-embedding methods map augmented views into a shared space while using distinct mechanisms to avoid representational collapse.Contrastive methods use large batches or momentum memory banks, whereas self-distillation uses asymmetric architectures, momentum updates, and stop-gradient operations.
- Contrastive learning: Contrastive learning treats each image as its own class by bringing positive pairs closer and pushing anchors away from negative samples.Training commonly uses InfoNCE or related losses, with negatives drawn from large batches or memory banks.
3. Methodology
ViTAMINS extends contrastive vision-transformer pretraining with synthetic hard negatives generated from challenging memory-queue features. It combines these negatives with standard contrastive objectives and momentum-encoded representations.
- ViTAMINS framework: ViTAMINS generates informative synthetic negatives within existing contrastive learning frameworks for vision transformers.The method adapts established synthetic-negative approaches without requiring complex architectures or training schemes.
- Representation pipeline: Two augmented views of each image produce query and key representations through separate online and target encoder-projector branches.The query additionally passes through a predictor, while all outputs are ℓ2-normalized.
- Negative sampling: A memory queue stores K = 4096 target-branch features from distinct images as negative samples.The queue requires memory O(K · d), where d is the embedding dimension.
- Synthetic hard negatives: Hardness is defined by query-negative similarity ℓ(ni) = q⊤·ni, and the top-N hardest queue negatives are selected for synthesis.Synthetic negatives are ℓ2-normalized before entering the representation space.
- Synthetic hard negatives: Six transformations generate synthetic negatives, including interpolation, extrapolation, mixup, noise injection, and adversarial perturbation.The complete synthetic set is denoted S and is incorporated alongside memory-based negatives.
- Contrastive objective: The combined memory and synthetic negative set contributes to the InfoNCE denominator, with temperature τ = 0.2.When S = ∅, the method reduces to the standard InfoNCE loss used by MoBY and MoCo-v3.
4. Main Results
ViTAMINS representations perform strongly across ImageNet classification, retrieval, copy detection, segmentation, and downstream transfer tasks. The method improves transformer representations across both classification and dense prediction evaluations.
- ImageNet evaluation: ViTAMINS outperforms competing methods in ImageNet linear and k-NN evaluation, including larger I-JEPA, V-JEPA, and iBOT models.It also surpasses MoBY and BYOL, which do not use synthetic hard negatives or negatives, respectively.
- Retrieval and copy detection: ViTAMINS achieves competitive image retrieval, with smaller models matching or exceeding larger architectures on revisited Oxford and Paris datasets.Retrieval uses frozen features with direct k-NN evaluation.
- Retrieval and copy detection: ViTAMINS surpasses DINO on copy detection measured by mean average precision on the Copydays strong subset.The evaluation follows the established protocol for copy detection.
- Semantic layout: ViTAMINS yields strong video instance segmentation on DAVIS-2017 using nearest-neighbor matching between consecutive frames without training.The evaluation measures spatial coherence on frozen features.
- Semantic layout: ViTAMINS produces sharper attention maps than DINO while separating foreground from background and capturing fine object details without DINO’s additional tricks.The visualizations include [CLS]-to-patch attention and patch self-attention.
- Transfer learning: ViTAMINS consistently improves COCO detection and instance segmentation and achieves the best mIoU across transformer architectures on ADE20K.COCO reports bounding-box and mask mean average precision, while ADE20K reports mean intersection over union.
- Transfer learning: In transfer classification, ViTAMINS wins on 8/11 datasets with ViT-S and 9/11 with Swin-T in linear probing, and on all 5 datasets in fine-tuning.The evaluated datasets span small images, fine-grained recognition, landscapes, and textures.
5. Ablation Study of ViTAMINS
The ablations test synthetic-negative strategies, regularization, implementation choices, and contrastive hyperparameters. Results support complementary synthetic transformations, asymmetric drop path, cooldown, and robust integration with existing pipelines.
- Synthetic hard negatives strategies: Combining all six synthetic-negative strategies yields the highest performance, exceeding the no-synthetic-negative baseline by +0.8% and +0.7%.Individual strategies vary in effectiveness, with S3 most impactful and S1 next; their combination provides complementary benefits.
- Drop path regularization: Higher online-encoder drop path with no target-encoder dropout performs best, outperforming both no regularization and symmetric drop path.The authors attribute this likely to more robust online representations while preserving target stability.
- Ablation setup: The ablations evaluate queue size, temperature, momentum, and online drop path rate using ViT-S and Swin-T on ImageNet-100 after 100 epochs.The default settings are K = 4096, τ = 0.2, mstart = 0.99, and online drop path rate = 0.2.
- Implementation tricks: SynCo cooldown, which disables synthetic negatives during the last 100 epochs, achieves optimal results, while fixed patch embeddings are unnecessary.Applying synthetics without warmup hurts because early representations are not yet sufficiently developed.
- Other hyperparameters: Performance remains stable across a wide range of queue sizes, temperatures, and momentum values using default MoBY hyperparameters.This supports adoption without architectural changes, extensive hyperparameter retuning, or additional hyperparameter-search overhead.
6. Conclusion
ViTAMINS improves vision-transformer representations by adding synthetic hard negatives to self-supervised contrastive learning. It yields stronger accuracy, transfer, retrieval, segmentation, and attention behavior than standard contrastive baselines, while avoiding DINO’s complex procedures.
- Overall findings: Synthetic hard negatives improve vision-transformer representations across ImageNet accuracy, k-NN, downstream tasks, attention maps, and video object segmentation.The method achieves strong video segmentation despite no video training.
- Comparison with DINO: Under identical 300-epoch regimes without multi-crop, ViTAMINS outperforms reproduced DINO across linear probing, k-NN, retrieval, copy detection, video segmentation, and dense prediction.The comparison covers all listed tasks in Tables 1 and 3 to 6.
- Emergent properties: Semantic segmentation properties arise in contrastive learning and are strengthened by synthetic negative generation, rather than being exclusive to self-distillation.The conclusion identifies sharper object-boundary alignment as an additional benefit.