Source-linked AI summary
CoViT: Instance-Correspondence Contrastive Learning for Vision Transformer
Yisen Wang, Zhirong Wu, Limin Wang
TL;DR
ViTs learn semantic representations but can struggle to distinguish different instances of the same category, limiting their use in instance-level tasks. CoViT mines instance-aware triplets using attention maps and embedding similarities; the paper reports gains across instance-level tasks and ViT backbones.
Problem
ViTs can represent different objects of the same category similarly, limiting their use in instance-level tasks that require precise object discrimination.
Source: §1 · p. 1Method
Results
Takeaways & Limitations
Experiments report consistent gains across several ViT backbones and downstream frameworks, demonstrating the value of explicit instance modeling for instance-level vision tasks.
Source: §5 · p. 9Future work will improve robustness to occlusion and extend CoViT to end-to-end self-supervised pre-training.
Source: §5 · p. 9
Abstract
from arXiv · showhide
Vision Transformers (ViT) excel in semantic understanding but fail to discriminate between object instances (e.g., identical embeddings for two dogs), limiting their use in instance-level tasks such as object detection and instance segmentation. We propose Contrastive Vision Transformer (CoViT), a self-supervised learning framework that injects instance-awareness into ViT through geometry-guided contrastive learning. CoViT uniquely coordinates ViT's attention maps and embeddings by constructing triplets: (1) Attention-guided masking: Refine multi-head attention via adaptive thresholding and morphological operations to generate instance masks, identifying foreground anchors; (2) Hardest contrastive mining: For each anchor, computing pairwise embedding similarities to select the intra-instance hardest positive (least similar patch within its mask) and inter-instance hardest negative (most similar patch from other instances), with intra-instance regions masked during negative search. These triplets drive a contrastive loss that simultaneously compresses intra-instance variance and expands inter-instance margins, forcing ViT to discern subtle geometric and appearance differences between instances. CoViT consistently achieves stable performance gains of over 2 AP points across multiple instance-level perception tasks by using ViT as backbone architecture. Notably, CoViT requires no extra decoders or labels, demonstrating that a pure ViT can learn instance-aware representations via inherent attention priors and targeted contrastive constraints. Code and models will be released.
1 Introduction
CoViT targets ViTs’ difficulty distinguishing instances within the same semantic category, which limits their utility in instance-level tasks. It mines instance-aware triplets from attention maps and embedding similarities, and reports gains on instance-level tasks while preserving semantic representation strength.
1 Introduction
- ViTs can produce roughly the same representation for different objects of the same category, restricting their utility in instance-centric applications such as instance segmentation.The introduction gives two dogs as an example of same-category instances.Source: §1 · p. 1
- Semantic-focused pre-training objectives learn invariant features but can suppress instance-specific cues, leaving attention focused on broad semantic relationships rather than boundaries between adjacent instances.The text names masked-patch reconstruction and consistency between augmented views as examples.Source: §1 · pp. 1–2
- CoViT uses refined attention maps to approximate instance regions and embedding similarities to mine intra-instance hard positives and inter-instance hard negatives for targeted contrastive optimization.Its dual objectives reduce distance to hard positives and increase the margin from hard negatives.Source: §1 · p. 2
- CoViT improves DINO-Base mask AP on COCO instance segmentation by 3.2% and few-shot instance recognition accuracy on PASCAL VOC by 12.7%.The framework is also reported to preserve the model’s original semantic representation strength.Source: §1 · p. 2
- The work reports versatility across instance-level tasks and ViT architectures, seeking to reconcile pre-trained semantic representations with instance-sensitive adaptation.The stated contributions include analyzing the limitation, proposing attention-and-similarity-guided triplet mining, and empirically validating it across tasks and architectures.Source: §1 · p. 2
2 Related Work
Related work describes a gap between semantic-focused ViT pre-training and instance-level discrimination, alongside contrastive and architectural approaches with distinct trade-offs. These directions motivate CoViT’s use of attention refinement and adaptive triplet mining without architectural alteration.
2 Related Work
- Existing self-supervised ViT paradigms are described as semantically oriented, with objectives that align poorly with discrimination between semantically identical entities.The discussion cites MAE [3] and DINO [4] [5] among the approaches and notes ViTDet [8] as demonstrating detection potential.Source: §2 · p. 2
- Contrastive methods such as MoCo [9] [10] [11] and SimCLR [12] [13] use negative sampling, while non-contrastive DINO [4] avoids it but is described as sacrificing local instance details.The passage says random or class-agnostic negatives can introduce noisy training signals; MaskContrast is described as hampered by semantic biases in attention, especially with occluded or overlapping instances.Source: §2 · pp. 2–3
- Architectural alternatives include task-specific designs and multimodal training, while the text argues that architectural modification can come at a cost to generalizability.The passage also notes that MultiMAE found joint training with auxiliary modalities can enhance instance discrimination.Source: §2 · p. 3
- These directions motivate CoViT’s entropy-guided morphological refinement of attention maps and adaptive triplet mining to extract instance regions and contrast intra-instance variance with inter-instance similarity.The proposed strategy is framed as instance discrimination without architectural alteration.Source: §2 · p. 3
3 Approach
CoViT uses ViT attention to localize candidate instances and embedding similarities to mine hard within-instance and cross-instance correspondences. A contrastive objective uses these triplets to promote within-instance cohesion and between-instance separation.
3.1 Overview
- CoViT alternates attention-based instance localization and embedding-based correspondence mining, then optimizes the resulting triplets to reduce within-instance variance and increase separation between instances.The framework uses ViT’s own attention and patch embeddings without manual instance-level annotations.Source: §3.1 · p. 3
3.2 Binary Attention Map Generation
- Query-specific thresholds binarize attention maps, and dilation followed by erosion connects fragmented foreground responses before connected components isolate candidate instances.Adaptive thresholds account for variation in attention magnitude across queries and images.Source: §3.2 · pp. 4–5
3.3 Correspondence Mining
- Within each instance, CoViT selects the least-similar patch as a hard positive and the most-similar patch from outside it as a hard negative, subject to mask and margin constraints.The mined patches must lie within refined masks to avoid edge artifacts.Source: §3.3 · pp. 5–6
3.4 Contrastive Learning
- The hybrid contrastive objective pulls an anchor toward intra-instance patches while separating it from confusing external patches, addressing heterogeneous positives and spatially localized negatives.The intra-instance term averages over all positive patches and includes the mined hard negative in its denominator.Source: §3.4 · pp. 6–7
4 Experiment
Experiments evaluate CoViT on COCO detection and segmentation and HICO-DET HOI detection, then examine its initialization, sampling, attention, morphology, and loss choices.
4.1 Experimental Setup
- Evaluation used COCO 2017 detection and segmentation and HICO-DET HOI detection, with ViT-B/16 initialized from MAE, CLIP, DINO, or supervised ImageNet-1K.Source: §4.1 · p. 7
4.2 Main Results
- ViTDet gained +1.9 points in detection AP@[.5:.95] (46.2 to 48.1) and +1.6 mask AP (41.4 to 43.0), with +3.3 boundary-localization accuracy.Source: §4.2 · p. 7, Table 1 · p. 8
- DEFR full mAP improved 2.54 points (32.35 →34.893), rare-category performance gained 2.58 points, and ViPLO reached 39.25 mAP (+2.03 over baseline).Source: §4.2 · pp. 7–8, Table 2 · p. 8
4.3 Ablation Studies
- Averaging all attention heads and dilation followed by erosion each led at 34.89 mAP; manual head selection, erosion-first, and repeated morphology performed worse.Source: §4.3 · p. 8, Table 3 · p. 9
- Triplet loss performed best, while maximum-variation intra-instance positives improved deformation and occlusion robustness; spatially constrained and neighboring-patch sampling had drawbacks.Source: §4.3 · pp. 8–9, Table 3 · p. 9
- MAE-ViT led the pretraining strategies at 34.89 mAP, while DINO and CLIP weakened spatial-relation modeling.Source: §4.3 · p. 9
- Scaling to ViT-H/14 raised mAP to 39.83, but a 4.3× parameter increase yielded only a 4.94 mAP gain.Source: §4.3 · p. 9
5 Conclusion and Future Work
CoViT addresses ViTs’ difficulty distinguishing same-category objects through instance-aware triplets, with gains reported across instance-level tasks and further work planned on occlusion robustness and end-to-end pretraining.
5 Conclusion and Future Work
- CoViT uses self-attention maps and patch similarities to mine instance-aware triplets, reducing intra-instance variation and enlarging inter-instance separation without extra annotations.Source: §5 · p. 9
- Detection, instance segmentation, and HOI detection experiments show consistent gains across several ViT backbones and downstream frameworks.Source: §5 · p. 9
- Future work will improve robustness to occlusion and extend CoViT to end-to-end self-supervised pre-training.Source: §5 · p. 9