Source-linked AI summary
VICRegL: Self-Supervised Learning of Local Visual Features
Adrien Bardes, Jean Ponce, Yann LeCun
TL;DR
Self-supervised vision has largely separated global features for classification from local features for segmentation, leaving a trade-off between the two. VICRegL combines global and local VICReg criteria with spatial and embedding-based local matching, improving segmentation while preserving classification performance. The method also shows that convolutional backbones, including ConvNeXt, can deliver strong self-supervised representations.
Problem
Existing self-supervised methods tend to emphasize either global invariant features for classification or local features for detection and segmentation.
Method
VICRegL combines a global VICReg loss on pooled representations with symmetric location-based and feature-based local matching before the final pooling layer.
Results
VICRegL improves segmentation while preserving classification accuracy, achieving 55.9 mIoU on Pascal VOC with ResNet-50, +8.1 mIoU over VICReg.
Takeaways & Limitations
Learning local features alongside global features significantly improves segmentation without sacrificing classification performance, and ConvNeXt backbones are competitive with vision transformers.
Takeaways & Limitations
Because convolutional feature maps have image-wide receptive fields, it remains unclear how much a position-specific feature vector represents objects at its corresponding input location.
Abstract
from arXiv · showhide
Most recent self-supervised methods for learning image representations focus on either producing a global feature with invariance properties, or producing a set of local features. The former works best for classification tasks while the latter is best for detection and segmentation tasks. This paper explores the fundamental trade-off between learning local and global features. A new method called VICRegL is proposed that learns good global and local features simultaneously, yielding excellent performance on detection and segmentation tasks while maintaining good performance on classification tasks. Concretely, two identical branches of a standard convolutional net architecture are fed two differently distorted versions of the same image. The VICReg criterion is applied to pairs of global feature vectors. Simultaneously, the VICReg criterion is applied to pairs of local feature vectors occurring before the last pooling layer. Two local feature vectors are attracted to each other if their l2-distance is below a threshold or if their relative locations are consistent with a known geometric transformation between the two input images. We demonstrate strong performance on linear classification and segmentation transfer tasks. Code and pretrained models are publicly available at: https://github.com/facebookresearch/VICRegL
1 Introduction
Self-supervised vision methods commonly learn either global invariant representations or local features, motivating VICRegL’s joint learning of both scales for classification and segmentation.
- Motivation: Recent methods learn view-invariant representations from differently transformed views of the same image using joint embedding architectures and specialized losses.Transformations include cropping, rescaling, and color jittering.
- VICRegL: VICRegL applies VICReg to both pooled global vectors and pre-pooling local vectors, matching local features using spatial or embedding-space relationships.The method is designed to learn global and local features simultaneously without segmentation masks.
- Results: Frozen-backbone evaluation shows that adding local features preserves classification performance while improving semantic segmentation accuracy.The evaluation tunes only a linear classification or segmentation head.
- Results: 55.9 mIoU on Pascal VOC with ResNet-50 is a +8.1 mIoU improvement over VICReg, while ConvNeXt-S reaches 67.5 mIoU, a +6.6 mIoU improvement.These results are reported for linear frozen semantic segmentation.
2 Related work
Related work distinguishes global representation learning from local feature learning, with local methods explicitly modeling image parts and commonly targeting segmentation.
- Global features: Global methods learn representations invariant to multiple views and avoid collapse through contrastive negatives, clustering, or related mechanisms.The cited global approaches use joint embedding architectures.
- Local features: Local methods explicitly learn feature sets describing small image regions and are therefore better suited to segmentation tasks.These methods commonly evaluate on segmentation benchmarks.
- Local features: Contrastive local-learning objectives can operate at pixel level or feature-map level to enforce consistency between corresponding pixels or groups of pixels.The passage distinguishes pixel-level and feature-map-level applications.
- Local features: Other approaches modify backgrounds or aggregate semantic content from multiple images to improve object localization, but rely on carefully handcrafted image constructions.These constructions alter background context or combine object instances.
3 Method
VICRegL extends VICReg to learn local and global representations simultaneously by matching local feature vectors using spatial and embedding-space information, while applying a global criterion after pooling. Its combined objective balances local and global learning and supports convolutional backbones, including multi-crop adaptations.
- VICRegL: feature vectors matching: VICRegL applies VICReg to both pooled global representations and unpooled local feature maps from two views of the same image.A convolutional encoder produces spatial feature maps; pooling supplies global representations while local features retain spatial structure.
- VICRegL: feature vectors matching: A local projector embeds feature maps into a smaller space while preserving localization information for local matching.The projected maps retain H × W spatial dimensions while changing the feature dimension.
- VICRegL: feature vectors matching: Location-based matching pairs features using their corresponding absolute positions in the seed image after view transformations.Each feature is matched to the spatially nearest coordinate, and only the top-γ pairs are retained.
- VICRegL: feature vectors matching: Feature-based matching pairs each local feature with its nearest neighbor in embedding space to capture long-range interactions.The top-γ pairs by l2-distance are retained, and this criterion matches features that need not occupy the same seed-image location.
- VICRegL: feature vectors matching: The final objective combines symmetrized location- and feature-based local losses with a global VICReg loss weighted by α.α controls the emphasis on global versus local features, and the authors report a trade-off supporting transfer to classification and segmentation.
- VICRegL with the ConvNeXt backbone: The matching procedure supports convolutional networks and is adapted to multi-crop training by using separate match counts for large and small views.The experiments use ResNet-50 and ConvNeXt backbones; multi-crop generates two large and N − 2 small views.
4 Experimental Results
VICRegL improves frozen linear segmentation while preserving classification performance across ResNet-50 and ConvNeXt evaluations. Ablations show that combining global and local criteria, using both location- and feature-based matching, and retaining VICReg regularization are important for strong local representations.
- Comparison with prior work: VICRegL strongly improves linear segmentation over VICReg while preserving most classification performance across ResNet-50 and ConvNeXt backbones.The evaluation uses frozen backbones with linear classification or segmentation heads, alongside fine-tuning comparisons.
- Comparison with prior work: +6.2 mIoU on Pascal VOC accompanies only a 0.3% classification-accuracy loss for VICRegL α = 0.9 with a frozen ResNet-50 backbone.VICRegL α = 0.75 reaches a +8.1 mIoU improvement on Pascal VOC.
- Comparison with prior work: +6.6 mIoU on Pascal VOC and +2.2 mIoU on ADE20K are obtained over VICReg with a frozen ConvNeXt-S backbone at α = 0.9.The reported improvements preserve most classification performance.
- Ablations: +3.4, +4.3, and +4.6 mIoU segmentation gains occur as α decreases to 0.9, 0.75, and equal local/global weighting, while classification accuracy drops by 0.2%, 1.2%, and 2.9%.Further decreasing α reduces performance on both tasks, indicating that global information remains important for localized tasks.
- Ablations: +4.3 mIoU is achieved when location- and feature-based matching are combined, compared with +2.9 mIoU for location-based matching and +1.7 mIoU for feature-based matching.The combined criterion produces the best local-feature quality, while classification accuracy is not affected across these ablations.
- Ablations: VICReg local variance and covariance regularization improve segmentation by +0.8 and an additional +1.3 mIoU, without significantly affecting classification accuracy.The local invariance, variance, and covariance criteria are evaluated as collapse-prevention components.
5 Conclusion
VICRegL learns local and global visual features jointly by matching representations across pixel and embedding spaces. The local criterion improves segmentation while preserving classification accuracy, though the local nature of convolutional features remains uncertain.
- VICRegL learns local and global visual features simultaneously by matching feature vectors using pixel-space and embedding-space distances.
- Adding the local criterion significantly improves segmentation performance while preserving classification accuracy.
- Convolutional networks with ConvNeXt backbones are competitive with vision transformers for self-supervised learning.
- Convolutional feature vectors may contain information from across the entire image rather than mainly from corresponding local objects.Their receptive fields cover the entire image, making their local character unclear.
A Additional results
Additional evaluations show that VICRegL performs strongly on frozen and fine-tuned detection and semi-supervised benchmarks. Its advantage is especially pronounced with frozen linear evaluation and ConvNeXt-B features.
- VICRegL significantly outperforms other methods with a frozen non-linear segmentation head and outperforms all compared methods in fine-tuning.Performance gaps are closer with a non-linear head than with a linear head.
- +3.7 AP over VICReg and +2.6 AP over DenseCL are achieved by VICRegL in frozen ResNet-50 object detection.With ConvNeXt-B, VICRegL also outperforms concurrent ViT-B methods by a significant margin.
- VICRegL significantly outperforms other methods on COCO semi-supervised detection and segmentation benchmarks.
B Additional visualizations
VICRegL’s local matching often connects distant regions that represent similar concepts, but some matches are less relevant because convolutional feature vectors have broad attention spans.
- Feature-based matching often pairs distant image regions that represent similar concepts, such as corresponding animal parts or repeated objects.
- Some selected matches are irrelevant because convolutional feature vectors cover the entire image rather than only the displayed local square.
C Method details
The method estimates each feature vector’s corresponding image location from the centers of a regular grid over the crop. This provides the spatial coordinates used for local matching.
- For a feature map with H × W vectors, VICRegL assigns each vector the center coordinate of its corresponding rectangle in an H × W grid.The grid has identical rectangle dimensions and approximately matches the crop size in the original image.
D Evaluation details
Evaluation comparisons used custom linear-evaluation scripts and the mmsegmentation library, with unavailable results reproduced from official pretrained models.
- The authors used custom scripts for linear evaluations and mmsegmentation for segmentation evaluations.
- When comparison results were unavailable, they downloaded official pretrained models and evaluated them by extensively sweeping hyper-parameters, especially learning rate.
D.1 Linear Evaluation
Linear evaluation trains a classifier on frozen-backbone representations using a standardized optimization and augmentation protocol.
- The evaluation trains a linear classifier on representations from a frozen backbone.
- Training uses SGD with batch size 256 for 100 epochs, selecting the best result across tested learning rates.
- Training images are randomly cropped and resized to 224 × 224, while validation images are resized to 256 × 256 and center cropped to 224 × 224.
- No color augmentation is used during either training or evaluation.
D.2 Linear Segmentation
Linear segmentation evaluation maps backbone feature representations to segmentation masks, using architecture-specific feature aggregation and dataset-specific training schedules.
- For ResNet-50, a linear layer maps the (2048, 32, 32) feature map to class channels before bilinear upsampling to (num_classes, 512, 512).
- ConvNeXt feature maps from four blocks are upsampled, concatenated into (1920, 128, 128), and mapped linearly to segmentation classes.
- The evaluation uses 40k iterations for Pascal VOC, 160k iterations for ADE20k, and separate learning-rate sweeps for frozen and fine-tuned regimes.
E Running time and memory usage
Adding VICRegL's local criterion introduces moderate time and memory overhead during ResNet-50 pretraining, mainly from covariance computations over output feature maps.
- VICRegL adds moderate computational burden in both time and memory compared with VICReg.
- The overhead mainly comes from computing covariance matrices for every feature vector in the output feature maps.