PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep ViT Features as Dense Visual Descriptors

Shir Amir, Yossi Gandelsman, Shai Bagon, Tali Dekel

arXiv:2112.05814v3cs.CV

TL;DR

The paper asks whether pretrained ViT representations can serve as effective dense visual descriptors beyond CNN features. It studies DINO-ViT features across layers and attention facets, then applies lightweight zero-shot methods to vision tasks, finding competitive results with supervised methods and large gains over unsupervised ones.

  • Problem

    Pretrained CNN features are widely used as visual descriptors, motivating the question of whether ViT features provide a stronger alternative.

  • Method

    The paper analyzes supervised and self-supervised ViTs across layers and attention facets, then directly applies lightweight zero-shot methods to their patch features.

  • Results

    The methods achieve competitive results with recent state-of-the-art supervised methods and outperform previous unsupervised methods by a large margin.

  • Takeaways & Limitations

    DINO-ViT features can support co-segmentation, part co-segmentation, and semantic correspondences without additional training or data.

Abstract

from arXiv · show

We study the use of deep features extracted from a pretrained Vision Transformer (ViT) as dense visual descriptors. We observe and empirically demonstrate that such features, when extractedfrom a self-supervised ViT model (DINO-ViT), exhibit several striking properties, including: (i) the features encode powerful, well-localized semantic information, at high spatial granularity, such as object parts; (ii) the encoded semantic information is shared across related, yet different object categories, and (iii) positional bias changes gradually throughout the layers. These properties allow us to design simple methods for a variety of applications, including co-segmentation, part co-segmentation and semantic correspondences. To distill the power of ViT features from convoluted design choices, we restrict ourselves to lightweight zero-shot methodologies (e.g., binning and clustering) applied directly to the features. Since our methods require no additional training nor data, they are readily applicable across a variety of domains. We show by extensive qualitative and quantitative evaluation that our simple methodologies achieve competitive results with recent state-of-the-art supervised methods, and outperform previous unsupervised methods by a large margin. Code is available in dino-vit-features.github.io.

1 Introduction

The paper investigates pretrained ViT features as dense visual descriptors, focusing on how self-supervised DINO-ViT representations encode localized, cross-category semantics. It uses lightweight zero-shot methods across several vision tasks and reports competitive performance with supervised methods.

  • Motivation: Deep features from pretrained neural networks have long served as visual descriptors, but have been studied mainly in CNN-based models.CNN features support tasks including image generation, manipulation, correspondences, tracking, and perceptual quality measurement.
  • Motivation: The paper asks whether ViT properties are reflected in internal representations and whether deep ViT features can replace deep CNN features as dense visual descriptors.The study examines ViT representations and demonstrates their use in real-world visual tasks.
  • Findings: DINO-ViT features encode high-level semantic object parts at high spatial resolution, share semantics across related classes, and gradually reduce positional bias across layers.The authors attribute these properties to the combination of ViT architecture and training supervision.
  • Applications: The paper applies simple zero-shot methods to DINO-ViT features for co-segmentation, part co-segmentation, and semantic point correspondences without further training.It also considers challenging part co-segmentation across unrestricted image sets and domains, including video applications.
  • Contributions: The contributions include localized semantics beyond saliency, lightweight methods for multiple tasks, and part co-segmentation across categories and domains with scarce training data.The work reports qualitative and quantitative evaluation against existing methods.

2 Related Work

Prior work established deep visual descriptors mainly through CNNs and explored DINO-ViT features largely as black-box, last-layer representations. This paper instead studies ViT features across layers and attention facets for zero-shot vision tasks, addressing generalization limitations of supervised approaches.

  • CNN-based Deep Features: CNN deep features support detection, segmentation, generation, and other tasks, but exhibit texture bias, lack positional information, and have restricted receptive fields.These limitations motivate examining alternative architectures for dense visual representation.
  • Vision Transformer: ViTs emerged as CNN alternatives with strong task performance and improved robustness to occlusions, adversarial attacks, and texture bias.The paper investigates whether these properties appear in ViT internal representations.
  • DINO-ViT: DINO-ViT was introduced as a label-free, self-distilled model whose attention heads focus on salient foreground regions, supporting retrieval and object segmentation.Subsequent work extended these features to object discovery, semantic segmentation, and category discovery.
  • Research Gap: Earlier DINO-ViT studies generally treated the model as a black box using only last-layer features, whereas this paper examines representations across layers and attention facets.The authors aim to connect understanding of ViT representations with zero-shot real-world applications.
  • Vision Tasks: Supervised co-segmentation methods perform strongly but are restricted by training-data notions of commonality and struggle with new inter-class scenarios.Part co-segmentation and semantic correspondence methods likewise commonly rely on trained CNNs or annotated data, motivating the paper’s zero-shot approach.

3 ViT Features as Local Patch Descriptors

ViTs represent each image patch with token, query, key, and value features throughout transformer layers, preserving spatial resolution while building global-context representations. The paper finds that DINO-ViT features provide localized, cross-category semantics, with keys offering cleaner descriptors and intermediate layers combining position with semantics.

  • Architecture: A ViT splits an image into n non-overlapping patches, projects them into d-dimensional tokens, adds learned positional embeddings, and processes them through L transformer encoder layers.A [CLS] token captures global image properties, while each layer contains normalization, self-attention, MLP, and skip-connection operations.
  • Patch Descriptors: Each patch is associated at every layer with a token, query, key, and value, all of which can serve as local patch descriptors.The paper focuses on keys and justifies this choice through ablations in later sections.
  • Semantics vs. Spatial Granularity: ViT maintains spatial resolution through all layers and gives every token a whole-image receptive field, enabling fine-grained semantic information at higher spatial resolution.CNN deep layers instead trade spatial resolution for semantics, with deepest feature maps 32 times smaller than the input image.
  • Representations Across Layers: Shallow ViT features are mostly positional, deep features become more semantic, and intermediate features contain both positional and semantic information.PCA visualizations show deep features separating dogs from backgrounds while shallow features group by spatial location.
  • Semantic Information Across Classes: DINO-ViT keys group semantically similar body parts across animal classes, whereas supervised ViT features group primarily by class regardless of body part.This indicates localized semantic information in DINO-ViT and global class emphasis in supervised ViT representations.
  • Different Representation Facets: Keys provide slightly cleaner representations than other ViT facets and show less sensitivity to background clutter, while keys and queries retain more intermediate-layer positional bias than values and tokens.Figure 4 compares similarity maps for tokens, queries, values, and keys at intermediate and final layers.

4 Deep ViT Features Applied to Vision Tasks

The paper applies lightweight, zero-shot procedures directly to DINO-ViT dense descriptors for co-segmentation, part co-segmentation, and semantic correspondences. The methods combine clustering, saliency-based voting, positional information, context aggregation, and higher-resolution feature extraction.

  • Methodological design: The overall approach deliberately uses simple, lightweight methodologies on extracted features without additional training or fine-tuning.The authors use this design to showcase the effectiveness of DINO-ViT representations.
  • Co-segmentation and part co-segmentation: DINO-ViT descriptors are clustered across images to discover common semantic segments, then saliency-based voting assigns clusters to foreground or background.Foreground descriptors are reclustered to obtain common object parts.
  • Co-segmentation and part co-segmentation: The procedure repeats clustering on foreground features alone, grouping descriptors of common semantic parts across images.The number of parts is set to the desired value, while co-segmentation clusters use the elbow method.
  • Point correspondences: Semantic correspondence uses mid-layer, position-sensitive features and log-binning to combine local descriptors with adjacent spatial context.This balances positional and semantic information while reducing ambiguity between similarly semantic points.
  • Point correspondences: Reliable correspondences are selected as Best Buddies Pairs whose descriptors are mutual nearest neighbors under cosine similarity.The method forms these pairs from binned descriptors in the two images.
  • Resolution increase: Overlapping patches with interpolated positional encodings increase ViT feature resolution at test time without additional training.The resulting finer spatial features benefit the paper’s applications.

5 Results

Across part co-segmentation, co-segmentation, and semantic correspondence tasks, the zero-shot DINO-ViT approach produces consistent semantic structures across challenging images and domains. It surpasses unsupervised baselines, remains competitive with supervised methods, and outperforms NBB by a large margin for correspondences.

  • 5.1 Part Co-segmentation: Part co-segmentation remains semantically consistent on challenging image pairs with appearance, class, and domain differences, including bicycle-unicycle and cat-tiger pairs.The method consistently identifies parts such as tires, spokes, chassis, and saddles.
  • 5.1 Part Co-segmentation: The method produces consistent animal-face parts across different classes on the 1.5K-image AFHQ test set using k = 10 parts.Examples include ears, foreheads, and whiskers receiving consistent labels across classes.
  • 5.1 Part Co-segmentation: On CUB, the method surpasses unsupervised methods by a large margin and is competitive with the supervised method of Choudhury et al.Evaluation uses landmark regression error, NMI, and ARI on bird images.
  • 5.3 Point Correspondences: Ablations show that keys outperform default DINO-ViT tokens and that intermediate features benefit correspondence more than deepest features.The findings support the use of layer-dependent positional and semantic properties.
  • 5.2 Co-segmentation: On intra-class co-segmentation datasets, the method is competitive with supervised methods and surpasses unsupervised methods by a large margin.The evaluation reports mean Jaccard index and mean precision across MSRC7, Internet300, and PASCAL-VOC.
  • 5.2 Co-segmentation: On the inter-class PASCAL-CO scenario, the method surpasses all other methods.PASCAL-CO contains forty sets of six images drawn from semantically related classes.
  • 5.3 Point Correspondences: Qualitative correspondence results are more robust than NBB to appearance, pose, and scale changes across intra-class and inter-class pairs.The comparison uses the VGG-based NBB method.
  • 5.3 Point Correspondences: On 360 random Spair71k pairs, the method outperforms NBB by a large margin and closes the gap toward supervised CATs.Performance is measured using Percentage of Correct Keypoint (PCK).

6 Conclusion

The paper reports that empirical observations about ViT internal features support lightweight zero-shot methods for several vision tasks. These methods achieve competitive results with state-of-the-art supervised approaches and enable part co-segmentation across classes, videos, and domains lacking training sets.

  • Conclusion: Lightweight zero-shot methods applied to ViT features achieve competitive results with state-of-the-art supervised methods.The conclusion also reports new empirical observations about ViT internal features.
  • Conclusion: Part co-segmentation extends across classes and image domains with scarce available training sets.The authors present these capabilities as a consequence of their feature observations and methods.
Loading 2112.05814v3…