PaperlayerView exact revision on arXiv

Source-linked AI summary

Region-Aware CLS Token Augmentation for Fine-Grained Image Retrieval

Ian de Holanda Cavalcanti Bezerra, Vivek Trivedy, Lucas Pascotti Valem, Longin Jan Latecki

arXiv:2610.10991v1cs.CVcs.LG

TL;DR

Single global image descriptors can miss localized details important for fine-grained retrieval, while using all patch embeddings is costly. The paper uses DINOv2-reg cue tokens to select ROI tokens and matches a compact token set with ColBERT-style alignment. The authors report improvements over the single-vector baseline, with the largest gains arising from ROI tokens.

  • Problem

    A single global descriptor may miss localized details needed for fine-grained image retrieval, while comparing all patch tokens is costly in computation and storage.

    Source: §I · p. 1

  • Method

    The method matches each DINOv2-reg [CLS] or register cue to its most similar patch, pools a nearby region into an ROI token, and uses ColBERT-style multi-vector matching.

    Source: §I · pp. 1–2

  • Results

    The compact 10-token representation improves over the corresponding DINOv2-reg single-vector baseline in evaluated settings, with the largest gains arising from ROI tokens.

    Source: §V · p. 5

  • Takeaways & Limitations

    Register tokens provide complementary retrieval information, and adding buddy-pooled ROI tokens further improves the compact representation.

    Source: §I · p. 2, §V · p. 5

Abstract

from arXiv · show

Image retrieval methods often rely on a single global semantic descriptor extracted from an image, e.g., the [CLS] token in vision transformers. However, trying to squeeze all the semantic information of an image into a single descriptor can hurt downstream retrieval performance, especially for fine-grained retrieval tasks. In this work, we augment the semantic tokens in the newer visual transformers, the global [CLS] token and the four register tokens, with a carefully selected collection of spatial tokens, aiming to capture the spatial region representation that characterizes the contents captured in each of the semantic tokens. We leverage the DINOv2-reg model, which includes register tokens that emergently learn object and part-based representations. For each "cue" token ([CLS] and each register token), we find a "buddy" image patch token and extract an N x N patch region to produce a set of localized ROI tokens. Our approach automatically captures important regions of interest without any external bounding boxes or saliency modules, purely by matching semantic tokens with their spatial representation regions. Furthermore, we incorporate these tokens into a multi-vector retrieval framework inspired by ColBERT, enabling fine-grained matching via a per-token alignment mechanism while avoiding the large storage cost of keeping all patch embeddings. Through extensive experiments, we find that (1) register tokens encode useful fine-grained details that can complement the [CLS] token; (2) automatically pooled ROI tokens further improve fine-grained discrimination; and (3) multi-vector retrieval with a small set of tokens improves over a DINOv2-reg single-vector baseline while remaining tractable for large-scale search. The code is available at https://github.com/IdhcbIan/Augmenting_CLS_with_ROI_tokens.

I. INTRODUCTION

A single global descriptor may miss localized details needed for fine-grained retrieval, while comparing every patch token is costly. The paper uses DINOv2-reg cue tokens to select compact ROI descriptors and combines them in a multi-vector retrieval framework.

I. INTRODUCTION

  • Single-vector descriptors may struggle with localized details in fine-grained retrieval, while comparing all image patch tokens is costly in computation and storage.The introduction also cites representational-capacity limits for single-vector embeddings [5].Source: §I · p. 1
  • DINOv2-reg register tokens carry object- and part-level features, and query [CLS] tokens are often more similar to another image’s register token than its [CLS] token.For COCO dog-class queries, the reported most-similar-token split is 64% register tokens versus 36% [CLS] tokens.Source: §I · p. 1, Figure 1 · p. 1
  • For each [CLS] or register cue, the method finds its most similar patch token and mean-pools a nearby region into an ROI token without external proposals, boxes, or saliency modules.The selected regions can capture parts such as a bird’s head or a car’s headlight.Source: §I · p. 1
  • The authors report that matching register tokens alongside [CLS] improves retrieval, and adding ROI tokens improves it further.They identify registers as complementary cues and localized ROI tokens as a further source of improvement.Source: §I · p. 2
  • ColBERT-inspired per-token max-sim alignment supports fine-grained matching with about 10 tokens per image, keeping similarity computation and storage tractable.The introduction notes that single-vector approximations can further accelerate multi-vector search [16].Source: §I · p. 2

II. RELATED WORK

Prior retrieval work spans hand-crafted and learned local features, ViT-based region representations, and methods for compressing multi-vector indexes. Fine-grained retrieval emphasizes matching parts or class-specific features to distinguish similar objects.

II. RELATED WORK

  • Local-feature methods include SIFT, CNN-based attentive descriptors in DELF [8], and DELG’s global and local descriptors.Other approaches learn pairwise similarity or use asymmetric estimation for retrieval,.Source: §II · p. 2
  • ViT-based approaches use [CLS] or other tokens to represent regions, while some combine or select patch features for fine-grained discrimination [10],, [12], [19], [20].DETR-like models can also learn bounding boxes through attention without proposal networks.Source: §II · p. 2
  • Multi-vector indexes are made more manageable through token clustering, centering, quantization, dimensionality reduction, or reducing token counts within the model, [22], [23], [24].The paper positions its approach as using model similarity scores to retain a compact, discriminative token set.Source: §II · p. 2
  • Fine-grained retrieval distinguishes subordinate categories with subtle inter-class and large intra-class variation, and prior work uses parts or attribute and hierarchical labels to focus on class-specific features, [8],, [27].The paper relates its register and ROI tokens to these part-focused strategies.Source: §II · p. 2

III. METHODOLOGY

The method extracts DINOv2-reg semantic tokens, uses them to locate corresponding patch regions, and trains retrieval with ColBERT-style token similarity. The training representation contains 10 tokens per image.

III. METHODOLOGY

  • The training flow extracts the [CLS] and register tokens from DINOv2-reg, then uses each as a cue to identify its most similar image patch token.The selected patch is termed the cue token’s buddy token.Source: Figure 3 · p. 2
  • Each ROI token is formed by mean-pooling a buddy token with image patch tokens in the surrounding region, yielding 10 tokens per image in the experiments.Source: Figure 3 · p. 2
  • Training uses standard triplet training with ColBERT-style similarity between image pairs.Source: Figure 3 · p. 2

A. Model and Forward Pass

The base DINOv2-reg backbone processes an image through patch embedding and transformer layers to produce tokens. The method aims for a small retrieval token set that balances global context and local detail.

A. Model and Forward Pass

  • The base DINOv2-reg vision transformer takes an image as input and, after patch embedding and transformer layers, produces [CLS], register, and image patch tokens for a compact retrieval representation.The stated design aim is to balance global context with fine-grained local detail.Source: A. Model and Forward Pass · pp. 2–3

B. Regions-of-Interest (ROI) Tokens

The method uses each semantic cue token to locate a related patch region, then pools that region into an ROI token. The final retrieval set combines these ROI tokens with the cue tokens.

B. Regions-of-Interest (ROI) Tokens

  • Each [CLS] or register cue token selects its most-similar patch as the center of a localized region.The patch is selected by dot-product similarity; the default ROI is an N × N patch region with N = 3.Source: B. Regions-of-Interest (ROI) Tokens · p. 3
  • Mean-pooling the selected region produces one ROI token, with boundary clamping and padded-position masking where applicable.Mean-pooling adds no learnable parameters and is described as smoothing and denoising patch embeddings; masking is used for INSTRE padding.Source: B. Regions-of-Interest (ROI) Tokens · p. 3
  • The retrieval set combines each cue token with its corresponding ROI token, totaling 2(R + 1) tokens.With R = 4 register tokens, the set contains 10 tokens.Source: B. Regions-of-Interest (ROI) Tokens · p. 3

C. Multi-Vector Training

The paper trains its multi-vector image representation with a ColBERT-inspired max-sim objective and a triplet-like margin. At inference, the compact token set keeps scoring tractable, with a single-vector shortlist proposed for further scaling.

C. Multi-Vector Training

  • The multi-vector representation uses ColBERT-inspired max-sim matching, and each token is L2-normalized.The score compares each query token with database tokens using its best match.Source: C. Multi-Vector Training · p. 3
  • The triplet-like objective pushes the positive image score above the negative image score by at least margin α.The positive image shares the query's class, while the negative image belongs to a different class.Source: C. Multi-Vector Training · p. 3
  • At inference, 10 tokens require 100 token similarities per image pair, versus 1 for a single vector and 40,401 for the full-token representation.A [CLS]-based ANN shortlist is proposed before max-sim reranking; integrating and benchmarking this two-stage approach remains future work.Source: C. Multi-Vector Training · p. 3

IV. EXPERIMENTS

The experiments assess retrieval improvements associated with register tokens, ROI-token selection, and ColBERT-style multi-vector training, alongside ablations.

IV. EXPERIMENTS

  • The evaluation covers retrieval benchmarks and ablations of register tokens, ROI-token selection, and ColBERT-style multi-vector training.Source: §IV · p. 3

A. Datasets

The evaluation uses five retrieval datasets spanning fine-grained categories, product retrieval, and instance-level retrieval. It reports Recall@K with K values that vary by dataset.

A. Datasets

  • The five datasets are CUB-200, In-Shop Clothes Retrieval, Cars-196, Stanford Online Products, and INSTRE.They cover birds, clothing, car models, products, and instance-level retrieval, respectively.Source: A. Datasets · p. 3
  • Recall@K is reported at K = {1, 2, 4, 8} for CUB, SOP, and Cars-196, and K = {1, 10, 20, 30} for In-Shop.Source: A. Datasets · p. 3

B. Implementation Details

The experiments train DINOv2-reg end to end with multi-vector triplet loss and dataset-specific settings, then evaluate retrieval with Recall@K and, in some cases, mAP.

Training setup

  • Training uses pretrained DINOv2-reg-base with four register tokens and backpropagates through hard buddy selection using a steep-softmax approximation.This trains both the vision encoder and ROI selection process on each dataset.Source: B. Implementation Details · pp. 3–4

Preprocessing and evaluation

  • Evaluation uses image size 224 and Recall@K as the primary metric, with mAP included in some comparisons, especially INSTRE.Additional augmentation is used where dataset queries apply cropping.Source: B. Implementation Details · p. 4

Optimization

  • AdamW uses a 10^-5 learning rate and 0.01 weight decay, with a triplet-loss margin of 0.5 for INSTRE and 0.3 for other datasets.Batch sizes are 512 for CUB, INSTRE, and Cars and 1024 for larger datasets; training runs 1,000 steps for CUB and Cars, 1,500 for In-Shop, and 10,000 for SOP.Source: B. Implementation Details · p. 4

Triplet selection

  • Hard-negative triplets pair each anchor with its least-similar same-class match and most-similar different-class match to resolve subtle class differences.A straight-through estimator with τ = 0.05 approximates gradients through buddy-patch selection; inference uses a strict hard arg max.Source: B. Implementation Details · p. 4

Data augmentation

  • Only INSTRE uses multi-scale padding augmentation to reduce the domain shift between cropped queries and full gallery images.A binary mask excludes canvas borders during buddy selection and ROI pooling, and evaluation takes the maximum similarity across scales.Source: B. Implementation Details · p. 4

C. Results and Ablations

The compact multi-vector representation improves over the DINOv2-reg [CLS] baseline in evaluated settings, with ROI tokens providing the larger gains. The section also reports comparisons on INSTRE and against local-feature methods, and examines storage, training, and ROI-window choices.

Overall ablation findings

  • The compact multi-vector representation improves over the DINOv2-reg [CLS] baseline in evaluated settings, and the 3 × 3 ROI window is a good experimental default.The ablations are limited to CUB and Cars-196 because each configuration requires retraining; comparisons in Table I span different backbones, pretraining schemes, and training objectives.Source: C. Results and Ablations · p. 4

Results on small objects (INSTRE)

Register vs. non-registers

  • Register tokens provide a modest gain, while buddy-pooled ROI tokens provide the larger improvement on CUB and a further gain on Cars-196.The results characterize registers as complementary cues, with the main benefit coming from combining them with localized ROI tokens.Source: C. Results and Ablations · pp. 4–5, Table III · p. 5

Single-vector vs. multi-vector training

  • The single-vector training setting underperforms the multi-vector method, supporting the importance of multi-vector scoring for local alignment in fine-grained retrieval.Both models are trained identically until scoring; the multi-vector loss aggregates scores from [CLS], register, and ROI tokens.Source: C. Results and Ablations · p. 5, Table III · p. 5

Ablation on region size

  • A local neighborhood around the buddy patch is preferable to a single patch, while larger ROI windows offer limited extra benefit and may depend on the dataset.The experiments use a 3 × 3 ROI window as the default setting.Source: C. Results and Ablations · p. 5, Table IV · p. 5

Memory consumption vs. performance

  • The compact representation uses more storage than a single vector but far less than storing all patch tokens, while retaining most of the observed benefit of compact multi-vector matching.The 1M-image storage figures are analytical projections, not million-image retrieval results; indexing overhead, latency, batching, and ANN implementation costs are excluded.Source: C. Results and Ablations · p. 5, Table V · p. 5

Comparison with local-feature methods

  • Our method obtains the highest R@k in five of six reported comparisons with local-feature methods.The comparison uses results reported in the respective papers; the approach uses a fixed set of 10 tokens in one backbone forward pass.Source: Table VI · p. 5, C. Results and Ablations · p. 5

V. CONCLUSION AND FUTURE WORK

The work concludes that DINOv2 register tokens provide complementary retrieval information and that a compact representation combining [CLS], registers, and buddy-pooled ROI tokens improves over the corresponding single-vector baseline. Future work will test alternative ROI selection strategies and additional model configurations.

Conclusion

  • The 10-token representation combines [CLS], four register tokens, and buddy-pooled ROI tokens, with the largest gains arising from ROI tokens.In evaluated settings it improves over the corresponding DINOv2-reg single-vector baseline without external modules.Source: §V · p. 5

Future work

  • Planned work includes comparing cue-guided ROI selection with attention and random selection, testing fewer-token and untrained configurations, and extending the mechanism to other encoders.Named extensions include CLIP, DINOv3, and registers added at inference.Source: §V · p. 5
Loading 2610.10991v1…