PaperlayerView exact revision on arXiv

Source-linked AI summary

A Mixed Diet Makes DINO An Omnivorous Vision Encoder

Rishabh Kabra, Maks Ovsjanikov, Drew A. Hudson, Ye Xia, Skanda Koppula, Andre Araujo, Joao Carreira, Niloy J. Mitra

arXiv:2602.24181v2cs.CVcs.AI

TL;DR

Vision foundation models do not naturally map different modalities of the same scene to a shared feature space. Omnivorous Vision Encoder distills cross-modal alignment into a pretrained encoder while preserving its discriminative representations, exceeding DINOv2 across 3D-relevant and semantic tasks and cross-modal alignment.

  • Problem

    Vision foundation models should map RGB, depth, and segmentation views of the same scene into a shared space without sacrificing discriminative power.

  • Method

    A parameter-efficient teacher-student framework updates only final processing blocks, maximizing cross-modal alignment while anchoring adapted features to a frozen teacher.

  • Results

    The Omnivorous encoder exceeds DINOv2 on all 3D-relevant and semantic Probe3D tasks, cross-modal alignment, and produces closely aligned RGB, depth, and segmentation features.

  • Takeaways & Limitations

    The modality-agnostic encoder generalizes to unseen visual modalities, supporting a more foundational vision model.

  • Takeaways & Limitations

    It remains unclear whether high-resolution fine-tuning would be required after training Omnivorous DINO.

Abstract

from arXiv · show

Pre-trained vision encoders like DINOv2 have demonstrated exceptional performance on unimodal tasks. However, we observe that their features are poorly aligned across different visual modalities. For instance, the feature embedding for an RGB image and its corresponding depth map of the same scene exhibit a cosine similarity that is nearly identical to that of two random, unrelated images. To address this, we propose the Omnivorous Vision Encoder, a post-training framework that learns a modality-agnostic feature space. We fine-tune the encoder with a dual objective: first, to maximize the feature alignment between different modalities of the same scene; and second, a distillation objective that anchors the learned representations to a fully frozen teacher. The resulting student encoder becomes "omnivorous" by producing more consistent embeddings for a given scene, regardless of the input modality (RGB, Depth, Segmentation, etc.). This approach enables robust cross-modal understanding while retaining the discriminative semantics of the original foundation model. Omnivorous model weights are available at https://github.com/google-deepmind/representations4d.

1. Introduction

The introduction argues that vision foundation models should map different modalities of the same scene to nearly identical representations, but DINOv2 exhibits poor cross-modal alignment. It proposes distilling alignment into an existing encoder through a parameter-efficient teacher–student recipe and data-centric hard positives.

  • Motivation: An omnivorous vision encoder should map RGB, Depth, and Segmentation views of one scene to nearly identical feature-space points.This goal mirrors the stability of human scene perception across viewing conditions.
  • Problem: DINOv2 features for paired RGB, Depth, and Segmentation images are poorly aligned, with RGB–depth cosine similarity often comparable to unrelated scenes.The passage formalizes this as cos(f(xr), f(xd)) ≈ cos(f(xr,1), f(xr,2)).
  • Motivation: The paper motivates cross-modal alignment by analogy to multilingual representation learning, linking abundant RGB data with scarcer, structure-rich depth and segmentation signals.The authors argue that aligning modalities could improve vision models’ generalization similarly to shared multilingual encoders.
  • Challenge: Alignment must avoid feature collapse, while existing contrastive methods such as CMC depend on large, modality-limited sets of negative examples.Collapse would destroy the encoder’s discriminative power.
  • Contribution: The proposed recipe distills cross-modal alignment into a pretrained foundation model using a lightweight, parameter-efficient teacher–student framework that preserves pretrained priors.The student is initialized from the pretrained encoder, with updates restricted to final high-level layers.
  • Data-centric contributions: Natural-palette colorization creates hard positives for depth and segmentation, preventing shortcut solutions based on low-level channel statistics and encouraging structural alignment.Standard grayscale or jet colormaps can expose superficial color-histogram cues.

2. Related Work

Prior work unifies visual modalities through shared encoders, cross-modal pretraining, adapters, or distillation, but usually co-trains or otherwise modifies the backbone. This work instead performs post-hoc cross-modal alignment with a lightweight projector on a fixed foundational encoder while preserving scene-level discrimination.

  • Unified encoders across visual modalities: Unified encoders process multiple modalities through shared backbones, latent-bottlenecked Transformers, or common input-output tokenization.Examples include Omnivore, Uni-Perceiver, Perceiver, Perceiver IO, and Unified-IO.
  • Aligning RGB, depth, and 3D representations: RGB–depth and 2D–3D methods align representations during pretraining using paired contrastive objectives or hybrid contrastive and masked-autoencoding schemes.CLIP2Point applies image–depth contrastive pretraining, while CoMAE combines cross-modal alignment with masked autoencoding.
  • Adapters and parameter-efficient alignment: Parameter-efficient methods preserve mostly frozen backbones by adding adapters or blockwise alignment modules for task adaptation and explicit cross-modal matching.ViT-Adapter provides a blueprint for projector-style modules, while MA-AVT performs blockwise contrastive alignment across audio-visual tokens.
  • Cross-modal distillation and source-free transfer: Cross-modal distillation transfers supervision across modality gaps using source-free transfer, paired task-irrelevant data, consistency losses, and structured student spaces.These approaches also motivate evaluation protocols such as RGB-only inference after multimodal training.
  • Our contribution: The proposed method differs by learning a single lightweight projector g on top of a fixed foundational backbone f ∗ to maximize cross-modal agreement while preserving scene-level discrimination.This targets post-hoc alignment of heterogeneous modalities rather than unified co-training or RGB-D pretraining.

3. Method

The method learns a unified, modality-agnostic embedding space using a shared encoder with a frozen backbone and trainable adapter. It combines symmetric cross-modal InfoNCE alignment with teacher anchoring to preserve pretrained semantics, supported by colorization and modality-mixup augmentation.

  • Unified encoder: A single shared-parameter encoder maps arbitrary modalities, including RGB, depth, and segmentation, into a common embedding space.The architecture freezes feature extraction f* and trains a modality-agnostic adapter g to produce aligned representations.
  • Teacher-student adaptation: The teacher-student framework initializes the student from the foundation model, freezes most layers, and updates only final processing blocks while keeping the teacher head frozen.The teacher provides a stable distillation anchor, while the student adapts shared backbone features across modalities.
  • Data processing: Training colorizes depth and segmentation maps using the RGB image, then applies independently sampled modality mixup with blending coefficients restricted to [0, 0.5].Evaluation sets the blending coefficients to zero, while training uses the interpolation to span Depth ↔ RGB ↔ Segmentation.
  • Cross-modal alignment: Symmetric InfoNCE alignment pulls together embeddings from different modalities of the same scene and separates embeddings from different scenes.The total alignment loss averages symmetric losses over modality pairs using cosine similarity and a learned temperature parameter.
  • Semantic preservation: The anchoring loss uses cosine distance to keep student outputs close to teacher outputs of the same modality, preserving the original representation’s discriminative semantics.A non-zero anchor weight prevents degenerate solutions; the default λanchor is 10.
  • Implementation: Losses are computed separately for class and dense tokens, with 64 dense tokens subsampled per image and intra-image dense tokens excluded as negatives.This implementation applies the objective to both global and spatial representations.

4. Experiments

Experiments show that Omnivorous substantially improves cross-modal retrieval and zero-shot modality transfer while preserving the semantic and geometric capabilities of DINOv2. Ablations reveal a tunable trade-off between alignment and retention of the teacher’s discriminative features.

  • 4.1 Cross-Modal Retrieval: Cross-modal retrieval evaluates RGB, Depth, and Segmentation features across six directed modality pairs on MOVi, ScanNet, and TartanAir.Features are extracted with both Global Average Pooling and the CLS token.
  • 4.1 Cross-Modal Retrieval: On ScanNet with GAP features, Omnivorous improves R@1 from 4.6% to 46.1% and reduces Median Rank to 2.0.On MOVi, R@1 reaches 86.2% versus 15.5% for DINOv2.
  • 4.2 Downstream Evaluation: With a Linear readout, Omnivorous reduces depth RMSE from 0.405 to 0.377 and improves δ1 accuracy from 0.875 to 0.896, while DPT performance remains at 0.297 RMSE.These results indicate preserved downstream geometric information.
  • 4.2 Downstream Evaluation: ImageNet k-NN accuracy remains effectively unchanged at 81.97% versus 81.94% for the teacher, while RP2K improves by +3.65%.The results on downstream transfer datasets are mixed.
  • 4.3 Cross-Modal Transfer: For heads trained on RGB and tested on Segmentation maps, Omnivorous achieves 0.532 RMSE versus 1.536 for DINOv2; on unseen NOCS, it achieves 1.075 versus 1.996.NOCS is out-of-distribution for both backbones, whereas Segmentation is within Omnivorous’s training distribution.
  • 4.4 Ablations: Varying λanchor exposes a Pareto frontier: λanchor=1.0 approaches 0.70 dense-feature alignment, whereas λanchor values of 10.0 or 100.0 recover discernibility while sacrificing alignment gains.The frozen DINOv2 baseline has cross-scene discernibility 0.80 and cross-modal alignment 0.28.

5. Discussion

The discussion finds that Omnivorous adaptation unifies RGB, Depth, and Segmentation features in a shared semantic space while preserving spatial geometry. It concludes that the approach improves across tested tasks and generalizes to unseen modalities, while identifying high-resolution fine-tuning and pre-training alignment as open questions.

  • Representation analysis: Adapted Depth and Segmentation feature maps align closely with RGB maps, unlike frozen DINOv2 features that occupy distinct subspaces.The aligned maps share consistent colors and structural details, providing qualitative evidence of a unified semantic space without discarding spatial geometry.
  • Future work: Future work could align visual modalities during encoder pre-training rather than through post-hoc adaptation.The paper suggests this may unlock deeper benefits than fine-tuning the final layers of an existing model.
  • Limitations: It remains unclear whether DINOv2’s final high-resolution fine-tuning step is required after training Omnivorous DINO.This is identified as a limitation of the current work.
  • Conclusion: Omnivorous DINOv2 exceeds DINOv2 on all 3D-relevant Probe3D tasks, semantic classification, and cross-modal alignment.The modality-agnostic encoder also generalizes to unseen visual modalities.

6. Training and Evaluation Details

This section summarizes the training configuration, then describes the training data pipeline and evaluation protocols.

  • Training configuration is summarized in Table 7.
  • The training data pipeline is described in Section 6.1.
  • Evaluation protocols are detailed in Section 6.2.

6.1. Data Pipeline

The data pipeline augments RGB images photometrically, naturally colorizes structural maps to prevent modality-specific shortcuts, and applies modality mixup to bridge geometry and texture. Structural maps are normalized, discretized into 64 bins, palette-smoothed, and re-rendered before mixing.

  • Photometric augmentation: Photometric augmentation perturbs RGB brightness, saturation, hue, and contrast to improve robustness to lighting variations and color shifts.Brightness and hue use additive deltas, while saturation and contrast use multiplicative factors.
  • Natural colorization: Natural colorization re-renders Depth and Segmentation maps using scene-specific RGB color distributions, preventing alignment through superficial channel statistics.The resulting hard positives force the encoder to attend to shared geometric content.
  • Natural colorization: The colorization algorithm normalizes structural maps, discretizes intensities into B = 64 bins, aggregates RGB colors per bin, smooths the palette, and re-renders each pixel.Palette smoothing uses a kernel of size 5, and palette normalization uses ϵ = 1e −6 for numerical stability.
  • Normalization: All [0,1] images are standardized using ImageNet-1k mean (0.485, 0.456, 0.406) and standard deviation (0.229, 0.224, 0.225).
  • Modality mixup: Modality mixup stochastically blends colorized structural maps with augmented RGB images to span a continuous spectrum between geometry and texture.The mixing coefficient αm is sampled uniformly from [0, αmax], with αmax = 0.5 so structural information remains dominant.

6.2. Evaluation Protocols

The evaluation uses task-specific protocols spanning high-resolution dense prediction, feature correspondence, linear probing, and k-NN transfer. Across tasks, frozen-backbone features are evaluated with modality- and resolution-appropriate preprocessing, decoders, matching rules, and metrics.

  • Monocular Depth Estimation: Dense depth evaluation uses high-resolution NYUv2 and NAVI inputs, center-cropped and padded to the nearest multiple of the ViT patch size p=14.NYUv2 images use 480×640 resolution and NAVI uses 512×512 resolution; the pad-to-patch strategy avoids interpolation artifacts.
  • Semantic Segmentation: Segmentation evaluation uses 512×512 sliding-window inference with stride 341, averaging overlapping-window logits before the final argmax.Training uses random resizing, horizontal flipping, photometric distortion, and random crops; performance is measured by mIoU.
  • 3D Feature Correspondence: 3D correspondence compares flattened 16×16 feature-token grids using bidirectional nearest-neighbor cosine matching and reports exact-token PCK@0.Each source token selects the target token with maximum cosine similarity, and PCK@0 requires the predicted token index to exactly match ground truth.
  • Linear Evaluation: ImageNet linear evaluation trains a single classifier on frozen features and reports validation top-1 accuracy using CLS or concatenated CLS-plus-GAP representations.The classifier projects feature dimension D to K = 1000 classes and is trained for 10 epochs over a sweep of learning rates.
  • k-NN Evaluation: ImageNet and transfer evaluation use cosine-similarity k-NN protocols, with weighted soft voting for ImageNet and hard k-NN classification for transfer datasets.Transfer datasets include iNaturalist, SOP, GLDv2, RP2K, and Food2k, using the same 256-to-224 center-crop preprocessing.

7. Extended Results · 7.1. Diagnostic Metrics

Extended diagnostic results show that Omnivorous training substantially improves cross-modal alignment, while reducing cross-scene discernibility. On ScanNet, <R1, R2> similarity increases from 0.198 to 0.259, reflecting a trade-off controlled by λ_anchor.

  • 7.1. Diagnostic Metrics: Table 8 evaluates Omnivorous training using cross-modal alignment and cross-scene discernibility metrics.The comparison covers performance before and after training.
  • 7.1. Diagnostic Metrics: The default Omnivorous DINO checkpoint greatly improves cross-modal alignment.This improvement is reported relative to the pre-training diagnostic results.
  • 7.1. Diagnostic Metrics: 0.198 to 0.259 <R1, R2> similarity is reported on ScanNet.The passage gives these values as an example of the alignment–discernibility change.
  • 7.1. Diagnostic Metrics: Cross-scene discernibility is sacrificed as cross-modal alignment improves.The passage explicitly characterizes the result as a trade-off.
  • 7.1. Diagnostic Metrics: The diagnostic trade-off is consistent with the trend shown in Fig 4a.Fig 4a presents the trade-off as a function of the λ_anchor loss weight.
  • 7.1. Diagnostic Metrics: λ_anchor loss weight controls the reported alignment–discernibility trade-off.The passage links the changing trade-off directly to this loss-weight parameter.

7.2. 3D Tasks

Across Probe3D tasks, Omnivorous matches DINOv2 across normals-estimation metrics and is more 3D-consistent in multiview correspondence. Qualitatively, it reduces over-segmentation and depth-prediction noise while improving resilience to texture and performance on flat surfaces.

  • Normals estimation: Omnivorous is consistently at par with DINOv2 across all normals-estimation metrics.The evaluation uses a DPT head.
  • Multiview 3D correspondence: Omnivorous is consistently more 3D-consistent than DINOv2 in multiview correspondence.The evaluation reports Percentage of Correct Keypoints at the 0.0 level across four fine-tuned blocks and frozen DINOv2 counterparts, though the performance gap varies by feature block.
  • Segmentation: Omnivorous reduces over-segmentation and is more resilient to textural details than DINOv2 in qualitative segmentation comparisons.These comparisons use linear-head segmentation predictions.
  • Depth estimation: Omnivorous reduces high-frequency noise in linear-head depth predictions and performs better on flat surfaces and flat objects placed on flat surfaces.The qualitative comparison includes cases such as a painting on a wall.

7.3. Ablations

The ablations show that Omnivorous training generalizes across teacher networks and student parametrizations. They also motivate freezing the first eight ViT blocks in the default configuration.

  • Teacher ablation: Omnivorous TIPS ViT-B/14 replaces DINOv2 as the pretrained teacher, using αmax = 0.5 and freezing the first 8 blocks.TIPS differs from DINOv2 in position-encoding shape and CLS-token count.
  • Teacher ablation: λanchor = 100 exceeds higher anchor values on depth and segmentation performance when distilling from TIPS.The result supports the framework’s generality across pretrained teacher networks.
  • Student parametrization: A zero-initialized four-block adapter trained atop a frozen backbone achieves comparable performance to fine-tuning the final four backbone blocks.The adapter-on-top student is larger because it preserves all teacher blocks and adds trainable adapter blocks.
  • Student parametrization: Both student parametrizations are evaluated with a linear head on the final layer, rather than DPT, because their intermediate block structures differ.The adapter-on-top setting has 16 blocks versus 12 for the fine-tuned-backbone setting.
  • Frozen blocks: The default Omnivorous ViT-B/14 setting freezes the preceding Lstop-gradient = 8 blocks while fine-tuning only the final blocks.Depth and segmentation are evaluated with both DPT and linear heads when selecting the frozen-block count.
Loading 2602.24181v2…