PaperlayerView exact revision on arXiv

Source-linked AI summary

DDMS: Discriminative Distillation of Multi-view Foundational Features into Single-view Models

Jeong-gi Kwak, Sho Kagami, Yuki Ono, Kwang Moo Yi

arXiv:2608.23850v1cs.CV

TL;DR

Pretrained 2D features have limited 3D awareness, while correspondence-focused adaptations can weaken semantic richness. DDMS fuses 2D and multi-view geometric features, refines them with discriminative ranking and semantic anchoring, and distills the teacher into a single-view encoder. Across correspondence, dense prediction, and explicit 3D lifting evaluations, it produces stronger 3D-aware features while preserving semantic utility.

  • Problem

    Pretrained 2D foundation features can be inconsistent across views and insufficiently separate geometrically distinct regions, limiting reliable multi-view correspondence.

  • Method

    DDMS builds a multi-view teacher from fused 2D and geometric features, refines it with geometry-supervised ranking and feature-space anchoring, and distills it into a single-view student.

  • Results

    DDMS improves cross-view consistency and local discriminability across correspondence, dense prediction transfer, and explicit 3D lifting evaluations while preserving semantic utility.

  • Takeaways & Limitations

    Multi-view geometry models provide effective supervision for stronger 3D-aware visual features that remain usable as general-purpose representations.

  • Takeaways & Limitations

    Prior adaptation methods expose a trade-off: improving consistency can smooth matching-critical distinctions, while improving correspondence can harm semantic richness and 3D consistency.

Abstract

from arXiv · show

Foundational visual features such as DINO have played a critical role across modern computer vision, and have recently become key components in multi-view feed-forward geometry estimators. In this work, we demonstrate that by re-distilling these multi-view models---their internal knowledge of 3D geometry---into a single-view estimator, we can obtain enhanced 3D consistent foundational features. Our key idea is to construct a multi-view teacher by fusing pretrained 2D foundation features with multi-view geometric features, and refining the fused representation with a discriminative ranking objective. Through our discriminative distillation framework, we enforce the learned features to be both 3D consistent and locally distinctive, while keeping them aligned with the feature space of the original foundation model to preserve the semantic structure of the pretrained representation. Consistency and local discriminability are critical for 3D computer vision problems such as forming semantic and geometric correspondences across images. To demonstrate the effectiveness of our method, we perform comprehensive experiments spanning multiple angles: direct feature analysis, dense prediction transfer, and explicit 3D lifting and rendering. Across these evaluations, our method consistently produces stronger 3D-aware foundation features that improve multi-view consistency and local discriminability while preserving the semantic transferability of the original representation.

1 Introduction

Pretrained 2D foundation features offer strong semantic utility but limited 3D awareness, motivating DDMS’s combination of multi-view geometric knowledge, discriminative ranking, and semantic anchoring. The resulting single-view representation targets improved consistency and local separability without sacrificing semantic transfer.

  • Motivation: 2D foundation features transfer well across tasks but can vary across viewpoints and fail to separate visually similar, geometrically distinct regions.These limitations hinder reliable cross-view feature association and aggregation.
  • Motivation: Existing approaches can improve 3D consistency while smoothing distinctions needed for image matching, or improve correspondence while harming semantic richness and 3D consistency.Fit3D illustrates the first trade-off, whereas MEF illustrates the second.
  • Method: DDMS combines 2D foundation features with multi-view geometric features to retain semantic utility while adding geometric awareness, then distills the result into a single-view student.The multi-view teacher is refined before distillation to make the representation practical at inference time.
  • Method: Geometry-supervised ranking promotes valid local 3D correspondences over incompatible candidates, while feature-space anchoring preserves alignment with the original semantic feature space.The objective jointly targets 3D consistency, fine-grained separability, and semantic preservation.
  • Results: DDMS demonstrates gains across geometric correspondence, semantic transfer, dense prediction, and 3D lifting evaluations.Its stated contribution is a more balanced representation across these evaluation dimensions.

2 Related Work

Related work establishes strong semantic transfer from visual foundation models, while multi-view geometry models provide feed-forward geometric reasoning. DDMS combines these directions through a multi-view teacher and single-view distillation framework.

  • Visual foundation models and 3D awareness: Visual foundation models provide dense features that transfer across recognition, dense prediction, segmentation, and correspondence-based tasks.Their broad utility comes from intermediate representations that function as dense visual descriptors.
  • 3D-aware feature learning and distillation: Recent methods add geometric supervision, 3D lifting, or heterogeneous teacher distillation to improve the limited 3D awareness of pretrained visual features.Fit3D and related approaches adapt pretrained DINOv2 features using explicit multi-view or 3D supervision.
  • Multi-view geometry and feed-forward 3D models: Multi-view geometry foundation models use cross-view reasoning to predict depth, point maps, camera parameters, tracks, and metric scene structure from one or more images.Feed-forward 3DGS methods likewise produce renderable 3D representations.
  • DDMS positioning: DDMS fuses pretrained 2D visual features with frozen multi-view geometry features to construct a teacher, then refines and distills it into a single-view student.The final student operates as a standard 2D backbone at inference time.

3 Method

The method builds a multi-view-aware teacher by combining semantic 2D foundation features with geometric multi-view features, refining them discriminatively, and distilling them into a single-view student. Evaluations show strong feature separation, geometric correspondence, and semantic transfer performance.

  • 3.2 Multi-view-aware Teacher Construction: The framework constructs a multi-view teacher by fusing pretrained 2D foundation features with geometry-aware features from a multi-view transformer.The representations are concatenated on a shared ViT patch grid and mapped back into the original feature space through a lightweight refinement module.
  • 3.2 Multi-view-aware Teacher Construction: Residual feature fusion injects cross-view geometric context while keeping the teacher close to the original foundational feature space.The fusion module and LoRA adapters are optimized while the geometry model remains frozen.
  • 3.3 Discriminative Feature Learning: The teacher ranks geometrically valid correspondences above incompatible candidates using geometry-filtered positive and negative samples.Candidates are filtered by reprojection validity, visibility/depth consistency, surface-normal agreement, and 3D proximity before applying a differentiable average-precision objective.
  • 3.4 Distilling Multi-view Priors into a Single-view Student: The refined teacher is distilled into a single-view student with feature-level supervision and a weak geometry-supervised ranking regularizer.At inference, only the student encoder is used for standard single-image feature extraction with distilled multi-view awareness.
  • 3.4 Distilling Multi-view Priors into a Single-view Student: The method achieves the best feature separation, substantially improves ScanNet geometric correspondence, and remains strong across Navi-Wild, SPair-71k, and DAVIS.Qualitative ScanNet and Navi-Wild results also report the most accurate correspondences with fewer errors.

4 Experiments

DDMS is evaluated through direct feature quality, dense prediction transfer, and explicit 3D aggregation, testing consistency, discriminability, and semantic utility across multiple benchmarks. The method improves correspondence and feature separation while retaining strong segmentation, depth, and 3D-rendering behavior.

  • Evaluation setup: DDMS is evaluated on direct feature quality, dense prediction transfer, and explicit 3D feature aggregation.Direct evaluations cover separation and correspondence; transfer uses frozen-feature linear probes; aggregation uses point-based and rendering-based strategies.
  • Direct feature quality: The method achieves the largest feature-separation margin across both backbone variants.Feature separation is measured as the gap between corresponding-feature similarity and average non-corresponding-feature similarity.
  • Direct feature quality: DDMS substantially improves ScanNet geometric correspondence, with gains remaining stable as viewpoint changes increase.Qualitative results show more accurate matches with fewer errors.
  • Direct feature quality: Across Navi-Wild, SPair-71k, and DAVIS-2017, the method remains consistently strong while maintaining semantic richness.These benchmarks test object-centric correspondence, semantic correspondence across instances, and temporal propagation.
  • Dense prediction transfer: For dense prediction, DDMS produces clean segmentation and coherent depth, matching or improving over SnD while outperforming it by a large margin in Table 1.The method performs comparably to SnD on semantic segmentation and mildly outperforms it on depth estimation, while remaining stronger on correspondence-related evaluations.
  • Ablation study: Removing either anchoring or discriminative learning causes suboptimal performance, while single-view teacher conditioning reduces the overall gain.Directly fine-tuning DINOv2 with the discriminative loss improves separation and correspondence but degrades segmentation.

5 Conclusion

DDMS transfers multi-view geometric knowledge into a single-view visual encoder through geometry-aware fusion, discriminative ranking, and semantic anchoring. Across correspondence, dense prediction transfer, and explicit 3D lifting, it improves consistency and local discriminability while preserving semantic utility.

  • Conclusion: DDMS transfers multi-view geometric knowledge into a single-view visual encoder.Its framework uses a multi-view teacher and a single-view student.
  • Conclusion: The framework combines geometry-aware feature fusion, discriminative ranking, and semantic anchoring.These components target cross-view consistency, local discriminability, and semantic utility.

A Additional Analysis

Additional analyses examine the learned representation's behavior and generality across separation, geometry-feature selection, foundation features, backbone generalization, and out-of-distribution transfer.

  • Additional analyses: The additional analyses cover pair-level feature separation, geometry feature selection, off-the-shelf 3D-aware features, backbone generalization, and out-of-distribution transfer.

A.1 Feature Separation Distribution

Pair-level analysis shows that DDMS improves feature separation consistently across ScanNet image pairs, and teacher construction benefits from an intermediate geometry-model feature stage. The appendix also compares discriminative objectives within the same teacher–student pipeline.

  • Feature Separation Distribution: The pair-level analysis treats each of 1,500 ScanNet test pairs as one sample and computes its average positive-negative margin.Table 5 reports mean and standard deviation, while Figure 9 visualizes the full distribution.
  • Feature Separation Distribution: DDMS shifts margins higher than Fit3D and SnD while maintaining a more compact spread than DINOv2 and MEF.The comparison indicates improved separation with more stable behavior across scene pairs.
  • Feature Separation Distribution: DDMS achieves the highest mean margin and lowest standard deviation across image pairs.This supports more consistent feature separation and the intended balance between cross-view consistency and local discriminability.
  • Choice of geometry feature layer: The middle-stage DA3 feature gives the best separation and correspondence performance for teacher construction.It has undergone cross-view interaction without the specialization of the final geometry-prediction features.
  • Choice of geometry feature layer: Early DA3 features provide weaker geometric context, while final features perform poorly as dense descriptors because of output-geometry specialization.Block 13 is the first cross-view-attention layer, block 19 is intermediate, and block 39 is final.
  • Discriminative objective: The ranking objective is compared with an InfoNCE variant while keeping teacher construction, feature anchoring, and student distillation fixed.The variants differ only in the geometry-supervised discriminative objective.

A.4 Evaluation of Off-the-Shelf 3D-Aware Foundation Features

Off-the-shelf 3D-aware representations are not automatically strong general-purpose dense features, whereas DDMS improves quality across backbones and remains effective without ground-truth geometry.

  • Off-the-Shelf Features: Off-the-shelf VGGT and DA3 features are not automatically strong general-purpose dense representations, while DDMS performs better with a lighter single-view encoder.DDMS also outperforms the comparably sized, single-view DUNE representation.
  • Downstream Pose Estimation: DDMS improves pose recall over original DINOv2 features on ScanNet, 7Scenes, and ETH3D using a shared matching and PnP-RANSAC pipeline.Ground-truth depth constructs reference 3D points so the comparison isolates descriptor quality.
  • Backbone Generalization: DDMS consistently improves feature separation, geometric correspondence, and semantic correspondence across DINOv2-L, DINOv2-B-reg, and DINOv3-B backbones.Semantic segmentation improves on DINOv2 variants and remains comparable on DINOv3.
  • Geometry Dependence: With DA3-predicted geometry instead of ground-truth depth and poses, feature separation and semantic segmentation remain largely unchanged, while correspondence decreases moderately.Accurate ground-truth geometry is beneficial but not strictly required.
  • Out-of-Distribution Transfer: Additional out-of-distribution linear-probe evaluations cover ADE20K and Pascal VOC segmentation and KITTI depth estimation.These benchmarks broaden evaluation beyond the indoor domains used in the main transfer experiments.

B Additional Qualitative Results

Additional qualitative results examine DDMS across correspondence, dense prediction, and explicit 3D aggregation settings, including out-of-distribution benchmarks and novel-view rendering.

  • Correspondence: Qualitative correspondence examples cover ScanNet geometric matching, Navi-Wild object matching, and SPair-71k semantic keypoint matching.The settings test shared-scene geometry, varied visual contexts, and matching across instances of the same category.
  • Dense Prediction: Additional dense prediction examples use frozen-feature linear probes for ADE20K, Pascal VOC, and KITTI.The benchmarks span semantic segmentation and outdoor depth estimation.
  • Dense Prediction: DDMS better preserves scene layout and object-level depth structure than DINOv2 in qualitative depth estimation examples.The comparison uses the same frozen-feature linear-probing setup.
  • 3D Aggregation: Additional 3D aggregation examples evaluate point segmentation after lifting features to point clouds and feature coherence after Gaussian splatting and novel-view rendering.The splatting comparison uses held-out novel views and a fixed uplifting procedure.

C Implementation Details

DDMS is trained with posed multi-view RGB-D supervision by fusing 2D foundation and DA3 geometric features, then distilled into a single-view student for inference.

  • Training Setup: Training uses posed multi-view RGB-D sequences from ScanNet++, with geometry used to construct supervision and candidate labels.The distilled student does not require geometry or multi-view inputs at inference.
  • Teacher Construction: The multi-view teacher fuses pretrained 2D backbone features with frozen intermediate features from Depth-Anything-3.The fused features are projected and refined before distillation.
  • Student Distillation: After teacher refinement, the representation is distilled into a single-view student initialized from the refined 2D foundation branch.The student uses a lightweight 3×3 convolutional refinement head and independently processes selected views.
  • Student Distillation: The teacher processes N = 4 views jointly, while M = 2 views supervise the student with λdisc = 0.1 for 20k optimization steps.Inference requires no multi-view inputs, camera poses, depth maps, or DA3 model.

D.1 Geometric and Semantic Feature Quality

DDMS is evaluated for local feature separation, geometric and semantic correspondence, dense prediction, 3D point segmentation, and rendered feature matching under standardized protocols.

  • Feature Separation: Feature separation measures the query-positive cosine similarity margin against the average similarity to 50 sampled negatives.Negatives exclude pixels within a 30-pixel radius of the positive location.
  • Geometric Correspondence: Geometric correspondence uses nearest-neighbor matching with Lowe-ratio ranking and evaluates retained matches using PCK thresholds on ScanNet and Navi-Wild.Navi-Wild restricts queries to object masks and viewpoint changes below 120°.
  • Semantic Correspondence: Semantic correspondence is evaluated through keypoint matching on SPair-71k and temporal mask propagation on DAVIS-2017.DAVIS uses frozen features without task-specific training and reports J, F, and their average.
  • Dense Prediction Transfer: Frozen-feature linear probes evaluate semantic segmentation and depth estimation with one-layer classifiers and shared protocol settings.Depth probing reports AbsRel, RMSE, and δ1 on ScanNet and NYUv2.
  • 3D Lifting and Rendering: 3D point segmentation averages features projected from posed RGB-D views, while LUDVIG uplifts image features to Gaussians for held-out-view rendering.These tests assess whether features remain useful after explicit 3D aggregation.

E Limitations and Broader Impact

DDMS improves pretrained visual features toward 3D consistency while preserving semantic transferability, but its training and inference scope impose important geometric and data limitations.

  • Limitations: The model uses posed RGB-D data and a geometry foundation model during training to construct multi-view teacher features and geometric candidate labels.
  • Limitations: Noisy or incomplete RGB-D annotations can produce noisy correspondence labels under occlusions, reflective or transparent surfaces, and dynamic objects.
  • Limitations: The distilled student performs efficient single-image inference but does not retain the multi-view attention mechanism of geometry foundation models.
  • Broader impact: DDMS can be applied to different pretrained 2D backbones and geometry models, with potential benefits for visual localization, SLAM, robotics, augmented reality, and scene understanding.
Loading 2608.23850v1…