Source-linked AI summary
Probing the 3D Awareness of Visual Foundation Models
Mohamed El Banani, Amit Raj, Kevis-Kokitsi Maninis, Abhishek Kar, Yuanzhen Li, Michael Rubinstein, Deqing Sun, Leonidas Guibas, Justin Johnson, Varun Jampani
TL;DR
Visual foundation models achieve broad 2D and semantic generalization, but it remains unclear whether their representations encode the 3D structure of scenes and objects consistently across views. This paper probes frozen representations for surface geometry and multiview correspondence, finding strong single-view encoding in many models but weak consistency under large viewpoint changes.
Problem
Visual foundation models are strong on 2D and semantic tasks, but their representation of 3D structure and cross-view consistency remains understudied.
Method
The paper evaluates frozen features from varied pretrained models using task-specific probes and zero-shot inference for depth, surface normals, and 3D correspondence.
Results
Models generally encode visible-surface depth and orientation, but struggle with multiview consistency at large viewpoint changes; vision-language models are a notable exception for surface properties.
Takeaways & Limitations
Current representations are more view-consistent than 3D-consistent: they can match semantic parts and nearby views without reliably incorporating global object pose.
Takeaways & Limitations
Comparisons are limited by differing training datasets and compute scales, minimal probing approaches, and coverage of only two basic aspects of 3D understanding.
Abstract
from arXiv · showhide
Recent advances in large-scale pretraining have yielded visual foundation models with strong capabilities. Not only can recent models generalize to arbitrary images for their training task, their intermediate representations are useful for other visual tasks such as detection and segmentation. Given that such models can classify, delineate, and localize objects in 2D, we ask whether they also represent their 3D structure? In this work, we analyze the 3D awareness of visual foundation models. We posit that 3D awareness implies that representations (1) encode the 3D structure of the scene and (2) consistently represent the surface across views. We conduct a series of experiments using task-specific probes and zero-shot inference procedures on frozen features. Our experiments reveal several limitations of the current models. Our code and analysis can be found at https://github.com/mbanani/probe3d.
1. Introduction
This paper asks whether visual foundation models represent the 3D structure depicted in images, beyond their demonstrated 2D and semantic capabilities. It evaluates 3D awareness through visible-surface geometry and consistency across views, finding substantial variation across models.
- 1. Introduction: Visual foundation models generalize across classification, segmentation, generation, and zero-shot visual tasks, but their representation of the 3D world remains unclear.The paper frames 3D representation as an open question despite strong image-level and downstream capabilities.
- 1. Introduction: 3D awareness comprises encoding visible-surface geometry and maintaining consistent representations across different views.The proposed criteria include estimating depth, surface orientation, and correspondence between views.
- 1. Introduction: The study evaluates depth, surface normals, and 3D correspondence at both scene and object levels using probes and zero-shot methods on frozen representations.The models span different large-scale pretraining objectives and are assessed without updating their underlying features.
- 1. Introduction: DINOv2 and StableDiffusion encode detailed depth and surface normals, whereas CLIP performs poorly despite strong semantic generalization.The results reveal a large variation in 3D awareness across visual foundation models.
- 1. Introduction: Models generally match objects and scenes under small viewpoint changes but perform poorly under large viewpoint variations.The analysis suggests that cross-view consistency is semantic: models match semantic parts but struggle with global object pose.
2. 3D Aware Visual Representations
The paper defines 3D-aware representations by their encoding of visible-surface properties and their consistency across views. This framing contrasts explicit geometric representations with modern learned feature representations whose 3D content is not directly known.
- 2. 3D Aware Visual Representations: Human perception and mental imagery motivate defining 3D awareness through encoded surface properties and 3D shape constraints.The relevant surface properties include depth and orientation.
- 2. 3D Aware Visual Representations: Early computer-vision representations explicitly modeled object geometry and spatial relationships, whereas recent models rely on dense feature grids or token sets.The 3D awareness of these newer learned representations remains unclear.
- 2. 3D Aware Visual Representations: The proposed evaluation tests single-view surface reconstruction and multiview consistency as distinct capabilities of 3D-aware visual representations.Single-view tests assess visible-surface depth and orientation, while multiview tests assess relationships between images of the same object or scene.
3. Experimental Setup
The experiments probe visual foundation models across single-view surface reconstruction and multiview consistency, using models spanning different supervisory signals. Results show strong variation: many models encode surface properties, but current representations often lose consistency across viewpoints.
- Evaluation Goals: The study evaluates whether models represent visible surfaces and remain consistent across views, using depth, surface normals, and 3D correspondence at scene and object levels.These questions are evaluated across multiple model types and supervisory signals.
- Models: The evaluated models span classification, language supervision, self-supervision, image generation, depth estimation, and class-agnostic segmentation objectives.Public checkpoints were selected with comparable model and training sizes whenever possible.
- Probe: A dense multiscale probe maps features from multiple layers to depth or surface normals instead of requiring 3D properties to be linearly encoded at one layer.The design follows a DPT-like decoder and accommodates information distributed across network locations.
- Single Image Surface Reconstruction: DINOv2 and StableDiffusion produce accurate, detailed depth and surface-normal predictions, whereas CLIP and MAE often produce blurry estimates or rely on coarse priors.DINOv2 captures details such as cow ears and chair legs, while CLIP can appear correct on scenes but inaccurate on objects.
- Single Image Surface Reconstruction: Single-view performance is strongly correlated across tasks and domains, but image-level and pixel-level error correlations can be much weaker.All single-view task correlations exceed 0.82, while DINOv2’s NYU error correlations are 0.37 at image level and 0.13 at pixel level.
- Multiview Consistency: Correspondence is accurate for small viewpoint changes but deteriorates sharply for larger changes, leaving current models insufficiently 3D consistent.StableDiffusion and SAM can drop from top performers to the worst models, while indoor-scene correspondence remains very low for wide baselines.
4. Related Work
Prior work has analyzed learned vision representations, generative-model geometry, and foundation-model features for 3D tasks, but existing probing methods and evaluations leave broad comparisons difficult.
- Representation analysis has examined what vision models learn and how their features transfer to visual tasks.
- Generative-model studies probe single-image geometry, but their techniques often do not extend readily to other visual models.
- Large-scale models have been used for correspondence, pose estimation, fine-tuned depth estimation, and text- or image-conditioned 3D reconstruction.
5. Discussion
The paper finds that visual foundation models encode visible-surface geometry but generally lack robust multiview consistency, while emphasizing limitations in comparisons, probing, and evaluation scope.
- The study defines 3D awareness as encoding visible-surface geometry and maintaining consistency across views, evaluated with trainable probes and zero-shot methods on frozen features.
- Models encode depth and surface orientation, but vision-language models are a notable exception; correspondence also degrades under large viewpoint changes.
- These findings indicate view-consistent rather than 3D-consistent representations, with view-dependent representations or strong image modeling offered as possible explanations.
- The analysis is limited by unequal training data and compute, minimal probing approaches, and coverage of only two basic aspects of 3D understanding.
- Future analysis should address higher-order properties including 3D shape perception, spatial reasoning, deformation, and dynamics.
- The work is presented as a first step toward understanding whether increasingly photorealistic and temporally consistent generation models learn world structure.
A. Additional Experimental Details
The paper summarizes the experimental setup in the main text and provides expanded setup details and design rationales separately to support replication and extension.
- The supplementary section expands the experimental setup and explains the rationale behind selected design choices.
- The authors plan to release code so others can replicate and extend the analysis.
- The code and analysis are associated with the project repository at github.com/mbanani/probe3d.
A.1. Visual Foundation Models
The study evaluates 26 publicly available checkpoints spanning ten learning objectives and five supervision forms, selected to cover major pretraining approaches at comparable scales.
- Models: The experiments include 26 checkpoints spanning ten learning objectives and five forms of supervision.
- Models: Models were selected for coverage of large-scale training approaches and comparable model and training scale.
- Models: Comparisons primarily use listed models, while all checkpoints contribute to correlations in Figure 8 and additional trends in Appendix B.
- Vision-language models: CLIP aligns image and text embeddings through a contrastive objective, with five checkpoints considered.
- Vision-language models: SigLIP replaces contrastive learning with an instance-wise sigmoid loss using image-text embeddings independently.
- Generative models: StableDiffusion uses text-conditioned image generation with a denoising objective, and features are extracted from UNet decoding blocks.
- Depth models: MiDaS models are trained on monocular depth datasets with a scale-invariant depth-estimation objective.
A.2. Evaluation Datasets
The evaluation uses complementary datasets spanning object-centric and scene-centric settings, with annotations and splits tailored to depth, normals, and correspondence tasks.
- NAVI: NAVI contains 36 objects across varied poses and environments, with aligned meshes providing depth and pose annotations.The dataset is extended with surface-normal annotations and excludes two objects lacking both multiview and wild-set images.
- NAVI: NAVI uses multiview images for training and validation, while wild-set images are reserved for testing and correspondence pairs span 0–120° relative rotation.
- NYU v2: NYU Depth v2 provides RGB-D indoor-scene videos with dense depth and semantic-segmentation annotations, plus surface normals for 1,449 labeled images.For depth estimation, unlabeled instances expand training to 24,231 images.
- ScanNet Pairs: ScanNet Pairs supplies 1,500 RGB-D image pairs as a correspondence benchmark, all used for testing because correspondence estimation requires no training.
- SPair 71k: SPair-71k contains image pairs from 18 PASCAL categories with class-specific keypoints and human-annotated viewpoint variation, including eight non-rigid categories.
A.3. Evaluation Tasks
The evaluation probes single-image 3D understanding and multiview consistency through depth, surface-normal, geometric-correspondence, and semantic-correspondence tasks.
- Task Overview: The four tasks evaluate the two targeted dimensions of 3D awareness: single-image 3D understanding and multiview consistency.Depth and surface-normal models are trained, whereas correspondence is evaluated directly from frozen features.
- A.3.1 Monocular Depth: Monocular depth estimation predicts a depth value for every image pixel, but scale ambiguity makes generalization across camera settings difficult.
- A.3.1 Monocular Depth: AdaBins represents depth with probabilities over 256 uniformly distributed bins, whose weighted bin centers produce the final depth estimate.The method uses a 0–10m range for NYU and a 0–1 range for NAVI.
- A.3.1 Monocular Depth: The multiscale convolutional depth probe combines features from several network stages rather than relying on a single layer.This design accommodates differing feature granularity and layer specialization across models.
- A.3.2 Surface Normals: Surface-normal estimation predicts a unit vector orthogonal to the surface at each pixel, using four probe outputs for three directional components and uncertainty.
- A.3.2 Surface Normals: Surface-normal performance is measured by per-pixel angular error, image-level RMSE, and accuracy thresholds of 11.25°, 22.5°, and 30°.
- A.3.3 Geometric Correspondence: Geometric correspondence matches pixels depicting the same 3D point across viewpoints, evaluated with 2D projection or 3D metric error.Scene-centric evaluation considers all pixels, whereas object-centric evaluation restricts pixels to the object mask.
- A.3.3 Geometric Correspondence: Correspondences come from nearest neighbors in feature space, are filtered with Lowe’s ratio test, and the top 1,000 matches are retained.Performance is reported as correspondence recall across viewpoint-change ranges, whose values are not directly comparable between objects and scenes.
B. Additional Results
Additional analyses show that training objective and architecture relate to 3D awareness, while dataset and scale effects are smaller or mixed. Keypoint correspondence also varies with viewpoint, deformation, and semantic structure, and controlled experiments remain necessary.
- Model architecture: Changing CLIP’s backbone from ViT to ConvNeXt produces the greatest improvement, qualitatively changing depth predictions from flat surfaces to depth-map-like outputs.This does not establish ConvNeXt as universally superior: DeiT’s ViT outperforms ConvNeXt under supervised ImageNet-22k training.
- Training augmentation: Augmentation boosts CLIP performance, but the gains are relatively small and its effect with ViT backbones remains unclear.
- Model scale: Larger backbones typically improve performance only marginally, without changing the reported patterns or correlations.
- Training objective and dataset: Training objective appears to drive more performance variation than dataset choice, although no single factor explains vision-language models’ low performance.The authors report that dataset effects are often mixed or marginal, while objective-related differences are larger.
- Keypoint correspondence: For horses and airplanes, unique keypoints remain accurately matched across large viewpoint changes, whereas semantically related or deformable parts become confused.Examples include horse knees and hooves, and airplane wheels, wings, and stabilizers; confusion is restricted to semantically equivalent classes for deformable parts.
C. Limitations
The study’s conclusions are constrained by uncontrolled checkpoint comparisons, a limited definition of 3D awareness, and reliance on frozen-feature probing rather than fine-tuning.
- Comparison constraints: Publicly available checkpoints differ in training recipes and datasets, making it unclear whether observed trends reflect major factors or minor implementation details.The authors note that ideally models would share architecture, data, and training recipe while varying only objectives, but the required resources and data are unavailable.
- Comparison constraints: Comparable model capacity and pretraining-data scale improve fairness, but the authors still call for more controlled model or dataset comparisons.They report that DINO consistently outperforms CLIP despite CLIP’s larger dataset, while CLIP performs similarly across WIT and LAION.
- Scope of 3D awareness: The analysis covers single-view surface reconstruction and multiview consistency, but not the full range of 3D understanding properties.Unexamined properties include complete 3D shape, deformation, and physical properties such as support and containment.
- Methodological boundary: The methodology uses linear probes and zero-shot analyses on frozen features, leaving transfer to fine-tuning setups for future study.The authors chose frozen features to inspect representations without adapting them to 3D tasks.