PaperlayerView exact revision on arXiv

Source-linked AI summary

A General Protocol to Probe Large Vision Models for 3D Physical Understanding

Guanqi Zhan, Chuanxia Zheng, Weidi Xie, Andrew Zisserman

arXiv:2310.06836v3cs.CV

TL;DR

Because large vision models are trained mainly on 2D images that project the physical world, the extent to which their features represent 3D scene properties remains an open question. The paper probes annotated real images with classifiers trained on pre-trained model features. Stable Diffusion and DINOv2 perform well on several properties, though less well on material and occlusion.

  • Problem

    Because large vision models are trained mainly on 2D images that project the physical world, it remains an open question how well they represent the 3D scenes depicted.

    Source: §1 · p. 1

  • Method

    The paper probes pre-trained model features with linear classifiers on property-annotated real-image datasets, selecting layers or diffusion time steps for region-pair questions.

    Source: §1 · p. 1, §3.2 · p. 4, §3.3 · p. 7

  • Results

    Stable Diffusion and DINOv2 perform well on geometry, support relations, lighting, and depth but less well on material and occlusion, and outperform OpenCLIP, DINOv1, and VQGAN across properties.

    Source: §1 · p. 2, §4.4 · p. 9

  • Takeaways & Limitations

    Property-predictive features may support shadow-object or support-relation prediction, while identifying weak properties may help spot Stable Diffusion-generated images.

    Source: §1 · p. 2

    Linear probes perform poorly for material and occlusion, leaving unclear whether the properties are weakly modelled or require a more expressive probe.

    Source: §5 · p. 9

Abstract

from arXiv · show

Our objective in this paper is to probe large vision models to determine to what extent they 'understand' different physical properties of the 3D scene depicted in an image. To this end, we make the following contributions: (i) We introduce a general and lightweight protocol to evaluate whether features of an off-the-shelf large vision model encode a number of physical 'properties' of the 3D scene, by training discriminative classifiers on the features for these properties. The probes are applied on datasets of real images with annotations for the property. (ii) We apply this protocol to properties covering scene geometry, scene material, support relations, lighting, and view-dependent measures, and large vision models including CLIP, DINOv1, DINOv2, VQGAN, Stable Diffusion. (iii) We find that features from Stable Diffusion and DINOv2 are good for discriminative learning of a number of properties, including scene geometry, support relations, shadows and depth, but less performant for occlusion and material, while outperforming DINOv1, CLIP and VQGAN for all properties. (iv) It is observed that different time steps of Stable Diffusion features, as well as different transformer layers of DINO/CLIP/VQGAN, are good at different properties, unlocking potential applications of 3D physical understanding.

1 Introduction

The paper asks whether large vision models trained on 2D images encode properties of the 3D scenes those images depict, and proposes a lightweight protocol to probe their features. Stable Diffusion and DINOv2 perform well on several properties but less well on material and occlusion.

Evaluation protocol

  • The protocol evaluates whether pre-trained model features represent 3D scene and viewpoint properties by training and testing simple classifiers on those features.It selects real-image datasets with property annotations, searches model layers or diffusion time steps, and evaluates the selected feature and classifier on a test set.Source: §1 · pp. 1–2

Scope

  • The investigation covers scene geometry, material, support relations, shadows, occlusion, and depth across Stable Diffusion, OpenCLIP, DINOv1, DINOv2, and VQGAN.The models span generative, self-supervised, weakly supervised, and autoregressive training approaches.Source: §1 · p. 2

Key findings

  • Stable Diffusion and DINOv2 perform well on scene geometry, support relations, lighting, and depth, but less well on material and occlusion.They generally outperform OpenCLIP, DINOv1, and VQGAN; the best Stable Diffusion time steps and transformer layers vary by property.Source: §1 · p. 2

Why probe these models?

  • The study aims both to clarify whether these models implicitly model 3D scenes and to identify features that could support property prediction.The authors also note that weak properties may help identify Stable Diffusion-generated images and indicate where further training could improve 3D modelling.Source: §1 · pp. 2–3

2 Related Work

Prior work has explored model representations, used large-scale models for downstream tasks, and studied individual 3D physical properties. This paper distinguishes its approach by probing a wider set of properties across models with a general, efficient protocol.

2.1 Exploration of Pre-trained Models

  • Earlier work visualized representations in discriminative and generative models, while this paper probes a wider range of geometric and physical properties across model layers and time steps.The authors also contrast their work with concurrent research on depth, surface normals, and geometric correspondence.Source: §2.1 · p. 3

2.2 Exploitation of Pre-trained Models

  • Prior work exploited large-scale models for tasks such as segmentation, depth, correspondence, and pose; this paper instead studies 3D physical understanding in models trained without 3D supervision.Source: §2.2 · p. 3

2.3 3D Physical Scene Understanding

  • Earlier 3D physical-scene studies usually addressed one or two properties and often trained a supervised model, whereas this work uses one model to predict multiple properties without training its features.Source: §2.3 · p. 3

3 Method – Properties, Datasets, and Classifiers

The method probes large vision models for physical and viewpoint-dependent properties by training classifiers on region features from real images. It spans scene geometry, material, support, shadows, occlusion, and depth, with datasets and probing choices matched to each property.

3 Method – Properties, Datasets, and Classifiers

  • The method applies linear probing to features from Stable Diffusion, OpenCLIP, DINOv1, DINOv2, and VQGAN across layers and, for Stable Diffusion, time steps.For each property, the protocol uses annotated real-image datasets and searches validation performance to select features and classifier settings.Source: §3 · p. 3, §3.3 · p. 7

3.1 Properties and Questions

  • Seven pairwise questions probe same or perpendicular planes, material, support, object-shadow relations, occlusion, and relative depth.These questions target geometry, surface material, physical support, lighting, and viewpoint-dependent relations.Source: §3.1 · p. 4

3.2 Datasets

  • Annotated real-image datasets provide region pairs and binary labels, with geometry, material, support, shadow, occlusion, and depth drawn from property-specific datasets.The datasets include ScanNetv2, DMS, NYUv2, SOBA, and Separated COCO; Table 1 reports image counts for train, validation, and test splits, capping test sets at 1,000 images when the original set is larger.Source: §3.2 · pp. 4–6, Table 1 · p. 4

3.3 Property Probing

  • The protocol extracts region features from Stable Diffusion and transformer layers, then uses linear SVMs and validation search to select layers, time steps, and regularization.Stable Diffusion features are extracted at different time steps and U-Net layers; other models use transformer layers. Region features are pooled over masks, and symmetric relations use absolute feature differences while support and depth use ordered differences.Source: §3.3 · pp. 6–7

4 Experiments

The experiments select linear probes over model-specific features and compare their performance across 3D properties. Stable Diffusion and DINOv2 representations outperform OpenCLIP, DINOv1, and VQGAN across the tested properties.

4.1 Implementation Details and Evaluation Metric

  • The evaluation balances positive and negative pairs, tunes SVM penalty C, and uses ROC AUC, which is insensitive to decision thresholds.The Stable Diffusion search spans timestep, U-Net layer, and C; other models are searched over transformer layer and C.Source: §4.1 · p. 7

4.2 Results for Stable Diffusion

  • Stable Diffusion performs well on scene geometry, support relations, shadows, and depth, but less well on material and occlusion; D3 is its best U-Net layer across properties.The best timestep varies by property and is usually below 400; Figure 5 illustrates material and occlusion prediction failures.Source: §4.2 · p. 8, Table 2 · p. 8, Figure 5 · p. 8

4.3 Results for CLIP/DINO/VQGAN Features

  • For OpenCLIP, DINOv1, DINOv2, and VQGAN, validation searches select transformer layer and SVM C, with different layers performing well for different properties.Table 3 reports grid-search results for material and support; results for other tasks are in the appendix.Source: Table 3 · p. 8, §4.3 · p. 9

4.4 Comparison of Different Features Trained at Scale

  • Stable Diffusion and DINOv2 representations outperform OpenCLIP, DINOv1, and VQGAN on all properties in the test-set comparison.Table 4 reports ROC AUC using the best grid-searched settings for each feature family.Source: Table 4 · p. 9, §4.4 · p. 9

5 Discussion and Future Work

The discussion frames the protocol as a way to examine whether large-scale image models represent 3D physical properties. It highlights promising feature choices while noting linear-probe limits and the study’s scope.

5 Discussion and Future Work

  • The protocol examines whether large-scale image-pretrained models have explicit feature representations for different 3D physical-scene properties.Different Stable Diffusion timesteps and DINOv2 layers handle several properties at state-of-the-art performance, suggesting downstream-task potential.Source: §5 · p. 9
  • Material and occlusion are not distilled into features usable by a linear probe, which may reflect weak modeling or a need for a more expressive probe.The appendix also reports such prediction failures on generated images.Source: §5 · p. 9
  • Probed Stable Diffusion features receive preliminary downstream-task evaluation, and properties the model handles poorly are also explored for spotting generated images.Source: §5 · p. 9
  • The study addresses real images, while contact relations, object orientation, and more nuanced asymmetric formulations remain possible properties for future investigation.Source: §5 · p. 9

A More Implementation Details

The appendix details Stable Diffusion feature extraction, dataset partitioning and filtering, and the linear-SVM evaluation setup.

A More Implementation Details

  • Stable Diffusion features are averaged across eight random-noise additions, computed in one batch, using an empty text prompt.The feature extraction follows DIFT [38].Source: A More Implementation Details · p. 14
  • For NYUv2 and ScanNetv2, training and validation images come from different scenes; when fewer images are used, splits are randomly sampled.Inaccurately annotated images are manually filtered, and regions smaller than 100 pixels are excluded.Source: A More Implementation Details · p. 14
  • Positive and negative pairs are balanced; depth pairs require a 1.2× average-depth difference, while perpendicular-plane pairs use specified angle ranges.Perpendicular pairs have angles between 85° and 95°; non-perpendicular pairs have angles below 60° or above 120°.Source: A More Implementation Details · p. 14
  • Feature vectors are L2-normalized before a linear SVM makes binary decisions using sign(wT v + b).The passage also identifies separate question groups for same-plane, perpendicular-plane, material, shadow, and occlusion questions versus support-relation and depth questions.Source: A More Implementation Details · p. 14

B More Stable Diffusion Generated Images

Stable Diffusion’s inpainting examples illustrate scene geometry, support relations, shadows, and depth, alongside occasional material and occlusion failures.

B More Stable Diffusion Generated Images

  • The examples broadly show Stable Diffusion representing several physical properties, while sometimes failing on material boundaries and occlusion.The authors describe these examples as matching their feature-probing findings.Source: B More Stable Diffusion Generated Images · p. 15
  • Failures to separate materials clearly or reconnect occluded parts are offered as cues for spotting Stable Diffusion-generated images.The examples cited are the second row of Figure 7 and first row of Figure 8.Source: B More Stable Diffusion Generated Images · p. 15
  • Masked plane intersections are inpainted as perpendicular planes meeting at corners across sofa, wall, cabinet, pillar, and fridge examples.The caption says the model generates the corner based on the unmasked portions of the planes.Source: Figure 6 · p. 16
  • Inpainting examples show supporting objects and shadows rendered correctly, but one material example has a mixed boundary between pancake and plate.Another material example distinguishes the materials clearly.Source: Figure 7 · p. 17
  • Occlusion examples include failures and successes in connecting separated parts; depth examples generate larger cars when closer and smaller cars when farther away.The cat-tail example fails, while separated people and oven parts are connected.Source: Figure 8 · p. 18

C Additional Results for CLIP/DINO/VQGAN Features Trained at Large Scale

The appendix identifies grid-search result tables for other large-scale model features across five property tasks, without supplying their cell values here.

C Additional Results for CLIP/DINO/VQGAN Features Trained at Large Scale

D Train/Val AUC Results of Stable Diffusion Features

This appendix table presents Stable Diffusion feature grid-search train/validation AUC results at the selected settings.

D Train/Val AUC Results of Stable Diffusion Features

E AUC Curves for Stable Diffusion Features Grid Search

The appendix provides AUC curves across Stable Diffusion layers and time steps for probing scene properties.

E AUC Curves for Stable Diffusion Features Grid Search

  • Figures 9 and 10 show curves for the same-plane and perpendicular-plane tasks.Both compare AUC across Stable Diffusion layers and time steps.Source: Figure 9 · p. 22, Figure 10 · p. 23
  • Figures 11 and 12 show curves for the material and support tasks.Both compare AUC across Stable Diffusion layers and time steps.Source: Figure 11 · p. 23, Figure 12 · p. 24
  • Figures 13 and 14 show curves for the shadow and occlusion tasks.Both compare AUC across Stable Diffusion layers and time steps.Source: Figure 13 · p. 24, Figure 14 · p. 25
  • Figure 15 shows curves for the depth task.The curves compare AUC across Stable Diffusion layers and time steps.Source: Figure 15 · p. 25

F Preliminary Results of Applying Probed Feature for Downstream Tasks

The paper preliminarily applies a probed Stable Diffusion feature to surface-normal estimation with iDisc, reporting better performance on most metrics after 1,000 iterations.

F Preliminary Results of Applying Probed Feature for Downstream Tasks

  • After 1,000 iterations, injected Stable Diffusion features improved most surface-normal metrics over iDisc’s reported numbers.The comparison is reported in Table 11.Source: F Preliminary Results of Applying… · p. 26, Table 11 · p. 26
  • The experiment injected the perpendicular-plane probe feature where iDisc’s encoder encodes image features.The authors report training for 1,000 iterations versus 45,000 total iterations for iDisc.Source: F Preliminary Results of Applying… · p. 26
  • The authors suggest probed features may also be used for instance shadow detection, material segmentation, and support-relation inference.These applications are proposed for future use if the features are injected or otherwise utilized.Source: F Preliminary Results of Applying… · p. 26

Related papers

Loading 2310.06836v3…