Source-linked AI summary
{\Phi}eat: Physically Grounded Material Feature Representation
Giuseppe Vecchio, Adrien Kaiser, Claudia Cuttano, Rouffet Romain, Rosalie Martin, Elena Garces, Tamy Boubekeur
TL;DR
General-purpose vision representations often emphasize semantics and do not explicitly model physical material properties needed for material reasoning. Φeat adapts a pretrained visual backbone by contrasting the same material under varied lighting and geometry. It provides material-aware features that outperform general-purpose backbones in material-centric applications.
Problem
General-purpose visual representations prioritize semantics and may not explicitly capture physical material properties such as reflectance, limiting their fit for material reasoning.
Source: §1 · p. 2Method
Φeat specializes a pretrained DINOv3 ViT [45] by contrasting renderings of the same material across varied geometry and lighting, with cross-material contrastive training.
Source: §1 · p. 2, §4.2 · pp. 7, 9Results
Φeat provides strong priors for material-centric tasks and outperforms general-purpose backbones in those applications without dense per-pixel labels during pretraining.
Source: §1 · p. 2Takeaways & Limitations
Physically inspired weak supervision can produce features tailored to material perception and transfer to downstream material tasks.
Source: §7 · p. 15Pretraining relies entirely on synthetic data, and material-centric adaptation incurs a moderate loss on general-purpose semantic tasks.
Source: §6 · p. 15
Abstract
from arXiv · showhide
While foundation models have emerged as general-purpose visual backbones, their representations are primarily optimized for semantics and lack explicit modeling of physical factors, such as reflectance, hindering their efficacy in tasks requiring explicit material reasoning. We introduce $\Phi$eat$, a novel material-grounded visual backbone that encourages a representation sensitive to material identity, including reflectance and mesostructure. Instead of relying on generic data augmentations, we pretrain our model by contrasting observations of the same material under controlled variations in lighting and geometry. This encourages invariance to extrinsic factors while preserving sensitivity to intrinsic material properties. We show that the resulting representation provides strong priors for material-centric tasks, including feature-based material selection and classification. Our results demonstrate that physically inspired weak supervision is an effective strategy for learning representations tailored to material perception.
1 Introduction
General-purpose vision representations prioritize semantic and geometric cues, leaving physical material properties less explicit. Φeat specializes a pretrained backbone with physically varied views of the same material to capture material identity while remaining robust to context.
1 Introduction
- Standard vision backbones often capture robust geometric and categorical cues but struggle to separate the physical factors that shape scene appearance.Generic photometric augmentations can destroy subtle material cues or encourage object-level consistency instead.Source: §1 · p. 2
- Material identity includes properties such as reflectance and mesostructure, while geometry, illumination, and local orientation are extrinsic factors that can change appearance.The paper notes that material identity may matter more than object labels in intrinsic decomposition, material capture, and robotics.Source: §1 · p. 2
- The method contrasts renderings of the same material across geometries and lighting, using physical triplets and cross-material contrast to favor intrinsic appearance over object semantics.The approach specializes a pretrained DINOv3 ViT [45].Source: §1 · p. 2
- Φeat captures intrinsic reflectance and surface structure while remaining robust to physical context, providing priors for material-centric tasks without dense per-pixel pretraining labels.The paper positions this as a material-grounded feature extractor for downstream applications.Source: §1 · p. 2
- The contributions include a weakly supervised backbone for physically grounded material features and a material-aware pretraining strategy using varied lighting and geometry.The data-generation contribution uses designed mesh templates to render many materials.Source: §1 · p. 3
2 Related Work
Prior work spans intrinsic-scene understanding, material recognition, and general-purpose visual encoders, but material reasoning often remains tied to task-specific supervision or semantic features. Φeat instead targets generic material-aware features through physically grounded weak supervision.
2 Related Work
- Intrinsic-scene and material methods have advanced, but many remain narrow regression tasks that rely on dense labels or controlled capture setups.Some recent methods address unknown lighting but still use task-specific supervision.Source: §2 · p. 3
- Φeat contrasts with approaches that depend on frozen semantic backbones, supervised layers, or task-specific pipelines by learning generic material-aware features directly.Its weak supervision aligns representations with physical appearance cues from the same material under changing lighting and geometry.Source: §2 · p. 4
- Foundation encoders learn transferable semantics or spatial features through image-text alignment, self-supervision, and teacher-student distillation, including DINOv3’s Gram anchoring.These methods provide the general visual representations that Φeat physically grounds.Source: §2 · p. 4
3 Data Collection and Curation
The data pipeline holds material identity constant while varying geometry and lighting, aiming to teach physical rather than contextual similarity. Semantically aligned templates and procedural materials support realistic, diverse synthetic renders.
3 Data Collection and Curation
- The dataset varies lighting and shape while keeping material identity fixed, encouraging features grounded in physical appearance rather than semantic or contextual similarity.The approach does not require anchoring the representation to a specific physical model such as a reflectance distribution function.Source: §3 · pp. 4–5
- Geometric templates are paired with semantically plausible material categories, such as cork on a rigid, low-curvature surface, to avoid unrealistic combinations.Figure 2 illustrates templates rendered with different materials; the full dataset has approximately one million renders.Source: §3 · p. 5, Figure 2 · p. 5
- The pipeline uses over 9,500 procedural materials across 21 appearance classes to synthesize approximately 36,000 unique sets of PBR maps.Instances are rendered on aligned geometries under four randomly selected HDR environment maps from a set of 20.Source: §3 · p. 5
- GPU-native path tracing uses 128 samples per pixel, denoising, and displacement-map tessellation; varying object and environment rotations adds view and light variation.Rendering transparent holes against environment maps adds variability, and the process yields roughly one million renders.Source: §3 · pp. 5–6
4 Method
Φeat replaces generic photometric augmentation with physically varied renderings of the same material, training for invariance to geometry and lighting while retaining material identity. Its training objectives combine teacher–student alignment with material-level contrast and auxiliary regularizers.
4 Method
- Φeat contrasts renderings of the same material across geometries and lighting, shifting pretraining toward physical invariance rather than semantic consistency across crops.The strategy replaces generic photometric jitter with structured extrinsic variation.Source: §4 · pp. 6–7, §4.2 · p. 9
4.1 Model Architecture
- The backbone uses DINOv3’s ViT, with 16 × 16-pixel patch tokens, a global [CLS] token, register tokens, and RoPE positional embeddings.Final-layer outputs include patch-level features and a global representation from [CLS].Source: §4.1 · p. 7
4.2 Training Objectives
- Training combines image-level and masked patch-level objectives with KoLeo, Gram anchoring, and material contrastive losses.The patch-level objective predicts teacher embeddings at masked student-patch positions.Source: §4.2 · pp. 7–8
- KoLeo encourages dispersion among normalized student features, while Gram anchoring aligns second-order patch relations to a frozen teacher.The KoLeo objective promotes a more uniform feature distribution on the unit hypersphere.Source: §4.2 · p. 8
- An in-batch InfoNCE term pulls global embeddings of the same material together and pushes embeddings of different materials apart.Its positives are other views of the same material; remaining batch samples are negatives.Source: §4.2 · pp. 8–9
- The total loss weights the image, iBOT, KoLeo, Gram, and contrastive objectives, with coefficients reported in Section 5.Together, the augmentation strategy and cross-material contrastive loss shift pretraining toward physical invariance across extrinsic conditions.Source: §4.2 · p. 9
5 Experiments
The experiments evaluate Φeat’s material alignment, feature separability, robustness, and retained semantic utility. They report stronger material-focused results than general-purpose baselines, alongside qualitative material coherence and an ablation showing the role of multi-render supervision and contrastive regularization.
5 Experiments
- Evaluations test material selection, k-NN separability, and prediction stability under lighting and geometry changes, against foundation-model baselines with ViT-B backbones.The study also measures retained semantic performance using ImageNet classification and ADE20K/Cityscapes segmentation.Source: §5 · p. 9, §5.1 · p. 10
5.1 Quantitative Results
- Φeat achieves the strongest overall material-selection and k-NN classification performance across the reported metrics, outperforming generic feature baselines including DINOv3.Material selection uses patch-level backbone features without supervised task adaptation; the classification evaluation uses k-NN retrieval.Source: Table 1 · p. 10, §5.1 · pp. 11–12
5.2 Semantic Performance
- Compared with DINOv3, Φeat loses 3.4 ImageNet accuracy points and 5.3/5.4 mIoU points on ADE20K/Cityscapes, while retaining much of its semantic utility.The authors frame this as a specialization tradeoff: DINOv3 remains stronger on purely semantic tasks, while Φeat offers more material-grounded features.Source: §5.2 · pp. 12–13, Table 2 · p. 14
5.3 Qualitative Results
- Across video frames, Φeat better preserves coherence and alignment with material boundaries than DINOv3 and PE Spatial.The examples show more uniform asphalt selection and identification of leather across different objects.Source: Figure 5 · p. 13, §5.3 · p. 14
- Φeat’s similarity maps and K-means clusters align with material boundaries and reflectance or texture, unlike baseline features that often follow semantic or spatial cues.The qualitative comparisons include patch similarity in images and propagated selections across video frames.Source: Figure 4 · p. 11, §5.3 · pp. 13–14
5.4 Ablation Study
- Single-render material supervision improves selection but reduces k-NN accuracy; multi-render training improves both, and adding contrastive regularization yields the strongest performance across metrics.The contrastive term is reported to restore separability through compact intra-material clusters and clearer inter-material boundaries.Source: Table 4 · p. 14, §5.4 · p. 15
6 Limitations
Φeat has limits in physical interpretability, training-data scope, semantic utility, and the range of modeled light transport.
6 Limitations
- Φeat does not explicitly disentangle its latent space into interpretable physical factors such as roughness, metalness, or albedo.The authors suggest that separating these factors could allow manipulation along specific physical dimensions.Source: §6 · p. 15
- Pretraining relies entirely on synthetic data, leaving integration of unlabelled real-world data through domain adaptation or self-training as an open challenge.Source: §6 · p. 15
- Material-centric adaptation incurs a moderate loss on general-purpose semantic tasks compared with the original backbone.The paper describes this as a specialization tradeoff between semantic utility and material alignment.Source: §6 · p. 15
- The current physical model focuses primarily on surface reflectance and shading; more complex light transport, including translucency and subsurface scattering, remains to be modeled.Source: §6 · p. 15
7 Conclusion
The conclusion presents Φeat as a weakly supervised backbone for physically grounded appearance representations, trained on renderings that vary geometry and lighting. It reports transfer to material-focused tasks and argues that material-aware pretraining can produce representations robust to extrinsic factors without semantic labels.
7 Conclusion
- Φeat learns physically grounded appearance representations from unlabelled data using weak supervision.Source: §7 · p. 15
- The method contrasts renderings of the same material across geometry and lighting conditions, combining DINO-based teacher–student alignment with contrastive regularization.The representation is described as capturing material identity while remaining invariant to shape and illumination.Source: §7 · p. 15
- The learned features transfer to fine-grained material selection, unsupervised segmentation, and clustering under changing illumination and geometry, without semantic labels.Source: §7 · p. 15