Source-linked AI summary
Glass Surface Detection Grounded in 3D Visual Geometry
Yiwei Lu, Ke Xu, Tao Yan, Xiaojun Chang, Radu Timofte, Rynson W. H. Lau
TL;DR
Glass surface detection is difficult because transparency and specularity create semantic and geometric ambiguity, while appearance-based methods can fail in such scenes. The paper grounds GSD in VGGT-derived 3D geometry and combines frequency-domain localization with geometry-aware segmentation. It reports state-of-the-art performance across seven benchmarks, strong generalization, and improved glass-scene reconstruction.
Problem
Transparency and specularity create semantic and geometric ambiguity, and existing appearance-based methods can struggle when glass lacks distinct visual patterns.
Method
The framework distills VGGT 3D priors into glass-aware supervision and uses FSAM and GeGB in a multitask glass detection head.
Results
The method achieves state-of-the-art performance across seven standard GSD benchmarks, generalizes to multimodal and video data, and substantially improves glass-scene reconstruction.
Takeaways & Limitations
Grounding GSD in 3D visual geometry provides a framework for detecting glass surfaces while representing their physical existence.
Takeaways & Limitations
Performance may fail when high-quality inferred geometric contexts are limited, such as when a windshield is too close to the camera.
Abstract
from arXiv · showhide
Glass surface detection (GSD) is critical for scene understanding and reconstruction, and yet remains challenging due to the transparency and reflectivity of glass surfaces. Existing GSD methods typically rely on 2D appearance cues, which may fail in geometrically ambiguous scenes. In this paper, we propose a paradigm shift: grounding GSD in 3D visual geometry to explicitly model the physical existence of glass surfaces. Our method first distills rich 3D priors from the visual geometry grounded transformer (VGGT) and generates glass-aware 3D representations. It then exploits multi-tasking learning with a novel glass detection head, consisting of two core modules: a Frequency Self-Attention Module (FSAM) that identifies glass-specific spectral features for glass surface localization, and a Geometry Grounding Block (GeGB) that selectively grounds 2D features in 3D geometry for glass surface segmentation. Extensive experiments demonstrate that our method achieves state-of-the-art performance across seven standard GSD benchmarks, generalizes well to video/multi-modal data, and substantially improves reconstruction in glass scenes. Code is available in https://github.com/YT3DVision/VGGT_GLASS.
1 Introduction
Glass surface detection is difficult because transparency and specularity create semantic and geometric ambiguity, while appearance-based methods can miss weak visual cues. The paper addresses this by grounding detection in 3D geometry with VGGT-derived representations and a glass-specific detection head.
- Transparency and specularity mix transmitted backgrounds with foreground reflections, creating ambiguity that challenges depth, optical flow, reconstruction, and navigation.
- Existing GSD methods mainly use RGB or multimodal appearance cues, but these approaches are limited when glass lacks distinctive visual patterns.
- The proposed paradigm explicitly reasons about geometric inconsistencies, including depth and surface-normal information, to model glass as a physical object.
- VGGT supplies large-scale 3D priors, including camera pose, depth, and point clouds, but directly applying it is difficult because it can see through glass and miss the surface.
- The framework generates refined pseudo-ground-truth 3D supervision and combines FSAM for frequency-domain localization with GeGB for geometry-aware segmentation.
- Experiments report state-of-the-art performance on seven benchmarks, generalization to multimodal and video data, and substantial improvements in glass-scene reconstruction.
2 Related Work
Prior GSD research uses RGB appearance cues or additional modalities, while recent foundation-model methods still primarily reason from 2D appearance. The paper positions 3D visual geometry as an alternative for robust detection in visually ambiguous scenes.
- RGB-based methods exploit contextual features, boundaries, reflections, ghosting effects, and temporal information, whereas multimodal methods add polarization, thermal infrared, near-infrared, or depth.
- Existing appearance-focused methods struggle when glass surfaces do not exhibit distinct visual patterns.
- LoRA adapts frozen foundation-model backbones through learnable low-rank matrices, reducing trainable parameters while preserving pretrained knowledge.
- The paper applies vanilla LoRA to adapt VGGT for GSD, preserving its 3D representations with minimal parameter overhead.
3 Method
The method grounds glass surface detection in 3D geometry through rectified VGGT-based supervision and a multi-task glass detection head. FSAM captures spectral appearance cues, while GeGB integrates complementary depth and point-cloud context for geometry-aware segmentation.
- 3.1 Pipeline: The method then performs end-to-end multi-task learning with a glass detection head grounded in 3D visual geometry.The adapted VGGT backbone extracts features that are processed for glass surface segmentation and boundary delineation.
- 3.1 Pipeline: The framework first generates rectified 3D pseudo-ground truth from VGGT depth and point clouds, using glass masks to correct glass-region geometry.Boundary-aware anchoring preserves perspective consistency, while planar interpolation supplies consistent depths within interior glass regions; corrected depths are combined with initial point clouds.
- 3.3 Glass Detection Head: The decoder progressively fuses glass appearance, depth, and point features to produce increasingly fine-grained glass segmentation and boundary results.Multi-scale VGGT token representations receive FSAM processing, while decoder stages use GeGB to enrich features with multimodal geometry.
- 3.3 Glass Detection Head: FSAM operates in the frequency domain to capture prominent spectral features and preliminarily localize glass regions.It targets glass-induced high-frequency suppression and produces glass features with aligned spatial and channel dimensions.
- 3.3 Glass Detection Head: GeGB grounds 2D appearance features in complementary depth and point-cloud features for glass-region reasoning and segmentation.Depth provides dense pixel-aligned distances, while point clouds provide local surface structure and volumetric context; spatial attention highlights discriminative geometric regions.
4 Experiments
Experiments evaluate the method across seven GSD benchmarks, multiple modalities, video, visual cases, reconstruction, ablations, and runtime. The method shows broad performance advantages, preserves generalization without extra sensors, improves glass-scene reconstruction, and runs interactively.
- Evaluation Setup: The evaluation covers seven standard benchmarks spanning single-image, RGB-Thermal, RGB-Depth, and video GSD datasets.The study reports IoU, F-measure, MAE, BER, and accuracy, comparing against GSD methods and foundation-model baselines.
- Single-Image GSD Results: Our method maintains a clear performance advantage across all evaluated single-image datasets and metrics.Competing methods were retrained under identical training settings; the comparisons include six single-image methods and three foundation models.
- Generalization Results: The method achieves comparable performance to RGB-Thermal approaches without extra thermal or depth sensors and outperforms RGB-Depth methods.These comparisons are reported on two multimodal GSD datasets.
- Generalization Results: The method consistently outperforms GlassWizard across all evaluation metrics on the video GSD dataset.GlassWizard leverages large-scale 2D diffusion priors, while GhostingNet can degrade when ghosting cues are weak or ambiguous.
- Visual Results: Visual comparisons show accurate segmentation in scenes with weak cues, full glass coverage, and partially occluded glass regions.Previous appearance-based methods may over- or underdetect glass in these challenging cases.
- Application: The method improves reconstruction in glass regions by incorporating planar geometric priors that represent the physical glass surface.Original VGGT tends to reconstruct objects behind transparent surfaces while neglecting the glass itself.
- Ablation Studies: Ablations show that removing model components or training objectives generally produces sub-optimal performance, while relying entirely on frequency information can over-detect glass.FSAM supports glass localization, GeGB resolves appearance ambiguities with geometric structure, and corrected 3D supervision provides consistent gains.
- Runtime Efficiency: The method runs interactively at around 8.5 FPS on a single NVIDIA RTX 4090 GPU.The runtime comparison includes GhostingNet, GlassWizard, and a SAM3-based baseline.
5 Conclusion
The paper grounds glass surface detection in 3D visual geometry using VGGT-derived priors and a multi-task glass detection head. It reports state-of-the-art performance across seven benchmarks, strong generalization, interactive runtime, and a limitation when geometric cues are insufficient.
- Conclusion: The framework uses VGGT-derived pseudo-ground-truth depths and points, then applies FSAM and GeGB in a multi-task glass detection head.FSAM characterizes glass surfaces through high-frequency attenuation, while GeGB grounds 2D frequency features in 3D geometric features.
- Conclusion: The model achieves new state-of-the-art performance across seven standard benchmarks and demonstrates strong generalization capability.It also improves reconstruction in glass scenes and runs interactively at around 8.5 FPS on a consumer-level GPU.
- Limitations: The method may fail when inferred geometric contexts are limited, such as when a windshield is too close to the camera.The stated limitation concerns reliance on high-quality 3D visual geometry cues.