Source-linked AI summary
Multi-View Reflective Surface Inspection via Semantic-Saliency Cross-Verification
Van-Giang Nguyen, Thanh-Tuan Tran, Xuan-Hieu Phan, Xiem HoangVan
TL;DR
Reflective cover-glass inspection is difficult because defect visibility varies across viewpoints and semantic evidence can remain spatially ambiguous. The paper uses a shared per-view expert that combines VLM semantic boxes with normal-reference saliency, then aggregates evidence across views without registration. On production data, association and additional views improve fixed-proposal detection and product-level recall, supporting their complementary roles.
Problem
Reflective smartphone cover glass is difficult to inspect from one fixed observation because defect visibility depends strongly on imaging condition.
Method
The framework combines VLM class-aware localization with normal-reference saliency, ranks fixed semantic proposals using spatial support, and aggregates per-view evidence without cross-view registration.
Results
Semantic–saliency association raises AP50 from 52.6% to 62.6%, while all three views raise Rprod@0.5 from 75.5% to 88.3%.
Takeaways & Limitations
The results support complementary roles for within-view semantic–saliency association and additional optical observations in reflective-surface inspection.
Takeaways & Limitations
Evaluation is limited to three physical viewpoints, one product family, and defective products only.
Abstract
from arXiv · showhide
Reflective smartphone cover glass is challenging to inspect from a single fixed viewpoint because defect visibility varies with viewing geometry and specular reflections. This gives rise to two practical challenges: defects may be weakly observable from certain viewpoints, while the available visual evidence may remain spatially ambiguous. To address these issues, we propose a multi-view inspection framework in which each RGB observation is processed by a shared per-view expert. A vision-language model (VLM) produces class-aware semantic boxes, while a normal-reference reconstruction branch provides class-agnostic saliency. Their spatial agreement is used as supporting evidence to rank semantic proposals without modifying their coordinates or treating saliency as ground truth. The resulting evidence records are combined at product level without cross-view registration. On 282 production-line images, semantic-saliency association improves $AP_{50}$ from 52.6% to 62.6% by re-ranking fixed semantic proposals. Across 94 products, cross-view evidence recall $R_{\rm prod}@0.5$ increases from 75.5% for the best single view to 88.3% using all three views. These results support the complementary roles of semantic-saliency cross-verification and additional optical observations in reflective-surface inspection.
I. INTRODUCTION
Reflective smartphone cover glass is difficult to inspect from one fixed viewpoint because defect visibility changes with imaging geometry, while semantic and spatial evidence have complementary limitations. The proposed framework combines multi-view optical evidence with semantic–saliency cross-verification and product-level aggregation.
- Motivation: Single-view inspection is challenged by dark, reflective surfaces and geometry-dependent defect contrast.Scratches and cracks may occupy only small image regions and remain inconspicuous under some illumination or viewpoints.
- Motivation: Supervised detectors require labeled examples for the defect vocabulary represented during training.Changes in inspection criteria therefore require corresponding annotations and model adaptation.
- Motivation: Vision-language models offer language-specified semantic grounding, while normal-reference methods provide class-agnostic spatial deviation evidence.The two cues address different limitations: semantic flexibility does not guarantee precise localization, and anomaly responses do not identify defect classes.
- Framework: Multiple observations expose reflective surfaces under different optical conditions rather than recovering geometry or establishing cross-camera correspondence.Each image remains in its native coordinate system, and only the resulting evidence is combined at product level.
- Framework: The framework cross-verifies per-view semantic proposals with saliency before aggregating evidence at product level without cross-view registration.The shared per-view expert combines VLM localization with normal-reference saliency, and the inspection context can retrieve the corresponding manufacturer rule.
II. METHOD
The method separates per-view evidence construction from product-level reasoning. Each observation becomes a common evidence record through semantic–saliency association before records are combined across views.
- II. METHOD: Each observation is converted into a common evidence record before any cross-view combination.This separates per-view evidence construction from product-level reasoning.
A. Per-View Semantic–Saliency Expert
A shared per-view expert extracts class-aware VLM localization and class-agnostic normal-reference saliency, then associates them spatially without altering semantic coordinates. The resulting evidence record preserves the cues needed for product-level reasoning.
- Per-view expert: The VLM supplies defect semantics and localization, while the normal-reference branch supplies spatial deviation evidence.The prediction paths remain separate until spatial association.
- Semantic localization: The VLM uses a fixed inspection prompt to produce defect classes and normalized semantic boxes mapped to image pixels.Invalid classes and malformed boxes are rejected without another language-model call.
- Normal-reference evidence: The reconstruction pathway compares an RGB observation with its normal-reference reconstruction to form a normal-deviation saliency map.Feature-space dissimilarities across levels are combined, then the map is thresholded and small connected components are removed.
- Semantic–saliency cross-verification: Connected saliency regions are boxed only to place class-agnostic responses and class-aware proposals in a common spatial representation.Their spatial consistency can then be measured directly.
- Semantic–saliency cross-verification: Saliency support ranks proposals without changing semantic coordinates or certifying prediction correctness.Weakly supported proposals are retained because reflections can create strong saliency and relational defects can produce weak residuals.
- Evidence record: The per-view evidence record preserves defect semantics, localization, spatial support, and visual evidence as the interface to product-level reasoning.The record summarizes the associated semantic and saliency evidence for the current observation.
B. Multi-View Product Decision
Each product view is processed independently into a shared evidence record, and product-level reasoning combines those records without requiring cross-view registration. A single view can support the final verdict, while retrieval supplies reporting rules only after visual inspection.
- B. Multi-View Product Decision: The same per-view expert converts each observation into an evidence record that can be reused across available views.Adding a view requires another pass through the unchanged expert and produces an additional record.
- B. Multi-View Product Decision: Views remain in native image coordinates because they vary defect visibility rather than reconstructing common geometry.Cross-view registration is therefore not required.
- B. Multi-View Product Decision: Semantic proposals are associated with normal-reference saliency as spatial support while preserving the original semantic boxes.The resulting evidence records are combined at product level.
- B. Multi-View Product Decision: A defect supported by one observation can contribute to the final verdict even when weak or absent in other views.The product-level formulation does not impose a majority rule.
- B. Multi-View Product Decision: Retrieved manufacturer rules and dispositions are attached to supporting evidence after the visual verdict and do not alter the visual decision.Retrieval populates the inspection report.
A. Common Protocol
The evaluation uses supervised and prompt-driven baselines under domain-specific protocols, with localization measured before multi-view aggregation. The SSGD screening subset is diagnostic, while the production-line setup supplies three fixed camera viewpoints.
- A. Common Protocol: The complete per-view expert is compared with Faster R-CNN, RetinaNet, YOLOv8-L, YOLOv11-L, and YOLOv12-L.Direct semantic localization also includes four VLMs and VT-ADL as an anomaly-localization reference.
- A. Common Protocol: AP50 and standard COCO AP evaluate per-image localization before multi-view aggregation.Direct VLM proposals receive a common fixed score, while expert proposals are ranked by o∗vj.
B. Inspection Evaluation
The evaluation separates per-view inspection from product-level aggregation, using SSGD and production-line data to assess semantic localization, evidence association, and detector comparisons.
- SSGD protocol: SSGD uses a fixed five-class subset of 1,310 images, while its 200-image VLM-screening subset is diagnostic and does not evaluate repeated multi-view products.Because SSGD contains individual images rather than repeated views, it evaluates only the per-view component.
- SSGD protocol: 57.1% mean AP50 is achieved by Gemini 3.6 Flash for direct VLM localization, rising to 64.6% with saliency-based ranking.The gain is nonuniform: Spot decreases from 64.0% to 60.5%, supporting soft ranking rather than hard acceptance.
- SSGD protocol: 40.4% versus 39.8% overall AP shows competitive single-view localization against supervised detectors on the 1,310-image SSGD protocol.YOLOv12-L has the highest AP and AP50, whereas the proposed expert has the highest AP75 and APS.
- Production-line protocol: 282 production-line images cover 94 defective products across three fixed RGB viewpoints at nominal 45°, 90°, and 135° orientations.The views are captured sequentially while each product remains stationary; configurations with N ∈{1, 2, 3} are evaluated.
- Production-line protocol: 52.6% macro AP50 for the strongest direct VLM increases to 62.6% for the complete expert, improving all four defect classes.The production-line results are per image and exclude product-level aggregation.
- Production-line protocol: The proposed method is higher in AP, AP75, APS, and APM, while YOLOv12-L remains higher at AP50 and APL on production-line images.The margins are small and do not establish overall superiority.
C. Ablation and Multi-View Evaluation
The ablation holds semantic coordinates fixed to isolate evidence ranking, then evaluates whether additional optical views provide complementary product-level evidence.
- Evidence association: 52.6% AP50 from semantic proposals alone rises to 58.6% with saliency ranking and 62.6% with explicit semantic–saliency region association.Because coordinates remain identical, the 10.0-point gain reflects evidence-based ranking, with region association adding 4.0 points over mean in-box saliency.
- Optical view complementarity: The multi-view metric counts a product when at least one selected view contains a correct-class prediction with IoU ≥0.50.It isolates optical view complementarity because cross-verification neither changes nor discards semantic proposals.
- Optical view complementarity: Rprod@0.5 reaches 75.5% for the best single view, 85.1% for the best pair, and 88.3% for all three views.The second view adds 9.6 percentage points and the third adds 3.2 points within the tested acquisition range.
- Caveats: Spatial agreement supports proposal ranking but is not a correctness certificate because reflection can affect both semantic and saliency cues.Semantic localization is preserved while saliency changes the support attached to each proposal.
- Caveats: Qualitative examples outside the evaluation datasets are consistent with evidence association, but they do not establish out-of-distribution generalization.The examples illustrate cross-verification behavior under additional appearance conditions.
IV. CONCLUSION
The framework combines within-view semantic–saliency association with product-level aggregation across complementary optical views, without changing semantic coordinates or requiring cross-view registration. Evaluation reports improved proposal ranking and higher product-level evidence recall, while remaining limited in scope.
- 62.6% AP50, up from 52.6%, results from semantic–saliency association with fixed semantic coordinates.The association changes proposal support while preserving the original semantic localization coordinates.
- The framework combines VLM semantic localization with normal-reference saliency through spatial support for reflective cover-glass inspection.Representative results show cross-verified evidence across three viewpoints and varied imaging conditions.
- The evaluation is limited to three physical viewpoints, one product family, and defective products only.The product-level VLM is not evaluated independently from evidence availability across views.