Source-linked AI summary
SpatialTrust: A Benchmark for Environmental Risk Recognition in Secure Authentication
Junbin Lu, Hsiang-Wei Huang, Saesha Wadhwa, Yu Ting Hsu, Jenq-Neng Hwang
TL;DR
Existing MLLM evaluations rarely test whether models can recognize, explain, and spatially ground environmental risks in secure-authentication scenes. SpatialTrust introduces a benchmark for these abilities and evaluates proprietary and open-source models, while SpatialTrustGuard improves Qwen3-VL-30B-A3B-Instruct from 36.78% to 41.12% overall. The results show that indirect-risk reasoning and spatial risk awareness remain challenging.
Problem
Existing MLLM evaluations rarely assess reliable recognition, localization, and explanation of environmental risks in spatially grounded authentication scenarios.
Method
SpatialTrust is a question-answering benchmark covering detection, direct and indirect factor identification, and direct and indirect factor explanation in verification captures.
Results
36.78% to 41.12%: SpatialTrustGuard improves Qwen3-VL-30B-A3B-Instruct’s overall score, while current models remain limited, especially on indirect factors and grounded explanations.
Takeaways & Limitations
Spatial risk awareness remains challenging for MLLMs, motivating dedicated benchmarks and structured inference methods for more trustworthy authentication systems.
Abstract
from arXiv · showhide
Visual environmental risk recognition plays an important role in secure authentication, where a user's surroundings may reveal sensitive information or introduce potential security risks. However, existing evaluations of multimodal large language models (MLLMs) rarely examine whether models can reliably recognize, localize, and explain such risks in spatially grounded authentication scenarios. We present SpatialTrust, a question-answering benchmark for evaluating environmental risk recognition in secure authentication. SpatialTrust assesses five complementary abilities: sensitive factor detection, direct factor identification, indirect factor identification, direct factor explanation, and indirect factor explanation. We evaluate both proprietary and open-source MLLMs and find that current models show limited performance, especially in understanding and explaining indirect risks, indicating that spatial risk awareness remains a challenging capability for MLLMs. In addition, we introduce SpatialTrustGuard, a structured QA-and-audit pipeline that improves Qwen3-VL-30B-A3B-Instruct from 36.78% to 41.12% overall. Our findings highlight the need for dedicated benchmarks and structured inference methods to improve the trustworthiness of MLLMs in secure authentication.
1. Introduction
SpatialTrust addresses the gap in evaluating whether MLLMs can recognize, distinguish, explain, and spatially ground environmental risks in authentication captures. Its evaluation finds substantial limitations, while SpatialTrustGuard improves Qwen3-VL-30B-A3B-Instruct’s overall score through structured QA and auditing.
- Authentication captures can expose background factors that leak information from the login computer or create privacy and security risks.
- The benchmark requires models to identify risks, explain them, and ground relevant visual evidence with bounding boxes.
- Current MLLMs struggle with distinguishing direct from indirect factors, detecting subtle background risks, avoiding unsupported identifications, and maintaining complete grounding.
- 36.78% to 41.12%: SpatialTrustGuard improves Qwen3-VL-30B-A3B-Instruct’s overall score using answer selection, conservative auditing, and grounded response generation.
- SpatialTrust evaluates five abilities: sensitive factor detection, direct and indirect factor identification, and direct and indirect factor explanation.
2. Related Work
Prior visual privacy and MLLM safety research addresses privacy attributes, image-level privacy, assistive images, harmful scenarios, and identity linkage. SpatialTrust instead targets spatially grounded information-leakage risks in laptop-camera-based login verification.
- Prior visual privacy work studies privacy attribute recognition, image privacy prediction, and private visual information in assistive settings.
- SpatialTrust focuses on laptop-camera verification, where observers or cameras in the background may leak information from the login computer.
- Recent MLLM evaluations examine image-based safety attacks, harmful queries, and individual-level privacy reasoning through identity linkage.
3. SpatialTrust Benchmark
SpatialTrust benchmarks spatially grounded recognition of environmental information-leakage risks in laptop-camera verification, focusing on visible background and non-primary-subject factors. It combines direct and indirect risk distinctions, five structured questions, edited authentication scenarios, and answer-plus-grounding evaluation.
- Benchmark scope: SpatialTrust evaluates five abilities: sensitive factor detection, direct and indirect factor identification, and direct and indirect factor explanation.Each item asks models to determine whether sensitive factors are present, classify them, explain them, and ground visible evidence with bounding boxes.
- Evidence grounding: All predictions must rely on visible evidence, avoid unsupported inferences, and provide bounding boxes when sensitive factors are visible and localizable.The benchmark therefore checks consistency among selected options, textual explanations, and spatial evidence.
- Risk definitions: Direct factors visibly create immediate disclosure, whereas indirect factors create a possibility or risk of disclosure without directly disclosing information in the current image.Examples include an observing person or aimed camera for direct risk, versus an open doorway, exposed layout, or potentially observing person for indirect risk.
- Benchmark construction: The benchmark is constructed from sampled video frames through preprocessing, identity-level face replacement, situation-level editing, QA generation, and human review.Situation-level diversification creates varied authentication scenarios and sensitive-factor configurations around a central user and surrounding scene.
- Dataset statistics: SpatialTrust contains 577 images and 2,885 QA items, including 577 answer-only and 2,308 grounded questions across single-choice, multiple-choice, and short-answer formats.At image level, 391 images contain at least one sensitive factor, while 186 contain none; direct and indirect categories can overlap, with 109 images containing both.
- Evaluation protocol: Evaluation combines exact-match or option-level answer scoring with grounding verification using Intersection over Union, with τ = 0.5 used in all experiments.For free-form answers, a structured LLM judge assesses semantic identification, explanation correctness, and unsupported predictions; extra incorrect options receive zero credit for the question.
4. SpatialTrustGuard
SpatialTrustGuard replaces one-pass answering with staged candidate selection, conservative verification, and grounded generation constrained by verified choices. The design separates recognition from explanation and localization while preventing unsupported factors from being introduced later.
- Method overview: SpatialTrustGuard decomposes inference into candidate selection, conservative self-verification, and final grounded answer generation.The pipeline is designed to improve consistency without fine-tuning or additional training data.
- Staged QA Decomposition: Candidate selection asks only whether sensitive factors exist and which direct or indirect factors are visible.Explanations and localization are postponed so basic recognition is separated from open-ended generation.
- Conservative Self-Verification: Conservative verification retains a selected factor only when its complete description is clearly supported by visible evidence.The verifier evaluates selected q2 and q3 options under an evidence-sufficiency criterion.
- Grounded Answer Generation: Final explanations and grounding are generated only from verified q2 and q3 choices, with unsupported categories reported as lacking visual evidence.Fixed semantic commitments prevent later generation from introducing new direct or indirect factors.
- Design rationale: Separating answer selection from explanation and localization reduces interference among detecting factors, classifying their type, explaining them, and locating evidence.One-pass prompting can produce mismatches between selected factors, explanations, and bounding boxes.
- Conservative Self-Verification: The verifier can delete unsupported candidates but cannot add missed factors, replace choices, or alter q1.Deletion-only editing targets false positives that could otherwise produce fluent explanations and plausible-looking grounding regions.
5. Experiments
Experiments evaluate eight proprietary and open-source MLLMs on five SpatialTrust question types using grounded evidence scoring. Models detect possible risks more reliably than they identify, explain, and localize them, while SpatialTrustGuard improves the Qwen3-VL baseline overall despite a detection decrease.
- Experimental setup: Eight proprietary and open-source MLLMs are evaluated on SpatialTrust using five task-level scores and an overall average.The five scores cover sensitive factor detection, direct and indirect factor identification, and direct and indirect factor explanation.
- Main results: Most models score around 66–69% on sensitive factor detection but drop substantially on factor identification, explanation, and grounding.The results indicate that detecting possible factors is easier than localizing evidence and explaining authentication-related leakage.
- Main results: Gemini-2.5-flash outperforms GPT-5.4 overall, mainly through stronger factor identification and explanation.Among open-source models, Qwen3-VL-30B-A3B-Instruct performs best overall at 36.78%.
- SpatialTrustGuard results: 41.12% is SpatialTrustGuard’s overall score, up from 36.78% for Qwen3-VL-30B-A3B-Instruct.Indirect factor identification rises from 28.65% to 34.20%, while direct and indirect factor explanation rise from 25.74% to 35.44% and 22.91% to 29.38%.
- SpatialTrustGuard results: 65.51% is SpatialTrustGuard’s sensitive factor detection score, down from 67.42% after conservative deletion-only verification.The overall score still improves because the benchmark emphasizes correct identification, explanation, and grounding.
- Ablation study: 38.96% is the score after adding Staged QA Decomposition, and Conservative Self-Verification raises it further to 41.12%.The ablation attributes the gains to intermediate answer states and additional auditing of unsupported factor claims.
6. Conclusion
SpatialTrust evaluates whether MLLMs can recognize, distinguish, explain, and spatially ground environmental information-leakage risks in secure authentication captures. Models remain limited, while SpatialTrustGuard improves performance through staged inference and conservative self-verification.
- SpatialTrust evaluates recognition, distinction, explanation, and visual grounding of environmental information-leakage risks in authentication captures.
- Current MLLMs remain limited, particularly in distinguishing direct from indirect sensitive factors and grounding explanations in relevant evidence.
- 41.12% overall, up from 36.78%, is achieved by SpatialTrustGuard through staged inference with conservative self-verification.
- The results suggest structured decomposition can improve consistency, while spatially grounded environmental risk recognition remains challenging for reliable, evidence-based MLLMs.