Source-linked AI summary
Evaluating Visual Language Models on Visual Grounding for Visually Impaired People
Fairooz Azim, Lydia Achour, Claudia Borg, Lina Rojas-Barahona
TL;DR
Visual grounding remains underexplored in VLMs evaluated on images taken by visually impaired people, although it matters for trustworthy assistive VQA. The paper extends an explainability framework into VG-SHAP and finds that models can answer correctly without grounding in the proper region, though segmentation quality limits that conclusion.
Problem
Method
Results
Takeaways & Limitations
For VIP assistive use, an answer’s correctness alone is insufficient when users need to verify its image evidence and understand uncertainty.
Source: §1 · p. 1The findings are constrained by segmentation-model quality, so the authors cannot yet attribute the observed grounding gap to VLMs’ grounding ability on VIP images.
Source: §7 · p. 4
Abstract
from arXiv · showhide
Visual grounding (VG) in visual question answering (VQA) is a fundamental capability of Vision-Language Models (VLMs), enabling the detection of the most relevant region in an image to answer the question. This reflects the model's capability of multimodal comprehension, which is crucial when interacting with visually impaired people (VIP). Indeed, visual grounding, when applied to assistive scenarios, is not merely a technical feature, but rather a trust and safety requirement. In this work, we study visual grounding in VLMs for visually impaired people. To this end, we assess open-source VLMs on visual grounding using the VizWiz-VQA-Grounding dataset and challenge, which contains real-life images taken by VIP. We propose VG-SHAP, which extends an existing explainability framework to provide visual grounding explanations, allowing the evaluation of black-box VLMs. Our results on the VizWiz-VQA-Grounding challenge show that despite generating correct answers, VLMs struggle to ground in the proper region in the image. However, these results are constrained by the quality of the segmentation model of our explainability framework and further research is necessary.
1 Introduction
Visual grounding matters for assistive VQA because users need to know that an answer comes from the relevant image region. The paper evaluates open-source VLMs and finds a gap between answer correctness and grounding, with results qualified by reliance on external segmentation models.
1 Introduction
- Grounding helps VIPs verify that answers about medication, dosage, or expiration dates come from the correct part of an image and communicate uncertainty.The introduction frames grounding as essential to trust and safety in critical assistive scenarios.Source: §1 · p. 1
- Grounding capabilities on VIP images remain underexplored, while many VLMs are assessed mainly for textual correctness and do not expose internal attention maps.This makes direct assessment of their visual reasoning difficult.Source: §1 · p. 1
- VLMs can answer correctly without grounding in the proper image region, but external segmentation models constrain the preliminary result.The authors say further investigation is needed before drawing definitive conclusions about reliability and trustworthiness.Source: §1 · p. 1
2 Related Work
The paper builds on model-agnostic, perturbation-based attribution and Shapley methods, extending V&L-SHAP to produce visual grounding explanations. It focuses on static VIP images and evaluates grounding with the VizWiz-VQA-Grounding challenge metrics.
2 Related Work
- Perturbation-based attribution masks or alters segmented image regions to measure their impact on a VLM’s output, enabling model-agnostic evaluation of black-box models.Shapley-based methods integrate segmentation and SHAP to score each image segment’s contribution, yielding a mask of critical visual evidence.Source: §2 · pp. 1–2
- The paper extends V&L-SHAP [2] from natural-language explanations to visual grounding and explores newer segmentation models.The segment contribution scores serve as a pixel-level measure of grounding.Source: §2 · p. 2
- Unlike EgoBlind [18], which evaluates video QA, this work studies grounding in static VIP images using VizWiz-VQA-Grounding [3] and its objective challenge metrics.The approach also extends a model-agnostic explainability framework.Source: §2 · p. 2
3 Visual Grounding for VIP
The paper evaluates visual grounding in realistic images taken by VIPs, using the VizWiz-VQA-Grounding dataset and challenge. It emphasizes the safety stakes of grounding and describes the mask-overlap metrics used for evaluation.
3 Visual Grounding for VIP
- VizWiz-VQA-Grounding [3] contains realistic images taken by VIPs, which can be blurry or out of focus, and its test annotations are withheld.Evaluation therefore uses submissions to the EvalAI leaderboard.Source: §3 · p. 2
- An answer derived from an irrelevant image correlation can be unusable and potentially dangerous for VIPs, who otherwise must accept it on faith.The paper connects verifiable grounding with trust, interpretability, and privacy preservation in real-world systems.Source: §3 · p. 2
- IoU measures overlap between predicted and ground-truth masks, while Dice is more sensitive to small overlaps and useful with imbalanced class distributions.IoU ranges from 0 to 1, with higher values indicating better alignment.Source: §3 · p. 2
4 Visual Grounding Explainability: VG-SHAP
VG-SHAP extends V&L-SHAP to produce visual grounding explanations as binary masks. It combines standardized image segments with Kernel SHAP attribution to assess a VLM’s answer without relying on internal model access.
4 Visual Grounding Explainability: VG-SHAP
- VG-SHAP formalizes its explanation as a binary mask used to compute visual grounding, rather than generating a natural-language explanation.Figure 1 depicts the framework’s core method, inputs, and outputs.Source: §4 · p. 2, Figure 1 · p. 2
- Feature extraction segments an image into meaningful, non-overlapping regions, replacing DFF with STEGO, SAM, SAM2, or GroundedSAM2.The resulting masks are standardized before being passed to the explainer.Source: §4 · p. 2
- Kernel SHAP uses weighted linear regression to estimate segment contributions by evaluating the VLM across feature-subset coalitions.The method prioritizes higher-weight coalitions through deterministic k-best sampling.Source: §4 · p. 2
5 Experimental Setup
The experiments move from exploratory zero-shot VQA and segmentation selection to evaluation on the VizWiz-VQA-Grounding validation set and official challenge test set.
Experimental workflow
- The study first qualitatively inspects zero-shot VQA, then explores segmentation models, and finally evaluates selected models on the official challenge.The test-set annotations are not publicly available.Source: §5 · p. 3
Preliminary zero-shot VQA
- Five open-source VLMs are qualitatively assessed on 50 randomly selected training samples using answer completeness, correctness, and hallucination as criteria.Models receive only the image and question, without prompt-induced constraints.Source: §5 · p. 3
- The preliminary zero-shot analysis is intended for exploratory model selection; the main grounding evaluation uses the validation set and official challenge benchmark.The 50-sample analysis is not the main grounding evaluation.Source: §5 · p. 3
Segmentation models evaluation
- SAM, SAM2, and GroundedSAM2 are compared for Gemma3-4B on 50 validation samples using 10-fold cross-validation, semantic metrics, and execution speed.A standardized assistive prompt is used, and speed is measured on an RTX 3090 server.Source: §5 · p. 3
VizWiz-VQA-Grounding challenge
- The two best prompted models from the preliminary steps are evaluated on the VizWiz-VQA-Grounding validation set and its official challenge test set.Source: §5 · p. 3
6 Results
The results report preliminary model behavior, compare segmentation approaches, and evaluate selected VLMs on the VizWiz-VQA-Grounding benchmark. Grounded-SAM2 performs best among the tested segmentation models, but grounding remains limited.
Preliminary zero-shot VQA
- Qwen2.5-3B, Molmo-7B, and Gemma3-4B generally produced standard-length, correct, visually grounded answers with minimal hallucination in the qualitative sample.LLaVA-7B showed weaker correctness and significant hallucination, while Phi-4-multimodal-instruct showed variable responses and some hallucinations.Source: §6 · p. 3
Segmentation models evaluation
- Grounded-SAM2 is faster and more accurate than SAM and SAM2, with IoU gains of +0.1 over SAM and +0.08 over SAM2.The comparison uses Gemma3-4B with P2 parameters; Table 2 defines t as execution time per image and v as speed.Source: §6 · p. 3, Table 2 · p. 4
- The reported runtime and speed comparison uses a single fold on a 4-GPU RTX 3090 server because resource allocation could not be matched on the HPC cluster.This qualifies the hardware comparison reported in Table 2.Source: §6 · p. 4
VizWiz-VQA-Grounding challenge
- Grounded-SAM2 outperforms the other segmentation models, but IoU remains relatively low on the VizWiz-VQA-Grounding evaluation.Table 3 reports results for Qwen2.5-VL-3B and Gemma3-4B-it on the validation set and official challenge test set.Source: §6 · p. 4, Table 3 · p. 4
Visual explanation example
- In one SAM explanation, 89 segments produced 289 possible coalitions, and the first 300 coalitions yielded a blue mask with cosine similarity 1.0.The authors report that this mask contained no noise observed with STEGO and covered the details in the answer.Source: §6 · p. 4, Figure 2 · p. 4
7 Conclusion and Future Work
The paper presents VG-SHAP as a model-agnostic framework for evaluating visual grounding in images taken by visually impaired people. Its findings suggest correct answers may lack correct-region grounding, but segmentation quality constrains that conclusion.
Conclusion
- VG-SHAP combines external segmentation with SHAP-based attribution to assess VLM grounding without access to internal weights or model architecture.The final grounding mask depends on the quality and granularity of the segmentation produced by the external model.Source: §7 · p. 4
Limitations
- SAM can generate excessive fine-grained segments that increase computational overhead and only partially cover relevant regions, lowering IoU and Dice even when answers are correct.Grounded-SAM outperforms the other SAM variants, but grounding scores remain low.Source: §7 · p. 4
Future work
- Because results are constrained by segmentation quality, the authors cannot conclude that VLMs have poor grounding on VIP images.They propose further investigation using attention-based interpretability in open-source models and correlation analysis with VG-SHAP.Source: §7 · p. 4
A Prompts
The appendix provides the assistive prompt used in the segmentation evaluation and challenge experiments.
Prompt
- The prompt addresses the model as an assistant for visually impaired users.It is identified as the prompt used in the segmentation-model evaluation and VizWiz-VQA-Grounding challenge.Source: A Prompts · p. 5
- It asks for a specific answer without extra explanation based on the image.The prompt emphasizes concise responses.Source: A Prompts · p. 5
- When an image is too blurry to answer reliably, the prompt instructs the model to respond “unanswerable.”Source: A Prompts · p. 5
B Qualitative Analysis
Qualitative analysis finds that correct answers can be grounded in incomplete or irrelevant image regions. Difficult image conditions and fragmented objects can compound segmentation errors in the final grounding mask.
Qualitative Analysis
- In blurry, occluded, or poorly illuminated images, segmentation errors can propagate into the final grounding mask, especially with fine-grained segmentations such as SAM.Source: B Qualitative Analysis · p. 5
- When relevant objects are fragmented into multiple regions, the explainer may select only a partial visual cue.Source: B Qualitative Analysis · p. 5
- A correct answer can still have a grounding mask that misses the annotated region, as in the Gemma3-4B-it soup example.Source: Figure 3 · p. 5, B Qualitative Analysis · p. 5