Source-linked AI summary
Towards Understanding the Use of MLLM-Enabled Applications for Visual Interpretation by Blind and Low Vision People
Ricardo E. Gonzalez Penuela, Ruiying Hu, Sharon Lin, Tanisha Shende, Shiri Azenkot
TL;DR
Prior work found BLV users dissatisfied with AI-generated visual interpretations, and it remains unclear how they use MLLM-enabled applications and whether these meet their needs. A two-week diary study collected 553 entries from 20 BLV participants; in a preliminary analysis, participants reported high satisfaction and trust, including in high-stakes uses.
Problem
Method
In a two-week diary study, 20 BLV participants used a developed MLLM-enabled visual interpretation application, producing 553 diary entries; the preliminary analysis covered 60 entries from six participants.
Source: §1 · p. 2Results
4.15 out of 5 mean satisfaction and 3.75 out of 5 mean trust; participants also used the application for high-stakes scenarios such as identifying medication and receiving dosage advice.
Source: §1 · p. 2Takeaways & Limitations
The authors plan to analyze all 553 entries to deepen understanding of BLV users’ interactions and inform more robust, user-centered visual interpretation systems.
Source: §1 · p. 2The analysis was limited to randomly sampled diary entries rather than participants’ complete diaries.
Source: §5 · p. 7
Abstract
from arXiv · showhide
Blind and Low Vision (BLV) people have adopted AI-powered visual interpretation applications to address their daily needs. While these applications have been helpful, prior work has found that users remain unsatisfied by their frequent errors. Recently, multimodal large language models (MLLMs) have been integrated into visual interpretation applications, and they show promise for more descriptive visual interpretations. However, it is still unknown how this advancement has changed people's use of these applications. To address this gap, we conducted a two-week diary study in which 20 BLV people used an MLLM-enabled visual interpretation application we developed, and we collected 553 entries. In this paper, we report a preliminary analysis of 60 diary entries from 6 participants. We found that participants considered the application's visual interpretations trustworthy (mean 3.75 out of 5) and satisfying (mean 4.15 out of 5). Moreover, participants trusted our application in high-stakes scenarios, such as receiving medical dosage advice. We discuss our plan to complete our analysis to inform the design of future MLLM-enabled visual interpretation systems.
1 INTRODUCTION
The paper examines how BLV people use MLLM-enabled visual interpretation applications and whether they meet users’ needs. In a preliminary analysis, participants reported high satisfaction and trust, including in high-stakes uses, while the authors plan to analyze the full dataset.
1 INTRODUCTION
- Prior work reported low satisfaction with AI-generated interpretations, while MLLMs can use photos and questions to produce more detailed descriptions.The earlier study reported mean satisfaction scores of 2.58 for understanding surroundings and 2.23 for identifying visual qualities such as shirt color, each on a five-point scale.Source: §1 · p. 1
- The study asks what BLV users do with MLLM-enabled interpretation apps, how well their needs are met, and how they converse with the systems.Source: §1 · p. 2
- The two-week study collected 553 diary entries from 20 participants, with this preliminary analysis covering 60 entries from six participants.The 60 entries were 10 randomly selected entries from each of six participants.Source: §1 · p. 2
- Mean satisfaction was 4.15/5 and mean trust was 3.75/5 across the analyzed diary entries.The scores were 4.15 (SD=1.10) for satisfaction and 3.75 (SD=1.03) for trust.Source: §1 · p. 2
- Participants used the application in high-stakes scenarios, including identifying medication and receiving dosage advice.Source: §1 · p. 2
- The authors plan to analyze all 553 entries to investigate users’ problem-solving strategies and interactions in complex tasks.Source: §1 · p. 2
2 RELATED WORK
Related work traces visual interpretation from human assistance to AI applications and MLLM-enabled tools. This paper extends prior research by examining how BLV users interact with conversational visual interpretation systems in everyday use.
2 RELATED WORK
- This work contributes evidence about how BLV users interact with AI-powered visual interpretation systems, building on studies of human collaborators and AI tools.Source: §2 · p. 2
2.1 Human-Powered Visual Interpretation Systems
- Earlier human-powered systems progressed from navigation support to crowdsourced and then real-time assistance for broader daily tasks.Real-time systems support interaction and feedback, while users of asynchronous crowdsourcing were limited to one query per photo.Source: §2.1 · p. 2
2.2 AI-Powered Visual Interpretation Systems
- AI visual interpretation enabled independent photo descriptions, but prior work reported low satisfaction and trust amid frequent errors.Earlier studies of MLLM answers also found that long-form responses could seem plausible to BLV participants despite being incorrect.Source: §2.2 · p. 3
- Research on MLLM-enabled tools documented adoption and accessibility barriers, but did not explain how BLV users employ visual interpretation apps for daily needs.Source: §2.2 · p. 3
3 METHODS
The study used VisionPal in a two-week diary study with 20 BLV participants, then analyzed a random subset of 60 entries and follow-up interviews. The application captured conversations and feedback to examine use cases, needs, and system responses.
3.1 Participants
- Twenty BLV participants used VisionPal for two weeks, and researchers analyzed 60 randomly sampled entries from six participants.The preliminary sample included five participants in the United States and one in Turkey; five identified as Blind and one as Low Vision.Source: §3 · p. 3, §3.1 · p. 3
3.2 Procedure
- The study combined an initial interview, a two-week diary period with at-least-daily use, and a follow-up interview.Diary entries contained photos, descriptions, chat messages, and survey responses; the survey asked about satisfaction, trust, context, information needs, and issues.Source: §3.2 · pp. 3–4
3.3 VisionPal: Our Data Collection Application
- VisionPal was built to resemble commercial apps and capture feedback immediately because those services lacked researcher data-collection access.Each diary submission followed a flow from photo capture and description through optional conversation and a feedback form.Source: §3.3 · pp. 4–5, Figure 1 · p. 4
- VisionPal used GPT-4o and provided an initial photo description followed by optional chat, with the photo and full conversation supplied as context for each response.Source: §3.3 · pp. 4–5
3.4 Preliminary Analysis
- Researchers coded conversations for answer-seeking behavior and response correctness, and coded interviews inductively before validating themes through affinity diagramming.Two researchers first coded 20 conversations and two interviews independently, discussed differences, and one researcher coded the remaining material; four researchers validated the themes.Source: §3.4 · p. 5
4 FINDINGS
Participants reported high satisfaction alongside some skepticism, and their interactions included follow-up conversations, occasional answer errors, and varied question types.
4.1 Satisfaction and Trust
- Satisfaction averaged 4.15 out of 5 and trust averaged 3.75 out of 5, though participants remained slightly skeptical.Forty-eight of 60 entries (80%) included survey responses; 12 entries had incomplete surveys.Source: §4.1 · p. 5
4.2 Conversations
- Participants started conversations in 45 entries (75%), with a mean conversation length of 3.24 messages; engagement varied by participant.Some participants exchanged at most one follow-up message each, while others engaged for up to 14 messages.Source: §4.2 · p. 5, Table 2 · p. 5
- Most participants got desired information within one or two messages, but complex tasks prompted longer exchanges and sometimes required users to correct the AI.During a braille embosser setup, the AI hallucinated braille labels on ink cartridges, leading P1 to guide and challenge it.Source: §4.2 · p. 6
4.3 Questions and Messages
- Of 80 questions, 58 (72.5%) were answered correctly, 15 (18.75%) incorrectly, 5 (6.25%) partially, and 2 (2.5%) went unanswered.P1 and P5 had many incorrect or partial answers, or answers that were not received.Source: §4.3 · p. 6, Table 4 · p. 6
- Participants also asked speculative questions, including about a rug’s likely region of origin and a textbook’s contents.These examples illustrate queries beyond directly identifying or verifying pictured objects.Source: §4.3 · p. 6
- Evidence-based questions were most common (41; 51.25%), followed by visual identification (17; 21.25%) and verification (14; 17.5%).Eight questions (10%) sought instruction, opinions, advice, or reasons, including a medication-dosage question.Source: §4.3 · p. 6, Table 3 · p. 6
5 DISCUSSION AND FUTURE WORK
In a preliminary analysis of 60 diary entries from six of 20 participants, users reported higher satisfaction and trust than in the authors’ earlier study, including in high-stakes and complex situations. The authors caution that positive impressions do not ensure accuracy and plan a fuller analysis of all 553 entries.
5 DISCUSSION AND FUTURE WORK
- The findings come from a preliminary analysis of 60 entries from six of 20 participants, and the authors plan to analyze the full 553-entry dataset.The current analysis used randomly sampled entries rather than participants’ complete diaries; planned coding will examine goals, locations, and output accuracy, alongside statistical tests of performance across use cases and goals.Source: §5 · pp. 6–7
- Satisfaction averaged 4.15 out of 5 in the current study versus 2.76 out of 5 in the previous study, while participants leaned toward trusting the application.The authors contrast this with many instances of dissatisfaction and distrust in the previous study.Source: §5 · p. 6
- Participants used the application in high-stakes and complex situations, including seeking medical dosage information and setting up a braille embossing machine.The authors suggest the application had reached a threshold of usability and trustworthiness for many daily tasks.Source: §5 · p. 7
- Positive trust and satisfaction did not guarantee accuracy: 18.75% of entries contained incorrect information, 6.25% partially correct information, and 2.5% no answer.The authors warn that measuring only trust and satisfaction may mislead or harm users; one example involved hallucinated braille labels during embosser setup.Source: §5 · p. 7
- The authors argue that up-to-date study of authentic conversations matters as visual assistants move toward wearable formats, to understand users’ questions, clarifications, and trust over time.They aim for a more robust, user-centered system grounded in BLV people’s experiences.Source: §5 · p. 7