PaperlayerView exact revision on arXiv

Source-linked AI summary

VIABench: A Comprehensive Video Benchmark Collected from Blind Individuals for Visual Impairment Assistance

Yunfeng Liu, Yuandong Yang, Jiarui Han, Zhenpeng Huang, Yuqing Tang, Xiangyu Zeng, Gangshan Wu, Limin Wang

arXiv:2607.14660v1cs.CV

TL;DR

Existing assistive datasets do not capture the complexity and dynamic nature of blind users’ everyday experiences. VIABench benchmarks three assistance tasks, and evaluation finds current MLLMs’ proactive capabilities remain nascent, especially relative to VQA.

  • Problem

    Existing assistive datasets fall short in capturing the complexity and dynamic nature of everyday experiences encountered by visually impaired users.

    Source: §1 · p. 1

  • Method

    VIABench is a real-world video benchmark covering Proactive Reminder, VQA, and Vision-Guided Interaction, with TPAD for efficient assessment of proactive alerts in long videos.

    Source: §1 · p. 3

  • Results

    Current MLLMs show nascent proactive capabilities, especially when contrasted with their VQA performance.

    Source: §6 · p. 8

  • Takeaways & Limitations

    The paper identifies advancing proactive-response mechanisms, enriching training data, and tailoring models for assistive scenarios as directions toward more capable and dependable assistive technologies.

    Source: §6 · p. 9

    Real-world visual assistance is open-ended, so strong VIABench performance does not guarantee readiness for rare events, edge cases, or situational challenges.

    Source: §11 · p. 20

Abstract

from arXiv · show

Visually impaired individuals (VIIs) encounter significant daily challenges due to limited access to visual information. Although Multimodal Large Language Models (MLLMs) have achieved impressive results on general vision and language tasks, their practical utility in real-world blind assistance still remains largely underexplored. To fill this gap, we introduce VIABench, a comprehensive video benchmark specifically designed to evaluate MLLMs in Visually Impaired Assistance scenarios using first-person videos recorded or shared by VIIs themselves. VIABench defines three core tasks, each targeting a distinct requirement in visual assistance. Proactive Reminder: Assesses the model's ability to interpret ongoing video content while proactively anticipating and verbally describing upcoming navigation-critical events; Visual Question Answering (VQA): Evaluates the model's capacity to answer user-posed questions about the environment or objects within the video; Vision-Guided Interaction: Tests context-aware reasoning to accomplish intentional interactions between user and environment. To ensure a robust and fair evaluation, we propose a rigorous benchmarking pipeline that supports both online (real-time) and offline settings. Our experiments demonstrate that current MLLMs still struggle to deliver comprehensive support for VIIs, especially in the Proactive Reminder task, which demands accurate anticipation and real-time responsiveness. We hope VIABench will drive future research toward developing customized MLLMs for real-world assistance, ultimately improving navigation and interaction experiences for visually impaired individuals. Code and data will be released at https://github.com/MCG-NJU/VIABench.

1. Introduction · 2. Related Work

VIABench is a real-world, egocentric video benchmark for visual impairment assistance that combines autonomous reminders with user-initiated tasks. The paper also introduces TPAD to evaluate reminder-style assistance and situates the benchmark among MLLMs and earlier assistive datasets.

1. Introduction

  • Traditional mobility aids provide only partial support, while traditional assistive vision algorithms have single-task designs and generalize poorly to real-world scenarios.MLLMs’ multi-task learning, zero-shot performance, and natural-language interaction make Jarvis-like visual assistance increasingly feasible.Source: §1 · p. 1
  • Earlier assistive datasets do not capture the complexity of everyday VII experiences: WAD [37] uses short curated clips, while EgoBlind [29] focuses on passive, prompted QA.WAD’s YouTube travel vlogs introduce a domain gap, and EgoBlind does not evaluate proactive assistance.Source: §1 · p. 1
  • The three tasks span proactive reminders, user-question answering from current and past context, and iterative guidance to complete an instructed goal.They represent increasing levels of assistive complexity, from autonomous alerts to question answering and action-oriented interaction.Source: §1 · pp. 2–3, Figure 1 · p. 2
  • VIABench collects long-form egocentric videos from visually impaired individuals and unifies three assistive tasks, capturing both autonomous and user-initiated behavior.The videos have greater total duration and temporal richness than prior visual-assistance video benchmarks.Source: §1 · p. 3
  • TPAD is a two-stage framework for efficiently and reliably evaluating reminder models’ balance between action and inaction in real-time streams.The challenge is to communicate critical events proactively while remaining silent when no relevant information is present; existing online benchmarks [15] [16] use costly pipelines and sparse event annotations.Source: §1 · p. 3

2. Related Work

  • General-purpose MLLMs, including Gemini 1.5 [21] and Qwen2-VL [25], perform strongly on video understanding benchmarks [3] [9] [13] [26] [39], but goal-oriented assistance remains underexplored.The paper focuses on applying MLLMs to visual impairment assistance.Source: §2 · p. 3
  • Prior VII datasets range from VizWiz’s [11] real-world images to video QA datasets EgoBlind [29] and VIEW-QA [20], but these videos focus on post-hoc QA.WalkVLM [37] explores proactive reminders using approximately 3-second curated YouTube clips, which provide insufficient temporal context and differ from real-world egocentric videos.Source: §2 · p. 3

3. VIABench

VIABench evaluates visual-assistance models across three complementary tasks using first-person videos from visually impaired individuals and simulated interaction scenarios. Its long-form, navigation-centered dataset also tests robustness to degraded egocentric footage.

3.1. Task Definition

  • Across three complementary tasks, responses should be accurate, concise, and informative: proactive reminders anticipate navigation events, VQA answers queries, and interaction adapts until goals are met.Proactive Reminder comprises 21 fine-grained sub-tasks; Vision-Guided Interaction is multi-turn and closed-loop.Source: §3.1 · pp. 3–4

3.3. Statistics and Comparative Analysis

  • 761 videos and 14,526 manually curated annotations span 46.9 hours; average video duration is 222 seconds (max: 1959s), versus 40s in EgoBlind and 3s in WalkVLM.The duration histogram highlights the collection’s range of short and long-form videos.Source: Table 1 · p. 4, §3.3 · p. 4, Figure 3 · p. 5

3.2. Data Collection and Annotation

  • A five-stage pipeline sourced real VII videos and added 231 purpose-filmed interaction clips, then used pilot labeling, vendor annotation with multi-stage QA, and English translation.The pilot involved 14 trained annotators labeling 150 videos and producing 3.5K annotations; final annotations were translated using GPT-4o.Source: §3.2 · p. 4, Figure 2 · p. 5

3.3. Statistics and Comparative Analysis

  • 530 real VII videos (~44 hours) and 231 simulated videos fill interaction coverage; annotations center directional cues and obstacles, aligning the dataset with navigation-focused real-world assistance.The simulated videos were designed to complete coverage of Vision-Guided Interaction scenarios.Source: §3.3 · pp. 4–5, Figure 3 · p. 5
  • 94% of the data (44 hours) comes from VII recordings often affected by occlusion, abnormal exposure, or camera rotation; robustness evaluation tests whether models flag unreliable input and request camera adjustment.The paper argues that missing such warnings can create safety-critical situations for visually impaired users.Source: Table 1 · p. 4, §3.3 · p. 5

4. Method

VIABench maps assistive needs to three evaluation tasks and introduces TPAD to make offline MLLMs usable for proactive reminder evaluation.

4.1. Adapting MLLMs to Assistive Scenarios

  • VIABench links navigation, blind questioning, and interaction to Proactive Reminder, VQA, and Vision-Guided Interaction, covering when to act, how to answer, and how to assist.The framework is intended to assess MLLMs against distinct cognitive and perceptual challenges in assistive scenarios.Source: §4.1 · p. 6
  • Proactive Reminder evaluates continuous monitoring of streaming video and timely hazard alerts, enabling assessment of offline models alongside online systems.The task addresses alerts for hazards such as steps, obstacles, or sidewalk boundaries, which users need without explicitly requesting them.Source: §4.1 · p. 6
  • VQA answers user queries using only video information available before query time, preventing access to future frames and supporting temporal grounding and causal reasoning.The benchmark adopts an online VQA setting following OVOBench [15].Source: §4.1 · p. 6
  • Vision-Guided Interaction simulates multi-turn assistance, requiring each response to use accumulated dialogue and current visual context to maintain coherent, adaptive, grounded guidance.The model must update its instructions as the scene evolves because users cannot visually verify their progress.Source: §4.1 · p. 6

4.2. Token-Level Prompt Activation Decoding

  • TPAD adapts offline MLLMs without fine-tuning by combining a forced-choice prompt and video frames, then using each frame’s final-token hidden state to estimate alert probability in one pass.Each frame’s hidden state is projected through the language-modeling head to obtain alert probability.Source: §4.2 · pp. 6–7, Figure 4 · p. 6
  • TPAD uses one holistic forward pass rather than one inference per segment, conditioning frame predictions on preceding visual context and improving stability over frame-isolated baselines.The authors report particular value in ambiguous or partially observable scenes, where local frame evidence is insufficient.Source: §4.2 · p. 7

5. Experiments

VIABench evaluates three assistance tasks across open-source, proprietary, and streaming MLLMs. Results show early-stage competence in real-world assistance, with particular difficulty in proactive reminders and vision-guided interaction.

5.1. Evaluation Details

  • The evaluation covers all three VIABench tasks across open-source foundation, proprietary, and online streaming MLLMs.Open-source models include InternVL3.5 [27], Qwen2.5VL, LLaVA-OneVision-1.5 [1], LLaVA-Video [38], MiniCPM-V 4.5 [35], MiniCPM-o 2.6 [32], and Kimi-VL [22], plus smaller variants. Proprietary baselines include GPT-5, GPT-4o, and Gemini-2.5 Pro [6]; streaming models include LiveCC [5], VideoLLM-Online [4], and StreamingVLM [30].Source: §5.1 · p. 7

5.2. Main Results

  • Proactive Reminder combines recall and response correctness, while GPT-5 evaluates VQA and Vision-Guided Interaction responses against ground truth.Reminder scoring requires triggering within ground-truth temporal intervals and producing accurate, context-aware guidance; proprietary models use frame-by-frame prompting without TPAD, while open-source models use TPAD and streaming models run in native streaming modes.Source: §5.2 · p. 7, §5.1 · p. 7
  • GPT-5 scores 28.8 on average, while Proactive Reminder and Vision-Guided Interaction performance remains below VQA.Models often fail to anticipate navigation-critical events or provide timely, context-aware guidance aligned with ground-truth intervals.Source: §5.2 · pp. 7–8, Table 2 · p. 8
  • Vision-Guided Interaction remains difficult because first-person videos can be low-quality or unusual in viewpoint, and models may misinterpret users’ needs.For example, a model asked to help locate an object may say “pick it up” instead of giving spatially specific guidance for a visually impaired user.Source: §5.2 · p. 8
  • Existing streaming models perform extremely poorly on Proactive Reminder despite their ability to generate proactive outputs.The paper attributes this to limited comprehension of complex scenarios and training data that is too narrow for diverse real-world guidance instructions, including Ego4D-Narration [10] and SoccerNet-Caption [17].Source: §5.2 · p. 8

6. Conclusion and Future Work

VIABench integrates online video understanding into evaluation of MLLMs for visual assistance, and its findings point to weak proactive capabilities relative to VQA. Future work should improve proactive response, training data, and models’ alignment with visually impaired users.

Conclusion

  • VIABench evaluates Proactive Reminder, VQA, and Vision-Guided Interaction, uses TPAD for efficient online evaluation, and finds current MLLMs’ proactive capabilities nascent relative to VQA.The experiments also reveal limitations in current proactive-response models.Source: §6 · p. 8

Future Work

  • Future work should improve MLLMs’ ability to anticipate navigation-critical events and provide timely, context-aware guidance in complex scenarios.The passage identifies improved model architectures as one avenue for more reliable proactive responses.Source: §6 · p. 8
  • Future work should use richer, more diverse video-text datasets covering varied instructions, complex interactions, and a wider range of real-world scenarios.Existing models are predominantly trained on dense video captions or automatically generated narrations, which offer limited exposure to real-world blind-assistance instructions and semantic complexity.Source: §6 · p. 8
  • Models optimized with blind-centered data could better align assistive guidance with users’ perspectives and intentions, improving its accuracy, safety, and utility.Current MLLMs may treat blind users as sighted and struggle with low-quality, blurry, or unconventional-viewpoint first-person videos; the broader aim is more capable, dependable assistance that supports autonomy and safety.Source: §6 · p. 9

Supplementary Material · 7. More Details of Evaluation

The evaluation uses task-specific inference protocols and defines temporal and semantic metrics for Proactive Reminder, VQA, and Vision-Guided Interaction. GPT-5 similarity scoring evaluates the quality of generated descriptions and answers.

7.1. Evaluation Settings

  • Offline Proactive Reminder models use TPAD, online models use default interfaces, and closed-source models use single-frame inference when required by API constraints.Source: §7.1 · p. 11
  • VQA uses video from the clip start to the question timestamp; Vision-Guided Interaction uses multi-turn VQA with prior ground-truth responses and timestamps in the dialogue history.Source: §7.1 · p. 11
  • TPAD and GPT evaluations sample video at 1 FPS, while other evaluations follow each model’s default frame rate and inference configuration.Source: §7.1 · p. 11

7.2. Primary Metric for Proactive Reminders

  • An event is detected when the model outputs a matching task prediction and associated prompt within the event’s annotated valid time interval.Source: §7.2 · p. 11
  • Proactive Detection Rate (PDR) measures the proportion of annotated events correctly detected, capturing task-relevant interventions made within the allowed temporal window.Source: §7.2 · p. 11

7.3. Description Accuracy with GPT-5

  • Mean Prompt Similarity (MPS) averages GPT-5 scores comparing generated prompts with ground-truth descriptions for matched events, penalizing mismatches in obstacle location, type, or behavioral instruction.Source: §7.3 · p. 11

7.4. Metric for Visual Question Answering and Vision-Guided Interaction

  • VQA and Vision-Guided Interaction Accuracy Score (VAS) is the mean GPT-5 semantic similarity score between model-generated answers and ground truth across test examples.Source: §7.4 · p. 11

8. Additional Results

VIABench’s additional results show that current models remain limited by proactive-event retrieval, real-time latency, and assistance that must adapt to a blind user’s intent and changing context. The analyses also find only marginal gains from more input frames and show that TPAD can reduce runtime without sacrificing detection accuracy.

8.1. Proactive Reminder

  • Offline Stage-1 recall stays below 45%, while LiveCC [5] reaches 75.8% through dense, often irrelevant outputs rather than task understanding.LLaVA-OneVision-1.5 [1] has the highest offline recall, but limited Stage-2 generation quality prevents this advantage from yielding strong overall performance.Source: §8.1 · p. 12
  • With a shared 32-frame window, InternVL3.5-8B [27] leads offline open-source Stage-2 generation, followed by InternVL3.5-4B and Kimi-VL-A3B [22].Generation remains challenging even when retrieval differences are controlled; Direction Deviation is weakest, while Exit/Entry Recognition performs better, underscoring difficulty with trajectory monitoring and directional reasoning.Source: §8.1 · p. 12

8.2. The impact of input frames

  • Increasing input frames yields only marginal VIABench improvements because visual assistance is situational and older observations quickly become outdated.The benchmark emphasizes proactive monitoring, intention-aware question understanding, and adaptive guidance rather than long-term temporal dependencies.Source: §8.2 · p. 12

8.3. Inference Latency

  • At 32 frames, Qwen2.5-VL-7B takes around 4.6 seconds per response and InternVL-3.5 around 15.5 seconds, while latency rises exponentially with frame count.VideoLLM-Online can reach 10 FPS, but its overall performance remains significantly lower, making it unsuitable for reliable assistance.Source: §8.3 · pp. 12–13, Figure 5 · p. 13

8.4. Qualitative Analysis

  • Closed-source GPT-5 and Gemini-2.5 Pro [6] more often reject mismatched queries and adapt guidance to a blind user, while open-source models hallucinate or assume the user can see.Examples include open-source models inventing escalator malfunctions and describing a washing-machine display as visible instead of providing usable non-visual guidance.Source: §8.4 · pp. 13–14, Figure 6 · p. 15

8.5. Efficiency Comparison on a Proactive Reminder Video Segment

  • TPAD processes a 53-second, six-event video in 76.85 seconds versus 344.64 seconds for conventional prompting, with identical detection accuracy.TPAD uses one forward pass with selected frames contextualized by preceding frames, while the baseline makes 53 frame-wise passes.Source: Figure 8 · p. 14, §8.5 · pp. 14–15

8.6. Example Responses of Online Models

  • Online models often respond too early, miss critical events, or narrate scene content unrelated to the user’s instructed reminder task.Their high recall mainly reflects frequent unsolicited outputs, not reliable interpretation of user intent or correct reminders.Source: §8.6 · p. 17, Figure 9 · p. 17

9. VIABench

VIABench frames MLLMs as real-time assistive agents for blind users, evaluating proactive reminders with timing rules intended to make alerts contextually appropriate and safe. Its Proactive Reminder suite covers fine-grained real-world navigation scenarios.

9.1. Overview

  • VIABench envisions MLLMs on smart glasses or smartphones to provide blind users with navigation alerts, situational visual answers, and multi-step interactive guidance.The goal is to support more independent and safe navigation in dynamic real-world environments.Source: §9.1 · p. 17

9.2. Timing of Model Alerts in the Proactive Reminder Task

  • Alerts should occur at event onset for immediate-alert tasks, but for other tasks only when the user is within a safe, actionable distance of the target or hazard.Examples include alerting at camera occlusion, walking-direction drift, or crosswalk departure, while traffic-light alerts should wait until the user is close enough to make a crossing decision.Source: §9.2 · p. 17
  • The safe alert range extends about 1.5–2 meters ahead and 50 cm to each side, while trajectory-intersecting hazards may require alerts even outside that zone.Figure 10 categorizes targets as requiring alerts, deferred until approach, or not requiring alerts; lateral obstacles may be deferred, but approaching hazards can trigger alerts.Source: §9.2 · pp. 17–18, Figure 10 · p. 18

9.3. Task List

  • Proactive Reminder evaluates online video understanding and timely, anticipatory feedback across fine-grained navigation scenarios commonly encountered by visually impaired people.The benchmark lists the tasks with abbreviations, full names, and concise descriptions in Table 6.Source: §9.3 · p. 18

10. Data Collection Details

VIABench’s data-collection process emphasizes authentic, rich first-person videos and precise annotations, using task-specific protocols and quality controls for real-time visual assistance.

10.1. General Guidelines

  • Annotators received manuals, real-data training videos, and onboarding sessions, while labels were restricted to visible evidence and decisions could not use future frames.These principles aimed to make annotations fine-grained, temporally precise, and aligned with online assistance constraints.Source: §10.1 · p. 18

10.2. Dataset Annotation Submission Guidelines

  • Each annotation records a task label, precise start and end times, and a natural-language reminder; obstacle alerts span from warning onset to the last moment evasive action remains feasible.Active-avoidance reminders describe the obstacle and its coarse direction or position, then give concrete avoidance guidance. Annotators were instructed to label every qualifying event.Source: §10.2 · pp. 18–19

10.3. Common Annotation Issues for Annotators

  • QA identified errors from relying on audio, subtitles, or prior knowledge and from using future actions to justify present prompts; guidance should respond to current visual hazards, not presumed destinations.For example, a turn prompt is warranted when the current scene leaves only one safe path, not merely because the user later turns.Source: §10.3 · p. 20

11. Limitations

VIABench’s task coverage and Interaction data collection leave limitations in its ability to represent real-world assistance. The authors plan to broaden coverage and collect real-world interactive footage with blind communities.

Limitations

  • Because real-world visual assistance is open-ended, strong VIABench performance does not guarantee readiness, and misjudgments or hallucinations may pose serious safety risks.Rare events, long-tail edge cases, and situational challenges cannot be fully enumerated in unconstrained, dynamic environments.Source: §11 · p. 20
  • Vision-Guided Interaction recordings use role-played simulations by sighted people, creating a domain gap that limits evaluation of generalization to real-world interactive contexts.The videos differ from the real-world blind-user recordings used for Proactive Reminder and VQA.Source: §11 · p. 20

Future Work

  • Future iterations plan to broaden task coverage and ethically collect real-world interactive footage in collaboration with blind communities.The authors intend this work to help keep VIABench comprehensive and representative.Source: §11 · p. 20

12. Prompt Templates

The section documents prompts for VIABench’s inference tasks and a GPT-5 judging approach that scores free-form assistance responses using task-specific criteria.

12.1. Inference Prompts

12.2. Evaluation Prompts

  • Because VIA requires accurate, informative, concise free-form feedback, the authors reject MCQ evaluation and use GPT-5 to score responses against ground truth, assigning Si ∈[0, 1] [8] [36].Source: §12.2 · p. 20
  • Generic similarity-based judging can overlook dangerous factual conflicts, such as misidentifying an escalator’s operating status, yet still award a non-zero score [8] [36].Source: §12.2 · p. 20
  • Task-specific, criteria-guided scoring asks GPT-5 to compare responses with ground truth while considering obstacle types, spatial locations, object states, and safety-critical cues.Source: §12.2 · p. 20
  • Scoring prompt templates cover the three main tasks, with detailed designs referenced in Figures 13 and 14.Source: Figure 13 · p. 22, §12.2 · p. 23

13. Task Examples

VIABench’s examples show Proactive Reminder directing safe navigation, VQA answering visually grounded questions, and Vision-Guided Interaction helping users manipulate the camera or target objects. The section illustrates these tasks with first-person video intervals and ground-truth instructions.

Proactive Reminder

  • Intersection Recognition requires naming each junction’s relative direction and adding only verified details; an example warns that a T-junction is approaching.The instruction prohibits invented information.Source: §13 · p. 27
  • Crossing examples cover signal visibility, crosswalk alignment, and reaching the opposite sidewalk, with concise prompts such as moving right from a crosswalk’s left edge.If no pedestrian signal is visible, the task calls for reporting that absence rather than guessing its state.Source: §13 · pp. 27–28
  • Other navigation prompts monitor route deviation or blockage, steps and slopes, road surfaces, tactile paths, and branching routes, giving timely directions or hazard warnings.Examples include warning of a downhill ahead and directing the user around a road depression.Source: §13 · pp. 29, 30, 31, 33
  • Further prompts identify entrances, camera obstruction or unusable views, public facilities, signs, and escalators, reporting relevant location, condition, or actionable guidance.For example, an escalator prompt specifies position, direction, and operating status, and warns when its end approaches.Source: §13 · pp. 33–36

Visual Question Answering

  • VQA answers the blind person’s spoken question concisely from visible video evidence and says “information not visible” when the answer never appears.Because real blind-person questions are scarce, the examples use simulated safety- or navigation-critical queries; one asks whether an escalator is directly ahead.Source: §13 · p. 37

Vision-Guided Interaction

  • Vision-Guided Interaction guides camera positioning or hand movement toward a target, then verifies completion and states the exact result.Examples direct the camera slightly right to find elevator buttons and the hand down three slots to reach a door-close button; completion confirms the fifth-floor button was found.Source: §13 · p. 38

Related papers

Loading 2607.14660v1…