PaperlayerView exact revision on arXiv

Source-linked AI summary

VIABench: A Comprehensive Video Benchmark Collected from Blind Individuals for Visual Impairment Assistance

Yunfeng Liu, Yuandong Yang, Jiarui Han, Zhenpeng Huang, Yuqing Tang, Xiangyu Zeng, Gangshan Wu, Limin Wang

arXiv:2607.14660v1cs.CV

TL;DR

Visually impaired people need visual assistance that captures the complexity and dynamics of everyday situations, but existing benchmarks do not fully reflect real-world use. VIABench evaluates MLLMs across proactive reminders, VQA, and vision-guided interaction, finding that current models remain limited—especially in proactive assistance.

  • Problem

    Existing assistive-vision benchmarks do not fully capture the complex, dynamic real-world experiences of visually impaired users.

  • Method

    VIABench uses real-world egocentric videos to evaluate MLLMs on Proactive Reminder, VQA, and Vision-Guided Interaction, with TPAD supporting proactive-alert evaluation.

  • Results

    Current MLLMs show nascent proactive capabilities and remain limited in proactive-response tasks, particularly compared with their VQA performance.

  • Takeaways & Limitations

    Future progress should advance proactive-response mechanisms, expand training data, and tailor models to assistive scenarios.

  • Takeaways & Limitations

    Strong VIABench performance does not guarantee real-world readiness because rare, long-tail situations remain incompletely represented and errors may create serious safety risks.

Abstract

from arXiv · show

Visually impaired individuals (VIIs) encounter significant daily challenges due to limited access to visual information. Although Multimodal Large Language Models (MLLMs) have achieved impressive results on general vision and language tasks, their practical utility in real-world blind assistance still remains largely underexplored. To fill this gap, we introduce VIABench, a comprehensive video benchmark specifically designed to evaluate MLLMs in Visually Impaired Assistance scenarios using first-person videos recorded or shared by VIIs themselves. VIABench defines three core tasks, each targeting a distinct requirement in visual assistance. Proactive Reminder: Assesses the model's ability to interpret ongoing video content while proactively anticipating and verbally describing upcoming navigation-critical events; Visual Question Answering (VQA): Evaluates the model's capacity to answer user-posed questions about the environment or objects within the video; Vision-Guided Interaction: Tests context-aware reasoning to accomplish intentional interactions between user and environment. To ensure a robust and fair evaluation, we propose a rigorous benchmarking pipeline that supports both online (real-time) and offline settings. Our experiments demonstrate that current MLLMs still struggle to deliver comprehensive support for VIIs, especially in the Proactive Reminder task, which demands accurate anticipation and real-time responsiveness. We hope VIABench will drive future research toward developing customized MLLMs for real-world assistance, ultimately improving navigation and interaction experiences for visually impaired individuals. Code and data will be released at https://github.com/MCG-NJU/VIABench.

1. Introduction · 2. Related Work

VIABench addresses the limited real-world applicability of existing assistive vision methods by benchmarking MLLMs on long-form egocentric videos from visually impaired individuals across three assistance tasks. The paper also introduces TPAD to evaluate proactive alert generation in continuous video streams, extending assessment beyond general-purpose video understanding benchmarks.

  • 1. Introduction: Visually impaired individuals face major obstacles in everyday tasks because canes and guide dogs provide only partial support and limited semantic or contextual information.Examples include crossing streets, finding store entrances, and reading menus.
  • 1. Introduction: Traditional assistive-navigation algorithms are constrained by single-task designs and poor generalization, despite MLLMs’ strong performance on existing video-understanding benchmarks.The cited MLLM examples include Gemini 1.5, Qwen2-VL, MiniCPM-V, and VideoChat.
  • 1. Introduction: Prior assistive datasets do not adequately capture the complexity and dynamic nature of visually impaired users’ everyday experiences, motivating a more realistic benchmark.The passage specifically identifies the WAD dataset as comprising 3-second video clips.
  • 1. Introduction: VIABench is a time-aligned, egocentric video benchmark collected from visually impaired individuals and designed for real-world assistive-vision evaluation.Its videos are large-scale, real-world, long-form, and temporally richer than prior visual-assistance video benchmarks.
  • 1. Introduction: The benchmark unifies three assistance tasks: Proactive Reminder, Visual Question Answering, and Vision-Guided Interaction.These tasks cover autonomous risk or cue notification, user-initiated questions, and iterative guidance for completing a target task.
  • 1. Introduction: The evaluation focuses on whether MLLMs can identify and communicate critical events proactively while remaining silent when no relevant information is present.This challenge concerns real-time video streams and the balance between action and inaction.
  • 1. Introduction: TPAD is introduced as a two-stage framework for efficiently and reliably evaluating proactive alert generation in long, continuous videos.The paper presents TPAD as addressing the core challenge of Reminder-style tasks.
  • 2. Related Work: General-purpose MLLMs demonstrate strong capabilities in temporal reasoning, event localization, and open-ended video question answering, but existing benchmarks primarily assess broad video understanding.Representative models include Gemini 1.5, Qwen2-VL, MiniCPM-V, and VideoChat.

3. VIABench

VIABench evaluates visually impaired assistance through three complementary tasks derived from first-person videos and practical scenarios. It combines long-form, authentic video data with navigation-centered annotations and realistic visual degradation.

  • Core Tasks: VIABench defines Proactive Reminder, Visual Question Answering, and Vision-Guided Interaction as complementary tasks requiring accurate, concise, and informative responses.Together, these tasks form a holistic evaluation framework for practical assistance scenarios encountered in first-person videos recorded by visually impaired individuals.
  • Proactive Reminder: Proactive Reminder requires models to recognize navigation-critical events and anticipate them before occurrence, providing timely, actionable support across 21 fine-grained sub-tasks.The task targets online video understanding in dynamic, real-world scenarios.
  • Vision-Guided Interaction: Vision-Guided Interaction is a multi-turn, closed-loop task requiring step-by-step instructions that adapt continuously to user actions and changing visual input.The guidance continues until the user achieves a specific physical goal.
  • Dataset Scale: 761 videos and 14,526 manually curated annotations provide 46.9 hours of footage, with 222-second average and 1959-second maximum durations.VIABench is described as the most temporally rich dataset in this domain, emphasizing long-form video.
  • Data Characteristics: 530 real VII videos and 231 informed simulated videos ground VIABench in authentic, navigation-centered scenarios while covering Vision-Guided Interaction.Annotations emphasize directional cues and obstacles, and 94% of the data comes from visually impaired users whose recordings often contain occlusion, abnormal exposure, and camera inversion or rotation.

4. Method

VIABench organizes evaluation around three assistive scenarios—navigation, blind questioning, and interaction—mapped to Proactive Reminder, VQA, and Vision-Guided Interaction tasks. It also introduces TPAD, a fine-tuning-free method that converts offline MLLMs into frame-wise proactive detectors using a single context-aware forward pass.

  • Assistive Scenarios and Tasks: VIABench evaluates navigation, blind questioning, and interaction through Proactive Reminder, Visual Question Answering, and Vision-Guided Interaction, respectively.Together, the tasks assess proactive perception, reactive reasoning, and interactive communication in real-world assistive scenarios.
  • Assistive Scenarios and Tasks: The Proactive Reminder task requires continuously monitoring streaming video and issuing timely warnings about navigation hazards without explicit user requests.The task targets scene understanding and deciding when an alert should be issued for hazards such as steps, obstacles, or sidewalk boundaries.
  • Assistive Scenarios and Tasks: The online VQA task answers user questions using only visual information available before each question, measuring accurate real-time responses to surroundings.Its design follows the online setting of OVOBench.
  • Assistive Scenarios and Tasks: Vision-Guided Interaction simulates multi-turn assistance in which models update coherent, vision-grounded instructions as the scene and accumulated dialogue evolve.Each annotated interaction segment forms a conversational turn requiring the next instructive response.
  • Token-Level Prompt Activation Decoding: TPAD transforms a pretrained offline MLLM into a frame-wise proactive detector without fine-tuning by scoring forced-choice alert tokens from frame hidden states.All prompt and video tokens are processed as one concatenated sequence, and a single forward pass produces frame-wise alert probabilities while conditioning predictions on preceding visual context.

5. Experiments

Experiments evaluate open-source, proprietary, and online streaming MLLMs across VIABench’s Proactive Reminder, VQA, and Vision-Guided Interaction tasks. Results show early-stage real-world assistance, with GPT-5 achieving only 28.8 on average and proactive response remaining especially difficult.

  • Evaluation Setup: Three MLLM categories are evaluated across all VIABench tasks, with Proactive Reminder requiring proactive outputs from the visual stream.The categories are open-source foundation, closed-source proprietary, and online streaming models.
  • Evaluation Setup: Open-source models use TPAD for Proactive Reminder, while smaller variants probe settings relevant to low-resource assistive devices.Evaluated smaller variants include InternVL3.5-1B, InternVL3.5-4B, and Qwen2.5VL-3B.
  • Evaluation Setup: Proprietary models use conventional frame-by-frame prompting without TPAD, whereas online streaming models are tested directly in their native streaming modes.The proprietary baselines include GPT-5, GPT-4o, and Gemini-2.5 Pro; online models include LiveCC, VideoLLM-Online, and StreamingVLM.
  • Key Findings: 28.8 is GPT-5’s average score, indicating only nascent competence in real-world visual assistance.Proactive Reminder and Vision-Guided Interaction perform notably below established VQA benchmarks.
  • Key Findings: Current models struggle to anticipate navigation-critical events and provide accurate, context-aware, multi-turn, vision-grounded guidance.The reported difficulties are linked to navigation-critical timing and the challenges of first-person videos captured by visually impaired users, which can be low-resolution or blurry.
  • Limitations: Existing proactive-response models perform extremely poorly because streaming designs have limited complex-scenario comprehension and their training data is limited.The passage identifies these as two main reasons for poor Proactive Reminder performance.

6. Conclusion and Future Work

The paper introduces VIABench for evaluating MLLMs as assistive tools for visually impaired individuals through online video understanding, with Proactive Reminder, VQA, and Vision-Guided Interaction tasks. It also highlights the need for blind-centered models and future work on proactive response, richer data, and assistive-scenario specialization.

  • Contributions: VIABench integrates online video understanding into MLLM evaluation for visually impaired individuals through Proactive Reminder, VQA, and Vision-Guided Interaction tasks.The benchmark is designed to assess MLLMs as assistive tools for visually impaired individuals.
  • Contributions: Token-Level Prompt Activation Decoding (TPAD) is proposed to facilitate robust online evaluation, particularly for proactive alerting.TPAD is described as an efficient method for online evaluation.
  • Limitations and Future Work: Models specifically optimized for visually impaired users are needed because current MLLMs often overlook blind users’ perspectives and struggle with low-quality, blurry, or unconventional first-person videos.Blind-centered training and fine-tuning could improve alignment with user needs, accuracy, safety, and overall assistance utility.
  • Future Work: Future directions include advancing proactive-response mechanisms, expanding and enriching training data, and tailoring models for assistive scenarios.These directions aim to support more capable and dependable MLLM-based assistive technologies and improve autonomy and safety for visually impaired individuals.

VIABench: A Comprehensive Video Benchmark Collected from Blind Individuals

The section presents VIABench as a comprehensive video benchmark for visual impairment assistance, collected from blind individuals.

  • The benchmark addresses visual impairment assistance.

7. More Details of Evaluation

VIABench evaluates its three tasks with task-specific video-processing protocols and semantic metrics. Proactive Reminder uses temporal event detection and prompt-quality scoring, while VQA and Vision-Guided Interaction use GPT-5-based answer similarity.

  • Evaluation Protocols: Proactive Reminder evaluates offline models with TPAD, online models through default inference interfaces, and closed-source models with API-constrained single-frame inference.VQA uses video from clip start to question timestamp with default sampling and inference, while Vision-Guided Interaction uses multi-turn VQA-style evaluation.
  • Proactive Reminder Metrics: An event is correctly detected when the model outputs a matching task prediction and associated prompt within its ground-truth interval [ts, te].The interval defines the valid time window for issuing a task-specific prompt.
  • Proactive Reminder Metrics: Proactive Detection Rate measures the proportion of annotated events correctly detected through task-relevant intervention within the allowed temporal window.It is defined as the size of correctly detected events divided by the set of all annotated events.
  • Proactive Reminder Metrics: Mean Prompt Similarity averages GPT-5 scores comparing generated and ground-truth prompts, with mismatches in obstacle location, type, or behavioral instruction penalized.Each similarity score lies in [0, 1] and is computed for matched events.
  • VQA and Vision-Guided Interaction Metrics: VQA and Vision-Guided Interaction use GPT-5 to score semantic similarity between generated answers and ground truth, averaging scores into the VQA/VGI Accuracy Score.The similarity scores range from 0 to 1 and are averaged across the respective test examples.

8. Additional Results

Additional results show that current models struggle with reliable, real-time visual assistance: proactive reminders remain difficult, frame increases offer marginal gains, and inference latency limits deployment. Models also exhibit failures in directional reasoning, hallucination robustness, and adapting guidance to blind users’ non-visual needs.

  • Stage 2 Ablation: Using identical 32-frame windows for up to 100 examples isolates Stage 2 generation quality by eliminating Stage 1 retrieval variability.Every model receives the same visual context immediately preceding the ground-truth trigger time.
  • Proactive Reminder: LiveCC achieves recall as high as 75.8%, but dense outputs resemble continuous captioning and produce low-quality, often irrelevant reminders.Frequent narration does not reliably identify correct reminder-triggering moments.
  • Task Analysis: Direction Deviation is the weakest sub-task, whereas Exit/Entry Recognition performs better, revealing persistent difficulty with precise directional reasoning and dynamic pathfollowing.Direction Deviation requires continuously tracking walking direction and issuing timely alerts about dangerous trajectory shifts.
  • Frame Scaling: Increasing input frames yields only marginal VIABench improvements because visual assistance is real-time and rapidly changing, limiting the usefulness of stale temporal context.This differs from conventional video benchmarks where larger temporal context often provides substantial gains.
  • Inference Efficiency: With 32 visual frames, Qwen2.5-VL-7B requires around 4.6 seconds per response, while InternVL-3.5 takes around 15.5 seconds, making always-on assistance impractical.Online models such as VideoLLM-Online can reach 10 FPS, but their significantly lower overall performance undermines reliable assistance.
  • Robustness and User Awareness: Open-source models hallucinate more under mismatched queries and often overlook blind users’ non-visual needs, unlike closed-source models that better ground responses and adapt guidance.GPT-5 and Gemini-2.5 Pro correctly read or help expose a washing-machine display, while InternVL3.5-8B and Qwen2.5-VL-7B assume the user can see it.

9. VIABench

VIABench aims to use MLLMs as real-time visual assistants that improve blind individuals’ independent and safe navigation while supporting situational questions and multi-step interactions. Its Proactive Reminder component evaluates online video understanding and anticipatory feedback through fine-grained real-world navigation tasks.

  • Motivation: VIABench envisions MLLMs deployed in smart glasses or smartphones as Jarvis-like assistants for safer, more independent real-world navigation.The assistant can proactively alert users to hazardous obstacles, answer situational visual questions, and engage in multi-step interactions.
  • Alert Timing: Alerts should follow contextually appropriate timing based on the blind user’s spatial relation to the target, avoiding irrelevant early or unsafe late warnings.The benchmark distinguishes immediate alerts at event onset from distance-aware alerts issued within a safe, actionable range.
  • Alert Timing: Immediate alerts are required for camera occlusion, walking-direction drift, and unintended crosswalk exits as soon as these events begin.These scenarios are explicitly marked for event-onset notification.
  • Safe Alert Range: The defined safe alert range covers approximately 1.5 to 2 meters ahead and around 50 cm on each side of the user.The front range corresponds to 4–6 average walking steps, while the lateral range is roughly one person-width per side.
  • Alert Timing: Targets outside the safe zone may still require alerts when their trajectories intersect the user’s future path and pose a risk.This edge-case rule complements distance-based alerting by accounting for predicted collision risk.
  • Proactive Reminder: Proactive Reminder is VIABench’s central component, evaluating online video understanding and timely anticipatory feedback across fine-grained real-world navigation scenarios.Each sub-task targets a distinct navigational situation commonly encountered by visually impaired individuals.

10. Data Collection Details

VIABench uses a carefully controlled data-collection and annotation pipeline centered on authentic first-person videos, task-specific protocols, and strict quality assurance. Its guidelines enforce visual-only, forward-only online reasoning and precise, comprehensive event labeling for real-time assistance.

  • Pipeline Design: VIABench’s data pipeline prioritizes authenticity, richness, and precision through principled sourcing, annotation methodology, and quality-control strategies.The section specifically emphasizes a carefully designed pipeline for real-world blind assistance evaluation.
  • Annotator Training: Annotators received manuals, real-data training videos, and formal onboarding sessions covering task definitions, boundary conditions, examples, and common ambiguities.These measures were intended to standardize task understanding and maintain consistent annotation quality.
  • Annotation Principles: Annotations followed visual-only and online temporal constraints, excluding audio and future frames so labels reflected only information available to a vision-based model.These principles produced fine-grained, temporally precise annotations aligned with real-time assistive-system constraints.
  • Submission Protocol: Each task instance required precise start and end timestamps, task-specific reminder content, and coverage of every qualifying event without omissions.For obstacle alerts, the interval spans from the earliest warning moment until evasive action is no longer feasible or the obstacle disappears; active-avoidance reminders include obstacle description and walking instruction.
  • Quality Assurance: Audits identified recurring errors involving nonvisual cues and future-event reasoning, reinforcing that labels must rely on current visual context rather than later user actions.The audit distinguishes real-time forward-only assistance from offline video review and destination-aware navigation.

11. Limitations

VIABench’s task coverage cannot fully represent the open-ended, dynamic nature of real-world visual assistance, and its simulated interaction videos introduce a domain gap from actual blind-user behavior. Future iterations will broaden coverage and collect interactive footage with blind communities.

  • Task coverage limitations: VIABench’s three task families cannot enumerate rare events, long-tail edge cases, and highly situational challenges in unconstrained assistive navigation.Strong performance on VIABench therefore does not guarantee real-world assistance performance.
  • Data limitations: Vision-Guided Interaction recordings use role-played simulation by sighted individuals, unlike the Proactive Reminder and VQA videos.Although designed to emulate realistic behavior patterns, the recordings introduce a domain gap relative to actual blind-user behaviors.
  • Data limitations: The simulated Interaction data currently limit evaluation of model generalization across real-world interactive contexts.This limitation specifically affects the Interaction component’s ability to assess behavior beyond the role-played setting.
  • Future directions: Future iterations will broaden task coverage and collaborate with blind communities to ethically collect real-world interactive footage.These efforts aim to keep VIABench comprehensive and representative for advancing assistive multimodal intelligence.

12. Prompt Templates

The section specifies prompt templates for proactive reminders, navigation queries, VQA, and vision-guided interaction, standardizing response formats across evaluation settings. It also introduces GPT-5, task-specific, criteria-guided judging to evaluate free-form assistance responses while accounting for safety-critical errors.

  • 12.1. Inference Prompts: Prompt templates cover proactive reminder sub-tasks, navigation-related queries, and VQA/VGI interactions.They define expected response formats for consistent evaluation across settings.
  • 12.1. Inference Prompts: The inference prompts standardize responses across different online and offline evaluation settings.The templates are designed to ensure consistency in how models respond.
  • 12.2. Evaluation Prompts: Because visual impairment assistance is free-form generation, MCQ evaluation is unsuitable and GPT-5 instead scores responses against ground-truth descriptions on Si ∈[0, 1].The LLM-as-a-judge paradigm evaluates whether responses are accurate, informative, and concise.
  • 12.2. Evaluation Prompts: Generic similarity-based judging can produce unreliable and overly optimistic scores when responses contain factual conflicts affecting user safety.Text similarity may miss errors such as misidentifying an escalator’s operating status.
  • 12.2. Evaluation Prompts: Task-specific, criteria-guided scoring asks GPT-5 to assess obstacle types, spatial locations, object states, and safety-critical cues.The criteria are tailored to each sub-task and compare model responses with the ground truth.
  • 12.2. Evaluation Prompts: Figure 11 details prompt templates for all Proactive Reminder sub-tasks.The figure documents the prompting design for the proactive reminder evaluations.
  • 12.2. Evaluation Prompts: Figure 12 presents prompt templates for VQA and vision-grounded interaction tasks.These templates address question answering and intentional environment interaction.

13. Task Examples

This section presents VIABench task definitions and annotated examples spanning intersection and signal recognition, path and crossing guidance, surface and route monitoring, camera and facility detection, and actionable text or escalator assistance.

  • Intersection Recognition: Intersection Recognition requires detecting every visible junction, reporting its relative direction, and optionally adding one verified traffic detail without inventing information.The example ground truth describes approaching a T-junction and advises caution.
  • Traffic and Crossing Guidance: Pedestrian Traffic Light and crosswalk tasks require identifying signal availability or phase and keeping the user aligned within the crosswalk.Examples include reporting no pedestrian traffic lights and correcting a user positioned at the crosswalk’s left edge.
  • Navigation Guidance: Crossing, trajectory, path, step, stair, and incline tasks provide timely instructions for reaching the opposite sidewalk, correcting drift, avoiding blocked routes, and handling elevation changes.Examples include preparing to cross, warning about veering toward a landscaped area, turning right along a tactile path, and recognizing upward or downward steps and slopes.
  • Surface and Path Monitoring: Road Surface Condition and Blind Path Recognition tasks monitor hazards, sidewalk position, tactile paving, and route branches to support safe alignment and route selection.Examples identify a road depression, recommend the sidewalk, locate tactile paving, and report a road branching on the right.
  • Scene and Camera Monitoring: Exit/Entry Recognition and Camera Content Anomaly tasks detect usable passages and navigation-compromising camera conditions, then report the issue and corrective action.Examples describe an open exit on the left, a blocked left camera view, and excessive camera exposure requiring adjustment.
  • Facilities and Information: Service Facility Detection, text reading, and escalator tasks identify public objects, read actionable signs with their positions, and characterize escalator location, direction, and operating state.Examples include chairs ahead on the right and instructions to turn right along tactile paving to board a vehicle.
Loading 2607.14660v1…