PaperlayerView exact revision on arXiv

Source-linked AI summary

VISTA: A Multimodal Egocentric Benchmark for Goal-Oriented Perception and Assistance to Blind and Low-Vision Users

Noé Mosca, Mohamed Rayan Barhdadi, Muhammad Hamza, Joshua Yeh, Suryodaya Bikram Shahi, Zhiyan Li, Selim-Antoine Lali, Weiyue Li, Hasan Kurban, Erchin Serpedin, Gang Luo, Yue Zhang, Minghan Li, Mengyu Wang

arXiv:2610.22822v1cs.CV

TL;DR

Existing benchmarks offer limited coverage of multimodal wearable inputs and goal-conditioned assistance for blind and low-vision users. VISTA introduces a sensor-rich egocentric benchmark and evaluates vision-language models alongside a multimodal baseline; current models still struggle to turn scene understanding into grounded guidance, while the baseline improves guidance scores across evaluated backbones.

  • Problem

    Existing benchmarks provide limited coverage of wearable sensor modalities and goal-conditioned assistance for everyday tasks faced by blind and low-vision users.

    Source: §1 · p. 1

  • Method

    VISTA collects egocentric recordings with synchronized sensor streams and three annotation types across ten assistive task categories, and evaluates zero-shot VLMs and a six-modality baseline.

    Source: §1 · p. 2, §3.1 · p. 4

  • Results

    Current VLMs frequently fail to translate visual observations into grounded, actionable guidance, while a sensor-input baseline with task adaptation improves guidance scores across all evaluated backbones.

    Source: §1 · p. 2, §7 · p. 10

  • Takeaways & Limitations

    The findings motivate models that connect perception of users’ surroundings with the actions needed to achieve their goals.

    Source: §7 · p. 10

    VISTA was collected in one geographic region with sighted participants simulating assistive scenarios, and the offline model does not yet support real-time assistance.

    Source: §6 · p. 9

Abstract

from arXiv · show

Blindness and low vision limit many people's ability to complete daily tasks, motivating AI assistants that provide useful guidance. Yet, existing benchmarks do not adequately test whether vision models can provide effective everyday assistance. In particular, i) existing benchmarks primarily rely on visual and textual inputs, with limited support for other modalities that are increasingly available in egocentric wearable devices such as gaze, inertial measurements, spatial trajectories, and others; and ii) existing evaluations emphasize scene understanding and question answering, with limited coverage of goal-conditioned guidance for blind and low-vision users across everyday tasks. To bridge this gap, we introduce VISTA (Visually Impaired Scene and Task Annotations), a multimodal egocentric benchmark for goal-oriented assistance for blind and low-vision users. VISTA contains $\sim$1,000 egocentric recordings captured with Project Aria glasses; each recording provides synchronized sensor streams, six of which serve as model inputs, and three different types of annotation, covering ten task categories (TC), from object localization to hazard detection. We evaluate 14 representative vision-language models zero-shot on VISTA and design a lightweight multimodal model baseline that integrates all six modalities. Experimental results show that, although existing models can often describe egocentric scenes, they remain limited in generating goal-directed assistive guidance. VISTA thus provides a benchmark for evaluating multimodal models that could improve everyday accessibility.

1 INTRODUCTION

VISTA targets the gap between describing what a scene contains and providing goal-directed, user-centered assistance. It combines synchronized egocentric sensor recordings and annotations with zero-shot model evaluations and a multimodal baseline.

1 INTRODUCTION

  • Existing benchmarks emphasize visual and textual inputs, while assistive evaluation has limited coverage of wearable sensing and guidance conditioned on a user's goal.Useful assistance also requires interpreting the egocentric scene from the user's perspective, identifying what matters for the current goal, and generating concise, actionable guidance.Source: §1 · pp. 1–2
  • VISTA provides about 1,000 Project Aria recordings across ten assistive task categories, with synchronized sensor streams and annotations for scene descriptions, visual questions, and goal-conditioned instructions.The instructions require reasoning about relevant objects, spatial relations, hazards, and social cues to generate actionable guidance.Source: §1 · p. 2
  • VISTA evaluates vision-language models zero-shot and introduces a lightweight baseline that combines RGB video with five sensor streams, for six input modalities total.The recorded streams include RGB, stereo SLAM and eye-tracking cameras, audio, IMU, barometer, and magnetometer.Source: §1 · p. 2, Figure 1 · p. 2
  • Current models often describe scenes but remain limited on task-oriented guidance; the multimodal baseline with task adaptation improves answer quality on every evaluated backbone, especially on guidance.The authors identify frequent errors on assistive tasks, particularly goal-conditioned guidance.Source: §1 · p. 2

2 RELATED WORK

Prior work covers egocentric understanding, visual assistance, and multimodal modeling, but these areas largely leave wearable-sensor integration and goal-conditioned assistance for blind and low-vision users unaddressed. VISTA combines sensor streams with task-specific annotations and evaluates open-ended responses for grounding, correctness, task completion, and accessible communication.

Egocentric video understanding

  • General egocentric datasets and QA benchmarks support activity recognition, long-video reasoning, memory, instruction help, and text reading, but mostly use RGB and target general users.Project Aria also records audio, IMU, gaze, and trajectory data, which VISTA combines with annotations for blind and low-vision users.Source: §2 · p. 2

Visual assistance for blind and low-vision users

  • Assistive datasets span single-image questions, egocentric video, walking, and navigation, while VISTA adds synchronized wearable sensors and three annotation types across ten task categories.The annotations include scene descriptions, question–answer pairs, and goal-conditioned instructions.Source: §2 · p. 3

Multimodal models and sensor fusion

  • Open VLMs generally accept visual and text inputs, while prior sensor-fusion tools include attention bottlenecks, a pretrained IMU encoder, and parameter-efficient adaptation.The paper combines these tools in a sensor-augmented baseline for VISTA.Source: §2 · p. 3

Evaluating open-ended assistance

  • Reference-based metrics measure overlap with answers but do not assess scene match, task completion, or usefulness; VISTA uses a vision-language judge to score grounding, correctness, completion, and accessible communication.The paper notes that LLM-as-a-judge methods score open-ended answers more flexibly and agree well with human ratings.Source: §2 · p. 3

3 VISTA BENCHMARK

VISTA is an egocentric benchmark built around assistive tasks for blind and low-vision users, combining synchronized wearable recordings with scene descriptions, question–answer pairs, and goal-directed instructions. Its annotations and task coverage support evaluation of scene understanding as well as actionable, safety-aware guidance.

3.1 OVERVIEW

  • VISTA contains recordings from first-generation Project Aria glasses across ten approximately balanced task categories and varied indoor and outdoor settings.Clips last roughly 10 seconds to one minute; the settings include kitchens, corridors, streets, transit-like locations, and parks.Source: §3.1 · p. 4

3.2 DATA COLLECTION SETUP

  • The glasses time-align multiple sensor streams; the multimodal baseline uses RGB plus five derived streams, while audio and raw SLAM and eye-tracking images are not model inputs.The five additional streams are IMU, magnetometer/barometer, eye gaze, hand tracking, and 6-DoF trajectory.Source: §3.2 · p. 4

3.3 DATA ANNOTATION PIPELINE

  • Each clip has three annotation types: scene descriptions, user-oriented question–answer pairs, and goal-instruction pairs stating an assistive goal and concrete steps.The goal-instruction layer distinguishes VISTA from conventional captioning or VQA and enables evaluation of guidance quality, specificity, and safety.Source: §3.3 · pp. 4–5
  • Eight trained annotators wrote clip descriptions, QA pairs, and goal-instruction pairs, followed by BLV-community review and a full revision pass.For 30 multiply labeled clips, BERTScore was 0.897 for QA pairs and 0.877 for scene descriptions; overlapping annotations were retained as multiple references.Source: §3.3 · p. 5
  • The release has 997 clips and approximately balanced coverage across ten tasks, with 97–102 clips per category and multiple annotation types.It contains 1,063 scene descriptions, 1,191 QA pairs, and 2,067 goal-instruction pairs.Source: §3.3 · p. 5

3.4 DATASET COMPARISON

  • VISTA combines egocentric video, open-ended QA, scene descriptions, and goal-instruction pairs in assistive scenarios designed around blind and low-vision users.The scenarios were recorded by sighted participants simulating those situations.Source: §3.4 · p. 6

4 MULTIMODAL BASELINE

The baseline augments a pretrained VLM with five synchronized sensor streams alongside RGB video. Temporal adapters, question-conditioned reliability gates, and bottleneck exchange fuse the streams into prompt tokens; training is limited to the LoRA adapters and sensor-fusion network.

4 MULTIMODAL BASELINE

  • The baseline combines RGB video with five sensor streams—IMU, magnetometer/barometer, eye gaze, hand tracking, and 6-DoF trajectory—to form six modalities.The streams complement visual observations with cues about attention, interaction, device and user movement, heading, and elevation.Source: §4 · pp. 6–7
  • Temporal adapters, question-conditioned reliability gates, and bottleneck exchange fuse sensor streams into prompt tokens alongside RGB-derived visual features.The architecture uses eight RGB frames; Figure 5 specifies 8 sensor tokens at stage 1 or 28 at stage 2.Source: §4 · p. 6, Figure 5 · p. 7
  • Only the language LoRA adapters and sensor-fusion network are trained; the VLM vision encoder remains frozen.Missing sensor observations are represented with explicit masks rather than treated as valid measurements.Source: §4 · p. 7, Figure 5 · p. 7

5 EXPERIMENTS

The experiments compare zero-shot VLMs on VISTA and test whether multimodal adaptation improves their answers. Adaptation raises overall scores across backbones, while goal-conditioned guidance remains a central challenge and some task-level trade-offs persist.

5.1 EVALUATION SETTING

  • The shared evaluation uses eight sampled frames and a fixed judge rubric scoring evidence grounding, correctness, task completion, and accessible communication.Potentially harmful instructions are flagged, capping the final score at 1.95.Source: §5.1 · p. 7, Table 2 · p. 8

5.2 ZERO-SHOT RESULTS

  • Guidance is harder than scene description and question answering; InternVL3-14B scores 3.81 on scene description but 2.91 on guidance.Across eight models evaluated on both benchmarks, scores span 3.11–3.58 on EgoBlind and 1.76–3.31 on VISTA.Source: §5.2 · p. 8
  • Even the highest-scoring zero-shot model, Gemini 3.1 Flash-Lite (3.75), has critical errors on 7.4% of items and 11.6% of navigation items.Critical errors are judge-flagged answers that could cause harm if followed.Source: §5.2 · p. 8

5.3 MULTIMODAL ENHANCEMENT

  • Every backbone improves overall after adaptation; InternVL3-2B rises from 2.75 to 3.54 (+28.7%).Table 3 presents the adapted results against pretrained backbones.Source: Table 3 · p. 9, §5.3 · p. 9
  • After adaptation, Qwen3-VL-4B scores 3.95, above the best zero-shot open-source model (3.33) and proprietary model (3.75).For the four models with the largest gains, all rubric components and all three response forms improve.Source: §5.3 · p. 9
  • Adapted Gemma 3-12B gains 5.5% overall, but its scene-description and question-answering scores fall while its guidance score rises.Its guidance score increases from 2.48 to 3.32, while scene description drops from 3.41 to 3.00 and question answering from 3.76 to 3.30.Source: §5.3 · p. 9

6 DISCUSSION AND FUTURE WORK

The discussion identifies both the promise and limits of multimodal assistance: adapted models gain most on guidance, but the sensor contributions are not isolated. The authors also note limits in the data, evaluation, and offline processing, and outline extensions for future work.

6 DISCUSSION AND FUTURE WORK

  • Adapted models gain most on guidance, where answers depend on the user’s position and motion, but the gains combine sensor fusion with task adaptation.The learned reliability gates do not measure each stream’s causal importance, which the authors leave for future work to isolate.Source: §6 · p. 9
  • VISTA was collected in one region with sighted participants simulating assistive scenarios, and its automatic evaluation cannot fully measure real-world usefulness or safety.The model processes clips offline and estimates trajectories from the full recording, so it does not yet support real-time assistance.Source: §6 · p. 9
  • Planned work includes broader participant and location coverage, audio, depth, and streaming inputs, plus human ratings of usefulness and safety.Proposed baseline extensions include compact environment-state tracking and risk-aware generation.Source: §6 · pp. 9–10

7 CONCLUSION

The conclusion presents VISTA as a ten-category multimodal egocentric benchmark and reports a persistent gap between scene understanding and actionable assistance. Its sensor-input baseline with task adaptation improves guidance across evaluated backbones.

7 CONCLUSION

  • VISTA benchmarks assistance to blind and low-vision users across ten egocentric task categories.The conclusion frames the benchmark around evaluating assistance, rather than scene understanding alone.Source: §7 · p. 10
  • Current VLMs frequently fail to translate visual observations into grounded, actionable guidance.The conclusion describes this as a persistent gap between scene understanding and assistance.Source: §7 · p. 10
  • A baseline combining synchronized sensor inputs with task adaptation improves guidance scores across all evaluated backbones.The authors state that these findings motivate models connecting perception of surroundings with actions needed to achieve user goals.Source: §7 · p. 10

A VISTA BENCHMARK DETAILS

The appendix details VISTA’s clips, sensor streams, annotations, and evaluation protocol. It describes ten task categories, three annotation types, and a shared judge process for benchmark scoring.

A.1 TASK CATEGORIES

  • VISTA contains 997 clips across ten categories, with 97 to 102 clips in each category.Table 4 gives a question–answer pair and a goal–instruction pair for each category.Source: §A.1 · p. 15, Table 4 · p. 16

A.2 RECORDED SENSOR STREAMS

  • Recorded streams include RGB video, audio, inertial measurements, SLAM-camera images and trajectories, eye-tracking images and gaze estimates, and environmental measurements.Figure 6 shows example signals; trajectory, orientation, and environmental scales vary by clip.Source: §A.2 · p. 15, Figure 6 · p. 17

A.3 ANNOTATION EXAMPLES

  • Each clip has a scene description, a question–answer pair, and a goal–instruction pair.Table 5 gives examples of all three annotation types.Source: §A.3 · p. 17, Table 5 · p. 18

A.4 ANNOTATION STATISTICS

  • Annotation-field counts can exceed clip counts because clips may have multiple references; the task split uses one scene-description item per clip.Table 6 reports annotation-field lengths, while Figure 7 shows text-length distributions and item counts by split.Source: §A.4 · p. 18, Table 6 · p. 18, Figure 7 · p. 19

B EVALUATION PROTOCOL

  • A shared judge engine and dataset-specific adapters provide the evaluation pipeline for VISTA and EgoBlind.The engine holds the judge model, rubric, and score aggregation; adapters load data, extract frames, and build prompts.Source: B EVALUATION PROTOCOL · p. 19

B.1 INFERENCE PROTOCOL

  • Zero-shot models receive no benchmark-specific fine-tuning, adapter, or in-context example; each item is represented by eight sampled frames and a task-specific prompt.VISTA samples uniformly over the full clip, while EgoBlind samples up to the annotated question timestamp.Source: §B.1 · p. 19

B.2 JUDGE MODEL AND SCORING

  • The judge accepts agreement with any reference, but benchmark scores are not directly comparable when reference counts differ.VISTA averages 1.01 references per item, compared with 2.24 for EgoBlind.Source: §B.2 · p. 19
  • The 0–5 judge score combines evidence grounding, answer correctness, task completion, and accessible communication; critical-error answers are capped at 1.95.Evidence grounding and answer correctness each account for 35% of the total score.Source: §B.2 · p. 20

B.3 FAIRNESS AND REPRODUCIBILITY

The evaluation uses a shared, blinded judging protocol, while repeated calls can produce different verdicts. The prompt specifies task-specific guidance and a four-part rubric, including criteria for flagging potentially harmful answers.

B.3 FAIRNESS AND REPRODUCIBILITY

  • All models share the test split, frame sampling, prompt template, judge, and rubric; predictions are frozen before judging, and the judge is blind to model identity.The rubric also instructs the judge to ignore response length and confidence.Source: §B.3 · p. 20
  • One judge evaluation is reported per prediction, but repeated calls on identical inputs can yield different verdicts, leaving scores subject to judge variability.The judging procedure can be rerun from its specified prompt and settings, and recorded verdicts are released for inspection.Source: §B.3 · pp. 20–21

B.4 JUDGE PROMPT

  • The judge receives eight sampled frames, the task data, prediction, and references, then scores evidence grounding, answer correctness, task completion, and accessible communication.Each dimension has its own point range, and points are assigned in increments of 0.05.Source: §B.4 · pp. 21–22
  • A critical error is an identifiable answer error that could cause immediate harm if followed; it requires a negative correctness judgment and caps the total score at 1.95/5.The four component scores remain available for audit.Source: §B.4 · p. 21
  • Task-specific guidance changes with the annotation type, assessing scene descriptions, direct answers, or feasible goal-directed actions grounded in the visible layout.For goal guidance, the prompt accepts alternative safe routes and judges the requested outcome rather than requiring a particular step count or numerical distance.Source: §B.4 · pp. 21–22

C.1 VISTA ZERO-SHOT RESULTS BY RESPONSE TYPE AND TASK CATEGORY

Guidance is the weakest response type for every model in the reported zero-shot results. Lower-scoring task categories vary by response type, with task instructions and distance estimation repeatedly among the weakest.

C.1 VISTA ZERO-SHOT RESULTS BY RESPONSE TYPE AND TASK CATEGORY

  • 2.91 is the best open-source guidance score (InternVL3-14B), below its 3.81 scene-description score and Gemma 3-27B’s 3.81 question-answering score.Table 8 breaks down the zero-shot VISTA results by response type.Source: §C.1 · p. 22, Table 8 · p. 23
  • Across open-source models, the lowest-scoring categories include task instructions and social cues for scene description, task instructions and distance estimation for question answering, and distance estimation and motion understanding for guidance.For scene description, the lowest averages are TC07 (3.39), TC05 (3.41), and TC10 (3.45).Source: §C.1 · p. 22, Table 9 · p. 24

S EG AC TC ACom S EG AC TC ACom

The category-level results identify task-specific weaknesses in question answering and guidance. On EgoBlind, the leading models retain the same ranking as on VISTA, though absolute scores are not directly comparable across benchmarks.

S EG AC TC ACom S EG AC TC ACom

  • 2.72, 2.75, and 2.80 are the lowest question-answering scores for TC07, TC06, and TC04, respectively.Source: Table 10 · p. 25
  • 2.08, 2.15, and 2.17 are the lowest guidance scores for TC06, TC10, and TC04, respectively.Source: Table 11 · p. 26

C.2 EGOBLIND ZERO-SHOT RESULTS

  • On EgoBlind, Qwen3-VL-8B (3.58) and Qwen3-VL-4B (3.56) are the strongest open-source models, while Gemini 3.1 Flash-Lite (3.88) is strongest overall.The ranking among models evaluated on both benchmarks matches VISTA, but absolute scores are not directly comparable because EgoBlind supplies more references per item.Source: §C.2 · p. 27, Table 12 · p. 27

D FAILURE ANALYSIS

Guidance is the most failure-prone response type, and hazard detection and navigation have the highest critical-error rates among task categories. Across all fourteen models, four recurring failure modes concern precise egocentric grounding and reasoning.

D.1 FAILURE STATISTICS

  • 17.5% of guidance items have critical errors, compared with 2.5% for scene description; hazard detection and navigation have the highest category-level critical-error rates.No model reaches 3/5 on 4.3% of items, including the strongest model, Gemini 3.1 Flash-Lite.Source: §D.1 · p. 27, Table 13 · p. 28

D.2 QUALITATIVE FAILURE CASES

  • Four failure modes recur across all fourteen models: egocentric body orientation, objects behind the wearer, dynamic and temporal reasoning, and metric distance estimation.Neither model scale nor model family removes these recurring failures.Source: §D.2 · p. 28, Figure 8 · p. 29
  • In the body-orientation case, 13 of 14 responses are flagged as errors that could cause immediate harm if followed.Source: Table 14 · p. 30
  • For the dynamic and temporal reasoning case, every model answers yes when the reference answer is no, while none is flagged for a critical error.Source: Table 16 · p. 31
  • In the metric-distance case, every model makes a claim the eight frames cannot verify.Source: Table 17 · p. 31

Related papers

Loading 2610.22822v1…