PaperlayerView exact revision on arXiv

Source-linked AI summary

EgoExo-Next:Benchmarking Vision-Language Models on Visual-Option Next-State and Cross-View Reasoning

Yutong Li, Molin Wang, Xiaotong Li, Yanyan Fang, Daoguo Dong, Ziyi Ye

arXiv:2610.04506v1cs.CVcs.AI

TL;DR

Existing video reasoning benchmarks largely test semantic events, temporal relations, or correspondence between observed views rather than future visual states. EgoExo-Next evaluates visual-state progression and cross-view correspondence across four connected tasks; the best model achieves 43.81% average accuracy versus 98.55% for humans, with the largest gap on the composed task.

  • Problem

    Existing benchmarks largely evaluate semantic events, temporal relations, or correspondence between already observed views, leaving direct evaluation of unobserved future visual-state reasoning underexplored.

    Source: §1 · pp. 1–2

  • Method

    EgoExo-Next is a visual-option benchmark that uses four temporally related image candidates to evaluate temporal state progression, cross-view correspondence, and their composition across four subtasks.

    Source: §1 · pp. 2–3, §3 · p. 4

  • Results

    43.81% average accuracy for the best model versus 98.55% for humans, with the largest gap on the composed Ego-to-Exo Next-State task.

    Source: §1 · p. 3

  • Takeaways & Limitations

    Current VLMs remain substantially below human performance in dynamic visual-state reasoning, particularly when temporal progression and cross-view reasoning are composed.

    Source: §5 · p. 10

Abstract

from arXiv · show

Vision-language models (VLMs) are increasingly evaluated for egocentric and cross-view video reasoning, yet existing benchmarks largely focus on semantic event understanding, temporal relations, or correspondence between already observed views, leaving their ability to reason directly about future visual states underexplored. We introduce EgoExo-Next, a visual-option benchmark for dynamic visual-state reasoning, where models must identify how an observed action trajectory subsequently appears rather than predict only an action label or textual description. EgoExo-Next contains 2,503 human-curated four-choice questions from six public egocentric and ego--exo video sources and comprises four interconnected subtasks that evaluate egocentric next-state prediction, bidirectional ego--exo state correspondence, exocentric next-state prediction, and their composition in Ego-to-Exo Next-State. Extensive evaluation of proprietary, open-source, and spatial reasoning VLMs reveals a substantial human--model gap, with the best model achieving 43.81\% average accuracy compared with 98.55\% for humans, and the largest degradation occurring on the composed Ego-to-Exo task. These results suggest that current VLMs remain substantially limited in dynamic visual-state reasoning, particularly when temporal progression and cross-view reasoning must be composed. The benchmark is publicly available at \url{https://huggingface.co/datasets/yutongli2024/EgoExo-Next}.

1 INTRODUCTION

Existing egocentric and ego–exo benchmarks mainly test event understanding, temporal relations, or matching observed views rather than identifying an unobserved future visual state. EgoExo-Next evaluates that capability across four tasks; the best model reaches 43.81% average accuracy versus 98.55% for humans, with the largest gap on the composed task.

1 INTRODUCTION

  • The composition is relevant to embodied settings where agents perceive egocentrically while demonstrations, supervision, or collaborative observations may come from exocentric views.The benchmark introduces a visual-option task to jointly evaluate temporal progression and viewpoint transformation.Source: §1 · pp. 1, 3
  • Existing benchmarks assess action or event understanding and correspondence between observed views, but do not directly test progression into an unobserved visual state.A semantically correct next-action prediction may still fail to identify the next scene configuration, including object configuration, pose, contact relations, and spatial layout.Source: §1 · p. 2
  • EgoExo-Next targets the subsequent visual state itself, using four temporally related image candidates and distractors from both earlier and later-future states.The candidate design requires distinguishing fine-grained progression even when states share high-level semantics.Source: §1 · p. 2
  • Its four subtasks cover egocentric progression, bidirectional ego–exo matching, exocentric progression, and their composition into a future exocentric state.The composed task infers an unobserved subsequent state from preceding egocentric observations and identifies its appearance from an exocentric viewpoint.Source: §1 · pp. 2–3
  • 43.81% average accuracy for the best model versus 98.55% for humans, with the largest gap on composed Ego-to-Exo Next-State.The task requires inferring an unobserved state from egocentric observations and identifying its exocentric appearance.Source: §1 · p. 3
  • The benchmark contains 2,503 human-curated four-choice questions from six public video sources, and evaluation finds performance depends on temporal context and cross-view evidence.Its design uses non-uniform temporal sampling and controlled earlier-state and later-future distractors.Source: §1 · p. 3

2 RELATED WORK

Prior work spans egocentric video question answering and paired ego–exo datasets, but these resources address varied temporal, planning, annotation, and cross-view learning goals. EgoExo-Next is situated among those existing benchmark and dataset lines.

2.1 EGOCENTRIC VQA BENCHMARKS

  • Egocentric VQA benchmarks address long-form temporal understanding, goal-oriented planning, multidimensional first-person reasoning, fine-grained temporal patterns, and cross-view understanding.Examples include EgoSchema, EgoPlan, EgoThink, EgoTempo, EgoProx, and EgoExoBench.Source: §2.1 · p. 3

2.2 EGO-EXO DATASETS

  • Paired ego–exo datasets range from synchronized daily-activity views to multi-agent, RGB-D, procedural, skilled-activity, asynchronous-learning, and full-body skill-assessment settings.The cited resources include Charades-Ego, Home Action Genome, LEMMA, H2O, Assembly101, Ego-Exo4D, EgoExoLearn, and EgoExo-Fitness.Source: §2.2 · pp. 3–4

3 EGOEXO-NEXT

EgoExo-Next evaluates visual-state reasoning across egocentric and exocentric perspectives, asking models to select target states from four visual candidates across four interconnected subtasks.

3.1 OVERVIEW

  • The suite tests egocentric and exocentric next-state progression, synchronized cross-view correspondence, and their composition in Ego-to-Exo Next-State.The composed task infers a future exocentric state from egocentric observations.Source: §3.1 · p. 5

3.2 BENCHMARK CONSTRUCTION

  • Construction begins with human-identified complete action segments, then forms subtask-specific observations and candidate states with controlled earlier-state and later-future distractors.Sampling intervals are not fixed, and human verification and quality filtering check that target states are visually distinguishable.Source: §3.2 · p. 5

3.2.1 DATA SOURCE

  • Six public sources span indoor and outdoor settings, with egocentric videos supporting temporal progression and paired or multi-view recordings supporting correspondence and composed reasoning.The sources cover cooking, object manipulation, sports, procedural activities, and human motion.Source: §3.2.1 · p. 5

3.2.2 QUESTION-ANSWER CONSTRUCTION

  • For next-state questions, the target is the nearest sampled future state; distractors may be earlier or later, and candidate sets vary in the number of future options.Equal groups contain one, two, three, or four future candidates, preventing selection based only on future status or temporal advancement.Source: §3.2.2 · p. 5

3.2.3 BENCHMARK STATISTICS

  • The 2,503 questions are nearly even across subtasks and indoor/outdoor scenes; Ego-Exo4D contributes the largest source share at 35.9% (899 questions).Figure 3 presents composition by subtask, scene type, and data source.Source: §3.2.3 · p. 6, Figure 3 · p. 6

3.2.4 QUALITY ASSURANCE AND FILTERING

  • Independent reviewers check generated QA groups for ambiguity, visual quality, distinguishable candidates, cross-view counterparts, and extraction failures; unclear groups are revised or removed.Correct-answer positions are balanced across A/B/C/D, and candidate order is randomized.Source: §3.2.4 · pp. 6–7

4 EVALUATION ON EGOEXO-NEXT

EgoExo-Next is evaluated in a zero-shot four-way multiple-choice setting across proprietary, open-source, and spatial reasoning models. Results show a substantial human–model gap, and controlled analyses test benchmark validity, visual-similarity shortcuts, and recurring model failures.

4.1 EXPERIMENT SETUP

  • Models are evaluated zero-shot on four-way multiple-choice questions, with mean accuracy across three runs and bidirectional Subtask 2 scores averaged.Source: §4.1 · p. 7

4.2 MAIN RESULTS

  • Gemini-3.1-Pro leads models evaluated across all four subtasks at 43.81% overall accuracy, compared with 98.55% for humans.Source: §4.2 · p. 7
  • GPT-6 reaches 62.23% Indoor and 58.14% Outdoor accuracy on Subtask 4, beating previous best results by 29.34 and 23.86 percentage points.Source: §4.2 · p. 8

4.3 BENCHMARK VALIDITY AND SHORTCUT ANALYSIS

  • In the Candidate Only control, average accuracy falls from 40.70% to 26.31%, close to random guessing, showing performance depends on more than candidate-side information alone.Source: §4.3 · p. 8
  • Last Frame Only and Shuffled Sequence lower temporal accuracy to 35.56% and 31.70%; Mismatched Observation lowers cross-view accuracy to 26.32%.Source: §4.3 · p. 8

4.4 VISUAL-SIMILARITY BASELINES

  • CLIP and DINO average 26.54% and 26.49%, respectively, only slightly above the 25% random baseline.Source: §4.4 · pp. 8–9

4.5 FAILURE CASE ANALYSIS

  • Failures include choosing earlier states, missing cross-view correspondences, confusing subtle state differences, and failing to compose temporal and viewpoint reasoning.Source: §4.5 · p. 9

5 CONCLUSION

EgoExo-Next evaluates dynamic visual-state reasoning across temporal progression, cross-view correspondence, and their composition. The accompanying materials clarify its benchmark positioning and provide additional diagnostic and implementation details.

5 CONCLUSION

  • EgoExo-Next contains 2,503 human-curated four-way questions from six public sources, and current VLMs remain substantially below humans, especially on composed Ego-to-Exo Next-State.Source: §5 · pp. 9–10

A APPENDIX OVERVIEW AND ORGANIZATION

  • Supplementary sections detail experimental protocols and prompts, diagnose composed-task capabilities, assess caption augmentation, and report implementation and participant-level human-evaluation results.Source: A APPENDIX OVERVIEW AND ORGANIZATION · p. 13
  • The appendix distinguishes direct prediction of an unobserved visual state from benchmarks targeting actions, events, relations, or text, with Ego-to-Exo composing progression and viewpoint transformation.Source: A APPENDIX OVERVIEW AND ORGANIZATION · p. 13

B EXPERIMENTAL SETUP

The evaluation uses zero-shot four-way multiple-choice questions with separate observation and candidate images. Prompts adapt to the task’s viewpoints, while Subtask 2 tests synchronized cross-view matching in both directions.

B EXPERIMENTAL SETUP

  • Models receive no in-context examples, task-specific fine-tuning, or gradient updates; the evaluated groups are proprietary, open-source, and spatial reasoning models.Source: B EXPERIMENTAL SETUP · pp. 13–14
  • Training-free CLIP and DINO baselines select the candidate most cosine-similar to the final observation frame, without using earlier observations or task-specific training.Source: B EXPERIMENTAL SETUP · p. 14

C.1 EVALUATION PROMPTS FOR SUBTASK1, 3, 4

  • A unified next-state prompt covers Subtasks 1, 3, and 4, with observation and candidate perspectives set for each task and an added external-view instruction for Subtask 4.Source: §C.1 · p. 14

C.2 EVALUATION PROMPT FOR SUBTASK 2

  • Subtask 2 presents a keyframe from one viewpoint and four candidates from the other, asking for the synchronized same-moment state in either direction.Source: §C.2 · p. 14

C.1 EVALUATION PROMPTS FOR SUBTASK1, 3, 4

  • The next-state prompt asks models to track action progression across ordered observations and choose the subsequent visual state from four trajectory candidates.Source: §C.2 · p. 15

D COMPARISON WITH EXISTING EGOCENTRIC AND SPATIAL REASONING BENCHMARKS

Existing egocentric and spatial benchmarks target actions, semantic answers, spatial relations, or other symbolic outputs. EgoExo-Next instead tests direct identification of unobserved visual states, including a task that composes temporal progression and viewpoint transformation.

D COMPARISON WITH EXISTING EGOCENTRIC AND SPATIAL REASONING BENCHMARKS

E DIAGNOSING COMPOSED EGO-TO-EXO NEXT-STATE REASONING

The diagnostic study tests how cross-view correspondence and egocentric temporal progression contribute to Ego-to-Exo Next-State, and finds that combining these abilities remains difficult.

E DIAGNOSING COMPOSED EGO-TO-EXO NEXT-STATE REASONING

  • The matched diagnostic subset contains 200 Subtask 4 groups, split evenly between Indoor and Outdoor examples, with Subtasks 1–3 derived from the same clips and synchronized views.The study also reports Subtask 4 accuracy conditional on all three diagnostic tasks being answered correctly.Source: E DIAGNOSING COMPOSED EGO-TO-EXO… · p. 16
  • 58.58% accuracy with both oracle inputs, versus 29.64% on original Subtask 4, while accuracy is 57.40% when all three matched diagnostic tasks are correct.The results support the diagnostic decomposition, but remaining errors indicate that unified egocentric-to-exocentric next-state reasoning remains difficult.Source: E DIAGNOSING COMPOSED EGO-TO-EXO… · pp. 16–17
  • Providing + Current Exo raises Subtask 4 accuracy to 43.24%, while + Next-State Ego raises it to 41.74%.The oracle inputs reduce the cross-view correspondence requirement and remove egocentric next-state prediction, respectively.Source: E DIAGNOSING COMPOSED EGO-TO-EXO… · p. 16

F EFFECT OF CAPTION AUGMENTATION

The study tests whether captions of candidate images add semantic cues to visual-state reasoning, finding higher overall accuracy with captions while cautioning against interpreting the gain as a direct language-bias measure.

F EFFECT OF CAPTION AUGMENTATION

  • The evaluation samples 400 questions, with 50 from each Indoor and Outdoor split across the four subtasks, and compares Image Only with Image + Caption.Qwen2.5-VL-72B generates captions for candidate images only; human reviewers verify and correct them, while observation inputs and evaluation prompts stay unchanged.Source: F EFFECT OF CAPTION AUGMENTATION · p. 17
  • 40.50% to 46.33% overall accuracy with caption augmentation, a gain of 5.83 percentage points; 34 of 40 model–setting pairs improve.Subtask 3 Outdoor shows the largest average gain, improving by 12.00 percentage points.Source: F EFFECT OF CAPTION AUGMENTATION · p. 17
  • Because captions are generated directly from candidate images, the gain supports the benefit of explicit semantic cues rather than directly measuring language bias.The authors state that explicit descriptions can complement visual information and help distinguish candidate states.Source: F EFFECT OF CAPTION AUGMENTATION · p. 17

G ADDITIONAL IMPLEMENTATION DETAILS AND RESULTS

The implementation details specify model deployment, zero-shot evaluation conditions, and repeated-run reporting.

G ADDITIONAL IMPLEMENTATION DETAILS AND RESULTS

  • Proprietary models and Qwen3-VL-235B use official APIs, while other open-source models run from official checkpoints on eight NVIDIA GeForce RTX 4090 GPUs.Source: §G.1 · p. 18
  • All models are evaluated zero-shot, without in-context examples, task-specific fine-tuning, or gradient updates, using the same instructions and output constraints.Source: §G.1 · p. 18
  • API evaluations use temperature 0 and a 32-token response limit; each model is run three times and mean accuracy is reported.Source: §G.1 · p. 18

G.2 HUMAN EVALUATION DETAILS

Human performance is measured using three participants who complete the benchmark independently, with results averaged across participants.

G.2 HUMAN EVALUATION DETAILS

  • Three adult participants, uninvolved in benchmark annotation or quality control, independently answer all 2,503 questions.Source: §G.2 · p. 18
  • Participants use the same observation and candidate images as the models, with randomized candidate order and no strict time limit.Source: §G.2 · p. 18
  • Human performance for each setting is the mean of the three individual accuracies, and Avg. is the unweighted mean across eight subtask scene settings.Source: §G.2 · p. 18

Related papers

Loading 2610.04506v1…