Source-linked AI summary
EgoExoBench: A Benchmark for First- and Third-person View Video Understanding in MLLMs
Yuping He, Yifei Huang, Guo Chen, Baoqi Pei, Jilan Xu, Tong Lu, Jiangmiao Pang
TL;DR
Whether MLLMs can integrate information across first- and third-person video remains an open question, despite the importance of cross-view reasoning to human intelligence. EgoExoBench evaluates that ability across 11 subtasks and finds that current models still struggle to bridge perspectives, while deliberate human performance substantially exceeds the best model.
Problem
Although cross-view knowledge transfer is central to human intelligence, MLLMs’ ability to integrate information across egocentric and exocentric views has not been systematically evaluated.
Source: §1 · pp. 1–2Method
EgoExoBench evaluates cross-view video understanding with over 7,300 question–answer pairs across 11 subtasks covering semantic alignment, spatial correspondence, and temporal reasoning.
Source: §1 · p. 2Results
Current MLLMs struggle to bridge perspectives, and deliberate human evaluators reach 90.1% accuracy, 42% above the best model.
Source: §4.2 · p. 7, §5 · p. 10Takeaways & Limitations
EgoExoBench offers an evaluation resource for research on embodied agents and collaborative systems seeking cross-view reasoning.
Source: §5 · p. 10The benchmark may not fully reflect the breadth of real-world ego–exo scenarios.
Source: §5 · p. 10
Abstract
from arXiv · showhide
Transferring and integrating knowledge across first-person (egocentric) and third-person (exocentric) viewpoints is intrinsic to human intelligence, enabling humans to learn from others and convey insights from their own experiences. Despite rapid progress in multimodal large language models (MLLMs), their ability to perform such cross-view reasoning remains unexplored. To address this, we introduce EgoExoBench, the first benchmark for egocentric-exocentric video understanding and reasoning. Built from publicly available datasets, EgoExoBench comprises over 7,300 question-answer pairs spanning eleven sub-tasks organized into three core challenges: semantic alignment, viewpoint association, and temporal reasoning. We evaluate 13 state-of-the-art MLLMs and find that while these models excel on single-view tasks, they struggle to align semantics across perspectives, accurately associate views, and infer temporal dynamics in the ego-exo context. We hope EgoExoBench can serve as a valuable resource for research on embodied agents and intelligent assistants seeking human-like cross-view intelligence.
1 Introduction
Cross-view reasoning lets people transfer knowledge between first- and third-person perspectives, but whether MLLMs can do this systematically remains unevaluated. EgoExoBench addresses that gap and finds current models still struggle to bridge perspectives.
1 Introduction
- Current benchmarks assess egocentric or exocentric video in isolation, leaving MLLMs’ ability to integrate information across viewpoints unevaluated.The paper frames this as an open question about whether MLLMs can perform cross-view reasoning like humans.Source: §1 · pp. 1–2
- EgoExoBench introduces a cross-view benchmark with over 7,300 question–answer pairs and 11 subtasks covering semantic alignment, spatial correspondence, and temporal reasoning.Its temporal tasks require aligning and predicting event sequences across asynchronous or overlapping streams.Source: §1 · p. 2
- Across 13 open- and closed-source MLLMs, models strong on single-view tasks drop on cross-view challenges, and explicit reasoning models still struggle to combine text reasoning with dual-view video.The authors identify this as a gap from the cross-view understanding needed for embodied agents and human–robot collaboration.Source: §1 · p. 2
2 Related Work
Prior work advances single-view video understanding, builds paired ego–exo datasets, and uses exocentric data to improve recognition. The paper positions EgoExoBench as an evaluation of cross-perspective reasoning that these efforts do not provide.
Video benchmarks
- Video benchmarks such as Next-QA [64], MLVU [78], and Video-MME [14] assess video understanding from one viewpoint at a time, while EgoExoBench evaluates cross-perspective reasoning.The paper also cites EgoSchema [40] for egocentric long-form understanding and HourVideo [7] for spatial intelligence.Source: §2 · p. 2
Ego-exo datasets
- Paired datasets include synchronized home and multi-step recordings [54, 47, 51, 26] and larger collections with rich annotations, but the latter lack downstream QA or reasoning benchmarks.Robotics datasets such as iGibson [32] and H2O [30] focus more narrowly on pose estimation and navigation.Source: §2 · p. 2
3 Benchmark
EgoExoBench evaluates cross-view video understanding through an 11-subtask suite, with questions built from paired video datasets using varied construction and filtering strategies. Its tasks cover semantic matching, spatial viewpoint translation, and temporal event reasoning.
3.1 Task Suite
- The 11 subtasks test semantic alignment, spatial correspondence, and sequence integration across egocentric and exocentric video streams.The suite includes tasks, actions, people, and objects; spatial translation; and event alignment and prediction.Source: §3.1 · p. 3
3.2 Data Construction Pipeline
- The pipeline draws on six paired datasets, mixes annotation-derived, LLM-generated, and human-built questions, then filters ambiguous or text-only-solvable items.The datasets include synchronized and asynchronous recordings; all questions use four choices.Source: §3.2 · pp. 3, 5
3.3.1 Ego-Exo Relation
- Four relation subtasks match tasks, actions, people, or objects across views, using challenging negatives to discourage scene, temporal, and identity shortcuts.Negatives include different tasks in similar settings, other objects in the same environment, or different people at the same timestamp.Source: §3.3.1 · p. 5
3.3.2 Ego-Exo View Transition
- Three spatial tasks identify the wearer, predict movement direction, and map body-part actions across first- and third-person perspectives.Wearer samples retain frames with at least four people; direction examples are manually verified, and body-part questions are filtered for text-only solvability.Source: §3.3.2 · pp. 5–6
3.3.3 Ego-Exo Temporal Reasoning
- Four temporal tasks forecast next actions, compare action order, align multistep sequences, and assess skill across viewpoints.Sequence-alignment pairs share at least two annotated steps, while skill questions compare candidates against expert demonstrations.Source: §3.3.3 · p. 6
3.4 Benchmark Statistics
- 7,330 four-way MCQs from six paired datasets span 11 subtasks; Figure 3 shows question counts by subtask and by dataset.The benchmark comprises three task categories.Source: §3.4 · pp. 6–7, Figure 3 · p. 7
4 Experiment
The experiments evaluate MLLMs on EgoExoBench and test whether reasoning prompts or reference-view context improve performance. Models remain well below human accuracy; CoT usually hurts, while reference videos help Action Prediction but do not consistently help Skill Evaluation.
4.1 Experiment Setup
- Evaluations used zero-shot multiple-choice prompts, accuracy, and no fine-tuning or gradient updates.The models received standardized prompts; open-source experiments ran on four A100 GPUs with 80GB memory each.Source: §4.1 · p. 7
4.2 Main Result
- Humans reached 90.1% average accuracy with three minutes per question, outperforming the best model by 42%.The largest gap was in Egocentric Wearer Identification; the smallest was in Skill Evaluation, which challenged human evaluators too.Source: §4.2 · pp. 7–8
- GPT-o4-mini achieved 48.0% overall accuracy among closed-source models, while Qwen2.5-VL-72B led open-source models at 47.0%.Table 1 also shows GPT-o4-mini ahead of Qwen2.5-VL-72B by 5% on average in Ego-Exo Relation; Qwen2.5-VL-72B reached 47.3% in View Transition and 37.4% in Temporal Reasoning. EgoGPT’s gains over similarly sized counterparts were marginal.Source: §4.2 · p. 7, Table 1 · p. 8
4.3 Can Reasoning Improve MLLM’s Performance?
- CoT degraded performance on most tasks; Qwen2.5-VL-72B lost 20% on Person Relation and 19% on Action Relation.The authors attribute the drops to text-only decomposition disrupting the interleaved visual-linguistic workflow required across video streams.Source: §4.3 · p. 8, Figure 4 · p. 9
- In an Egocentric Wearer Identification example, GPT-4o first identified a nearby person correctly, then misidentified that person as the camera wearer.The example illustrates a loss of continuity between its first-person observation and subsequent cross-view inference.Source: §4.3 · p. 8, Figure 5 · p. 9
4.4 Can MLLMs Leverage Cross-Perspective Guidance?
- Adding a reference video consistently improved Action Prediction, including an 8.2% gain for Qwen2.5-VL, but Skill Evaluation results varied.Qwen2.5-VL dropped 1.5% on Skill Evaluation with the reference; the authors suggest domain-specific expertise and subtle quality judgments may outweigh extra visual context.Source: §4.4 · p. 9, Table 2 · p. 9
5 Conclusion
EgoExoBench evaluates cross-view video understanding in MLLMs through paired egocentric–exocentric recordings and over 7,300 questions. The evaluation finds that current models struggle to bridge perspectives, while the benchmark may not capture the full breadth of real-world ego–exo scenarios.
5 Conclusion
- EgoExoBench is a large-scale benchmark designed specifically to evaluate cross-view video understanding in multimodal LLMs.It aggregates paired egocentric–exocentric recordings and more than 7,300 multiple-choice questions across eleven subtasks.Source: §5 · p. 9
- The benchmark probes semantic alignment, spatial correspondence, and sequence integration, and current MLLMs struggle to bridge perspectives despite strong single-view performance.Chain-of-thought prompting and additional cross-perspective guidance offer limited improvements.Source: §5 · p. 10
- The benchmark may not fully reflect the breadth of real-world ego–exo scenarios.The authors identify this as a boundary for the benchmark and leave it for future work.Source: §5 · p. 10
- The authors hope the benchmark will support research on embodied agents and collaborative systems requiring human-like cross-view intelligence.Source: §5 · p. 10
A Additional details and benchmark statistics
The appendix section outlines additional benchmark statistics, task-specific question-answer construction details, and human-evaluation performance.
A Additional details and benchmark statistics
- The appendix first presents additional EgoExoBench statistics.Source: A Additional details and benchmark… · p. 14
- It then details question-answer construction for each task.Source: A Additional details and benchmark… · p. 14
- It concludes with detailed human-evaluation performance information.Source: A Additional details and benchmark… · p. 14
A.1 Benchmark Statistics
EgoExoBench is built from six publicly available ego–exo datasets, with 11 subtasks across three task categories. The appendix details how its relation, view-transition, and temporal-reasoning questions are constructed.
A.1 Benchmark Statistics
- Six public ego–exo datasets contribute to 11 subtasks across three task categories, with their QA-pair distribution summarized by dataset and subtask.Figure 6 summarizes the number of QA pairs each dataset contributes to every subtask.Source: §A.1 · p. 14, Figure 6 · p. 15
A.2 Ego-Exo Relation
- Ego–Exo Relation tests cross-view matching at task, action, object, and person levels using distractors designed to require relevant visual cues.Task-level distractors share a general scene type; other subtasks use temporally aligned or same-time clips, with bounding boxes marking the person in person-relation candidates.Source: §A.2 · pp. 14–15
A.3 Ego–Exo View Transition
- Ego–Exo View Transition tests cross-view spatial translation through wearer identification, direction prediction, and body-part action questions.Wearer candidates include at least four people; direction questions are human-annotated, while body-part questions are generated and filtered with LLMs.Source: §A.3 · pp. 15, 18
A.4 Ego–Exo Temporal Reasoning
- Ego–Exo Temporal Reasoning covers next-action prediction, action ordering, sequence alignment, and skill evaluation across video streams.Construction uses action or keystep annotations and filters for plausible order, unambiguous answers, or proficiency differences, depending on the subtask.Source: §A.4 · pp. 18, 21, 22
A.5 Human Performance Evaluation.
The human baseline is measured on 330 questions sampled evenly across the 11 subtasks. Two graduate students complete the same video-based inputs as the MLLMs.
A.5 Human Performance Evaluation.
- 330 questions are sampled, with 30 drawn from each subtask, to estimate human-level performance on EgoExoBench.The evaluation subset spans all 11 subtasks.Source: §A.5 · p. 24
- Two graduate students who did not construct the data answer the same questions, options, and videos provided to MLLMs.They may pause, replay, and rewatch without time constraints; their average accuracy is the human baseline.Source: §A.5 · p. 25
B Experiments
The experiments examine whether MLLMs distinguish concatenated video inputs, report additional EgoExoBench and benchmark-comparison results, and provide CoT and reference-video ablations.
B Experiments
- The experiments cover video-input boundary recognition, further EgoExoBench results and benchmark comparisons, plus CoT and reference-video ablations.The overview lists these as the planned experimental topics.Source: B Experiments · p. 25
B.1 Preliminary Study: Can MLLMs Distinguish Multiple Video Inputs?
A preliminary experiment tests whether MLLMs distinguish multiple video streams, a capability needed for EgoExoBench comparisons. All evaluated models score above 77% accuracy, well above the 10% random-guess expectation.
B.1 Preliminary Study: Can MLLMs Distinguish Multiple Video Inputs?
- The task presents five clips, including an exact duplicate pair, to test whether models treat concatenated inputs as distinct streams.The authors describe this capability as critical because EgoExoBench requires comparing or relating information across multiple videos.Source: §B.1 · p. 25
- All evaluated models achieve above 77% accuracy on identifying duplicate clips among five video inputs, versus 10% expected under random guessing.The evaluation uses 100 questions and eight uniformly sampled frames per video; a correct response must identify the matching pair precisely.Source: §B.1 · p. 25, Table 3 · p. 25
B.2 Main Results
Gemini 2.5 Pro achieves the highest reported EgoExoBench accuracy, while the appendix also details Chain-of-Thought prompting and examples of cross-video reasoning successes and failures.
B.2 Main Results
- 51.7% average accuracy gives Gemini 2.5 Pro the highest EgoExoBench performance, 3.7% above second-best GPT-4o-mini.Table 4 reports the benchmark comparison.Source: §B.2 · p. 25, Table 4 · p. 27
- Qwen2.5VL-72B and InternVL3-78B outperform GPT-4o on certain widely used video-understanding benchmarks.Table 5 compares MLLM performance across EgoExoBench and existing benchmarks.Source: §B.2 · p. 25, Table 5 · p. 27
B.3 Experimental Details of CoT Prompting
- CoT prompting appends “Let’s think step by step.” with temperature 0, top-p 1, and top-k 1; Qwen2.5-32B extracts the final answer.Figure 19 illustrates the answer-extraction prompt.Source: §B.3 · pp. 25, 27, Figure 19 · p. 27
- The identical-video-pair task asks a model to identify which two of five videos are most likely identical.Figure 18 illustrates the task.Source: §B.3 · p. 26, Figure 18 · p. 26
- GPT-4o’s successful cases compare key actions for Task Relation and track object-interaction order and fine-grained actions for Sequence Alignment.The Sequence Alignment example distinguishes actions such as covering and unwrapping.Source: §B.3 · p. 27, Figure 20 · p. 28
- Failure examples show GPT-4o mislocalizing the active hand across views and selecting a person whose proximity contradicts its earlier observation.These cases concern Body Part Action Understanding and Person Relation.Source: §B.3 · p. 27
B.4 Experimental Details on Reference Video Usage.
The reference-video ablation removes that video while retaining the original input and compares GPT-4o predictions with and without the reference for Action Prediction and Skill Evaluation.
B.4 Experimental Details on Reference Video Usage
- The ablation removes the reference video while retaining the original input, changing only prompt components needed to reflect its absence.All other experimental settings remain unchanged.Source: §B.4 · p. 28
- A Sequence Alignment example shows successful cross-video comparison by tracking key-object interaction order and distinguishing fine-grained actions.The example highlights the model’s reasoning steps.Source: Figure 21 · p. 29
- Figures 26 and 27 compare GPT-4o predictions with and without the reference video for Action Prediction and Skill Evaluation.Correct and incorrect predictions are highlighted in the examples.Source: §B.4 · p. 29, Figure 26 · p. 33, Figure 27 · p. 34
- Failure examples show inconsistent cross-view localization of the active hand and a person selection contradicting the model’s earlier distance assessment.The figures mark correct and incorrect or inconsistent reasoning steps.Source: Figure 22 · p. 30, Figure 23 · p. 31
- The Action Prediction prompt uses the exocentric video as a gold standard for action order and details to predict the next egocentric action.The prompt asks the model to choose the option matching the expected next action.Source: §B.4 · p. 32