PaperlayerView exact revision on arXiv

Source-linked AI summary

Does Local Video Understanding Transfer Across Encounters? The EgoGears Benchmark

Yuedong Tan, Lei Qi, Yu Liu, Di Wen, Ruiping Liu, Xiaoye Wang, Yufan Chen, Junwei Zheng, Chengzhi Wu, Chen Zhang, Zhihang Chen, Haiwen Sun, Zongwei Wu, Radu Timofte, Danda Pani Paudel, Kunyu Peng

arXiv:2609.37938v1cs.CVcs.AIcs.RO

TL;DR

Aggregate cross-video accuracy does not distinguish local perception from failures to bind and compose evidence across encounters. EgoGears pairs single- and multi-video questions from repeated outdoor routes to examine that transition. Models perform better on individual observations than on relating them, with evidence binding and ordered route understanding remaining bottlenecks.

  • Problem

    It is unclear whether understanding individual observations supports consistent reasoning across separate encounters with changed viewpoints, motion, or illumination.

    Source: §1 · pp. 1–2

  • Method

    EgoGears pairs single- and multi-video questions from repeated human-collected outdoor traversals to assess local evidence, observation identity, correspondence, and route relationships.

    Source: §1 · pp. 2–3

  • Results

    Current MLLMs perceive individual observations better than they relate them; clip–evidence binding and ordered route understanding remain key bottlenecks.

    Source: §5 · p. 10

  • Takeaways & Limitations

    Explicit reasoning aids relation verification more than trajectory reconstruction in the evaluated setting.

    Source: §5 · p. 10

    Recordings come from a small number of participants and locations in one country, limiting representation of other regions, climates, terrains, body types, or devices.

    Source: A SOCIETAL IMPACT AND LIMITATIONS · p. 13

Abstract

from arXiv · show

Embodied systems must make knowledge acquired during one encounter usable in another despite changes in viewpoint, motion, and illumination. Yet aggregate cross-video accuracy conflates failures of local perception with failures to preserve observation identity, establish correspondence, and compose evidence, obscuring whether local video understanding actually transfers. We introduce EgoGears, a complementary single- and multi-video benchmark designed to diagnose this transition. It contains 567 single-video and 1,487 multi-video questions derived from 126 human-collected egocentric recordings covering 39 outdoor routes. Repeated traversals across movement speeds and lighting conditions ground comparisons in shared physical environments; 531 questions require alignment across independent recordings. Single-video questions measure the local visual, spatial, and motion evidence available to a model, while multi-video questions test whether evidence remains bound to the correct observation and can be composed into consistent route relationships. We report 29 single-video and 31 multi-video MLLM configurations across six model families in the main leaderboard. Among the 20 configurations evaluated comparably on both splits, every model performs worse on multi-video questions, with a mean decrease of 22.5 percentage points, and the gap persists when answer format and scoring are held fixed. The gap is not explained simply by additional videos or recording boundaries. The central bottlenecks are observation--evidence binding and ordered route-state tracking. The code and benchmark are publicly available at https://github.com/lei-qi-233/EgoGears.

1 INTRODUCTION

EgoGears pairs local video understanding with cross-video evaluation to test whether models preserve observation identity and compose evidence across changing outdoor encounters. Its evaluations reveal persistent binding and route-reasoning difficulties, while explicit reasoning helps spatial relations more than trajectory questions.

1 INTRODUCTION

  • Cross-video understanding matters for embodied systems that must connect separate, partial encounters despite changes in viewpoint, motion, and illumination.Repeated traversals can reverse landmark order or change how a familiar landmark appears.Source: §1 · pp. 1–2
  • Single-video questions assess local visual, spatial, and motion evidence, while multi-video questions test observation identity, correspondence, and composition of route relationships.Multi-video tasks include landmark, direction, and route-segment relations; distractors perturb direction, temporal order, route phase, and evidence attribution.Source: §1 · pp. 2–3
  • EgoGears pairs 567 single-video and 1,487 multi-video questions from 126 recordings across 39 routes, including 531 questions requiring alignment across independent recordings.Repeated traversals vary movement speed and illumination while anchoring observations to shared physical environments.Source: §1 · pp. 2–3
  • 33.3% is the strongest multi-video configuration’s score on multi-step turning over whole-video evidence; explicit reasoning gains are larger for spatial relations than trajectory questions.Across eight matched single-video comparisons, reasoning improves every configuration by a median of 7.8 percentage points; gains are 14.0 versus 3.7 points for ego-relative spatial relations and trajectory-grounded questions, respectively.Source: §1 · p. 3
  • The paper frames joint single- and multi-video evaluation as a way to assess whether local understanding supports consistent reasoning across encounters.The evaluation is intended to help distinguish errors in local perception from failures of evidence attribution, correspondence, or reasoning over aligned observations.Source: §1 · pp. 2–3

2 RELATED WORK

Prior egocentric benchmarks cover action, long-video memory, object interaction, and some spatial or cross-video tasks, but route-level reasoning across repeated outdoor encounters remains underexplored. EgoGears targets that gap while evaluating MLLMs whose dynamic-video processing and cross-video reasoning remain challenging.

2 RELATED WORK

  • Existing benchmarks do not directly test whether knowledge remains spatially and temporally coherent when the same physical environment is revisited under changed viewpoints, directions, speeds, or illumination.Prior work includes long-video temporal memory, synchronized multi-view reasoning, and cross-video activity reasoning, but often targets different settings or tasks.Source: §2 · p. 3
  • MLLMs use key-frame selection and token compression for long dynamic videos, yet frame processing can discard landmarks needed for continuous state tracking.The cited discussion also notes challenges from lighting changes and reliance on language priors; comparable evaluations show lower multi-video than single-video accuracy, even with answer format and scoring held fixed.Source: §2 · pp. 3–4

3 EGOGEARS DATASET

EgoGears evaluates first-person outdoor route understanding across local clips and repeated encounters, with question design and review intended to ground answers in specified visual evidence. Its dataset combines varied route recordings with complementary single- and multi-video tasks.

3.1 QA PAIRS DESIGN

  • EgoGears targets local, continuous-temporal, and cross-clip understanding of first-person routes, asking models to connect entities with temporal position, spatial relation, route phase, and motion state.The design focuses on walking, fast walking, and running footage with camera displacement, viewpoint change, and motion-induced instability.Source: §3.1 · p. 5
  • Each question links an observation target and contextual condition to an inferred relation, using evidence windows that jointly cover the asked relation.Closed-set options use near-miss distractors that perturb direction, order, route phase, entity state, or subject-relation assignment; exact-set evaluation requires selecting all and only correct options.Source: §3.1 · p. 5

3.2 COLLECTION METHODOLOGY

  • Recordings cover shared outdoor routes across movement speeds and day/night conditions, and question generation uses clips or recordings to test local understanding and cross-observation relations.Review checks evidence availability, comparison requirements, duplication, and leakage; reviewers construct separate clip timelines for multi-video correspondence, and unresolved questions are removed.Source: §3.2 · p. 6

3.3 DATASET COMPOSITION

  • The collection contains 126 recordings across 39 routes; only 9 routes include all six movement-speed and lighting combinations.Coverage varies: 29 routes have multiple recordings, 21 cover all movement categories, and 12 cover both lighting conditions alongside all movement categories.Source: §3.3 · p. 6
  • EgoGears-Single contains 567 questions grounded in one continuous clip, while EgoGears-Multi contains 1,487 questions, including 531 requiring correspondence across independent recordings.Multi-video questions span 29 fine-grained types and test association of evidence with the correct observation; both subsets use variable answer sets and exact-set scoring.Source: §3.3 · p. 7

4 EXPERIMENTS

Experiments compare single- and multi-video model configurations using exact-set accuracy and test how performance varies across input structures, recording conditions, and reasoning modes. Ordered route understanding remains weak, while reasoning gains are selective and cross-video difficulty depends on the model.

4 EXPERIMENTS

  • The evaluation covers 29 single-video and 31 multi-video configurations, using fixed-denominator exact-set accuracy that counts invalid or unfinished outputs as incorrect.Options are deterministically shuffled and models receive a common structured-answer instruction.Source: §4 · p. 7
  • Input-structure difficulty is model-specific: some configurations score higher on cross-video questions, while Gemini and every Qwen3.5 configuration score higher on multi-caption questions.The ordering persists after chance normalization; the supplied comparison gives InternVL3.5-241B κ = 31.6 versus 10.6 for cross-video questions.Source: §4 · pp. 7–8
  • On the nine fully crossed routes, night-time changes single-video accuracy by −8.0 pp and multi-video accuracy by −0.3 pp; running is harder than walking in both settings.The multi-video night-time effect may partly reflect a floor, and running changes accuracy by −8.1 pp versus fast walking for single-video and −4.8 pp versus slow walking for multi-video.Source: Figure 3 · p. 7, §4 · p. 9
  • Turns and route shape are weak in both settings: the strongest single-video trajectory score is 53.1%, and multi-step turning reaches only 33.3% for Gemini-3.8-Flash.The single-video result cannot be attributed solely to exact multi-select scoring because 91 of 96 trajectory questions are single-choice.Source: §4 · p. 9
  • Reasoning improves all eight matched single-video configurations by a median 7.8 percentage points, with larger gains for ego-relative spatial relations (14.0) than trajectory understanding (3.7).For Qwen3-VL-235B, multi-select accuracy rises from 42.6% to 64.0%, compared with 60.6% to 62.8% on single-choice questions.Source: §4 · p. 9
  • Qualitative cases show reasoning can correct single-video object selection, but cross-encounter success still requires consistent place, direction, and route-phase representations.Reasoning corrects Gemma-4-31B's answer to {B, C, F}, while Qwen3-VL-235B still selects A and omits F.Source: §4 · pp. 9–10, Figure 4 · p. 9

5 CONCLUSION

EgoGears pairs repeated outdoor egocentric traversals with evaluation across encounters. The conclusion identifies evidence binding and ordered route understanding as key bottlenecks, while explicit reasoning helps relation verification more than trajectory reconstruction.

5 CONCLUSION

  • Clip–evidence binding and ordered route understanding remain key bottlenecks for current MLLMs.The benchmark complements local video understanding with reasoning across repeated outdoor encounters.Source: §5 · p. 10
  • Explicit reasoning aids relation verification more than trajectory reconstruction.Source: §5 · p. 10
  • EgoGears evaluates repeated outdoor egocentric traversals alongside reasoning across encounters.Source: §5 · p. 10

A SOCIETAL IMPACT AND LIMITATIONS

The benchmark may inform route-assistance and mobility research, but its data and evaluation have boundaries. The authors also note privacy risks and caution that benchmark accuracy does not establish deployment safety.

A SOCIETAL IMPACT AND LIMITATIONS

  • Public-space clips may expose bystanders, vehicles, or private property, and route matching could in principle reveal where someone moves from shared videos.The authors intend EgoGears for video-understanding research, not surveillance or re-identification; clips are downscaled, audio is removed, and faces and plates are to be blurred.Source: A SOCIETAL IMPACT AND LIMITATIONS · p. 13
  • The recordings come from few participants and locations in one country, limiting representation of other regions, climates, terrains, body types, and recording devices.Only nine routes fully cross speed and lighting conditions; the authors describe that condition analysis as descriptive.Source: A SOCIETAL IMPACT AND LIMITATIONS · p. 13
  • Closed-set offline accuracy does not establish safe real-time use; deployment would require oversight, calibrated uncertainty, and validation in the target environment.The benchmark does not evaluate open-ended answers or streaming interaction.Source: A SOCIETAL IMPACT AND LIMITATIONS · p. 13

B IMPLEMENTATION DETAILS AND MODEL CONFIGURATIONS

The implementation treats each checkpoint and inference mode as a separate configuration, supplies labeled clips without audio, and uses deterministic prompts and parsing. Its scorer counts invalid, incomplete, or truncated outputs as incorrect.

B.1 MODELS AND CONFIGURATIONS

  • A configuration is one checkpoint paired with an inference mode; the leaderboards report 29 single-video and 31 multi-video configurations.The multi-video set includes 3 Gemini API models, 18 open-source base runs, and 10 open-source explicit-reasoning runs.Source: §B.1 · p. 13

B.2 VISUAL INPUT AND PROMPTING

  • Models receive separate, labeled evaluation clips as images without audio, with deterministically shuffled options and a JSON-only answer instruction.The multi-video prompt introduces each independent clip with its label; the single-video prompt states that frames are chronological.Source: §B.2 · pp. 13–14

B.3 INFERENCE AND ANSWER PARSING

  • Inference uses model-specific reasoning parsers, while the scorer accepts only the last post-reasoning JSON answer and counts truncated, unparseable, missing, or incomplete outputs as incorrect.Multi-video runs use the same prompts, visual input, and option shuffling; small score differences among open-source configurations should not be read as capability rankings.Source: §B.3 · p. 14

B.4 METRICS

EgoGears uses exact-set accuracy as its primary metric and supplements it with chance normalization and Jaccard partial credit. Partial credit raises scores but barely changes configuration rankings.

B.4 METRICS

  • Exact-set accuracy requires the selected option set to match the reference exactly, while Jaccard gives partial credit for overlap and penalizes missing or extra options.Jaccard is the mean per-question |P ∩ G|/|P ∪ G|.Source: §B.4 · p. 14
  • On multi-video questions spanning independent recordings, the uniform-random exact-set baseline is 4.47%.Chance-normalized accuracy compares observed exact accuracy with the question-specific random-selection baseline.Source: §B.4 · p. 14
  • Partial credit raises scores by median 9.2 points on single-video and 18.3 points on multi-video questions, while configuration rankings remain highly consistent (Spearman ρ ≥0.98 on both splits).The authors conclude that exact-set accuracy conclusions do not depend on metric strictness.Source: §B.4 · p. 14

B.5 SINGLE- VERSUS MULTI-VIDEO COMPARISON

Across the 20 configurations evaluated comparably, multi-video scores are lower even when answer format and scoring are controlled. Evaluation prompts are fixed across direct-answer and reasoning modes, differing only in how visual input is introduced.

B.5 SINGLE- VERSUS MULTI-VIDEO COMPARISON

  • Every comparable configuration scores lower on multi-video questions under all four answer-format and scoring comparisons.The mean gap falls from 22.5 points to 14.6 points for single-choice questions and 13.3 points under partial credit, but remains positive for every configuration.Source: §B.5 · p. 14, Table 5 · p. 16

B.6 EVALUATION PROMPTS

  • Both splits use fixed templates unchanged between direct-answer and reasoning modes; only the visual-input introduction differs.In multi-video prompts, each clip’s frames are preceded by its anonymized label.Source: §B.6 · p. 15

B.7 EVALUATION TAXONOMIES

The section organizes questions by evidence structure and capability, then describes acquisition, candidate generation, screening, verification, and human review. The released benchmark retains questions after automated and human checks.

B.7 EVALUATION TAXONOMIES

  • Multi-video questions are classified by evidence unit and by capability, with the axes reported separately because some unit–capability intersections are empty or sparse.The unit axis distinguishes questions using one recording from anonymous, cross-video, or cross-recording evidence; capability groups cover seven reasoning areas.Source: §B.7 · p. 17, Table 6 · p. 18

C.1 ACQUISITION AND EVIDENCE PREPARATION

  • Recordings use chest-mounted forward-facing smartphones at 1080p and 30 fps, with repeated traversals targeted across movement and lighting conditions.Realized coverage is partial: 54 recordings are explicitly out-and-back, and 9 routes cover all six movement–lighting combinations; clips are encoded as 480p H.264 without audio, and identities and route metadata are concealed.Source: §C.1 · p. 17

C.2 MODEL-ASSISTED CANDIDATE GENERATION

  • For multi-video candidates, structured observations of route phases, turns, landmarks, events, environmental changes, and spatial relations support questions spanning global, local, and independent-recording evidence.Single-video candidates are generated from 60-second clips with visually grounded captions and structured fields; model-assisted generation is used for both benchmark components.Source: §C.2 · pp. 17–18

C.3 AUTOMATED SCREENING FOR EGOGEARS-MULTI

  • Multi-video screening checks question structure and leakage, shuffles options and clip order, tests text-only guessability, and verifies options against footage without proposed answers.The automated process produces a human-review package of 1,526 questions after evidence-availability and deduplication checks.Source: §C.3 · pp. 18–19

C.4 VERIFICATION AND SAMPLING FOR EGOGEARS-SINGLE

  • Single-video verification labels options against evidence and retains answer-correct candidates for quota-based sampling, yielding 698 questions for human review.Of 6,452 verified candidates, 5,081 were answer correct, 1,364 wrong, and 7 insufficient; sampling used a fixed seed and capped each source recording at 15 questions.Source: §C.4 · pp. 19–20

C.5 HUMAN REVIEW, ADJUDICATION, AND RELEASE

  • Human reviewers inspect the associated clips and assess option support, while adjudication can repair answers, wording, option sets, or evidence problems before renewed checks.The multi-video release contains 1,487 questions after review of 1,526 items; the single-video release retains 567 of 698 reviewed candidates, with modifications re-verified against footage.Source: §C.5 · p. 20

C.6 REPRODUCIBILITY AND VALIDITY CHECKS

The validity checks test score reproducibility and whether adjudication affects a model comparison. They also illustrate cross-recording and whole-video errors involving viewpoint reversal, correspondence, and distant observations.

C.6 REPRODUCIBILITY AND VALIDITY CHECKS

  • Qwen3-VL-8B’s separate-cluster scores were 40.9% versus 40.7% in direct-answer mode and 51.0% versus 52.6% in reasoning mode.The reasoning-mode difference was explained mainly by higher residual truncation (60 versus 38), showing that serving conditions interact with long reasoning outputs.Source: §C.6 · pp. 20–21
  • On 445 human-confirmed questions, Gemini models average 80.9% versus 34.0% across 28 open-source configurations, and Gemini-3.8-Flash leads the strongest open model by 20.5 points.Gemini’s adjudication role contributed an estimated 1.5 points to overall accuracy, and configuration ranking on the human-confirmed subset nearly matches the full set (Spearman ρ = 0.99).Source: §C.6 · p. 21, Table 7 · p. 21
  • Cross-recording comparison requires reversing a travel-direction-dependent relation while preserving building identity; recognizing shared visual content alone does not ensure a consistent coordinate and temporal frame.In the illustrated cases, reasoning corrected Gemma-4-31B but not Qwen3-VL-235B; only the strongest displayed configuration produced the exact set in the second panel.Source: §C.6 · pp. 21–22
  • Whole-video endpoint matching requires linking distant observations and selecting the landmark that establishes the match; model outcomes differ across the illustrated configurations.Qwen3-VL-235B succeeded in both modes, while Gemma-4-31B and Gemini-3.1-Pro selected the same incorrect alternative.Source: §C.6 · p. 22

C.7 ADDITIONAL COMPUTATION CAN RESCUE OR DESTABILIZE PREDICTIONS

Explicit reasoning can rescue or destabilize both single- and multi-video answers, rather than reliably improving them. It helps when evidence is available but correctly bound; it cannot repair missing motion evidence, failed place correspondence, or incorrect route-event segmentation.

C.7 ADDITIONAL COMPUTATION CAN RESCUE OR DESTABILIZE PREDICTIONS

  • Explicit reasoning acts mainly as an evidence-selection and consistency mechanism, not a guaranteed source of new visual information.It helps when needed facts are present but bound to the wrong option, side, or clip; it fails with missing motion evidence, unestablished place correspondence, or unsupported reinterpretation.Source: §C.7 · p. 22
  • Reasoning produces mixed outcomes in single-video cases, correcting some answers while replacing correct answers with unsupported objects or interpretations.Qwen3-VL-235B and two other models are rescued in the examples, while GLM-4.6V regresses in both spatial and trajectory cases.Source: §C.7 · p. 22, Figure 7 · p. 23
  • Multi-video reasoning likewise rescues some answers but regresses or leaves errors unresolved, and it cannot compensate for incorrect route-event segmentation.In one “no turns” case, two models fail in both modes while Qwen3.5-122B is correct in both.Source: §C.7 · p. 22, Figure 8 · p. 24, Figure 9 · p. 24
Loading 2609.37938v1…