PaperlayerView exact revision on arXiv

Source-linked AI summary

EgoTools: Towards Tool-Centric Reasoning in Real-World Egocentric Videos

Shulin Tian, Junsu Kim, Shuai Liu, Hao Li, Yujiao Shen, Sihan Li, Zhe Yang, Yeongon Kim, Feiyu Li, Jialin Wu, Yichi Zhang, Wenhui Wang, Runmao Yao, Yuhao Dong, Zhaoxi Chen, Fangzhou Hong, Antonino Furnari, Jingkang Yang, Hongyuan Zhu, Ziwei Liu

arXiv:2609.39378v1cs.CV

TL;DR

Tool-use understanding requires reasoning about affordances, physical relations, procedure, and causal effects, yet real-world egocentric data and diagnostic benchmarks for this reasoning have been lacking. EgoTools provides both through an annotated corpus and four-track benchmark; fine-tuning on its data improves Qwen3-VL-8B-Instruct accuracy on the full benchmark.

  • Problem

    Current multimodal video models remain limited in egocentric tool-use reasoning, and real-world egocentric data and diagnostic benchmarks that foreground tool-mediated reasoning are scarce.

    Source: §1 · p. 2

  • Method

    EgoTools combines 100 hours of annotated tool-centric egocentric recordings with a 1,000-question benchmark covering four tracks from perception and geometry to procedure and causal reasoning.

    Source: §1 · p. 3, §3 · p. 5

  • Results

    10.9 points: fine-tuning Qwen3-VL-8B-Instruct improves full-benchmark accuracy from 50.0% to 60.9%, with gains on three tracks and Spatial Reasoning the exception.

    Source: §1 · p. 3

  • Takeaways & Limitations

    EgoTools provides actionable supervision for tool-centric reasoning as well as a diagnostic evaluation resource for egocentric tool-use understanding.

    Source: §1 · p. 3, §5 · p. 15

    The EgoPlan-Bench and EgoThink results do not isolate format coverage from other fine-tuning effects because the training subsets mirror those benchmarks’ answer formats.

    Source: §4.3 · p. 12

Abstract

from arXiv · show

Real-world embodied tasks, from everyday activities to professional procedures, require agents to act under physical constraints while tracking evolving object and task states. Tool use sits at the heart of such tasks, as many everyday and professional activities are tool-mediated. Understanding them requires reasoning about affordances, hand-tool-object geometry, procedural progress, and causal effects on target objects. Yet despite strong performance on perception-oriented video tasks such as captioning and general video QA, current multimodal video models remain limited in this form of tool-centric embodied reasoning. Progress in this direction has been limited by the lack of real-world egocentric data and diagnostic benchmarks. To address this gap, we introduce EgoTools, the first comprehensive suite for egocentric tool-use understanding. It consists of two complementary components: EgoTools-Data, a large-scale corpus of 100 hours of tool-centric egocentric recordings with synchronized audio, dense captions, reasoning-heavy narrations, and supplementary 3D information; and EgoTools-Bench, a diagnostic benchmark of 1,000 QA pairs across four tracks that cover tool-use understanding from perception and geometry to procedure and causal reasoning. Experimental results show that current models still struggle to ground tool use in visual evidence: Gemini-3.1-Pro achieves 66.9% overall accuracy but only 51.7% on Perception & Grounding. Beyond evaluation, we validate EgoTools-Data as a training resource. On the full 1,000-question benchmark, full supervised fine-tuning improves Qwen3-VL-8B-Instruct from 50.0% to 60.9%, under strict source-video separation. Together, these results establish EgoTools as a unified resource for both training and diagnostic evaluation of real-world egocentric tool-use understanding.

1. Introduction

Tool-use understanding requires linking visible hands, tools, and objects to contact, spatial coordination, procedural progress, and physical outcomes. EgoTools combines a real-world annotated corpus with a diagnostic benchmark, and fine-tuning on its data improves benchmark accuracy while leaving Spatial Reasoning as an exception.

1. Introduction

  • Understanding tool-mediated activity requires reasoning about tool–target contact, spatial coordination, procedural progress, and actions’ physical outcomes—not only recognizing visible actions and objects.The egocentric viewpoint foregrounds hands, tools, and their immediate consequences as task states evolve.Source: §1 · p. 2
  • Current multimodal video models remain limited in egocentric tool-use reasoning, while existing data and benchmarks rarely foreground tools or connect tool choice to object-state changes and causal outcomes.This leaves both an evaluation gap and a shortage of dense training signals for tool-mediated reasoning.Source: §1 · pp. 2–3
  • EgoTools pairs 100 hours of tool-centric egocentric recordings with a 1,000-pair benchmark spanning affordance and causality, perception and grounding, procedural dynamics, and spatial reasoning.The recordings include synchronized audio, dense captions, tool-centric narrations, and supplementary 3D information.Source: §1 · p. 3
  • Fine-tuning Qwen3-VL-8B-Instruct on EgoTools-derived instruction data raises full-benchmark accuracy from 50.0% to 60.9%, with gains on three tracks except Spatial Reasoning.Training uses non-benchmark videos under strict source-video separation.Source: §1 · p. 3

2. Related Work

Prior egocentric datasets and benchmarks cover activities, objects, interactions, and broader video reasoning, but explicit tool-use reasoning remains under-specified. EgoTools complements these resources with human-centered first-person evidence and a benchmark focused on tool-mediated reasoning.

Egocentric video datasets and benchmarks

  • Existing egocentric resources cover action, object, hand–object, procedural, industrial, and skilled-activity understanding, while many benchmarks organize questions around activities, events, or plans.Instructional-video tasks also examine detours and step differences involving tools, but do not systematically test tool choice, substitutes, or adaptation to changing object states.Source: §2 · pp. 3–4

Physical tool-use reasoning

  • EgoTools-Data pairs egocentric recordings with participant-provided tool-centric narrations and focal-tool grounding, while EgoTools-Bench evaluates complementary aspects of tool-use understanding on held-out videos.The benchmark complements robotics and MLLM/VLM work by using continuous human-centered tasks where tools are selected, substituted, and adapted.Source: §2 · p. 4

Egocentric video datasets and benchmarks

  • Figure 2 describes a 1,000-pair benchmark across four tracks and qualitatively contrasts its tool-, action-, object-, and state-change-focused vocabulary with EPIC-KITCHENS-100 and Ego4D.Table 1’s caption defines its feature marks and scope: embodied interaction benchmarks are contrasts, not directly comparable egocentric video corpora.Source: Table 1 · p. 4, Figure 2 · p. 5

3. EgoTools Data Suite

EgoTools derives a source-video-disjoint training corpus and diagnostic benchmark from a 100-hour egocentric corpus, combining textual and 3D annotations across diverse tool-use tasks. Its benchmark uses human-crafted and spatial QA, quality control, and a distribution that emphasizes reasoning capabilities while remaining long-tailed across domains.

3. EgoTools Data Suite

  • EgoTools comprises a 100-hour real-world corpus and a source-video-separated instruction-tuning corpus and 1,000-question, 8-way benchmark built from a 40.34-hour reserved pool.The benchmark has 900 human-crafted questions and 100 human-verified spatial questions across four tracks.Source: §3 · p. 5

3.1. Data Collection and Preprocessing

  • The corpus spans seven tool-use domains and combines structured task-guided recordings with open-ended participant-driven tasks to capture both consistent task steps and adaptive tool use.Open-ended tasks capture spontaneous tool choices, procedural adaptation, recovery, and substitutions; the benchmark pool is held out from training construction.Source: §3.1 · p. 6

3.2. Multimodal Annotation and Curation

  • Dense captions track actions and progress, while tool-centric narrations foreground tool choice, intent, and target effects; selected mentions receive 2D grounding points for tracking.For 3D annotations, reconstructed depth supports recursive VLM-guided segmentation, whose masks are lifted to 3D for object boxes, trajectories, and spatial QA.Source: §3.2 · p. 7, Figure 4 · p. 7

3.3. Video Instruction-Tuning Data Construction

  • The 184,679-example instruction corpus labels the capability each example supervises, covering grounding, reasoning, and descriptive tasks.69,760 examples directly supervise the four benchmark abilities; other examples include episode-level questions, general video QA, captioning, narration completion, and open-ended QA.Source: §3.3 · p. 8

3.4. Benchmark Construction and Quality Control

  • The benchmark combines 900 human-crafted questions with 100 spatial questions generated from 3D annotations and human-verified, with clips localized around anchor moments.Independent review found 94.4% of sampled items had a uniquely best answer supported by visible evidence; ambiguous items were reviewed and failing items revised or replaced.Source: §3.4 · pp. 8–9

3.5. Distribution

  • The 1,000 benchmark questions are distributed across Affordance & Causality (363), Perception & Grounding (236), Procedural Dynamics (222), and Spatial Reasoning (179).The benchmark is long-tailed across domains, with most questions from Kitchen videos; the authors therefore focus primary analysis on reasoning capabilities rather than domain-level performance.Source: §3.5 · p. 9

4. Experiments

The experiments compare human and model performance on EgoTools-Bench, test EgoTools-Data as training supervision, and analyze how tracks respond to different evidence and model configurations. They find a substantial tool-use reasoning gap, alongside gains from fine-tuning and distinct visual and temporal evidence needs across tracks.

4.1. Experimental Setup

  • Models receive sampled video frames, with audio-enabled models also receiving synchronized non-narration audio; human results use free video seeking and replay.The training evaluation uses a single full-parameter fine-tuning stage on the language-model component, with the visual encoder and multimodal aligner frozen; training videos are disjoint from benchmark videos.Source: §4.1 · p. 10

4.2. Main Results

  • Human experts score 83.2%, compared with 38.9%–50.0% for open-source instruct models; GLM4.1V-Thinking leads open-source thinking models at 50.7%.EgoTools-Bench uses eight answer choices, making chance performance 12.5%.Source: §4.2 · p. 10
  • Qwen3-VL-8B-Instruct scores 50.0% on EgoTools-Bench versus 69.0% on EgoSchema and 61.0% on EgoThink, showing that broad activity understanding does not ensure tool-use reasoning.Across evaluated models, added model or input capacity does not close the gap.Source: §4.2 · p. 11

4.3. Training Utility of EgoTools-Data

  • 60.9% versus 50.0%: fine-tuning Qwen3-VL-8B-Instruct on EgoTools data improved EgoTools-Bench accuracy by 10.9 points under source-video separation.Affordance & Causality, Perception & Grounding, and Procedural Dynamics improve, while Spatial Reasoning decreases by 11.0 points; the authors identify spatial reasoning as a limitation of the training recipe.Source: §4.3 · pp. 11–12, Table 3 · p. 13

4.5. Ablations and Error Analysis

  • 51.7% versus 42.7%: ordered video outperforms text-only input overall, though text-only scores remain above the 12.5% chance level.This supports visual evidence as necessary beyond answer-choice patterns or language priors, while leaving room for residual commonsense or answer-option cues.Source: Table 4 · p. 13, §4.5 · pp. 13–14

4.4. Analysis

  • 51.7%: Gemini-3.1-Pro’s Perception & Grounding score remains 18–23 points below its scores on the other three tracks, marking visual grounding as a bottleneck.Open-source instruct models score 35.6%–45.9% on Perception & Grounding; this track asks about concrete visual details such as contact points and local state changes.Source: §4.4 · pp. 12–13
  • Model variants have heterogeneous track strengths: Qwen3-VL-8B gains most over its 4B counterpart on Procedural Dynamics and Spatial Reasoning, but only 0.5 points on Perception & Grounding.The variants differ in training and sometimes input budgets, so these comparisons do not isolate effects of scale, audio, or deliberation.Source: §4.4 · p. 13

4.5. Ablations and Error Analysis

  • Multi-frame coverage and temporal order contribute differently: middle-frame input costs 4.8 points, while shuffling frames costs 2.4 points overall.Spatial Reasoning benefits most from multi-frame coverage, while Procedural Dynamics is most sensitive to temporal order; Perception & Grounding improves by 13.9 points over text-only input. Figure 6 also shows shared errors, including 0.63 overlap between Qwen3-VL-8B-Instruct and Qwen3-VL-8B-Thinking.Source: §4.5 · pp. 14–15, Figure 6 · p. 15

5. Conclusion

EgoTools reframes egocentric video understanding around how tools mediate action, procedure, causality, and state change. Fine-tuning on EgoTools-derived data improves benchmark accuracy, while the conclusion notes remaining spatial-reasoning and visual-grounding limitations.

5. Conclusion

  • 50.0% to 60.9%: EgoTools-derived supervised fine-tuning improves Qwen3-VL-8B-Instruct, with gains on three tracks while Spatial Reasoning remains a limitation.The conclusion also notes a substantial gap to human performance, particularly for precise visual grounding of tool-mediated interactions.Source: §5 · p. 15
  • EgoTools centers embodied video understanding on physical tools, since recognizing activities alone does not capture how tools mediate action, causality, procedure, and state change.The suite combines real-world recordings, dense hierarchical annotations, long-horizon spatial grounding, and diagnostic QA.Source: §5 · p. 15

A. Contributors and Acknowledgments

  • The authors report contributions spanning data collection, benchmark construction, training, evaluation, annotation, logistics, and project supervision.Shulin Tian, Junsu Kim, Shuai Liu, and Hao Li contributed equally; Ziwei Liu served as corresponding author.Source: A. Contributors and Acknowledgments · p. 21

B. Ethics

  • Participation was voluntary and consent covered applicable collection, annotation, research-use, and release activities, but no formal IRB or institutional ethics approval was sought.The project describes consent, privacy, anonymization, and release safeguards.Source: B. Ethics · p. 21
  • The public release is limited to consent-cleared, anonymized materials; raw fisheye video is withheld, and the research-use license excludes surveillance and participant identification.Some audio and 3D assets are released only when their authorization and privacy review permit it.Source: B. Ethics · p. 22

C.1. Data Collection Settings

EgoTools uses structured task-guided and open-ended participant-driven recordings to balance procedural control with natural variation. Contributor statistics are aggregate, anonymized, and drawn from a documented subset.

C.1. Data Collection Settings

  • Structured task-guided sessions use prepared goals and key steps while leaving participants free to perform manipulations naturally.The design yields coherent procedural boundaries and observable state changes useful for benchmark construction.Source: §C.1 · p. 22
  • Open-ended sessions give participants a broad goal and capture spontaneous tool choice, substitution, recovery, and state-dependent manipulation.Examples include switching tools after an unsuccessful attempt or adapting as a target bends, tears, loosens, or becomes unstable.Source: §C.1 · p. 23

C.2. Anonymized Contributor Statistics

  • Contributor statistics cover a documented subset, and role counts are non-exclusive because contributors may serve multiple roles.The aggregate reporting omits names, institutions, contact details, public profiles, and exact per-video contribution counts.Source: §C.2 · p. 23, Table 6 · p. 23, Table 7 · p. 23

D.1. View Rectification Settings

EgoTools standardizes a rectified first-person view for annotation, training, evaluation, and release, then pairs tool-centric narrations with selected visual grounding. Narration guidance emphasizes tool choice, alternatives, targets, actions, and rationale.

D.1. View Rectification Settings

  • The rectified front-left RGB stream is the canonical egocentric view across annotation, instruction-data construction, benchmark construction, evaluation, and public release.Synchronized views are used to normalize the spatial representation while preserving hand–tool–object interactions and workspace context.Source: §D.1 · p. 24
  • Figure 7 illustrates rectification; its displayed example uses 1024 × 1024 output, 0.78125 zoom, 90.0° field of view, and −5.0° pitch correction.The listed parameters describe the displayed example, not a general setting claim.Source: §D.1 · p. 24, Figure 7 · p. 24

D.2. Narration Annotation Guidelines

  • Narrations target meaningful tool-use decisions and state-dependent behavior rather than every visible action, including tool selection, switching, adaptation, preparation, and recovery.The guidance asks annotators to explain the selected tool, target, action, and rationale, with an alternative when applicable.Source: §D.2 · p. 25
  • Korean and Chinese source narrations are translated and normalized into corrected English text used as the primary narration signal for instruction-data construction.Final benchmark questions and choices are in English and reviewed against timestamped visual evidence.Source: §D.2 · p. 25
  • Selected narration instances receive 2D keyframe grounding points that link mentioned focal tools or target objects to visible instances and support tracking and localization.Grounding annotations are available only for selected instances and are released progressively as finalization, authorization, and privacy review proceed.Source: §D.2 · p. 26

E.1. Dataset and Benchmark Statistics

EgoTools-Data provides a large, multi-domain egocentric corpus, while EgoTools-Bench reserves source videos for evaluation and tests four tool-use reasoning tracks. The release and benchmark have explicit scope boundaries, and quality control repairs valid annotations before discarding them.

E.1. Dataset and Benchmark Statistics

  • EgoTools contains 646 videos totaling 100.37 hours across seven tool-use domains, with 361,332 hierarchical captions and 6,519 tool-centric narrations.A 40.34-hour subset of high-quality, tool-dense source videos is reserved for benchmark construction and evaluation.Source: §E.1 · p. 26, Table 8 · p. 26
  • The benchmark is dominated by Kitchen questions and is not domain-balanced, so small domain slices are not used for primary performance claims.Its construction prioritizes complementary reasoning capabilities and challenging tool-mediated interactions rather than statistical balance across collection domains.Source: §E.1 · p. 27, Table 9 · p. 27
  • EgoTools-Bench contains 1,000 8-way QA pairs—900 human-crafted and 100 human-verified spatial questions—across four reasoning tracks.The tracks include 363 Affordance & Causality, 236 Perception & Grounding, 222 Procedural Dynamics, and 179 Spatial Reasoning questions.Source: §E.1 · p. 27

E.2. Release and Privacy Considerations

  • The public release provides rectified video and cleared annotations, but excludes raw fisheye streams; grounding, audio, and 3D assets may vary by example.Those components require annotation finalization, release authorization, and privacy review.Source: §E.2 · pp. 27–28
  • EgoTools does not exhaustively cover all tools, environments, participant populations, cultural practices, or professional procedures.The benchmark’s Kitchen concentration should not be treated as representative of the full corpus’s domain distribution.Source: §E.2 · p. 28

F.1. Fix-First Quality Control

  • Fix-first quality control audits structural defects, repairs clear and repairable issues, and sends cases with uncertain meaning or answer validity to human review.The structural audit uses 22 binary flags; content-empty submissions and verbatim duplicates are removed.Source: §F.1 · p. 29

F.2. Eight-Choice Normalization and Anti-Shortcut Checks

The benchmark standardizes eight-choice items and checks for answer shortcuts and length bias; related work distinguishes EgoTools-Bench by its focus on physical tool-use reasoning in natural first-person videos.

F.2. Eight-Choice Normalization and Anti-Shortcut Checks

  • Items are normalized to eight options by distractor count, then checked for shortcut patterns, answer-position imbalance, and answer-length bias.Failed items go to bounded regeneration or human review.Source: §F.2 · p. 29
  • After repair, every item satisfies |z| ≤1.5, and mean standardized deviation shifts from cached +2.21 to −0.15.Items with |z| > 1.5 are flagged for repair.Source: §F.2 · p. 30
  • Final options are deterministically shuffled by QA identifier, reducing positional priors while preserving reproducible evaluation and a 12.5% chance baseline.Source: §F.2 · p. 30

G. Additional Related Work

H.1. Single-Stage Training Procedure

The procedure fine-tunes Qwen3-VL-8B-Instruct on EgoTools-derived examples while freezing its visual modules, with benchmark media excluded from training.

H.1. Single-Stage Training Procedure

  • The 184,679-example training mixture includes 170,595 multiple-choice, 5,000 short open-ended, and 9,084 captioning or narration-completion examples.Next-action answer letters are balanced across four positions at 1,250 examples each.Source: §H.1 · p. 31
  • Training uses no benchmark media or QA annotations, and media overlap with the benchmark reference set is zero, although some question strings match generic templates.Source: §H.1 · p. 31
  • “Full SFT” updates all parameters of Qwen3-VL-8B-Instruct’s language-model component while keeping its vision encoder and multimodal projector frozen.Source: §H.1 · p. 31
  • Training runs for one epoch with AdamW at 2.3 × 10−6, effective batch size 128, and 1,443 optimizer steps; each video uses 64 uniformly sampled frames.The language-model component is trained with bfloat16 precision and gradient checkpointing.Source: §H.1 · p. 31

H.2. Training Data Composition

The training-data section reports a 184,679-example mixture and identifies the capability-level breakdown in Table 10, followed by protocols for evaluating three comparison benchmarks.

H.2. Training Data Composition

  • The 184,679-example training corpus is summarized by supervised capability in Table 10.The table reports mixture shares rounded to one decimal place.Source: §H.2 · p. 32, Table 10 · p. 32

H.3. Evaluation Protocols

  • Evaluation uses all 5,031 EgoSchema questions, the 3,343-question EgoPlan-Bench validation set, and macro-averaged accuracy over EgoThink’s twelve single-image subtasks.EgoThink subtasks are judged by gpt-4o-2024-11-20.Source: §H.3 · p. 32
Loading 2609.39378v1…