Ego-R1: Chain-of-Tool-Thought for Ultra-Long Egocentric Video Reasoning

• Egocentric videos span hours to weeks and require reasoning across diverse temporal contexts, modalities, and long-range dependencies.Source: §1 · p. 2
• Prior long-context methods remain computationally difficult at day-long durations, while compression and sampling can miss key events.Source: §1 · p. 2
• Ego-R1 uses fine-tuned LLMs and reinforcement learning for dynamic, tool-driven reasoning over ultra-long egocentric videos.Source: §1 · p. 2
• Ego-R1 Data provides 25K CoTT reasoning traces and 4.4K QA instances for supervised fine-tuning and reinforcement learning, respectively.Source: §1 · p. 2
• Extensive experiments show that Ego-R1 extends effective temporal coverage from a few hours to a week on ultra-long egocentric videos.Source: §1 · p. 2

• Existing egocentric benchmarks cover temporal understanding, object interactions, and episodic memory, typically over minutes, with some extensions reaching hours.Source: §2 · pp. 2–3
• Week-long EgoLife provides longer videos, but its question-answering tasks lack requirements for deep visual reasoning.Source: §2 · p. 3
• Existing long-video approaches process unified inputs, sample frames without guaranteed relevance, or analyze frames in isolation.Source: §2 · p. 3
• Tool-integrated reasoning and reinforcement learning have been explored as scalable approaches for improving language-model tool use, including in multimodal domains.Source: §2 · p. 3
• Existing datasets lack adequate high-quality Chain-of-Thought annotations for long-video understanding, motivating Ego-CoTT-25K.Source: §2 · p. 3

Egocentric Long Video Reasoning

• Long-range egocentric reasoning links actions separated by hours or days through persistent strategies, habits, and dispersed cues.Source: §3 · p. 3

Egocentric Long Video Reasoning Tasks

• The task supplies several days of video and a query time, requiring analysis of all preceding content before answering.Source: §3.1 · p. 4

Dynamic Tool-Calling

• Ego-R1 empowers an LLM to invoke specialized perception tools on demand instead of placing the entire long context into an MLLM.Source: §3.2 · p. 4
• H-RAG segments videos into 30-second clips, aligns captions with ASR logs, and hierarchically aggregates summaries for temporal retrieval.Source: §3.2 · p. 4
• The Video-LLM analyzes local windows from a few seconds to ten minutes, while the VLM extracts fine-grained frame-level details.Source: §3.2 · pp. 4–5

• Ego-R1 Data enables agentic tool use by conditioning dynamic tool selection on current observations and past actions.Source: §4 · p. 5

Chain-of-Tool-Thought (CoTT)

• Each CoTT trajectory contains sequential reasoning steps in which the agent generates thoughts and tool calls based on the query and previous observations.Source: §4.1 · p. 5
• The action space includes H-RAG for long-range retrieval, Video-LLM for short-range understanding, and VLM for framewise image understanding.Source: §4.1 · pp. 5–6
• The observation space collects outputs from RAG, Video-LLM, and VLM tools, which guide subsequent reasoning steps.Source: §4.1 · p. 6

Data Generation

• The dataset contains 4.4K QA pairs, 2.9K selected questions, 25K CoTT traces, and an average of 7.42 tool calls per task.Source: §4.2 · p. 6
• Generated tool calls include fully specified arguments, are pre-verified for syntax, executed externally, and returned as observations for multi-turn use.Source: §4.2 · p. 6

• Ego-R1 uses a two-stage strategy to train a model that invokes multi-turn tools for long-form video reasoning.The framework is designed for structured long-chain reasoning in which tool calls collaboratively solve complex problems.Source: §5 · p. 6, Figure 3 · p. 7

Stage 1: Supervised fine-tuning (SFT)

• Stage 1 uses supervised fine-tuning on synthetic CoTT data to establish correctly formatted tool calls.CoTT presents stepwise interactions as reasoning followed by either a tool call or an answer.Source: §5.1 · pp. 6–7

Stage 2: Reinforcement learning (RL)

• Stage 2 applies Gradient-Regularized Policy Optimization to improve multi-turn tool-calling capabilities.GRPO maximizes expected final task reward while regularizing policy-gradient variance across reasoning steps.Source: §5.2 · p. 7
• The policy conditions each reasoning-token decision on the question, current observation, and preceding sequence while KL regularization limits drift from the original model.The objective also uses standardized group-based advantage estimates to guide optimization.Source: §5.2 · p. 7
• Training generates rollout trajectories by sequentially executing tools, ending when a valid answer is produced or the maximum step limit is reached.These stepwise observations support generalization to multi-turn tool usage and produce the final Ego-R1 Agent.Source: §5.2 · p. 7, Figure 3 · p. 7

• Table 2 compares Ego-R1 with video-understanding models across multiple QA benchmarks, using bold and underlined values to mark the best and second-best results.The evaluation caption states that a clean EgoLifeQA subset was retained because some QA pairs supported CoTT generation and training.Source: Table 2 · p. 8

• Ego-R1 achieves the best or second-best score on three of four datasets despite using far fewer parameters than most competitors.Source: §6.2 · p. 9

Exocentric setting

• 64.9% accuracy on VideoMME (long) is the highest among open-weight models and second overall, behind Gemini-1.5-Pro at 67.4%.Source: §6.2 · p. 9

Egocentric settings

• 46.0% accuracy on Ego-R1 Bench exceeds Gemini-1.5-Pro by 7.7% and LLaVA-Video by 17.0% on 44.3-hour egocentric videos.Source: §6.2 · p. 9

Analysis

• Agent-based approaches remain in the 32–36% range on Ego-R1 Bench, below Ego-R1’s 46%, while CoTT enables long-horizon reasoning with a compact 3B model.Source: §6.2 · p. 9

Ablation Study

• Vanilla RL without intermediate CoTT supervision reduces task accuracy to 0%, whereas CoTT SFT significantly improves both task and tool-call format accuracy.Source: §6.3 · p. 9

• Ego-R1’s Chain-of-Tool-Thought decomposes long-horizon video reasoning into modular, tool-grounded steps.Source: §7 · p. 9

Outlook

• The conclusion suggests hybrid symbolic-neural architectures may be important for open-world, ultra-long video understanding and transparent human-AI collaboration.Source: §7 · p. 10

Tool Use

• The framework integrates a text-based tool and two visual-based tools for distinct temporal and perceptual dimensions of egocentric understanding.Source: A Hierarchical RAG · p. 17

Hierarchical RAG

• Hierarchical RAG builds summaries from 30-second clips through day-level summaries and retrieves information top-down from coarse to fine temporal levels.Source: A Hierarchical RAG · p. 17, Figure 4 · p. 18
• The agent selects an initial retrieval level and whether to continue to finer granularities based on returned summaries.Source: A Hierarchical RAG · p. 17

Tool Use

• The framework is modular because its visual perception components can integrate a range of state-of-the-art visual understanding models.Source: §B.1 · p. 17

• GPT-4.1 and AutoGen are used to systematically generate CoTT data with structured message passing and standardized function-calling interfaces.Source: §B.2 · p. 18

• RAG indexing granularity is adjusted proportionally to each benchmark’s average question timespan.Source: §B.3 · p. 18

EgoLifeQA

• EgoLifeQA uses week → day → hour → 10-minute retrieval levels for its 44.3-hour setting.Source: §B.3 · p. 18

VideoMME

• VideoMME long videos use 30-second clips aggregated into 10-minute summaries, producing a two-level 10-minute → 30-second hierarchy.Source: §B.3 · p. 18

EgoSchema

• EgoSchema uses 30-second clips summarized from 1 FPS frames, without additional hierarchical summarization because its questions cover limited timespans.Source: §B.3 · pp. 18–19

Tool-use ablation

• Using only RAG degrades performance, supporting the combination of retrieval and visual tools for long-horizon reasoning.Source: Table 4 · p. 19

• Table 4 evaluates how different tool-use configurations affect Ego-R1.The ablation includes comparisons between different video_llm modules.Source: §C.1 · p. 19
• 43.7% to 46.0%: replacing LLaVA-Video with Gemini-1.5-Pro improves Ego-R1 Bench performance.The passage attributes this improvement to stronger localized visual understanding.Source: §C.1 · p. 19
• The ablation indicates that the video_llm module is an important tool-use configuration choice.Source: §C.1 · p. 19

C.2 Qualitative Results

• Structured reasoning chains expose how collaborative tool use contributes to performance on complex video reasoning tasks.The visual demonstrations present explicit CoTT outputs as an interpretable reasoning process.Source: §C.2 · p. 19
• Ego-R1 produces more detailed and interpretable step-by-step reasoning chains than Video-R1 in successful cases.The comparison attributes these chains to dynamic tool-calling and explicit CoTT outputs.Source: §C.2 · p. 19, Figure 5 · p. 20
• Ego-R1 can fail when a later tool call does not adjust the temporal range after relevant information is found.In the reported failure, Step 1 identified information near DAY2_15500000, but the subsequent call did not refine the range.Source: Figure 5 · p. 20
• Tool-call functions in AutoGen define how the agent uses tools, while training syntax guides structured reasoning steps and valid calls.The prompts align tool usage with the Chain-of-Tool-Thought process.Source: Table 5 · p. 21, Table 6 · p. 22
• The training prompt requires internal reasoning before tool calls and uses structured tags for thoughts, tool queries, returned information, and answers.The specified tools are rag, video_llm, and vlm, with tool information returned between designated tags.Source: D Future Works · p. 22
• The system prompt provides a concrete RAG query format using keywords, start_time, and query_time fields.The example includes screwdriver and applause as keywords and specifies DAY1 time boundaries.Source: D Future Works · p. 22