Source-linked AI summary
Ego-R1: Chain-of-Tool-Thought for Ultra-Long Egocentric Video Reasoning
Shulin Tian, Ruiqi Wang, Hongming Guo, Penghao Wu, Yuhao Dong, Xiuying Wang, Jingkang Yang, Hao Zhang, Hongyuan Zhu, Ziwei Liu
TL;DR
Ultra-long egocentric videos require reasoning across extended temporal, multimodal contexts. Ego-R1 addresses this with Chain-of-Tool-Thought, dynamic tool use, and two-stage SFT/RL training, achieving strong results on week-long and other long-video benchmarks.
Problem
Ultra-long egocentric videos demand long-horizon reasoning across diverse temporal contexts, modalities, and dependencies.
Method
Ego-R1 combines Chain-of-Tool-Thought with dynamic specialized-tool invocation and two-stage supervised fine-tuning and reinforcement learning.
Results
Ego-R1 effectively handles ultra-long egocentric videos and extends demonstrated temporal coverage from a few hours to a week.
Takeaways & Limitations
Modular, tool-grounded reasoning provides a foundation for video understanding across extended temporal coverage.
Abstract
from arXiv · showhide
We introduce Ego-R1, a novel framework for reasoning over ultra-long (i.e., in days and weeks) egocentric videos, which leverages a structured Chain-of-Tool-Thought (CoTT) process, orchestrated by an Ego-R1 Agent trained via reinforcement learning (RL). Inspired by human problem-solving strategies, CoTT decomposes complex reasoning into modular steps, with the RL agent invoking specific tools, one per step, to iteratively and collaboratively answer sub-questions tackling such tasks as temporal retrieval and multi-modal understanding. We design a two-stage training paradigm involving supervised finetuning (SFT) of a pretrained language model using CoTT data and RL to enable our agent to dynamically propose step-by-step tools for long-range reasoning. To facilitate training, we construct a dataset called Ego-R1 Data, which consists of Ego-CoTT-25K for SFT and Ego-QA-4.4K for RL. Furthermore, our Ego-R1 agent is evaluated on a newly curated week-long video QA benchmark, Ego-R1 Bench, which contains human-verified QA pairs from hybrid sources. Extensive results demonstrate that the dynamic, tool-augmented chain-of-thought reasoning by our Ego-R1 Agent can effectively tackle the unique challenges of understanding ultra-long egocentric videos, significantly extending the time coverage from few hours to a week.
1 Introduction
Ultra-long egocentric videos create challenges for multimodal, long-horizon reasoning, motivating Ego-R1’s dynamic tool-driven framework, training data, and week-long evaluation benchmark.
- Egocentric videos span hours to weeks and require reasoning across diverse temporal contexts, modalities, and long-range dependencies.
- Prior long-context methods remain computationally difficult at day-long durations, while compression and sampling can miss key events.
- Ego-R1 uses fine-tuned LLMs and reinforcement learning for dynamic, tool-driven reasoning over ultra-long egocentric videos.
- Ego-R1 Data provides 25K CoTT reasoning traces and 4.4K QA instances for supervised fine-tuning and reinforcement learning, respectively.
- Extensive experiments show that Ego-R1 extends effective temporal coverage from a few hours to a week on ultra-long egocentric videos.
2 Related Work
Related work establishes broad egocentric benchmarks and multimodal tool-use methods, but existing approaches remain limited for long-duration, deeply reasoned video understanding.
- Existing egocentric benchmarks cover temporal understanding, object interactions, and episodic memory, typically over minutes, with some extensions reaching hours.
- Week-long EgoLife provides longer videos, but its question-answering tasks lack requirements for deep visual reasoning.
- Existing long-video approaches process unified inputs, sample frames without guaranteed relevance, or analyze frames in isolation.
- Tool-integrated reasoning and reinforcement learning have been explored as scalable approaches for improving language-model tool use, including in multimodal domains.
- Existing datasets lack adequate high-quality Chain-of-Thought annotations for long-video understanding, motivating Ego-CoTT-25K.
3 Egocentric Long Video Reasoning via Dynamic Tool-Calling
Ego-R1 treats egocentric long-video reasoning as linking dispersed evidence across extended timelines, using dynamic tool calling to combine retrieval, local video analysis, and frame-level perception.
- Egocentric Long Video Reasoning: Long-range egocentric reasoning links actions separated by hours or days through persistent strategies, habits, and dispersed cues.
- Egocentric Long Video Reasoning Tasks: The task supplies several days of video and a query time, requiring analysis of all preceding content before answering.
- Dynamic Tool-Calling: Ego-R1 empowers an LLM to invoke specialized perception tools on demand instead of placing the entire long context into an MLLM.
- Dynamic Tool-Calling: H-RAG segments videos into 30-second clips, aligns captions with ASR logs, and hierarchically aggregates summaries for temporal retrieval.
- Dynamic Tool-Calling: The Video-LLM analyzes local windows from a few seconds to ten minutes, while the VLM extracts fine-grained frame-level details.
4 Ego-R1 Data: Chain-of-Tool-Thought (CoTT) for Video Reasoning
Ego-R1 Data trains dynamic Chain-of-Tool-Thought reasoning by representing each step as thought, tool call, and observation, supported by specialized tools and generated traces.
- Ego-R1 Data enables agentic tool use by conditioning dynamic tool selection on current observations and past actions.
- Chain-of-Tool-Thought (CoTT): Each CoTT trajectory contains sequential reasoning steps in which the agent generates thoughts and tool calls based on the query and previous observations.
- Chain-of-Tool-Thought (CoTT): The action space includes H-RAG for long-range retrieval, Video-LLM for short-range understanding, and VLM for framewise image understanding.
- Chain-of-Tool-Thought (CoTT): The observation space collects outputs from RAG, Video-LLM, and VLM tools, which guide subsequent reasoning steps.
- Data Generation: The dataset contains 4.4K QA pairs, 2.9K selected questions, 25K CoTT traces, and an average of 7.42 tool calls per task.
- Data Generation: Generated tool calls include fully specified arguments, are pre-verified for syntax, executed externally, and returned as observations for multi-turn use.
5 Ego-R1 Agent: Towards Tools Integrated Video Understanding Agent
Ego-R1 trains a language model for long-form video reasoning through structured, multi-turn tool calls. Its two-stage strategy combines CoTT-based supervised fine-tuning with reinforcement learning that optimizes iterative tool use and reasoning.
- Ego-R1 uses a two-stage strategy to train a model that invokes multi-turn tools for long-form video reasoning.The framework is designed for structured long-chain reasoning in which tool calls collaboratively solve complex problems.
- Stage 1: Supervised fine-tuning (SFT): Stage 1 uses supervised fine-tuning on synthetic CoTT data to establish correctly formatted tool calls.CoTT presents stepwise interactions as reasoning followed by either a tool call or an answer.
- Stage 2: Reinforcement learning (RL): Stage 2 applies Gradient-Regularized Policy Optimization to improve multi-turn tool-calling capabilities.GRPO maximizes expected final task reward while regularizing policy-gradient variance across reasoning steps.
- Stage 2: Reinforcement learning (RL): The policy conditions each reasoning-token decision on the question, current observation, and preceding sequence while KL regularization limits drift from the original model.The objective also uses standardized group-based advantage estimates to guide optimization.
- Stage 2: Reinforcement learning (RL): Training generates rollout trajectories by sequentially executing tools, ending when a valid answer is produced or the maximum step limit is reached.These stepwise observations support generalization to multi-turn tool usage and produce the final Ego-R1 Agent.
- Table 2 compares Ego-R1 with video-understanding models across multiple QA benchmarks, using bold and underlined values to mark the best and second-best results.The evaluation caption states that a clean EgoLifeQA subset was retained because some QA pairs supported CoTT generation and training.
6 Experiments
Experiments evaluate Ego-R1 across long-video benchmarks and training regimes, showing strong performance across exocentric and egocentric settings and the importance of CoTT supervision.
- Ego-R1 achieves the best or second-best score on three of four datasets despite using far fewer parameters than most competitors.
- Exocentric setting: 64.9% accuracy on VideoMME (long) is the highest among open-weight models and second overall, behind Gemini-1.5-Pro at 67.4%.
- Egocentric settings: 46.0% accuracy on Ego-R1 Bench exceeds Gemini-1.5-Pro by 7.7% and LLaVA-Video by 17.0% on 44.3-hour egocentric videos.
- Analysis: Agent-based approaches remain in the 32–36% range on Ego-R1 Bench, below Ego-R1’s 46%, while CoTT enables long-horizon reasoning with a compact 3B model.
- Ablation Study: Vanilla RL without intermediate CoTT supervision reduces task accuracy to 0%, whereas CoTT SFT significantly improves both task and tool-call format accuracy.
7 Conclusion and Outlook
The paper concludes that modular, tool-grounded reasoning supports ultra-long egocentric video understanding, using hierarchical retrieval and adaptable visual tools. It also points toward broader applications of transparent multimodal reasoning.
- Ego-R1’s Chain-of-Tool-Thought decomposes long-horizon video reasoning into modular, tool-grounded steps.
- Outlook: The conclusion suggests hybrid symbolic-neural architectures may be important for open-world, ultra-long video understanding and transparent human-AI collaboration.
- Tool Use: The framework integrates a text-based tool and two visual-based tools for distinct temporal and perceptual dimensions of egocentric understanding.
- Hierarchical RAG: Hierarchical RAG builds summaries from 30-second clips through day-level summaries and retrieves information top-down from coarse to fine temporal levels.
- Hierarchical RAG: The agent selects an initial retrieval level and whether to continue to finer granularities based on returned summaries.
- Tool Use: The framework is modular because its visual perception components can integrate a range of state-of-the-art visual understanding models.
B.2 Data Generation
The paper uses AutoGen with GPT-4.1 to systematically generate Chain-of-Tool-Thought training data through structured tool interactions.
- GPT-4.1 and AutoGen are used to systematically generate CoTT data with structured message passing and standardized function-calling interfaces.
B.3 RAG Construction for Other Benchmarks
RAG construction is adapted to each benchmark’s temporal span and question setting, using hierarchical levels for longer videos and simpler indexing for shorter ones.
- RAG indexing granularity is adjusted proportionally to each benchmark’s average question timespan.
- EgoLifeQA: EgoLifeQA uses week → day → hour → 10-minute retrieval levels for its 44.3-hour setting.
- VideoMME: VideoMME long videos use 30-second clips aggregated into 10-minute summaries, producing a two-level 10-minute → 30-second hierarchy.
- EgoSchema: EgoSchema uses 30-second clips summarized from 1 FPS frames, without additional hierarchical summarization because its questions cover limited timespans.
- Tool-use ablation: Using only RAG degrades performance, supporting the combination of retrieval and visual tools for long-horizon reasoning.
C.1 Ablation Studies
Table 4 ablates tool-use configurations in Ego-R1, including the choice of video_llm module. Replacing LLaVA-Video with Gemini-1.5-Pro improves Ego-R1 Bench performance from 43.7% to 46.0%.
- Table 4 evaluates how different tool-use configurations affect Ego-R1.The ablation includes comparisons between different video_llm modules.
- 43.7% to 46.0%: replacing LLaVA-Video with Gemini-1.5-Pro improves Ego-R1 Bench performance.The passage attributes this improvement to stronger localized visual understanding.
- The ablation indicates that the video_llm module is an important tool-use configuration choice.
C.2 Qualitative Results
Ego-R1 presents explicit step-by-step reasoning through dynamic tool use and structured CoTT outputs. Qualitative comparisons show more detailed reasoning than Video-R1 in successful cases, while also exposing a temporal-range adjustment failure.
- C.2 Qualitative Results: Structured reasoning chains expose how collaborative tool use contributes to performance on complex video reasoning tasks.The visual demonstrations present explicit CoTT outputs as an interpretable reasoning process.
- C.2 Qualitative Results: Ego-R1 produces more detailed and interpretable step-by-step reasoning chains than Video-R1 in successful cases.The comparison attributes these chains to dynamic tool-calling and explicit CoTT outputs.
- C.2 Qualitative Results: Ego-R1 can fail when a later tool call does not adjust the temporal range after relevant information is found.In the reported failure, Step 1 identified information near DAY2_15500000, but the subsequent call did not refine the range.
- C.2 Qualitative Results: Tool-call functions in AutoGen define how the agent uses tools, while training syntax guides structured reasoning steps and valid calls.The prompts align tool usage with the Chain-of-Tool-Thought process.
- C.2 Qualitative Results: The training prompt requires internal reasoning before tool calls and uses structured tags for thoughts, tool queries, returned information, and answers.The specified tools are rag, video_llm, and vlm, with tool information returned between designated tags.
- C.2 Qualitative Results: The system prompt provides a concrete RAG query format using keywords, start_time, and query_time fields.The example includes screwdriver and applause as keywords and specifies DAY1 time boundaries.