PaperlayerView exact revision on arXiv

Source-linked AI summary

InternVideo3: Agentify Foundation Models with Multimodal Contextual Reasoning

Ziang Yan, Sheng Xia, Jiashuo Yu, Yue Wu, Tianxiang Jiang, Songze Li, Kanghui Tian, Yicheng Xu, Yinan He, Kai Chen, Limin Wang, Yu Qiao, Yi Wang

arXiv:2606.12195v1cs.CV

TL;DR

Long-horizon multimodal video reasoning remains less developed than text-dominant agentic AI, despite requiring sustained temporal understanding and iterative evidence use. InternVideo3 addresses this gap with closed-loop Multimodal Contextual Reasoning and efficient context handling, achieving strong results across video reasoning benchmarks, especially long-horizon tasks.

  • Problem

    Open multimodal agents remain less developed for long-horizon video tasks requiring sustained temporal understanding, iterative interaction, and evidence grounding.

  • Method

    InternVideo3 uses evolving-context Multimodal Contextual Reasoning with tool use and memory, plus M2LA to compress KV-cache states while preserving multimodal tokens.

  • Results

    InternVideo3 performs strongly across short-video, long-video, and spatiotemporal benchmarks, with especially notable gains on long-horizon tasks.

  • Takeaways & Limitations

    The results suggest that effective long-horizon multimodal agency requires using context as an evolving reasoning substrate, not merely adding more frames or scaling static QA.

  • Takeaways & Limitations

    Newer open-weight multimodal models surpass InternVideo3 on some benchmarks, so the paper does not claim state-of-the-art overall multimodal agency or frontier open-model performance.

Abstract

from arXiv · show

Recent progress in foundation models has shifted toward agentic behavior involving multi-step reasoning and tool use. However, open-source efforts largely focus on text-dominant settings, leaving long-horizon multimodal tasks underexplored. This gap is evident in video tasks requiring sustained temporal understanding and iterative interaction. We present InternVideo3, a framework enhancing these capabilities via Multimodal Contextual Reasoning (MCR). MCR treats understanding as a closed-loop process over a shared, evolving context containing observations, instructions, reasoning, tool actions, and memory. This frames long-video understanding as evidence accumulation and verification. To ensure efficiency, we introduce Multimodal Multi-head Latent Attention (M^2LA), a token-preserving reparameterization compressing KV-cache states while retaining the full token stream. Our staged training includes continued pretraining, short-to-long supervised fine-tuning, rule-based reinforcement learning, and on-policy distillation. Experiments show InternVideo3 achieves strong performance on benchmarks like Video-MME, MLVU, and EgoSchema. We further instantiate the model as a video agent with retrieval tools, demonstrating robust evidence-grounded behavior. Our results suggest that efficient context handling and closed-loop reasoning are vital for adapting open multimodal models toward long-horizon visually grounded agency.

1. Introduction

InternVideo3 addresses the limited open-source progress on visually grounded long-horizon reasoning by framing video understanding as recursive multimodal contextual reasoning. It combines an efficient attention reparameterization and staged training with strong benchmark results and a retrieval-enabled video-agent instantiation.

  • Multimodal Contextual Reasoning (MCR) represents observations, instructions, reasoning, tool actions, feedback, and memory in a shared evolving context.This formulation turns long-video understanding into a recursive process rather than a one-shot video-to-answer mapping.
  • M2LA compresses KV-cache states while preserving the full multimodal token stream, enabling longer multimodal rollouts under constrained hardware budgets.The method targets the growing cache required to retain observations, reasoning traces, tool outputs, and memory across long decoding processes.
  • The staged training pipeline combines continued pretraining, short-to-long long-video supervised fine-tuning, rule-based reinforcement learning, and on-policy distillation.The recipe is designed to strengthen reasoning over dense visual evidence and extended temporal dependencies without requiring frontier-scale base models.
  • InternVideo3 achieves strong results on short-video, long-video, and spatiotemporal reasoning benchmarks, with notable gains on Video-MME, MLVU, and EgoSchema.A retrieval- and verification-enabled video-agent instantiation illustrates more robust evidence-grounded behavior.

2. Related Work

Related work spans multimodal language models for video understanding, agentic tool use, world models, long-context modeling, and reasoning-oriented interaction. Despite progress, long-video tasks still challenge sustained temporal reasoning and require systems to decide what evidence to gather and when it is sufficient.

  • Multimodal Large Language Models: Multimodal language models have expanded from image-text alignment and image understanding to general video understanding and multimodal reasoning.Recent open-source video MLLMs perform strongly on standard short-to-medium video benchmarks, but performance degrades on genuinely long videos or tasks requiring sustained temporal reasoning.
  • Agents: Agent research extends interleaved reasoning and action to program synthesis, tool use, GUI interaction, mobile interfaces, and multimodal environments.In multimodal settings, agents must identify missing perceptual evidence, choose where to look next, and judge whether current evidence is sufficient to answer or act.
  • World Models and Predictive Video Learning: World models learn compact internal state representations supporting prediction, planning, and decision making in partially observed environments.This line includes latent dynamics models for reinforcement learning, action-conditioned predictive models, generative simulators, and self-supervised predictive representation learning.
  • Context Modeling and Engineering: Long multimodal sequence methods use sparse sampling, hierarchical temporal modeling, retrieval, summarization, memory tokens, and context compression.Systems including MovieChat, LongVU, and Gemini 1.5 employ retrieval, summarization, or compression for long videos and documents.
  • Reasoning, Reflection, and Test-Time Interaction: Reasoning methods combine prompting, reflection, process supervision, reinforcement learning, and test-time scaling, while multimodal systems couple reasoning with perception.Visual search and grounding methods indicate that reasoning can guide which evidence to inspect within an image or video.
  • Reasoning, Reflection, and Test-Time Interaction: This work integrates reasoning, perception updates, tool feedback, and memory into one evolving multimodal context for closed-loop evidence gathering and belief updates.The formulation treats deciding whether more evidence is needed and selecting the next perception or tool operation as part of the same process.

3. Method

InternVideo3 improves long-horizon multimodal reasoning through three complementary components: Multimodal Contextual Reasoning, efficient attention reparameterization, and staged training. Its central formulation treats understanding as a closed-loop process of observation, reasoning, action, feedback, and contextual updating.

  • Multimodal Contextual Reasoning: Multimodal Contextual Reasoning represents observations, intermediate reasoning, tool actions, feedback, and memory within a shared evolving context.It reframes long-horizon understanding as an iterative process rather than a one-pass video-to-answer mapping.
  • Efficient Attention: M2LA is an efficient attention reparameterization that reduces KV-cache cost for long multimodal rollouts.The design makes repeated long-context rollouts practical while preserving the framework’s evolving contextual state.
  • Staged Training: The staged training recipe restores pretrained capability after attention conversion and then specializes the model for long-horizon multimodal reasoning.Training is presented as the third complementary component of the method.

3.1. Multimodal Contextual Reasoning

Multimodal Contextual Reasoning (MCR) models long-horizon multimodal understanding as an evolving, shared context that accumulates evidence through reasoning, tool interaction, memory, and verification. It implements this as a closed-loop rollout that repeatedly updates visual evidence, actions, feedback, and context until producing an answer or termination action.

  • Multimodal Contextual Reasoning: MCR represents observations, instructions, reasoning traces, tool actions, external feedback, and memory within a shared context that grows over time.The context records what the model observed, inferred, and acted upon, as well as remaining uncertainty.
  • Closed-Loop Rollout: At each rollout step, the model conditions on current visual evidence, action traces, tool feedback, and accumulated context to produce an intermediate reasoning state or response.The process begins from an initial multimodal observation and user query, then proceeds through multiple steps.
  • Closed-Loop Rollout: New reasoning, actions, and feedback are appended to the context, with aggregation optionally combined with summarization or compression to control growth during long rollouts.The rollout terminates when the model emits a final answer or termination action.
  • Closed-Loop Rollout: In practice, MCR alternates autoregressively between intermediate reasoning tokens, tool calls, and final answers while appending tool outputs and memory summaries to the same multimodal context.This operationalizes the closed-loop process as a single evolving sequence.
  • Actions and Tools in MCR: MCR supports perceptive, tool, memory, verification, and termination actions, unifying diverse agentic behaviors within one sequence-modeling framework.Examples include temporal zoom-ins, ASR, segmentation, temporal grounding, web search, context summarization, evidence checks, and final responses.

3.2. Long-Context Efficient Attention with M2LA

M2LA improves long-context efficiency by compressing cached attention states instead of dropping multimodal tokens. It reconstructs head-specific keys and values from compact latent representations while preserving the full token stream for extended MCR rollouts.

  • Efficiency outcome: M2LA reduces KV-cache memory while retaining the original multimodal token stream, enabling substantially longer MCR rollouts.The bottleneck arises as visual evidence and agent-like reasoning traces accumulate in the evolving context.
  • Core design: M2LA stores one compact latent vector per token and reconstructs head-specific keys and values on demand, replacing standard GQA-style attention blocks.This shifts efficiency from token reduction to in-attention KV-state compression.
  • Core design: RoPE-aware positional aggregation and low-rank factorization compress positional, content-key, and value channels while preserving positional structure and multimodal tokens.Content keys and values are cached in a compact latent space and reconstructed through learned up-projections.
  • Adaptive compression: Modality-aware adapters and layer-wise, modality-wise latent dimensions adapt shared representations to text–vision differences and provide a controllable memory–accuracy trade-off.Latent rank is reduced more aggressively where reconstruction fidelity is less critical.
  • Qwen3 compatibility: The concentrated distribution of Qwen3 RMSNorm coefficients enables a Global-Norm-Linear approximation that makes Qwen3-style architectures compatible with M2LA conversion.This approximation reproduces the original normalized outputs with high fidelity.

3.3. Training Recipe for Long-Horizon Multimodal Reasoning

InternVideo3 uses staged training to recover capabilities after M^2LA conversion, progressively extend temporal context, and strengthen long-horizon reasoning. The recipe culminates in on-policy distillation that provides dense supervision on student-visited trajectories and reduces exposure bias.

  • Continued Pretraining: Lightweight continued pretraining recovers language ability and multimodal alignment disrupted by the M^2LA parameterization change.The mismatch is especially pronounced before the vision encoder and decoder adapt to the new latent attention pathway.
  • Short-to-Long Supervised Fine-Tuning: Short-to-long supervised fine-tuning progressively increases temporal resolution and context length instead of directly training on maximal-length videos.The curriculum addresses unstable optimization caused by simultaneously learning temporal reasoning and extremely long-context attention.
  • Short-to-Long Supervised Fine-Tuning: The curriculum begins at 2 fps and 512 frames, then advances to 4 fps and 2048 frames, expanding context capacity from approximately 32k to 256k tokens.The first stage establishes basic temporal understanding, while the second supports extended durations and finer temporal detail.
  • Rule-Based Reinforcement Learning: Rule-based GSPO trains on video QA and temporal grounding, using group-normalized rewards, uncertainty filtering, and a clipped objective with KL regularization.Temporal grounding is scored by interval IoU against ground-truth moments, while video QA correctness is checked against reference answers.
  • On-Policy Distillation: On-policy distillation uses Qwen3-235B to evaluate student-generated trajectories, transferring stronger reasoning and reducing exposure bias over long reasoning trajectories.The distillation set retains reasoning-heavy examples where the teacher is correct or more complete and the student is incorrect, incomplete, or weakly grounded.

3.4. Video-Agent Instantiation

InternVideo3 is instantiated as a video agent that iteratively combines perception, memory retrieval, tool use, and answer verification within an evolving context. MCR supplies the reasoning loop, M2LA enables longer histories, and staged post-training supports the practical agent capability.

  • Agent Workflow: The agent alternates coarse perception, memory retrieval, targeted re-perception, tool use, and answer verification in a closed-loop process.This extends MCR beyond static question answering by treating video understanding as iterative evidence accumulation.
  • Memory and Routing: Hierarchical memory stores sampled frames, scene boundaries, clip captions, timestamps, and optional subtitle or OCR signals for semantic or temporal retrieval.The rollout maintains an evidence window, retrieved memories, and a compact belief summary.
  • Memory and Routing: The controller invokes video segmentation, ASR, web search, temporal grounding, summarization, verification, and answering, appending tool outputs to the evolving context.Question-type predictions provide a soft prior for deciding whether existing memory suffices or an external tool is needed.
  • Verification and Efficiency: Before termination, recursive verification checks evidential support and unresolved conflicts, triggering focused retrieval or re-perception when support is insufficient.The mechanism reduces hallucination and promotes evidence-grounded responses.
  • Verification and Efficiency: M2LA reduces the KV-cache footprint of accumulated evidence, feedback, and belief updates, allowing longer histories and additional recursive evidence gathering within one rollout.Simple questions can terminate after a coarse pass, whereas harder temporal or causal questions can trigger further evidence collection.

4. Data Curation

InternVideo3 curates stage-specific multimodal data, including a 16M-sample continued-pretraining corpus and a ∼7.2M-sample supervised-fine-tuning corpus. Its long-video SFT data uses hierarchical annotation to preserve local evidence and global temporal coherence, while post-training data targets verifiable rewards and teacher–student capability gaps.

  • Continued Pre-Training: The continued-pretraining corpus contains 16M multimodal samples, corresponding to around 13.5B tokens, to restabilize the M2LA-converted backbone and recover general multimodal capability.This stage precedes downstream long-video training and is not intended to teach new task-specific behaviors.
  • Supervised Fine-Tuning: The ∼7.2M-sample SFT corpus spans long- and short-video understanding, image understanding, STEM reasoning, code generation, documents, UI grounding, conversation, and translation.The corpus is designed to support reasoning while remaining compatible with broader multimodal and agentic workloads.
  • Long-Video SFT Curation: The long-video SFT corpus contains 379K videos averaging 15.8 minutes, totaling ∼100K hours, and is curated for long-context multimodal reasoning.Its hierarchical pipeline uses scene-consistent clips, fine-grained captions, and merged narratives preserving temporal order, cross-scene entity consistency, and event structure.
  • Long-Video SFT Curation: Hierarchical narratives support over 1M QA pairs across perception and recognition, spatial-temporal understanding, event and action reasoning, and holistic semantics.The QA design encourages recognition, state tracking, distant-event connections, and coherent high-level interpretation of long videos.
  • Post-Training Data: Post-training data emphasizes informative long-horizon examples: rule-based RL uses temporal grounding and multiple-choice QA with verifiable rewards, while OPD filters for clear teacher–student capability gaps.Temporal-grounding RL retains IoU values in [0.1, 0.7] and samples 5K examples; multiple-choice RL produces 10K questions with verifiable correctness signals.

5. Experiments · 5.1. Main Results

InternVideo3 is evaluated across long-video, short-video, and spatiotemporal benchmarks, alongside analyses of agentic reasoning and the staged training recipe. It achieves strong open-weight results across these settings, particularly on sustained temporal reasoning, while remaining non-dominant on selected benchmarks.

  • 5. Experiments: Experiments cover standard multimodal benchmarks, video-agent reasoning with retrieval, grounding, and verification tools, and analyses of M2LA and staged post-training components.The evaluation reports official benchmark metrics and task success or answer accuracy under fixed interaction budgets and tool sets.
  • 5.1.1. Long-Video Understanding: InternVideo3 achieves the best open-weight results on Video-MME (73.8), MLVU (77.3), VRBench (69.4), and EgoSchema (76.6).It remains close to the top tier on VideoMME-v2 and LongVideoBench, with scores of 27.6 and 66.8, respectively.
  • 5.1.1. Long-Video Understanding: +8.7 on Video-MME, +6.2 on LongVideoBench, +4.5 on MLVU, +9.3 on LVBench, +17.5 on VRBench, and +12.7 on EgoSchema versus InternVideo2.5-7B.The corresponding InternVideo3 versus InternVideo2.5-7B scores are 73.8 vs. 65.1, 66.8 vs. 60.6, 77.3 vs. 72.8, 55.7 vs. 46.4, 69.4 vs. 51.9, and 76.6 vs. 63.9.
  • 5.1.2. Short-Video Understanding: InternVideo3 obtains a Short QA Avg. of 69.0, outperforming Eagle2.5-8B (67.0), InternVideo2.5-7B (66.5), VideoChat-Flash-7B (66.4), and Qwen3-VL-8B (65.3).It also achieves the best open-weight results on PerceptionTest (81.4) and Tomato (37.4), and matches the best reported open-weight score on NextQA (85.5).
  • 5.1.2. Short-Video Understanding: InternVideo3 improves over InternVideo2.5-7B by +0.6 on NextQA, +6.5 on PerceptionTest, +4.5 on Tomato, and +3.9 on TempCompass.It remains essentially comparable on MVBench and MotionBench, while TempCompass is below Keye-VL-1.5-8B (75.5) and Eagle2.5-8B (74.4).
  • 5.1.3. Spatiotemporal Intelligence: InternVideo3 achieves the best open-weight results on all three temporal grounding benchmarks: 59.9 on QVHighlights, 50.4 on Charades-STA, and 47.9 on ANet.Relative to Qwen3-VL-8B, the gains are +0.5, +2.1, and +1.1 on these benchmarks, respectively.
  • 5.1.3. Spatiotemporal Intelligence: InternVideo3 scores 68.1 on VSIBench, 27.6 on MMSIBench, and 30.7 on MMSIBench-Video, with the strongest spatial result on VSIBench.It outperforms all listed baselines on VSIBench and achieves the best listed result on MMSIBench-Video, but trails InternVL3.5-8B (30.5) on MMSIBench.

5.2. Inference Efficiency · 5.3. Agentic Video Exploration

M2LA improves long-context inference efficiency by accelerating decoding, reducing KV-cache memory, and extending executable context lengths. InternVideo3’s video-agent demonstrations show that MCR supports iterative, evidence-grounded exploration through targeted tool use and verification.

  • 5.2. Inference Efficiency: M2LA remains executable through 768K prefill tokens, whereas Qwen3-VL runs out of memory from 512K onward.For 256K and longer inputs, both models use 64K chunked prefill without external inference acceleration.
  • 5.2. Inference Efficiency: M2LA achieves 39.96 tok/s versus 21.74 tok/s for Qwen3-8B at 32K prefill, a 1.84× decoding speedup under identical single-H200 settings.Decode length is fixed to 16K tokens, with ten measured runs after one warm-up run and no external inference acceleration.
  • 5.2. Inference Efficiency: Decode throughput improves by 4.12× at 128K, 4.77× at 256K, and 5.01× at 384K prefill tokens, while runtime falls by more than 4× at 256K–384K.At 32K, M2LA reduces end-to-end runtime by 1.83×.
  • 5.2. Inference Efficiency: M2LA reduces the KV-cache footprint by roughly 50% across context lengths, supporting its largest gains in decode-dominated long-horizon multimodal reasoning.The relevant setting is batch size 1 with bf16 precision, where long intermediate traces, tool calls, and responses must preserve large multimodal contexts.
  • 5.3. Agentic Video Exploration: InternVideo3 is instantiated as a video agent with segmentation, ASR, temporal grounding, search, summarization, and verification tools for qualitative recursive evidence gathering.Large-scale quantitative agent evaluation is left to future work.
  • 5.3. Agentic Video Exploration: The agent grounds answers in selected visual evidence across attribution, thematic linkage, equipment-performance reasoning, and implicit emotional-state inference.These four representative demonstrations cover complementary forms of long-horizon multimodal reasoning.
  • 5.3. Agentic Video Exploration: For long-form questions, MCR consults hierarchical memory, identifies relevant or uncertain segments, issues targeted tool calls, and verifies before termination.Speech-heavy questions trigger ASR, event-centric questions trigger temporal grounding, and low-confidence answers trigger verification; compared with single-pass inference, iteration recovers incomplete evidence more reliably and reduces coarse-summary errors.
  • 5.3. Agentic Video Exploration: MCR treats agentic video understanding as deciding whether evidence is sufficient, what additional evidence is needed, and how new evidence updates the current belief state.This closed-loop formulation extends beyond generating an answer from a fixed prompt.

5.4. Ablation Studies · 5.5. Discussion

Ablations show that recovery training, explicit long-context exposure, and broad long-video supervision contribute complementary gains, while MCR at inference improves Video-MME but remains a proof of concept. The discussion frames long-horizon multimodal agency as requiring an evolving reasoning substrate rather than context length or static QA scaling alone.

  • 5.4. Ablation Studies: The full recipe reaches the best overall average of 66.8, showing complementary gains from continued pretraining, long-context training, and curated long-video supervision.The ablations remove CPT, Long Ctx., and LV Data to identify the recipe’s key ingredients.
  • 5.4. Ablation Studies: Removing CPT causes the largest Video-MME drop (-4.2), degrades MLVU by -3.0, and lowers the overall average from 66.8 to 64.7.Continued pretraining serves as an important recovery stage after M2LA conversion.
  • 5.4. Ablation Studies: Removing long-context training lowers the average to 65.4 and produces the largest LongVideoBench drop (-3.0), confirming the need for explicit extended-context exposure.The effect on LVBench is small, but broader long-video capability is noticeably weaker.
  • 5.4. Ablation Studies: Architectural conversion alone is insufficient; recovery training, explicit long-context exposure, and broad long-video supervision contribute differently, with their combination yielding the strongest long-horizon performance.The ablation results support the design logic of the complete training recipe.
  • 5.4. Ablation Studies: Agentic inference with MCR increases Video-MME from 73.1 to 75.8 (+2.7) by enabling evidence revisitation and intermediate-conclusion verification.This preliminary result instantiates MCR at test time rather than only as a training framework.
  • 5.4. Ablation Studies: The agentic gain is not uniform across long-video benchmarks, so the authors present it as a proof of concept rather than evidence of general agentic superiority.Current public benchmarks primarily target direct question answering and may not consistently reward multi-step retrieval.
  • 5.5. Discussion: Longer context alone is insufficient; the model must use it as an evolving reasoning substrate linking events, relevant evidence, and distant observations coherently.The strongest gains occur when tasks require maintaining an account of what happened and how evidence relates across the video.
  • 5.5. Discussion: InternVideo3 improves most on long-video and selective temporal reasoning tasks, supporting long-horizon multimodal agency as a distinct capability beyond scaling static multimodal QA.The discussion does not claim uniform gains across all short-form tasks.

6. Conclusive Remarks

InternVideo3 frames long-horizon multimodal understanding as closed-loop contextual reasoning and combines efficient attention with staged post-training to support sustained video understanding. Its results support context efficiency and multimodal verification as important capabilities for visually grounded agency, while positioning the work as a step rather than a complete solution.

  • Core perspective: InternVideo3 models video understanding as evidence accumulation, context updating, tool interaction, and self-correction over evolving multimodal observations.This reframes multimodal understanding as recursive contextual reasoning with intermediate conclusions and external feedback.
  • Efficiency and training: M2LA compresses KV-cache states while preserving the full multimodal token stream, enabling practical long-context rollouts.The training recipe combines continued pretraining after attention conversion, short-to-long supervised fine-tuning, rule-based reinforcement learning, and on-policy distillation.
  • Empirical results: InternVideo3 achieves strong short-video and long-video benchmark performance, with notable gains on Video-MME, MLVU, and EgoSchema.Relative to InternVideo2.5-7B, it substantially improves several representative long-video benchmarks without moving to a much larger or newer backbone family.
  • Broader significance: The work treats video-centric long-horizon reasoning as a testbed for maintaining, updating, and verifying multimodal contextual state relevant to visually grounded agency.The authors explicitly present InternVideo3 as a step toward multimodal agents, not a complete solution to multimodal agency.
  • Broader significance: The paper identifies context efficiency and closed-loop multimodal reasoning as broader lessons for long-horizon multimodal rollouts amid rapidly evolving model architectures.It does not claim frontier-wide architectural finality as newer model families adopt stronger native long-context architectures and broader post-training recipes.

7. Limitations, Discussion, and Future Work

The discussion identifies limitations in model currency, efficiency generality, conceptual scope, tool dependence, and evaluation coverage. Future work targets broader efficient-attention adaptations, multimodal agent settings, predictive world models, and resource-efficient open-weight adaptation.

  • Rapidly Evolving Open-Weight Frontier: Newer open-weight multimodal families surpass InternVideo3 on several benchmarks and often provide stronger native long-context designs.This rapid ecosystem evolution limits the currency of the comparison landscape.
  • Efficiency: M^2LA offers one practical GQA-to-latent-attention route, but newer models with native MLA, linear, or hierarchical attention require broader efficiency strategies.The paper maintains that efficient context handling remains important for long-horizon multimodal reasoning and deployment.
  • Conceptual Novelty of MCR: MCR combines established ideas such as closed-loop reasoning, tools, memory, self-correction, and evidence gathering into a clear formulation for long-horizon multimodal reasoning.Its contribution is framed as formulation and extensibility rather than inventing these ingredients from scratch.
  • World Modeling: MCR models contextual belief states rather than a full action-conditioned simulator, making it practical while limiting scope relative to predictive world models.Future work could combine contextual reasoning with stronger predictive models to compare possible future outcomes.
  • Tool Dependence and System Fragility: External ASR, grounding, segmentation, retrieval, and search tools can propagate errors into shared context because their interfaces are not jointly optimized with the base model.Suggested remedies include tighter model–tool integration, uncertainty-aware use, and improved calibration of tool-derived evidence.
  • Scope of the Current Evaluation: The evaluation covers video understanding and a perception-tool agent setup but omits GUI, browser, mobile, and embodied-robotics settings while under-measuring recursive evidence gathering and verification.Future benchmarks should treat visually grounded long-horizon reasoning as a distinct multimodal-agency capability axis.
Loading 2606.12195v1…