Source-linked AI summary
VideoLoop: Looped Working Memory Against Semantic Thrashing in Long-Form Video Agents
Jinfa Huang, Jianming Xu, Jingyang Lin, Zhengyuan Yang, Jiebo Luo
TL;DR
Long-form video agents must gather sparse evidence across extended timelines, yet append-only memory can dilute key evidence as it grows. VideoLoop couples video reasoning with an inner loop that retrieves stored artifacts and rewrites bounded working memory. It improves accuracy across three long-form video benchmarks and transfers gains across LVLM backbones.
Problem
Long-form video evidence is sparse across non-contiguous segments, and append-only agent memory can dilute key evidence as it grows.
Source: §1 · p. 1Method
VideoLoop pairs an outer multimodal video agent with an inner orchestrator that retrieves filesystem artifacts and rewrites bounded working memory after each step.
Source: §1 · p. 2Results
VideoLoop improves across three long-form video benchmarks and multiple LVLM backbones; with Gemini 3.1 Pro, it gains +4.5 points on VideoMME (long).
Source: §1 · p. 3, §4.2 · p. 8Takeaways & Limitations
The paper concludes that long-horizon agents should curate memory rather than accumulate it.
Source: §5 · p. 9
Abstract
from arXiv · showhide
Long-form video understanding requires multimodal agents to iteratively gather evidence over many reasoning steps. However, most existing agentic methods suffer from semantic thrashing: as append-only working memory grows, attention to key evidence collapses, and the agent loses access to what it has already found. First, we provide a structural argument showing that append-only memory can incorporate newly observed target evidence, but cannot remove accumulated noise or prevent ordered context growth without a rewrite operator. Second, motivated by this analysis, we propose VideoLoop, a multimodal agent with two coupled loops. The outer loop reasons over the video and the inner loop, after each step, retrieves artifacts from an unbounded filesystem of past observations and intermediate analysis, and rewrites a bounded working memory. Extensive experiments demonstrate the effectiveness of VideoLoop, which improves four popular LVLM backbones in a plug-and-play manner, with an average gain of 4.2% points over baseline on VideoMME (long). Further analysis of working memory suggests that VideoLoop mitigates semantic thrashing: on the hardest quarter of VideoMME (long) questions, a blind judge that reads only the agent's context answers 81.1% correctly, versus 60.9% for the append-only agent. With Gemini 3.1 Pro, VideoLoop reaches 88.3% on VideoMME (long), 88.8% on VideoMMMU, and 80.9% on LongVideoBench (long).
1 INTRODUCTION
Long-form video agents must gather sparse evidence across non-contiguous segments, but append-only memory can dilute evidence as observations accumulate. VideoLoop pairs video reasoning with a memory-rewriting loop and improves results across long-form video benchmarks and LVLM backbones.
1 INTRODUCTION
- Long-form video evidence is sparse and scattered across non-contiguous segments, while LVLMs struggle to reliably ingest massive contexts.Iterative agents address this setting by observing video regions, updating evidence memory, and deciding where to look next.Source: §1 · p. 1
- On VideoMME (long), VideoLoop reaches 85.8% versus 81.9% for the append-only agent, a +3.9-point gain with Gemini 3 Flash.The figure describes retrieval as first peaking and then collapsing as append-only context grows.Source: Figure 1 · p. 2
- Append-only memory can dilute relevant evidence as it grows, and cannot remove accumulated irrelevant evidence or bound the cost of retaining past observations without a rewrite operator.The paper terms this structural failure semantic thrashing.Source: §1 · p. 2
- VideoLoop stores observations in an unbounded filesystem and uses an inner orchestrator to retrieve relevant artifacts and rewrite bounded working memory after each outer step.The outer loop observes the video; the inner loop maintains the working memory.Source: §1 · pp. 2–3, Figure 1 · p. 2
- VideoLoop improves accuracy across three long-form video benchmarks and transfers its gains across diverse LVLM backbones without retraining.The paper also reports Gemini 3.1 Pro results of 88.3% on VideoMME (long), 88.8% on VideoMMMU, and 80.9% on LongVideoBench (long).Source: §1 · pp. 2–3
2 RELATED WORK
Prior video agents use iterative controllers, predefined pipelines, and varied memory mechanisms to gather and integrate evidence across long temporal spans. VideoLoop distinguishes itself by exploring in a coding sandbox and assigning memory maintenance to a dedicated inner-loop agent.
Agentic Video Understanding
- Unlike predefined pipelines and prebuilt indices, VideoLoop writes code in a sandbox to direct exploration without upfront video preprocessing or a fixed schema.It composes retrieval and analysis routines using a minimal toolset.Source: §2 · p. 3
Memory Mechanisms in Video Agents
- VideoLoop separates video exploration from working-memory maintenance through a dedicated inner-loop agent that retrieves evidence, consolidates it across segments and sections, and edits memory under a fixed token budget.The agent uses a persistent filesystem to retrieve prior evidence.Source: §2 · p. 3
- Prior memory approaches include pruning, learned retention and deletion, hierarchical multimodal memory, knowledge-graph consolidation, and active context management.VideoLoop differs by combining filesystem retrieval, joint consolidation, and section-level editing under a fixed budget.Source: §2 · p. 3
3 METHODOLOGY
The section characterizes semantic thrashing as a consequence of append-only memory that cannot remove noise, then motivates decoupling video exploration from memory management. VideoLoop combines persistent artifact storage with bounded, repeatedly rewritten working memory.
3.1 SEMANTIC THRASHING PROBLEM
- Append-only updates can add newly observed target evidence but cannot remove non-target content, so noise accumulates and can obscure relevant evidence under bounded context.The proposed structural diagnostic separates newly added target evidence from new noise; the motivation for rewriting is to remove redundancy and recover missing target evidence.Source: §3.1 · p. 4
3.2 VIDEO AGENT WITH PERSISTENT SANDBOX
- VideoLoop separates video exploration from memory management: the outer agent saves observations and artifacts to a persistent filesystem, while the inner orchestrator retrieves evidence and updates working memory.The external store retains observations and intermediate artifacts losslessly; the working memory is bounded and rewritten after each step.Source: §3.2 · pp. 4–5, Figure 2 · p. 5
3.3 DUAL-LOOP BOUNDED WORKING MEMORY
- The policy context combines the query, rewritten memory, and a capped window of recent turns, keeping total context size bounded independently of iteration.The memory orchestrator keeps working memory within a predefined token budget below the effective context capacity.Source: §3.3 · p. 5
3.4 FILESYSTEM-BASED MEMORY ORCHESTRATION
- A three-tier hierarchy pairs six-section bounded working memory with a compressed action manifest and a filesystem that preserves extracted artifacts losslessly.The manifest bridges the filesystem and working memory; the orchestrator can retrieve relevant files and make UPDATE, APPEND, or DELETE edits to memory sections.Source: §3.4 · p. 6
- Memory divergence decreases only when removed noise and imported missing evidence together outweigh accidentally lost target evidence and newly added noise.This is a condition on rewrite quality, not an unconditional guarantee.Source: §3.4 · p. 6
4 EXPERIMENTS
Across three long-video benchmarks, VideoLoop improves over native backbones. Component and memory analyses report gains in accuracy, evidence retrievability, and token efficiency.
4.1 EXPERIMENT SETUP
- Experiments compare native LVLM inference and video-agent systems across three long-video benchmarks, using long subsets where specified.The evaluation includes 900 VideoMME (long) questions, 900 VideoMMMU questions, and 564 LongVideoBench (long) questions.Source: §4.1 · pp. 6–7
4.2 MAIN RESULTS
- 88.3% on VideoMME (long), 88.8% overall on VideoMMMU, and 80.9% on LongVideoBench (long) with Gemini 3.1 Pro; gains over its native model are +4.5, +4.2, and +3.2 points, respectively.Table 1 compares VideoLoop with native LVLMs and prior video-agentic systems across the benchmarks.Source: Table 1 · p. 7, §4.2 · pp. 7–8
4.3 ANALYSIS OF VIDEOLOOP DESIGN COMPONENTS
- In the Gemini 3 Flash VideoMME ablation, full VideoLoop improves +3.9 points over append-only (+9.3 on Q4), while filesystem access adds +2.4 over dual-loop rewriting.Dual-loop rewriting alone reaches 83.3% from append-only’s 81.9%; adding filesystem access raises accuracy to 85.8%.Source: Table 2 · p. 8, §4.3 · p. 8
4.2 MAIN RESULTS
- Across Gemini 3.1 Pro, Gemini 3 Flash, Kimi K2.5, and MiMo-V2-Omni, VideoLoop improves over each native model by +4.5, +5.1, +3.5, and +3.7 points, respectively.The results indicate compatibility with different policy models.Source: Figure 3 · p. 8, §4.2 · p. 8
4.4 FURTHER ANALYSIS
- Full VideoLoop reaches 85.8% accuracy at 618.2K tokens per question versus append-only’s 81.9% at 614.9K, a +0.5% token overhead.The full design reduces input tokens from 584.6K to 559.0K, while dual-loop alone has a +5.2% token overhead.Source: Table 3 · p. 9, §4.4 · p. 9
- Compared with append-only, VideoLoop views fewer frames, uses fewer iterations, and distributes frame inspection more evenly across video timelines.The analysis associates these patterns with reduced redundant evidence collection and iterative overhead.Source: Figure 5 · p. 9, §4.4 · p. 9
- On Q4, blind-judge final-snapshot accuracy is 81.1% for VideoLoop versus 60.9% for append-only; across quartiles, VideoLoop stays stable or improves while append-only plateaus or degrades.The blind judge answers from frozen context snapshots without access to the video, filesystem, or tools.Source: §4.4 · pp. 8–9, Figure 4 · p. 8
5 CONCLUSION
VideoLoop pairs an outer multimodal agent with an inner LLM that rewrites bounded working memory from a sandboxed artifact store after every step. It improves accuracy on three long-form video benchmarks and supports the principle that long-horizon agents should curate, not accumulate, memory.
5 CONCLUSION
- VideoLoop pairs an outer multimodal agent with an inner LLM that rewrites bounded working memory after every step from a sandboxed artifact store.The framework is designed for long-form video understanding.Source: §5 · p. 9
- VideoLoop improves accuracy on three long-form video benchmarks, with gains transferring across multiple LVLM backbones in a plug-and-play manner.Source: §5 · p. 9
- For long-horizon agents, the paper concludes that memory should be curated rather than accumulated.Source: §5 · p. 9