PaperlayerView exact revision on arXiv

Source-linked AI summary

VideoTapestry: Query-Adaptive Memory Refinement for Multi-Agent Long-Video Understanding

Yucheng Liu, Yufei Yin, Mingxiao Feng, Jiajun Deng, Wengang Zhou, Houqiang Li

arXiv:2610.06672v1cs.CVcs.AI

TL;DR

Long-video understanding must preserve global context while finding decisive evidence across extended temporal spans, a challenge complicated by the distinct limitations of query-led exploration and query-independent memory. VideoTapestry uses scale-aligned agents to refine a preconstructed hierarchy with query-relevant observations, then assembles a composite memory. It improves accuracy over direct GPT-5.5 inference across four benchmarks.

  • Problem

    Long-video questions require preserving global context while locating decisive evidence at appropriate granularity, and integrating question-specific evidence with existing memory remains challenging.

    Source: §1 · pp. 1–2

  • Method

    VideoTapestry is a training-free multi-agent framework that refines a preconstructed hierarchical memory through coarse-to-fine query-guided observations and assembles the refined layers for final reasoning.

    Source: §1 · p. 2

  • Results

    VideoTapestry improves accuracy over direct GPT-5.5 inference by 17.2, 14.9, 9.8, and 7.0 percentage points on LVBench, LongVideoBench (Long), Video-MME (Long), and EgoSchema, respectively, with the best reported results among compared methods on all four.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The results support organizing global video structure alongside selectively refined query-relevant evidence for final reasoning.

    Source: §5 · p. 9

    Retrieval remains constrained by the initial segmentation and parent-child structure, so incorrect event grouping or temporal boundaries may hinder access to relevant evidence.

    Source: D LIMITATIONS AND FUTURE WORK · p. 22

Abstract

from arXiv · show

Long-video understanding places substantial demands on memory, as answering questions often requires retrieving information distributed across extended temporal spans. Existing approaches broadly follow two paradigms: query-driven exploration, which is sensitive to localization errors, and query-independent memory construction, which may omit question-specific details. We introduce VideoTapestry, a training-free multi-agent framework that adapts a preconstructed hierarchical video memory through coarse-to-fine, query-driven refinement. The preconstructed memory organizes video content into three levels, capturing global narrative context, event-level temporal structure, and fine-grained relational evidence, respectively. To support coarse-to-fine localization and observation, we assign a specialized agent to each level, keeping retrieval and refinement within a scale-specific context. Guided by the query, these agents revisit relevant video regions and enrich layer-wise memories with targeted multimodal observations. Their refinements are assembled according to the original hierarchy into a composite query-adaptive memory, preserving global context in a compact form while retaining fine-grained evidence along query-relevant branches for final reasoning. Compared with direct GPT-5.5 inference, VideoTapestry achieves absolute accuracy gains of 17.2%, 14.9%, 9.8%, and 7.0% on LVBench, LongVideoBench (Long), Video-MME (Long), and EgoSchema, respectively, achieving the state-of-the-art results among all competitors.

1 INTRODUCTION

Long-video questions require preserving global context while locating decisive evidence at the right granularity, but query-led exploration and query-independent memory each have limitations. VideoTapestry refines a preconstructed hierarchy with query-guided agents and reports gains over direct GPT-5.5 inference across four benchmarks.

1 INTRODUCTION

  • Query-driven exploration can miss key segments through inaccurate localization, while query-independent memory offers broad coverage but may omit question-specific details.The two paradigms provide complementary strengths, motivating integration of new evidence while limiting irrelevant content.Source: §1 · pp. 1–2
  • VideoTapestry uses a training-free multi-agent framework to refine a fixed, preconstructed hierarchy through coarse-to-fine, query-driven observation.Overview, Skim, and Focus agents work at distinct semantic levels, guided by candidate memories and upstream inspection objectives.Source: §1 · pp. 2–3
  • Layer-wise memories retain a global scaffold while refining question-relevant branches, then assemble along the original parent-child relations for final reasoning.An independent Answer Agent predicts from the assembled memory without further retrieval or observation.Source: §1 · pp. 2–3
  • VideoTapestry improves accuracy over direct GPT-5.5 inference by 17.2, 14.9, 9.8, and 7.0 percentage points on LVBench, LongVideoBench (Long), Video-MME (Long), and EgoSchema, respectively, and achieves the best reported results among compared methods on all four.Controlled ablations also support the effectiveness of query-adaptive memory refinement and scale-aligned multi-agent collaboration.Source: §1 · p. 3
  • The framework’s query-adaptive memory strategy integrates newly observed evidence into selected branches while preserving global context and the original hierarchy.The framework aligns agents with memory levels and coordinates refinement through branch selection and inspection-objective transfer.Source: §1 · p. 3

2 RELATED WORK

The related work covers approaches to long-video context limits, hierarchical video memory, and agentic video reasoning. VideoTapestry is positioned as adapting hierarchical memory through targeted re-observation and scale-aligned agents.

2.1 LONG-VIDEO UNDERSTANDING

  • Long-video research expands context windows, compresses spatiotemporal tokens, and improves temporal coverage, while practical context budgets still motivate external memory and query-conditioned retrieval.Examples include LongVU, LongViTU, Hour-LLaVA, Video Panels, and Tempo.Source: §2.1 · p. 3

2.2 HIERARCHICAL VIDEO MEMORY

  • Hierarchical and graph-based video memories organize descriptions, clips, or temporal segments across semantic levels and relations; VideoTapestry adapts a query-independent hierarchy through targeted re-observation.Its composite memory retains a compact global scaffold and refined query-relevant branches.Source: §2.2 · p. 3

2.3 AGENTIC AND MULTI-AGENT VIDEO REASONING

  • Agentic systems plan, search, retrieve, and selectively observe video, while multi-agent approaches distribute perception, localization, verification, and reasoning across specialized roles.VideoTapestry emphasizes the final hierarchical representation: scale-aligned agents refine relevant branches and assemble layer-wise memories for an independent Answer Agent.Source: §2.3 · pp. 3–4

3 METHOD

VideoTapestry refines a fixed hierarchical video memory through query-guided, scale-aligned agents that inspect and update temporary layer memories. The refinements are assembled under the original hierarchy for an independent answer agent.

3.1 QUERY-ADAPTIVE MEMORY AND SCALE-ALIGNED ORGANIZATION

  • Three levels represent global stages, localized events, and fine-grained relational evidence, with Overview, Skim, and Focus agents assigned to the corresponding scales.Selected parent identifiers determine which descendants are exposed at the next level.Source: §3.1 · pp. 4–5

3.2 LAYER-WISE TOOL-AUGMENTED MEMORY REFINEMENT

  • Each visited layer follows Plan→Observe→Update→Decide, producing a refinement batch before selecting the next scope or stopping.Agents use a Visual Observation Tool; the Overview Agent additionally accesses transcript evidence in the evaluated Video-MME configuration.Source: §3.2 · p. 5

3.2.1 OVERVIEW AGENT: SUPER-LEVEL MEMORY REFINEMENT

  • The Overview Agent uses retrieval outputs as localization hints, associates observations with SuperEvents, protects evidence-linked events and neighbors, and compresses remaining descriptions.The protected nodes retain their original descriptions and receive available evidence; unavailable evidence fields are omitted.Source: §3.2.1 · pp. 5–6

3.2.2 SKIM AGENT: MACRO-LEVEL MEMORY REFINEMENT

  • The Skim Agent rewrites selected MacroEvent descriptions with observed evidence while preserving relevant static facts and compressing unrelated content.Missing or invalid updates retain the original text, and node identifiers, parent relations, semantic levels, and temporal ranges remain fixed.Source: §3.2.2 · p. 6

3.2.3 FOCUS AGENT: SUBGRAPH-LEVEL MEMORY REFINEMENT

  • The Focus Agent inspects selected local Subgraphs and updates their serialized descriptions, then terminates hierarchical exploration.The operation changes temporary text rather than underlying graph nodes or edges; missing or invalid updates retain the original text.Source: §3.2.3 · p. 6

3.3 COMPOSITE MEMORY ASSEMBLY AND ANSWER GENERATION

  • Assembly retains the refined Super-level scaffold and places selected refined descendants beneath their original parents; the independent Answer Agent predicts without further retrieval or observation.The serialized context includes the question, options, and a compact handoff trace that excludes complete observation and update logs.Source: §3.3 · p. 6

4 EXPERIMENTS

VideoTapestry improves on direct and prior methods across four long-video benchmarks, while ablations and backbone tests examine its components and generality.

4.1 BENCHMARKS

  • VideoTapestry is evaluated on LVBench, LongVideoBench, Video-MME (Long), and a 500-question EgoSchema subset.The benchmarks cover long videos and questions, with EgoSchema comprising egocentric reasoning clips.Source: §4.1 · p. 7

4.2 EXPERIMENTAL SETUP

  • The training-free evaluation keeps model parameters and preconstructed memory fixed; Direct uses the same backbone and examples without the agentic framework.Ablations use a fixed 200-question Video-MME Long subset, and comparisons include direct VLM and agentic baselines.Source: §4.2 · pp. 7–8

4.3 MAIN RESULTS

  • With frozen GPT-5.5, VideoTapestry gains 7.0–17.2 percentage points over same-backbone Direct and beats the strongest prior result in each column by 0.8–4.4 points.It also leads across all six LVBench dimensions, with gains of 3.9–9.6 points over the strongest prior result.Source: §4.3 · p. 8

4.4 ABLATION STUDIES

  • In query-time memory refinement, removing visual observation, selected-window ASR, structured memory refinement, or multi-agent collaboration reduces accuracy by 3.0, 5.5, 1.5, or 4.5 points, respectively.With the full exploration fixed, combining the refined Super-level scaffold and refined paths reaches 87.0%, versus 85.0% for Refined Super Only, 80.5% for Refined Paths Only, and 79.5% for Static Memory Only.Source: §4.4 · pp. 8–9

4.5 BACKBONE GENERALIZATION

  • With memory and tool interfaces fixed, VideoTapestry improves every tested backbone by 7.0–9.5%, with the largest gain on Qwen3.6-27B.The tested backbones include GPT-5.5, GPT-5, Qwen3.7-Plus, and Qwen3.6-27B.Source: §4.5 · p. 9

5 CONCLUSION

VideoTapestry uses coarse-to-fine agents to refine a preconstructed hierarchy into query-adaptive memory. The conclusion reports consistent improvements over direct and agentic baselines on long-video benchmarks.

5 CONCLUSION

  • VideoTapestry uses coarse-to-fine retrieval and targeted observation to refine query-relevant content at complementary semantic granularities.The Overview, Skim, and Focus agents assemble their layer-wise refinements with a global scaffold for final reasoning.Source: §5 · p. 9
  • The composite query-adaptive memory preserves video-wide structure while enriching relevant branches for final reasoning.The conclusion describes the framework as training-free and hierarchical.Source: §5 · p. 9
  • Experiments on long-video benchmarks show consistent improvements over the same-backbone direct baseline and representative agentic methods.Table 4 is identified as a generalization comparison across backbone models.Source: §5 · p. 9, Table 4 · p. 9

A ADDITIONAL RESULTS

The additional examples trace refinement from broad localization to fine-grained verification and evidence aggregation across distant video branches.

Additional Qualitative Results

  • The illustrated trajectories proceed from coarse localization to fine-grained verification, including a state-by-state check for a brief door-opening event.The plan calls for inspecting the complete six-second sequence after the initial interval is selected.Source: A ADDITIONAL RESULTS · p. 13

Case 1

  • In the stairwell example, agents narrow inspection to 27–33 seconds, verify the man descends without opening the door or window, and answer C.The refined evidence places the descent at 27.5–30.5 seconds and the approach to the closed door at 30.5–32.5 seconds.Source: A ADDITIONAL RESULTS · p. 13

Case 2

  • In the piano example, agents verify distinct performers across two related branches roughly 25 minutes apart, aggregate the evidence, and answer D..Source: A ADDITIONAL RESULTS · p. 13

B.1 EVALUATION AND IMPLEMENTATION SETTINGS

The appendix describes baseline fairness and the visual, audio, and memory procedures used during VideoTapestry evaluation.

B.1 EVALUATION AND IMPLEMENTATION SETTINGS

  • Direct baselines receive uniformly sampled frames, while VideoTapestry observations follow its agent-specific sampling and temporal-selection procedures.GPT-family direct models receive 50 frames; VideoTapestry Overview samples up to 120 frames in six-frame grids, while Skim and Focus use selected-interval frames.Source: §B.1 · p. 14
  • All experiments use hierarchical memories built following the MemDreamer procedure to isolate query-time refinement from offline memory construction.The MemDreamer baseline is separately reproduced.Source: §B.1 · p. 14
  • For GPT-family experiments, only the Answer Agent uses high reasoning effort; for Qwen-based experiments, thinking is enabled for planning, decision, and final answering.Other GPT-family calls use the default setting, and other Qwen calls use non-thinking mode.Source: §B.1 · p. 14
  • The ASR tool accesses only agent-selected windows, applying Whisper when subtitles are unavailable or retrieving subtitle text when they are provided.At most three selected windows are accessed rather than transcribing the full video.Source: §B.1 · p. 14

B.2 DETAILED INFERENCE PIPELINE

VideoTapestry’s inference pipeline processes a static hierarchy through layer-aligned agents, then assembles the refined memory for answer generation. Within each visited layer, an agent plans, observes, updates valid memory nodes, and decides whether to stop or hand off to the next level.

B.2 DETAILED INFERENCE PIPELINE

  • Algorithm 1 propagates candidate branches and inspection instructions across levels, stops when directed or when no valid descendants remain, and assembles composite memory for the Answer Agent.The Answer Agent predicts from the assembled memory without further retrieval or observation.Source: §B.2 · p. 15
  • Algorithm 2 applies one Plan–Observe–Update–Decide batch and one decision at each visited layer.The procedure records plans, observations, and updates in a local trajectory.Source: §B.2 · p. 15
  • At the Super level, Overview observes a merged video window with up to 120 frames, plans selected-window ASR, and uses up to three normalized transcript windows.If ASR ranges are not proposed, the visual window is used; evidence-linked SuperEvents and valid immediate temporal neighbors inform the updated memory.Source: §B.2 · p. 15
  • At lower levels, visual observation is restricted to planned ranges intersected with candidate intervals, using up to 50 individual frames before valid requested nodes are updated.At the Subgraph level, Focus is the final retrieval layer and must stop descending.Source: §B.2 · p. 15

B.3 PROMPTS USED IN VIDEOTAPESTRY

This appendix section presents the prompts used across VideoTapestry’s planning, observation, memory-updating, decision, and answer-generation stages. They guide hierarchy-based retrieval and the use of query-adaptive memory, distinguishing global-summary from detail questions while reserving answers for the final stage.

B.3.1 HIERARCHICAL OBSERVATION PLANNING

  • Planning prompts direct agents to refine the current layer’s static summary through bounded, query-relevant observation and updates, rather than treating old memory as visual verification.The agent is instructed not to answer the question and to return structured planning fields.Source: §B.3.1 · p. 16
  • For overview localization, prompts treat locator outputs only as unverified time hints, requiring visual confirmation and a complete, question-relevant observation window.They prefer complete joint matches, broaden the window when anchors are unresolved, and cap calls at two observations and two updates.Source: §B.3.1 · p. 16

B.3.2 VISUAL OBSERVATION TOOL

  • The visual observation prompt limits the tool to the supplied instruction, requesting directly relevant visible facts and explicit uncertainty instead of inference or answer selection.It requires a structured response with observation results, uncertainty, and relevance to the question.Source: §B.3.2 · p. 18
  • The ASR tool uses only the supplied timestamped transcript, reports supported spoken facts or uncertainty, and copies directly supporting transcript lines exactly.It receives neither the original question nor answer options and returns an empty evidence string when no reliable support exists.Source: §B.3.2 · p. 18

B.3.3 MEMORY UPDATE TOOL

  • Memory-update prompts keep visual and ASR observations separate, preserve memory metadata, and restrict updates to objective facts supported by referenced sources.They prohibit option-tendency judgments and direct the final answer agent, not the updater, to judge the options.Source: §B.3.3 · pp. 19–20
  • The macro-event and subgraph update prompt preserves concrete details in relevant memories, summarizes irrelevant memories in at most 10 characters, and forbids answer-choice steering.Updates must not invent information beyond the old memory or observations.Source: §B.3.3 · p. 20

B.3.4 HIERARCHICAL DECISIONS AND INTER-AGENT HANDOFF

  • Hierarchical decision prompts ask agents whether updated memory is sufficient or whether to descend, specifying child-memory selection and handoff fields without answering the question.When stopping, agents must return no child memories and a null next instruction.Source: §B.3.4 · p. 21
  • L1 routing prompts classify Macro information as main idea, temporal order, identity, action verification, spatial relation, or causal result.The next instruction is one sentence telling the Skim Agent what type of Macro information to find, and the selected type is returned as template_type.Source: §B.3.4 · p. 21
  • Only global main-idea, title, or summary questions may stop at Super; all other question types continue to Macro.The prompt directs other question types to proceed to Macro.Source: §B.3.4 · p. 21
  • When retrieval continues from Macro to Subgraph, the prompt calls for a short instruction.The provided passage introduces the instruction format but does not include its contents.Source: §B.3.4 · p. 21
  • For non-global detail questions, retrieval should continue to Subgraph after at least one Macro observation; old Macro text alone is insufficient to terminate.The handoff prompt asks the agent to track what was found and what still needs verification.Source: §B.3.4 · p. 21
  • At L2 (Subgraph), the prompt directs retrieval to stop and sets stop_descending to true.Focus is specified as the final retrieval layer.Source: §B.3.4 · p. 21

B.3.5 FINAL ANSWER AGENT

  • The final-answer agent is instructed to return strict JSON based on the provided memory.The prompt specifies JSON keys, but the supplied passage does not show which keys.Source: §B.3.5 · p. 21
  • The final-answer prompt supplies the raw question with options and the serialized query-adaptive video memory.Source: §B.3.5 · p. 21

C MEMORY ABLATION PROTOCOL

The protocol separates a static-retrieval comparison from controlled tests of final memory composition. The controlled variants share the same completed exploration, so only the contents passed to the Answer Agent differ.

C MEMORY ABLATION PROTOCOL

  • All Table 3(b) configurations use the same 200-question Video-MME Long subset, GPT-5.5, and preconstructed hierarchical memory.The protocol distinguishes the static-retrieval baseline from controlled final-memory comparisons.Source: C MEMORY ABLATION PROTOCOL · p. 22
  • Static Memory Only retrieves the hierarchy without Visual, ASR, or Update calls, and compares against VideoTapestry’s combined query-time observation and refinement effects.It passes retrieved static contents and the retrieval trace to the Answer Agent without compressing or rewriting node contents.Source: C MEMORY ABLATION PROTOCOL · p. 22
  • The controlled variants fix VideoTapestry’s exploration trace and compare refined Super-only, selected-path-only, and combined Super-plus-path memories.They do not rerun retrieval, observation, or updating, isolating the contribution of the final serialized memory components.Source: C MEMORY ABLATION PROTOCOL · p. 22

D LIMITATIONS AND FUTURE WORK

The paper identifies constraints from the fixed hierarchy, computational costs, and reliance on model-generated evidence interpretations. It proposes structural revision, budget-aware evaluation, and provenance tracking or selective re-verification as future directions.

D LIMITATIONS AND FUTURE WORK

  • Retrieval remains constrained by the fixed hierarchy’s initial segmentation and parent–child structure, so incorrect event groupings or temporal boundaries may hinder access to relevant evidence.Alternative memory constructors and selective structural revision are proposed as future directions.Source: D LIMITATIONS AND FUTURE WORK · p. 22
  • Offline memory construction and additional query-time model and tool calls mean the reported accuracy gains do not establish computational-efficiency advantages.Future work could use budget-aware observation and refinement while accounting separately for offline and per-query costs.Source: D LIMITATIONS AND FUTURE WORK · p. 22
  • Errors in model-generated interpretations of visual and transcript evidence may propagate into refined memory and subsequent decisions.Prompts and separate recording of visual and transcript observations do not guarantee correct interpretation or fully resolve conflicts; provenance tracking and selective re-verification are proposed.Source: D LIMITATIONS AND FUTURE WORK · p. 22

Related papers

Loading 2610.06672v1…