Source-linked AI summary
Caption-once, Frames-on-Demand: Visual-Need Routing for Budget-Aware Agentic Long Video Understanding
Weitong Cai, Hang Zhang, Yukai Huang, Yiqiao Xie, Shan Gao, Jiankang Deng, Songcen Xu, Jifei Song, Zhensong Zhang
TL;DR
Long-video understanding must preserve temporal structure and fine-grained visual attributes under tight budgets. CFD builds reusable narrative memory offline, then routes bounded frame retrieval only to perceptual queries; experiments report competitive accuracy with substantially lower online visual cost.
Problem
Long-video understanding must reason over hours of content under tight compute and bandwidth budgets while retaining both temporal structure and fine-grained visual attributes.
Method
CFD performs one offline edge captioning pass to build cached event and clip memories, then uses story-first cloud reasoning and a Visual-Need Router for bounded keyframe retrieval.
Results
CFD achieves competitive accuracy while substantially reducing online visual processing, including a reduction from 6.63 to 5.76 frames/query with routing.
Takeaways & Limitations
Visual access can be allocated selectively: language supports long-horizon temporal reasoning, while raw frames are reserved for attribute-critical questions.
Takeaways & Limitations
CFD depends on offline caption quality and shot-boundary segmentation, which may miss transient visual cues and do not directly suit continuous egocentric recordings.
Abstract
from arXiv · showhide
Long-video understanding on edge devices must reason over hours of content under tight compute and bandwidth budgets. Subsampling visual tokens loses temporal structure, while text-only video memories lose fine-grained visual attributes. We observe a visual-textual duality: language memories carry long-range temporal structure better than dense frames, while pixels remain decisive for attribute-level perception. Building on this insight, we propose Caption-once, Frames-onDemand (CFD), a budget-aware edge-cloud agentic framework. The edge runs a single offline captioning pass that builds a dual-track narrative index, an event-level story skeleton plus a clip-level micro-log, cached and reused across queries without re-captioning. At query time, a cloud-side MLLM reasons over the index in a story-first loop centered on a lightweight Visual-Need Router: a per-query gating module that triggers bounded keyframe retrieval only for perceptual questions (appearance, on-screen text, attribute disambiguation) and keeps temporal-structural questions in language space. The router turns visual access into a first-class, query-conditioned cost, capping per-query frame consumption regardless of video length. Experiments on long-video benchmarks demonstrate strong accuracy-efficiency trade-offs while substantially reducing online visual processing.
1 Introduction
Long-video QA must balance whole-video temporal coverage against fine-grained visual evidence under tight inference budgets. CFD addresses this tension by caching narrative memories offline and retrieving bounded visual evidence selectively at query time.
- Long-video QA is a joint modeling and resource-allocation problem because systems must preserve query-relevant memory while avoiding missed decisive events or attributes.
- Visual compression reduces token cost but risks losing long-video coverage, whereas textual translation improves scalability while losing fine-grained visual evidence.
- CFD performs a single, question-agnostic offline pass that builds reusable event-level and clip-level language memories cached across queries without re-captioning.
- At query time, the system reasons story-first, backtracks through narrative memory, and retrieves bounded keyframes only when perceptual inspection is needed.Temporal-structural questions can remain in language space when narrative memory is sufficient.
- CFD reaches competitive accuracy on Video-MME and InfiniBench while using roughly an order of magnitude fewer frames per question than prior agent-based methods.The framework supports selective rather than uniform visual access.
2 Method
CFD separates broad temporal reasoning from fine-grained visual verification through cached narrative memories and query-conditioned frame retrieval. Its story-first agentic loop enriches localized events, invokes the Visual-Need Router, and bounds visual cost per query.
- Three-Tier Memory Architecture: CFD uses a three-tier memory architecture that separates offline narrative indexing from online visual verification under a controlled visual budget.Event Memory provides global coverage, Clip Memory adds targeted local detail, and Visual Working Memory supplies pixel evidence when language is insufficient.
- Three-Tier Memory Architecture: A single offline edge captioning pass creates reusable event-level and clip-level memories, eliminating captioner calls for later queries on the same video.Event captions form a global story skeleton, while clip captions preserve micro-actions, state changes, and visible text for selective injection.
- Story-First Reasoning Loop: The answering agent first reasons from narrative memory, while localization selects one unexplored relevant event and injects overlapping clip details only when that event is explored.Selective, cumulative clip injection adds dense local information without placing the full clip memory in the reasoning context.
- Visual-Need Router: The Visual-Need Router triggers bounded keyframe retrieval for appearance, text, layout, or object-disambiguation questions, but skips frames for temporal and narrative questions.When it declines visual inspection, the loop clears the visual working memory and continues without visual cost.
- Visual-Need Router: The router makes visual access an explicit per-query cost-control gate rather than an automatic consequence of interaction depth.Applications prioritizing attribute fidelity can disable the router and always route to frames.
3 Experiment
CFD is evaluated on Video-MME and InfiniBench, including component, budget, captioner-size, routing, and iteration studies. It achieves competitive benchmark accuracy with substantially fewer online frames, while visual access remains important for fine-grained attributes.
- Experimental setup: CFD uses identical hyperparameters across benchmarks, covering Video-MME duration splits and hour-long LVBench and InfiniBench narratives without dataset-specific tuning.The evaluated benchmark setup includes Video-MME without subtitles and four InfiniBench grounding skills.
- Comparison with other methods: 67.5 overall accuracy on Video-MME uses 5.8 frames per question, exceeding reported agent-based baselines while using far fewer frames than dense inference.Qwen3-VL-32B reaches 75.9 overall at 768 frames per question; CFD targets the bounded-budget accuracy-efficiency frontier.
- Comparison with other methods: 55.10 on InfiniBench Chronological Understanding exceeds Qwen3-VL's 48.44 with 14.8 frames per question, while Scene Transitions remains competitive at 52.40 versus 53.97.CFD scores 58.90 on Global Appearance and 56.20 on Character Actions, below dense Qwen3-VL for these attribute-heavy tasks.
- Ablation study: Scaling the offline captioner from 2B to 8B improves overall accuracy from 62.6 to 67.5, whereas scaling to 32B degrades long-video accuracy to 61.0 from 63.6.The reported interpretation attributes the degradation to verbosity accumulation that weakens localization.
- Ablation study: 99.1% fewer frames than 768-frame inference reduces usage to 6.63 frames per query; routing further lowers usage to 5.76 while increasing accuracy from 67.2 to 67.5.The router improves average accuracy by 2.2 points over no-frame reasoning and slightly exceeds always-frame inference.
- Ablation study: The router's 5.76 frames per query and 67.5 accuracy outperform the outcome oracle's 0.64 frames per query and 69.5% accuracy only in visual-cost efficiency, not accuracy.The oracle is an empirical upper bound based on cases where always-frame inference corrects no-frame errors.
4 Conclusion
The paper concludes that CFD combines reusable narrative memory with sparse, query-triggered frame retrieval. Across long-video benchmarks, this design retains competitive accuracy while substantially reducing online visual cost.
- Conclusion: The framework separates reusable narrative indexing from online visual verification, making visual processing conditional on the query rather than uniformly dense.The conclusion describes the system as budget-aware and edge-cloud.
- Conclusion: CFD achieves competitive accuracy while substantially reducing online visual cost by indexing video once and retrieving sparse frames only when language alone cannot ground an answer.The conclusion presents this as the paper's central accuracy-efficiency outcome.
Limitations
CFD is limited by offline caption quality, event segmentation, sparse keyframe coverage, and its current assumptions about recording types and deployment hardware.
- Caption quality and segmentation sensitivity: Question-critical details absent from offline captions cannot be recovered by text-only backtracking, and bounded keyframe sampling may miss transient cues, small objects, overlays, or subtle motion.Failure remains possible when both textual localization and sparse visual retrieval miss the needed evidence.
- Caption quality and segmentation sensitivity: Event segmentation quality constrains localization: coarse segments bury evidence, whereas fine segments fragment coherent episodes and weaken later retrieval.The limitation is tied to the upstream segmentation stage rather than the routing policy alone.
- Egocentric and content-dependent segmentation: CFD's shot-boundary segmentation is suited to edited video but not continuous egocentric or wearable recordings, which require a different event detector.The paper identifies replacing shot-boundary detection as follow-up work.
- Deployment scope: The stated edge setup uses a server-class device or powerful desktop for captioning, so smartphone or smart-glasses deployment and end-to-end hardware measurements remain future work.The paper defines edge as inference-tier separation rather than mobile-class execution.
A Reasoning Loop Algorithm
Algorithm 1 formalizes a story-first loop that operates on dual-track narrative memory and a bounded visual working memory, invoking answering, localization, and routing under fixed budgets.
- Reasoning loop: The loop uses event memory, clip memory, and visual working memory while coordinating Answering, Localization, and Visual-Need Router agents.Iterations are bounded by T, and visual usage is controlled by Nfr and FIFO capacity Nwm.
- Reasoning loop: The algorithm's explicit controls are an iteration budget T, a per-query frame budget governed by Nfr, and a fixed FIFO working-memory capacity Nwm.These controls constrain repeated localization and on-demand frame retrieval.
B Datasets
The evaluation uses Video-MME and InfiniBench to assess multimodal understanding across varied video durations, domains, and question types.
- Video-MME contains 900 videos and 2,700 expert-annotated multiple-choice questions spanning six visual domains and 30 subcategories.The benchmark includes three duration splits: short, medium, and long.
- Video-MME evaluates short-to-long videos, with average durations ranging from 82.5 seconds to 2,385.5 seconds.
- InfiniBench targets long-form understanding in movies and television episodes with an average video duration of 53 minutes.
- The benchmarks cover temporal perception, spatial reasoning, action and object recognition, information synopsis, scene transitions, and global appearance tracking.
C Related Work
Prior long-video systems compress visual inputs, translate videos into language memories, or use agents for iterative search and retrieval. CFD’s algorithm instead organizes story-first reasoning, temporal localization, clip enrichment, visual-need routing, and on-demand frame extraction.
- Long-video research includes visual compression methods that reduce token counts and language-surrogate methods that represent videos as captions, transcripts, or textual memories.
- CFD reasoning procedure: Algorithm 1 begins with a story-first answer, then performs temporal localization and clip-enriched re-answering before visual-need routing and on-demand frame extraction.
- CFD reasoning procedure: The procedure performs multimodal re-answering after frame retrieval and returns a fallback answer when the query budget is exhausted.
- CFD emphasizes query-conditioned visual access and explicitly separates its cost-control strategy from prior language-memory and agentic pipelines.
- Agentic systems treat videos as searchable environments through iterative planning, retrieval, hierarchical memory, and query-adaptive keyframe selection.
D Results on LVBench
On LVBench, CFD achieves strong performance across long-form video understanding dimensions while using a bounded, dataset-agnostic pipeline. Its overall score is competitive with leading systems, especially on key information retrieval.
- LVBench contains 103 long-form YouTube videos totaling 117 hours, averaging 4,101 seconds per video across six domains and 1,549 questions.
- 52.9 overall places CFD above all listed open-source agent systems and within 0.4 points of AdaReTaKe-72B’s 53.3 on LVBench.CFD scores 44.4 for MemVid, 41.3 for VCA, 28.8 for VideoTree, and 29.3 for VideoAgent, while VideoLucy reaches 58.8 with a closed-source reasoning model.
- 63.9 on Key Information Retrieval is CFD’s strongest LVBench dimension, supported by timestamped textual anchors in the dual-track narrative index.
- Identical hyperparameters across benchmarks indicate that CFD’s indexing and routing strategy generalizes across substantially different durations and evaluation protocols.The comparison spans Video-MME duration splits, LVBench, and InfiniBench.
E Further Ablation Study
Further analyses examine model scale, backbone transfer, routing behavior, efficiency, and qualitative inference trajectories. They show that CFD can combine strong accuracy with selective visual use, while exposing routing and latency limitations.
- Model scale: 67.5 overall at 32B versus 49.7 at 2B shows performance scaling with cloud reasoning model size across Video-MME duration splits.The 2B-to-8B gap is 9.2 points, while the 8B-to-32B gap is 8.6 points.
- Efficiency and limitations: The Router adds an LLM call, so lower frame usage does not necessarily reduce wall-clock latency in every configuration.
- Qualitative analysis: The direct-answer example resolves a high-level question from event memory alone, avoiding localization, routing, and visual retrieval.
- Routing diagnostics: Among 1,876 decidable questions, the Router reaches 68.5% routing accuracy and 95.5% recall for frame-beneficial cases.
- Efficiency and limitations: Remaining failure modes include mixed temporal-perceptual questions, unnecessary routing, and missed evidence after localization or sparse sampling.
- Routing diagnostics: 76.98→82.73 on OCR and 70.37→75.93 on Spatial Perception show gains from adding on-demand pixels in the matched diagnostic comparison.
F More Implementation Details
CFD combines one-time, question-agnostic edge indexing with query-time narrative reasoning, routing only visually perceptual uncertainties to sparse keyframe retrieval. Its implementation includes structured memories, specialized agents, explicit confidence outputs, and diagnostic cost and router analyses.
- Captioning Agent: The offline Captioning Agent runs once per video on the edge, constructing structured Event Memory and Clip Memory rather than recaptioning for each question.Event Memory includes scene, entities, chronology, state changes, text, and retrieval anchors; Clip Memory uses a separate prompt regime.
- Answering Agent: The answering agent returns confidence, an answer option or abstention, supporting time intervals, and an evidential reason from narrative and optional visual evidence.Its prompt is provided in Table 23.
- Visual-Need Router: The Visual-Need Router selects sparse keyframes for appearance, color, text, layout, or fine-grained state cues, while retaining language-only reasoning for temporal structure and narrative questions.The router is prompted using the selected event, question, and prior failure reason to decide whether visual evidence will improve correctness.
- Inference trajectory: CFD’s story-first inference trajectory moves from coarse event memory to narrower clip memory and then invokes visual backtracking only when textual evidence remains insufficient.Figure 5 presents this progression on a Video-MME example.