Source-linked AI summary
Finding the Right Evidence: Factor-Guided Coarse-to-Fine Reasoning for Long Videos
Baixuan Xu, Yinyui Xu, Tianshi Zheng, Zhaowei Wang, Weiqi Wang, Haochen Shi, Jiayu Liu, Qing Zong, Xiyu Ren, Xinyu Geng, Zhitao He, Yangqiu Song
TL;DR
Long-video QA often retrieves context relevant to the question but misses sparse evidence that separates plausible answers. PACE uses question-derived factors to build an answer-independent evidence index, then uses candidate-derived contrastive cues for verification. It reaches 42.6% accuracy and 66.9% annotated-cue recall on MMR-V, with consistent gains over DVD across four broader benchmarks.
Problem
Long-video QA lacks reliable recovery of sparse, decision-critical evidence that discriminates among plausible answers, despite retrieving question-relevant context.
Method
PACE builds a clip-level evidence database from question-derived factors without candidate answers, then queries it with candidate-derived contrastive cues.
Results
42.6% accuracy and 66.9% annotated-cue recovery on MMR-V accompany consistent gains over DVD on LVBench, Video-MME, EgoSchema, and LongVideoBench.
Takeaways & Limitations
Option-aware evidence acquisition transfers beyond the diagnostic MMR-V setting and is associated with improved evidence recovery rather than answer-side priors alone.
Takeaways & Limitations
PACE has higher inference latency and computational cost than end-to-end models because it requires multiple model-interaction rounds and tool invocation.
Abstract
from arXiv · showhide
While LVLMs rapidly improve, long-video question answering still remains challenging: relevant evidence is sparse, and question-relevant context often fails to provide cues that discriminate the correct answer from plausible alternatives. Diagnostic analysis on a manually annotated subset of MMR-V shows that prior agentic systems substantially improve cue retrieval over direct VLM inference yet fail to achieve a corresponding gain in answer accuracy, indicating that the bottleneck lies in option-discriminative evidence rather than topical relevance alone. We propose PACE (Progressive Acquisition of Critical Evidence), a factor-guided framework for long-video evidence acquisition. PACE proceeds in two stages: it first indexes clip-level descriptions guided by question-derived factors without observing the candidate answers; it then uses the candidate answers to derive contrastive cues and queries the index for verification. On MMR-V with the open-source Qwen3-VL backbone, PACE achieves 42.6% accuracy, outperforming direct inference and prior agentic baselines including Deep Video Discovery (DVD). On the same diagnostic subset, PACE recovers 66.9% of the annotated cues, providing empirical evidence that its gains are associated with improved evidence recovery rather than stronger answer-side priors alone. Consistent gains over DVD on LVBench, Video-MME, EgoSchema, and LongVideoBench suggest that option-aware evidence acquisition transfers beyond MMR-V. Code is available at https://github.com/HKUST-KnowComp/PACE.
1 Introduction
Long-video QA is bottlenecked by sparse evidence that distinguishes plausible answers, not merely by the amount of question-relevant context. PACE addresses this through question-conditioned indexing followed by candidate-aware verification, improving cue recovery and accuracy.
- Motivation: Decision-critical evidence is sparse and visually subtle, so question-relevant context may fail to discriminate among plausible answers.Current pipelines target question relevance without directly targeting option discrimination.
- Motivation: The dachshund example shows that abundant clips of the boy with the dog do not distinguish four candidate explanations, whereas a brief photograph provides the decisive cue.The photograph shows the ex-girlfriend with the same dog and is easily missed by question-only retrieval.
- Diagnostic Analysis: 56.2% cue recovery for DVD versus 43.2% for direct Qwen3-VL and 30.6% for VideoTree did not yield higher accuracy, with all systems reaching roughly 54%.The diagnostic indicates that recovered evidence was relevant to the question but not consistently option-discriminative.
- Approach: PACE indexes clip descriptions using question-derived factors before seeing candidate answers, then derives contrastive cues from the candidates for evidence verification.Its factors cover entities, actions, attributes, and temporal anchors, while the two stages preserve a write-without-options, read-with-options asymmetry.
- Results: 42.6% accuracy with Qwen3-VL versus 39.5% for DVD and 66.9% annotated-cue recovery show PACE’s gains on MMR-V.PACE also yields consistent gains over DVD on LVBench, Video-MME, EgoSchema, and LongVideoBench.
2 Related Work
Long-video research has progressed from larger and compressed visual representations toward agentic exploration and retrieval. PACE differs from these approaches by separating question-conditioned indexing from candidate-aware verification.
- 2.1 Long Vision-Language Models: Long-video vision-language models have progressed from frame aggregation toward hierarchical compression, dynamic token pruning, and increasingly scaled foundation models.The related work also notes specialized architectures for long-video understanding.
- 2.2 Agentic Long-Video Reasoning: Agentic frameworks actively explore videos through dense-caption retrieval, query-adaptive hierarchies, and state-action browsing.Deep Video Discovery is identified as the most directly comparable open-source agentic system.
- 2.2 Agentic Long-Video Reasoning: Unlike prior retrieval systems, PACE conditions evidence acquisition on the question for indexing and on candidate answers for contrastive verification.This partition addresses the option-blind retrieval gap quantified in the introduction.
3 The PACE Framework
PACE separates long-video evidence acquisition into question-conditioned indexing and option-aware verification. It builds an answer-independent evidence database from question-derived factors, then uses candidate-derived contrastive factors to retrieve and test discriminative evidence.
- PACE separates evidence acquisition into a question-conditioned indexing stage and an option-aware verification stage.The stages deliberately differ in whether they can observe the candidate answers.
- The evidence database M stores clip identifiers, textual evidence records, and supported question-derived factors.It connects both stages while preserving the distinction between indexing and verification.
- Question-Conditioned Evidence Indexing: Stage 1 extracts a compact factor set from the question covering entities, actions, relations, attributes, and temporal anchors.These factors condition clip-level descriptions that are embedded and inserted into M without observing candidate answers.
- Option-Aware Verification: Stage 2 derives contrastive criteria from the question and candidate options to test whether indexed evidence distinguishes among alternatives.Each criterion identifies an option subset that it can rule in or out.
- Option-Aware Verification: A reasoning agent retrieves evidence from M using contrastive factors and invokes frame inspection when textual records leave a factor unresolved.PACE adds QUERYDECOMPOSE and EVIDENCERETRIEVE to inherited browsing, search, and inspection tools.
- PACE keeps M independent of candidate answers, avoiding answer-side priors during indexing and rebuilding the database for each option.Question-anchored factors determine what to record, whereas option-anchored factors determine what to test.
4 Experiments
PACE is evaluated on MMR-V and broader long-video benchmarks through controlled comparisons, diagnostics, robustness tests, ablations, and factor-budget analyses. It improves accuracy and evidence recovery over DVD, while results show that question-conditioned indexing and option-aware verification are complementary and that compact factor sets work best.
- Main Results on MMR-V: 42.6% overall accuracy on MMR-V exceeds VideoTree at 34.4% and DVD at 39.5%.The gain over DVD is +3.1 points in both implicit- and explicit-reasoning subsets.
- Main Results on MMR-V: PACE’s largest category-level gains over DVD are +11.8 in Life and +5.8 in TV, while it trails DVD in Art and Philosophy.Art scores 28.1 versus 28.8, and Philosophy scores 25.6 versus 32.6.
- Needle-Recall Diagnosis: PACE reaches 66.9% needle recall and 57.0% accuracy on the diagnostic subset, whereas DVD reaches 56.2% recall and 54.0% accuracy.DVD’s recall advantage over direct Qwen3-VL does not produce a corresponding accuracy gain, while PACE aligns higher recall with higher accuracy.
- Transfer Results: PACE improves over DVD on all four transfer benchmarks: +0.3 on LVBench, +0.5 on LongVideoBench, +0.2 on EgoSchema, and +1.7 on Video-MME.The largest transfer gain appears on Video-MME.
- Backbone Robustness: Across three backbone configurations, PACE outperforms DVD by 3.4, 2.1, and 3.1 absolute points, respectively.The improvement has the same direction in every tested configuration.
- Ablation Studies: Removing question-conditioned indexing lowers accuracy to 41.8%, removing option-aware verification lowers it to 41.5%, and their combined losses total 1.9 points.The two stages therefore appear complementary rather than independently additive.
- Factor Impact on the Indexing Stage: Accuracy peaks with one or two coarse factors, then falls to 36.1% at three factors and 35.0% at five factors.The paper attributes this decline to narrower captioner attention and premature cue filtering under a fixed context window.
- Tool Use and Cost: Most episodes terminate in exactly four steps, and reasoning consumes more tokens per query than indexing consumes per clip.83.4% of episodes use four steps; average reasoning cost is 68,164 prompt and 10,262 completion tokens per query versus 17,584 and 1,109 during indexing.
5 Conclusion
PACE addresses option-blind retrieval in long-video question answering through a write-without-options, read-with-options asymmetry. It attains 42.6% accuracy and 66.9% needle recall on MMR-V, with consistent gains across four broader long-video benchmarks and three backbone configurations.
- PACE addresses the option-blind retrieval bottleneck through a write-without-options, read-with-options asymmetry.The indexing stage builds an evidence database from question-derived factors before observing candidate answers, while verification queries it using contrastive cues derived from those candidates.
- 42.6% accuracy and 66.9% needle recall are achieved on MMR-V.These results are reported with the Qwen3-VL backbone.
- Consistent gains are reported across four broader long-video benchmarks and three backbone configurations.
Limitations
PACE has efficiency, task-scope, backbone-dependence, and evaluation-setting limitations. Its multi-round agentic process increases latency and computational cost, while its evidence-acquisition focus and multiple-choice design constrain broader applicability.
- PACE incurs higher inference latency and computational cost than end-to-end models because it uses multiple model interactions and tool invocations.The paper suggests distilling reasoning trajectories into lighter-weight models to improve efficiency.
- PACE’s gains may be smaller when the main bottleneck is abstract semantic, symbolic, or thematic interpretation rather than sparse evidence acquisition.Narrowing context around query-relevant factors may also filter out subtle cues weakly aligned with the query’s surface form.
- PACE depends on the underlying VLM’s perception and instruction-following ability, so severe visual ambiguity, OCR failures, or weak grounding may propagate through reasoning.
- The current implementation is evaluated primarily in multiple-choice settings with an explicitly provided candidate answer space.How to instantiate the hypothesis space for open-ended QA remains an open question.
Ethics Statement
The work uses public video-understanding benchmarks and reports no collection or annotation of new private video data involving human subjects. It also acknowledges inherited model-bias, computational-impact, and potential-misuse concerns.
- The study uses publicly available MMR-V, LVBench, VideoMME, and EgoSchema benchmarks under their usage licenses and terms of service.
- No new personally identifiable information or private video data involving human subjects was collected or annotated for this study.
- PACE may inherit social biases or stereotypes from the pretrained Vision-Language Models on which it operates.The paper advises caution and safety filtering for real-world deployment.
- Iterative tool use and multi-turn inference consume more computational resources and energy than single-pass models.
- Advanced video understanding could be repurposed for unauthorized surveillance or privacy intrusion.The paper states that PACE is developed for information retrieval and improving video accessibility.
B Detail of Dataset
The evaluation covers MMR-V and four broader long-video benchmarks spanning diverse durations, extreme long-video understanding, referring reasoning, and egocentric comprehension. Baselines include proprietary and open-source VLMs alongside agentic frameworks.
- Video-MME: Video-MME’s Long subset without subtitles contains 300 videos and 900 questions, covering videos from 11 seconds to 1 hour.
- LVBench: LVBench tests long-term memory and information extraction over content spanning several hours, with 1,549 questions across 103 videos.
- LongVideoBench: LongVideoBench provides 6,678 multiple-choice questions over 3,763 videos, while this study evaluates a 564-question, 188-video validation subset.
- EgoSchema: EgoSchema’s validation set contains 500 videos and an equivalent number of associated questions.
- Baselines: Baselines span proprietary VLMs, open-source VLMs, and open-source agentic frameworks including VideoTree and Deep Video Discovery.Deep Video Discovery is the primary comparison system.
D Full Comparison on General Benchmarks
This section presents a comprehensive comparison of PACE with additional foundation models and video-agent frameworks across broader long-video benchmarks, with detailed results in Table 7.
- PACE is compared against additional foundation models on broader long-video benchmarks.
- The evaluation also compares PACE with video-agent frameworks.
- Detailed performance results are reported in Table 7.
E Analysis of Limitations in Art and Philosophy Categories
PACE performs relatively worse on Art and Philosophy questions because its concrete, entity-centric evidence acquisition can conflict with abstract, metaphor-driven reasoning. Retrieved captions may therefore steer the model toward surface-level evidence, as illustrated by the metaphor-understanding case study.
- Limitations: PACE has a relative performance limitation on Art and Philosophy questions.The limitation is associated with the abstract and metaphor-driven nature of questions in these categories.
- Mechanism: PACE decomposes queries into entity- and event-centric factors, which benefits questions grounded in identifiable factual cues.Art and Philosophy questions more often depend on long-range themes, symbolism, or metaphorical connections rather than explicit entities.
- Failure mode: Concrete, decomposed captions may steer the model toward surface-level evidence and away from intended abstract reasoning.This degradation is illustrated by the metaphor-understanding case study in Table 14.
G Impact of Clip Duration
Clip duration materially affects PACE’s performance during contextual environment construction. Ten-second clips provide the best reported trade-off between contextual fragmentation, retrieval noise, caption coarseness, and temporal specificity.
- Ablation results: 42.6% accuracy is achieved with T = 10s, outperforming 5s at 37.2% and 20s at 38.1%.The study identifies 10s as the optimal clip duration.
- Interpretation: Ten-second clips balance context preservation and temporal specificity for evidence discovery.Shorter clips fragment context and increase retrieval noise, whereas longer clips produce overly coarse captions.
- Experimental setting: The clip-duration ablation is conducted on MMR-V during contextual environment construction.
I Case Study
The case studies illustrate PACE’s successful and failed evidence-acquisition trajectories across factual, multi-step, and metaphor-understanding questions. They show both concrete retrieval of decisive evidence and failure when the question requires abstract interpretation.
- Case-study coverage: PACE’s case studies include correct standardized four-step and multiple-step trajectories, alongside failed standardized and metaphor-understanding trajectories.Accurate and useful content is distinguished from inaccurate or irrelevant content using the stated color coding.
- Factual evidence retrieval: For the card-passing question, query decomposition identifies card function and hidden-device presence as available evidence types.The decisive factor concerns whether the card’s purpose derives from intrinsic properties or a separate attached device.
- Factual evidence retrieval: Evidence retrieval finds hands manipulating a circuit board and a small chip under blue lighting at 00:02:15–00:02:19.A chip attached to the playing card is also identified at 00:00:12–00:00:16 as the hidden device.
- Retrieval failure: For the finger-trick question, global browsing finds no explicit description of the specific trick in the provided clips.
- Metaphor understanding: The mirror case retrieves a metaphorical connection between the adult man and his younger self through the reflection.The opening sequence shows a child appearing in the ornate mirror’s reflection.