PaperlayerView exact revision on arXiv

Source-linked AI summary

TRACE: Temporal Retrieval with Anchored and Convergent Evidence for Long-Horizon Video Understanding

Pengyiang Liu, Junbo Niu, Xiaoyang Hu, Zhongyue Shi, Zitian Wang, Linjiang Huang, Si Liu

arXiv:2608.22516v1cs.CVcs.CL

TL;DR

Long-horizon video answers may depend on visual evidence distributed across multiple intervals, yet existing evaluations rarely audit whether decoded frames cover all necessary evidence. The paper introduces VES-Bench and TRACE to evaluate and improve evidence-supported answering, with TRACE reaching ECA@2 = 50.7 at 98.7 frames on a same-backbone audit.

  • Problem

    Long-horizon questions requiring event ordering or counting depend on seeing every relevant interval, but decoded-frame coverage is rarely audited.

  • Method

    The paper introduces VES-Bench to audit jointly necessary evidence coverage and TRACE, a training-free agent using evidence growth trajectories and final answer replay.

  • Results

    TRACE reaches ECA@2 = 50.7 at 98.7 frames on a same-backbone audit, exceeding uniform decoding at 128 frames and approaching uniform decoding at 256 frames at lower frame cost.

  • Takeaways & Limitations

    Evidence-supported evaluation pairs answer correctness with targeting, coverage breadth, and joint correctness, while TRACE combines trajectory-based acquisition with final answer replay.

  • Takeaways & Limitations

    VES-Bench covers only Temporal Ordering and Event Counting, and TRACE depends on backbone visual recognition, with Event Counting accuracy plateauing near 40% on Gemini-3-Flash.

Abstract

from arXiv · show

A long-video answer is evidence-supported only when the frames decoded from the video cover every event the answer depends on. Existing evaluations score final-answer correctness or predicted evidence intervals, but the frames a method decodes before answering are rarely audited, so correct answers can still rest on incomplete observation. We introduce VES-Bench, a 600-question benchmark of Temporal Ordering and Event Counting items over 348 public long videos. Each item carries a jointly necessary set of evidence intervals, letting us audit at three strictness levels whether a method's decoded frames cover every one of them. We also propose TRACE, a training-free agent that grounds answers in raw visual clips, builds an evidence bundle round by round, and stops only when the answer stabilises as the bundle grows and a final pass over the same clips returns the same answer. Under a same-backbone audit, TRACE answers 50.7% of questions correctly with at least two decoded frames inside every evidence interval, at 98.7 frames per question: over 10 points above uniform decoding at 128 frames (40.2%), and within 2.6 points of uniform decoding at 256 frames at 0.39x its frame cost, while reaching the highest answer accuracy in the audit (63.5%). TRACE also stays competitive on Video-MME (86.1), LVBench (75.6), and LongVideoBench (75.1).

1 Introduction

Long-horizon questions often require visual evidence distributed across multiple video intervals, so answering correctly depends on observing every relevant interval. TRACE and VES-Bench address this gap by combining evidence-grounded stopping with audits of jointly necessary evidence coverage.

  • Temporal ordering and event counting require seeing every relevant interval before visual evidence can distinguish the answer choices.
  • TRACE is a training-free agent that grounds answers in raw visual clips and stops when the answer trajectory stabilises as the evidence bundle grows.
  • VES-Bench audits whether decoded frames cover each question’s jointly necessary evidence intervals at three strictness levels.
  • At matched budgets on a shared backbone, trajectory-based stopping yields higher evidence-supported correctness than fixed-sampling and agent baselines.

2 Related Work

Prior long-video work scales processing, searches or grounds temporal evidence, and evaluates answers through accuracy, probes, predicted intervals, or multi-clip questions. TRACE instead connects test-time observation control with an audit of the source frames actually decoded.

  • Long-Horizon Video Understanding: Long-video methods address input scale through longer context, visual-token compression, or sparse memory, including systems that process more video in one pass.
  • Agent-based reasoning and temporal grounding: Figure 2 depicts TRACE converting the full video into visual anchors, constructing prefix bundles, and using reconciliation actions to adjust observed evidence.
  • Agent-based reasoning and temporal grounding: Agent-based frameworks treat long-video understanding as multi-step observation, search, tool use, or stopping decisions.
  • Agent-based reasoning and temporal grounding: TRACE controls observation through a trajectory-level stopping criterion, while VES-Bench audits decoded source-frame timestamps against annotated evidence intervals.
  • Long-video QA benchmarks: Grounding-aware evaluations commonly score predicted evidence intervals or clues, or construct questions requiring cues from multiple non-overlapping clips.

3 Method

TRACE builds raw-visual evidence bundles incrementally and analyzes how answers change as new anchors are added. Its Trajectory Reconciler selects a minimal stable, evidence-sufficient prefix, then final answer replay verifies the stopping decision.

  • TRACE initializes a global storyboard and iteratively proposes temporal candidates, extracts anchors, forms prefix bundles, and reconciles the resulting answer trajectory.
  • A visual anchor is a short raw-visual clip localized within a candidate interval, with an observation strategy that adapts frame rate and spatial resolution.
  • Constructing the Evidence Growth Trajectory: Prefix bundles create a monotonic sequence whose consecutive inputs differ only by newly added visual evidence.
  • Constructing the Evidence Growth Trajectory: Evidence Assembly independently evaluates each prefix from raw clips, the question, and answer options, recording an answer, status, supported facts, and remaining evidence needs.
  • The Trajectory Reconciler seeks the earliest prefix unaffected by later flips, degradation, or contradiction while leaving no unmet evidence need.
  • TRACE can STOP, DROP, REFINE, or EXPAND evidence, and final answer replay checks the accepted stable prefix using the same raw clips.

4 VES-Bench

VES-Bench evaluates whether long-video answers are supported by the visual evidence actually observed, using jointly necessary intervals across Temporal Ordering and Event Counting questions. It combines a semi-automatic construction pipeline with an audit that measures targeting, interval coverage, and evidence-supported correctness.

  • VES-Bench contains 600 four-choice questions from 348 public Ego4D and YouTube videos, evenly split between Temporal Ordering and Event Counting.
  • A question is evidence-closed when removing any single interval leaves at least two answer options visually compatible with the remaining evidence.
  • Temporal Ordering items require one evidence interval per ordered event, while Event Counting items list every target-event occurrence.
  • The benchmark is built through proposal, visual verification, leave-one-out sufficiency, shortcut, and adjudication stages.
  • The average video lasts 434 seconds, and each question carries 3.4 evidence intervals averaging 16.2 seconds, making evidence localisation central.
  • The audit logs distinct decoded source-frame timestamps and reports targeting, coverage breadth, joint correctness, and supplied-frame budget without ground-truth intervals during inference.

5 Experiments

TRACE improves evidence-supported correctness over same-backbone uniform and agent baselines while remaining competitive on standard long-video benchmarks. Its gains combine targeted evidence acquisition, broader interval coverage, and final answer replay.

  • Public benchmarks: TRACE reaches 86.1 on Video-MME, 75.6 on LVBench, and 75.1 on LongVideoBench using Gemini-2.5-Pro.These answer-only benchmarks serve as a competitiveness check, while evidence-supported analysis uses VES-Bench.
  • VES-Bench main results: 40.2 ECA@2 at 128 uniform frames rises to 53.3 at 256 frames, while Acc decreases from 57.5 to 56.7.Cov@2 increases from 74.7 to 95.5, showing that additional uniform frames strengthen coverage without improving accuracy beyond this budget.
  • VES-Bench main results: 50.7 ECA@2 at 98.7 frames makes TRACE more evidence-supported than AVP and Uniform-128 at comparable budgets.TRACE records Cov@2 = 78.5 and reaches within 2.6 ECA@2 points of Uniform-256 at 0.39× its frame cost.
  • VES-Bench main results: TRACE has the highest ECA@𝑘 among same-backbone methods at comparable frame budgets across 𝑘=1, 2, 3.Its ECA@2 advantage is accompanied by lower Uncov-Corr than AVP and Uniform-128, at 12.8 versus 19.3 and 17.3.
  • VES-Bench main results: 78.0% of correctly answered, Cov@2-satisfied TRACE predictions change when one evidence interval is masked, versus 6.9% when random non-evidence segments are masked.Event Counting accuracy plateaus near 40% across Gemini-3-Flash budget points, indicating a recognition-related residual error.
  • Ablations: Removing evidence growth reduces ECA@2 from 50.7 to 27.5, while removing final answer replay reduces ECA@2 by 8.7 points.The ablation separates trajectory-based acquisition for targeting and coverage from replay for additional evidence-supported correctness.

6 Conclusion

TRACE and VES-Bench target evidence-supported long-horizon video understanding by pairing evidence auditing with trajectory-based stopping. TRACE reaches ECA@2 = 50.7 at 98.7 frames on a same-backbone audit.

  • TRACE and VES-Bench address evidence-supported long-horizon video understanding.
  • VES-Bench pairs answer correctness with targeting, coverage breadth, and joint correctness across three audit strictness levels.
  • 50.7 ECA@2 is achieved by TRACE at 98.7 frames on a same-backbone audit.

Limitations

The study’s scope, backbone dependence, recognition limits, computational overhead, and annotation process constrain how its results should be interpreted.

  • VES-Bench covers only Temporal Ordering and Event Counting, leaving looser evidence-boundary families for future work.
  • The headline audit uses Gemini-3-Flash because it requires source-frame timestamps and full inference-loop control.
  • TRACE depends on backbone visual recognition, with Event Counting accuracy plateauing near 40% from 64 to 256 frames.
  • Trajectory inference adds prefix-bundle reasoning calls, although rule-triggered STOP keeps retained anchors small.
  • Human judgment remains part of evidence-set-closure annotation despite the semi-automatic pipeline.

Ethics Statement

The benchmark uses videos from Ego4D and public YouTube, and releases annotations, evidence boundaries, observation logs, and source videos under stated licensing terms.

  • VES-Bench draws videos from Ego4D and public YouTube.
  • Annotations, evidence-set boundaries, and per-question observation logs are released under CC BY-NC-SA 4.0 for non-commercial research.
  • Source videos accompany the release so observation logs can be replayed frame by frame.
  • Users are bound by the release terms and the licenses of the source datasets listed in Appendix K.

A Implementation Details and Per-question Cost

TRACE uses prompt-based modules and a trajectory controller to assemble visual evidence, manage observation actions, and account for per-question cost under controlled backbone audits.

  • Implementation Details: TRACE wraps prompt-based test-time modules around the evaluation backbone and exchanges structured records across search and control stages.
  • Implementation Details: Final answer replay consumes only raw visual clips, the question, and answer options, then emits an answer alone.
  • Per-question Cost: TRACE consumes 98.7 frames per question, with frame budget serving as a proxy for observation cost because visual input dominates each call.
  • Trajectory Reconciler Pipeline: The Trajectory Reconciler consumes the evidence trajectory, unexplored segments, and remaining budget, emitting STOP, DROP, REFINE, or EXPAND.
  • Trajectory Reconciler Pipeline: Deterministic flags propose STOP from a minimal stable prefix, while unresolved control choices invoke one backbone call.
  • Trajectory Reconciler Pipeline: DROP removes conflicting anchors, REFINE changes an anchor’s observation strategy, and EXPAND proposes unexplored candidate intervals.
  • Per-question Cost: The audit holds visual recognition fixed by reproducing agent baselines on a shared backbone and logging frames through a shared decoding wrapper.
  • Per-question Cost: Confidence intervals use bootstrap resampling over 600 questions, with paired bootstrap comparisons and video-cluster robustness checks.

H Annotation Pipeline Details

VES-Bench is constructed through staged visual verification, sufficiency testing, shortcut rejection, and adjudication, with explicit agreement and closure guarantees. The resulting audit excludes items solvable from language priors, chance, or coarse gist, while not claiming minimal evidence sets.

  • Pipeline: 600 accepted items remain from 1,024 proposals after five stages of visual verification, leave-one-out probing, shortcut testing, and adjudication.The candidate pool contracts to approximately 820, 700, 640, and finally 600 items; the combined accept rate is 58.6%.
  • Pipeline: Stage 2 removes the most items overall, Stage 3 is the largest closure-specific filter, and Stage 4 removes items solvable from text or gist alone.Stage 4 tests language-only and gist-only shortcuts after visual grounding.
  • Agreement: 91.3% agreement is reported between human and Claude 3.5 Sonnet on leave-one-out sufficiency, while boundary IoU averages 0.83 over accepted items.The shortcut probe rejects an item when either language-only or gist-only selection exceeds the 25% chance baseline on a held-out repeat.
  • Closure guarantee: The closure guarantee requires human and independent vision-LLM agreement that removing any single interval leaves at least two visually compatible options.It does not guarantee that the annotated evidence set is minimal sufficient.
  • Shortcut validation: A held-out text-only GPT-4-Turbo scores 25.3% overall on VES-Bench, within the bootstrap interval of the 25% random baseline.Scores are 25.0% for Temporal Ordering and 25.7% for Event Counting.

J Ablation Across Density Levels

TRACE’s coverage and evidence-supported correctness decline as the required evidence density increases across all ablations. Removing the growth trajectory causes the largest coverage loss at density k=3.

  • Density effects: Cov@k and ECA@k both fall as k rises from 1 to 3 for every TRACE variant.The ablation covers density levels k=1, 2, and 3.
  • Component ablation: At k=3, removing the growth trajectory reduces Cov@k from 60.3 to 30.5, the largest share loss among ablated components.The comparison is reported for the TRACE component ablation on VES-Bench.

K Dataset Licenses

VES-Bench materials are released for non-commercial research under CC BY-NC-SA 4.0, alongside the licenses governing the source datasets.

  • Release terms: The VES-Bench annotations and source videos are released under CC BY-NC-SA 4.0 for non-commercial research.The release includes questions, options, evidence-set boundaries, and per-question observation logs; reproducing the audit requires the source videos and exact decoded timestamps.
  • Source datasets: Users downloading the release are also bound by the licenses of the source datasets listed in Table 8.Table 8 reports the source-dataset licenses.
Loading 2608.22516v1…