TRACE: Temporal Retrieval with Anchored and Convergent Evidence for Long-Horizon Video Understanding

• Temporal ordering and event counting require seeing every relevant interval; omitting one can leave answer choices visually indistinguishable, so locating a single relevant moment is insufficient (Ben-Ami et al., 2025; Liu et al., 2026a; Yu et al., 2025; Kwan et al., 2026).Long-Horizon Video Understanding questions often distribute evidence across multiple intervals (Mangalam et al., 2023; Wu et al., 2024; Wang et al., 2025a; Zhou et al., 2025; Li et al., 2026b).Source: §1 · p. 2
• Fixed-sampling and long-context methods cannot recover intervals missed by their initial schedule, whereas agent-based methods iteratively search for relevant clips (Chen et al., 2025b; Shen et al., 2025; Shu et al., 2025; Song et al., 2024; He et al., 2024; Ataallah et al., 2024; Tang et al., 2025; Wang et al., 2024a; Yang et al., 2025; Wang et al., 2025d; Li et al., 2026a; Wang et al., 2025c; Zuo et al., 2025; Yuan et al., 2025; Xu et al., 2026).The introduction identifies complementary agent blind spots: targeting intervals without covering them all, or spending more frames without targeting the question.Source: §1 · p. 2
• TRACE is a training-free agent that retains raw visual clips and stops exploration when answers stabilize as its evidence bundle grows, with a final pass over the same clips verifying the answer.This contrasts with text-chain agents that lose visual grounding and visual-only agents that stop using self-reported confidence (Wang et al., 2024a; Yang et al., 2025).Source: §1 · p. 2
• VES-Bench contains 600 four-choice Temporal-Ordering and Event-Counting questions with jointly necessary evidence sets and audits decoded-frame coverage at three strictness levels.The audit makes evidence-supported correctness measurable for each prediction, addressing evaluations that score answer correctness or predicted intervals but do not audit frames actually decoded.Source: §1 · p. 2
• On a shared backbone, TRACE combines targeted evidence gathering with density at an agent-comparable frame budget, surpasses agent baselines, approaches dense uniform decoding at a fraction of its frame cost, and remains competitive on three long-video benchmarks.The introduction attributes evidence-supported correctness at matched budgets to trajectory-based stopping rather than raw budget.Source: §1 · p. 2

Long-Horizon Video Understanding

• Long-horizon video methods address input scale through longer context, visual-token compression, or sparse memory, building on general vision-language models.Examples include LongVILA, LongVU, and Video-XL for single-pass context, and MovieChat, MA-LMM, Goldfish, and InternLM-XComposer2.5-OmniLive for compact memory or retrieval representations (Chen et al., 2025b; Shen et al., 2025; Song et al., 2024; He et al., 2024; Ataallah et al., 2024; Zhang et al., 2024).Source: §2 · p. 2

Agent-based reasoning and temporal grounding

• Agent frameworks choose where and how to observe and when to stop, while temporal-grounding methods predict query-relevant intervals; TRACE sits between these lines.The cited agent frameworks include VideoAgent, VideoTree, VCA, VideoMind, DeepVideoDiscovery, VideoLucy, AVP, and LensWalk (Wang et al., 2024a, 2025c; Yang et al., 2025; Liu et al., 2026b; Zhang et al., 2025; Zuo et al., 2025; Wang et al., 2025d; Li et al., 2026a); temporal grounding reports interval metrics such as tIoU and Recall@k (Gao et al., 2017; Lin et al., 2023).Source: §2 · p. 3
• TRACE controls observation with trajectory-level stopping, while VES-Bench checks decoded source-frame timestamps against annotated evidence intervals.The audit operationalizes grounding through observed-frame logs rather than answer-level hallucination probes.Source: §2 · p. 3

Long-video QA benchmarks

• Prior grounding-aware protocols score predicted intervals or clues, or questions requiring fusion across non-overlapping clips; VES-Bench instead logs frames actually decoded before an answer.This complements evaluations that probe grounding through hallucination-style questions, but measures observed-frame logs (Xiao et al., 2024; Qin et al., 2025; Ben-Ami et al., 2025).Source: §2 · p. 3

3 Method

• Algorithm 1 iterates Temporal Proposal, Evidence Assembly, and Trajectory Reconciler over anchors and prefix bundles until replay confirms a stable answer or the budget runs out.It initializes a coarse storyboard and unexplored segments, updates state with DROP, REFINE, or EXPAND, and otherwise returns a replayed least-conflicted fallback prefix.Source: §3 · pp. 3–4

3.1 From Full Video to Visual Anchors

• TRACE forms raw-clip visual anchors through temporal proposal, local extraction, and prioritization of clips likely to distinguish choices or fill evidence gaps.A 32-frame uniform storyboard previews the full video, but its frames are part of the VLM input; anchor observation strategies adapt frame rate and resolution to the evidence sought.Source: §3.1 · p. 4

3.2 Constructing the Evidence Growth Trajectory

• Evidence Assembly evaluates each growing prefix independently from raw clips, the question, and options, so trajectory changes reflect newly added visual evidence.Re-supplying each whole bundle costs the sum of prefix frame counts; four anchors of 5, 4, 4, and 3 frames require 43 frames across four calls.Source: §3.2 · pp. 4–5

3.3 Trajectory Reconciliation and Control

• TRACE accepts the earliest prefix only when later anchors do not flip or weaken the answer, contradict visual facts, or reveal unmet evidence needs.The reconciler can DROP invalid or redundant anchors, REFINE observations, or EXPAND to unexplored intervals; STOP is confirmed by replay, while budget exhaustion returns a least-conflicted prefix without claiming sufficiency.Source: §3.3 · p. 5

4 VES-Bench

• VES-Bench contains 600 four-choice questions from 348 public Ego4D and YouTube videos, evenly split between Temporal Ordering and Event Counting.The Ego4D videos are attributed to Grauman et al. (2022).Source: §4 · p. 5

4.1 Evidence-Closed Questions

• Each evidence-closed question has jointly necessary intervals: ordering items need one interval per event, while counting items need every target-event occurrence.Removing any single interval leaves at least two answer options visually compatible with the remaining evidence.Source: §4.1 · p. 5

4.2 Construction and Verification

• A five-stage semi-automatic pipeline proposes, verifies, probes, and adjudicates items, accepting 600 from a candidate pool of 1,024.Visual grounding and leave-one-out sufficiency are checked, and items solvable from the question and options alone or with a global video summary are rejected.Source: §4.2 · p. 6

4.3 Audit Metrics

• The audit uses decoded source-frame timestamps to measure targeting (EP, AR), coverage breadth (Cov@k), and correct answers with covered evidence (ECA@k).Cov@2 and ECA@2 are headline measures; Cov@1/Cov@3 and ECA@1/ECA@3 provide sensitivity bounds. The visual budget Fr charges frames supplied across backbone calls, including repeated timestamps.Source: §4.3 · pp. 6–7

5.1 Public Benchmarks

• TRACE scores 86.1 on Video-MME, 75.6 on LVBench, and 75.1 on LongVideoBench as an answer-only competitiveness check.TRACE uses Gemini-2.5-Pro to match AVP’s reported backbone.Source: §5.1 · p. 8, Table 1 · p. 8

5.2 VES-Bench Main Results

• From 128 to 256 uniform frames, ECA@2 rises 40.2→53.3 and Cov@2 rises 74.7→95.5, while Acc falls 57.5→56.7.Across the sweep, uniform decoding cannot condition on question content, and added frames can strengthen coverage without improving accuracy.Source: Table 2 · p. 8, §5.2 · p. 9
• VCA’s targeting (AR = 1.23) yields Cov@2 = 5.8 and ECA@2 = 3.3, while AVP’s 101.3 frames yield ECA@2 = 39.7, near Uniform-128’s 40.2.The baselines fail differently: VCA targets without covering, whereas AVP covers without targeting.Source: Table 2 · p. 8, §5.2 · p. 9
• At 98.7 frames, TRACE reaches ECA@2 = 50.7, above Uniform-128 (40.2) and within 2.6 points of Uniform-256 (53.3) at 0.39× its frame cost.VES-Bench comparisons use Gemini-3-Flash as the shared backbone to isolate observation-policy effects.Source: §5.2 · pp. 8–9, Table 2 · p. 8, Figure 4 · p. 9
• Masking a necessary interval flips 78.0% of 304 correct-and-covered answers, versus 6.9% after equal-sized non-evidence masking; TRACE’s per-family gains concentrate on Temporal Ordering.Event Counting accuracy plateaus near 40% across Gemini-3-Flash budget points, indicating a recognition-related residual error that more frames alone do not address.Source: §5.2 · p. 10, Table 3 · p. 10

5.3 Ablations

• Removing evidence-growth trajectory lowers EP from 29.1 to 19.4, ECA@2 from 50.7 to 27.5, and Acc from 63.5 to 55.0.Fixed-round stopping and removing EXPAND retain EP between 25 and 27 but lower Cov@2 below 50; removing final answer replay reduces ECA@2 by 8.7 points while leaving EP and Cov@2 essentially unchanged.Source: §5.3 · p. 10, Table 4 · p. 10

5.4 Qualitative Analysis

• On a three-scene ordering item, TRACE keeps acquiring until the late frog interval is substantively observed, then accepts a stable prefix only after replaying its raw clips.The correct order is unsupported unless the late interval is also observed, and increasing density requires revisiting it substantively rather than glancing at it once.Source: §5.4 · p. 11

6 Conclusion

• ECA@2 = 50.7 at 98.7 frames on a same-backbone audit for TRACE.Source: §6 · p. 11
• VES-Bench pairs answer correctness with targeting, coverage breadth, and joint correctness, audited at three strictness levels.Source: §6 · p. 11
• TRACE grounds its stopping decision in the evidence growth trajectory and final answer replay.Source: §6 · p. 11

• VES-Bench covers only Temporal Ordering and Event Counting, leaving families with looser evidence boundaries for future work.Source: Limitations · p. 11
• The headline audit uses Gemini-3-Flash for per-question source-frame timestamps and full inference-loop control; the authors expect qualitative ordering to hold on other backbones.Source: Limitations · p. 11
• TRACE depends on backbone visual recognition, and Event Counting Acc plateaued near 40% from 64 to 256 frames on Gemini-3-Flash.Additional frames do not address this recognition bottleneck.Source: Limitations · p. 11
• Trajectory inference adds prefix-bundle reasoning calls that scale with retained anchors, kept small by rule-triggered STOP.Per-question backbone-call breakdown and frame budget are in Appendix A.Source: Limitations · p. 11
• Evidence-set closure annotation depends on human judgment; the semi-automatic pipeline reduces but does not remove this dependence.Source: Limitations · p. 11

Ethics Statement

• VES-Bench uses Ego4D (Grauman et al., 2022) and public YouTube videos; annotations, evidence boundaries, and observation logs are released under CC BY-NC-SA 4.0 for non-commercial research.Source videos are included so observation logs can be replayed frame by frame; download also binds users to the source datasets’ licenses listed in Appendix K.Source: Ethics Statement · p. 11
• Ego4D footage depicts identifiable people, so the benchmark inherits that dataset’s consent and de-identification terms and adds no new recording.Users must also comply with the licenses of the source datasets listed in Appendix K.Source: Ethics Statement · p. 11
• Four authors handled annotation—three annotators and one adjudicator—for roughly 470 person-hours.Source: Ethics Statement · p. 11
• The audit metrics use decoded-frame timestamps and annotated interval boundaries, not attributes of people shown in the videos.Source: Ethics Statement · p. 11

Content of Appendices

• Appendices A–D cover implementation details and per-question cost, the Trajectory Reconciler pipeline, trajectory versus voting and self-consistency, and same-backbone baseline reproduction.Source: Content of Appendices · p. 15
• Appendices E–K address statistical reporting, targeting, backbone configuration, annotation, blind LLM validation, density-level ablations, and dataset licenses.Source: Content of Appendices · p. 15

A Implementation Details and Per-question Cost

• TRACE uses prompt-based modules around each evaluation backbone, reuses a 32-frame storyboard, and costs 98.7 frames per question versus AVP’s 101.3.Final answer replay consumes only raw visual clips, the question, and answer options, and emits an answer alone.Source: A Implementation Details and… · p. 15

B Trajectory Reconciler Pipeline

• TR rule-triggers STOP at the earliest stable, answerable, uncontradicted prefix with no unmet needs; otherwise, one backbone call selects DROP, REFINE, or EXPAND.REFINE adjusts an existing anchor’s observation strategy, while EXPAND selects unexplored intervals; the backbone acts as a classifier over this limited action set.Source: B Trajectory Reconciler Pipeline · pp. 15–16

C Trajectory vs. Majority Voting and Self-Consistency

• Removing the growth trajectory reduced EP from 29.1 to 19.4 and ECA@2 from 50.7 to 27.5 while retaining the prefix-bundle structure.Unlike majority voting and self-consistency (Wang et al., 2023), TRACE fixes the question, options, prompt, and decoding setting while visual evidence grows, exposing unmet needs for targeted expansion or refinement.Source: C Trajectory vs. Majority Voting and… · p. 16

D Same-Backbone Reproduction of Agent Baselines

• AVP and VCA were reproduced on Gemini-3-Flash with shared timestamp logging; reproduced AVP on Gemini-2.5-Pro stayed within 0.6 points of reported public-benchmark results.The reproduction follows published pipelines and ported prompts, keeping intermediate planning or scoring on the same backbone family.Source: D Same-Backbone Reproduction of Agent… · pp. 16–17

E Statistical Reporting

• Confidence intervals use 1,000-question-set bootstrap resamples, paired bootstrap compares methods, and video-cluster bootstrap checks robustness when videos contribute multiple questions.The resampling set is the 600 VES-Bench questions; Table 6 reports 95% intervals at k=2.Source: E Statistical Reporting · p. 17

F Targeting Reference

• VES-Bench’s EPref is 13.7%, a dataset-wide constant used to anchor each method’s Allocation Ratio.It is computed as the average, over 600 questions, of evidence-interval union length divided by source-video duration.Source: F Targeting Reference · p. 17

G Backbone Configuration

• TRACE uses Gemini-2.5-Pro on public benchmarks and Gemini-3-Flash for the VES-Bench audit, where full-loop control and cost constrain the backbone choice.Table 2 includes a same-frame Gemini-2.5-Pro Direct VLM result at 64 frames for comparison across protocols.Source: G Backbone Configuration · p. 17

H Annotation Pipeline Details

• Construction uses models outside the VES-Bench audit backbones, including Qwen2.5-VL-72B for captioning, GPT-4o for proposals, and Claude 3.5 Sonnet for the vision probe.The cited models are Qwen2.5-VL-72B (Bai et al., 2025b), GPT-4o (Hurst et al., 2024), and Claude 3.5 Sonnet; captioning windows are 30 seconds with 5-second overlap. Proposals follow the Temporal Ordering and Event Counting schema and pair each interval with a rationale for verification.Source: H Annotation Pipeline Details · pp. 17–18
• Table 7 reports TRACE component ablations on VES-Bench using Cov@k and ECA@k at k=1, 2, and 3.The TRACE row matches Table 4.Source: Table 7 · p. 18
• Of 1,024 proposed items, 600 survive five-stage adjudication, a combined accept rate of 58.6%.The pool falls to approximately 820 after visual verification, 700 after the leave-one-out probe, and 640 after the shortcut probe; Stage 2 removes the most items overall, while Stage 3 is the largest closure-specific filter.Source: H Annotation Pipeline Details · p. 18
• Four authors staff annotation: two of three annotators independently review each item, while an adjudicator resolves disagreements.Mean time is 13.8 minutes per item for visual verification and the leave-one-out probe combined; total annotator time across 1,024 candidates is approximately 470 hours.Source: H Annotation Pipeline Details · p. 18
• Evidence-boundary IoU agreement is 0.83, and human–Claude 3.5 Sonnet leave-one-out sufficiency agreement is 91.3%.The shortcut probe rejects an item if either GPT-4o (language-only) or Claude 3.5 Sonnet (gist-only) selects the correct option above the 25% chance baseline on a held-out repeat.Source: H Annotation Pipeline Details · p. 18
• The pipeline rejects language-prior, chance-solvable, and gist-solvable items, but does not guarantee minimal evidence sets.Under the audit, humans and an independent vision LLM must agree that no leave-one-out partial set uniquely determines the answer; removing any single interval leaves at least two options visually compatible.Source: H Annotation Pipeline Details · p. 18

I Blind LLM Validation

• Blind text-only GPT-4-Turbo scores 25.3% overall, within the bootstrap interval of the 25% random baseline, with ECA@k equal to 0 by construction.Scores are 25.0% for Temporal Ordering and 25.7% for Event Counting; the test uses only question stems and options, with no visual or retrieved context.Source: I Blind LLM Validation · p. 18

J Ablation Across Density Levels

• At k=3, removing the growth trajectory lowers Cov@k from 60.3 to 30.5, while Cov@k and ECA@k fall as k rises for every variant.The largest share of Cov@k is lost when the growth trajectory is removed.Source: J Ablation Across Density Levels · p. 18

K Dataset Licenses

• VES-Bench annotations are released under CC BY-NC-SA 4.0 for non-commercial research.The annotations include questions, options, evidence-set boundaries, and per-question observation logs.Source: K Dataset Licenses · p. 18
• Source videos are included because reproducing the audit requires the exact frames corresponding to decoded source-frame timestamps.Downloading the release binds users to CC BY-NC-SA 4.0 and the source-dataset licenses listed in Table 8.Source: K Dataset Licenses · pp. 18–19, Table 8 · p. 19