HERBench: A Benchmark for Multi-Evidence Integration in Video Question Answering

1 Introduction

• Established VideoQA scores can reflect language priors or single-cue shortcuts, while single-cue questions do not measure multi-evidence integration.The paper argues that controlling evidential requirement distinguishes integration from reliance on isolated cues [12].Source: §1 · p. 1
• MRFS measures the minimum number of frames needed for a correct answer, and HERBench has the highest mean MRFS among the benchmarks considered.The canonical MRFS comparison supports ER-focused diagnostics.Source: §1 · pp. 1–2
• Adaptive frame selectors still lag ground-truth keyframes, and models achieve only modest accuracy even when given those frames because they struggle to integrate all critical evidence.The authors identify both missed evidence and underweighting of necessary frames as bottlenecks.Source: §1 · pp. 1–2
• HERBench’s 26,806 questions across twelve compositional subtasks enforce at least three distinct, non-redundant visual cues.The Temporal Shot Ordering example depicts four non-overlapping cues dispersed across a video.Source: §1 · pp. 1–2, Figure 1 · p. 2

2 Related Work

• Video-LLMs have evolved from feature pooling [24] to alignment modules and instruction tuning [28], yet audits report continued reliance on language priors or single-frame shortcuts [12] [31].The passage also notes large token capacities in Gemini 2.5 [29] and GPT-4o.Source: §2 · p. 3
• VideoQA benchmarks span short-clip recognition, longer contexts [11], and multi-step reasoning, while HERBench controls the number of cues to combine.Its questions require at least three non-overlapping cues from distinct video moments; oracle frames help disentangle retrieval from fusion failures.Source: §2 · p. 3

3.1 Task Taxonomy

• HERBench groups 12 tasks into four reasoning families, with every task requiring at least three distinct, temporally separated cues.The families are Temporal Reasoning & Chronology, Referring & Tracking, Global Consistency & Verification, and Multi-Entity Aggregation & Numeracy.Source: §3.1 · p. 3, Figure 2 · p. 4

3.2 Benchmark Construction

• Three complementary streams—object tracking and trajectory analysis, shot segmentation, and human-verified ground-truth integration—produce refined data for task programming and quality control.Questions undergo token-level similarity checks, manual leakage review, blind-LLM filtering, and expert verification; validation accuracy was 88.8% with full-video access and 95.7% with oracle frames.Source: §3.2 · pp. 5–6

3.3 Benchmark Statistics

• HERBench contains 26,806 five-way questions from 336 videos across 12 tasks and four reasoning families; videos average 395 seconds, with a 60–2100-second range.Source: §3.3 · p. 6

3.4 Evidential Requirement & the MRFS Metric

• 5.49 mean MRFS makes HERBench highest among the compared benchmarks under the canonical protocol; MINERVA follows at 5.14.The shared comparison uses Qwen2.5-VL, AKS, and a 16-frame budget; HERBench combines language debiasing, structurally required temporal fusion, absence checking, and oracle-frame availability.Source: §3.4 · pp. 6–7, Table 1 · p. 7

Setup and results

• All models received the same 16 uniformly sampled frames, and the controlled-frame evaluation indicates that poor performance reflects integration failure, not merely lack of access to evidence.Source: §4 · pp. 7–8

Results

• 38.2% mean accuracy across 13 models; Ovis-2.5-9B [4] reached 42.1% and LLaMA-4-Scout-17B 31.4%, versus the 20% random baseline.Source: Table 2 · p. 8, §4 · p. 8
• Ovis-2.5-9B scored 79.7% on AGBI and 76.2% on AGAR, while mean accuracy was 23.0% on AC and 23.2% on MEGL; TSO scores fell to 0.1%.The passage characterizes AGBI and AGAR as single-entity tracking tasks and AC, MEGL, and TSO as requiring multi-cue aggregation or temporal ordering.Source: §4 · p. 8

5 Analysis

• The analysis frames two questions: how frame-selection strategies affect retrieval, and whether models aggregate the correct frames once retrieval uncertainty is removed.Source: §5 · p. 8

5.1 Isolating the Evidence Retrieval Bottleneck

• With Oracle Frames, accuracy remains below 50%; AKS and BOLT-ITS outperform uniform sampling on many tasks but still trail the oracle configuration.The comparison averages accuracy across tasks and three representative models; the oracle-frame setup is intended as a best-case retrieval setting.Source: §5.1 · pp. 8–9

5.2 Evidence Aggregation with Oracle-Only Frames

• The oracle-only study supplies manually curated ground-truth frames to isolate aggregation from retrieval uncertainty; human annotators reached 95.7% accuracy in the same presentation format.The human result indicates that the curated oracle frame-set is generally sufficient to resolve the questions.Source: §5.2 · p. 9
• Under oracle-only inputs, incorrect predictions concentrate importance on one frame (Top-1 shares frequently near 0.8), while correct predictions are more balanced, near 0.5.This pattern indicates that models can misallocate attention among distributed cues even when evidence-bearing frames are provided.Source: Figure 5 · p. 9, §5.2 · p. 10

• Across 13 state-of-the-art MLLMs, accuracy ranges from 31.4–42.1%, only modestly above the 20% chance baseline.Source: §6 · p. 10
• HERBench contains 26,806 questions across 12 tasks, each requiring at least three distinct, temporally separated visual cues; it has the highest mean MRFS among compared benchmarks.MRFS is the minimum frame-set required for a correct answer.Source: §6 · p. 10
• Failures reflect both retrieval deficits, where selectors miss necessary cues, and fusion deficits, where models fail to combine available cues, often collapsing onto one frame.Source: §6 · p. 10
• HERBench provides a principled benchmark for studying high-evidential-requirement video understanding and identifies substantial headroom beyond single-cue success.Source: §6 · p. 10

7.1 Track Ranking and Selection

• RF-DETR-L with DeepSORT achieved the highest tracking scores on the curated DanceTrack subset, with HOTA of 47.1% and IDF1 of 53.2%.The authors selected this configuration for their data-generation pipeline.Source: §7.1 · p. 14, Table 4 · p. 14
• TrackRank selects 6–10 salient entities per video using a composite score, unlike simple duration-based ranking, aggregating track duration, size, associated objects, center distance, motion, and appearance exceptionality.Source: §7.1 · pp. 14–15
• The pipeline links RF-DETR detections with DeepSORT tracks, removes outliers, interpolates gaps, smooths trajectories, and ranks salient tracks for downstream descriptors.Figure 6 depicts this tracking, post-processing, and ranking sequence.Source: §7.1 · p. 14, Figure 6 · p. 15
• Round-robin selection across perturbed ranking weights reduces redundant selections, and manual validation excludes phantom detections and identity switches.Source: §7.1 · p. 15

7.2 Decoupled Descriptor Generation

• A-cards describe static appearance from the first and last trajectory portions, while B-cards describe behavior from the middle, leaving at least a 30-frame temporal gap between cue sources.The prompts prohibit position or action details in appearance descriptions and prohibit appearance details in path descriptions.Source: §7.2 · pp. 15–16, Figure 7 · p. 16

7.3 Spatial Operations and Region Definitions

• For region-localized counting, a track visits an ROI when at least 50% of its box lies inside it; answer options use binned count ranges to absorb residual tracking noise.The bins span approximately ±40% around the true count on average, while retaining a spatial-counting requirement.Source: §7.3 · pp. 16–17

7.4 Scene Card Perturbations

• For scene verification, negative cards alter or add 2–5 atomic details while remaining plausible, creating contrasts with faithful scene descriptions.The examples include modified clothing patterns, backgrounds, or added objects.Source: §7.4 · p. 17, Figure 8 · p. 17

7.5 Corpus-Plausible Foil Generation

• False Action Memory and False Object Memory use corpus-plausible absent actions or objects as distractors, while Action Counting varies the correct count’s rank across options.Action Sequence Integrity uses swapped or rotated five-event timelines and checks the perturbations against event logs to exclude sequences that actually occur.Source: §7.5 · pp. 17–18

7.6 Text-Only Bias Filtering Details

• Text-only filtering removes questions correctly answered by at least three of four blind LLMs, rejecting approximately 10% of candidates.The filter targets questions answerable from language priors, including object-color co-occurrences.Source: §7.6 · p. 18

7.7 Human Verification Protocol

• Experts audit a stratified 15% sample for three-frame compliance, answer uniqueness, and descriptor disentanglement; violations are rejected, yielding a 17.8% rejection rate.The construction-time audit checks that A-cards and B-cards do not leak information from one another.Source: §7.7 · p. 18

7.8 Human Validation and Oracle-Frame Study

• 88.8% full-video accuracy and 95.7% oracle-frame accuracy show that humans can answer the questions and that curated frames are generally sufficient.The full-video study also reports Fleiss’ κ = 0.74; sparse or temporally localized tasks show the largest oracle-frame improvements.Source: §7.8 · pp. 18–19

7.9 Dataset Statistics

• HERBench contains 26,806 questions from 336 videos averaging 395 seconds, with evidence spanning a mean of 101.1 seconds per question.Questions average 65.5 tokens and are balanced across five multiple-choice options.Source: §7.9 · p. 19

8.1 Extended MRFS Analysis

• TSO has the highest reported task MRFS at 9.05, while even lower-demand RLPC (3.11) and AGAR (3.85) require multiple frames.FAM (6.77) and SVA (6.74) also have high MRFS; the overall weighted mean is 5.49.Source: §8.1 · p. 19
• 35.9% accuracy on HERBench (5.49 MRFS) contrasts with 87.5% on TemporalBench (2.21 MRFS), amid an inverse relationship between evidential demand and model accuracy.The ten-benchmark comparison reports R2 = 0.82 for the fitted trend; AGQA falls below that trend, suggesting other factors can also depress performance.Source: §8.1 · p. 19, Figure 9 · p. 20

8.2 Full Frame-Selection Ablation

• +3–6 absolute accuracy points is the typical oracle-frame gain over the best learned selector, yet models still struggle on fusion-bound tasks.Learned selectors provide moderate gains over uniform sampling, especially on sparse-evidence tasks; oracle-frame access does not resolve most failures, while helping more on temporally global tasks.Source: §8.2 · p. 21, Table 7 · p. 21

8.3 MRFS robustness across backbones and frame selectors

• HERBench consistently has the highest MRFS across tested backbone and selector configurations, supporting MRFS as a robust dataset-level measure of evidential requirement.The comparisons use four backbones with AKS and three selectors with Qwen2.5-VL, on stratified 50% benchmark samples.Source: §8.3 · p. 21, Table 8 · p. 22

Temporal Reasoning & Chronology

• Figures 10–12 illustrate temporal shot ordering, multi-person duration reasoning, and action-sequence integrity through examples involving chronology and event order.The examples respectively involve ordering four non-overlapping shots, comparing visible-time intervals, and choosing among plausible event-sequence permutations.Source: §9 · p. 22, Figure 10 · p. 23, Figure 11 · p. 24

Referring & Tracking

• Figures 13–15 show appearance-grounded behavior interaction, attribute recognition, and localization trajectory examples centered on tracking a described target.The examples ask who interacts with the target, which attributes apply, or how the target enters or exits the scene.Source: §9 · p. 22, Figure 13 · p. 25, Figure 14 · p. 26

Global Consistency & Verification

• Figures 16–18 illustrate false-action and false-object memory alongside scene verification and arrangement.These examples require identifying a plausible action or object that is absent, or checking faithful and perturbed shot descriptions.Source: §9 · p. 22, Figure 16 · p. 27, Figure 17 · p. 27

Multi-Entity Aggregation & Numeracy

• Figures 19–21 exemplify multi-entity grounding, action counting across a video, and counting unique people entering through specified regions.The examples cover verifying which described individuals appear and counting action–object instances or people by entry region.Source: §9 · p. 22, Figure 19 · p. 28, Figure 20 · p. 29