PaperlayerView exact revision on arXiv

Source-linked AI summary

Video-Index: A Curated Meta-Benchmark for Video Understanding

Enxin Song, Yinuo Xu, Shusheng Yang, Wenhao Chai, Jiatao Gu

arXiv:2610.00960v1cs.CV

TL;DR

Video benchmarks can award high scores without models demonstrating the claimed capability, as stronger language priors, rich frames, and retrieved questions create shortcuts. The paper audits benchmarks with a five-level attack pyramid and builds Video-Index from screened items. Newer benchmarks break earlier, capability claims break at different levels, and the released benchmark contains 840 hardest verified items.

  • Problem

    Video benchmark scores may be earned through language priors, rich single frames, or retrieval over previously asked items rather than the capability they claim to measure.

    Source: §1 · p. 2

  • Method

    The attack pyramid tests five attacker sets with increasing access, while an agent-assisted pipeline screens items and uses deterministic selection and a red-team gate to compose benchmarks.

    Source: §1 · p. 2

  • Results

    Newer benchmarks break earlier, each capability claim breaks at its own attack level, and Video-Index retains 840 hardest verified items.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Scores should be interpreted above the strongest tested shortcut, and resistance holds only against the attacks that were tested.

    Source: §6 · p. 15

    The audit covers English multiple-choice benchmarks; open-ended and generative video benchmarks fall outside its protocol.

    Source: C Limitations · p. 36

Abstract

from arXiv · show

A video benchmark should reward the capability it claims to measure, yet models can exploit answer options, question text, or partial visual evidence. We introduce the attack pyramid, five levels of shortcut attacks with increasing access to each item, and audit 115 video benchmarks with it. On 35 benchmarks, attackers that never see a frame approach full-video accuracy. On 51 benchmarks with temporal probes, shuffled frames keep a median 96% of full-video accuracy. Near-duplicate questions make up at least half the items in 63 benchmarks. We screen 505,518 question-answer pairs from 112 of them into an audited pool. Agents turn evaluation requests into specifications, and a deterministic selector with a red-team gate composes reproducible benchmarks. We release Video-Index, the 210 hardest verified items under these attacks in each of four capability groups, 840 items from 76 sources. With the same fixed input, Claude Opus 5 outscores every open-source model by over 37 percentage points, and agent tools add about 20 more, yet all systems leave room to improve efficiency and accuracy. Blog: https://www.enxinsong.com/blog/video-index/ GitHub: https://github.com/Espere-1119-Song/Video-Index Hugging Face: https://huggingface.co/datasets/Video-Index/Video-Index

1 Introduction

Benchmark scores can look like evidence of video capability even when models exploit language priors, rich single frames, or retrieved questions. The attack pyramid tests shortcuts at increasing access levels, and the resulting pipeline retains hard, verified items for reproducible benchmarks.

1 Introduction

  • The attack pyramid tests five attacker sets, from answer options alone to shuffled or truncated video, and keeps items no tested attack solves.The levels progressively add question text, other evaluation items, and degraded visual access.Source: §1 · p. 2
  • Newer benchmarks break earlier, while each capability claim breaks at its own attack level.The audit covers 115 video benchmarks.Source: §1 · p. 2
  • Four-stage screening reduces the 505,518-item pool to 840 hardest items for Video-Index.The retained answers were verified against video frames.Source: Figure 2 · p. 2, §1 · p. 2
  • The agent-assisted pipeline combines screening, specification generation, deterministic selection, and red-team validation to compose reproducible, on-demand benchmarks with lockfiles.Source: §1 · p. 2

2 What scores certify

The paper treats benchmark scores as evidence for claimed capabilities and tests how much of that score attackers can achieve without the capability. Its attack pyramid defines progressively broader access levels and a tolerance-based breaking threshold.

2 What scores certify

  • Exploitability is the largest accuracy margin over chance achieved by an attacker in the declared set.The maximizing attacker may differ across benchmarks.Source: §2 · p. 5
  • The nested attack levels progress from options, to question text and other evaluation items, then degraded visual input and temporally perturbed video.Attackers do not see the ground truth of the evaluated item.Source: §2 · p. 5
  • The protocol applies every measured attack level to every benchmark, although a capability’s threat model could exempt some levels.The paper gives knowledge claims as a case that could exempt the text level.Source: §2 · p. 5
  • A benchmark breaks at the first level where exploitability reaches the reference model’s full-protocol accuracy gap over chance minus 5%.The 5% tolerance is described as about the 95% sampling margin of a 300-item accuracy.Source: §2 · p. 5

3 Auditing with shortcut attacks

The audit probes 115 video benchmarks under options-, text-, pool-, frame-, and order-level shortcuts. Many scores survive without full-video evidence: shuffled frames retain 96% median accuracy on 51 benchmarks, while 38 benchmarks remain unbroken.

3.1 Benchmarks and protocol

  • From a census of 605 benchmarks and 721,324 question–answer pairs, the audit retained 115 English multiple-choice benchmarks without audio and sampled up to 300 items each.Gemini 3.5 Flash supplied reference accuracy at 32 frames, with 128-frame probes for long-video benchmarks near chance.Source: §3.1 · p. 5

3.2 Options without questions

3.3 Questions without video

  • Question text raised accuracy by up to 71% over chance; among 98 option-level passers, blind readers beat reference on 4 and broke 13.Source: §3.3 · p. 6

3.4 Learning from evaluation items

  • As evaluation items accumulated, the strongest pool attacker narrowed the video gap, and five benchmarks broke.Source: Figure 4 · p. 7, §3.4 · p. 7

3.6 Order and temporal coverage

  • Shuffling retained a median 96% of full-video accuracy and a contiguous tenth retained 78% among 51 benchmarks, while 38 benchmarks remained unbroken.Source: §3.6 · pp. 7–8

3.5 Single frames and captions

  • Captions came within δ of reference on 23 benchmarks and exceeded reference on 9, versus one-frame tolerance on 8 of the 78 benchmarks with all probes.Source: §3.5 · p. 7, Figure 5 · p. 8

4 What the audit finds

The audit finds that newer video benchmarks are more often vulnerable before visual input, while shortcut patterns and model errors differ across capability groups. It also tests the effects of additional visual input and finds substantial duplication in many benchmarks.

4.1 Newer benchmarks break earlier

  • 35 of 115 benchmarks break before any visual input, and the share rises from 19% among releases through 2024 to 34% among releases in 2025 and 2026.The rise comes from text and pool levels, where newer releases account for 17 of 18 breaks; blind readers break 12 of 89 newer benchmarks against 1 of 26 older ones.Source: §4.1 · p. 8, Figure 7 · p. 10

4.2 Each claim breaks differently

  • Reasoning and knowledge benchmarks often break on text or captions, perception benchmarks most often at the frame level, and spatial and physical benchmarks on options or temporal perturbation.Temporal benchmarks split: 6 of 18 break on options, while 10 survive every level; the authors recommend matching audits to claimed capability and screening options even for temporal claims.Source: §4.2 · pp. 8–9, Figure 8 · p. 11

4.3 Frames or pixels

  • 11 benchmarks favor higher pixel resolution, 3 favor denser frame sampling, and most long-video benchmarks gain from larger frame budgets.Among 30 benchmarks whose videos average at least 300 seconds, 18 gain at least δ in the reference sweep and 9 still gain at its 600-frame limit; these gains also depend on the reference model’s long-context ability.Source: §4.3 · p. 9, Figure 9 · p. 11

4.4 Errors cluster in few capabilities

  • The two most frequent error-attribution categories hold a median 71% of a benchmark’s attributions, while capability profiles overlap weakly, with mean cosine similarity 0.29.Fine-grained action leads on 27 benchmarks, spatial relation on 13, and temporal order and action counting on 7 each, counting ties; covering observed gaps therefore takes several sources.Source: §4.4 · p. 10

4.5 Duplicates reduce effective size

  • A 200-item draw has a median of 32 effective items, and near-duplicate questions make up at least half the items in 63 benchmarks.Weighting by inverse duplicate-cluster size moves accuracy by more than 1% on 93 benchmarks; a tenth of items use videos appearing in another benchmark, and PAI-Bench-U and PhysicalAI share 191 questions and yield nearly identical scores across conditions.Source: §4.5 · pp. 10–11

5 Rebuilding video evaluation

The authors build an audited pool and a request-driven selector, then release Video-Index as a difficult evaluation set. Its results show a large gap between proprietary agents and open models, with additional gains from agent tools.

5.1 Screening every item

  • Attacks remove 76% of distinct questions, leaving 62,142 with complete screening and labeled videos for a 10,000-item coverage composition.The screen uses text-only, options-only, and single-frame attacks, then ordered and shuffled 32-frame attacks; three recently added benchmarks are excluded.Source: §5.1 · pp. 11–12

5.2 Composition from a request

  • A specification agent converts a request into a confirmed pool query, and a round-robin selector balances duration, event type, and source while recording relaxations for scarce coverage cells.After selection, a red-team gate removes solved items and refills for bounded rounds; a lockfile records the pool snapshot, query, selector version, and seed.Source: §5.2 · p. 12

5.3 Video-Index

  • Fixed-input Claude Opus 5 outscores every open model by more than 37%, and tools raise the same model by 19.8% on matched items.Video-Index contains 840 hardest verified questions from 76 sources, with 210 items per capability group and one item per video; the gate holds the strongest attacker to 13%, against 19% for ungated selections at a matched item budget.Source: §5.3 · pp. 13–14, Table 1 · p. 15
  • On Video-Index, GPT-6-Astra, Claude Fable 5.1, and Claude Opus 5 often tile frames into grids but choose the same sampling strategy on fewer than half of items.Claude Fable 5.1 extracts far more images than GPT-6-Astra without reaching its accuracy, and all three score lower on videos past ten minutes.Source: §5.3 · p. 15

6 Conclusion and Discussion

The paper concludes that benchmark scores should be interpreted alongside shortcut resistance, and that leading proprietary agents still leave room to improve accuracy and evidence selection. Its appendices document the audit, the benchmark set, and related work.

6 Conclusion and Discussion

  • A score certifies the claimed capability only above its strongest shortcut, and the authors recommend reporting the breaking level beside the score.Resistance holds only against tested attacks, so audits and benchmarks should evolve with new models and shortcuts.Source: §6 · p. 15

Appendix

  • The appendices document the audit probes, capability breakdowns, pool construction, Video-Index selection, and per-benchmark reports.They also state the limitations of the audit and Video-Index.Source: Appendix · pp. 33–34

A Audited benchmarks

B Related work

  • Related work covers video-understanding benchmarks and prior shortcut audits, benchmark construction, dataset artifacts, adversarial data, and evidence-grounding or evidence-seeking systems.The paper distinguishes Video-Index’s multiple-choice filtering from direct evidence-grounding checks, describing the approaches as complementary.Source: B Related work · pp. 34–36

C Limitations

The audit and Video-Index cover English multiple-choice benchmarks, exclude audio-dependent benchmarks, and bound their findings to the tested attackers.

C Limitations

  • The audit covers English multiple-choice benchmarks; open-ended and generative video benchmarks fall outside its protocol.Video-Index also retains multiple-choice format, which the option-level audit shows can leak answers.Source: C Limitations · p. 36
  • Audio-dependent benchmarks are excluded, leaving audio-visual claims untested because attackers receive no speech, subtitles, or sound.Source: C Limitations · p. 37
  • Exploitability is a lower bound: stronger or newer attackers may raise it and lower a benchmark’s breaking level.Surviving every level establishes resistance only to the attackers tested.Source: C Limitations · p. 37

D Qualitative examples

The audit illustrates how options, text, other items, captions, and altered frames can expose shortcuts, and details its scoring and screening protocol. Its examples also contrast these vulnerabilities with difficult Video-Index items.

D Qualitative examples

  • Examples show distinct shortcuts: options-only and blind readers exploit answer wording, copied questions, or specific titles, while captions or one frame can suffice.Conversely, agents solve a reversed butterfly transformation that fixed-input systems miss, while every tested system misses a long-video price-tag count.Source: D Qualitative examples · pp. 37–39

E Attack levels as bounded observers

  • Attack levels model bounded observers with progressively more input; exploitability measures their score, while breaking level marks the first level approaching the reference.Option permutations test sensitivity to answer layout, whereas frame shuffling tests dependence on temporal order.Source: E Attack levels as bounded observers · p. 40

F Per-benchmark option and text results

  • Option attacks and blind readers exceed chance on most audited benchmarks, but above-chance accuracy alone does not meet the break criterion.Figure F.1 separates fixed-position from options-only attacks and compares two blind readers; each panel ranks its own stronger attack.Source: F Per-benchmark option and text results · p. 40, Figure F.1 · p. 42

G Experimental settings

H Scoring and screening protocol

  • Scoring verifies errors through identical-frame reasoning reruns, rule-based pre-classification, independent judges, and dense-frame checks; offset-induced answer flips are classified as unstable.The screen removes 368 near-duplicates across 115 benchmarks and 34,274 items; its frame-offset self-check flips 108 of 1,103 failures (9.8%).Source: H Scoring and screening protocol · pp. 40, 42, 43, Table H.1 · pp. 43–45

I Option-level attacks

J Text-level attacks

  • Blind lift correlates −0.38 with Qwen2.5-7B min-k% across 110 benchmarks; median blind lift is +14% in 2023, +8% in 2024, +21% in 2025, and +13% in 2026.Exposure indicators are indirect; release dates need not date source material, and question matches show public text rather than answer access.Source: Table J.4 · p. 58, J Text-level attacks · p. 61

K Pool-level attacks

Pool-level attacks test reuse and retrieval across items, while frame- and order-level probes test whether visual detail and temporal order are necessary. Breaking patterns vary by capability and change under claim-matched attacks.

K Pool-level attacks

  • Near-duplicate analysis finds 923 shared videos and 2,893 cross-benchmark video pairs, plus 412 near-duplicate and 345 verbatim question pairs.Effective-size rankings agree across visual, text, and joint embedding spaces at Spearman 0.88, 0.96, and 0.90, respectively.Source: K Pool-level attacks · pp. 61–62, Table K.1 · p. 62, Table K.2 · p. 63
  • Cross-pool attacks reach 28.5% exploitability at full scale, compared with 23.7% for within-pool attacks.Exploitability is the largest final-stage margin among the listed attackers.Source: Table K.3 · p. 63

L Frame-level attacks

  • Frame captions recover more reference accuracy than a single image; on MMWorld accuracy is 67% with one frame and 68% with 32, versus 59% and 78% on EgoSchema.Source: L Frame-level attacks · pp. 73–74

M Order-level attacks

  • The long-video frame-budget sweep uses identical items, prompt, and scorer, but its reference sweep ends at 600 frames because of the request limit.The long-video set contains benchmarks whose sampled videos average at least 300 seconds.Source: M Order-level attacks · p. 80, Table M.2 · p. 82

N Breaking levels by fine capability

  • Object identity breaks at the frame level on five of seven benchmarks, while temporal order survives on eight of 18 and spatial relation survives on 11 of 24.Multi-hop reasoning breaks on text or a frame for 15 of 29 benchmarks and survives on six; fine-grained action survives on three of ten.Source: N Breaking levels by fine capability · pp. 83–84, Figure N.1 · p. 84

O Claim-matched breaking levels

P Robustness to tolerance and reference model

The audit’s headline patterns persist under alternative tolerances and reference models, while resolution and frame-rate gains are generally small.

P Robustness to tolerance and reference model

Q Resolution, frame rate, and frame budget

R Errors introduced by video input

Video input can introduce errors even while improving overall accuracy: the blind reader corrected some errors made by the video reference, and adding video lowered total accuracy on 10 benchmarks.

R Errors introduced by video input

  • 21% of the 9,651 attributed reference errors were answered correctly by the blind reader, with the share ranging from 17% to 30% across the middle half of benchmarks.On IntPhys 2 [77], it reached 88%, although both conditions remained near chance and succeeded on different items.Source: R Errors introduced by video input · p. 87
  • Video lowered total accuracy on 10 benchmarks, including drops above 9% on LongTimeScope and VideoOdyssey [138].Figure R.1 shows that video introduced individual errors on most benchmarks, but lowered total accuracy on only these 10.Source: R Errors introduced by video input · p. 87
  • Shortcut auditing and error attribution address complementary failure modes: errors made by the video model can be answered correctly without video.The blind reader's corrections show that video input can overturn a correct text-only answer.Source: R Errors introduced by video input · p. 87

S Attribution of reference errors

Reference errors include label problems that better models cannot resolve, as well as capability-related errors that remain after benchmarks survive shortcut attacks. Among fine-attributed errors, language priors dominate most surviving benchmarks.

S Attribution of reference errors

  • 5% of 6,863 attributed failures arose from labels, including ambiguous answers and mismatched videos; the median unrecoverable score gap was 0.8%.That gap exceeded 10% on TempCompass and UCF101-AD [119].Source: S Attribution of reference errors · p. 87
  • Language priors were the largest fine error category on 30 of 35 surviving benchmarks, with a median share of 54.6%.VSI-Bench [5] instead attributed 58% to spatial relation and metric scale; Figure S.1 summarizes capability-error shares.Source: S Attribution of reference errors · p. 87
  • Among 38 surviving benchmarks, 30 had label-error attribution, but the median label-error share was 0%.Across all benchmarks, the median label-error share was 2.1%, and it exceeded a quarter on one benchmark.Source: S Attribution of reference errors · p. 87
  • Most named profiles concentrate their non-prior errors in two categories, with separate tables covering perception, temporal, spatial, reasoning, and unresolved categories.Benchmarks with fewer than 20 capability instances show no shares; a dash indicates an instance exists, while a blank cell marks none.Source: S Attribution of reference errors · p. 88, Table S.1 · p. 88, Table S.2 · p. 91
  • Attribution tables use Gemini 3.5 Flash or Claude Opus 5 failures and exclude disputed, unstable, and unverified instances from their denominators.Per-benchmark rows require at least 20 errors; five first-table rows use predictions made with letter-only option lists.Source: S Attribution of reference errors · p. 102, Table S.6 · p. 102, Table S.7 · p. 104
  • Protocol screens flag misleading answer formats and metadata: four sources had blind accuracy below chance, and 33 benchmarks omitted chance metadata.Micro versus macro averaging reversed 79 of 1,344 system pairs; eight benchmarks were flagged by three of four independent screens.Source: S Attribution of reference errors · p. 108

T Benchmark recommendations by capability

Thirty-eight of 115 benchmarks survive every tested attacker level, providing a certificate only against those attackers. Survivors are distributed across four capability groups, with the strongest benchmarks ranked by their margin below the breaking threshold.

T Benchmark recommendations by capability

  • Of 115 benchmarks with certificates, 38 survive every attacker level; this includes 13 of 45 perception, 10 of 18 temporal, and 7 of 20 spatial and physical benchmarks.Source: T Benchmark recommendations by… · p. 108
  • A survivor resists only the attackers tested and certifies nothing beyond them.Its report card identifies the level at which it came closest to breaking.Source: T Benchmark recommendations by… · p. 109
  • Tables T.1–T.4 rank the ten strongest benchmarks in each capability group by breaking level and then margin below the breaking threshold.The groups are perception, temporal, spatial / physical, and reasoning / knowledge.Source: T Benchmark recommendations by… · p. 109

U Pool construction

The pool construction produces Video-Index by filtering items against shortcut attackers, verifying answers against video frames, and selecting difficult items across four capability groups. A separate coverage composition and deterministic benchmark-building process support broader and reproducible evaluations.

U Pool construction

  • Approved sources are pinned to revisions and audited before joining a pool snapshot; lockfiles record the query, selector settings, seed, item identifiers, and residual exploitability.The watcher scans registries, reports, and release pages every twelve hours; later snapshots can support reissued compositions.Source: U Pool construction · p. 113
  • From 63,630 candidates, the pipeline retains the 3,000 lowest worst-case attacker percentiles, then selects 210 verified items per capability group in increasing percentile order.The five attacker views include options, question text, a single frame, 32 frames, and shuffled frames; reference-model accuracy does not determine final selection.Source: U Pool construction · pp. 114–115
  • Verification keeps 468 of 840 initial candidates and 1,208 of the next 2,095 hardest labeled items, yielding a verified pool of 1,676.The verification process drops contradicted, short-video not-verifiable, and definition-dependent items.Source: U Pool construction · p. 115
  • 840 final census items from 76 benchmarks have a median worst-case attacker percentile of 0.14 and maximum of 0.23; the reference model answers 477 correctly (56.8%).The items comprise 468 retained from the initial census and 372 replacements; the reference score measures performance on this selected set.Source: U Pool construction · p. 115
  • The evaluation protocol samples video at 1 frame per second up to 512 frames, while the blind protocol averages accuracy across four option permutations.Video gain is defined as video accuracy minus blind accuracy.Source: U Pool construction · p. 115
  • On the coverage census, reference and blind accuracy are 66.7% and 49.1%, versus 56.8% and 21.3% on Video-Index.Both video and blind accuracy are lower on the released composition than on the coverage census.Source: U Pool construction · p. 116, Table U.2 · p. 116
  • The 10,000-item coverage composition balances duration, scene, and question types across 9,688 videos and 9,045 scene clusters; every duration group includes all 19 scene types.Scarce duration–scene cells permit relaxed selection caps.Source: U Pool construction · pp. 117–118
  • Video-Index separates all video-versus-blind pairs and 77% of known-order pairs; strongest-attacker accuracy is 13%, versus 19% without gating.Gain-based rank agreement with the census is Kendall τ = 0.56 at 840 items, compared with τ = 0.34 at 105 and 0.49 at 420.Source: U Pool construction · p. 127, Figure U.3 · p. 127, Table U.10 · p. 127

V Agent behavior on Video-Index

Agent evaluation on Video-Index compares tool-enabled and fixed-input performance, while the accompanying results describe inspection strategies, duration effects, and full audit tables. The reported patterns include divergent inspection paths despite similar answers and lower accuracy on long videos.

V Agent behavior on Video-Index

  • 70.6% accuracy and 24.9% blind accuracy for tool-enabled Claude Opus 5 compare with 56.8% with video and 21.3% blind under the fixed 1 fps / 512-frame protocol.The tool-enabled evaluation uses maximum effort, model-directed sampling, and counts failures as incorrect.Source: V Agent behavior on Video-Index · p. 128
  • Figure 13 pairs leaderboard accuracy with session-log effort, reporting runtime intervals from first to third quartiles and independent operation-use percentages.Tables V.1–V.3 cover agent tool use, sampling strategies, and sampling, image counts, and accuracy by video duration.Source: V Agent behavior on Video-Index · p. 128, Table V.1 · p. 128, Table V.2 · p. 129
  • Contact sheets appear in 99–100% of Astra, Fable, and Opus sessions, compared with 69% for Gemini Pro and 35% for Flash-Lite.Fable decodes every stored frame in 72.2% of sessions, Gemini Pro uses an explicit frame-rate filter in 47.6%, and Flash-Lite retains native extraction resolution in every logged session.Source: V Agent behavior on Video-Index · p. 128
  • Pairwise answer agreement among Astra, Fable, and Opus is 76.8–80.6%, while strategy agreement is 33.0–43.8%.Beyond ten minutes, accuracy is 67.1% for Astra, 50.9% for Fable, and 46.6% for Opus, versus 88.7%, 84.4%, and 86.2% below 30 seconds; image-production quartile associations do not show that producing more images causes worse answers.Source: V Agent behavior on Video-Index · p. 128

W Taxonomy and full results

  • Tables W.2–W.3 rank benchmark-level audit results by the 32-frame-minus-blind accuracy gap, while Table W.1 summarizes capability groups and audit statistics.The full-results tables report lift, attributed-error shares, diversity, and membership; some diversity, near-duplicate, and membership cells are blank where not computed.Source: W Taxonomy and full results · p. 130, Table W.1 · p. 130, Table W.2 · p. 130

X Benchmark report cards

This section presents benchmark-level report cards pairing attack-break or survival behavior with diagnostics of effective size, duplication, video dependence, protocol issues, pool coverage, and Video-Index selection.

X Benchmark report cards

  • Each card reports attack gaps, breaking level, reference and shortcut accuracies, error categories, effective size, near-duplicate share, video dependence, protocol issues, and pool contribution.The cards regenerate from the same exported tables as the paper.Source: X Benchmark report cards · p. 137
  • AoTBench’s 77% reference accuracy at 32 frames survives every attack level, while Artifact-Bench’s 9% breaks at the option level.The two cards report chance accuracies of 47% and 17%, respectively.Source: X Benchmark report cards · p. 137
  • Breaking levels vary: COVER breaks at the text level, DSR-Bench and EgoSchema at the order level, and ExpVid at the frame level.Their reported reference accuracies are 59%, 41%, 77%, and 65%, respectively; COVER, DSR-Bench, and EgoSchema use 32 frames, while ExpVid uses 32 frames.Source: X Benchmark report cards · pp. 138–139
  • ExpVid’s effective size is 4 of 200 with 100% near-duplicates, compared with COVER’s 52 of 200 and 8%.ExpVid reports +6% video dependence and 6% protocol issues; COVER reports +8% video dependence and 31% protocol issues.Source: X Benchmark report cards · pp. 138–139
  • LongTimeScope reports -18% video dependence and 88% protocol issues, while GasVideo-1000 reports +29% and 71%, respectively.Their reference accuracies are 16% and 75%, respectively, at 32 frames.Source: X Benchmark report cards · pp. 139–140
  • Neptune’s 81% reference accuracy survives every attack level despite +30% video dependence and 67% near-duplicates.It has an effective size of 30 of 200, a coverage pool of 68, and a Video-Index count of 1.Source: X Benchmark report cards · p. 143
  • NExT-QA’s 83% reference accuracy breaks at the frame level, while StreamingBench’s 62% reference accuracy also breaks there.Their effective sizes are 56 of 200 and 19 of 200, respectively; near-duplicates are 14% and 88%.Source: X Benchmark report cards · pp. 143, 147, 148
  • SciVideoBench has an effective size of 8 of 200 and 97% near-duplicates, while its 65% reference accuracy survives every attack level.Its video dependence is +9%, coverage pool is 28, and Video-Index count is 3.Source: X Benchmark report cards · pp. 145–146
  • SpatialTree has an effective size of 3 of 200 and 100% near-duplicates, with +43% video dependence; its 62% reference accuracy breaks at the option level.Its coverage pool and Video-Index counts are both 0.Source: X Benchmark report cards · p. 147
  • TemporalBench has 34 of 200 effective items and a coverage pool of 75; its 77% reference accuracy breaks at the option level.Its near-duplicate rate is 58%, video dependence is +20%, and Video-Index count is 13.Source: X Benchmark report cards · p. 148
  • Attack break levels vary: v-HUB and Video-MME break at the frame and order levels, respectively, while VCR-Bench and ViMUL-Bench survive every level.The cards also report option-level breaks for VideoMathQA and text-level breaks for VideoSIAH-Eval.Source: X Benchmark report cards · pp. 149, 150, 152, 153
  • Video dependence ranges from -10% for VideoOdyssey to +48% for v-HUB, while VideoMathQA records -5%.These figures appear alongside other per-benchmark diagnostics, including effective size, near-duplicates, and protocol issues.Source: X Benchmark report cards · pp. 149, 152
  • Protocol issues account for 82% of VITATECS errors, versus 0% for Vinoground and Causal2Needles.Other cards report 2% for VCR-Bench and 50% for v-HUB.Source: X Benchmark report cards · pp. 149, 150, 152, 153, 155
  • Near-duplicates reach 100% for CAViAR and VURB, compared with 5% for Video-MME.The card for VideoMathQA reports 63% near-duplicates.Source: X Benchmark report cards · pp. 150, 152, 155
  • Across the cited entries, reported effective sizes span 4 to 64 of 200; near-duplicate rates range from 8% to 100%, and video dependence ranges from −1% to +32%.Across these entries, Video-Index values include 0, 24, 28, and 45.Source: X Benchmark report cards · pp. 151, 152, 153, 155
  • Near-duplicates reach 100% in ContPhy, E.T. Bench, and ReXTime, whose effective sizes are 8, 9, and 2 of 200, respectively.ContPhy has 45 items in its coverage pool and contributes 1 Video-Index item; it survives every attack level.Source: X Benchmark report cards · pp. 156, 161
  • Cards report breaks at the option, text, frame, and order levels: examples include EgoPlan-Bench, HLV-1K, CineTechBench, and iSafetyBench.Other cards report breaks at the text level for LongVideo-Reason and the option level for FineBench.Source: X Benchmark report cards · pp. 156–159
  • KFS-Bench and SmartHome-Bench each contribute 21 Video-Index items, from coverage pools of 305 and 257, respectively.KFS-Bench has 29 effective items of 200 and 41% near-duplicates; SmartHome-Bench has 40 of 200 and 51% near-duplicates; both survive every level.Source: X Benchmark report cards · pp. 159, 162
  • +41% video dependence is reported for NaturalBench-Video, while FineBench reports -4% and protocol issues in 12% of errors.NaturalBench-Video survives every attack level; FineBench breaks at the option level.Source: X Benchmark report cards · pp. 157, 160
  • ScaleLong contributes 109 Video-Index items from a coverage pool of 505 and survives every attack level.Its effective size is 43 of 200, with 12% near-duplicates, +8% video dependence, and 0% of errors attributed to protocol issues; its reference is 64% at 32 frames (chance 25%).Source: X Benchmark report cards · p. 161
  • TaskMeAnything-v1 has effective size 4 of 200, near-duplicates 99%, video dependence +15%, and protocol issues 29% of errors; its 62% reference at 32 frames breaks at the order level.Its coverage pool is 18 and Video-Index includes 1 item.Source: X Benchmark report cards · p. 162
  • TOC-Bench, UCF101-AD, and VANE-Bench break at the frame level; their references at 32 frames are 50%, 47%, and 88%, respectively.Their video dependence values are +30%, +15%, and +13%, respectively.Source: X Benchmark report cards · pp. 162–163
  • UrbanVideo-Bench, VELOCITI, and VGenST-Bench each survive every level, with video dependence +39%, -2%, and +1%, respectively.Their references are 54%, 94%, and 88% at 32 frames, respectively.Source: X Benchmark report cards · pp. 163–164
  • VAU-Bench, VF-Eval, and VideoEspresso break at the text level; their references at 32 frames are 94%, 84%, and 80%, respectively.Their video dependence values are +11%, +28%, and +30%, respectively.Source: X Benchmark report cards · pp. 163–165
  • VNBench (25% chance), VELOCITI (50%), and VGenST-Bench (27%) each survive every level; their respective references at 32 frames are 74%, 94%, and 88%.Source: X Benchmark report cards · pp. 164–165
  • VRU-Accident has video dependence +48% and breaks at the pool level, while Video-LevelGauge has video dependence +42% and breaks at the order level.VRU-Accident has reference 76% at 32 frames; Video-LevelGauge has reference 89% at 32 frames.Source: X Benchmark report cards · pp. 164–165

Related papers

Loading 2610.00960v1…