PaperlayerView exact revision on arXiv

Source-linked AI summary

ActionLens: Diagnosing Spatial-Temporal Binding Failures in Vision-Language Models

Gueter Josmy Faure, Min-Hung Chen, Hao Ping Wang, Timothée Lardy, Hung-Ting Su, Winston H. Hsu

arXiv:2609.34547v1cs.CVcs.AIcs.CL

TL;DR

Video benchmarks often score highly while leaving spatial-temporal binding—the association of actions with the right people and moments—under-tested. ActionLens evaluates that capability with five diagnostics and deterministic annotations. The strongest open-weight model reaches 68.8% on the full set, while scoring 65.9% against pooled humans’ 91.0% on the reviewed subset.

  • Problem

    Popular video benchmarks often emphasize scene-level recognition, leaving limited evaluation of whether models bind actions to the correct person and moment.

    Source: §1 · p. 2

  • Method

    ActionLens evaluates five spatial-temporal binding diagnostics with deterministic answers derived from per-second, per-person annotations and a 14-round human quality-engineering process.

    Source: §1 · p. 2

  • Results

    The strongest open-weight model scores 68.8% on the full set; on the human-reviewed subset, it scores 65.9% versus 91.0% for the pooled human reference.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Diagnostic scores distinguish targeted weaknesses, including actor disambiguation and gaze detection, rather than reducing performance to a single aggregate measure.

    Source: §1 · p. 2

    Movie casting and editing constrain the actions and contexts represented, so benchmark scores should not be used to infer suitability for surveillance or other consequential decisions.

    Source: ETHICS STATEMENT · p. 10

Abstract

from arXiv · show

Video-capable vision-language models score above 80\% on popular benchmarks yet struggle with spatial-temporal binding: associating the right action with the right person at the right moment. We introduce ActionLens, a diagnostic benchmark of 6,701 multiple-choice video questions spanning five targeted diagnostics: transition detection, actor-specific identification, concurrent action binding, directed interaction reasoning, and gaze detection. Ground-truth answers are derived deterministically from 1.58 million per-second, per-person annotations. Fourteen rounds of human quality engineering raised answer clarity from 53% to above 90% human accuracy. Across 20 VLMs, the full-set leader scores 68.8%; on the human-reviewed subset, it scores 65.9% versus 91.0% for the pooled human reference. Gaze detection remains near chance against 89.6% human accuracy. On actor disambiguation, reference-interface controls show that relational descriptions recover 5.55--13.25 points over static coordinates, confirming a substantial numeric-parsing penalty; yet visual boxes still lead every model by 1.15--6.50 points, exposing a residual unboxed actor-resolution gap. A binding-trap analysis shows models systematically select the wrong actor's action. ActionLens provides diagnostic measurements of these distinct failure modes across model families and scales for direct comparison. We release all data, code, and evaluation scripts at https://anonymous.4open.science/r/lmms-eval-2276

1 INTRODUCTION

ActionLens targets spatial-temporal binding—the ability to connect the right action to the right person at the right moment—which popular video benchmarks often do not test. Its evaluation finds substantial model–human gaps and distinct weaknesses across binding diagnostics.

1 INTRODUCTION

  • Popular video benchmarks often exceed 80% accuracy while emphasizing scene-level recognition rather than asking which co-present person performs a particular action.The target capability is spatial-temporal binding: associating the right action with the right agent at the right moment.Source: §1 · p. 2
  • ActionLens comprises 6,701 video questions across five diagnostics, with deterministic answers from 1.58 million per-second, per-person annotations and 14 rounds of human quality engineering.The review process raised question clarity from 53% to above 90% human accuracy; the benchmark also includes controls and analyses targeting shortcuts and binding errors.Source: §1 · p. 2
  • The strongest open-weight model scores 68.8% on the full set and 65.9% on the human-reviewed subset, versus 91.0% for pooled humans.Gaze detection remains near chance for most models despite 89.6% pooled-human accuracy; relational descriptions recover 5.55–13.25 points over static coordinates, but visual boxes lead every model by 1.15–6.50 points.Source: §1 · p. 2
  • ActionLens treats each score as a measure of a targeted operation, making specific weaknesses more actionable than an aggregate score alone.The paper frames naming a failure as a first step toward fixing it.Source: §1 · p. 2

2 RELATED WORK

Prior video benchmarks cover diverse question-answering, temporal, and spatial tasks, but do not jointly test per-person grounding with temporal transitions and programmatically verified answers. ActionLens builds on dense person-action annotations and controlled spatial-binding comparisons.

2 RELATED WORK

  • Existing benchmarks span question-answering, temporal reasoning, interaction, and long-video evaluation, but typically assess scene-level or other task-specific capabilities.Examples include NExT-QA, STAR, MVBench, VideoMME, and EgoSchema.Source: §2 · p. 3
  • No existing benchmark jointly requires per-person spatial grounding, temporal transition detection, and programmatically verified ground truth.Benchmarks with multiple people still treat the scene holistically rather than binding actions to specific individuals.Source: §2 · p. 3
  • AVA supplies 1.58 million per-second, per-person action annotations, from which ActionLens derives ground-truth answers deterministically.The paper contrasts this approach with recent benchmarks relying on LLM-generated labels.Source: §2 · p. 3
  • Spatial-grounding work spans static-image relationships and 3D scenes; ActionLens extends entity-binding questions to video through controlled comparisons of boxes, coordinates, and relational descriptions.Winoground highlights role confusion, while VSR evaluates spatial relations in static images.Source: §2 · p. 3

3 THE ACTIONLENS BENCHMARK

ActionLens is designed to isolate distinct spatial-temporal binding operations using deterministic labels and questions that resist shortcuts. Its construction combines five diagnostics, iterative human review, and a dataset spanning 6,701 questions.

3.1 DESIGN PHILOSOPHY

  • ActionLens decomposes fine-grained video understanding into targeted operations with unambiguous answers, using deterministic ground truth from AVA v2.2’s 1.58 million per-second, per-person annotations.Answer generation requires no human labeling, distinguishing the benchmark from work using LLM-generated labels.Source: §3.1 · pp. 3–4

3.2 THE FIVE DIAGNOSTICS

  • Transition Sensitivity tests whether models detect a tracked person’s before-to-after action change rather than temporal reversal or either static action.Clips last 4–10 seconds; boundaries require two stable seconds before and after the change, and auditory-only transitions are excluded.Source: §3.2 · p. 4
  • Actor Disambiguation asks which action belongs to a person in a colored box, placing the other person’s action among the distractors to test person-action binding.All distractors are verified false for the queried person at the query second.Source: §3.2 · p. 4
  • Concurrent Binding asks for both simultaneous actions of one person, with physically compatible distractor pairs that prevent biomechanical-impossibility shortcuts.Distractors also exclude actions that are true of the highlighted person.Source: §3.2 · p. 4
  • Directed Interaction Reasoning varies which person holds each communication role and visible co-action; Gaze Detection asks which highlighted person is watching someone.Gaze labels use AVA’s per-person binary annotations with a three-second temporal consistency filter.Source: §3.2 · pp. 4–5

3.3 QUALITY ENGINEERING

  • Fourteen human-review rounds raised pilot accuracy from 53% to 74% in rounds 1–6, then addressed subtler identity, action-pair, and temporal-label issues.The process pruned Temporal Localisation and standalone Talker-Listener Detection; Interaction Reasoning retained talk/listen labels in composite choices with visible co-actions.Source: §3.3 · p. 5
  • The most over-represented answer position deviates from the 25% baseline by at most 5.1% for Interaction Reasoning and 3.4% for other tasks.Interaction Reasoning’s slight imbalance reflects a 21% three-option subset, which leaves option D under-represented by design.Source: §3.3 · p. 5

3.4 DATASET STATISTICS

  • The benchmark contains 6,701 questions from 64 videos, spans 73 actions, and uses a 1,194-item pooled-human-reference subset.Main-task clips last five seconds except variable-length Transition Sensitivity clips; the reference subset includes all 194 transition items plus 250 randomly selected items per other diagnostic.Source: §3.4 · p. 6

4 EXPERIMENTS & ABLATIONS

Across 20 VLMs, ActionLens reveals large gaps from human performance and uneven results across diagnostics. Interface and frame-budget controls further show that actor references and temporal sampling affect performance differently across tasks.

4.1 RESULTS

  • Gaze Detection reaches 89.6% for the pooled human reference, versus 47.2% for the best model condition; on the full set, Qwen3.5-27B leads at 43.3%.Every other full-set model scores below 40%, and several are near chance.Source: §4.1 · p. 6
  • Actor-disambiguation scores span 34.0% to 67.7%, and most tested models select the wrong actor’s action more often than uniform distractor guessing predicts.The paired video-cluster bootstrap finds a 22.7-point gap between Qwen3.5-27B and LLaVA-Video-32B, while their 2.25-point lead over InternVL3.5-38B is unresolved.Source: §4.1 · p. 6
  • 65.9% for Qwen3.5-27B on the human-reviewed subset trails the pooled human reference’s 91.0%, while its full-set score leads at 68.8%.Table 2 reports per-diagnostic and average accuracy for the models and pooled human reference.Source: §4.1 · p. 6, Table 2 · p. 7
  • Qwen3.5-4B and InternVL3.5-38B both score 63.6% overall, while video-level resampling confirms larger cross-family gaps but not small leaderboard separations.Qwen3.5-27B leads InternVL3.5-38B by 5.18 points in question-micro accuracy (95% CI [3.61, 6.75]); training and input differences co-vary, so the comparison identifies no single cause.Source: §4.1 · pp. 6–7

4.2 REFERENCE INTERFACE IS AN ACTOR-BINDING BOTTLENECK

  • On actor disambiguation, relational descriptions improve over static coordinates by 5.55–13.25 points but remain 1.15–6.50 points below visual boxes for every tested model.For two-person interaction reasoning, there is no universal ordering of unboxed references; visual boxes lead for four models, while Qwen3.5-27B scores highest with coordinates at 79.58%.Source: §4.2 · pp. 8–9

4.3 WHAT’S THE OPTIMAL FRAME BUDGET

  • All three frame-sweep models improve substantially from 1 to 8 frames, then plateau; doubling the budget from 32 to 64 yields essentially no gain.The authors identify 32 frames as a sufficient operating point for the main evaluation; Figure 3’s temporal-control panel reports little gain beyond 8–16 frames.Source: Figure 3 · p. 8, §4.3 · p. 9
  • Transition Sensitivity benefits most from additional frames, while Concurrent Action Binding can be largely solved from one well-chosen frame; InternVL3.5-8B and Qwen3.5-4B also differ in frame sensitivity.InternVL3.5-8B’s weighted average is 57.8% at 1 frame and 57.6% at 32, whereas Qwen3.5-4B gains 13.5 points from 1 to 8 frames.Source: §4.3 · p. 9

4.4 CAN MODELS SOLVE ACTIONLENS FROM TEXT ALONE?

  • Removing video lowers question-micro accuracy by 15.2–33.4 points across five models, while transition questions can invite guesses from familiar action sequences and gaze remains difficult with video.Actor disambiguation, concurrent binding, and interaction reasoning decline for every model when the clip is removed.Source: §4.4 · p. 9

5 CONCLUSION

ActionLens makes distinct video-model failures measurable: the conclusion highlights wrong-actor selection, gaze near chance, and a remaining gap between unboxed relational descriptions and visual boxes. The paper also documents benchmark scope, evaluation setup, quality engineering, and limits on interpreting some diagnostics.

5 CONCLUSION

  • 13 of 16 models were more likely to select the wrong actor’s action than guess randomly, while every model still trailed visual boxes using ordinary relational descriptions.The conclusion also reports a 25-point gap between the best open-weight model and pooled human reference on the same 1,194 questions, and gaze near chance.Source: §5 · p. 9
  • Movie casting and editing limit represented actions and contexts, so benchmark scores should not be used to infer suitability for surveillance or other consequential decisions.Source: ETHICS STATEMENT · p. 10

B.1 MODELS / B.2 PROMPT TEMPLATE / B.3 VIDEO SAMPLING AND HARDWARE

  • The appendix identifies the evaluated models and uses one shared multiple-choice template, with 32-frame sampling by default and model-specific hardware settings.The template uses no chain-of-thought, few-shot examples, or task-specific system prompt; 21% of Interaction Reasoning items omit option D.Source: §B.1 · p. 13, §B.2 · p. 13, §B.3 · p. 13

B.4 METRIC DETAILS

  • Question-micro accuracy weights questions across the 6,701-item set, while diagnostic-macro accuracy averages the five diagnostic accuracies equally.Correctness is whether the first predicted option letter matches the ground-truth label.Source: §B.4 · p. 14

C QUALITY ENGINEERING: THE 14 ROUNDS

C.1 DETAILED FIX LOG

  • Quality fixes excluded target-person actions from Actor Disambiguation distractors, limited Concurrent Binding to exactly two visual actions, and removed auditory-only transitions.Interaction Reasoning distractors were structured variants; when a scene alternative was unavailable, the question used three options.Source: §C.1 · pp. 14–15
  • Later fixes used AVA person IDs to prevent identity drift and required stable action regions around transition boundaries, producing variable-length clips averaging 7.1 seconds.The revision log also addressed confusable transitions and shared co-actions that could leave multiple valid answers.Source: §C.1 · p. 15

C.2 RETIRED DIAGNOSTICS

  • Talker/Listener Detection was retired because muted-video questions could rely on speech cues; Interaction Reasoning uses talk/listen labels only in composite choices and does not independently validate every role label.Temporal Localisation was also retired because action-onset answers lacked a defensible tolerance window.Source: §C.2 · p. 15

D DATASET COMPOSITION

The dataset composition and reference-control analyses describe how ActionLens balances its diagnostics and how model performance varies with actor references and model family. The supplied passages also report a systematic association between actor-disambiguation accuracy and binding-trap responses.

D DATASET COMPOSITION

  • The reported distribution assigns 2% to changes and 3% to others, reflecting AVA’s annotation density rather than a design choice; it indicates no single trivially recognisable pattern dominates this diagnostic.Source: D DATASET COMPOSITION · p. 16
  • Interaction Reasoning uses role-swap, co-action-swap, and scene-alternative distractors; 21% of items use three options when a scene alternative is unavailable.Unique co-actions help prevent role-swap distractors from remaining factually correct.Source: D DATASET COMPOSITION · pp. 16–17
  • Gaze Detection uses per-person watch labels rather than directional gaze, filters for temporal consistency, and balances four gaze categories at 224 items each.The four categories fall within 1.5 percentage points of the uniform 25% baseline.Source: D DATASET COMPOSITION · p. 17, Figure 5 · p. 17

E STATIC-COORDINATE REFERENCE: FULL RESULTS

F RELATIONAL REFERENCE CONTROL

  • In crowded scenes, relational references trail visual boxes by 0.8–4.1 points for two-person scenes and 8.1–13.1 points for four-person scenes.Relational and static-coordinate conditions both use unboxed video; only the target description changes.Source: F RELATIONAL REFERENCE CONTROL · p. 18, Table 7 · p. 18

G MODEL FAMILIES AND INPUT HANDLING

  • Actor-disambiguation scaling differs across model families: InternVL3.5-38B exceeds InternVL3.5-4B by 10.70 points, while Qwen3.5-27B versus Qwen3.5-4B has an interval spanning zero.The reported interval for the Qwen3.5 difference is [−1.34, 4.66]; the family comparisons motivate hypotheses, not single-cause attribution.Source: G MODEL FAMILIES AND INPUT HANDLING · p. 19

H BINDING TRAP ANALYSIS: FULL RESULTS

  • Actor Disambiguation accuracy is negatively correlated with binding-trap rate; the trap rate measures incorrect answers choosing the other person’s correct action, against a 33.3% random expectation.Source: H BINDING TRAP ANALYSIS: FULL RESULTS · p. 19

I PER-DIAGNOSTIC FRAME SWEEP

The frame-sweep results show that frame needs differ by diagnostic: transition detection benefits most from additional frames, while Concurrent Binding is a snapshot task. Most model-diagnostic combinations largely saturate by 8–16 frames.

I PER-DIAGNOSTIC FRAME SWEEP

  • Binding-trap rate is the fraction of incorrect Actor Disambiguation responses selecting the other person’s correct action; points above the 33.3% random-expectation line indicate systematic bias toward that action.Source: Figure 7 · p. 20, Table 9 · p. 20
  • Transition Sensitivity is the most frame-hungry diagnostic, while Concurrent Binding peaks at one frame.Actor Disambiguation and Interaction Reasoning show strong frame sensitivity for Qwen3.5-4B but near-flat curves for InternVL3.5-8B.Source: I PER-DIAGNOSTIC FRAME SWEEP · p. 19, Figure 8 · p. 21
  • Most model-diagnostic cells stop changing after 8–16 frames, with Transition Sensitivity the primary exception.The heatmap encodes model-diagnostic pairs by rows, frame counts by columns, and higher accuracy with darker cells.Source: I PER-DIAGNOSTIC FRAME SWEEP · p. 20, Figure 9 · p. 21

J TEXT-ONLY ABLATION: FULL RESULTS

The text-only ablation finds that wording alone does not recover person-action associations for actor, concurrent, or interaction tasks. Transition wording can provide familiar action-sequence cues, limiting what that diagnostic measures.

J TEXT-ONLY ABLATION: FULL RESULTS

K LIMITATIONS

The reported results are bounded by the benchmark’s source domain, the smaller transition diagnostic, the human-reference protocol, and sampling uncertainty.

K LIMITATIONS

  • Because ActionLens uses only Hollywood-film videos from AVA v2.2, its performance gaps may not transfer directly to sports, surveillance, or egocentric video.Those domains can differ in action distributions, camera motion, and occlusion patterns.Source: K LIMITATIONS · p. 22
  • Transition Sensitivity has 194 items, so conclusions about it have lower statistical power than conclusions about the other four diagnostics.Its size is constrained by AVA’s annotation density for clear action-boundary transitions.Source: K LIMITATIONS · p. 22
  • The 91.0% human reference comes from one response per item on the retained 1,194-item set, and author involvement in quality review leaves potential familiarity bias uncontrolled.Questions were iteratively filtered for visual answerability.Source: K LIMITATIONS · p. 22
  • GPT-5.2 and Gemini 3 Flash were evaluated only on the 1,194-item review subset because of API cost, limiting direct comparison with open-source full-set scores.The two scopes should not be treated as equivalent for direct model comparisons.Source: K LIMITATIONS · p. 22
  • Although decoding is deterministic, scores can change with the sampled source videos, and small model differences are unstable when paired intervals cross zero.The 6,701 questions are concentrated in 64 videos; paired video-cluster bootstrap intervals and video-subset analyses address this uncertainty.Source: K LIMITATIONS · pp. 22–23

L STATISTICAL UNCERTAINTY AND VIDEO-LEVEL ROBUSTNESS

The analysis tests robustness to video sampling, explores portability to sports footage, compares prompting strategies, and checks whether target-box color affects actor-disambiguation answers.

L STATISTICAL UNCERTAINTY AND VIDEO-LEVEL ROBUSTNESS

  • Across 64 videos, cluster-interval half-widths are 1.29–2.53 times question-level intervals, while equal-video weighting shifts question-micro scores by at most 1.58 points.Leave-one-video-out diagnostic-macro rankings have at least 0.996 correlation with the full ranking; sampling one question per video–diagnostic cell yields mean correlation 0.941 [0.882, 0.981].Source: L STATISTICAL UNCERTAINTY AND… · pp. 23–24

M PORTABILITY TO SPORTS VIDEO

  • In a 500-candidate MultiSports pilot, wrong-actor errors recur in all three models but remain below the 33.3% uniform-choice expectation; these unaudited results are directional evidence only.The pilot candidates have not passed ActionLens’s visual audit and failure-coded validation.Source: M PORTABILITY TO SPORTS VIDEO · p. 24

N PROMPTING SENSITIVITY

  • Reasoning prompts improve gaze detection and modestly help InternVL3.5-4B overall, but lower both Qwen3.5 models’ overall accuracy; Qwen3.5-27B interaction accuracy falls from 76.4% to 39.6% with plan-and-solve.Neither tested reasoning prompt closes the gap to the 91.0% pooled human reference.Source: N PROMPTING SENSITIVITY · p. 25

O TARGET-BOX COLOR SWAP

  • There is no consistent aggregate preference for red or blue target boxes, though predictions agree across color conditions on 77.5–92.3% of items and wrong-actor choices remain frequent.The color swap kept the queried person, clip interval, options, answer letter, and option order fixed.Source: O TARGET-BOX COLOR SWAP · p. 25

P SCOPE OF THE HUMAN-REVIEW SUBSET

The human-review subset broadly preserves matched-model comparisons with the full benchmark, but its absolute accuracy is not interchangeable with full-set accuracy.

P SCOPE OF THE HUMAN-REVIEW SUBSET

  • The 1,194-item subset includes all 194 Transition Sensitivity questions and 250 fixed-seed, round-robin samples from each other diagnostic, covering all 64 source videos in aggregate.The comparison uses six open models evaluated under the same protocol on both scopes.Source: P SCOPE OF THE HUMAN-REVIEW SUBSET · p. 25
  • Across six models evaluated on both scopes, mean absolute full/subset differences are 0.70 points for diagnostic macro and 2.03 for question micro, with a maximum question-micro shift of 3.20 points.The full-set leader remains the subset leader.Source: P SCOPE OF THE HUMAN-REVIEW SUBSET · pp. 25–26, Table 17 · p. 26
  • Because the subset changes diagnostic proportions, question-micro and diagnostic-macro aggregates need not shift together, and subset accuracy should not stand in for full-set accuracy.Micro weights questions equally; macro weights the five diagnostics equally.Source: Table 17 · p. 26, P SCOPE OF THE HUMAN-REVIEW SUBSET · p. 26

Q REPRODUCIBILITY

The paper provides the materials and settings needed to reproduce ActionLens evaluations, alongside its human-reference collection protocol.

Q REPRODUCIBILITY

  • The release includes the full pipeline, task configurations, raw evaluation logs, and scripts for one-command evaluation.The listed release location is the ActionLens project repository.Source: Q REPRODUCIBILITY · p. 26
  • Runs use temperature=0, do_sample=False, and num_beams=1, making results deterministic given identical model weights and hardware.Source: Q REPRODUCIBILITY · p. 26
  • Four non-author raters independently answered disjoint items in the 1,194-item review subset, watching each full clip without a time limit.The pooled human reference is 1,086/1,194 correct (91.0%).Source: Q REPRODUCIBILITY · p. 26

Related papers

Loading 2609.34547v1…