Source-linked AI summary
STRAND: Benchmarking and Improving Object-Centric Spatio-Temporal Monitoring in Video Large Language Models
Thong Nguyen, Tri Cao, Khoi Le, Cong-Duy Nguyen, Quynh Vo, See-Kiong Ng, Bryan Hooi Kuen-Yew
TL;DR
Video MLLM evaluations often score only final answers, leaving unresolved whether models tracked the entities and events that support them. STRAND pairs targets with sub-questions and scores them jointly, while an object-centric framework reasons over trajectories; the framework narrows the target-to-faithfulness gap, with gains persisting under matched-budget comparisons.
Problem
Method
Results
The framework narrows the gap between target accuracy and Faithful Accuracy, and gains persist under matched frame-, call-, or token-budget controls.
Source: §1 · p. 2Takeaways & Limitations
Whether explicit monitoring continues to pay off as backbones improve remains an open question.
Source: §7 · p. 9The contribution of individual components is backbone-dependent, and the authors state that controls needed to isolate the stronger-backbone explanation are lacking.
Source: §6.3 · p. 9
Abstract
from arXiv · showhide
While multimodal large language models (MLLMs) have advanced video understanding, they remain highly prone to hallucinations in dynamic scenes. We argue this stems from a failure in spatio-temporal monitoring, the ability to persistently track object identities, states, and relations over time. Existing benchmarks obscure this deficit by relying on single final-answer evaluations for queries that can often be resolved via local visual cues or statistical priors. To rigorously diagnose this, we introduce STRAND, a benchmark of human-verified object-centric facts that evaluates intermediate reasoning by decomposing queries into sub-questions, distinguishing genuine temporal understanding from coincidental correctness. Crucially, we score models with Faithful Accuracy, an unconditional joint metric that credits a prediction only when the target answer and every prerequisite sub-question are correct, so that a model cannot inflate its score by being selectively consistent on the small subset of targets it happens to answer correctly. To address failure modes exposed by STRAND, we further propose an object-centric framework that explicitly constructs and reasons over structured object trajectories via chunk-wise state extraction and temporal aggregation. Extensive experiments, including backbone-, frame-, call-, and token-matched comparisons against both end-to-end MLLMs and modular video harnesses, demonstrate that our object-centric framework significantly reduces hallucinated answers and improves spatio-temporal reasoning consistency over state-of-the-art MLLMs. The code, model, and data have been made available at nguyentthong.github.io/strand.
1 INTRODUCTION
Video MLLMs can produce plausible but incorrect accounts of objects, relations, and events, while final-answer evaluation does not show whether they tracked the right entities and events. STRAND evaluates supporting facts as well as target answers, and the proposed trajectory-based framework improves consistency under matched-budget comparisons.
1 INTRODUCTION
- Dynamic-scene understanding requires persistent tracking because object identities, states, relations, and events change across time.The introduction connects this need to hallucinated objects, relations, and events in video MLLMs.Source: §1 · p. 1
- STRAND exposes what answer-only benchmarks obscure by pairing target questions with sub-questions about the facts supporting them.A correct target answer receives credit only when every supporting sub-question is also correct, using all targets in the denominator.Source: §1 · p. 2, Figure 1 · p. 2
- The framework represents videos as object trajectories, extracting states in temporal chunks, linking identities across chunks, and answering from relevant trajectories.Trajectories are built once per video and reused for its questions.Source: §1 · p. 2
- The framework narrows the gap between target accuracy and Faithful Accuracy, with gains persisting under controls matching frame, call, or token budgets.Permuting trajectory order reduces target accuracy from 77.1 to 65.3 on one backbone and from 74.0 to 54.1 on another.Source: §1 · pp. 2–3
2 RELATED WORK
Prior work advances video MLLMs, studies hallucination and mitigation, and raises the difficulty of temporal and structured reasoning benchmarks. STRAND differs by evaluating supporting facts jointly with final answers.
Video Multimodal Large Language Models
- Video MLLMs have progressed from instruction-tuned short-video assistants toward systems emphasizing long-context temporal grounding and reasoning.The section also describes open-weight, reasoning-focused, and proprietary frontier systems.Source: §2 · p. 3
Video Hallucination Benchmarks and Mitigation
- Video hallucination research targets errors in objects, actions, temporal grounding, structural reasoning, and temporal drift, with mitigation at training and inference time.Approaches include alignment, decoding strategies, and retrieval-augmented architectures.Source: §2 · p. 3
Long-Video and Multi-Hop Reasoning Benchmarks
- Long-video and multi-hop benchmarks extend temporal horizons, chain evidence across moments, or make queries object-centric, but still grade final answers alone.STRAND instead scores supporting facts jointly with the answer.Source: §2 · p. 3
Object-Centric and Structured Video Understanding
- Recent object-centric work reports improved grounding from modeling spatio-temporal interactions, while structured pipelines decompose video reasoning into multiple stages.The described stages include planning, temporal localization, evidence verification, and answer generation.Source: §2 · p. 3
3 OBJECT-CENTRIC SPATIO-TEMPORAL FORMULATION
Direct video-question mapping can entangle object identity, temporal dynamics, and reasoning signals in multi-scene videos. The paper instead represents persistent entities through time-indexed states and reasons over their trajectories.
3 OBJECT-CENTRIC SPATIO-TEMPORAL FORMULATION
- Directly mapping a video and question to an answer can entangle identities, temporal dynamics, and reasoning signals across multiple scenes, without an explicit mechanism for tracking state changes.The section contrasts this implicit representation with structured reasoning over trajectories.Source: §3 · p. 3
- Each persistent entity is represented by a trajectory of observable states over time, including unary properties and typed relations.Identity is separated from state, and answering is formulated as reasoning over trajectories.Source: §3 · p. 3
- The STRAND construction pipeline anchors timestamps, registers entities, extracts typed facts, and links identities before composing and verifying questions.Source: Figure 2 · p. 4
4 STRAND CONSTRUCTION
STRAND evaluates object-centric temporal monitoring through human-verified video facts, decomposed questions, and unconditional joint scoring. Its construction combines trained annotation, generated and reviewed questions, and blind baselines for interpreting accuracy.
4 STRAND CONSTRUCTION
- STRAND targets the gap between object-centric temporal monitoring and benchmarks that can be solved from a frame or language priors and graded only on final answers.The benchmark is intended to reveal whether models resolve temporal dependencies or guess.Source: §4 · p. 4
4.1 EXPERT-DRIVEN FACT ANNOTATION
- Fifty trained annotators construct video fact records, reaching 0.88 Cohen’s kappa agreement on object identities and state transitions.They anchor key timestamps, register task-relevant entities, record typed states and relations, and link identities into trajectories.Source: §4.1 · p. 4
4.2 AUTOMATIC QA GENERATION AND DECOMPOSITION
- An LLM generates reasoning chains and target questions from symbolic facts, then decomposes targets into atomic sub-questions probing supporting facts.Targets are designed so event order distinguishes two plausible attributions; 361 of 382 temporally constrained targets (94.5%) include an ordering sub-question meeting that condition.Source: §4.2 · pp. 4–5
4.3 EVALUATION METRICS
- Faithful Accuracy credits a target only when its answer and every supporting sub-question are correct, over a fixed denominator of all targets.Unlike Conditional Consistency, it does not depend on which targets the model answered correctly, enabling direct cross-model comparison.Source: §4.3 · p. 5
4.4 STATISTICS AND CHANCE BASELINES
- STRAND contains 88 videos, 977 targets, and 2,516 sub-questions; blind baselines reach 19.1% Faithful Accuracy for majority class, 21.4% for question-only, and 31.2% for caption-only.On ordering sub-questions, the question-only baseline reaches 49.3%, no better than guessing.Source: §4.4 · p. 6
4.5 HUMAN VERIFICATION
- Annotators review generated questions for clarity, sufficiency, relevance, and visual grounding, discarding unsupported assumptions or unreliable identity bindings.An alternative sufficient fact set was identified for 5.8% of targets.Source: §4.5 · p. 6
5 OBJECT-CENTRIC TRAJECTORY REASONING
The framework constructs object trajectories once per video, then retrieves question-relevant trajectories to answer queries. Chunk-wise state extraction and temporal aggregation turn observations into temporally ordered, linked object records.
5 OBJECT-CENTRIC TRAJECTORY REASONING
- The system reuses structured object trajectories across questions about a video rather than rereading the full frame sequence for each query.Trajectory construction uses a fixed, uniformly sampled frame budget.Source: §5 · p. 6
5.1 CHUNK-WISE STATE EXTRACTION
- Object states are extracted independently from sampled temporal chunks, enabling extraction to parallelise.The chunk-level records describe object states as relational predicates with their arguments, not merely attributes.Source: §5.1 · pp. 6–7
5.2 TEMPORAL AGGREGATION
- A deterministic symbolic aggregator links noisy, duplicate observations into time-ordered trajectories using visual similarity, temporal constraints, confidence filtering, conflict resolution, and singleton retention.Its fixed hyperparameters are not learned weights, and the default aggregator makes no VLM call.Source: §5.2 · p. 7
5.3 RETRIEVAL AND ANSWERING
- For each question, the system retrieves relevant trajectories and answers from them, directly addressing object identities, state transitions, and temporal relations.Only the retrieved trajectory subset varies by query among video-derived inputs; the uniformly sampled frame budget is fixed.Source: §5.3 · p. 7
6 EXPERIMENTS
On STRAND, the experiments compare the framework with end-to-end models, test transfer to external benchmarks, and examine matched budgets and component ablations. Results show stronger STRAND scores alongside a remaining event-ordering weakness and backbone-dependent ablation effects.
6.1 EXPERIMENTAL SETUP
- The evaluation covers STRAND, VideoHallucer, Video-MME, and EgoSchema, with STRAND’s Faithful Accuracy as the primary metric.Target Accuracy, Sub-question Accuracy, and conditional Consistency are also reported for STRAND.Source: §6.1 · p. 7
6.2 MAIN RESULTS
- For Gemini-3-Flash, individual-fact sub-questions score 78.8% versus 61.4% for event-order questions; Qwen3-VL-235B scores 75.4% versus 56.2%.The ordering gap is about 18 points across both backbones.Source: §6.2 · pp. 7–8
- 59.4% Afaith, 77.1% Atarget, 76.3% Asub, and 81.3% Acons: Gemini-3-Flash exceeds Gemini-3.1-Pro by 20.5, 15.7, 11.4, and 12.1 points, respectively.No end-to-end model exceeds 68.4% Atarget; Afaith is the primary STRAND metric.Source: §6.2 · p. 7, Table 1 · p. 8
- The Gemini-3-Flash instantiation leads the strongest baseline by 9.3 points on VideoHallucer and 1.6 on EgoSchema, but trails Gemini-3.1-Pro by 0.5 on Video-MME.Qwen improves on its extractor backbone run end-to-end by 11.6 points on VideoHallucer; these external results test generality rather than claim state of the art.Source: Table 2 · p. 8, §6.2 · p. 8
6.3 ABLATION STUDY
- At equal 64-frame, model-call, and token budgets, the framework reaches 54.2 versus 41.5, 49.3 versus 39.8, and 52.8 versus 45.1 Afaith, respectively.The passage presents these matched-resource comparisons as the basis for the design claim.Source: §6.3 · p. 9
- For answerer frame reduction, retrieval removal, and LLM summarization, Qwen3-VL-235B loses 10.4, 9.4, and 11.3 points; Gemini-3-Flash losses of 0.6, 0.1, and 0.4 are ties.Lowering extractor density to 8 frames per chunk degrades both backbones; symbolic identity linking preserves low-salience details by never regenerating states.Source: §6.3 · p. 9
6.4 QUALITATIVE ANALYSIS
- A baseline example answers a target correctly while failing its identity sub-question, illustrating shortcut reasoning that target-only scoring hides.The quantitative form of the comparison is the gap between Atarget and Afaith in Table 1.Source: §6.4 · p. 9
7 CONCLUSION
STRAND evaluates object-centric spatio-temporal monitoring by decomposing target queries into supporting sub-questions and scoring them jointly. The proposed framework builds query-relevant object trajectories, with results showing gains across question types.
7 CONCLUSION
- STRAND pairs compositional target questions with atomic sub-questions and uses Faithful Accuracy to distinguish faithful reasoning from coincidental correctness.The benchmark begins with human-verified facts about objects, states, relations, and when they change.Source: §7 · p. 9
A IMPLEMENTATION DETAILS
- The pipeline uses 15-second chunks, samples 60 frames per chunk for Eϕ, and evaluates 64 frames at Gη.Identity-linking across temporal boundaries is enabled in structured JSON pipelines and disabled in unstructured prose variants.Source: A IMPLEMENTATION DETAILS · p. 13
B QUESTION-TYPE BREAKDOWN
- The framework improves performance across all four question types, with the largest gains against Gemini-3.1-Pro averaging 20.8 points.Identity tracking and action sequencing remain the hardest categories for every evaluated model.Source: B QUESTION-TYPE BREAKDOWN · p. 13
C ERROR-TYPE BREAKDOWN
The error breakdown shows that the framework suppresses visual hallucination and state misattribution, leaving temporal-order errors as its strongest remaining failure mode. The supplied external-benchmark results are described as on par with or better than existing methods.
C ERROR-TYPE BREAKDOWN
- Temporal-order errors account for 53.5% of the strongest framework’s sub-question errors, a 19.4-point margin over the next category.These are shares of each model’s own errors, not absolute error rates; visual hallucination and state misattribution are suppressed.Source: C ERROR-TYPE BREAKDOWN · p. 13
D.1 PERFORMANCE ON OTHER VIDEO BENCHMARKS
- The framework performs on par with or better than existing video-understanding methods on VideoHallucer, Video-MME, and EgoSchema.Table 6 reports results on those external benchmarks.Source: §D.1 · p. 13, Table 6 · p. 14
D.2 ADDITIONAL ABLATION STUDY
Ablations and controlled analyses test the framework’s representation, temporal linking, retrieval, and resource costs. They report benefits from structured, ordered trajectories while also identifying accuracy, retrieval, and efficiency trade-offs.
D.2 ADDITIONAL ABLATION STUDY
- Removing text trajectories or visual frames degrades performance, while 15-second chunks balance event-level extraction against cross-chunk interactions.Thirty-second chunks dilute event-level state extraction; 7.5-second chunks fragment cross-chunk interactions.Source: §D.2 · p. 14
- Replacing structured JSON with prose degrades accuracy; dense captioning yields 72.4% for Gemini, and shuffled trajectories yield 65.3%.The authors attribute the prose limitation to obscured entity tracking, timestamps, identity links, temporal alignments, and multi-object relations.Source: §D.2 · p. 14
D.3 IDENTITY-LINKING AND TEMPORAL AGGREGATION ABLATION
- For identity linking, a 30-second temporal constraint increases false-positive matches, while a 5-second constraint fragments trajectories; the 15-second, 0.75-confidence base with bipartite matching yields optimal Faithful Accuracy.The threshold adjustment exposes a trade-off between trajectory completeness and identity precision.Source: §D.3 · pp. 14–15, Table 7 · p. 15
D.4 BUDGET-MATCHED COMPARISONS
- Across matched frame, call, token, and backbone settings, explicit trajectory construction yields higher Faithful Accuracy than end-to-end MLLMs.The comparisons use a Qwen3-VL backbone for both the baseline and framework, varying representation within each control.Source: §D.4 · p. 15, Table 8 · p. 15
D.5 SINGLE-FRAME SOLVABILITY AND SHORTCUT AUDIT
- With a single center frame, audited frontier models reach only 28.2%–32.7% target accuracy and below 15% Faithful Accuracy.The audit concludes that the tested queries require multi-step spatio-temporal reasoning for the audited models.Source: §D.5 · p. 16, Table 10 · p. 16
D.6 TRAJECTORY RETRIEVAL ANALYSIS
- Retrieval reduces mean trajectory triplets from 145.2 per video to 24.6 per query, reaching 88.4% precision and 92.1% recall against gold chain facts.In 12.4% of queries, timeline clutter misleads the unfiltered generator while filtering succeeds; dropping one gold triplet reduces target accuracy on those queries to 51.2%.Source: §D.6 · p. 16
D.7 EFFICIENCY AND INFERENCE COST
- At 39.7 questions per video and 12.2 extraction calls on average, the framework uses 91.6 calls versus 39.7 for an end-to-end baseline, a 2.3× overhead.Construction is a one-off per-video cost, so its amortized cost per question falls as more questions are asked; Qwen per-query latency is 2.9 seconds versus 2.8 seconds end-to-end.Source: §D.7 · p. 17
E DATA PROVENANCE, LICENSING, AND ANNOTATION ETHICS
The benchmark draws on freely licensed public video and uses visually grounded annotations without biometric or identity attributes. Its appendix also documents the prompts and reproducibility limits of its generation process.
E DATA PROVENANCE, LICENSING, AND ANNOTATION ETHICS
- All 88 videos come from Wikimedia Commons, and the released files carry per-file attribution while fact annotations and QA pairs use CC-BY-NC 4.0.The videos are redistributed under their respective free-content licences.Source: E DATA PROVENANCE, LICENSING, AND… · p. 17
- People appear incidentally in public footage; annotations use visually grounded identifiers and record no biometric or identity attributes.The study reports a 50-annotator Prolific pool paid $15 per hour, with protocol and working conditions approved by the institutional ethics board.Source: E DATA PROVENANCE, LICENSING, AND… · p. 17
- STRAND is intended as a diagnostic benchmark and not for training surveillance or identity-tracking systems.Source: E DATA PROVENANCE, LICENSING, AND… · p. 17
F PROMPTS
- The appendix provides prompts for extraction, aggregation, retrieval, answering, and question decomposition; exact seeding is limited to open-weight models served through vLLM.API-served models are repeatable in distribution rather than byte for byte, and rerunning generation reproduces candidate questions rather than the human-verified released set.Source: F PROMPTS · pp. 17–19
G LIMITATION
The paper identifies efficiency, benchmark scope, and missing controls as boundaries on its claims. It also cautions that some baseline comparisons and consistency metrics do not support straightforward conclusions.
G LIMITATION
- The framework adds preprocessing overhead, particularly for extended videos or dense tracking, and trades strict real-time applicability for accuracy.The authors leave efficiency improvements for future work.Source: G LIMITATION · p. 19
- At 88 videos and 977 targets, aggregate comparisons are well resolved, but per-domain claims are not; domain breakdowns are descriptive only, and no stable model ordering within any single domain is claimed.Source: G LIMITATION · p. 19
- Three same-backbone controls remain absent, leaving the effects of sub-question prompting, modularity, and identity linking unisolated.The missing controls are a prompted raw MLLM, a modular baseline without trajectories, and a no-identity-linking ablation.Source: G LIMITATION · p. 19
- STRAND’s trajectory-centered design may favor the proposed framework; the authors limit their generality claim to the support provided by external-benchmark results.A benchmark built around a non-trajectory representation would be a stronger test, but was not run.Source: G LIMITATION · p. 19
- Yes/no targets and sub-questions are judge-free and atomic, but remain vulnerable to answer priors and leading prompts despite label balancing and prior-only baselines.The authors identify contrastive and open-form probes as the most valuable extension.Source: G LIMITATION · pp. 19–20
- The full-context control lowers Qwen3-VL-235B target accuracy from 74.0% to 61.2%, but this is a single-backbone result that the authors do not generalise.The ablation varies context use, not chunk length alone.Source: G LIMITATION · p. 20
H.2 WHICH COMPARISON SUPPORTS THE DESIGN CLAIM
- The structured-pipeline baselines use much smaller backbones, so those comparisons do not isolate harness design from model capacity.The authors do not rest their design claim on these rows.Source: §H.2 · p. 20
H.3 CONDITIONAL CONSISTENCY IS NOT COMPARABLE ACROSS MODELS
- Acons can favor models that answer fewer, easier targets; the authors therefore make consistency claims using unconditional Afaith instead.Cosmos-Reason2-8B has the table’s highest Acons while answering 22.4 points fewer targets correctly than Gemini-3-Flash.Source: §H.3 · p. 20
H.4 ABLATION LIMITATIONS AND MISSING COUNTERFACTUALS
The extended discussion narrows the mechanism claim to what the available controls support and documents constraints on the benchmark, representation, and pipeline. Its ablations show that component importance differs by backbone.
H.4 ABLATION LIMITATIONS AND MISSING COUNTERFACTUALS
- Three controls remain untested: sub-question prompting on a matched raw MLLM, a same-backbone modular baseline without trajectories, and disabling identity linking.The missing identity-linking ablation means cross-chunk binding is not isolated as the mechanism.Source: §H.4 · p. 20
H.5 TWO INSTANTIATIONS OF THE AGGREGATOR
- Deterministic concatenation merges linked chunk observations without regenerating states, whereas LLM-summarize can drop low-salience states and performs worse in the ablation.The discussion attributes that loss to summarization dropping states later found decisive.Source: §H.5 · p. 21
H.6 WHAT RETRIEVAL CONTRIBUTES
- Removing retrieval costs Gemini-3-Flash 0.1 points, a tie, while Qwen degrades meaningfully; the authors view retrieval as a context-length aid, not a source of accuracy.The discussion says retrieval matters when the answerer cannot reliably attend over the full trajectory set.Source: §H.6 · p. 21
H.8 ON THE UNIQUENESS OF A DECOMPOSITION
- The verified sub-questions form a sufficient, not unique, reasoning path: alternative sufficient fact sets occur for 5.8% of targets, while annotators agree exactly on 92.0% of sampled minimal sets.Agreement was measured on 100 sampled targets, with Fleiss’ κ = 0.87.Source: §H.8 · p. 21
H.9 ANSWER FORMAT AND CHANCE BASELINES
- Binary items support atomic prerequisite checks and judge-free grading, but remain vulnerable to answer priors; question-only and caption-only baselines reach 21.4% and 31.2% Afaith, respectively.A majority-class predictor reaches 19.1% Afaith; ordering questions fall to 49.3% for the question-only predictor.Source: §H.9 · p. 22
H.10 DATASET STATISTICS
- STRAND has 88 videos, 977 targets, and 2,516 sub-questions; its scale supports aggregate comparisons but not strong per-domain claims.With video-clustered bootstrap intervals, ablation differences below 2.0 points are treated as ties.Source: §H.10 · p. 22
H.11 IDENTITY-LINKING ALGORITHM
- Identity linking filters candidate matches by temporal proximity and confidence, resolves conflicts one-to-one, and sorts linked observations into trajectories.An unmatched observation is retained as a singleton trajectory.Source: §H.11 · p. 22
H.15 BACKBONE DEPENDENCE OF THE MECHANISM
- For Qwen3-VL-235B, dropping the answerer to 8 frames costs 10.4 points, removing retrieval filtering costs 9.4, and LLM-summarize aggregation costs 11.3; the corresponding Gemini-3-Flash changes are 0.6, 0.1, and 0.4 points, each within a tie.The authors restrict the claim that each pipeline stage contributes to the Qwen setting; for Gemini, they claim only that temporal ordering is necessary.Source: §H.15 · p. 24
I STRAND SAMPLES
The STRAND examples illustrate target questions paired with supporting sub-questions that test temporal and visual grounding. Additional open-weight baselines add measurements without changing the reported ordering.
I STRAND SAMPLES
- Figures 6–8 illustrate multi-step interactions, temporal ordering, and actor disambiguation through target questions and prerequisite sub-questions.The examples require tracking event sequences and distinguishing objects or actors across time.Source: I STRAND SAMPLES · p. 25, Figure 7 · p. 26, Figure 8 · p. 27
I.1 ADDITIONAL OPEN-WEIGHT BASELINES
- Qwen3-VL-8B Think reaches 42.5 Afaith, 61.3 Atarget, 70.1 Asub, and 77.4 Acons; none of the four additional models changes the table ordering.The other models are Qwen3-VL-32B Instruct, Qwen3.5-9B, and Qwen3-VL-8B Instruct.Source: §I.1 · p. 26