Source-linked AI summary
What's the Catch? Evaluating Temporal Consistency in Vision-Language Models
Marek Hradil, Danae Sánchez Villegas
TL;DR
The paper asks whether VLMs genuinely capture temporal structure, a question obscured by benchmarks that can permit shortcut exploitation. It introduces TimeCatch, a controlled anomaly-detection benchmark separating temporal frame swaps from Gaussian-noise frame corruptions across four datasets. VLMs perform well on frame-level anomalies but remain near chance on temporal anomalies, unlike humans, indicating difficulty integrating information across frames.
Problem
It remains unclear whether strong VLM performance on video and image-sequence benchmarks reflects temporal reasoning or shortcut exploitation.
Method
TimeCatch evaluates temporal grounding through detection and localization of adjacent-frame swaps and Gaussian-noise frame anomalies across synthetic and real-world datasets.
Results
VLMs reliably detect and often localize frame-level anomalies but perform near chance on temporal anomaly tasks, while humans achieve near-ceiling performance on both.
Takeaways & Limitations
Current VLMs can identify anomalies within individual frames but struggle to reason about temporal consistency across image sequences.
Takeaways & Limitations
TimeCatch isolates adjacent-frame-swap violations and does not comprehensively evaluate long-range dependencies, causal event reasoning, or motion continuity.
Abstract
from arXiv · showhide
Vision-language models (VLMs) achieve strong performance on video and image-sequence benchmarks, yet it remains unclear whether they capture temporal structure. To study this question, we formulate temporal grounding as an anomaly detection problem, providing a simple and controlled evaluation that directly tests sensitivity to temporal consistency. We introduce TimeCatch, where temporal anomalies are created by swapping consecutive frames and frame-level anomalies by replacing a frame with Gaussian noise. Models are evaluated on anomaly detection and localization tasks across four synthetic and real-world datasets, alongside a human study. Our evaluation reveals a substantial gap between frame-level and temporal anomaly detection. While VLMs consistently detect frame-level anomalies and often localize them accurately, they perform near chance on temporal anomaly detection and only modestly above chance on localization. Humans, in contrast, achieve near-ceiling performance on both tasks. Additional analyses across model scales, prompting strategies, sequence lengths, and visual similarity suggest that these failures cannot be explained solely by limitations in perception or model capacity. Together, these findings indicate that current VLMs can identify anomalies within individual frames but struggle to integrate information across frames to reason about temporal consistency. TimeCatch provides a controlled benchmark for evaluating temporal grounding in vision-language models.
1 Introduction
Although VLMs perform strongly on many video benchmarks, it remains unclear whether they capture temporal structure rather than exploiting shortcuts. TimeCatch addresses this gap by testing temporal consistency separately from frame-level anomaly perception.
- Motivation: Existing video benchmarks may not reveal whether successful performance reflects temporal reasoning or shortcut exploitation.This matters in applications where visual information evolves over time, including autonomous driving, robotics, medical imaging, and video understanding.
- TimeCatch: TimeCatch treats temporal grounding as anomaly detection in image sequences, distinguishing temporal inconsistencies from within-frame corruption.Temporal anomalies swap consecutive frames, whereas frame-level anomalies replace a frame with Gaussian noise.
- TimeCatch: The benchmark evaluates anomaly detection and localization across four datasets spanning synthetic and real-world domains.It is designed as a controlled evaluation of temporal grounding.
- Findings: VLMs reliably detect frame-level anomalies but perform near chance on temporal anomaly detection and only modestly above chance on temporal localization.Humans achieve near-ceiling performance on both temporal tasks.
- Findings: The results indicate that current VLMs reason effectively about individual frames but struggle with temporal consistency across image sequences.The paper presents this as evidence of a substantial gap between human and VLM performance.
2 Related Work
Prior temporal-reasoning benchmarks often use indirect question-answering or discriminative tasks that can permit shortcuts. TimeCatch instead isolates temporal consistency with controlled perturbations while preserving the underlying visual content.
- Existing benchmarks: Question-answering and discriminative benchmarks measure temporal reasoning indirectly and do not necessarily require detecting temporal-consistency violations.Ordering-based approaches are more closely related but differ from anomaly detection over image sequences.
- Datasets: The benchmark covers synthetic CLEVRER and CRAFT sequences alongside real-world DriveLM and MTL-AQA sequences.The four datasets span synthetic and real-world domains.
- Evaluation challenges: Models can achieve strong benchmark performance by processing only a subset of frames or skipping large portions of image sequences.These findings motivate evaluations that reduce opportunities for shortcut exploitation.
- Evaluation challenges: Temporal-reasoning evaluation must isolate temporal consistency from other information sources while minimizing shortcut opportunities.These are identified as central challenges for evaluating temporal reasoning.
- TimeCatch: TimeCatch addresses these challenges by introducing controlled temporal perturbations without changing the underlying visual content.This design evaluates temporal consistency in isolation.
3 Evaluation Framework
TimeCatch evaluates temporal grounding through four controlled anomaly tasks over image sequences. It contrasts adjacent-frame swaps with Gaussian-noise corruption across curated synthetic and real-world datasets.
- Tasks: TimeCatch includes temporal anomaly detection, temporal anomaly localization, frame-level anomaly detection, and frame-level anomaly localization.The tasks require recognizing whether a sequence is inconsistent with plausible temporal progression, without forecasting or domain-specific knowledge.
- Temporal anomalies: Temporal anomalies are created by swapping a randomly selected consecutive frame pair, and localization requires predicting the swap location.Detection asks whether the resulting sequence contains a temporal anomaly.
- Frame-level controls: Frame-level anomalies replace a uniformly sampled frame with Gaussian noise, providing a control condition for separating anomaly perception from temporal reasoning.Localization requires identifying the corrupted frame.
- Dataset curation: Dataset selection requires interpretable events, a single temporal trajectory, distinguishable consecutive-frame changes, and a consistent viewpoint.These criteria reduce confounds such as scene cuts, domain expertise, and ambiguous event structure.
- Datasets: The benchmark combines synthetic CLEVRER and CRAFT data with real-world DriveLM and MTL-AQA data.Sequences are formed from four dataset test splits spanning both domains.
- Sampling: Main experiments use mostly 4–8-frame sequences, while longer CRAFT sequences with 9–16 frames support sequence-length analysis.Video datasets are temporally subsampled and filtered so temporal anomalies remain perceptually meaningful.
4 Experimental Setup
The experiments evaluate five open-weight VLMs with a unified zero-shot prompting protocol. Detection and localization are both measured with accuracy under constrained output formats.
- Models: Five open-weight VLMs are evaluated: Qwen2.5-VL-7B, Qwen3-VL-8B, Gemma-4-E4B, InternVL3-8B, and InternVL3.5-8B.The models are recent VLMs with native support for multi-image inputs.
- Metrics: Both anomaly detection and localization are evaluated using accuracy.Detection uses balanced classes, while localization measures the fraction of sequences with the correct anomaly position.
- Evaluation protocol: A unified prompting protocol and constrained decoding restrict detection outputs to {yes,no} and localization outputs to valid frame indices.All models are evaluated in a zero-shot setting.
5 Results
VLMs perform strongly on frame-level anomalies but remain near chance on temporal anomalies, whereas humans substantially outperform them on temporal detection and localization.
- 57.4% is the highest temporal anomaly detection accuracy, while 43.2% is the highest temporal anomaly localization accuracy across evaluated models.
- Frame-level anomaly detection and localization reach 99.6% and 99.4%, respectively, demonstrating substantially stronger performance than on temporal anomalies.
- Human Study: Humans achieve 75.0–91.7% temporal anomaly detection accuracy and 83.3–88.9% localization accuracy across datasets.
- Human Study: No evaluated model exceeds 70% temporal detection accuracy or 50% temporal localization accuracy on any dataset.
- Human Study: In a representative MTL-AQA example, the human correctly localizes the swapped frames while all evaluated VLMs fail.
6 Analysis
Across analyses, temporal anomaly performance remains near chance despite changes in visual similarity, model scale, prompting, and sequence length, while localization errors show a systematic position bias.
- Visual Similarity: Larger LPIPS differences produce only modest gains, with temporal anomaly accuracy remaining far below human levels across all bins.
- Localization Errors: Qwen3-VL localization predictions show a strong bias toward position 2 regardless of the true anomaly location.
- Model Scale: Scaling Qwen3-VL from 2B to 32B improves temporal anomaly performance only marginally, leaving detection near chance and localization modestly above chance.
- Prompting: Alternative prompts have little effect, while reasoning-enabled models show mixed results without a consistent overall benefit.
- Sequence Length: Increasing sequence length does not improve temporal anomaly accuracy, whereas frame-level detection remains consistently high.
7 Implications for Temporal Grounding
TimeCatch isolates temporal consistency by changing frame order without changing visual content, exposing a limitation that persists across several tested conditions.
- Existing video benchmarks may overestimate temporal reasoning because models can identify objects, actions, or anomalous frames without modeling temporal relationships.
- TimeCatch provides a direct test of temporal consistency by modifying only frame order while leaving visual content unchanged.
- The limitation persists across visual similarity, prompting strategy, model scale, and sequence length, motivating TimeCatch as a complementary evaluation setting.
8 Conclusion
The paper introduces TimeCatch to evaluate temporal grounding and finds that VLMs detect frame-level anomalies reliably but struggle with temporal inconsistencies, unlike humans.
- TimeCatch is a controlled benchmark evaluating temporal grounding through temporal and frame-level anomaly detection.
- VLMs reliably detect and localize frame-level anomalies but struggle to recognize temporal inconsistencies, whereas humans achieve near-ceiling performance on both temporal tasks.
- The authors propose TimeCatch as a complementary benchmark for tracking progress in temporal reasoning.
Limitations
TimeCatch isolates temporal grounding through adjacent frame swaps, but does not represent the full range of temporal reasoning required for real-world video understanding.
- TimeCatch evaluates temporal consistency specifically through violations introduced by swapping adjacent frames.
- The benchmark does not capture long-range dependencies, causal event reasoning, or continuity of motion.
- The authors position TimeCatch as a complementary diagnostic rather than a comprehensive evaluation of temporal reasoning.
Ethical Considerations
The study uses public research datasets, recruits human annotators through Prolific, and collects only task annotations while providing structured instructions and interfaces.
- The study uses publicly available datasets and follows their respective licenses and terms of use.
- Participants completed image-sequence detection or localization tasks using scene descriptions, onboarding materials, and an interactive viewer.
- The study collected no personal or location data beyond the annotations.
- Human participants were recruited and compensated through Prolific, with 24 participants spanning four datasets and task configurations.
- Attention checks were included, and submissions failing two or three checks were rejected before human-study results were computed.
D Additional Results
Additional analyses examine targeted fine-tuning, scene-description effects, perceptual similarity, localization biases, and qualitative success and failure modes. Fine-tuning improves temporal anomaly detection and suggests supervision may matter beyond model scale, while other analyses provide limited evidence that visual differences or scene descriptions explain temporal-performance limitations.
- Targeted fine-tuning: Fine-tuning Qwen3-VL-2B on CLEVRER improves temporal anomaly detection on held-out CLEVRER and transfers to unseen CRAFT, DriveLM, and MTL-AQA datasets.The experiment uses 80% of CLEVRER for training and 20% for validation, evaluating the remaining datasets without further adaptation.
- Targeted fine-tuning: The fine-tuned 2B model matches or exceeds zero-shot Qwen3-VL-8B performance, suggesting supervision contributes to the observed limitation beyond model scale alone.The authors describe these experiments as preliminary and limited to a single model family.
- Prompt and perceptual analyses: Removing scene descriptions affects frame-level localization more than temporal anomaly detection, whose performance changes only modestly.Table 5 reports positive values as improvement without descriptions and negative values as degradation.
- Prompt and perceptual analyses: Temporal anomaly accuracy remains close to chance across most LPIPS-distance bins, so larger visual differences between swapped frames do not consistently improve detection.The apparent increase in the 0.8–0.9 bin is likely driven by its small sample size (n = 13).
- Localization analyses: Localization confusion matrices reveal model-specific swap-position biases alongside more uniformly distributed predictions consistent with near-random guessing.Qwen3-VL and InternVL3.5 show position biases, whereas Gemma-4, Qwen2.5-VL, and InternVL3 produce more uniform predictions.
- Qualitative analysis: Qualitative examples include cases where humans and models both succeed, sequences without swaps are correctly identified, and cases where models are correct while humans are wrong.The examples span CRAFT, MTL-AQA, and Drive-LM detection settings.