PaperlayerView exact revision on arXiv

Source-linked AI summary

TempCloze: Can Video-LLMs Identify the Missing Middle?

Wenqi Pei, Henry Hengyuan Zhao, Yilai Liu, Jiahao Meng, Han Chen, Ziyu Wang, Hongyang Du

arXiv:2609.01515v1cs.CVcs.AI

TL;DR

Existing Video-LLM temporal benchmarks can be influenced by linguistic shortcuts, motivating TempCloze’s direct visual missing-middle task. The benchmark evaluates same-source semantic, alignment, and progression distinctions across 1,521 videos, and finds temporal Alignment to be the primary bottleneck while revealing sensitivity to evaluation conditions.

  • Problem

    Language-mediated temporal benchmarks may allow models to exploit option wording, answer correlations, or language priors instead of visual temporal evidence.

  • Method

    TempCloze asks Video-LLMs to select the true missing middle from beginning and ending clips using same-source distractors targeting Semantic, Alignment, and Progression.

  • Results

    Alignment is the primary bottleneck across evaluations of 10 proprietary and 21 open-source Video-LLMs, while models often recognize plausible event content and progression.

  • Takeaways & Limitations

    Error and sensitivity analyses show that current Video-LLMs’ temporal reasoning is not fully stable across candidate order, context direction, visible span, frame density, and test-time scaling.

  • Takeaways & Limitations

    TempCloze is a controlled missing-middle diagnostic with fixed candidates, not a complete measure of open-ended, dialogue-based, audio-grounded, or unconstrained future reasoning.

Abstract

from arXiv · show

Temporal reasoning benchmarks for Video-LLMs are often mediated by language, leaving room for linguistic shortcuts from option wording, answer correlations, or language priors. To reduce such shortcuts, we introduce TempCloze, a video cloze benchmark for evaluating visual temporal reasoning in Video-LLMs. Given the beginning and ending clips of a video, models must identify the true missing middle from four candidates. TempCloze contains 1,521 carefully filtered videos from seven sources, mainly long-take and egocentric videos. We construct same-source distractors along three dimensions: Semantic asks what event should happen, Alignment probes when it should occur, and Progression tests how it should unfold, while shared scenes and objects reduce appearance cues. Our evaluation of 10 proprietary and 21 open-source Video-LLMs reveals Alignment as the primary bottleneck: models often recognize plausible semantic content and local event progression but struggle with temporal alignment. We further conduct error pattern and behavioral sensitivity analyses on TempCloze-Mixed and TempCloze-Hard with four representative models to examine where errors arise and how candidate order, context direction, visible span, frame density, and test-time scaling influence model choices.

1 Introduction

TempCloze introduces a video cloze benchmark that evaluates visual temporal reasoning by asking Video-LLMs to identify a missing middle clip while reducing linguistic and appearance shortcuts. Across broad model evaluations, temporal Alignment emerges as the central weakness, with additional analyses exposing sensitivity to error types and presentation factors.

  • Benchmark motivation and design: Language-mediated benchmarks can permit shortcuts from option wording, answer correlations, and language priors rather than visual evidence.The paper motivates comparing video segments directly instead of relying on textual descriptions.
  • Benchmark motivation and design: TempCloze evaluates visual temporal reasoning by requiring models to identify the true missing middle between beginning and ending clips.The benchmark contains 1,521 videos and asks models to choose among candidate clips.
  • Benchmark motivation and design: Same-source distractors share scenes and objects while testing Semantic, Alignment, and Progression: what happens, when it occurs, and how it unfolds.Alignment distractors shift or expand the ground-truth span, while Progression distractors reverse, reorder, or repeat the segment.
  • Evaluation and findings: 1,521 videos from seven sources form the filtered benchmark evaluated with 10 proprietary and 21 open-source Video-LLMs.The sources emphasize long-take, egocentric, and fine-grained motion videos.
  • Evaluation and findings: Alignment is the main bottleneck: models recognize plausible event content and progression but struggle substantially with temporal placement.Error analyses identify Alignment as the most misleading alternative.
  • Evaluation and findings: Behavioral analyses find unstable choices under candidate reordering, stronger reliance on beginning context, endpoint dependence, frame-density dilution, and model-dependent test-time-scaling gains.Test-time scaling does not change the relative ordering among dimensions.

2 Related Work

Prior video benchmarks evaluate temporal reasoning within broader VideoQA and long-context settings, while earlier cloze work primarily targets textual completion or representation learning. TempCloze instead uses cloze formatting to evaluate temporal reasoning directly in Video-LLMs.

  • Temporal reasoning benchmarks: General video understanding benchmarks incorporate temporal reasoning alongside questions about actions and spatio-temporal relations.Examples include TGIF-QA, NExT-QA, ActivityNet-QA, LongVideoBench, MVBench, and VideoMME.
  • Cloze for video understanding: Cloze tasks remove part of a context and ask models to infer the missing content.This format appears in both language-oriented video understanding and video representation learning.
  • Cloze for video understanding: Earlier video cloze-related methods use textual completion or withheld-content prediction as training objectives.VideoBERT, VCP, MaskFeat, and VideoMAE apply cloze-like prediction to learn video representations.
  • Cloze for video understanding: TempCloze repurposes video cloze formatting for evaluating temporal reasoning in Video-LLMs rather than training representations.This distinguishes its evaluation objective from the cited prior cloze methods.

3 TempCloze

TempCloze frames temporal reasoning as selecting the missing middle clip between a video's beginning and ending, using distractors that separately test event content, timing, and unfolding. Built from seven public sources and filtered to 1,521 videos, the benchmark reduces appearance cues through same-source clips and continuous-video sampling.

  • 3.1 Overview: A video is decomposed into beginning B, middle M, and ending E, with models selecting M from candidates given (B,E).The task represents the video as V = [B | M | E] and asks the model to fill the missing interval.
  • 3.3.2 Distractor Generation: TempCloze distractors target Semantic, Alignment, and Progression, testing what event occurs, when it occurs, and how it unfolds.The dimensions isolate content, temporal compatibility, and fine-grained event development.
  • 3.2 Data Sources: The benchmark draws on seven public sources, including long-take and egocentric datasets that provide temporally continuous videos.MiraData and LVD-2M are long-take sources, while EgoLife and FAVOR-Bench are egocentric sources.
  • 3.3.1 Video Filtering: 1,521 videos remain after duration, language-model, quality, sharpness, and optical-flow filtering.Gaps are sampled from the central 50% of videos and occupy 20%–40% of full duration, preserving context on both sides.
  • 3.3.2 Distractor Generation: Semantic distractors use non-overlapping same-duration clips, Alignment distractors shift or expand the target interval, and Progression distractors reverse, reorder, or repeat subevents.These constructions preserve relevant visual content while varying the temporal property under evaluation.

4 Experiments

Experiments show that Alignment is the central difficulty for current Video-LLMs, while error analyses reveal structured temporal failure modes and sensitivity to evaluation conditions.

  • Experimental Setup: 10 proprietary and 21 open-source Video-LLMs are evaluated, with four representative models used for detailed behavioral analyses.The auxiliary subsets are TempCloze-Mixed with 300 videos and TempCloze-Hard with 150 high-error instances.
  • Main Results: Semantic accuracy averages 70.73% for proprietary models and 34.00% for open-source models, while Progression averages 67.72% and 36.97%, respectively.Qwen3.5-397B reaches 75.94% on Semantic and 78.24% on Progression.
  • Main Results: Alignment accuracy averages 48.13% for proprietary models and 26.54% for open-source models, making it the primary bottleneck.The strongest scores are 76.92% for Seed1.8 and 51.55% for Qwen3.5-35B, below the 98.00% Human Baseline.
  • Dimension-Specific Errors: Alignment errors most often select Expanded clips, whereas Progression errors most often select Reversed clips.Seed1.8-I selects Expanded in 75% of its Alignment errors, and GPT-5.4 selects Reversed in 67% of its Progression errors.
  • Mixed-Dimension Errors: In mixed-dimension evaluation, Alignment receives the highest error share, and competition across dimensions is harder than evaluating a single dimension.Semantic can be more distracting than Progression despite generally slightly higher Semantic accuracy.
  • Behavioral Sensitivity: Candidate reordering changes selected clips substantially despite little accuracy variation, with CFR ranging from 32.4% to 60.7% and FR from 25.9% to 40.2%.Alignment and Progression are generally less stable than Semantic when candidates are visually similar but temporally distinct.
  • Behavioral Sensitivity: Models often rely more on beginning-only context than ending-only context, while longer visible spans and denser frame sampling can dilute Alignment evidence.Seed-1.6 reaches 80% on Progression with beginning-only context versus 46% with ending-only context; Alignment declines as span and sampling density increase.
  • Behavioral Sensitivity: Test-time scaling improves all models by varying amounts but does not change the relative ordering of dimensions, with Alignment remaining the bottleneck.Gemini2.5-Pro and Qwen3.5-397B-A17B gain roughly 30 Overall pass@k points from k = 1 to k = 5.

5 Conclusion

TempCloze tests visual temporal reasoning by selecting a missing middle from visually similar, same-source candidates across Semantic, Alignment, and Progression dimensions. Results show temporal Alignment remains the central bottleneck, motivating video-native mechanisms that represent temporal structure explicitly.

  • Conclusion: TempCloze evaluates missing-middle identification from surrounding clips rather than natural-language descriptions, using same-source candidates across three temporal dimensions.Semantic targets the event, Alignment its timing, and Progression its development; shared scenes and objects limit appearance shortcuts.
  • Conclusion: Many Video-LLMs recognize semantic content and event progression but struggle to place the missing middle precisely between the endpoints.Alignment is the hardest dimension, and this ordering remains largely unchanged under test-time scaling.
  • Conclusion: Models remain sensitive to presentation order and context direction, indicating that current gains do not reflect stable reasoning.The reported sensitivity analyses also examine visible span, frame density, and test-time scaling.
  • Conclusion: Future Video-LLMs need video-native mechanisms that integrate both temporal directions, select key evidence, and represent temporal structure explicitly.Future benchmarks should also target longer videos while controlling linguistic shortcuts.

Limitations

TempCloze is a controlled diagnostic of visually grounded temporal fit, not a complete measure of video understanding. Its conclusions are bounded by the fixed candidate format, selected video domains, and the changing behavior of evaluated Video-LLMs.

  • Scope: TempCloze does not directly cover open-ended generation, dialogue-based narrative understanding, audio-grounded inference, or unconstrained future prediction.The benchmark focuses on visually grounded temporal fit between observed beginning and ending clips.
  • Scope: Performance should be interpreted relative to the designed alternatives, not as a general measure of free-form video reasoning.The fixed candidate format constrains what the benchmark measures.
  • Design boundary: Because candidates come from the same source video, the benchmark measures exact temporal compatibility rather than only broad event plausibility.Alignment distractors may contain relevant content while still failing to match the removed interval.
  • Scope: Results are a snapshot of current Video-LLMs because APIs, decoding behavior, and video input limits can change over time.The source mix emphasizes long-take, egocentric, and fine-grained motion videos and cannot cover every domain or camera style.

LLM Usage Statement

The paper formalizes controlled distractors and sampling procedures for TempCloze, then reports checks and baselines used to characterize candidate selection. These passages also include an explicit statement about the role of LLMs in the experiments.

  • LLM Usage Statement: The authors state that LLMs served as experimental subjects but did not generate the paper’s core ideas, methods, designs, or conclusions.They attribute the technical contributions and analyses to the authors.
  • Distractor Formalization: The ground-truth interval lies in the central 50% of the video and lasts 20%–40% of its duration.This preserves context on both sides while defining the target span.
  • Distractor Formalization: Advanced and Deferred shift the target by half its duration, while Expanded extends it by half a duration on both sides.The perturbation amount is δ = ℓ/2 ∈[0.1T, 0.2T].
  • Distractor Formalization: Semantic distractors use same-duration clips outside the missing interval, whereas Alignment distractors stay near the target while perturbing its boundaries.The sampling procedure retains only valid intervals inside the source video.
  • Distractor Formalization: Progression distractors alter the target span’s internal order through partitioning, permutation, or repetition.These operations test how the event develops rather than only which event appears.
  • Diagnostics: 18,252 candidate clips produced zero exact matches between candidate edge frames and adjacent context boundaries.The check covers 1,521 videos per dimension.
  • Diagnostics: The edge baseline assigns only slightly above-uniform ground-truth probabilities, so boundary appearance is not substantially decisive overall.Mean probabilities average Semantic, Alignment, and Progression and are not top-1 accuracy.
  • Diagnostics: The single-frame baseline is compared with the original multi-frame evaluation for Seed1.8 and Gemini2.5-Pro.Table 8 reports all values as percentages.

C Human Baseline

A strict human baseline shows that people solve TempCloze with high accuracy across all three dimensions. The estimate requires both independent annotators to select the ground-truth clip for a question to count as correct.

  • Protocol: Five annotators completed the task after a short briefing using the same beginning, ending, and four-candidate inputs as model evaluation.The estimate used 100 randomly selected videos and two independent judgments per question.
  • Human Baseline: Humans achieve 96/100 Semantic, 98/100 Alignment, and 97/100 Progression questions under the strict agreement rule.Every disagreement is counted as incorrect.
  • Human Baseline: The strict human baseline reaches 291/300 = 97.0% mean dimension accuracy.The mean aggregates the three dimension-level results.
  • Human Baseline: At the video level, humans solve at least one dimension for 100/100 videos, at least two for 99/100, and all three for 92/100.These counts summarize performance across the sampled videos.

D.1 Dataset Overview

The dataset combines varied long-take, everyday, and egocentric video sources with staged filtering and central-gap generation. Human validation indicates that the retained videos are generally suitable for temporal cloze evaluation.

  • Filtering: Duration filtering removes the most videos, while LLM suitability and motion filtering are the most selective later stages.For sources without captions, fine-grained captions are generated before suitability screening.
  • Gap generation: Retained videos undergo central missing-span sampling, Farnebäck optical-flow validation, and up to three resampling attempts.Reported stage percentages are conditional on the preceding filter, whereas the total column is relative to the original source input.
  • Human validation: 300 videos receive human suitability ratings from five annotators using a five-point scale.Each annotator reviews 60 videos, and the review interface is implemented in Streamlit.
  • Human validation: The overall mean rating is 4.13/5, with 78.0% of ratings positive and 94.7% at least acceptable.Unusable ratings occur 0 times, indicating that clearly invalid cases were removed by the filtering pipeline.

E Statistics by Sources

Source-level results show strong Semantic performance but weaker Alignment, with proprietary models substantially outperforming open-source models on complete three-dimension solutions. Errors are often partial, especially for proprietary systems.

  • Error structure: High ≥1 accuracy alongside much lower 3/3 accuracy indicates that failures are often partial.Qwen3.5-35B-A3B varies by source and is strongest on Daily-Omni and Video-TT.
  • Joint dimensions: Alignment-involving pairs are weaker than Semantic-plus-Progression pairs for both proprietary and open-source models.Proprietary models score 40.54% on S+A and 37.39% on A+P versus 53.67% on S+P; open-source models show the same ordering at lower levels.
  • Model leaders: Seed 1.8 leads proprietary joint accuracy, while Qwen3.5-35B-A3B and Qwen3.5-397B-A17B lead among open-source models.The strongest proprietary models reach 67.22%–70.81% in 3/3 accuracy, compared with 35.77% for Qwen3.5-397B-A17B.
  • Model groups: The proprietary–open-source gap is 1.9× in mean dimension accuracy and 4.6× in 3/3 accuracy.The reported values are 62.19% versus 32.51% for mean dimension accuracy and 33.24% versus 7.15% for 3/3 accuracy.

F.3 Effect of Thinking

The thinking analysis compares matched reasoning variants and finds that extra reasoning helps when it preserves visual grounding and improves temporal timing. Its benefits are therefore model-dependent rather than uniform.

  • Evaluation setup: Thinking variants are compared against matched nonthinking or instruct baselines using percentage-point changes and accuracy–token tradeoffs.The analysis mixes proprietary reasoning modes with open-source thinking checkpoints, using matched pairs where possible.
  • Proprietary models: Seed 1.8 gains 5.15 mean-accuracy points and 13.87 full-joint points, mainly through a 14.99-point Alignment gain.Semantic changes little, indicating that the main improvement is associated with temporal alignment.
  • Open-source models: Qwen3VL thinking checkpoints improve Alignment at matched sizes but can lose more on Semantic and Progression.Qwen3VL-32B-T gains 14.47 Alignment points while dropping 26.30 Semantic points and 37.53 Progression points relative to Qwen3VL-32B-I.
  • Open-source models: KimiVL-A3B improves mean accuracy by 3.24 points through Semantic gains but slightly lowers full-joint accuracy.This contrasts with cases where improved individual dimensions do not translate into solving all three dimensions together.
  • Conclusion: Overall, thinking is not uniformly beneficial; it helps when extra reasoning improves temporal timing without losing visual or progression constraints.The benchmark’s cloze setup requires choosing the middle segment that connects ordered beginning and ending clips through visible temporal continuity.
Loading 2609.01515v1…