Source-linked AI summary
Improving Spatial-Temporal Reasoning in Video-Language Models with Structured Video Prompting
Sadegh Mohammadian
TL;DR
Video-language models struggle to track events over time and ground answers in precise spatial regions, motivating better organization of visual evidence. The paper introduces structured video prompting, a training-free input transformation that adds spatial and temporal anchors without changing model parameters or decoding. Across two benchmarks and two open models, structured inputs improve performance in several settings, with gains varying by model and task.
Problem
VLMs remain brittle when questions require temporal event tracking, precise spatial grounding, or connecting evidence across multiple moments.
Method
Structured video prompting deterministically augments video frames with explicit spatial and temporal structure while preserving the question and model inference procedure.
Results
Structured video prompting improves performance across both evaluated benchmarks and models, with gains varying by model and benchmark.
Takeaways & Limitations
The findings suggest that how visual evidence is organized at inference time contributes to some VLM reasoning failures and offers a practical improvement direction.
Takeaways & Limitations
The comparison keeps decoding parameters and video sampling settings fixed across baseline and structured conditions.
Abstract
from arXiv · showhide
Video-language models (VLMs) remain brittle on tasks that require tracking events over time and grounding answers in specific spatial regions. We propose that part of this limitation can be addressed through better organization of visual evidence at inference time. We introduce structured video prompting, a training-free inference-time method that augments the input video with lightweight spatial structure and temporal structure, providing explicit anchors for organizing evidence across space and time without changing model weights or decoding and without altering the question prompt in the main comparison. We evaluate this approach on two complementary video benchmarks and two open video-language models. Across these settings, structured inputs improve performance in several cases, with gains varying by model and task. Our findings suggest that some failures of VLMs arise not only from reasoning capacity, but also from how video evidence is presented at inference time. These results highlight structured video prompting as a simple and practical direction for improving video understanding.
1 Introduction
Video-language models can fail even when relevant evidence is present because spatial and temporal evidence is weakly organized. The paper studies structured video prompting as a training-free input transformation and finds performance improvements in several evaluated settings.
- VLMs struggle with event order, precise spatial grounding, and connecting evidence across multiple video moments.
- Implicit temporal positions and unlabeled spatial regions make it harder to locate relevant evidence and track events across frames.
- Structured video prompting adds explicit spatial and temporal anchors without changing the model, weights, or inference decoding.
- The evaluation compares baseline and structured versions of the same videos on complementary temporal and spatial reasoning benchmarks.
- Structured videos improve performance in several cases, with gains varying by model and benchmark.
2 Related Work
Prior work has improved video reasoning mainly through model-centric changes, while structured-input research has focused largely on images. This paper extends lightweight structured visual inputs to video-language reasoning without changing the model.
- Recent video-language models and benchmarks have advanced multimodal understanding while reliable video reasoning remains unresolved.
- Image-based work frames some vision-language failures as binding problems involving object, attribute, and relation associations.
- Most video-reasoning approaches intervene in architecture, training, attention, decoding, or internal visual-temporal representations.
- This work extends structured-input methods from images to video by adding lightweight spatial and temporal anchors directly to frames.
3 Method
The method transforms each video frame with explicit spatial and temporal scaffolding, then compares baseline and structured inputs under otherwise fixed inference settings. The transformation uses a 4 × 4 labeled grid and visible frame indices.
- Structured video prompting is a training-free preprocessing method that adds lightweight spatial and temporal cues to the input video.
- Structured Video Transformation: The transformation is deterministic and applied frame-wise using the current frame index and total video length.
- Structured Video Transformation: A light 4 × 4 grid divides each frame into 16 row-major cells with unique identifiers for localized spatial reference.
- Structured Video Transformation: A temporal band displays each frame’s position, such as Frame t/T, making temporal location visible within the video.
- Structured Video Transformation: The structured frame preserves original visual content while placing small labels, thin grid lines, and the temporal band outside the original frame region.
- Inference Setup: The baseline receives (V, q), whereas the structured condition receives (Ṽ, q) with the same question and fixed inference settings.
4 Experiments
The experiments compare structured and baseline video inputs across two benchmarks and two open VLMs, with fixed evaluation settings. Structured prompting improves performance overall, while component effects vary by task and model.
- Experimental Setup: The evaluation covers TemporalBench and MMSI-Video-Bench using Qwen2.5-VL-7B and InternVL3.5-8B under controlled baseline and structured-input conditions.The main comparison uses fixed sampling and decoding settings; structured inputs add spatial and temporal cues without changing the question prompt.
- Main Results: Across both benchmarks and both models, structured video prompting improves performance over the baseline.The reported gains vary across benchmarks and models rather than following a uniform magnitude.
- Main Results: On TemporalBench, Qwen2.5-VL-7B improves from 60.75% to 63.00%, a gain of 2.25 percentage points, while InternVL3.5-8B improves from 63.25% to 66.00%, a gain of 2.75 points.
- Main Results: On MMSI-Video-Bench, Qwen2.5-VL-7B improves from 28.25% to 33.25%, a gain of 5.00 points, while InternVL3.5-8B improves from 31.75% to 34.00%, a gain of 2.25 points.
- Category-Level Results: The largest category-level gains occur in Cross-Video Reasoning for Qwen2.5-VL-7B and Prediction for InternVL3.5-8B, while other categories show smaller or inconsistent changes.
- Component Ablation: The complete structured method combines a numbered spatial grid, visual frame identifiers, and temporal instruction components in the ablation.The five conditions include the base video, grid only, visual frame IDs only, text-only frame-index instruction, and the complete structured input.
- Component Ablation: The complete structured method improves overall accuracy from 28.25% to 33.25% and Cross-Video Reasoning from 28.17% to 36.62%, exceeding either component alone.The spatial grid primarily improves Spatial Construction, while visual frame identifiers provide the strongest isolated result on Motion Understanding.
- Component Ablation: Performance on Planning and Prediction remains unchanged across all ablation conditions, indicating that benefits are concentrated in evidence localization and cross-frame organization.
5 Conclusion
The paper introduces structured video prompting as a training-free input transformation that adds explicit spatial and temporal structure to video frames. Across two benchmarks and two open models, it improves performance in several settings and motivates further study of adaptive structures, longer videos, and larger models.
- Structured video prompting adds explicit spatial and temporal structure directly to video inputs without training.
- Across TemporalBench and MMSI-Video-Bench, the lightweight transformation improves performance in several settings for Qwen2.5-VL-7B and InternVL3.5-8B.
- The findings suggest that some video-language-model failures reflect how visual evidence is organized at inference time, not only model limitations.
- Future work can examine why the intervention works and whether it scales to adaptive structures, longer videos, and larger models.
Qualitative Analysis
A qualitative MMSI-Video-Bench example contrasts raw and structured video prompting for navigation. The structured input changes the model's answer from an incorrect left turn to the correct right turn by making spatial and temporal evidence more explicit.
- The example uses a gym walk-through in which the model must infer navigation from the final frame.
- The raw video leads Qwen2.5-VL-7B to select incorrect option A, whereas structured prompting leads it to select correct option B.
- Explicit spatial and temporal anchors help organize sampled-frame evidence and support the correct right-turn inference.