Source-linked AI summary
VideoLoom: A Video Large Language Model for Joint Spatial-Temporal Understanding
Jiapeng Shi, Junke Wang, Zuyao You, Bo He, Zuxuan Wu
TL;DR
Fine-grained video understanding remains fragmented between spatial and temporal capabilities, with limited datasets providing consistent annotations for both. VideoLoom addresses this gap with LoomData-8.7k, a unified SlowFast-token Video LLM, and LoomBench, achieving strong or state-of-the-art results across spatial and temporal benchmarks. The suite is designed for joint spatial-temporal video understanding and comprehensive evaluation.
Problem
Existing Video LLMs and datasets often address spatial and temporal understanding separately, while consistent fine-grained spatial-temporal annotations remain scarce.
Method
The paper introduces LoomData-8.7k, VideoLoom with fast and slow visual tokens, and LoomBench for jointly modeling and evaluating spatial-temporal understanding.
Results
VideoLoom achieves state-of-the-art or competitive performance across spatial and temporal benchmarks, including 63.1 J &F on ReVOS and 48.3 R1@0.7 on Charades-STA.
Takeaways & Limitations
The VideoLoom suite supports unified spatial-temporal understanding and evaluates temporal, spatial, and combined question types across diverse video tasks.
Takeaways & Limitations
The annotation pipeline still contains multiple interdependent stages, motivating further automation to improve efficiency and reliability.
Abstract
from arXiv · showhide
This paper presents VideoLoom, a unified Video Large Language Model (Video LLM) for joint spatial-temporal understanding. To facilitate the development of fine-grained spatial and temporal localization capabilities, we curate LoomData-8.7k, a human-centric video dataset with temporally grounded and spatially localized captions. With this, VideoLoom achieves state-of-the-art or highly competitive performance across a variety of spatial and temporal benchmarks (e.g., 63.1 J&F on ReVOS for referring video object segmentation, and 48.3 R1@0.7 on Charades-STA for temporal grounding). In addition, we introduce LoomBench, a novel benchmark consisting of temporal, spatial, and compositional video-question pairs, enabling a comprehensive evaluation of Video LLMs from diverse aspects. Collectively, these contributions offer a universal and effective suite for joint spatial-temporal video understanding, setting a new standard in multimodal intelligence.
1 Introduction
VideoLoom addresses the lack of unified fine-grained spatial-temporal understanding by combining a consistently annotated dataset, a SlowFast-token Video LLM, and a joint evaluation benchmark. It achieves strong performance across spatial and temporal video benchmarks.
- Existing Video LLMs commonly target either temporal grounding or spatial perception, leaving joint fine-grained understanding insufficiently addressed.
- Joint modeling is difficult because spatial tasks require high-resolution inputs, whereas temporal tasks require denser frame sampling under fixed computational budgets.
- LoomData-8.7k provides consistent spatial and temporal annotations through a character-centric, shot-guided pipeline that tracks characters and generates action descriptions.
- VideoLoom combines uniformly sampled multi-frame inputs with high-resolution keyframes using fast and slow visual tokens to balance temporal coverage and spatial precision.
- LoomBench evaluates joint understanding with questions requiring temporal grounding and spatial segmentation simultaneously.
- 63.1 J &F on ReVOS and 48.3 R1@0.7 on Charades-STA are among VideoLoom's reported results across spatial and temporal benchmarks.
2 Related Work
Prior video understanding work largely separates temporal localization from spatial localization, while VideoLoom targets both fine-grained capabilities in one framework. Its annotation pipeline operationalizes this goal through staged shot, mask, tracking, and temporal-action processing.
- Video datasets are generally divided into temporal-focused resources with timestamp-aligned descriptions and spatial-focused resources with masks or trajectories.
- Video LLMs have progressed from coarse tasks such as captioning and retrieval toward fine-grained modeling of object interactions and temporal dynamics.
- The annotation pipeline consists of shot partition, spatial mask annotation, shot merging, and temporal action annotation.
- Temporal-focused models develop timestamp-based localization, whereas spatial-focused models ground visual regions as trajectories.
- VideoLoom unifies fine-grained temporal understanding and spatial perception within a single Video LLM rather than restricting them to separate or task-specific systems.
3 Method
VideoLoom advances joint spatial-temporal video understanding through LoomData-8.7k, a unified Video LLM architecture, and LoomBench. Its method combines automatic character-centric annotation, SlowFast visual tokens, and MLLM-SAM2 integration to support temporal grounding and spatial segmentation.
- LoomData-8.7k: The annotation pipeline partitions shots, annotates character masks, merges temporally continuous shots, and generates timestamp-aligned actions.It uses multiple visual foundation models to associate main-character actions with spatial locations.
- LoomData-8.7k: LoomData-8.7k provides 8,710 shots with timestamp-aligned action descriptions and dense spatial masks.Each video averages 102.2 seconds, 6.0 shots, and 41.3-word temporal descriptions.
- SlowFast Visual Tokens: VideoLoom combines dense low-resolution frames with sparse high-resolution keyframes through fast and slow visual tokens.Fast tokens provide temporal coverage, while slow tokens preserve spatial detail.
- MLLM-SAM2 Architecture: InternVL3 processes SlowFast tokens and text to produce responses, timestamps, and a [SEG] embedding, while SAM2 generates and propagates masklets.The [SEG] token connects the language model to SAM2’s mask decoder.
- LoomBench: LoomBench evaluates temporal grounding, spatial perception, and combined tracklet localization with When, Where, and Combined questions.When targets timestamps, Where targets person masklets within segments, and Combined targets the corresponding tracklet segment.
4 Experiments
Experiments evaluate VideoLoom across temporal, spatial, image, and joint spatial-temporal benchmarks, alongside ablations and qualitative visualizations. Results show competitive performance, benefits from SlowFast tokens and LoomData-8.7k, and accurate temporal localization and spatial segmentation.
- Temporal benchmarks: VideoLoom is evaluated on temporal video grounding, dense video captioning, and video highlight detection benchmarks.The temporal evaluation is intended to comprehensively assess temporal understanding.
- Temporal benchmarks: 48.3 R1@0.7 on Charades-STA and 63.3 HIT@1 on QVHighlights demonstrate state-of-the-art or competitive temporal performance.VideoLoom surpasses unified TimeSuite and task-specific HawkEye on the reported comparisons, while lagging behind UniTime on Charades-STA.
- Spatial benchmarks: 51.7 on MeVIS, 71.3 on RefYTVOS, and 63.1 on ReVOS in J &F show strong referring video object segmentation performance.VideoLoom outperforms tracking-oriented Video LLMs across these spatial benchmarks.
- Image benchmarks: VideoLoom achieves the best results on RefCOCO, RefCOCO+, RefCOCOg, and Grand-f image segmentation benchmarks.The reported metrics are cIoU, AP50, and mIoU.
- Ablation studies: SlowFast tokens improve unified performance by 4.8 mIoU on QVHighlights and 3.2 J &F on MeVIS over standalone spatial or temporal models.Using only slow or fast tokens causes substantial degradation on temporal or spatial tasks, respectively; the joint Slow/Fast setting is more balanced but still below specialized models.
- Ablation studies: LoomData-8.7k improves joint spatial-temporal understanding by +5.0 J &Fbi-fore and yields consistent gains across spatial, temporal, and general visual benchmarks.The ablation compares VideoLoom trained with LoomData-8.7k against training on existing spatial and temporal datasets, while controlling additional VQA data.
5 Conclusion
The VideoLoom suite combines a dataset, unified Video LLM, and benchmark for joint spatial-temporal understanding. Its design includes interleaved frame identifiers, fast and slow visual tokens, and SAM2-based mask generation.
- Conclusion: VideoLoom comprises LoomData-8.7k, a unified Video LLM with MLLM-SAM2, and LoomBench for comprehensive spatial-temporal evaluation.The suite covers timestamp-aligned action descriptions, fine-grained spatial masks, temporal locations, and spatial masks.
- Conclusion: LoomBench evaluates Video LLMs across When, Where, and Combined question types for spatial-temporal understanding.
- Limitations and future work: The annotation pipeline still contains multiple stages with interdependent components, motivating future automation with stronger multimodal models and agents.The stated goal is to improve annotation efficiency and reliability.
- Model design: VideoLoom uses interleaved frame IDs with fast visual tokens and concatenates this sequence with slow tokens as LLM input.The frame IDs represent temporal positions, while the slow-token pathway supports mask generation through SAM2.
- Model design: The SAM2 mask decoder generates keyframe masks from visual features and an embedding provided through the MLLM's [SEG] token.
B.2 Additional Implemental Details
Implementation settings vary the number of fast-token frames across datasets according to video duration and annotation structure.
- Frame sampling: Fast-token frame counts are selected by dataset, with a maximum of 128 frames.The settings account for video duration and the temporal structure of annotations.
- Frame sampling: Charades-STA uses 64 sampled frames because its videos typically last around 30 seconds.
- Frame sampling: YouCook2 uses 128 uniformly sampled frames because its videos often exceed 2 minutes.
- Frame sampling: QVHighlights samples at 2 FPS, typically producing around 75 frames for its 2-second annotation intervals.
B.3 Training Data
The training-data pipeline combines multiple datasets with a manually verified annotation process designed to reduce manual involvement. Verification filters missing or incorrect annotations and removes redundant labels.
- Training data: VideoLoom is jointly trained for 1,315K iterations on the listed training datasets.The passage reports advanced performance across all these tasks.
- Manual verification: Manual verification uses two rounds of simple judgments to minimize manual involvement.The process covers filtering and correction after spatial mask annotation.
- Manual verification: The first round discards videos with missing annotations because completing missing tracklets requires extensive, non-scalable manual annotation.
- Manual verification: The second round filters incorrectly annotated videos and removes redundant annotations from retained shots.Incorrect labels include masks assigned to other objects when the main character is absent.
C.2 Statistics for LoomData-8.7k
LoomData-8.7k jointly annotates temporal timestamps and spatial masks on nearly 2-minute videos, with fine-grained temporal partitioning and broadly distributed shots.
- Dataset statistics: LoomData-8.7k provides joint temporal-timestamp and spatial-mask annotations for nearly 2-minute videos.
- Dataset statistics: Each LoomData-8.7k video contains an average of 6.0 segments with tracklets.This is reported as comparable to current spatial-temporal datasets.
- Dataset statistics: LoomData-8.7k enables fine-grained temporal partitioning rather than only rough temporal-overlap labels.
- Dataset statistics: Shot lengths vary widely, while normalized center timestamps are distributed almost evenly across videos.
D.1 Analysis on Bidirectional Foreground J&F
The analysis motivates Bidirectional Foreground J&F for evaluating joint spatial-temporal localization on LoomBench, where background segments can distort standard J&F. The proposed metric remains stable across queried-segment lengths and VideoLoom outperforms the TimeSuite and Sa2VA baseline across reported components.
- Metric motivation: Bidirectional Foreground J&F evaluates joint spatial-temporal understanding for Combined questions on LoomBench.It is introduced alongside experiments varying queried-segment lengths and component-level analysis.
- Metric motivation: Standard J&F rises sharply for shorter queried segments, whereas Bidirectional Foreground J&F remains stable across segment lengths.The comparison covers 0-20%, 20-60%, and 60-100% queried-segment proportions.
- Metric motivation: Background segments can receive J&F=1 when the predicted mask is None, inflating whole-video scores and making them sensitive to foreground-query proportion.This effect prevents accurate assessment of spatial-temporal capabilities on LoomBench.
- Metric motivation: Referring VOS commonly uses standard J&F because existing videos are often foreground throughout, unlike LoomBench.Bidirectional Foreground J&F avoids extensive background computation for LoomBench evaluation.
- Results: VideoLoom outperforms the TimeSuite and Sa2VA baseline across all reported Bidirectional Foreground J&F component metrics.The analysis reports components computed over both predicted masklets and ground-truth regions.
D.2 Ablation on Non-Human Categories
The ablation examines VideoLoom's segmentation across human and non-human RefDavis17 categories in a zero-shot setting. Adding LoomData improves both categories by 2.3 J&F, despite the dataset primarily targeting humans.
- Experimental setup: The ablation evaluates human and non-human RefDavis17 categories separately in a zero-shot referring VOS setting.Results are reported by object class.
- Results: Human-class segmentation surpasses non-human segmentation with or without LoomData.This category ordering is reported for both ablation conditions.
- Results: +2.3 J&F improves human-class segmentation when LoomData is incorporated.The reported gain is relative to the corresponding condition without LoomData.
- Results: +2.3 J&F improves non-human-class segmentation when LoomData is incorporated.The paper reports the same gain for non-human categories despite LoomData's human-centric focus.
- Interpretation: The results indicate that detailed textual descriptions contribute to semantic comprehension across categories.The paper connects this observation to VideoLoom's generalization beyond the human class.
E.2 Qualitative Results and Failure Cases
Qualitative results show VideoLoom handling diverse spatial-temporal instructions, while revealing failures on complex queries requiring sub-action or occurrence-level localization. The examples cover referring segmentation, temporal grounding, event localization, and highlight detection.
- Qualitative results: VideoLoom follows diverse spatial-temporal instructions across multiple qualitative tasks.Additional results are presented across the paper's spatial-temporal task examples.
- Failure cases: The model occasionally mislocalizes spatial-temporal content when queries target sub-actions or the n-th occurrence.These cases are identified as complex joint-understanding scenarios.
- Failure cases: The reported failure mechanism is difficulty identifying complete temporal intervals for lengthy motion sequences, causing potentially misaligned spatial-temporal localization.The authors identify temporal action grounding as a likely source and leave further exploration to future work.
- Temporal video grounding: For temporal grounding, VideoLoom predicts frames 0 to 26 for the query about putting books onto a desk.The output follows the requested start- and end-frame format.
- Temporal video grounding: For multi-event localization, VideoLoom outputs consecutive frame intervals and descriptions for seven cooking actions.The sequence spans frames 6 to 108, including cutting sashimi, adding ingredients, and molding sushi.
- Referring video segmentation: VideoLoom identifies a baby horse for the referring segmentation prompt by returning a segmentation output.The displayed response is “It is [SEG].”