LLaVA-ST: A Multimodal Large Language Model for Fine-Grained Spatial-Temporal Understanding

Hongyu Li; Jinyu Chen; Ziyu Wei; Shaofei Huang; Tianrui Hui; Jialin Gao; Xiaoming Wei; Si Liu

Source-linked AI summary

LLaVA-ST: A Multimodal Large Language Model for Fine-Grained Spatial-Temporal Understanding

Hongyu Li, Jinyu Chen, Ziyu Wei, Shaofei Huang, Tianrui Hui, Jialin Gao, Xiaoming Wei, Si Liu

arXiv:2501.08282v2Jan 14, 2025cs.CV

TL;DR

Existing MLLMs struggle to localize temporal ranges and spatial coordinates together because coordinate alignment and fine-grained feature compression are difficult. LLaVA-ST addresses these challenges with aligned positional embeddings, separate spatial-temporal packing, progressive training, and ST-Align data, achieving strong performance across 11 benchmarks.

Problem
Existing MLLMs cannot uniformly perform spatial, temporal, and interleaved fine-grained multimodal understanding, while joint localization complicates coordinate alignment and feature compression.
Method
LLaVA-ST combines Language-Aligned Positional Embedding, separate spatial-temporal feature packing, the 4.3M-sample ST-Align dataset, and progressive coarse-to-fine training.
Results
LLaVA-ST achieves strong or outstanding fine-grained multimodal understanding across 11 benchmarks covering temporal, spatial, and spatial-temporal interleaved tasks.
Takeaways & Limitations
The paper presents a unified MLLM for simultaneously processing temporal and spatial dimensions in fine-grained multimodal understanding tasks.

Abstract

from arXiv · show

Recent advancements in multimodal large language models (MLLMs) have shown promising results, yet existing approaches struggle to effectively handle both temporal and spatial localization simultaneously. This challenge stems from two key issues: first, incorporating spatial-temporal localization introduces a vast number of coordinate combinations, complicating the alignment of linguistic and visual coordinate representations; second, encoding fine-grained temporal and spatial information during video feature compression is inherently difficult. To address these issues, we propose LLaVA-ST, a MLLM for fine-grained spatial-temporal multimodal understanding. In LLaVA-ST, we propose Language-Aligned Positional Embedding, which embeds the textual coordinate special token into the visual space, simplifying the alignment of fine-grained spatial-temporal correspondences. Additionally, we design the Spatial-Temporal Packer, which decouples the feature compression of temporal and spatial resolutions into two distinct point-to-region attention processing streams. Furthermore, we propose ST-Align dataset with 4.3M training samples for fine-grained spatial-temporal multimodal understanding. With ST-align, we present a progressive training pipeline that aligns the visual and textual feature through sequential coarse-to-fine stages.Additionally, we introduce an ST-Align benchmark to evaluate spatial-temporal interleaved fine-grained understanding tasks, which include Spatial-Temporal Video Grounding (STVG) , Event Localization and Captioning (ELC) and Spatial Video Grounding (SVG). LLaVA-ST achieves outstanding performance on 11 benchmarks requiring fine-grained temporal, spatial, or spatial-temporal interleaving multimodal understanding. Our code, data and benchmark will be released at Our code, data and benchmark will be released at https://github.com/appletea233/LLaVA-ST .

1. Introduction

Existing MLLMs handle spatial or temporal localization well in isolation but do not uniformly support their simultaneous, interleaved use. LLaVA-ST addresses this gap with coordinate alignment, feature compression, progressive training, and new data and benchmarks.

Existing MLLMs specialize in either spatial localization or fine-grained temporal understanding, rather than uniformly handling spatial, temporal, and interleaved tasks.
Joint spatial-temporal localization expands the coordinate space and complicates cross-modal alignment, while available datasets often lack spatiotemporal tubes paired with linguistic descriptions.
Naive feature compression methods such as Q-Former or pooling lose spatial relationships and fine-grained details needed for simultaneous spatial-temporal comprehension.
LLaVA-ST uses Language-Aligned Positional Embedding to embed textual spatial-temporal coordinates into visual features and Spatial-Temporal Packer to separately compress temporal and spatial information.
ST-Align contains 4.3M samples, including 228K new spatial-temporal interleaved samples, and supports progressive training across content, coordinate, and multi-task alignment stages.
LLaVA-ST reports outstanding performance across benchmarks covering temporal, spatial, and spatial-temporal interleaved understanding tasks.

3. Model Architecture

LLaVA-ST processes images and videos by mapping visual features into textual space, adding language-aligned coordinate embeddings, and compressing video features with separate spatial and temporal streams. Its architecture feeds the resulting features into an LLM for unified fine-grained multimodal understanding.

Overview: The visual encoder extracts features from uniformly sampled video frames, while image inputs use one frame and bypass temporal processing.The extracted features are mapped into textual space with an MLP before positional embedding and packing.
Language-Aligned Positional Embedding: LAPE represents temporal and spatial coordinates with special language tokens whose embeddings and output-layer vectors become visual positional embeddings.The coordinate vocabulary includes temporal anchor tokens and separate width- and height-coordinate tokens.
Language-Aligned Positional Embedding: LAPE interpolates the coordinate-derived positional embedding to match the visual feature shape before adding it to visual features.For images, only the spatial coordinates are used because temporal relationships are unnecessary.
Spatial-Temporal Packer: STP first reduces spatial resolution, then separately compresses spatial and temporal dimensions to produce ˆFs and ˆFt for the LLM.The first reduction uses point-to-region attention, with pooled point features querying regional features to preserve detailed information.
Spatial-Temporal Packer: Point-to-region attention preserves more fine-grained information than direct pooling, supporting localization of spatiotemporal coordinates.The resulting spatial and temporal features are flattened and input to the LLM.

4. ST-Align and Training Strategy

ST-Align provides approximately 4.3M samples for spatial-temporal fine-grained multimodal understanding, while LLaVA-ST uses progressive content, coordinate, and multi-task training stages. The dataset includes spatial-temporal interleaved tasks alongside broader multimodal data.

Dataset: ST-Align comprises approximately 4.3 million training samples for spatio-temporal fine-grained multimodal understanding.The dataset addresses data involving fine-grained multimodal understanding with spatiotemporal interleaving.
Evaluation: The benchmark tables report spatial-temporal interleaved task results and Charades-STA TVG results, with LLaVA-ST described as state of the art for TVG.Table 2 covers the ST-Align benchmark, while Table 3 covers Temporal Video Grounding.
Training Strategy: The training pipeline consists of three sequential stages: content alignment, coordinate alignment, and multi-task instruction tuning.The stages are organized by data quality and visual-text alignment granularity.
Training Strategy: The multi-task stage incorporates 39 high-quality annotated datasets spanning REC, TVG, Video QA, STVG, ELC, and SVG.These datasets provide general visual question answering and fine-grained multimodal understanding capabilities.

5. Experiments

LLaVA-ST performs strongly across spatial, temporal, spatial-temporal, and video question-answering benchmarks using one model with prompt modifications. Its ablations support the roles of LAPE, STP, and spatial-temporal interleaved training data.

Main Comparisons: The same LLaVA-ST model achieves outstanding performance across 12 benchmarks by modifying prompts for different evaluations.The experiments cover multiple multimodal understanding capabilities.
Spatial-Temporal Interleaved Tasks: In STVG, LLaVA-ST reaches 43.8 mtIoU and 22.8 msIoU, exceeding Grounded-VideoLLM by 10.8 and GroundingGPT by 13.6, respectively.It also improves ELC mtIoU, msIoU, and METEOR by 27.3, 19.7, and 18.7 over other models’ best performance, and raises SVG msIoU by 14.6 over GroundingGPT.
Temporal Video Grounding: LLaVA-ST improves Charades-STA TVG mIoU by 6.9 over Grounded-VideoLLM.This result evaluates temporal localization of event start and end timestamps from language descriptions.
Referring Expression Comprehension: On RefCOCO+, LLaVA-ST improves accuracy by 2.1% on validation and 2.4% on test-A against Groma-7B at the same model size.The REC evaluation uses RefCOCO, RefCOCO+, and RefCOCOg.
Video Question Answering: LLaVA-ST attains the highest average VCG-Bench score of 3.32 and the best average MVBench score, improving by 4.8 over previous models.These results cover open-ended and multi-choice video question answering.
Ablation Study: Adding spatial-temporal interleaved data increases TVG mIoU by 3.9 over temporal-only data, whereas adding spatial data alone decreases it by 2.3.RefCOCO validation accuracy remains at the same level in the compared settings.

6. Conclusion

LLaVA-ST combines coordinate-aware visual representations, separate spatial and temporal compression, a large training dataset, and progressive training for fine-grained multimodal understanding. Experiments and ablations report strong performance and support the proposed modules and data strategy.

Conclusion: LLaVA-ST processes temporal and spatial dimensions together for fine-grained multimodal understanding tasks.The model is presented as capable of simultaneous processing across both dimensions.
Conclusion: LAPE embeds coordinate special tokens from text into visual features, while STP separately compresses temporal and spatial resolutions through point-to-region attention.These components target coordinate alignment and preservation of spatiotemporal context during video feature compression.
Conclusion: ST-Align and progressive training sequentially support content alignment, coordinate alignment, and multi-task ability.The conclusion presents these resources and stages as part of the overall approach.
Conclusion: Experimental results show strong fine-grained understanding across multiple benchmarks, while ablations support the effectiveness of LAPE, STP, and spatiotemporally interleaved data.The reported evidence covers both model components and training data.

Supplementary Material

The supplementary material identifies a training-stage ablation in which S.2 and S.3 data can be mixed for comparison.

Training-Stage Ablation: The supplementary training-stage ablation uses “S.2 + S.3” to denote mixed training data from stages S.2 and S.3.The notation is defined in the Table 9 caption.

7. Spatial Temporal Packer

The packers architecture processes region and pooled features with MLPs and cross-attention, while packert uses a similar design with temporal pooling.

7. Spatial Temporal Packer: Packers processes region and pooled features through MLPs before cross-attention.The pooled feature is subsequently added to the output through a residual connection.
7. Spatial Temporal Packer: A residual connection adds the processed pooled feature to the output feature.
7. Spatial Temporal Packer: Packert shares the packers architecture but applies temporal pooling.

8. More Ablations

The ablations show that separating progressive training stages and preserving temporal and spatial resolutions are important for fine-grained performance.

8. More Ablations: Table 10 compares packers and packert ablations using their output feature resolutions in spatial and temporal dimensions.
8. More Ablations: 2.9 msIoU is lost when Stage 2 and Stage 3 data are mixed.TVG accuracy changes minimally on Charades-STA and ST-Align under this mixed strategy.
8. More Ablations: TVG accuracy on Charades-STA drops from 38.5 to 8.8 when only spatially fine-grained features from packert are retained.The ablation highlights the role of higher temporal resolution in temporal fine-grained understanding.

9. ST-Align Data Construction

ST-Align combines grounded image descriptions, derived referring-expression data, segmentation-derived boxes, and metrics for spatial-temporal interleaved evaluation.

9. ST-Align Data Construction: GranD contributes 350K images with Dense Grounded Caption annotations describing each object.
9. ST-Align Data Construction: Language descriptions are paired with bounding-box phrases to enrich REC and REG datasets.Segmentation regions from Muse are converted into object bounding boxes using their bounding rectangles, with low-quality descriptions filtered.
9. ST-Align Data Construction: tIoU measures temporal overlap, while sIoU averages bounding-box overlap over overlapping predicted and ground-truth intervals.The benchmark also reports thresholded and mean variants, including tIoU@0.5, mtIoU, sIoU@0.5, and msIoU.

11. Implementation Details for Baselines

The baseline implementations use staged temporal and spatial localization for grounding tasks and structured prompts for event localization and captioning.

11. Implementation Details for Baselines: GroundingGPT first locates a video’s temporal span from event descriptions, then localizes the target spatially frame by frame.
11. Implementation Details for Baselines: GroundingGPT’s ELC implementation locates the temporal range while simultaneously handling the event description.
11. Implementation Details for Baselines: Grounded-VideoLLM and VtimeLLM use a prompt that requests an event caption and an end frame from a specified start frame.The requested output is formatted as caption and end-frame fields.

12. Visualized Results

The visualized results illustrate LLaVA-ST’s spatial-temporal, temporal, image-level, instruction-following, and multi-turn dialogue capabilities, alongside prompts for constructing fine-grained training tasks.

Temporal video understanding: The examples also visualize temporal video grounding and temporal referring for video inputs.These are identified in Figures 10 and 11.
Image understanding: Image-level demonstrations include dense grounded captioning, referring expression comprehension, and region captioning.Figures 12 and 13 present these fine-grained image-understanding results.
Dialogue and instruction following: Figures 14 and 15 show that LLaVA-ST can follow general language instructions and engage in multi-turn open-ended video question answering.The section includes two examples of multi-turn open-ended video QA.
Instruction-data prompts: GranD-generated prompts request detailed image descriptions with interleaved bounding boxes for corresponding answer parts.The prompt set includes multiple paraphrased requests for detailed or comprehensive image descriptions.
Instruction-data prompts: The instruction prompts include temporal questions asking when a described event occurs in a video.Example formulations request the event’s timestamp or time interval.