PaperlayerView exact revision on arXiv

Source-linked AI summary

Video-XL-Pro: Reconstructive Token Compression for Extremely Long Video Understanding

Xiangrui Liu, Yan Shu, Zheng Liu, Ao Li, Yang Tian, Bo Zhao

arXiv:2503.18478v2cs.CV

TL;DR

Long-video MLLMs face rising visual-token and memory costs, while existing compression lacks quality criteria and uses fixed ratios. Video-XL-Pro combines self-supervised ReCoT with dataset pruning and query-aware selection. Its 3B model outperforms most 7B models across long-video benchmarks and processes over 8K frames on one A100 with over 95% Needle-in-a-Haystack accuracy.

  • Problem

    Longer videos increase visual-token and memory costs, while existing compression lacks an established quality criterion and typically applies fixed ratios regardless of video information intensity.

    Source: §1 · p. 1

  • Method

    Video-XL-Pro uses self-supervised ReCoT with DTS and SGM to produce compact video tokens, plus video-data pruning and query-aware token selection for MLLM training.

    Source: §1 · p. 2

  • Results

    With 3B parameters, Video-XL-Pro outperforms most 7B video MLLMs across several long-video benchmarks and processes over 8K frames on one A100 with over 95% Needle-in-a-Haystack accuracy.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The authors conclude that Video-XL-Pro improves video-understanding quality and efficiency through compact tokens, higher-quality training data, and precise localization of query-relevant tokens.

    Source: §5 · p. 9

Abstract

from arXiv · show

Despite advanced token compression techniques, existing multimodal large language models (MLLMs) still struggle with hour-long video understanding. In this work, we propose Video-XL-Pro, an efficient method for extremely long video understanding, built upon Reconstructive Compression of Tokens (ReCoT), a learnable module that leverages self-supervised learning to generate comprehensive and compact video tokens. ReCoT introduces two key components: (i) Dynamic Token Synthesizer (DTS): DTS generates pseudo-video tokens from static image tokens by learning intra-token relationships, which are then used in masked video modeling. (ii) Semantic-Guided Masking (SGM): SGM adaptively masks redundant visual tokens to facilitate more effective reconstructive learning. To improve training efficiency in MLLMs fine-tuning, we introduce a video-specific dataset pruning strategy and design a simple yet Query-aware Selector that enables the model to precisely locate query-relevant video tokens. With only 3B parameters, Video-XL-Pro outperforms most 7B models trained on larger datasets across multiple long video understanding benchmarks. Moreover, it can process over 8K frames on a single A100 GPU while maintaining high-quality performance.

1. Introduction

Long-video processing drives visual-token and memory costs, while existing compression lacks direct quality criteria and typically uses fixed ratios. Video-XL-Pro proposes learnable reconstructive compression alongside training-data pruning and query-aware selection.

1. Introduction

  • Longer videos increase visual-token counts and memory costs, while existing compression lacks an established quality criterion and typically uses fixed ratios that ignore differences in video information intensity.The passage identifies these issues for videos exceeding three minutes.Source: §1 · p. 1
  • DTS uses lightweight spatio-temporal attention to learn intra-token relationships and dynamic motion, addressing image encoders’ difficulty capturing motion, especially in high-FPS videos.The paper notes that common image encoders struggle with dynamic motion, while video encoders have limited adaptability to arbitrary frames.Source: §1 · p. 2
  • SGM adaptively masks tokens using intra-frame information density and inter-frame spatio-temporal significance, preserving meaningful regions for reconstructive learning.Its learnable selection is intended to avoid random masking of redundant or low-information regions.Source: §1 · p. 2
  • Joint image-video pretraining produces compact ReCoT tokens, while frame-wise similarity differences guide video-data pruning and a RAG-based selector locates query-relevant tokens.The pruning strategy is intended to improve fine-tuning efficiency and generalization by filtering redundant or low-information videos.Source: §1 · p. 2
  • Despite 3B parameters, Video-XL-Pro outperforms most 7B video MLLMs across several long-video benchmarks, processes over 8K frames on one A100, and exceeds 95% accuracy on Needle-in-a-Haystack.ReCoT also achieves comparable or superior performance to SigLIP on ImageNet, UCF101, and Kinetics.Source: §1 · p. 2

2. Related work

Prior MLLM work develops ways to align visual and language representations, while video MLLMs extend these systems to multiframe inputs and long-video token reduction.

2. Related work

  • MLLM approaches combine visual and textual representations using cross-attention, Q-Former modules, or direct projection of visual features into the LLM embedding space.Examples include Flamingo, BLIP-2, MiniGPT-4, and LLaVA.Source: §2 · p. 2
  • Video MLLMs encode and concatenate multiframe features, apply temporal attention, or compress tokens; some methods use memory mechanisms or query-conditioned cross-attention.The cited examples include MovieChat, MA-LMM, LLaMA-VID, Chat-UniVi, and LongVLM.Source: §2 · p. 2
  • Video-XL uses LLM key-value sparsification and a visual summarization token, whereas Figure 2 presents Video-XL-Pro as separate reconstructive-compression and MLLM-training stages.The latter stage includes video dataset pruning and query-aware selection.Source: §2 · p. 3, Figure 2 · p. 3

3. Method

Video-XL-Pro first learns compact visual tokens through self-supervised reconstruction, then fine-tunes an MLLM with query-aware selection and video-data pruning. ReCoT combines DTS and SGM, while variable frame sampling supports training on longer videos.

3.1. Overview

  • The two-stage method pretrains reconstructive token compression, then fine-tunes an MLLM using the pretrained SigLIP-DTS encoder.The overview also introduces video dataset pruning and a Query-Aware Selector to extend supported context length.Source: §3.1 · p. 3

3.2. Reconstructive Token Compression

  • DTS follows SigLIP, merges tokens from four frames into one frame’s worth, and reconstructs compressed features with an MSE loss.Under SGM, SigLIP tokens are compressed by a factor of four; the DTS training description reports 48 A100 GPU hours.Source: §3.2 · pp. 3–4
  • SGM scores tokens using temporal and spatial queries, then randomly masks low-scoring tokens to guide DTS toward efficient reconstruction.Its scores combine information from the previous frame with the semantic average of the current frame’s tokens.Source: §3.2 · p. 4

3.3. MLLM Training

  • The Query-Aware Selector scores visual tokens against the text query and drops irrelevant tokens when input length exceeds the LLM’s limit.Video Dataset Pruning groups videos by task, samples 20 frames, and filters high-redundancy examples using SigLIP similarity.Source: §3.3 · p. 4
  • Variable sampling uses up to 360 frames, sampling short videos at varying rates and long videos at 1 fps.Figure 3 compares the training-video duration distributions for variable and uniform sampling.Source: §3.3 · p. 5, Figure 3 · p. 5

4. Experiment

The experiments evaluate Video-XL-Pro on long-video understanding and temporal grounding benchmarks. The 3B model performs competitively against larger models, including on MLVU, Video-MME, and V-STaR.

4.1. Training Data

  • Pretraining uses 2M image-text samples and 400K video samples, followed by fine-tuning on image and pruned video datasets.The pretraining data is filtered for image quality and video length; the fine-tuning mix includes Bunny, ShareGPT-4o, OCR-VQA, and pruned video data.Source: §4.1 · pp. 5–6

4.2. Implementation Details

  • Pretraining trains only the projector, while fine-tuning updates all parameters; experiments use 32 A800-80GB GPUs.Videos are sampled at 1 FPS, with a uniform 240-frame limit during fine-tuning.Source: §4.2 · p. 6

4.3. Benchmarks and Metrics

  • Evaluation covers MLVU, Video-MME, VNBench, LongVideoBench, TempCompass, Charades-STA, and V-STaR.The benchmarks span video question answering, retrieval, temporal understanding, and temporal grounding.Source: §4.3 · p. 6

4.4. Main Results

  • The 3B model outperforms most 7B models on MLVU and leads open-source models on TempCompass, while reaching 60.0% and 64.3% Video-MME accuracy.The Video-MME figures are for the no-subtitles and with-subtitles settings, respectively; TempCompass scores exceed Qwen2.5-VL-3B by 1.3% and Apollo 7B by 0.8%.Source: §4.4 · p. 6

4.5. Temporal Grounding

  • Video-XL-Pro shows highly competitive temporal-grounding performance on V-STaR, nearly outperforming existing 7B MLLMs.The comparison is against methods using more complex timestamp-perception architectures.Source: §4.5 · p. 6

4.6. Extra-Long Evaluation

Video-XL-Pro handles more than 8,000 frames on one A100 while retaining near-99% needle-retrieval accuracy. The experiments also report faster training and inference and assess the model’s compression, masking, pruning, and selector components.

4.6. Extra-Long Evaluation

  • Beyond 8,000 frames, Video-XL-Pro maintains nearly 99% needle-retrieval accuracy on one A100, versus Video-XL’s 2,048-frame limit.The evaluation also reports that LongVA supports fewer than 400 frames, while LLaVA-NexT-Video and LongLLaVA cannot support videos exceeding 1,000 frames.Source: §4.6 · p. 6, Figure 4 · p. 7

4.7. Efficiency Analysis

  • Video-XL-Pro improves training speed by more than 2 times and inference speed by nearly 3 times versus the baseline, alongside performance gains.Figure 5 presents the training and inference efficiency comparison.Source: §4.7 · p. 6, Figure 5 · p. 7

4.8. Ablation Studies

  • With DTS, the pretrained encoder outperforms SigLIP on multiple image benchmarks and remains better on video classification after fourfold compression.The evaluation uses frozen features with a trainable two-layer MLP classifier.Source: Table 4 · p. 7, §4.8 · p. 7
  • With ReCoT, the model performs better on image and video benchmarks despite producing fewer visual tokens.The ablation uses 50K images and 50K videos and evaluates MLVU, VideoMME, MME, and MMB.Source: §4.8 · p. 7, Table 5 · p. 9
  • At a fixed DTS compression rate, Semantic-Guided Masking improves image and video understanding benchmarks over random masking.The comparison is reported in Table 6.Source: §4.8 · p. 7, Table 6 · p. 9
  • Pruning 100K SFT examples yields results comparable to randomly selected 200K data, particularly on MLVU.The authors report that the pruned data supports state-of-the-art performance with a relatively small training scale.Source: §4.8 · p. 8, Table 7 · p. 9
  • The Query-Aware Selector compresses tokens to 50% without information loss and retains considerable accuracy at 90% compression.Its rates are evaluated on MLVU, VideoMME, and LongVideoBench.Source: Figure 7 · p. 9, §4.8 · p. 9

4.9. Qualitative Results

A qualitative comparison on videos longer than 30 minutes shows Video-XL-Pro locating key information and details more accurately than Qwen2.5-VL-7B.

4.9. Qualitative Results

  • The qualitative comparison uses ultra-long videos exceeding 30 minutes and Qwen2.5-VL-7B as the baseline.Figure 6 presents the comparison.Source: §4.9 · p. 9
  • Video-XL-Pro accurately locates key information in the long videos and presents its details.The authors describe this as evidence of robustness in long-video understanding.Source: §4.9 · p. 9
  • In contrast, Qwen2.5-VL-7B struggles with long-video content and fine-grained details.Source: §4.9 · p. 9

5. Conclusion

Video-XL-Pro uses ReCoT and supporting data and selection methods to improve long-video understanding. The 3B-parameter model outperforms larger models on multiple long-video benchmarks.

5. Conclusion

  • The 3B-parameter Video-XL-Pro outperforms larger models on multiple long-video understanding benchmarks.The conclusion attributes this to a method designed for efficient long-video understanding.Source: §5 · p. 9
  • ReCoT uses self-supervised learning to generate comprehensive, compact video tokens through Dynamic Token Synthesizer and Semantic-Guided Masking.The conclusion says these components improve video-understanding quality and efficiency.Source: §5 · p. 9
  • Video-XL-Pro also uses video-specific dataset pruning and a post-processing mechanism to improve training-data quality and locate query-relevant video tokens.Source: §5 · p. 9

Overview of Supplementary Material

The supplementary material covers training datasets, experimental settings, and additional results.

Overview of Supplementary Material

A. Training Datasets

Video-XL-Pro uses a two-stage fine-tuning process and video datasets pruned for training. ReCoT is trained self-supervised on one million data points from publicly accessible datasets.

A. Training Datasets

  • Video-XL-Pro freezes SigLIP, DTS, and the LLM while training the projection module, then performs full-parameter fine-tuning.Video datasets used in both stages are compressed with the Video Dataset Pruning method.Source: A. Training Datasets · p. 13
  • The two-stage training uses video-text pairs from multiple publicly accessible databases, including LLaVA-OneVision, LAION-2M, Charades, CinePine, OpenViD, and Ego4d.The passage also lists ShareGPT4o, OCRVQA, Next-QA, and LLaVA178k.Source: A. Training Datasets · p. 13
  • Using only 1 million data points from publicly accessible datasets including Bunny, llava178k, and OCRVQA, ReCoT achieves superior performance to original SigLIP features across multiple image and video benchmarks.Source: A. Training Datasets · p. 13

B. Experimental Settings & Additional Results

The supplementary material describes Video-XL-Pro’s fine-tuning hyperparameters and points to a table containing them.

B. Experimental Settings & Additional Results

Loading 2503.18478v2…