Source-linked AI summary
Kangaroo: A Powerful Video-Language Model Supporting Long-context Video Input
Jiajun Liu, Yibing Wang, Hanghang Ma, Xiaoping Wu, Xiaoqi Ma, Xiaoming Wei, Jianbin Jiao, Enhua Wu, Jie Hu
TL;DR
Long-video understanding is limited by scarce high-quality video data and excessive visual-feature compression. Kangaroo addresses these gaps with curated training data, curriculum training, and progressively larger visual inputs, achieving state-of-the-art performance with 8B parameters and outperforming some larger and proprietary models on long-video benchmarks.
Problem
Long-video understanding is limited by insufficient high-quality video data and aggressive compression of visual features.
Method
Kangaroo builds curated datasets and uses curriculum training with progressively increased resolution and input frames for video-language learning.
Results
With 8B parameters, Kangaroo achieves state-of-the-art performance on most benchmarks and competitive results on others, surpassing some larger and proprietary models on long-video benchmarks.
Takeaways & Limitations
Kangaroo demonstrates strong comprehensive and long-video understanding while preserving more visual context through higher resolution and more input frames.
Abstract
from arXiv · showhide
Rapid advancements have been made in extending Large Language Models (LLMs) to Large Multi-modal Models (LMMs). However, extending input modality of LLMs to video data remains a challenging endeavor, especially for long videos. Due to insufficient access to large-scale high-quality video data and the excessive compression of visual features, current methods exhibit limitations in effectively processing long videos. In this paper, we introduce Kangaroo, a powerful Video LMM aimed at addressing these challenges. Confronted with issue of inadequate training data, we develop a data curation system to build a large-scale dataset with high-quality annotations for vision-language pre-training and instruction tuning. In addition, we design a curriculum training pipeline with gradually increasing resolution and number of input frames to accommodate long videos. Evaluation results demonstrate that, with 8B parameters, Kangaroo achieves state-of-the-art performance across a variety of video understanding benchmarks while exhibiting competitive results on others. Particularly, on benchmarks specialized for long videos, Kangaroo excels some larger models with over 10B parameters and proprietary models.
I. INTRODUCTION
Kangaroo addresses long-video understanding by combining curated video-language data with curriculum training and expanded visual context. With 8B parameters, it achieves state-of-the-art performance on most benchmarks and competitive results on others.
- Current video-language methods are limited by scarce high-quality video data and aggressive visual-token compression that loses critical information.
- Kangaroo combines a high-quality dataset, curriculum training, increased resolution, and more input frames to address long-video understanding challenges.
- The data curation system collects, filters, and captions image and video data for vision-language pre-training and instruction tuning.
- With 8B parameters, Kangaroo achieves state-of-the-art performance on most video understanding benchmarks and competitive results on others.
- On long-video benchmarks, Kangaroo outperforms certain models with over 10B parameters and proprietary models.
II. RELATED WORK
Related work extends multimodal models through modality alignment, specialized datasets, visual-token compression, and long-context adaptation. Despite these efforts, long-video understanding remains constrained by limited high-quality video data.
- Broader LMM research advances modality alignment, spatial grounding, specialized datasets, and multi-task generalization through architectural and training strategies.
- Existing video LMMs commonly encode frames and either aggregate visual features or concatenate projected frame features before language modeling.
- Long-video methods use token compression, memory modules, timestamp-aware encoders, or sliding mechanisms to accommodate extended inputs.
- Stepwise approaches extend context sizes or exploit long-context language models, but limited access to high-quality video data remains an obstacle.
III. DATA CURATION
Kangaroo’s data curation system filters low-quality videos before generating frame-level captions and coherent video descriptions. The pipeline is designed to improve the quality of video-text pre-training data.
- The system discards original image captions judged low quality and creates a higher-quality image pre-training dataset through re-captioning.
- 1) Text Coverage Filter:: Text coverage filtering detects text regions in three frames and removes videos whose maximum text coverage exceeds a preset threshold.
- The curation pipeline collects videos, filters low-quality content, generates frame captions, summarizes them into video descriptions, and removes repetitive descriptions during refinement.
- Frame-level captions are summarized by a language model into coherent descriptions for video pre-training.
2) Face Coverage Filter:
Kangaroo filters videos with excessive facial content, insufficient temporal variation, unsuitable durations, or imbalanced categories to improve training-data quality.
- 2) Face Coverage Filter:: Videos with excessively large face coverage are excluded after detecting facial regions in five uniformly sampled frames with YOLOv8.
- Videos with optical-flow magnitude below a threshold are classified as close-to-static and filtered out.
- Videos are duration-filtered by removing overly short videos and segmenting clips from long videos.
- Category imbalance is reduced by resampling majority categories until each remains below 1% of the final dataset.
C. Video Caption Generation
Kangaroo’s video caption generation pipeline samples frames across each video, generates frame-level captions, and synthesizes them into video-level descriptions. The resulting pre-training dataset exceeds 60 million video-caption pairs, but some captions remain repetitive or insufficiently restructured.
- Videos are divided into five segments, one frame is sampled per segment, and frame captions are synthesized into a cohesive video caption.
- The dataset’s category distribution is analyzed after balancing, alongside the instruction-tuning dataset’s distribution.
- Over 60 million video-caption pairs form the large-scale video pre-training dataset.
- Some generated captions simply concatenate frame descriptions or repeat highly correlated content, reducing informative coverage of the full video.
- Sentence-overlap filtering uses pairwise IoU similarity to remove captions with excessive semantic redundancy before refinement.
- The instruction-tuning dataset spans multiple video-understanding tasks and uses task-specific prompts to improve instruction following.
1) Video Caption:
Kangaroo’s instruction-tuning data combines captioning, video question answering, and conversational video-understanding examples. The collection is organized across languages and task types, with preprocessing applied before training.
- 1) Video Caption:: The instruction-tuning dataset includes short captions and detailed descriptions in both English and Chinese.
- 2) Video QA:: Its video QA data covers multiple-choice and open-ended questions spanning fundamental, egocentric, reasoning, action-recognition, and long-video tasks.
- Conversational data includes single- and multi-round interactions in English and Chinese, with additional Chinese conversations generated from high-quality descriptions.
- The dataset composition is summarized statistically and reported in Table I.
- Videos shorter than five seconds are excluded, and all remaining videos are resampled to 4 fps to accelerate training data loading.
A. Architecture
Kangaroo combines a vision encoder, projector, spatial-temporal patchification, and an LLM, while adding timestamp-based temporal information to sampled video frames. Frame features are concatenated and projected into the LLM embedding space.
- A. Architecture: Kangaroo uses a vision encoder, multi-modal projector, spatial-temporal patchify module, and LLM as its core architecture.
- A. Architecture: The model uniformly samples video frames and applies sinusoidal temporal position encoding using each frame’s timestamp.
- A. Architecture: Timestamp-based encoding augments visual features with temporal information rather than relying only on frame indices.
- A. Architecture: Visual features from all frames are concatenated with special tokens to model temporal inter-dependencies.
- A. Architecture: The concatenated visual sequence is projected into the LLM embedding space as the visual branch output.
B. Curriculum Training
Kangaroo uses curriculum training that progressively increases task complexity, input resolution, and frame count to build long-video processing ability. The pipeline spans visual-language pre-training, refinement, instruction tuning, and long-video tuning.
- Progressive training equips a text-based LLM to process increasingly complex long-video inputs.
- Stage I: Visual-language Pre-training: Image and video pre-training first connects language concepts with visual elements and temporal information.
- Stage II: Pre-training Refinement: Pre-training refinement increases inputs to 16 × 448 × 448 frames to address inconsistent captions and repetitive content.
- Stage III: Instruction Tuning: Instruction tuning fine-tunes the entire model across multiple tasks after preliminary visual-linguistic alignment.
- Stage III: Instruction Tuning: Dynamic sampling selects 16–64 frames according to video duration, covering long videos while avoiding redundancy in short ones.
- Stage IV: Long Video Tuning: Long-video tuning raises the maximum input to 160 frames and includes short videos to preserve comprehension across video lengths.
V. EXPERIMENTS
Experiments evaluate Kangaroo across diverse video-understanding tasks using standardized greedy decoding and comparisons with open-source and proprietary models. Qualitative examples examine reasoning, dialogue, bilingual interaction, and long-video comprehension.
- C. Qualitative Results: Qualitative reasoning examples test accurate answers grounded in visual details, including camera angles, actions, and counterfactual scenarios.
- C. Qualitative Results: Long-video examples assess global content understanding and precise localization of relevant segments and details.
- Standardized evaluation uses greedy decoding and compares Kangaroo with open-source and proprietary models.
- Evaluation covers a wide range of video-understanding tasks organized into three benchmark categories.
1) General Video Understanding:
Kangaroo performs strongly across general, specialized, and long-video benchmarks. It reaches state-of-the-art or competitive performance among open-source models and exceeds some larger or proprietary systems on selected evaluations.
- 1) General Video Understanding:: Kangaroo achieves state-of-the-art performance on MMBench-Video and VideoVista among similarly sized open-source LMMs.
- 1) General Video Understanding:: On VideoMME, Kangaroo excels across all sub-tasks and outclasses some models with over 10B parameters.
- 1) General Video Understanding:: With identical input-frame counts, Kangaroo outranks most multi-image LMMs, including Claude-3, and performs close to GPT-4V.
- 2) Specialized Tasks:: Kangaroo achieves open-source state-of-the-art results on EgoSchema and TempCompass and outperforms GPT-4V on EgoSchema.
- 3) Long Video Understanding:: On long-video benchmarks, Kangaroo outperforms other 7B/8B models and surpasses some larger and proprietary models on certain metrics.
- C. Qualitative Results: Qualitative examples show accurate reasoning, bilingual multi-round dialogue, and comprehension of global and segment-level long-video content.
VI. CONCLUSION
Kangaroo combines curated video data, high-resolution inputs, and more input frames for long-video understanding. Its evaluation shows strong performance across comprehensive benchmarks and selected advantages over larger or proprietary models.
- Kangaroo combines a large-scale curated video dataset with high-resolution inputs and more frames to preserve visual details in long videos.
- Extensive evaluation demonstrates superior performance across diverse comprehensive video-understanding benchmarks.
- On specialized long-video benchmarks, Kangaroo outperforms larger models with over 10B parameters and proprietary models on certain metrics.
- Future work targets additional modalities, larger language models, and more complex tasks such as counting, visual grounding, and temporal localization.