Source-linked AI summary
LongVILA: Scaling Long-Context Visual Language Models for Long Videos
Yukang Chen, Fuzhao Xue, Dacheng Li, Qinghao Hu, Ligeng Zhu, Xiuyu Li, Yunhao Fang, Haotian Tang, Shang Yang, Zhijian Liu, Ethan He, Hongxu Yin, Pavlo Molchanov, Jan Kautz, Linxi Fan, Yuke Zhu, Yao Lu, Song Han
TL;DR
Long-context VLMs need coordinated data, training, and systems support to process long videos, while existing approaches leave parts of this problem underdeveloped. LongVILA combines staged context and video training with MM-SP, achieving million-token retrieval, strong benchmark performance, and efficient scaling.
Problem
Existing long-context VLM approaches use short-context training or token compression, leaving comprehensive training frameworks and dataset design insufficiently addressed.
Method
LongVILA combines a five-stage training curriculum with MM-SP, a multi-modal sequence-parallel framework for memory-intensive long-context VLM training and inference.
Results
LongVILA extends feasible video processing from 8 to 2048 frames, while achieving 99.8% accuracy across 6,000-frame videos and strong performance across popular video benchmarks.
Takeaways & Limitations
LongVILA provides a full-stack approach for long-context visual-language modeling that combines model capability with scalable distributed training.
Takeaways & Limitations
Existing VLM sequence-parallel systems face workload imbalance from modality heterogeneity and network inefficiency across intra-node and inter-node communication.
Abstract
from arXiv · showhide
Long-context capability is critical for multi-modal foundation models, especially for long video understanding. We introduce LongVILA, a full-stack solution for long-context visual-language models by co-designing the algorithm and system. For model training, we upgrade existing VLMs to support long video understanding by incorporating two additional stages, i.e., long context extension and long video supervised fine-tuning. However, training on long video is computationally and memory intensive. We introduce the long-context Multi-Modal Sequence Parallelism (MM-SP) system that efficiently parallelizes long video training and inference, enabling 2M context length training on 256 GPUs without any gradient checkpointing. LongVILA efficiently extends the number of video frames of VILA from 8 to 2048, achieving 99.8% accuracy in 6,000-frame (more than 1 million tokens) video needle-in-a-haystack. LongVILA-7B demonstrates strong accuracy on 9 popular video benchmarks, e.g. 65.1% VideoMME with subtitle. Besides, MM-SP is 2.1x - 5.7x faster than ring style sequence parallelism and 1.1x - 1.4x faster than Megatron with a hybrid context and tensor parallelism. Moreover, it seamlessly integrates with Hugging Face Transformers.
1 INTRODUCTION
LongVILA addresses gaps in long-context VLM training with a full-stack pipeline and MM-SP system. It extends video understanding to million-token contexts while achieving strong benchmark and retrieval results.
- Long-context VLMs matter for processing information-rich inputs such as long documents, codebases, and hour-length videos.
- Existing approaches often rely on long-context LLMs trained with short-context data or token compression, leaving training frameworks and dataset design underdeveloped.
- LongVILA combines a five-stage training curriculum with MM-SP for memory-intensive long-context VLM training and inference.
- 99.8% accuracy was achieved on 6,000-frame needle-in-a-haystack videos containing more than 1 million tokens, while LongVILA-7B reached 65.1% on VideoMME with subtitle.
- 2.1× to 5.7× speedups over ring-style sequence parallelism and 1.1× to 1.4× over Megatron’s hybrid strategy were reported.
2 RELATED WORKS
Prior VLM architectures and sequence-parallel systems provide the foundations for LongVILA. The related work highlights trade-offs between token reduction, communication efficiency, and scalability.
- Visual language model architecture: Encoder-decoder VLMs connect vision encoders to language decoders through multi-modal projectors that can reduce tokens per image or frame.
- Visual language model architecture: LongVILA follows the VILA-1.5 model architecture and training pipeline.
- Sequence parallelism and hybrid strategy: Sequence parallelism distributes long sequences across devices because long-context examples can exceed single-device memory.
- Sequence parallelism and hybrid strategy: Ulysses can outperform ring-style sequence parallelism through All-to-All communication, but its scalability is limited by the number of attention heads.
3 LONGVILA TRAINING PIPELINE
LongVILA uses staged training, long-video instruction data, and MM-SP to adapt VLMs to extensive video contexts. The pipeline first extends language-model context, then supports long-video supervised fine-tuning.
- Training curriculum: The five stages are multi-modal alignment, large-scale pre-training, supervised fine-tuning, LLM context extension, and long supervised fine-tuning.
- Training curriculum: Stages 1–3 progressively train the projector, general multi-modal capacity, and short-data instruction following.
- Long-video data: Instruction data is generated by segmenting videos into approximately 10-second clips, captioning them with VILA-1.5, and producing question-answer pairs with an LLM.
- Context extension: Stage 4 extends LLM context to 262,144 using continued pre-training with a progressive 8,192-to-262,144 schedule and 17B tokens.
- Long-video data: The long-video supervised fine-tuning dataset contains 15,292 videos with captioning and question-answer samples across varied video and question categories.
- System motivation: Long-video samples can contain hundreds or thousands of frames, motivating MM-SP because a 1,400-frame sequence can reach around 274k tokens.
4 MULTI-MODAL SEQUENCE PARALLELISM
MM-SP addresses long-video training and inference bottlenecks caused by modality heterogeneity, network heterogeneity, and limited sequence-parallel scalability. It combines customized sharding, 2D attention, and distributed inference to improve workload balance, scalability, and efficiency.
- Limitations of Existing Systems: Long-context VLM training requires sequence parallelism because a single sequence can exceed one device’s memory, while fully sharded data parallelism does not distribute its activations.This motivates a custom sequence-parallel system for memory-intensive long-context training.
- Modality Heterogeneity: A single video frame typically expands from one placeholder into around 256 visual tokens, so treating visual and text placeholders identically creates GPU workload imbalance.The mismatch arises because visual inputs are encoded into multiple real tokens during training.
- Networking Heterogeneity: NVLink provides 900 GB/s intra-node bandwidth versus 50 GB/s for InfiniBand inter-node bandwidth, an 18× difference that makes uniform P2P communication inefficient.The cited design can fail to hide communication overhead and may slow the computation kernel.
- Limited Maximal Sequence Length: DeepSpeed-Ulysses is limited by attention-head parallelism, reaching at most 8 sequence-parallel devices for Llama-3 8B’s 8 Key-Value heads without replication.Even with Key-Value head replication, the cited maximum is 32 devices, limiting support for extremely long sequences.
- MM-SP Workflow: MM-SP adopts 2D attention and a two-stage sharding strategy that distributes video frames for balanced image encoding before token-level sharding for language modeling.The workflow adds dummy tokens so sequences can be evenly divided for ring-based attention.
- Distributed Inference: MM-SP also provides distributed VLM inference by managing decoding-time tensors and terminating the distributed process when the machine holding the last token signals completion.Its inference mode is described as more efficient than Hugging Face pipeline parallelism.
5 EXPERIMENTAL RESULTS
LongVILA is evaluated across video understanding benchmarks and long-context system tasks, including throughput, maximum sequence length, inference latency, and sharding efficiency. MM-SP provides strong scalability and speed, while LongVILA supports accurate retrieval and competitive benchmark performance on long videos.
- Training System: 2.1× to 5.7× speedup over ZIGZAG-RINGATTN and 3.1× to 4.3× over Megatron-LM CP are achieved on 32 H100 GPUs.The method is also 1.1× to 1.4× faster than Megatron-LM’s hybrid strategy.
- Training System: Over 2 million context length is supported on 256 GPUs without activation checkpointing, with approximately 8× higher context lengths than DeepSpeed-Ulysses.The comparison progressively increases per-GPU sequence length until an out-of-memory error occurs.
- Inference System: 8.2× inference speedup and 2.9× longer supported sequences are achieved over HuggingFace Pipeline on 8 H100 GPUs.MM-SP uses all GPUs jointly, whereas HuggingFace Pipeline activates only one GPU at a time and produces imbalanced activation allocation.
- Effect of Two-Stage Sharding: Two-stage sharding yields a 1% to 7% speedup over image-count-only sharding, primarily in longer captioning tasks.The baseline suffers workload imbalance because it does not shard according to text-token counts.
- General Video Understanding: 65.1% overall VideoMME accuracy with subtitle is reported for LongVILA using 256 frames.The corresponding score without subtitle is 60.1%.
- Needle-in-a-Haystack: 99.8% accuracy is achieved across 6,000 frames containing more than 1 million tokens in the needle-in-a-haystack experiment.The 32-frame baseline could not accurately retrieve images beyond 32 frames.
6 CONCLUSION
LongVILA combines a five-stage training pipeline, curated long-video data, and MM-SP to extend visual-language models to long contexts. It supports 2048 frames, achieves strong video benchmark and needle-in-a-haystack results, and scales context length to 2 million tokens with faster distributed processing.
- Conclusion: LongVILA extends the feasible frame count from 8 to 2048 using curated long-video datasets and a five-stage training pipeline.The pipeline combines model training stages with the distributed MM-SP system.
- Conclusion: 65.1% accuracy with subtitle is reported for LongVILA-7B on VideoMME.The conclusion describes strong performance across popular video benchmarks.
- Conclusion: 99.8% accuracy is achieved in 6,000-frame needle-in-a-haystack videos containing more than 1 million tokens.The result concerns fine-grained information capture in long videos.
- Conclusion: MM-SP scales context length to 2 million tokens, with 2.1× to 5.7× speedups over ring sequence parallelism and 1.1× to 1.4× over hybrid Megatron parallelism.The comparison covers hybrid context and tensor parallelism for Megatron.
A.1 LONGVILA-CAPTION
LongVILA-Caption evaluates how increasing video frame counts affects long-video captioning, using both quantitative scores and qualitative examples. More frames improve caption detail and contextual coverage.
- Benchmark: The LongVILA-Caption benchmark contains 100 long videos with captions verified through human examination.Evaluation considers correctness, detailed orientation, and contextual understanding.
- Quantitative results: Average caption scores rise from 2.00 to 3.26 as training and evaluation frames increase from 8 to 256.The reported frame settings are 8, 128, and 256.
- Qualitative examples: With 256 frames, sports captions capture player actions, jersey colors, and celebratory hugging missing from the 8-frame baseline.The qualitative comparison shows richer event and narrative descriptions at higher frame counts.
- Qualitative examples: With 256 frames, technology captions describe a snowy road, multiple car views, and close-ups of controls absent from the 8-frame baseline.The additional details include the ignition button, gear shift, and dashboard elements.