Source-linked AI summary
SlowFast-LLaVA: A Strong Training-Free Baseline for Video Large Language Models
Mingze Xu, Mingfei Gao, Zhe Gan, Hong-You Chen, Zhengfeng Lai, Haiming Gang, Kai Kang, Afshin Dehghan
TL;DR
Training-free Video LLMs often process few frames and rely on the LLM to model temporal dependencies, limiting their ability to capture video detail. SlowFast-LLaVA uses complementary low-rate, detail-rich and high-rate, spatially pooled pathways. Across eight benchmarks, it outperforms existing training-free methods and matches or exceeds fine-tuned models on some benchmarks.
Problem
Many Video LLMs process only a limited number of frames and lack explicit temporal modeling, making it difficult to capture fine-grained spatial and temporal content across videos.
Source: §1 · p. 2Method
SF-LLaVA is a training-free model built on LLaVA-NeXT that aggregates low-rate, spatially detailed Slow features with high-rate, spatially pooled Fast features.
Source: §1 · p. 2Results
Across eight benchmarks, SF-LLaVA outperforms existing training-free methods and is on par with or better than fine-tuned models on some benchmarks.
Source: §1 · p. 2, §5 · p. 10Takeaways & Limitations
The authors present SF-LLaVA as a simple, strong baseline for Video LLMs and its design ablations as insights for future video representation research.
Source: §5 · p. 11Uniform frame sampling can miss key frames, and SF-LLaVA cannot precisely detect a video moment’s start and end times.
Source: §4.6 · p. 10
Abstract
from arXiv · showhide
We propose SlowFast-LLaVA (or SF-LLaVA for short), a training-free video large language model (LLM) that can jointly capture detailed spatial semantics and long-range temporal context without exceeding the token budget of commonly used LLMs. This is realized by using a two-stream SlowFast design of inputs for Video LLMs to aggregate features from sampled frames in an effective way. Specifically, the Slow pathway extracts features at a low frame rate while keeping as much spatial detail as possible (e.g., with 12x24 tokens), and the Fast pathway operates on a high frame rate but uses a larger spatial pooling stride (e.g., downsampling 6x) to focus on the motion cues. As a result, this design allows us to adequately capture both spatial and temporal features that are beneficial for detailed video understanding. Experimental results show that SF-LLaVA outperforms existing training-free methods on a wide range of video tasks. On some benchmarks, it achieves comparable or even better performance compared to state-of-the-art Video LLMs that are fine-tuned on video datasets. Code has been made available at: https://github.com/apple/ml-slowfast-llava.
1 INTRODUCTION
SlowFast-LLaVA addresses training-free Video LLMs’ limited frame coverage and lack of temporal modeling with complementary slow and fast input pathways. Across eight benchmarks, it outperforms existing training-free methods and is on par with or better than fine-tuned models on some benchmarks.
1 INTRODUCTION
- Existing Video LLMs often process only a few frames and lack temporal modeling designed for motion, limiting their ability to capture fine-grained spatial and temporal content across a video.The cited examples use 6 frames for IG-VLM and 16 for PLLaVA.Source: §1 · p. 2
- SF-LLaVA uses low-frame-rate, spatially detailed Slow features and high-frame-rate, spatially pooled Fast features to combine visual semantics with motion cues.The examples use 8 frames with 24 × 24 tokens on Slow and 4 × 4 tokens per Fast frame.Source: §1 · p. 2
- The model uniformly samples video frames, extracts features independently, processes them through both pathways, and feeds their concatenation to the LLM.The Slow pathway samples a subset of frames; the Fast pathway uses all sampled frames.Source: §1 · p. 2
- Across eight benchmarks, SF-LLaVA outperforms existing training-free methods and is on par with or better than fine-tuned models on some benchmarks.The evaluation spans Open-Ended VideoQA, Multiple Choice VideoQA, and Text Generation.Source: §1 · p. 2
2 RELATED WORK
Related work spans general-purpose image LLMs, video-specific LLMs, and training-free approaches that adapt image LLMs for video. The paper identifies limited frame coverage and insufficient temporal modeling as key drawbacks of training-free methods.
2 RELATED WORK
- Image LLMs connect pretrained visual and text models in different ways, including Flamingo’s interleaved inputs, BLIP-2’s Q-Former, and LLaVA’s connector and instruction-data pipeline.Other work explores component and data ablations, additional modalities, and 3D representations.Source: §2 · p. 2
- Video LLMs use varied strategies, including pooled frame features, video descriptions and appearance embeddings, shared visual-language projectors, and fine-tuning image LLMs on video data.PLLaVA and LLaVA-NeXT-Video are described as fine-tuned approaches.Source: §2 · p. 3
- Training-free methods adapt image LLMs for video but often handle only a few frames and rely on the LLM to model temporal dependencies.The cited examples process 4 frames in FreeVA and 6 in IG-VLM; the paper proposes a SlowFast design that can take more frames, such as 50.Source: §2 · p. 3
3 SLOWFAST-LLAVA
SF-LLaVA adapts a training-free Video LLM pipeline with two complementary feature pathways: Slow preserves spatial detail at a lower frame rate, while Fast retains temporal coverage with coarser spatial features. Their aggregated representation is passed to the LLM alongside the prompt and question.
3 SLOWFAST-LLAVA
- SlowFast-LLaVA builds on LLaVA-NeXT without further fine-tuning and uses two-stream inputs to capture detailed spatial semantics and long-range temporal context within commonly used LLM token budgets.The examples use 8 Slow frames with 12 × 24 tokens each and 64 Fast frames with 4 × 4 tokens each.Source: §3 · p. 3
3.1 PRELIMINARIES: TRAINING-FREE VIDEO LLMS
- A training-free Video LLM uses a pretrained image LLM without further fine-tuning, and its pipeline samples video frames, extracts visual features, aggregates them, and feeds them to the LLM with the question.The aggregator is used to leverage temporal prior knowledge and reduce video tokens to avoid exceeding the LLM token limit.Source: §3.1 · pp. 3–4
- The pipeline’s prompt is a system instruction that guides the LLM, and its design is adapted from image to video scenarios.The paper says it experiments with different prompts and discusses instruction design later.Source: §3.1 · p. 4
3.2 SLOWFAST ARCHITECTURE
- SF-LLaVA uniformly samples frames, extracts features independently, and processes them in two pathways before combining them as a video representation.Slow samples fewer frames and applies spatial pooling with a small stride to retain detail; Fast keeps all frames and uses a larger spatial pooling stride to focus on temporal context and motion cues.Source: §3.2 · p. 4
- Flattened Slow and Fast features are concatenated as the aggregated video representation and passed to the LLM with text tokens.The pathways are not separated with special tokens.Source: §3.2 · pp. 4–5
4 EXPERIMENTS
SF-LLaVA is evaluated across video question-answering and text-generation benchmarks, where it compares favorably with training-free baselines and is competitive with fine-tuned models. The experiments also examine pathway, pooling, and prompt choices, and identify limits in fine-grained temporal grounding and uniform frame sampling.
4.1 BENCHMARKS AND METRICS
- The evaluation covers open-ended VideoQA, multiple-choice VideoQA, and text generation, using GPT-3.5-Turbo-0125 for GPT-assisted assessment.Open-ended VideoQA uses accuracy and answer-quality scores; multiple-choice uses accuracy, while text generation assesses five dimensions.Source: §4.1 · p. 5
4.2 IMPLEMENTATION DETAILS
- Experiments use LLaVA-NeXT 7B and 34B models, with 50 uniformly sampled frames and 3,680 SlowFast visual tokens at the default setting.Inference on the 34B model reaches the 80GB GPU-memory limit at this token count.Source: §4.2 · p. 5
4.3 MAIN RESULTS
- Across Open-Ended VideoQA, Multiple Choice VideoQA, and Text Generation, SF-LLaVA outperforms training-free baselines and is competitive with fine-tuned models.On Open-Ended VideoQA it is on par with fine-tuned methods on most benchmarks; on EgoSchema, the 34B model exceeds an SFT method by 2.5%.Source: §4.3 · pp. 5–6, Table 1 · p. 6, Table 2 · p. 7
4.4 DESIGN CHOICES OF SLOWFAST
- Removing either pathway underperforms the corresponding tested alternatives, including comparisons designed to match the number of input tokens.For the Slow-path comparison, SF-LLaVA beats a single-stream baseline using 3,200 tokens versus 3,104 tokens.Source: §4.4 · pp. 7–8
- More frames in either pathway improve performance on the tested QA datasets, while pooling trades spatial detail against context and memory limits.Spatial pooling beats temporal pooling when token counts are matched, especially on EgoSchema; 1 × 1 Fast-path pooling may suit videos over 30 minutes.Source: §4.4 · pp. 8–9
4.5 DESIGN CHOICES OF PROMPT
- Input-structure prompts improve results on both datasets, task instructions help EgoSchema but not ActivityNet-QA, and structured-answer prompts also improve results.SF-LLaVA uses the input-data prompt for all tasks and task instructions only for Multiple Choice VideoQA.Source: §4.5 · p. 10
4.6 ERROR ANALYSIS
- SF-LLaVA cannot localize precise event start and end times, and uniform sampling can miss brief moments; dynamic sampling is suggested as a direction.The paper also notes that additional fine-tuning on fine-grained temporal datasets and multimodal inputs could improve this capability.Source: §4.6 · p. 10
5 CONCLUSION
SF-LLaVA is a training-free Video LLM built on LLaVA-NeXT, using a SlowFast two-stream design to represent detailed spatial semantics and long-range temporal context. Across eight video benchmarks, it outperforms existing training-free methods and is on par with or better than state-of-the-art fine-tuned models on some benchmarks.
5 CONCLUSION
- Across eight video benchmarks, SF-LLaVA outperforms existing training-free methods and is on par with or better than state-of-the-art fine-tuned models on some benchmarks.The paper presents SF-LLaVA as a simple but strong baseline for Video LLMs.Source: §5 · pp. 10–11
- Built on LLaVA-NeXT without additional fine-tuning, SF-LLaVA uses two-stream inputs to capture detailed spatial semantics and long-range temporal context.The SlowFast design aggregates frame features as a video representation.Source: §5 · p. 10
- The authors propose that ablations of SF-LLaVA's design choices can provide insights for future research on video representations for Multimodal LLMs.Source: §5 · p. 11