Source-linked AI summary
Temporal Gains, Spatial Costs: Revisiting Video Fine-Tuning in Multimodal Large Language Models
Linghao Zhang, Jungang Li, Yonghua Hei, Sicheng Tao, Song Dai, Yibo Yan, Zihao Dongfang, Weiting Liu, Chenxi Qin, Hanqian Li, Xin Zou, Jiahao Zhang, Shuhang Xun, Haiyun Jiang, Xuming Hu
TL;DR
The paper asks whether Video-SFT transfers video improvements to static-image understanding in MLLMs, a question that remains insufficiently examined. It systematically analyzes this balance across architectures, scales, and frame budgets, develops a theoretical account, and studies adaptive frame allocation. The results show a recurring temporal trap: video performance improves while image performance often weakens, with Hybrid-Frame partially mitigating the trade-off.
Problem
It remains unclear whether progress in video understanding reliably transfers to image understanding in MLLMs.
Method
The paper evaluates Video-SFT across architectures, parameter scales, and frame budgets, provides a conservative shared-parameter analysis, and studies instruction-aware adaptive frame allocation.
Results
Video-SFT improves temporal or video understanding but often weakens spatial or image capability, while larger temporal budgets can intensify the trade-off.
Takeaways & Limitations
Preserving spatial capability remains a central challenge for joint image–video training, although adaptive frame allocation can partially mitigate the temporal trap.
Takeaways & Limitations
The study covers representative MLLMs and standard benchmarks but not the full space of architectures, training schemes, evaluation settings, or streaming and interactive scenarios; Hybrid-Frame is heuristic.
Abstract
from arXiv · showhide
Multimodal large language models (MLLMs) are typically trained in multiple stages, with video-based supervised fine-tuning (Video-SFT) serving as a key step for improving visual understanding. Yet its effect on the fine-grained evolution of visual capabilities, particularly the balance between spatial and temporal understanding, remains poorly understood. In this paper, we systematically study how Video-SFT reshapes visual capabilities in MLLMs. Across architectures, parameter scales, and frame sampling settings, we observe a consistent pattern: Video-SFT reliably improves video performance, but often yields limited gains or even degradation on static image benchmarks. We further show that this trade-off is closely tied to temporal budget: increasing the number of sampled frames generally improves video performance, but does not reliably improve static image performance. Motivated by this finding, we study an instruction-aware Hybrid-Frame strategy that adaptively allocates frame counts and partially mitigates the image-video trade-off. Our results indicate that Video-SFT is not a free lunch for MLLMs, and preserving spatial understanding remains a central challenge in joint image-video training.
1 Introduction
The paper examines whether Video-SFT transfers video gains to static-image understanding in MLLMs. It identifies a recurring image–video trade-off, links it to temporal budget, and studies adaptive frame allocation as a partial mitigation.
- Video-SFT is widely used to improve video understanding within unified image–video MLLMs.
- The paper addresses whether video progress reliably transfers to static-image understanding, an assumption not previously examined systematically.
- Across architectures, scales, frame settings, and benchmarks, Video-SFT improves video performance but often provides limited image gains or degrades image performance.
- Increasing sampled frames generally improves video performance but does not reliably improve image performance.
- The authors call this recurring image–video trade-off the temporal trap and relate it to temporal budget.
- The instruction-aware Hybrid-Frame Strategy adaptively allocates frame counts and partially mitigates the image–video trade-off.
2 Related Work
Prior work has studied unified visual modeling and conflicts across tasks or modalities, but image–video conflicts under Video-SFT remain comparatively underexplored. This paper targets that gap with systematic analysis and adaptive frame allocation.
- Recent MLLMs increasingly process images and videos within shared architectural and training frameworks.
- Continual tuning studies report gradient conflicts, negative transfer, and catastrophic forgetting when models adapt across tasks or modalities.
- Compared with prior work on text–vision conflicts, image–video conflicts under Video-SFT remain underexplored.
- This paper systematically analyzes the balance between image and video capabilities and evaluates adaptive frame allocation as a partial mitigation.
3 Experimental Setup and Overview
The study evaluates how Video-SFT affects image and video understanding across model architectures, scales, and frame budgets. It uses curated video training data and established image and video benchmarks.
- The experiments compare image understanding and video understanding under Video-SFT.
- Model architecture: The study spans Qwen2.5-VL, LLaVA-Next-Video, and LLaVA-1.5 architectures.
- Model scale: Qwen2.5-VL is evaluated at 3B, 7B, 32B, and 72B parameter scales.
- Frame sampling setting: Video-SFT samples videos uniformly at 8, 16, 32, or 64 frames.
- Training data: The curated training set contains 20,000 videos from 10 sources and includes descriptions, open-ended questions, and multiple-choice questions.
- Evaluation benchmarks: Image evaluation uses MME, MMStar, MMBench, and POPE, while video evaluation uses Video-MME, MVBench, TempCompass, and Video-MMMU.
4 The Temporal Trap behind Visual Modality Conflict
Video-SFT creates a systematic temporal trap: video performance improves while static-image understanding often degrades, despite shared visual encoders. The conflict varies with architecture, model scale, and frame budget, with larger models and adaptive temporal allocation offering partial mitigation.
- Video-SFT enhances video performance while often degrading image performance, revealing a conflict between temporal adaptation and spatial visual reasoning.The paper names this recurring image–video trade-off the temporal trap.
- Impact of Model Architecture: Qwen2.5-VL remains comparatively stable, whereas LLaVA-1.5 exhibits the largest image-benchmark drop and LLaVA-NeXT-Video a smaller gap.The temporal trap nevertheless persists across all evaluated architectures.
- Impact of Model Scale: At 72B, post-SFT image performance becomes comparable to or slightly better than the base model on most image benchmarks, but 3B–32B results fluctuate.Scale partially mitigates the negative effect, but the mitigation is not strictly monotonic and larger models can be prohibitively costly.
- Impact of Model Architecture: After Video-SFT, smaller Qwen2.5-VL models show more dispersed target-object attention, while larger models retain more localized spatial focus.The attention shift suggests that larger models better preserve stable spatial representations under the temporal trap.
- Impact of Fine-Tuning Frame Count: Increasing training frames consistently improves video benchmarks but produces diminishing returns, while image benchmarks show dataset-dependent patterns rather than reliable gains.MME remains below the base model across frame settings; MMStar improves with slowing gains, while MMBench and POPE increase then decrease.
- Impact of Fine-Tuning Frame Count: Redundant temporal information during Video-SFT can disrupt static visual representations and weaken image-task generalization, producing the temporal trap.
5 Theoretical Analysis
The analysis explains how shared-parameter Video-SFT can improve video objectives while harming image objectives, especially as temporal specialization grows with frame budget. It motivates adaptive, sample-wise frame allocation to reduce redundant temporal exposure.
- Gradient conflict: A single Video-SFT update directly optimizes the video objective, while image preservation depends on how its gradient aligns with the image gradient.Positive alignment indicates cooperative directions; negative alignment indicates local conflict.
- Gradient conflict: Negative alignment between video and image gradients can make a Video-SFT update improve video performance while degrading spatial capability.The analysis treats this as a local sufficient condition under shared-parameter optimization, not a complete internal mechanism.
- Gradient conflict: Repeated small updates can accumulate measurable spatial degradation when Video-SFT is applied late without explicitly optimizing the image objective.This remains possible even with a smaller learning rate when alignment is persistently biased.
- Temporal budget: Larger temporal budgets can intensify conflict when temporally specialized updates gain weight faster than shared spatial benefits.The proposed decomposition separates a shared component, a temporally specialized component, and residual sampling or redundancy effects.
- Adaptive allocation: Sample-adaptive allocation uses fewer frames for temporally simple samples and more for temporally demanding ones, avoiding redundant temporal exposure.The analysis identifies the sample-wise minimal sufficient temporal budget as the target allocation.
- Connection to empirical findings: The empirical pattern is consistent with the analysis: video gains can coexist with image degradation, and increasing frame counts can shift transfer from cooperative to conflicting.These propositions provide a principled lens on the observed image–video trade-off.
6 Hybrid-Frame Strategy
The Hybrid-Frame Strategy adaptively allocates frames using instruction-aware predictors to preserve necessary temporal evidence while reducing redundancy. It improves the image–video balance across frame budgets and architectures while retaining video gains.
- Strategy design: Hybrid-Frame allocates enough frames for task-relevant temporal evidence while avoiding redundant temporal exposure.The strategy compares DINOv2-based allocation with predictors built on Qwen2.5-VL-3B and Qwen3-VL-8B.
- Strategy design: Both VLM-based predictors outperform the DINOv2-based strategy in reliability, with Qwen3-VL-8B achieving the best overall performance.Qwen2.5-VL-3B remains competitive, suggesting instruction-aware allocation can work with a smaller predictor.
- Empirical results: Hybrid-Frame achieves the best MMStar and POPE accuracy while outperforming fixed 32- or 64-frame budgets and maintaining strong video gains.This result is reported for Qwen2.5-VL-7B using the Qwen3-VL-8B-based strategy.
- Empirical results: Reducing redundant frames after temporal sufficiency is reached can improve the image–video trade-off without weakening useful supervision.The strategy is therefore presented as an effective intervention that also reduces unnecessary training cost.
- Cross-architecture results: Hybrid-Frame remains effective across Video-SFT budgets on LLaVA-1.5-7B, indicating its benefit is not tied to one architecture.The paper characterizes adaptive frame allocation as broadly useful for reducing the image–video trade-off in unified MLLMs.
7 Conclusion
Video-SFT improves temporal understanding across architectures, scales, and frame budgets, but often weakens spatial capability. Adaptive frame allocation can partially mitigate this trade-off, while preserving spatial understanding remains unresolved.
- Across architectures, parameter scales, and frame budgets, Video-SFT improves temporal understanding but often weakens spatial capability.
- The image–video trade-off is closely tied to temporal budget, with adaptive frame allocation partially mitigating it.
- Preserving spatial capability under Video-SFT remains a central challenge for joint image–video training.
Limitations
The study covers representative MLLMs and standard image–video benchmarks, but not the full space of architectures, training schemes, evaluation settings, or streaming and interactive scenarios. Its Hybrid-Frame strategy is heuristic rather than fully principled.
- The study covers representative MLLMs, not the full space of architectures, training schemes, or evaluation settings.
- The evaluation excludes streaming inputs and interactive multimodal reasoning, which may involve different modeling and analysis.
- The proposed Hybrid-Frame strategy is heuristic rather than fully principled.
A.2 Experiment Design
The experiments examine Video-SFT across model architectures, parameter scales, frame counts, and visual task categories. Results show consistent degradation in static image perception, especially for fine-grained spatial tasks, while cognitive reasoning is comparatively robust.
- Experiment Design: Three experiments target model architecture, model scale, and training frame counts to assess Video-SFT’s effect on MLLM visual capability.
- Static Image Perception: Across evaluated models, Video-SFT degrades static image perception, with fine-grained tasks suffering the most significant drops.
- Static Image Perception: Celebrity recognition in MME dropped by 80.59 points for LLaVA-1.5-7B, 52.35 points for LLaVA-Next-Video, and 54.71 points for Qwen2.5-VL-2.5-7B.
- Cognitive Reasoning: Qwen2.5-VL-7B improved by +7.50 on MME numerical calculation, code reasoning, and text translation after Video-SFT.
- Cognitive Reasoning: On MMStar, Math and Science & Technology remained stable or improved slightly after Video-SFT.
B.2 Impact of Input Differences during Training and Inference on Static Visual Capability
Controlled repeated-image experiments show that static visual degradation persists even when training and inference use identical frame counts. Increasing redundant frames hurts image performance, while Hybrid-Frame allocation preserves stronger video performance under an 8-frame inference budget.
- Controlled Input Experiment: A controlled experiment repeats one static image across the Video-SFT frame count to test whether modality mismatch explains image-performance degradation.
- Controlled Input Experiment: For every frame count, Video-SFT models underperform Base models, showing degradation persists under identical input modalities.
- Redundant Frames: In the Base model, increasing repeated images from 8 to 64 reduced MME from 2360 to 2272, MMStar from 60.20 to 58.00, and MMBench from 87.83 to 85.74.
- Task-Level Effects: Figure B reports the largest Video-SFT performance drop in Fine-Grained Perception compared with General Understanding and Visual Reasoning.
- Hybrid-Frame Strategy: With fixed 8-frame inference, Hybrid-Frame achieved 63.94% on MVBench, outperforming models trained with 16, 32, and 64 frames.
- Hybrid-Frame Limitations: The Hybrid-Frame strategy selects among 8, 16, 32, and 64 frames using textual instruction, but its current design is limited by fixed discrete choices and text-only awareness.
C.2 General-Purpose Multimodal Models
Unifying image and video processing is presented as an important step for general-purpose MLLMs, but Video-SFT can create conflicts between visual modalities. The section analyzes this conflict and introduces Hybrid-Frame as a preliminary adaptive strategy.
- Unifying images and videos is identified as a crucial step toward MLLMs that process multiple modalities within one framework.
- The study systematically examines whether Video-SFT benefits static-image and temporal visual tasks jointly across current MLLMs.The experiments compare models before and after Video-SFT across tasks on the MMStar and MMBench benchmarks.
- Video reasoning improvements may fail to transfer to image understanding or even degrade it, revealing a conflict between temporal and spatial capabilities.The paper suggests temporal supervision might undermine spatial generalization, while qualifying this explanation as a possibility.
- Hybrid-Frame adaptively estimates the frame count needed for each instruction by assessing temporal evidence and spatial granularity.Its prompt considers event duration, motion continuity, causality, object interaction, and fine-grained visual attributes.