Source-linked AI summary
A Visual Dependence-Aware Framework for Multimodal Unsupervised Continual Post-Training
Kaichen Li, Zhilin Zhu, Jianhao Huang, Zhengqin Lai, Baochen Xiong, Zibo Shao, Yaguang Song, Linhui Xiao, Xiaoshan Yang, Changsheng Xu
TL;DR
Existing unsupervised post-training for MLLMs overlooks heterogeneous token-level visual dependence, motivating MU-CPT for sequential learning from unlabeled multimodal streams. The paper proposes VDA, combining VC-OT for preserving old-task visual dependence with VMA for visually grounded new-task adaptation. Across six tasks, VDA achieves the strongest reported stability–plasticity balance and the highest AvgAcc of 62.5%.
Problem
Existing MLLM unsupervised post-training methods optimize target tokens uniformly, while deployed models must learn sequentially from non-stationary unlabeled multimodal data.
Method
VDA uses VC-OT to preserve old-task visual-dependence structure and VMA to emphasize visually attributed tokens during new-task learning.
Results
VDA achieves the highest AvgAcc of 62.5%, with AvgLA of 63.0 and forgetting of 0.6, yielding the best reported stability–plasticity balance.
Takeaways & Limitations
Token-level visual dependence provides a unified basis for reducing cross-modal forgetting while retaining strong new-task plasticity in MU-CPT.
Takeaways & Limitations
The formulation assumes that reference mass in the language-dependent set is negligible and need not be preserved as visual attribution.
Abstract
from arXiv · showhide
In this paper, we explore a novel task of Multimodal Unsupervised Continual Post-Training (MU-CPT), enabling deployed MLLMs to continually evolve from streaming unlabeled data. Existing unsupervised post-training methods for MLLMs typically optimize target tokens uniformly, overlooking their heterogeneous visual dependence (VD). However, we reveal that token-level VD is crucial for MU-CPT. Specifically, its structural distortion serves as an indicator of cross-modal catastrophic forgetting, and its inherent heterogeneity acts as a compass to guide new-task learning. Leveraging this property, we propose a Visual Dependence-Aware (VDA) framework with two main components. First, Visually Constrained Optimal Transport (VC-OT) formulates the VD structural distortion of old-task VD during new-task learning as an optimal transport problem to mitigate cross-modal forgetting. By designing a region-aware ground cost and a dependence-stratified transport penalty, it prevents global shifts in visual focus while strictly prohibiting visual reliance from degenerating into language bias. Second, Visually Modulated Adaptation (VMA) exploits VD heterogeneity to emphasize visually grounded new-task learning, promoting new-task plasticity. Together, our method simultaneously maintains old-task stability and new-task plasticity during challenging MU-CPT. Extensive experiments under our MU-CPT setting validate the effectiveness of VDA.
Introduction
The paper introduces MU-CPT for MLLMs learning sequentially from unlabeled, non-stationary multimodal streams, and proposes VDA to balance old-task stability with new-task plasticity through token-level visual dependence.
- Problem Setting: MU-CPT enables MLLMs to sequentially learn distinct multimodal tasks without labeled answers while preserving previously acquired capabilities.The setting targets continuously evolving, non-stationary unlabeled data streams.
- Motivation: Existing unsupervised post-training methods optimize target tokens uniformly, overlooking heterogeneous token-level visual dependence in multimodal learning signals.The paper measures VD by comparing token log-likelihood under real versus information-free counterfactual visual input.
- Motivation: Token-level visual dependence is heterogeneous across text tokens, while new-task learning structurally distorts old-task visual dependence.Figure 2 distinguishes highly visually dependent tokens from those with low visual dependence.
- Proposed Framework: VDA uses visual dependence as a unified signal to harmonize old-task stability and new-task plasticity during MU-CPT.The framework addresses cross-modal attributional forgetting caused by visual-dependence distortion.
- Proposed Framework: VC-OT formulates cross-modal attributional-forgetting mitigation as optimal transport that preserves visual-dependence strength and structure.Its region-aware cost regulates redistribution using visual spatial similarity, while a cross-set penalty blocks transport toward vision-independent tokens.
Related Work
Prior work covers continual learning, continual self-supervised learning, multimodal continual learning, and unsupervised post-training, with unsupervised post-training for MLLMs emerging as a related direction.
- Continual Learning: Continual learning balances new-task plasticity and old-task stability through regularization, rehearsal, and parameter-isolation approaches.Continual self-supervised learning extends this focus to unlabeled streams, mainly for unimodal representation learning.
- Continual Learning: Multimodal continual learning studies sequential visual question answering and prompt-based adaptation.These approaches address continual learning in multimodal settings.
- Unsupervised Post-Training: Unsupervised post-training adapts models to unlabeled inputs using self-generated learning signals, including self-consistency, response consistency, input perplexity, and entropy.The cited methods include LSMI, ScPO, TLM, and related semantic or token-level entropy approaches.
- Unsupervised Post-Training: For MLLMs, related unsupervised post-training work includes SeVa, while existing unsupervised approaches are primarily developed for static, closed-world data settings.The paper contrasts these settings with continuously evolving, non-stationary multimodal streams.
Methodology
MU-CPT trains MLLMs sequentially on answer-unlabeled multimodal tasks while preserving previous capabilities. VDA operates at token-level visual dependence, combining VC-OT for old-task stability with VMA for visually grounded new-task plasticity.
- Problem Formulation: MU-CPT sequentially learns distinct multimodal tasks from answer-unlabeled data while acquiring current capabilities and preserving previous ones.The updated model is evaluated on held-out image-question pairs from all encountered tasks.
- Token-Level Visual Dependence: VDA uses token-level visual-dependence heterogeneity as an anchor for balancing continual-learning stability and plasticity.Unlike sequence-level methods that optimize target tokens uniformly, VDA addresses MU-CPT at the token level.
- Visually Constrained Optimal Transport: VC-OT treats old-task visual-dependence distortion as optimal transport, preserving dependence strength and structure through region-aware costs and dependence-stratified penalties.The method separates visually attributed tokens H from vision-independent tokens L and considers intra-H redistribution and H→L transfer.
- Token-Level Visual Dependence: Visual dependence compares token prediction under real and information-free counterfactual visual inputs; positive values indicate visual facilitation, while non-positive values indicate primarily language-driven prediction.The comparison keeps the target token and preceding textual context unchanged.
- Visually Constrained Optimal Transport: Region-aware transport permits mild redistribution between visually similar tokens while penalizing global shifts in visual focus.Attention-distribution discrepancy across selected decoder layers is measured with Jensen-Shannon Divergence, with greater differences receiving higher cost.
- Visually Constrained Optimal Transport: Dependence-stratified transport assigns maximum penalty to H→L transfers, preventing visual reliance from moving onto language-driven tokens.The penalty uses JSD cost 1, corresponding to completely non-overlapping visual attention distributions.
- Visually Modulated Adaptation: VMA reweights incoming-task token losses to amplify visually attributed tokens while retaining a unit contribution for others and neutralizing average sequence scale.It uses current visual dependence, stop-gradient modulation, and sequence-level normalization.
Experiments
Experiments evaluate VDA in MU-CPT across six multimodal tasks, measuring overall performance, stability, plasticity, visual attribution, and VD preservation. VDA achieves the strongest reported balance between learning new tasks and retaining prior capabilities.
- Experimental Setup: VDA is evaluated on six tasks in a fixed sequential order, with all seen test sets measured after each learning stage.The tasks span OCR, scientific diagrams, financial charts, compositional reasoning, driving scenes, and medical VQA.
- Evaluation Metrics: AvgAcc is the primary stability–plasticity metric, while AvgF measures forgetting and AvgLA measures immediate post-learning task performance.These metrics respectively summarize final accuracy, previous-task performance drops, and new-task learning.
- Overall Comparison: 62.5% AvgAcc makes VDA the strongest method, exceeding ScPO by 3.7 points and SEEKR-MLLM by 2.4 points.VDA also improves the frozen Qwen2.5-VL-7B backbone by 8.6 points under shifting, answer-unlabeled data.
- Stability–Plasticity Analysis: VDA reaches 63.0% AvgLA and 0.6 forgetting, combining the highest plasticity with the lowest forgetting among the compared continual-learning methods.SEEKR-MLLM has AvgLA of 62.5% but greater forgetting, whereas CL-MoE has AvgLA of 59.5% with lower adaptation.
- Ablation Study: VMA improves plasticity, VD-strength preservation improves stability, and their combination yields the complete model’s 62.5% AvgAcc, 63.0% AvgLA, and 0.6 AvgF.The ablation identifies complementary roles for visually modulated adaptation and the two VD-preservation terms.
- Cross-Modal Comprehension: VDA maintains an RRAR of 4.204 after the full sequence, exceeding the baseline by 0.499; removing VMA or VC-OT lowers RRAR to 3.986 or 3.914.RRAR tracks attention allocated to question-relevant visual regions on TextVQA.
- VD Preservation Strategies: VC-OT achieves comparable VD preservation while regulating visually consistent and harmful redistribution differently, producing low AvgF and the highest global AvgAcc.Token-level L1 has the lowest AvgF of 0.4 but reaches only 59.1 AvgAcc, indicating a trade-off from rigid positionwise matching.
- Hyperparameter Analysis: AvgAcc is best at LoRA rank r=128, while a buffer of 1,000 already achieves 62.5% AvgAcc before larger-buffer gains saturate.Higher ranks eventually increase forgetting, whereas larger buffers generally improve performance with diminishing returns.
Conclusion
VDA enables MLLMs to continually learn from non-stationary, unlabeled multimodal data while reducing forgetting and retaining strong plasticity across six tasks.
- Experiments on six tasks show that VDA reduces forgetting while retaining strong new-task plasticity in MU-CPT.