LLaVAFlow: Preserving Latent Alignment Flow for Parameter-Efficient Multimodal Fine-Tuning

Muyao Yuan; Muyan Jiao; Jiangyong Ying; Weizhan Zhang; Yuanhong Zhang; Lan Ma; Yuan Gao; Haipeng Du

Source-linked AI summary

LLaVAFlow: Preserving Latent Alignment Flow for Parameter-Efficient Multimodal Fine-Tuning

Muyao Yuan, Muyan Jiao, Jiangyong Ying, Weizhan Zhang, Yuanhong Zhang, Lan Ma, Yuan Gao, Haipeng Du

arXiv:2608.26820v1Aug 27, 2026cs.CV

TL;DR

Visual instruction tuning can cause catastrophic forgetting by disrupting pretrained cross-modal alignment, which existing weight-level approaches overlook. LLaVAFlow extracts a latent alignment flow with an information bottleneck and transfers it between pretrained and fine-tuned MLLMs. Experiments on VQA and captioning report preserved alignment flow, improved downstream performance, and mitigated generalization loss.

Problem
Visual instruction tuning disrupts pretrained cross-modal alignment and generalization, while existing forgetting-reduction methods largely overlook this modality-dependent information.
Method
LLaVAFlow uses an Alignment Flow Module and information-theoretic distillation to compress alignment-relevant relations and maximize mutual information between pretrained and fine-tuned alignment flows.
Results
Experiments on VQA and captioning show that LLaVAFlow preserves alignment flow, improves downstream performance, and mitigates generalization loss.
Takeaways & Limitations
LLaVAFlow is presented as a plug-and-play framework for retaining pretrained alignment while supporting downstream adaptation in MLLMs.

Abstract

from arXiv · show

While Multimodal Large Language Models (MLLMs) exhibit strong generalization, visual instruction tuning for downstream tasks inevitably causes catastrophic forgetting, impairing overall generalization. While existing methods regulate weight updates to reduce forgetting, they overlook the fundamental cross-modal alignment in MLLMs. Based on prior work and our observations, we argue that cross-modal alignment is implicitly captured in the information-compression trajectory. To preserve the alignment flow embedded in the trajectory, we propose LLaVAFlow, an information-theoretic distillation framework. First, we compress the mutual information between the extracted relations and MLLM embeddings, encouraging a learnable module to produce a refined alignment flow that benefits downstream tasks. Second, we maximize the mutual information between the extracted alignment flows of the pretrained and fine-tuned MLLMs, enabling the transfer of compact alignment information. Extensive experiments show that LLaVAFlow is an effective plug-and-play framework that preserves alignment flow and enhances both downstream performance and generalization.

1 Introduction

Visual instruction tuning can disrupt pretrained cross-modal alignment and cause catastrophic forgetting in MLLMs. LLaVAFlow preserves the latent alignment flow through information-theoretic distillation, improving downstream performance while mitigating generalization loss.

Motivation: MLLM fine-tuning risks catastrophic forgetting because visual instruction tuning disrupts the pretrained feature space, modality alignment, and generalization.The problem is intensified by complex architectures, modality gaps, and task interference.
Alignment Flow: Alignment flow describes how visual features progressively evolve into textual semantics within the LLM backbone.Logit Lens exposes this trajectory by mapping hidden states of visual tokens to semantic words during the forward pass.
Challenges: Existing distillation methods inadequately preserve modality-dependent feature flows because they rigidly mimic final distributions or individual features.The alignment flow is latent, and backbone representations contain noise irrelevant to alignment and downstream tasks.
Method: LLaVAFlow uses an Alignment Flow Module to extract alignment information from inter-embedding relations and compress irrelevant information through mutual-information optimization.The refined flow is designed to retain alignment-relevant signals while discarding redundant noise.
Method: The framework transfers compact alignment information by maximizing mutual information between pretrained and fine-tuned alignment flows.This distillation objective is intended to preserve structural knowledge while allowing acquisition of new task-specific knowledge.
Results: Experiments on VQA and captioning show that LLaVAFlow preserves alignment flow, improves downstream performance, and mitigates generalization loss.The reported validation covers both downstream task performance and retention of broader generalization.

2 Related Works

Prior approaches address catastrophic forgetting mainly through weight updates or conventional multimodal distillation, but they do not adequately preserve cross-modal alignment. LLaVAFlow is positioned as an orthogonal approach to this limitation.

Continual Learning: Continual learning methods are commonly categorized as rehearsal-based, regularization-based, or architecture-based strategies for mitigating catastrophic forgetting.For foundation models, the focus has shifted toward preserving pretrained general knowledge during task-specific fine-tuning.
Weight-Level Methods: Weight-focused methods constrain or prune updates to reduce conflicts with prior knowledge, but they overlook modality alignment during fine-tuning.The cited methods include regularization, pruning, and selective updating of a small fraction of weights.
LLaVAFlow: LLaVAFlow complements existing strategies by preserving essential alignment information rather than focusing only on weight updates.This positions the framework as orthogonal to prior weight-level methods.
Knowledge Distillation: Earlier multimodal knowledge-distillation methods provide transfer insights but are inadequate for capturing complex cross-modal alignment in modern MLLMs.Recent work has begun adapting distillation to multimodal foundation models, motivating alignment-specific approaches.

3 Preliminaries

The preliminaries introduce matrix-based Rényi α-entropy and its mutual-information formulation for estimating information directly from data. Gram matrices and normalized kernel matrices provide the required representation without inaccessible density estimation.

Rényi Entropy: Rényi’s α-entropy generalizes Shannon entropy and measures information and interactions among variables.It is defined for a continuous variable through its probability density function.
Matrix-Based Entropy: Matrix-based Rényi α-entropy estimates entropy from the eigenspectrum of a Gram matrix in a reproducing kernel Hilbert space.This avoids requiring an explicitly accessible probability density function.
Gram Matrices: A Gram matrix is constructed from pairwise kernel evaluations over sampled scalar or vector data.Its entries are K_ij = κ(x_i,x_j).
Normalized Matrices: The entropy formulation uses a positive semi-definite matrix with unit trace, constructed in practice from a normalized kernel matrix.The normalized construction preserves positive semi-definiteness and satisfies tr(A) = 1.
Joint Entropy: For multiple variables, matrix-based joint entropy combines normalized kernel matrices for the variables and their Hadamard product.The variables are represented using positive, infinitely divisible kernels over joint samples.
Mutual Information: Mutual information is expressed through entropy, while the matrix-based formulation avoids high-dimensional density estimation and offers an efficient data-based computation.The mutual information between two variables is denoted I_α(A1; A2).

4 Method

LLaVAFlow uses an Alignment Flow Module and information-bottleneck distillation to extract, compress, and preserve multimodal alignment during parameter-efficient fine-tuning. The method approximates alignment flow from input and final embeddings, then transfers refined alignment information from a frozen teacher to the student.

A frozen pretrained MLLM serves as teacher while a parameter-efficiently fine-tuned copy serves as student, with shared AFM parameters regulating adaptation.
Alignment Flow Module: The AFM extracts alignment flow from embeddings before the first layer and after the final layer, using DPI to avoid computation over every hidden state.The final-layer embedding is described as retaining nearly all task-relevant information from the input, reducing computational cost while preserving essential alignment information.
Information Bottleneck Distillation: Flow Refinement Loss compresses mutual information between the teacher’s extracted flow and backbone embeddings, suppressing redundancy and noise while retaining essential multimodal structure.The objective uses matrix-based mutual information for computational tractability and a Frobenius-norm approximation with α=2.
Information Bottleneck Distillation: Alignment Preservation Loss maximizes mutual information between pretrained and fine-tuned alignment flows, transferring relevant alignment knowledge through relaxed imitation rather than rigid output matching.The loss is implemented with shared AFM parameters in teacher and student models and is intended to reduce interference with the task loss.
Alignment Flow Module: The AFM models a token-to-token relation map from normalized input and output embeddings, combining similarity with adaptive attention-based residual refinement.Each relation-map entry reflects relevance between an input token and an output token.

5 Experiments

Experiments evaluate LLaVAFlow across downstream and upstream tasks, alignment trends, semantic trajectories, ablations, and hyperparameter settings. The results indicate that it improves task performance while preserving pretrained alignment and generalization across diverse baselines.

Experimental Setup: Experiments cover VQA and captioning, with Source measuring upstream retention, Target measuring fine-tuned-task performance, and Avg averaging both.The evaluation uses ScienceQA and COCO-Caption, while upstream retention is measured on OKVQA, OCRVQA, GQA, and TextVQA.
Main Results: LLaVAFlow consistently enhances Source and Target performance across adapter tuning, distillation, and parameter-regularization baselines.The gains span LoRA, DoRA, LLaVA-KD, LLaVA-Mod, DARE, Model Tailor, and LoRASculpt, indicating broad plug-and-play compatibility.
Interpretation of LLaVAFlow: In layers 1–25, LLaVAFlow closely matches pretrained LLaVA’s cosine similarity and normalized Euclidean distance, while deeper layers diverge for task adaptation.The shallow layers primarily capture general knowledge, whereas deeper layers encode more task-specific knowledge.
Interpretation of LLaVAFlow: Logit Lens trajectories show that LLaVAFlow preserves most of pretrained LLaVA’s semantic flow and correctly maps a cat image patch to the token “cat.”LoRASculpt loses critical vision-language alignment along the trajectory and produces an incorrect prediction in the cited example.
Ablation Studies: Under rank 64, Alignment Preservation alone raises source performance from 55.67 to 58.71, while Flow Refinement raises target accuracy from 88.49 to 88.52.The ablation attributes source recovery primarily to alignment preservation and target adaptation gains to the information-bottleneck-based flow refinement loss.
Ablation Studies: At rank 64, multilayer Attention modules reduce source accuracy to as low as 38.61, whereas the lightweight single-layer AFM better balances adaptation and alignment retention.The paper attributes the degradation to over-parameterized projection modules absorbing alignment patterns instead of transferring them into the student backbone.

6 Conclusion

LLaVAFlow addresses catastrophic forgetting during visual instruction tuning by distilling latent cross-modal alignment flow. Its information-theoretic framework combines lightweight flow extraction with mutual-information-based refinement and preservation, achieving strong downstream performance while retaining general knowledge.

LLaVAFlow is an information-theoretic distillation framework designed to mitigate catastrophic forgetting during visual instruction tuning.
The Alignment Flow Module extracts latent cross-modal alignment from feature propagation, while information bottleneck compression suppresses redundant noise.
Alignment Preservation maximizes mutual information between pretrained and fine-tuned alignment flows to retain structural cross-modal knowledge during adaptation.
Experiments on VQA and captioning show strong downstream performance and effective retention of the MLLM’s general knowledge.