Wings: Learning Multimodal LLMs without Text-only Forgetting

1 Introduction

• MLLMs often forget how to respond to text-only instructions after multimodal training, with performance declining substantially on text-only evaluation.The problem arises despite the strong visual captioning, understanding, and reasoning capabilities of MLLMs.Source: §1 · p. 1
• Text-only and interleaved conversations are practically important, but replaying additional data or deploying separate LLMs increases data, computation, memory, or deployment costs.Users may begin with text-only questions and later add images, while text remains important even in multimodal instructions.Source: §1 · p. 2
• MLLM-LAWS analysis links text-only performance to attention shifts between text before and after inserted image tokens, where negative correlation indicates overfocus on visual tokens.Across more than 100 MLLMs, stronger text-only models show more similar attention to the surrounding textual segments.Source: §1 · p. 2
• WINGS adds parallel visual and textual learners within each attention block to compensate for attention shifts while preserving complementary modality focus.Visual learners first support image-text alignment; textual learners are later integrated through attention-based routing, using efficient LoRRA modules.Source: §1 · p. 2
• WINGS demonstrates comprehensive performance across text-only, visual-question-answering, and Interleaved Image-Text benchmarks.The IIT benchmark spans settings from text-only-rich to multimodal-rich question answering.Source: Figure 1 · p. 2, §1 · p. 3

2.1 Granting Sight to Large Language Models

• MLLMs extend pretrained LLMs with visual encoders and projectors, but continued multimodal learning can produce competitive modality shifts and catastrophic text-only forgetting.Prior mitigation approaches use additional text-only or interleaved data, while WINGS targets preservation of language ability with fewer resources and much lower overhead than large MoE expansions.Source: §2.1 · p. 3

2.2 Capturing the Attention Shift with MLLM-LAWS

• MLLM-LAWS measures layer-wise attention proportions on text before the image, the image itself, and text after the image.Visual feature tokens are inserted into the textual feature sequence, dividing the surrounding text into before-image and after-image regions.Source: §2.2 · pp. 3–4, Figure 2 · p. 4
• Attention to textual tokens typically rises and then falls across layers, while visual attention often follows the opposite trend; forgetting disrupts the post-image textual pattern.The deviation indicates that attention to text after the image has shifted relative to attention to text before the image.Source: §2.2 · p. 4, Figure 2 · p. 4
• Across more than 100 diverse MLLMs, attention shift correlates with text-only performance degradation.The analysis motivates mitigating shifted attention weights with additional learners.Source: §2.2 · p. 4

3 WINGS: Flying to Generality with Low-Rank Residual Attention Learners

• WINGS adds parallel visual and textual learners to each attention block, with an attention-weight router combining their outputs to compensate for shifted attention.The architecture uses complementary modality-specific learners alongside the main branch.Source: §3 · p. 4, Figure 3 · p. 5, §3.2 · p. 6

3.1 Revisit the Training Pipeline of the MLLM

• MLLM inputs combine visual tokens from an encoder and projector with textual instruction tokens, which may surround visual features at arbitrary sequence positions.The visual span is indexed between vstart and vend, while the remaining positions represent textual features.Source: §3.1 · p. 5

3.2 Visual and Textual Learners Weave WINGS

• The method motivates modality-specific auxiliary structures because shifted attention can overemphasize visual features and obscure mixed-modality information.WINGS therefore uses a visual learner to extract visual information and a symmetrical textual learner to balance the auxiliary contributions.Source: §3.2 · pp. 5–6

3.3 Stable Training Recipe

• Training first fine-tunes the projector and visual learners for image-text alignment, then updates the LLM while jointly learning textual learners and attention-based routing.The vision encoder remains fixed throughout this staged recipe.Source: Figure 4 · p. 6, §3.3 · p. 6

3.2 Visual and Textual Learners Weave WINGS

• LoRRA learners use low-rank mappings and residual attention, preserving cross-attention structure while reducing computational demands.The low-rank matrices are formed from smaller factors, and zero initialization makes the learner output zero at training start.Source: §3.2 · p. 6

3.3 Stable Training Recipe

• Table 1 evaluates WINGS against eight baseline MLLMs across six domains and 20 datasets under the same training data and matched backbone settings.The comparison includes different LLMs, visual encoders, and full-parameter or LoRA fine-tuning strategies.Source: Table 1 · p. 7

4.1 Toward Comprehensive Text-only and Multimodal Performance

• WINGS reduces text-only forgetting while maintaining strong visual-question-answering performance under matched data and resource conditions.Multimodal fine-tuning causes forgetting on 12 of 16 text-only datasets, with decreases up to 9.70 on MMLU, 8.91 on CMMLU, and 13.33 on RACE-High; WINGS improves several affected datasets.Source: §4.1 · p. 8
• WINGS achieves state-of-the-art performance on 13 of 18 datasets while outperforming LLaVAv1.5 on text-only and multimodal tasks.Table 2 compares equal-scale and efficient MLLMs across 18 representative datasets.Source: §4.1 · p. 8, Table 2 · p. 8

4.2 Interleaved Image-Text (IIT) Benchmark

• The IIT benchmark samples six multi-turn configurations ranging from text-only conversations to visual questions following text-only queries.Configurations include (T), (T, T), (T, T, T), (T, T, V), (T, V), and (V), with (T, T, V) placing a visual question after two text-only queries.Source: §4 · p. 7, §4.2 · p. 9

4.3 Ablation Studies

• On IIT settings, WINGS surpasses LLaVAv1.5 and the same-backbone Qwen-SigLIP model, especially in text-centric dialogues.The benchmark covers contexts from text-rich to multimodal-rich question answering.Source: §4.3 · p. 9, Figure 5 · p. 9
• A higher projector learning rate combined with a lower rate for other components provides the best balance between text-only and multimodal abilities.Lower rates preserve text-only proficiency but weaken multimodal performance, whereas higher rates improve multimodal ability without preserving text-only performance as well.Source: §4.3 · p. 9, Figure 5 · p. 9
• Visual learners alone slightly preserve text-only ability but reduce multimodal performance, supporting the need for the full learner design.The ablation attributes the text-only preservation to minimizing disruption to the LLM.Source: §4.3 · p. 9, §4.1 · p. 9

• WINGS combines visual and textual learners with Low-Rank Residual Attention to alleviate text-only forgetting.The method first learns visual features, then co-trains visual and textual learners using routing based on shifted attention weights.Source: §5 · p. 9
• WINGS performs strongly on text-only, visual-question-answering, and IIT benchmarks while maintaining text-only performance with limited resources.The conclusion also reports further performance enhancement in well-resourced settings.Source: §5 · p. 9

• WINGSbase uses LLaVAv1.5-consistent data, whereas WINGSpro adds visual QA and text-only QA datasets; WINGS1.8B shares WINGSpro’s training set.Source: A Experimental Setups and… · p. 18
• The model structures employ Qwen1.5 [7] and SigLIP [134] as their foundations.Source: A Experimental Setups and… · p. 18
• The two-stage schedule uses learning rates of 1e−3 and 2e−6, with the projector set to 1e−5 in the second stage.Training uses batch size 32, AdamW, and a cosine schedule.Source: A Experimental Setups and… · p. 18
• WINGSbase and WINGSpro train for approximately 1.5 or 6 days on 8× A100 GPUs, while WINGS1.8B takes about 5 days on 4× A100 GPUs.Source: A Experimental Setups and… · p. 18

• Figure 6 compares parameter-efficient modules across IIT settings, including visual questions following different amounts of preceding text-only dialogue.Source: Figure 6 · p. 18
• Parameter-efficient methods such as LoRA and Prefix-tuning reduce text-only forgetting but curb cross-modal positive transfer.Source: B Additional Experimental Results · p. 18

• WINGS can integrate with any multimodal mixed-input MLLM as a universal plugin.Source: C Discussion · p. 18
• Competitive reuse among multiple expert groups may improve attention allocation without requiring MLP experts three orders of magnitude larger.Source: C Discussion · p. 18
• The discussion relates this minor attention update to variants of LoRA and identifies future exploration of MLLMs.Source: C Discussion · p. 18

• Restarting feature-alignment training to integrate visual learners incurs extra costs.Source: D Limitation & Broader Impact · p. 18
• Edge deployment is limited; WINGS1.8B offers a solution at the expense of performance.Source: D Limitation & Broader Impact · p. 19
• WINGS still requires some text-only data replay to enhance overall performance.Source: D Limitation & Broader Impact · p. 19
• Figure 7 tracks layer-wise attention proportions for image-before, image-itself, and image-after tokens from shallow to deep layers.Source: Figure 7 · p. 19
• The IIT benchmark represents dialogues as text-only and visual QA pairs, with sampled questions potentially sharing a category such as Sociology.Source: Figure 8 · p. 19