Visual Representation Alignment for Multimodal Large Language Models

Heeji Yoon; Jaewoo Jung; Junwan Kim; Hyungyu Choi; Heeseong Shin; Sangbeom Lim; Honggyu An; Chaehyun Kim; Jisang Han; Donghyun Kim; Chanho Eom; Sunghwan Hong; Seungryong Kim

Source-linked AI summary

Visual Representation Alignment for Multimodal Large Language Models

Heeji Yoon, Jaewoo Jung, Junwan Kim, Hyungyu Choi, Heeseong Shin, Sangbeom Lim, Honggyu An, Chaehyun Kim, Jisang Han, Donghyun Kim, Chanho Eom, Sunghwan Hong, Seungryong Kim

arXiv:2509.07979v2Sep 9, 2025cs.CV

TL;DR

MLLMs remain limited on vision-centric tasks because text-only supervision can discard fine-grained visual information. VIRAL regularizes internal visual representations toward pretrained VFMs, and experiments show consistent improvements across standard multimodal benchmarks, averaging 9.4% over the baseline.

Problem
Text-only supervision provides only indirect guidance for visual tokens and can discard fine-grained information needed for vision-centric tasks.
Method
VIRAL aligns internal MLLM visual representations with features from pretrained vision foundation models.
Results
VIRAL yields consistent improvements across standard multimodal benchmarks, averaging 9.4% over the baseline.
Takeaways & Limitations
Representation alignment helps preserve fine-grained visual semantics and supports more accurate spatial reasoning and object grounding.
Takeaways & Limitations
Benefits are constrained by the representational capacity of the alignment target, with marginal or negative change on MMVP when encoder features underperform.

Abstract

from arXiv · show

Multimodal large language models (MLLMs) trained with visual instruction tuning have achieved strong performance across diverse tasks, yet they remain limited in vision-centric tasks such as object counting or spatial reasoning. We attribute this gap to the prevailing text-only supervision paradigm, which provides only indirect guidance for the visual pathway and often leads MLLMs to discard fine-grained visual details during training. In this paper, we present VIsual Representation ALignment (VIRAL), a simple yet effective regularization strategy that aligns the internal visual representations of MLLMs with those of pre-trained vision foundation models (VFMs). By explicitly enforcing this alignment, VIRAL enables the model not only to retain critical visual details from the input vision encoder but also to complement additional visual knowledge from VFMs, thereby enhancing its ability to reason over complex visual inputs. Our experiments demonstrate consistent improvements across all tasks on widely adopted multimodal benchmarks. Furthermore, we conduct comprehensive ablation studies to validate the key design choices underlying our framework. We believe this simple finding opens up an important direction for the effective integration of visual information in training MLLMs.

1 INTRODUCTION

MLLMs perform well broadly but remain limited on vision-centric tasks because text-only supervision under-supervises visual representations and can discard fine-grained information. VIRAL addresses this problem by aligning internal MLLM visual representations with vision foundation models, yielding consistent benchmark improvements.

Stronger vision encoders and projectors improve vision-centric abilities but are constrained in scalability and efficiency.VIRAL instead targets the training signal and internal visual representations.
Text-only visual instruction tuning indirectly supervises visual tokens, raising whether the prevailing setup adequately preserves visual information.Both the visual pathway’s substantial input contribution and its language-mediated supervision motivate revisiting the training paradigm.
Text-only supervision can discard visual attributes such as color, exact counts, and spatial layout that support fine-grained understanding.The paper contrasts efficient language alignment with loss of richer, structured vision-encoder representations.
Visual representations trained only with text rapidly diverge from input vision-encoder representations, while preserving alignment substantially improves fine-grained visual understanding.The paper calls this divergence visual representation misalignment.
VIRAL directly supervises the visual pathway by aligning MLLM representations with pretrained VFM features using cosine-similarity regularization.The alignment is intended to preserve encoder-provided details while adding visual knowledge from stronger VFMs.
VIRAL improves performance by an average of 9.4% over the baseline across standard multimodal benchmarks.The authors also report comprehensive ablations and analysis validating their design choices.

2 RELATED WORK

Related work examines MLLM information flow and ways to improve visual information, especially through encoders and token efficiency. VIRAL complements these directions by supervising intermediate visual representations where fine-grained semantics emerge.

Prior work identifies a hierarchy in which early layers aggregate global context, middle layers capture spatial features, and later layers integrate multimodal information.Middle layers are particularly associated with visual understanding, grounding, and propagation of insufficient visual information.
Most prior efforts improve visual information at the input stage through stronger or multiple vision encoders or by reducing visual-token overhead.These directions primarily modify frozen-encoder inputs or improve efficiency.
VIRAL complements input-stage approaches by aligning intermediate MLLM features with pretrained VFM embeddings to preserve semantically meaningful visual content.The method focuses particularly on middle layers where fine-grained semantics emerge.

3 PRELIMINARIES

MLLMs combine a pretrained language model and vision encoder through a vision-language projector, then generate text conditioned on visual and textual representations. Their two training stages share a language-modeling objective but differ in which parameters are updated.

An MLLM uses a vision encoder to extract patch features, a projector to map them into the language-model embedding space, and token embeddings for text.The resulting visual and textual embeddings are combined for multimodal processing.
During inference, the language model autoregressively generates text conditioned on visual representations, the prompt, and previously generated tokens.
Vision-language pretraining optimizes only projector parameters, whereas visual instruction tuning jointly optimizes the projector and language-model parameters.Both stages use the same language-modeling objective despite their different parameter updates.
The shared training objective maximizes the log-likelihood of text outputs.

4 METHODOLOGY

The method investigates whether preserving visual representations improves multimodal understanding, then introduces VIRAL to align intermediate MLLM visual features with encoder or stronger VFM features. Alignment-based regularization outperforms residual reinjection in representation similarity and multimodal benchmark performance, while gains are limited when encoder features are weak.

4.2 IS PRESERVING VISUAL INFORMATION BENEFICIAL?: Visual representations increasingly diverge from encoder features after early layers, although middle layers attenuate this decline and can show slight alignment increases.CKNNA measures similarity between internal MLLM visual representations and original encoder features; the middle-layer pattern motivates applying preservation at intermediate depths.
4.2 IS PRESERVING VISUAL INFORMATION BENEFICIAL?: Reinjecting projected visual features at layer 16 preserves higher encoder-feature alignment and generally improves standard benchmark performance over baseline training.The model uses the original text loss only, with layer 16 selected in a 32-layer LLaVA model based on the observed middle-layer behavior.
4.3 VISUAL REPRESENTATION ALIGNMENT FOR MLLMS: Alignment loss outperforms residual reinjection in both CKNNA similarity and multimodal benchmark performance.The comparison indicates that explicit feature-level consistency provides stronger fine-grained semantic preservation than a residual connection alone.
4.3 VISUAL REPRESENTATION ALIGNMENT FOR MLLMS: Encoder-feature preservation has a limitation on MMVP, where performance shows only marginal improvement or a slight drop because CLIP-like features underperform.This motivates replacing the original encoder target with more informative VFM features, whose inductive biases and representational capacity can better support visual semantics.
4.3 VISUAL REPRESENTATION ALIGNMENT FOR MLLMS: VIRAL aligns intermediate MLLM visual representations with stronger pretrained VFM features, using explicit regularization to preserve richer visual semantics.A learnable projection maps MLLM visual representations into the VFM feature space, and the alignment objective supplements language-modeling supervision.

5 EXPERIMENTS

Experiments evaluate VIRAL across multimodal benchmarks, ablations, attention analyses, robustness tests, qualitative examples, and training efficiency. Across identical settings, VIRAL consistently improves performance, especially on fine-grained vision-centric tasks, while preserving general multimodal capability.

5.1 EXPERIMENTAL SETTINGS: VIRAL consistently outperforms the baseline across vision-centric, hallucination, and general multimodal benchmarks, with the largest gains on fine-grained vision-centric tasks.The comparison uses identical training settings and aligns intermediate MLLM features with VFM targets.
5.2 MAIN RESULTS: Results remain improved across SigLIPv2, larger Vicuna-1.5-13B, and Qwen2.5-7B configurations, indicating applicability across vision encoders, model scales, and language backbones.The authors attribute the broader pattern to regularizing intermediate visual representations.
5.3 ABLATION STUDY: Ablations show that target VFM choice and alignment layer matter: stronger VFMs improve spatial-task performance, while the 16th layer of the 32-layer model performs consistently better across benchmarks.The target-layer study evaluates every fourth layer, and the VFM study compares DINOv2, CLIP, and other stronger foundation models.
5.2 MAIN RESULTS: VIRAL produces more focused, semantically aligned attention than LLaVA-1.5-7B, as measured through text-to-image attention maps and spatial entropy across layers and heads.The analysis describes baseline attention as dispersed and VIRAL attention as more discriminative and contextually focused.
5.5 ROBUSTNESS ANALYSIS: VIRAL improves sensitivity to spatial relationships under visual-token permutation tests on CV-Bench2D, directly probing whether spatial cues are retained.The robustness evaluation counts correct predictions on 788 spatial reasoning tasks.
5.6 QUALITATIVE RESULTS: Qualitative analyses show VIRAL answering counting and spatial-position questions correctly where LLaVA-1.5-7B often fails, alongside more structured and semantically coherent PCA embeddings.The PCA comparison uses visual representations from the 16th layer.
5.7 TRAINING EFFICIENCY: Across three CLIP-based model configurations, VIRAL converges faster and surpasses baseline performance within 3K of 5.2K training steps.Training efficiency is measured by averaging CV-Bench2D and MMVP accuracy every 1K steps.

6 CONCLUSION

VIRAL aligns internal MLLM visual representations with pretrained vision foundation models. This regularization preserves fine-grained visual semantics and supports more accurate spatial reasoning and object grounding.

6 CONCLUSION: VIRAL aligns internal MLLM visual representations with pretrained vision foundation models to preserve fine-grained visual semantics.The approach is presented as a simple regularization strategy for training MLLMs.
6 CONCLUSION: The preserved visual information enables more accurate spatial reasoning and object grounding.

A ADDITIONAL IMPLEMENTATION DETAILS

The implementation evaluates VIRAL with diverse vision foundation models, cosine alignment, and benchmark-specific protocols. Additional analyses test how visual-token order affects visual understanding.

Vision foundation models: VIRAL uses DINOv2, CLIP, Depth Anything v2, RADIO-v2.5, and SAM as pretrained vision foundation models for representation supervision.The models cover patch sizes 14 and 16 and are configured to match LLaVA’s 576 visual tokens.
Loss function and weighting: The alignment loss uses cosine similarity and is balanced with language modeling through a default weight of λ = 0.5.The cosine similarity compares the aligned representations while language modeling remains part of the training objective.
Benchmark settings: Evaluation covers CV-Bench2D, MMVP, What’s Up, POPE, MME, and MM-Star using task-specific reporting protocols.CV-Bench reports overall 2D accuracy, MMVP averages pair accuracy over 10 runs, and 3D CV-Bench tasks are outside scope.
Patch permutation: Randomly permuting visual tokens makes image attributes difficult to interpret, testing whether MLLMs use the visual information originally available.The experiment permutes vision-encoder features before the projector and language model.

B EXTENDED EXPLORATION OF THE PILOT STUDY

The extended pilot study compares feature re-injection and post-projection alignment as alternatives for preserving visual information. Post-projection alignment improves over baseline but remains weaker than using raw encoder features, whereas direct residual re-injection generally harms performance.

Extended pilot study: The pilot exploration compares pre-projection residual connections with the proposed post-projection visual representation alignment.Both strategies target visual information loss within the language model through different feature pathways.
Residual connection with pre-projection features: Direct residual re-injection of raw pre-projection features generally performs worse than baseline because the features are insufficiently aligned with language representations.A lightweight adapter provides dimensional compatibility, but the residual connection disrupts intermediate vision–language alignment.
Visual representation alignment with post-projection features: Post-projection alignment generally improves vision-centric benchmark performance over baseline but underperforms leveraging raw vision-encoder features.The comparison is reported in Table 4-(e), with the authors attributing the gap to weaker visual-information preservation after projection.

C ADDITIONAL ABLATION STUDIES

The ablations examine target-layer placement and alternative feature-alignment objectives. They identify a single intermediate target layer as especially effective and compare direct alignment with relation-based alignment.

Number of target layers: Alignment applied solely at the 16th layer performs best among the evaluated 15–17 and 14–18 multi-layer target ranges.The result supports targeting a specific visual-processing pathway rather than uniformly aligning multiple layers.
Alignment objectives: The study compares direct visual representation alignment with a feature-relation objective based on mean squared error between self-similarity matrices.The alternative objective aligns relations among VFM features and transformed intermediate representations rather than directly matching representations.

D COMPARISON WITH OTHER TRAINING OBJECTIVES

The comparison with reconstructive training objectives separates the effects of supervision location from objective choice. VIRAL outperforms both ROSS variants across benchmarks, with the authors attributing its advantage to intermediate-layer supervision and semantically rich VFM alignment.

Comparison with reconstructive objective: ROSS applied at an intermediate layer outperforms the default final-hidden-state version, particularly on vision-centric benchmarks.Both ROSS variants improve over baseline, indicating that supervising visual pathways contributes to performance.
Comparison with reconstructive objective: VIRAL significantly outperforms both ROSS variants across all benchmarks.The authors distinguish VIRAL’s direct alignment with pretrained visual representations from ROSS’s reconstruction-based objectives.
Comparison with reconstructive objective: The authors attribute VIRAL’s advantage to two choices: supervision at critical intermediate layers and semantically rich VFM signals instead of reconstruction objectives.Together, these choices are presented as enabling more effective visual representation learning for multimodal understanding.

E APPLICABILITY OF VIRAL TO OTHER MLLM ARCHITECTURES

VIRAL’s motivation remains relevant for tiled-image MLLMs: visual alignment declines across LLaVA-NeXT layers, supporting application of the method to this architecture.

Applicability to tiled-image MLLMs: LLaVA-NeXT preserves original image resolution by independently encoding high-resolution tile grids to capture fine-grained visual details.
Applicability to tiled-image MLLMs: Alignment scores decline across LLaVA-NeXT layers between input visual features and layer-wise representations, mirroring the pattern observed in Figure 2(d).The scores are measured with CKNNA and averaged across representations from tiled image splits.
Applicability to tiled-image MLLMs: The observed misalignment in tiled processing suggests that VIRAL’s core strategy for mitigating vision-information loss can extend to LLaVA-NeXT.

F ADDITIONAL VISUALIZATIONS AND RESULTS

Additional visualizations show that VIRAL produces more structured internal representations, distills organization from vision foundation models, and focuses attention more accurately on relevant image regions. The method’s applicability is constrained by architectures that disrupt spatial grids and by token-pruning approaches that exploit redundancy.

Layer-wise internal representations: VIRAL produces more semantically coherent and structured internal visual representations than LLaVA-1.5-7B, especially in middle and later layers.Figure 10 compares layer-wise PCA visualizations across intermediate representations.
VFM-aligned representations: Aligning with CLIP, DINOv2, SAM, DAv2, or RADIO distills the more semantically structured organization of VFM features into the MLLM’s internal representations.Figure 11 visualizes representations aligned with these different VFMs.
Cross-attention behavior: VIRAL’s cross-attention maps focus more accurately and locally on regions relevant to the multimodal context than the baseline.This qualitative observation supports the attention analysis associated with spatial entropy results.
Limitations: VIRAL is less directly applicable to Resampler or Q-Former architectures because they disrupt the spatial grid required for representation alignment.Effective alignment also requires a resolution-adjustable grid matching the MLLM’s expected number of visual tokens.
Limitations: Token-pruning methods that exploit visual-token redundancy may yield reduced benefits when combined with VIRAL because alignment strengthens each token’s semantic utility and inter-token relationships.