PaperlayerView exact revision on arXiv

Source-linked AI summary

Mind the Heads: Topological Representation Alignment for Multimodal LLMs

Davide Caffagni, Alberto Compagnoni, Federico Melis, Sara Sarto, Pier Luigi Dovesi, Mark Granroth-Wilding, Marcella Cornia, Lorenzo Baraldi

arXiv:2606.23885v1cs.CVcs.AIcs.CLcs.MM

TL;DR

MLLMs still struggle with foundational visual reasoning, while existing representation-alignment methods typically overlook the Transformer’s fine-grained structure by targeting a fixed language-backbone layer. HeRA aligns selected attention heads using a contrastive proxy for local topological similarity, and achieves a +3.6-point average improvement on demanding vision-centric benchmarks while mitigating visual hallucinations.

  • Problem

    MLLMs remain challenged by foundational visual reasoning, while existing representation-alignment methods typically target a fixed language-backbone representation without accounting for the model’s internal structure.

  • Method

    HeRA uses MKNN-guided selection and a contrastive objective to align individual attention heads with an external vision encoder’s local topological structure.

  • Results

    +3.6 points is HeRA’s average improvement on demanding Vision-Centric benchmarks, outperforming the next-best method’s +2.8 points.

  • Takeaways & Limitations

    HeRA improves challenging vision-centric performance while serving as a regularizer against visual hallucinations and over-reliance on linguistic priors.

  • Takeaways & Limitations

    The evaluation uses the computationally tractable LLaVA pipeline rather than newer frontier proprietary MLLMs.

Abstract

from arXiv · show

Representation alignment has emerged as an effective approach to improve Multimodal Large Language Models (MLLMs) by regularizing their internal representations toward those of an external vision encoder. However, existing methods typically align a fixed layer of the language backbone, overlooking the fine-grained structure of Transformer models. In this work, we propose Head-Wise Representation Alignment (HeRA), a method that enforces cross-modal alignment at the level of individual attention heads. Our approach is grounded in the Platonic Representation Hypothesis, focusing on preserving the topological structure of representations (i.e., their local neighborhood relationships) across modalities. Following the Mutual K-Nearest Neighbor (MKNN) alignment metric, we introduce a contrastive objective that acts as a differentiable proxy for matching local structures. HeRA applies this objective during multimodal training to specific attention heads in the LLM, selected by their alignment score according to the MKNN metric. Counterintuitively, we find that aligning the least aligned heads yields the largest gains. Extensive evaluations across multiple MLLMs and 18 benchmarks demonstrate that HeRA consistently improves performance on challenging vision-centric tasks and serves as an effective regularizer against visual hallucinations by naturally curbing the over-reliance on linguistic priors. Our code is publicly released.

1 Introduction

HeRA addresses the limitations of fixed-layer representation alignment in MLLMs by selecting individual attention heads using topological cross-modal consistency and MKNN diagnostics. Across 18 benchmarks under LLaVA [19], it consistently improves challenging vision-centric performance and regularizes against visual hallucinations [10] [39].

  • Problem and related work: Representation alignment regularizes MLLM language-model representations toward an external vision encoder through cross-modal distillation [2] [38] [46].Despite progress in MLLMs [1] [19] [34] [41], foundational visual reasoning remains limited, and existing methods commonly align a fixed backbone representation.
  • Method motivation: The approach is grounded in the Platonic Representation Hypothesis [9] [13], which posits that semantically similar inputs preserve local neighborhood structure across modalities.This frames representation alignment as preserving the local geometry, or topology, of representation spaces.
  • Method motivation: HeRA enforces cross-modal alignment at the level of individual attention heads, using pre-computed MKNN scores to select which heads to target.Unlike approaches that align a fixed, coarser layer [46], HeRA uses head-level selection to account for the language backbone’s internal structure.
  • Method motivation: HeRA targets the least aligned heads, which yields the largest gains by strengthening misaligned components while preserving already aligned structures.This selection is motivated by the complex interaction between alignment and the organization of pre-trained LLMs with strong language priors.
  • Results: HeRA improves performance consistently across 18 benchmarks [34] under the LLaVA [19] framework, especially on challenging vision-centric tasks, without sacrificing and often improving general visual question answering.The topological alignment also acts as a regularizer against visual hallucinations [10] [39].

2 Related Work

Prior work motivates local, rather than global, representation alignment across modalities and highlights vision-centric supervision as a way to improve MLLM visual understanding. Existing alignment methods generally target visual features from fixed language-backbone layers, while this work studies a new objective primarily on LLaVA and also on Qwen3 models.

  • The Platonic Representation Hypothesis: The Platonic Representation Hypothesis [13] proposes that models across architectures, modalities, and objectives converge toward structurally similar latent spaces, with local neighborhood relationships preserved across modalities [9].The cited concurrent work argues that structural consistency is local rather than global, so absolute global geometry may differ.
  • Vision-Centric Supervision in MLLMs: Vision-centric supervision methods align MLLM representations with a teacher vision encoder, but typically operate on visual features from a fixed, hard-coded language-backbone layer.The passage identifies this fixed-layer design as a common pattern in recent representation-alignment approaches, including JARVIS [2].
  • Implementation Context: This work studies a novel representation-alignment objective on the LLaVA [19] framework for computational tractability and additionally applies it to state-of-the-art Qwen3 [45] models.The broader MLLM research context includes massive datasets, post-training, stronger LLM backbones, and natively multimodal models [1] [41] [35].

3 Proposed Method

HeRA aligns multimodal representations with a frozen vision encoder’s local topology by applying a differentiable contrastive objective to selected Transformer attention heads. It combines this head-wise alignment loss with the standard language-modeling objective, targeting fine-grained interventions rather than a fixed language-model layer.

  • MKNN alignment metric: MKNN measures alignment through agreement between textual and visual k-nearest-neighbor sets, with higher scores indicating preserved local topological structure.Text representations use average-pooled language-model outputs, while visual representations use CLS embeddings from the vision encoder.
  • Contrastive topological alignment: The method trains multimodal representations to match the teacher vision encoder’s local neighborhoods by pulling target nearest neighbors closer and separating other batch samples.The multi-target InfoNCE objective is a differentiable proxy for the non-differentiable MKNN metric, with learnable temperature τ controlling distribution sharpness.
  • Head-wise alignment objective: HeRA applies contrastive alignment independently to selected attention-head representations, enabling atomic interventions that reduce conflicts between language modeling and representation alignment.Different heads specialize in different roles, so head-level control provides finer granularity than aligning a fixed intermediate layer.
  • Training objective: The final objective sums standard language modeling with the selected-head alignment loss, balanced by the fixed coefficient λ.The multimodal architecture combines an LLM, pretrained vision encoder, and projector, with visual embeddings concatenated to text embeddings before generation.
  • Head selection: The method boosts alignment of poorly aligned heads while preserving the strongest heads, as shown by the reported best-performing selection strategy.This selection behavior is presented as the method’s preferred way to allocate head-wise alignment.

4 Experiments

Across diverse LLMs, HeRA consistently preserves or improves core language, knowledge, and OCR abilities while delivering its strongest gains on vision-centric benchmarks and reducing visual hallucinations. Ablations show that contrastive alignment of the five least-aligned attention heads is most effective, while fixed-layer or excessive-head alignment is weaker.

  • Objective and Granularity: Contrastive alignment outperforms standard middle-layer cosine alignment, and selecting the five least-aligned heads yields larger gains than selecting the five most-aligned heads on both evaluated LLMs.The head ranking is based on MKNN alignment with the vision encoder; the supplied passage reports a striking advantage for the worst-5 strategy but truncates its example values.
  • Number of Heads to Align: Aligning five heads performs best, whereas aligning one or three gives modest gains and aligning ten causes regression, particularly for Qwen2.5-3B, suggesting excessive alignment conflicts with language modeling.After training, worst-5 heads markedly improve their vision-language alignment without reducing top-5 alignment, supporting the Platonic Representation Hypothesis [13].
  • Results on Cambrian Benchmarks: HeRA consistently preserves or improves General, Knowledge, and OCR performance while substantially improving Vision-Centric results across LLM architectures, including +2.3 points for Vicuna-7B, +3.4 points for Qwen2.5-14B, and a 58.9 Vision-Centric average for Qwen3-14B.These results are reported across the Cambrian benchmark suite and models spanning multiple architectural generations and parameter scales.
  • Results on Hallucination Benchmarks: HeRA reduces hallucination rates on CHAIR-MSCOCO and AMBER while improving AMBER accuracy and F1 and nearly all HallusionBench metrics, except Qwen3-4B on that benchmark.The improvements indicate stronger visual grounding even though hallucination mitigation is not explicitly optimized by the contrastive loss.
  • Comparison with Representation Alignment Strategies: Compared with fixed-layer strategies, HeRA’s targeted head-wise topological alignment is more effective; VIRAL is the only compared method that regresses against LLaVA.The comparison distinguishes HeRA’s local-structure objective from CMAR’s global pointwise CKA matching.
  • Teacher Vision Encoder: Using SigLIP2 itself as the teacher is mostly ineffective, consistent with concurrent work [46] [47] finding unsupervised vision encoders better representation teachers than language-supervised encoders.The teacher-encoder experiment uses Qwen3-8B with SigLIP2 as the primary vision encoder.

5 Conclusion

HeRA enhances MLLMs through topological representation alignment, using an MKNN-based contrastive proxy to align selected attention heads with an external vision encoder. Across multiple architectures and benchmarks, targeting the least aligned heads yields the most substantial gains.

  • HeRA enhances Multimodal Large Language Models through topological representation alignment guided by the Platonic Representation Hypothesis.
  • HeRA uses a contrastive proxy for the MKNN metric to align specific attention heads with an external vision encoder.
  • Targeting the least aligned attention heads yields the most substantial gains across evaluations spanning multiple architectures and benchmarks.

A Additional Implementation Details

The implementation follows LLaVA-1.5’s two-stage training recipe and constructs HeRA supervision from visual neighborhood structure, with stage-specific token pooling and learned temperature scaling.

  • Training Details: Training follows LLaVA-1.5 [19]: the projector is trained on 558k image-caption pairs before jointly optimizing the language model and projector on LLaVA-Instruct-665k.The first stage freezes the language model, and the reported optimizer, learning rate, and batch size match LLaVA-1.5 [19].
  • Contrastive Learning Details: Teacher [CLS] representations form a Gram matrix whose top-k visual neighbors receive uniform multi-positive contrastive targets, while all other samples receive zero probability.The targets are derived from pairwise dot products between teacher-encoder representations within each batch.
  • Contrastive Learning Details: Student representations are averaged across selected heads and stage-specific text tokens: all caption tokens in stage one, but only <ASSISTANT> tokens in stage two.The second-stage tokens are those contributing to the language-model loss.
  • Contrastive Learning Details: The temperature τ is learned in logarithmic scale and initialized to 0.07.
  • LLM Details and Selected Heads: Table 5 lists the publicly accessible LLM checkpoints and the selected attention heads, ordered by increasing MKNN alignment score.Head identifiers use LXHY notation, where X denotes the layer and Y the head index.

B Evaluation Benchmarks

The evaluation uses the Cambrian Evaluation Suite [34] across 18 benchmarks spanning general perception, knowledge reasoning, OCR, and visual abilities, alongside three hallucination-focused datasets. The hallucination evaluation measures object-, sentence-, cognition-, and coverage-level tendencies.

  • Cambrian Evaluation Suite: The Cambrian Evaluation Suite [34] covers 18 benchmarks organized into General, Knowledge, OCR, and Vision categories for evaluating diverse MLLM capabilities.The listed scope includes perception, knowledge reasoning, OCR and chart understanding, and core visual abilities.
  • Evaluation Analyses: Additional evaluation comparisons examine DINOv2 versus SigLIP2 vision encoders and the effects of applying LHeRA at different LLaVA training stages.These analyses are presented in Table 6 and Table 7, respectively.
  • Hallucination Datasets: Hallucinatory tendencies are evaluated on AMBER [39], CHAIR-MSCOCO [49], and HallusionBench [10].CHAIR-MSCOCO evaluates object- and sentence-level hallucination rates on descriptions for 500 MSCOCO [18] validation images.
  • Hallucination Datasets: The hallucination benchmarks capture cognition through overlap with human-hallucinated objects and coverage through an additional AMBER generative-task measure.The supplied passage defines these AMBER dimensions as Cog and Cover, respectively.

C Additional Experiments · C.1 Additional Ablation Studies and Results

The additional experiments examine HeRA across vision encoders, LLaVA training stages, language models, alignment strategies, and the full set of 18 VQA benchmarks. They further show that directly using DINOv2-L without representation alignment is ineffective as a vision encoder.

  • C.1 Additional Ablation Studies and Results: Directly plugging DINOv2-L into an MLLM without representation alignment is ineffective compared with the evaluated SigLIP2 alternative.Table 6 compares DINOv2-L and SigLIP2 as vision encoders, testing whether alignment can be omitted.
  • C.1 Additional Ablation Studies and Results: The appendix provides detailed VQA results for HeRA under the LLaVA training recipe across different language models.These comparisons are summarized in Table 8.
  • C.1 Additional Ablation Studies and Results: The appendix compares multiple representation alignment strategies for multimodal language models using detailed VQA results.The strategy comparison is summarized in Table 9.
  • C.1 Additional Ablation Studies and Results: HeRA is evaluated when applied to both training stages of the LLaVA recipe, whose multimodal inputs and objectives differ.The first stage uses aligned image-caption pairs, making it a suitable setting to examine representation alignment.
  • C.1 Additional Ablation Studies and Results: The full evaluation reports results across all 18 VQA benchmarks, extending beyond the vision-centric datasets emphasized in the main paper.The reported benchmark categories include General, Knowledge, and OCR, in addition to vision-centric VQA datasets.
  • C.1 Additional Ablation Studies and Results: The additional results cover RealWorldQA, MMVP, Blink, V*, and CVBench alongside General, Knowledge, and OCR benchmark categories.The main paper reported detailed scores for the listed vision-centric datasets while leaving category averages for the extended results.

C.2 Additional Analyses

Additional analyses show that HeRA’s gains come from targeting initially poorly aligned attention heads, whose local cross-modal topology it uniquely improves. This pattern persists in larger Qwen2.5 models, while feature-level and competing alignment methods largely fail to alter the Worst-5 heads.

  • Additional MKNN Head-Wise Analysis: Standard multimodal training largely preserves the base LLM’s relative head alignment: naturally aligned heads remain aligned, while poorly aligned heads remain poorly aligned.Across methods, Top-5 alignment generally increases, but most scores remain close to the unregularized LLaVA baseline.
  • Additional MKNN Head-Wise Analysis: HeRA applied to the Worst-5 heads substantially increases their MKNN alignment without sacrificing the Top-5 heads, whereas applying it to the Top-5 heads leaves the Worst-5 heads unchanged and yields suboptimal performance.Worst-5 and Top-5 heads are selected by pretraining MKNN alignment scores on Qwen2.5-3B; the selection remains predictive after standard multimodal training.
  • MKNN Alignment With Larger Qwen2.5 Models: Individual attention heads in Qwen2.5-7B and Qwen2.5-14B consistently show higher natural visual alignment than representations from any layer of the same model.The larger-model analyses compare both models against the DINOv2-L teacher using the MKNN metric, extending the earlier Qwen2.5 analysis.
  • MKNN Analysis with Different Representation Alignment Methods: HeRA is the only compared method that significantly increases cross-modal alignment for initially poorly aligned Worst-5 heads; CMAR and feature-matching methods produce no meaningful structural changes.CMAR targets cross-modal topology, whereas feature matching focuses on feature-level similarity and does not effectively modify local neighborhood structure.

C.3 Qualitative Results

HeRA produces more accurate and better-grounded answers than LLaVA and ROSS across General, Knowledge, OCR, and Vision-Centric categories, correcting perceptual errors in representative samples. Nevertheless, it can still misinterpret fine-grained visual details, including counts, materials, shapes, and spatial relationships in cluttered scenes.

  • Qualitative comparison: HeRA consistently delivers more accurate and better-grounded answers than LLaVA [19] and ROSS [38] across General, Knowledge, OCR, and Vision-Centric categories, correcting baseline perceptual errors.The comparison uses Qwen3-8B and SigLIP2.
  • Failure cases: HeRA occasionally misinterprets fine-grained visual details, including counting small instances, identifying ambiguous materials and shapes, and inferring precise spatial relationships in cluttered scenes.These failure cases remain despite the model's qualitative improvements.

D Limitations and Societal Impacts

The study primarily evaluates HeRA within the computationally tractable LLaVA pipeline, while an initial Qwen3-VL-4B study suggests promising gains on demanding vision-centric tasks. The authors report no direct societal harms specific to HeRA and associate it with improved visual grounding and fewer object hallucinations.

  • Limitations: HeRA is evaluated mainly with the LLaVA pipeline because its computational tractability enables extensive ablations and rigorous evaluations, despite newer proprietary MLLMs.This limits the architectural scope of the main evaluation.
  • Limitations: On Qwen3-VL-4B [1], fine-tuned on an 83k-sample Cambrian split from FineVision [42], HeRA shows promising results, especially on demanding vision-centric tasks.The study applies HeRA directly to a modern multimodal architecture as an initial exploration beyond LLaVA.
  • Societal Impacts: The authors identify no direct negative societal impacts specific to HeRA, arguing that improved visual grounding and reduced object hallucinations support more reliable, factual vision-language systems.This is presented as the expected societal impact of the representation-alignment technique.
  • Limitations: Qualitative comparisons span Cambrian categories, while separate examples document HeRA failure cases on VQA tasks.The supplied figures cover General, Knowledge, OCR, and Vision-Centric samples and VQA failures.
Loading 2606.23885v1…