PaperlayerView exact revision on arXiv

Source-linked AI summary

Lost in Embeddings: Information Loss in Vision-Language Models

Wenyan Li, Raphael Tang, Chengzu Li, Caiqi Zhang, Ivan Vulić, Anders Søgaard

arXiv:2509.11986v1cs.CVcs.CL

TL;DR

The extent to which VLM connectors preserve visual information and affect model performance remains under-investigated. The paper evaluates projection-related changes using neighborhood overlap and patch-level embedding reconstruction. It reports 40–60% divergence in nearest-neighbor rankings and links patch-level reconstruction loss to captioning performance and visually grounded question-answering failures.

  • Problem

    How well VLM connectors preserve visual information in latent representations, and how that information loss relates to downstream performance, has received limited investigation.

    Source: §1 · p. 1

  • Method

    The paper measures changes in image-embedding neighborhoods before and after projection and reconstructs original visual embeddings from projected embeddings to localize patch-level loss.

    Source: §1 · p. 2

  • Results

    Nearest-neighbor rankings diverge by 40–60%, while patch-level reconstruction loss negatively impacts captioning and explains failures in fine-grained visually grounded question answering.

    Source: §9 · p. 9

  • Takeaways & Limitations

    Patch-level reconstruction visualizes local information loss and offers interpretable explanations of model behavior on visually grounded tasks.

    Source: §9 · p. 9

    The findings may be specific to the connector-based VLMs analyzed and may not generalize to architectures using cross-attention for modality fusion.

    Source: Limitations · p. 10

Abstract

from arXiv · show

Vision--language models (VLMs) often process visual inputs through a pretrained vision encoder, followed by a projection into the language model's embedding space via a connector component. While crucial for modality fusion, the potential information loss induced by this projection step and its direct impact on model capabilities remain understudied. We introduce two complementary approaches to examine and quantify this loss by analyzing the latent representation space. First, we evaluate semantic information preservation by analyzing changes in k-nearest neighbor relationships between image representations, before and after projection. Second, we directly measure information loss by reconstructing visual embeddings from the projected representation, localizing loss at an image patch level. Experiments reveal that connectors substantially distort the local geometry of visual representations, with k-nearest neighbors diverging by 40--60\% post-projection, correlating with degradation in retrieval performance. The patch-level embedding reconstruction provides interpretable insights for model behavior on visually grounded question-answering tasks, finding that areas of high information loss reliably predict instances where models struggle.

1 Introduction

The paper examines how connector projections may degrade visual information and affect downstream performance, an area that has received limited investigation. It proposes geometric and patch-level analyses to quantify these changes.

2 VLMs and Connectors

  • VLMs combine pretrained vision and language models, using connectors such as MLPs or attention-based modules to project visual representations into language-compatible embedding sequences.Connectors are also called projectors.Source: §1 · p. 1

1 Introduction

  • Limited investigation has quantified how well connectors preserve visual information, while projection may lose critical details that constrain language-model reasoning.The paper treats both unrecoverable information and changes in local geometry as possible forms of information loss.Source: §1 · p. 1
  • The paper introduces a framework that measures connector-induced information change through neighborhood overlap and reconstruction of original visual embeddings from projected embeddings.The reconstruction localizes high-loss regions at the image-patch level.Source: §1 · p. 2

2.1 Formalizing Encoders and Connectors

  • The formalization treats a connector as mapping visual embeddings into text-compatible dimensions and defines information loss over the connector projection.The neighborhood-overlap example compares an image’s nearest neighbors before and after projection.Source: §2.1 · p. 3, Figure 2 · p. 3

3 Quantifying Information Loss

The section quantifies projection-related information loss through neighborhood preservation and patch-level reconstruction, complementing geometric assessment with spatially localized distortion.

3 Quantifying Information Loss

  • Two complementary methods measure information loss: k-nearest-neighbor overlap tracks structural preservation, while embedding reconstruction measures patch-level distortion.The overlap compares image neighborhoods before and after projection; reconstruction uses projected representations to recover original vision embeddings.Source: §3 · p. 3
  • KNOR measures how well image-embedding k-NN relationships are preserved before and after connector projection.Lower average overlap indicates greater geometric information loss; higher overlap indicates more faithful retention.Source: §3.1 · pp. 3–4

3.2 Embedding Reconstruction

  • KNOR captures structural loss but cannot identify fine-grained visual-feature loss at the patch level.The reconstruction approach addresses this by attempting to recover original vision embeddings from projected representations.Source: §3.2 · p. 4
  • A reconstruction model maps connector outputs back to original vision embeddings, minimizing loss between the reconstructed and original embeddings.The model is trained on projected representations; per-patch loss is squared Euclidean distance between the original and reconstructed embedding.Source: §3.2 · p. 4
  • Patch-wise reconstruction loss reveals the spatial distribution of information loss across an image.The total loss sums the patch-wise losses across patches and images.Source: §3.2 · p. 4

4 Experimental Setup

The experiments assess three open-weight connector-based VLMs using two information-loss measures across six datasets and task types. Reconstruction models are tailored to connector designs and trained on COCO.

4 Experimental Setup

  • The evaluation applies both information-loss measures to three open-weight connector-based VLMs across six datasets spanning question answering, captioning, and retrieval.The experiments assume greater structural loss reduces neighborhood overlap and greater patch-wise loss increases reconstruction error.Source: §4 · p. 4

4.2 Evaluation Datasets

  • Six datasets cover complementary visual-understanding tasks, including grounded question answering, fine-grained bird retrieval, and image captioning.The evaluation includes VizWiz Grounding VQA, CUB-200-2011, Flickr30k, and the COCO Karpathy test set.Source: §4 · p. 4, §4.2 · p. 4

4.1 Pretrained VLMs

  • LLaVA preserves patch count with a two-layer MLP, Idefics2 uses a perceiver resampler, and Qwen2.5-VL merges each four neighboring patches.The evaluated variants are LLaVA and Qwen2.5-VL 7B-instruct and Idefics2-8B-instruct.Source: §4.1 · p. 4

4.3 Embedding Reconstruction Models

  • Overlap ratios below 0.62 for all models across three datasets and k=10, 50, and 100 indicate poorly preserved neighbor rankings.The datasets are SeedBench validation, a 10,000-sample VQAv2 validation subset, and VizWiz grounding VQA validation.Source: Figure 3 · p. 5
  • Reconstruction models use larger capacity than connectors; LLaVA uses an MLP, while sequence-compressing connectors use transformer-based models.Models train on COCO 2017 for 30 epochs with early stopping, a learning rate of 1e−4, dropout 0.1, and batch size 128.Source: §4.3 · p. 5

5 Neighbor Rankings and Structural Information are Not Preserved

The section measures how connector projection changes image-neighbor structure and tests whether those changes correspond to retrieval performance. Neighborhood overlap is low across models, but retrieval outcomes differ, including improved post-projection retrieval for Qwen2.5-VL.

5.1 Low Overlap Ratio for All Models

  • About 50% neighborhood overlap for LLaVA and Idefics-2; LLaVA reaches 61.6% at k=100.The overlap ratios average across unique images in the evaluation datasets.Source: §5.1 · p. 5
  • For Qwen2.5-VL, post-projection neighbors are more semantically similar despite low overlap, while substantial neighbor reordering appears across models.The proposed explanation is continued vision-encoder training and patch merging, but the authors frame it as a suspicion.Source: Figure 4 · p. 6, §5.1 · p. 6

5.2 Image Retrieval Evaluation

  • 41.4% (LLaVA) and 18.8% (Idefics2) R@5 degradation after projection contrasts with improved Qwen2.5-VL retrieval.The comparison is zero-shot retrieval on CUB using L2 distance.Source: §5.2 · p. 6, Table 2 · p. 6
  • R@5 and 100-neighbor overlap correlate positively per sample: about 0.1 for Qwen2.5-VL and LLaVA, and 0.3 for Idefics2.All reported correlations have p-values below 1e−3.Source: §5.2 · pp. 6–7, Table 2 · p. 6

6 Reconstruction and Model Behavior

The authors reconstruct original patch embeddings from projected representations to measure and locate information loss. Across captioning and grounded question answering, loss-performance associations vary by model and, for some models, by patch relevance.

6 Reconstruction and Model Behavior

  • Patch-level reconstruction compares original visual embeddings with embeddings reconstructed from their projected counterparts, with greater reconstruction loss indicating harder recovery.The reconstruction models are trained on paired pre- and post-projection embeddings from COCO 2017 training images.Source: §6 · p. 7, §6.1 · p. 7

6.1 Reconstruction Loss Impacts Captioning

  • For LLaVA and Idefics2, higher per-image reconstruction loss corresponds to worse captioning CIDEr; Qwen-VL shows no obvious per-image correlation.The reported negative correlations for LLaVA and Idefics2 have p-values below 1e−5.Source: §6.1 · p. 7

6.2 Loss at Patch-level Visual Features Explains Question Answering Behaviors

  • For VizWiz grounding VQA, relevant-patch loss negatively correlates with LLaVA accuracy, while Idefics2 accuracy declines with loss in any patches; Qwen-2.5-VL has no significant correlation.A high-loss patch in the example containing the number “8” suggests loss of detail needed to identify it.Source: §6.2 · p. 8, Figure 5 · p. 8

7 Analysis

The analysis uses Procrustes alignment as a structural-fidelity baseline and tests reconstruction-model size and architecture. The reported alignment errors are high, while a 27M-parameter MLP suffices for LLaVA reconstruction.

Procrustes analysis

  • Procrustes analysis aligns mean-pooled pre- and post-projection embeddings after PCA brings them to the same dimension.An orthogonal transformation is learned to minimize Euclidean distance between the embedding sets.Source: §7 · pp. 8–9
  • 16.62 for LLaVA and 4.41 for Qwen2.5-VL are the reported Procrustes alignment errors, indicating difficulty preserving geometry with rigid transformations.The authors contrast this linear baseline with lower errors from their nonlinear reconstruction approach.Source: §7 · p. 9, Table 5 · p. 9

Ablation on Reconstruction Model Size and Structure

  • The 27M three-layer MLP suffices for reconstructing LLaVA visual embeddings; larger models do not improve validation loss.The compared alternatives are a 39M five-layer MLP and a 40M Transformer.Source: §7 · p. 9

8 Related Work

Prior work studies contrastive-model representation limitations and connector effects on downstream performance, but the paper identifies a remaining question about fine-grained visual information loss in connector-based models.

8 Related Work

  • Prior studies examine modality gaps in contrastive-based VLMs and show that connector choice affects downstream performance.Other work probes whether connector-based latent spaces retain information needed for classification.Source: §8 · p. 9
  • The unresolved issue is whether connectors lose fine-grained visual information needed for grounding and question answering.Source: §8 · p. 9
  • The paper claims to be the first to quantify connector information loss from the representation perspective and identify where and what visual information is lost.Source: §8 · p. 9

9 Conclusion and Future Work

The study finds substantial connector-related shifts in visual representation geometry and patch-level losses associated with weaker captioning and fine-grained visual question answering. It proposes connector properties and future directions while noting important scope and measurement limitations.

9 Conclusion and Future Work

  • Nearest-neighbor rankings diverge by 40–60%, and patch-level reconstruction loss is associated with weaker captioning and fine-grained visually grounded question answering.The patch-level analysis also provides visual explanations of model behavior.Source: §9 · p. 9
  • Effective connectors should preserve or improve image semantics and retain visual information relevant to text context; embedding reconstruction loss could be explored as pretraining regularization.The authors also propose exploring dynamic projection layers and better visual feature selection.Source: §9 · p. 9

Ethics Statement

  • The authors foresee no ethical concerns and state that the study provides no artifacts usable in real-world scenarios.Source: Ethics Statement · p. 10

Limitations

  • Findings may not generalize beyond analyzed connector-based VLMs in the 7B–8B range, and reconstruction cannot isolate connector loss from reconstruction-model learning limits.Pixel-level reconstruction results were inconclusive, and the metrics lack a formal theoretical characterization.Source: Limitations · p. 10

A Connectors in Autoregressive Vision-Language Models

The appendix describes the tested connector architectures and reports auxiliary retrieval, neighborhood-overlap, and reconstruction evaluations. These results include method robustness checks and a reconstruction-fidelity comparison across models.

A Connectors in Autoregressive Vision-Language Models

  • Idefics2 uses a perceiver resampler, LLaVA a two-layer MLP, and Qwen2.5-VL a patch merger that combines neighboring patches before projection.LLaVA preserves patch count, while Qwen2.5-VL reduces the image-feature sequence length.Source: A Connectors in Autoregressive… · p. 12

B Ablation on Index Method for k-NN Overlap Ratio

C.1 CUB image retrieval performance

  • CUB retrieval performance is consistent across L2 distance and inner-product similarity measures.The reported comparison covers both similarity measures, without supplied metric values.Source: §C.1 · p. 12

C.2 Reconstruction loss on VQA datasets

  • LLaVA’s projected embeddings have the highest reconstruction fidelity among tested models on VizWiz, SeedBench, and FoodieQA.Overall reconstruction loss reflects the difficulty of recovering information encoded in visual representations; results are consistent with COCO and Flickr30k.Source: §C.2 · p. 12

D Visualization

The visualizations examine embedding alignment, patch-level loss examples, and pre- versus post-projection neighborhood and retrieval results. The reported alignment visualization shows that a learned linear transformation does not align LLaVA’s embeddings well.

D.1 Visualization for Procrustes Analysis

  • Table 9 reports reconstruction loss on VizWiz, SeedBench, and FoodieQA, with LLaVA showing the lowest reconstruction error among tested models.The caption says this result is consistent with observations on COCO and Flickr30k.Source: Table 9 · p. 13
  • A learned linear transformation does not align LLaVA’s pre- and post-projection embeddings well in the Figure 6 visualization.Source: §D.1 · pp. 12–13, Figure 6 · p. 13

D.2 Patch-level Loss Visualization for Vizwiz Grounding VQA

  • Additional high-reconstruction-loss patches are shown in examples of model failures on questions requiring recognition of text in objects.Source: §D.2 · p. 13

D.3 Visualization of Neighborhood Reordering

  • Figures 10 and 11 compare pre- and post-projection neighbor searches and CUB retrieval; Table 8 reports zero-shot CUB recall using L2 distance and inner product.Source: §D.3 · p. 13, Table 8 · p. 13

E Image Reconstruction with Different Embeddings

The appendix explores reconstructing images from visual representations using a fine-tuned VAE decoder and an alignment module. For LLaVA, pixel-wise reconstruction loss is lower before projection, although reconstructed contours are similar.

E Image Reconstruction with Different Embeddings

Related papers

Loading 2509.11986v1…