PaperlayerView exact revision on arXiv

Source-linked AI summary

CoME-VL: Scaling Complementary Multi-Encoder Vision-Language Learning

Ankan Deria, Komal Kumar, Xilin He, Imran Razzak, Hisham Cholakkal, Fahad Shahbaz Khan, Salman Khan

arXiv:2604.03231v1cs.CV

TL;DR

Single-encoder CLIP-style VLMs can miss fine-grained visual cues, while combining encoders risks redundancy and higher token costs. CoME-VL fuses SigLIP and DINO using entropy-guided layer selection, orthogonality-regularized mixing, and RoPE-enhanced cross-attention. It consistently improves visual understanding and grounding over single-encoder baselines, with complementary layer ranges contributing to performance.

  • Problem

    Single CLIP-style visual representations can underemphasize fine-grained cues, while naive multi-encoder fusion can add redundant features and visual-token cost.

  • Method

    CoME-VL combines SigLIP and DINO through entropy-guided layer selection, orthogonality-regularized fusion, and RoPE-enhanced cross-attention.

  • Results

    CoME-VL consistently improves grounding and visual-understanding performance over strong baselines, with the best results combining DINO late-layer features and SigLIP features across layers.

  • Takeaways & Limitations

    SigLIP mainly contributes semantic features, DINO richer grounding features, and their fusion improves performance without adding much burden to the LLM.

  • Takeaways & Limitations

    CoME-VL increases average inference time from 1.26s to 1.52s per sample compared with the Molmo baseline.

Abstract

from arXiv · show

Recent vision-language models (VLMs) typically rely on a single vision encoder trained with contrastive image-text objectives, such as CLIP-style pretraining. While contrastive encoders are effective for cross-modal alignment and retrieval, self-supervised visual encoders often capture richer dense semantics and exhibit stronger robustness on recognition and understanding tasks. In this work, we investigate how to scale the fusion of these complementary visual representations for vision-language modeling. We propose CoME-VL: Complementary Multi-Encoder Vision-Language, a modular fusion framework that integrates a contrastively trained vision encoder with a self-supervised DINO encoder. Our approach performs representation-level fusion by (i) entropy-guided multi-layer aggregation with orthogonality-constrained projections to reduce redundancy, and (ii) RoPE-enhanced cross-attention to align heterogeneous token grids and produce compact fused visual tokens. The fused tokens can be injected into a decoder-only LLM with minimal changes to standard VLM pipelines. Extensive experiments across diverse vision-language benchmarks demonstrate that CoME-VL consistently outperforms single-encoder baselines. In particular, we observe an average improvement of 4.9% on visual understanding tasks and 5.4% on grounding tasks. Our method achieves state-of-the-art performance on RefCOCO for detection while improving over the baseline by a large margin. Finally, we conduct ablation studies on layer merging, non-redundant feature mixing, and fusion capacity to evaluate how complementary contrastive and self-supervised signals affect VLM performance.

1 Introduction

The paper argues that single CLIP-style visual representations can miss fine-grained cues and that multi-encoder fusion must control redundancy and token cost. CoME-VL combines complementary encoders through layer selection, orthogonality-regularized fusion, and efficient cross-attention.

  • Single CLIP-style representations can underemphasize precise localization, color attributes, and subtle spatial relationships needed for fine-grained multimodal understanding.
  • Naive multi-encoder fusion can introduce redundant features and substantially increase the visual-token burden processed by the LLM.
  • CoME-VL combines SigLIP and DINO to capture complementary semantic and grounding information for multimodal learning.
  • The framework uses entropy-guided layer selection, orthogonality-regularized fusion, and RoPE-enhanced cross-attention to align and merge features without directly concatenating visual patches.
  • CoME-VL consistently improves grounding and visual-understanding performance over strong baselines while maintaining efficient inference.

2 Related Work

Related work develops multimodal systems by connecting pretrained visual representations with language models, while highlighting limitations of single-encoder approaches for fine-grained grounding. Table 1 positions CoME-VL as a multi-encoder alternative on PixMo counting and pointing tasks.

  • Vision-language systems commonly align pretrained visual representations with large language backbones to support multimodal understanding and generation.
  • Table 1 compares recent models on PixMo counting and pointing, showing limited grounding for single-encoder CLIP/SigLIP models and improved accuracy for CoME-VL.
  • Many multimodal systems rely on a single CLIP-style visual representation, which can underrepresent localized cues.

3 Preliminary Analysis

Preliminary analyses identify complementary spatial and semantic properties across SigLIP2 and DINOv3, while motivating fusion that avoids redundancy, token inflation, and spatial misalignment.

  • Single-Encoder Architectures: Single-encoder VLMs show systematic failures on fine-grained grounding tasks such as counting and pointing.
  • Complementary Encoders: SigLIP2 supplies semantic alignment while DINOv3 contributes fine-grained spatial cues, motivating the dual-encoder design.
  • Spatial Entropy Analysis: Spatial entropy decreases with depth, while SigLIP2 remains more object-centric and DINOv3 increasingly focuses on semantically discriminative regions.
  • Semantic Feature Analysis: Figure 2 compares four representative layers per encoder using top-30% spatial-attention masks and attention-rollout visualizations for selective layers.
  • Redundancy in Multi-layer Fusion: Naive multi-layer fusion wastes representational capacity because neighboring transformer layers encode highly similar information.
  • Fusion and Alignment: Concatenating visual tokens increases LLM processing cost, while mismatched encoder grids make naive cross-attention spatially unreliable for precise grounding.

4 Methodology

CoME-VL fuses hierarchical representations from complementary SigLIP2 and DINOv3 encoders, reducing redundancy while aligning heterogeneous token grids before injecting compact visual tokens into a decoder-only LLM.

  • Multi-layer Visual Representations: The framework uses SigLIP2 for semantic alignment and DINOv3 for fine-grained spatial cues, extracting hierarchical features across encoder depths.SigLIP2 uses contrastive vision-language pretraining, whereas DINOv3 captures self-supervised structural information.
  • Multi-layer Visual Representations: Entropy-guided analysis defines encoder-specific candidate layer sets and supports adaptive soft aggregation across depth.The aggregation combines broader contextual semantics with finer spatial cues.
  • Orthogonality-Regularized Multi-layer Mixing: Orthogonal Layers project selected layer outputs before aggregation, preserving feature scale while encouraging complementary rather than redundant mixing.For square matrices, the projections use skew-symmetric parameterizations through matrix exponential or Cayley transforms.
  • RoPE-Enhanced Alignment: RoPE-enhanced cross-attention aligns SigLIP2 queries with DINOv3 keys and values across heterogeneous spatial grids.A gated residual connection combines the cross-attention output with SigLIP2 features, with the gate initialized to zero.
  • RoPE-Enhanced Alignment: The fused visual tokens are projected into a decoder-only LLM without increasing its context length, supporting generation and grounding with compact inputs.The connector keeps the output token count fixed at the SigLIP token count, unlike direct concatenation.

5 Experiments

CoME-VL is evaluated on PixMo and RefCOCO using complementary visual encoders, with experiments covering quantitative performance, qualitative grounding, and component ablations. The results show gains from combining semantic and spatial representations, multi-scale layers, RoPE alignment, and redundancy reduction.

  • Quantitative Results: CoME-VL achieves 57.24% on Charts, 66.94% on Diagrams, 70.75% on Tables, and 81.84% on Others on PixMo.
  • Quantitative Results: CoME-VL reports 58.56% @3px and 75.94% @5px on Pointing, the only model in the comparison reporting these metrics.
  • Quantitative Results: On RefCOCO, CoME-VL reaches 92.57% on val, 95.36% on testA, and 90.51% on testB, improving over CLIP-to-DINO by +0.84, +1.30, and +1.66 points, respectively.
  • Ablation Studies: RoPE alignment improves localization over the Molmo baseline, orthogonality-regularized fusion adds a further gain, and their combination performs best.
  • Ablation Studies: Combining deeper SigLIP layers (0–27) with higher-level DINO layers (10–23) yields the best PixMo performance, while restricting SigLIP to layers 22–27 degrades several grounding tasks.

6 Conclusion

CoME-VL combines contrastive and self-supervised visual representations to improve visual understanding and grounding while limiting the token burden on the LLM. Its remaining cost is a modest inference-time overhead relative to Molmo.

  • 6 Conclusion: CoME-VL improves visual understanding and grounding by jointly leveraging contrastively trained and self-supervised visual representations.The framework combines SigLIP and DINO through entropy-guided layer selection, orthogonality-regularized fusion, and RoPE-enhanced cross-attention.
  • 6 Conclusion: SigLIP contributes semantic features, whereas DINO contributes grounding features; the best results combine DINO’s late-layer features with SigLIP features across layers.
  • 6 Conclusion: 1.52s per sample is CoME-VL’s average inference time, up from 1.26s for the Molmo baseline.This overhead remains below COMM’s approximately 2.2s/sample when CLIP and DINO features are directly concatenated.

A Methods Details

The referring-expression detection task maps an image and text query to a bounding box, with the language model generating its coordinates autoregressively as specialized tokens.

  • Methods Details: The input consists of an image I and a referring text query T describing a target region.
  • Methods Details: The prediction target is a bounding box b = [x_min,y_min,x_max,y_max] localizing the region described by T.
  • Methods Details: Bounding-box localization is formulated as autoregressive generation, with the language model producing coordinates as specialized tokens.

A.1 Orthogonal Layer (OL)

Orthogonal Layer fusion preserves information from selected encoder depths while reducing redundancy through normalized, near-isometric projections and learned mixing weights.

  • A.1 Orthogonal Layer (OL): Intermediate representations across deep encoder layers can be highly correlated, making multi-layer fusion redundant.The OL objective is to preserve useful information from multiple depths while reducing overlap among layer-specific directions.
  • A.1 Orthogonal Layer (OL): The orthogonal projection is constrained to be semi-orthogonal, acting as a near-isometry that limits distortion during fusion.
  • A.1 Orthogonal Layer (OL): For each selected layer, LayerNorm is followed by an orthogonal projection into the mixing dimension.The encoder hidden states have shape R^(T×d), while projected representations have shape R^(T×m).
  • A.1 Orthogonal Layer (OL): Selected projected layer features are mixed using learned nonnegative weights normalized by a softmax.The weights sum to 1, and the resulting representation Z_e is used by downstream modules.

A.3 Bounding-box encoding as tokens

The appendix describes tokenized bounding-box outputs and contrasts concatenation with cross-attention, emphasizing controlled sequence length and explicit spatial alignment for heterogeneous encoder grids.

  • Bounding-box encoding as tokens: Molmo-style detection serializes each bounding box into a short sequence of discrete coordinate tokens for autoregressive prediction.This enables a model without native detection outputs to generate boxes token by token.
  • Bounding-box encoding as tokens: Boxes use corner coordinates ordered as (x1,y1,x2,y2), with top-left and bottom-right corners constrained by x1 ≤ x2 and y1 ≤ y2.The ordering is enforced during preprocessing and decoding, including endpoint swapping when needed.
  • Bounding-box encoding as tokens: Coordinates are normalized to [0,1], quantized into B uniform bins, and mapped to reserved coordinate-vocabulary tokens.The described implementation uses B = 1000 unless otherwise specified.
  • Fusion and Alignment in Multi-Encoder VLMs: Cross-attention keeps the LLM sequence length fixed at N_t, with per-layer complexity O(N_tN_v) when attending to visual memory.This avoids the context-length growth and quadratic cost associated with concatenating text and visual tokens.
  • Fusion and Alignment in Multi-Encoder VLMs: RoPE-based alignment matches SigLIP queries to DINO keys and values using relative spatial offsets while keeping the output token count equal to the SigLIP token count.Optional DINO pooling can further reduce the visual-token count and address patch-size mismatch.

C Semantic Feature Analysis Per Layer

The authors analyze SigLIP2 and DINOv3 layer by layer to inspect how semantic and spatial attention evolves with depth. This qualitative analysis avoids cross-layer aggregation and gradient-based weighting.

  • Layer-wise attention rollout independently examines semantic and spatial features learned by SigLIP2 and DINOv3.The analysis inspects visual focus separately at each transformer layer.
  • The method computes attention rollout independently for each transformer layer rather than aggregating attention across layers or applying gradient-based weighting.

C.1 Layer-wise Attention Rollout Visualization

The rollout visualizations track how token-to-token relevance changes through deeper layers of DINOv3 and SigLIP2. They reveal complementary behavior: SigLIP2 provides broad semantic coverage, whereas DINOv3 concentrates spatially coherent discriminative information in deeper layers.

  • C.1 Layer-wise Attention Rollout Visualization: Attention heads are mean-aggregated, low-importance links are pruned, and recursive matrix multiplication propagates token relevance within each layer.Class-to-patch relevance is then reshaped into a spatial heatmap over the input image.
  • C.1 Layer-wise Attention Rollout Visualization: The final class-to-patch attention map is reshaped to the spatial grid, normalized, and overlaid on the input image as a heatmap.Figures 7–10 show these layer-specific rollout results for deeper DINOv3 and SigLIP2 layers.
  • C.1 Layer-wise Attention Rollout Visualization: The rollout visualizations highlight distinct spatial and semantic behaviors across depth for DINOv3 and SigLIP2.
  • C.1 Layer-wise Attention Rollout Visualization: SigLIP2 captures diverse semantic cues with broad spatial coverage in early layers, while DINOv3 concentrates discriminative, spatially coherent information in deeper layers 10–23.This complementarity supports using early SigLIP2 layers for semantic diversity and later DINOv3 layers for spatial information.
Loading 2604.03231v1…