PaperlayerView exact revision on arXiv

Source-linked AI summary

Do VLMs Need Vision Transformers? Evaluating State Space Models as Vision Encoders

Shang-Jui Ray Kuo, Paola Cascante-Bonilla

arXiv:2603.19209v1cs.CVcs.LG

TL;DR

The paper asks whether SSM vision backbones can replace the ViT-family encoders commonly used in VLMs, given limited controlled evidence and the importance of spatially grounded reasoning. It evaluates matched backbone swaps and dense-task adaptations under a controlled LLaVA-style setup, finding strong SSM performance and identifying scaling and interface-stability limits. The results position SSM encoders as a competitive alternative while motivating architecture-agnostic stabilization strategies.

  • Problem

    VLM vision encoders are usually ViT-family models, while confounded comparisons leave it unclear whether backbone architecture itself limits useful visual evidence and spatial grounding.

  • Method

    The paper performs controlled LLaVA-style swaps of frozen SSM and ViT-family vision encoders, then evaluates detection- and segmentation-pretrained adaptations under matched settings.

  • Results

    VMamba achieves the strongest overall performance under matched IN1K/224 settings, dense objectives generally improve VQA and localization, and ImageNet accuracy or naive scaling is unreliable for downstream VLM quality.

  • Takeaways & Limitations

    SSM vision encoders are a strong alternative to ViT-family encoders, but downstream gains depend jointly on backbone architecture, pretraining objective, and stable vision–language interfaces.

  • Takeaways & Limitations

    Comparable contrastive- and SSL-pretrained SSM backbones are unavailable, so that direction is outside the main-paper scope.

Abstract

from arXiv · show

Large vision--language models (VLMs) often use a frozen vision backbone, whose image features are mapped into a large language model through a lightweight connector. While transformer-based encoders are the standard visual backbone, we ask whether state space model (SSM) vision backbones can be a strong alternative. We systematically evaluate SSM vision backbones for VLMs in a controlled setting. Under matched ImageNet-1K initialization, the SSM backbone achieves the strongest overall performance across both VQA and grounding/localization. We further adapt both SSM and ViT-family backbones with detection or segmentation training and find that dense-task tuning generally improves performance across families; after this adaptation, the SSM backbone remains competitive while operating at a substantially smaller model scale. We further observe that (i) higher ImageNet accuracy or larger backbones do not reliably translate into better VLM performance, and (ii) some visual backbones are unstable in localization. Based on these findings, we propose stabilization strategies that improve robustness for both backbone families and highlight SSM backbones as a strong alternative to transformer-based vision encoders in VLMs.

1. Introduction

The paper addresses limited architectural diversity and confounded comparisons in VLM vision encoders by systematically evaluating SSM backbones against transformer families in a controlled setting. It finds that SSM encoders are strong alternatives, while performance depends on pretraining objectives, scale, and interface stability.

  • Motivation: VLM studies commonly use frozen vision encoders and lightweight connectors, but vision-backbone choices remain concentrated in ViT-family architectures.The modular design typically updates only the connector and LLM during instruction tuning.
  • Motivation: Controlled comparisons are needed because prior evaluations often change pretraining objectives, multimodal pipelines, resolution, tokenization, or connector design simultaneously.These confounds make it difficult to attribute downstream differences to the vision architecture itself.
  • SSM Backbones: SSM vision backbones use structured multi-directional scans over 2D grids and show strong dense-prediction performance, making them plausible providers of spatially informative visual tokens.The paper identifies controlled inclusion of SSM encoders in VLM backbone swaps as an unexplored evaluation direction.
  • Findings: Under matched IN1K/224 swaps, VMamba achieves the strongest overall performance, consistently leading localization benchmarks while remaining competitive on open-ended VQA.The evaluation uses a controlled LLaVA-style setting with frozen vision encoders.
  • Findings: Dense pretraining improves VQA and localization across SSM and Transformer families, while ImageNet accuracy and naive scaling do not reliably predict downstream VLM quality.Some backbone configurations also exhibit sharp localization degradation that motivates stabilization strategies.

2. Preliminaries

The VLM combines a vision encoder, connector, and decoder-only language model under a controlled training and evaluation setup. Images become visual tokens, the connector maps them into the LLM space, and the model generates text from concatenated visual and prompt embeddings.

  • Experimental Framework: The experimental framework keeps the VLM pipeline identical while swapping only the vision-backbone checkpoint to isolate encoder effects.Shared settings cover the experiments unless a change is explicitly noted.
  • Model Architecture: The model consists of a vision encoder, lightweight connector, and decoder-only language model initialized from pretrained checkpoints.The language model used is Vicuna-7B.
  • Model Architecture: The vision encoder maps image x_img to visual tokens f_img = V_ω(x_img) in R^(L×d_vision), with token count and dimension determined by the backbone and image resolution.L denotes the number of visual tokens and d_vision their dimension.
  • Model Architecture: The connector maps visual tokens into the LLM embedding space as e_img = C_ψ(f_img) using a two-layer GELU multilayer perceptron.The resulting embeddings have dimension d_text.
  • Model Architecture: Visual and prompt embeddings are concatenated along the sequence dimension, after which the decoder-only language model autoregressively generates output text.The combined input has sequence length L+N.
  • Training Setup: Training uses one-stage instruction tuning with fixed optimization hyperparameters, random seed, and 665K multimodal instruction-tuning examples.Images are letterbox-resized, and training runs on 4× NVIDIA H200 GPUs with FSDP.
  • Evaluation: Evaluation covers six VQA benchmarks and four localization benchmarks, with weighted averages reported for VQA, localization, and overall performance.Dataset-specific preprocessing and thresholds follow the referenced prior setup.

3. Investigating Different Vision Encoders

The evaluation isolates vision-backbone effects through matched IN1K/224 swaps, then examines detection- and segmentation-adapted checkpoints under their pretraining geometries. Across these regimes, VMamba leads matched performance, dense adaptation can help but may destabilize localization, and larger or more accurate ImageNet backbones do not reliably improve VLM results.

  • Evaluation setup: Matched IN1K/224 swaps isolate architectural effects, while dense-objective comparisons study detection and segmentation adaptation under pretraining geometries.The dense-objective comparisons are not perfectly matched architectural swaps because input geometry and token length differ.
  • Matched backbone swaps: VMamba achieves the strongest overall matched performance, with VMamba-T/S consistently leading localization and VMamba variants reaching top aggregate VQA.The matched setup uses 224×224 inputs and L=196 visual tokens across backbone families.
  • Matched backbone swaps: Higher ImageNet accuracy and naive scaling do not reliably predict VLM performance: MaxViT-L worsens localization, and VMamba-B does not improve overall performance over VMamba-T/S.VMamba and MambaVision improve at small sizes but degrade at larger scales.
  • Dense objectives: Detection adaptation can improve VQA and localization when stable, but ViTDet-L/H and VMamba-T/B exhibit sharp localization degradation called localization collapse.The observed instability depends on the adapted configuration rather than uniformly affecting all detection-pretrained checkpoints.
  • Dense objectives: Segmentation adaptation yields more consistently strong localization across scales, with adapted VMamba generally outperforming DeiT (ViT-Adapter) baselines.These comparisons primarily reflect dense-objective effects because checkpoints are evaluated at differing pretraining geometries.

4. Analysis, Diagnosis, and Stabilizations

The analysis links VQA and localization, shows that dense-task adaptation can help but may trigger localization collapse, and attributes recovery to stabilizing the vision–language interface. Overall, backbone architecture, pretraining objective, and interface jointly determine downstream VLM quality.

  • 4.1. Localization Matters for General VQA: VQA-v2, GQA, POPE, and TallyQA correlate moderately to strongly with localization metrics, with Pearson r values from 0.65 to 0.80.Localization benchmarks are highly consistent with each other in the checkpoint analysis.
  • 4.2. VMamba is Strong Under the Matched IN1K/224 Setting: VMamba-T produces sharper token–region similarity and tighter grounding boxes than ViT-S under matched IN1K/224 settings.The qualitative alignment pattern is consistent with quantitative localization gains.
  • 4.3. Dense-Task Objectives Help: Dense detection or segmentation objectives often improve both localization and VQA, with the largest gains for ViT-family baselines lacking built-in spatial inductive bias.VMamba also benefits, but its improvements are typically more moderate because IN1K pretraining already preserves spatial structure.
  • 4.4. Failure Modes: Higher ImageNet accuracy and larger backbones do not reliably predict VLM quality, while detection adaptation can cause sharp localization collapse in ViTDet-L/H and VMamba-T/B.Related backbone variants can retain rich spatial information, suggesting collapse is more likely an interface-transfer failure than absent visual features.
  • 4.5.2. Transmission Bottleneck Test: A stronger 3-layer connector recovers localization in some collapse cases, but it does not resolve every failure, including VMamba-B and detection-pretrained VMamba-T.The results indicate that connector expressivity is important but insufficient on its own.
  • 4.5.3. Utilization Bottleneck Test: Square 512×512 inputs eliminate collapse and improve localization and VQA for detection-pretrained VMamba-T/B, while combining square geometry with a stronger connector yields the most consistent gains.VMamba-S slightly degrades at 512×512 because that geometry differs from its pretraining resolution; deeper mechanistic analysis remains future work.
  • 4.6. Summary: Backbone, Objective, and Interface: VLM performance is jointly determined by backbone architecture, pretraining objective, and interface stability, with spatially strong backbones and dense objectives helping only when spatial signals transfer reliably.The objective and interface factors are described as broadly architecture-agnostic, whereas architecture remains important for robustness under matched classification pretraining.

5. Related Works

Related work has used SSMs mainly for language or multimodal fusion, while vision encoders often remain ViT- or CNN-based. Fully Mamba-based contrastive vision–language pretraining exists, but the cited approaches do not study vision-backbone architecture effects in downstream generative VLM behavior.

  • SSMs in Multimodal Models: Prior multimodal systems primarily replace transformer text backbones or fusion blocks with Mamba-style layers while retaining ViT- or CNN-style vision encoders.These systems target sequence-modeling efficiency on the language and fusion sides.
  • SSMs in Multimodal Models: CLIMP replaces both vision and text encoders with SSMs for contrastive representation learning, rather than evaluating downstream generative VLM behavior.The cited related work therefore does not analyze the effect of different vision-backbone architectures in generative VLMs.

6. Conclusion

The paper finds that SSM-based VMamba is a strong, size-efficient alternative to ViT-family encoders, while backbone objectives and interface design materially affect VLM performance and stability.

  • VMamba is a strong alternative to ViT-family encoders in controlled LLaVA-style VLMs with frozen vision encoders.
  • Under matched IN1K/224 swaps, VMamba-T/S consistently lead grounding benchmarks, while dense-task pretraining improves VQA and localization across backbone families.
  • ImageNet accuracy and naive backbone scaling are unreliable predictors of downstream VLM quality.
  • Some high-resolution detection-pretrained configurations suffer sharp localization collapse from a vision–language interface failure mode.
  • Increasing connector capacity and adjusting interface geometry can complementarily recover localization and overall performance.

A. Training Setup and Hyperparameters

The experiments use a shared instruction-tuning recipe with frozen vision encoders and compare feature-extraction stages under matched spatial and token-count conditions.

  • Training Setup and Hyperparameters: All models use the same one-stage instruction-tuning recipe, training only the language model and connector while freezing the vision encoder.
  • Training Setup and Hyperparameters: Training runs for 1 epoch with global batch size 128, AdamW, learning rate 2 × 10−5, weight decay 0.1, and fixed hyperparameters across experiments.
  • Training Setup and Hyperparameters: Images use letterbox resizing, while incomplete batches are retained and variable-length inputs are truncated or padded as needed.
  • Vision Encoder Feature Extraction Stage: Hierarchical backbones begin at (H/4) × (W/4) resolution and downsample spatial dimensions by 2 at each subsequent stage.
  • Vision Encoder Feature Extraction Stage: At 224 × 224 resolution, stage 3 has 14 × 14 spatial size and 196 tokens, matching ViT/16 exactly.
  • Vision Encoder Feature Extraction Stage: Stage 4 often underperforms stage 3 on grounding and localization despite higher input resolution and comparable vision-token counts.

C. VMamba vs Vim

VMamba is selected as the representative SSM backbone because it consistently outperforms Vim in the VLM setting, while feature-stage choice substantially affects downstream results.

  • C. VMamba vs Vim: VMamba consistently performs better than Vim in VLM experiments and is adopted as the representative SSM-based backbone.
  • C. VMamba vs Vim: The comparison is restricted to Vim and VMamba because they have established architectures and mature public implementations.

E. Linear Probing Results

Linear probing only partially explains downstream VLM behavior: stronger global-feature probes do not reliably predict VQA or localization performance, especially when spatial fidelity matters.

  • E. Linear Probing Results: Linear probing trains a classifier on frozen global image features extracted using CLS tokens or mean pooling over spatial tokens.
  • E. Linear Probing Results: Weighted probing scores average performance across probing benchmarks using dataset-size weighting.
  • E. Linear Probing Results: Higher probing performance does not reliably predict stronger VLM performance across backbone families.
  • E. Linear Probing Results: ViT-B probes better than ViT-S yet performs worse overall, while larger MaxViT variants probe better than MaxViT-T but yield weaker localization and overall VLM performance.
  • E. Linear Probing Results: Dense-objective checkpoints can improve downstream VLM performance despite modest or lower probing scores than classification-pretrained counterparts.

F.1. Unfair Comparisons

The supplementary comparisons change multiple experimental factors at once, so they should not be treated as evidence for the main controlled conclusions. They nevertheless provide context on how pretraining objectives, stabilization, and inference setup affect VLM behavior.

  • Scope of comparisons: The supplementary tables vary pretraining dataset, input resolution, visual-token count, and model size simultaneously, limiting controlled architectural comparisons.These results are reported as supplementary reference rather than evidence for the main controlled conclusions.
  • Classification pretraining: Larger-scale classification pretraining or higher-resolution features can improve ViT VQA, but these changes do not consistently improve localization.MambaVision does not show the same trend, and increased scale or resolution can degrade both VQA and localization.
  • Dense-objective pretraining: Dense-objective pretraining reinforces downstream gains, while VMamba variants remain consistently stronger on localization in additional large-scale comparisons.Very large models can achieve the strongest VQA results in the reported group.
  • Alternative objectives: Stabilized fusion of DINOv2 and SigLIP becomes strongest in its comparison group, whereas direct fusion can be unstable for localization.The stabilized variants use larger vision stacks and higher token budgets than standard VMamba, so they are reported separately.
  • Inference profiling: Inference profiling rebuilds each encoder at target square resolutions and measures host, GPU, end-to-end latency, and peak memory at batch size 1.The first 100 of 200 iterations are treated as warm-up to estimate steady-state cost more stably.

G.2. Observations

At comparable scale, ViT and VMamba have similar practical inference costs, while VMamba offers stronger main-task performance and ViTDet is substantially heavier. Increasing resolution exposes different scaling and memory behaviors across the backbones.

  • Overall trade-offs: VMamba and ViT operate in a broadly similar efficiency regime, whereas ViTDet is substantially heavier and reaches memory limits earlier.This contrasts a scale-matched baseline with a higher-capacity alternative.
  • Memory scaling: ViTDet runs out of memory at much lower resolutions than ViT and VMamba, making memory a practical limit for this larger family.The observation is consistent with ViTDet's much larger model size.
  • Resolution scaling: VMamba's GPU vision latency grows more gradually with resolution than ViT's, while ViTDet has larger host-side vision overhead.The figure reports six views spanning host-side, GPU, end-to-end, and peak-memory costs.
  • Stagewise cost: The language stage dominates overall inference time for most settings, with vision-stage differences becoming more visible at very high resolutions or for ViTDet.End-to-end latency curves for ViT and VMamba are closer to LLM-stage than vision-stage curves.
  • Overall trade-offs: VMamba achieves stronger VLM performance than similarly scaled ViT at similar practical inference cost, while leaving more room than ViTDet for scaling.ViTDet remains competitive in VLM performance but reaches out-of-memory much earlier.

H. Exhaustive Results

The exhaustive-results section organizes VQA and localization benchmarks across all evaluated backbone families and separates standard, alternative-pretraining, and task-adapted settings. These tables provide family-specific coverage rather than a single additional controlled comparison.

  • Backbone coverage: Tables 23–36 provide exhaustive VQA and localization results for VMamba, ViT, MaxViT, MambaVision, Vim, ViTDet, and ViT-Adapter.Each family is assigned separate VQA and localization tables.
  • Backbone coverage: The VMamba, ViT, MaxViT, MambaVision, Vim, ViTDet, and ViT-Adapter families are each represented by dedicated benchmark pairs.The table sequence covers VQA followed by localization for every listed family.
Loading 2603.19209v1…