VL-Mamba: Exploring State Space Models for Multimodal Learning

1 Introduction

• Transformer self-attention has computational and memory costs that grow quadratically with sequence length, motivating alternatives for long-sequence modeling.The introduction identifies Mamba’s input-dependent selection and linear scaling in sequence length as relevant advantages.Source: §1 · pp. 1–2
• VL-Mamba is competitive with similarly sized MLLMs and outperforms larger 7B and 13B LLaVA-1.5 models on some popular benchmarks.The authors also report competitive performance against existing MLLMs and describe the model as a new framework option beyond Transformer-based architectures.Source: §1 · p. 2
• The proposed VL-Mamba replaces a Transformer-based language backbone with pretrained Mamba and introduces a MultiModal Connector with a Vision Selective Scan module.The connector uses bidirectional and cross-scan mechanisms; the authors also study combinations of vision encoders, Mamba variants, and connectors.Source: §1 · p. 2
• The authors report extensive experiments on diverse multimodal benchmarks and open-source the code to promote research on state space models for multimodal learning..Source: §1 · p. 2

2.1 State Space Models (SSMs)

• S4 and its variants address computational barriers in earlier state space models through parameterizations suited to continuous-time, recurrent, and convolutional views.DSS constrains the state matrix to be diagonal, while S4D provides a mathematical analysis of DSS initialization.Source: §2.1 · pp. 2–3
• Mamba adds input-dependent selection and hardware-aware computation to S4, while prior work adapts state space models to vision with bidirectional or cross-scan designs.Vision Mamba uses bidirectional SSMs, VMamba uses cross-scan, and U-Mamba combines CNNs and SSMs for biomedical image segmentation.Source: §2.1 · p. 3

2.2 Multimodal Large Language Model (MLLM)

• Prior MLLMs use approaches such as gated cross-attention, Q-Former, or visual-token projection, while VL-Mamba instead uses a pretrained Mamba language model and explores 2D selective scans and component combinations.The section notes that Transformer attention has high inference complexity for long sequences in multimodal learning.Source: §2.2 · p. 3

3.1 Preliminaries

• SSMs map inputs to outputs through a hidden state; discretization makes them usable in deep learning, while Mamba’s input-dependent selective scan addresses the fixed-parameter LTI formulation.Mamba derives B, C, and ∆ from input data to dynamically focus on sequence information.Source: §3.1 · p. 4

3.2.1 Overall Architecture

• The overall architecture sends image features from a pretrained vision encoder through a randomly initialized MMC, then combines its output with tokenized text for Mamba LLM response generation.Source: §3.2.1 · p. 5

3.2.2 Vision Encoder

• A ViT vision encoder converts each raw image into a sequence of visual patch features.Source: §3.2.2 · p. 5

3.2.3 MultiModal Connector (MMC)

• The MMC explores MLP, VSS-MLP, and VSS-L2 variants to bridge 1D sequence processing and 2D non-causal visual information.VSS combines a 2D scan mechanism with a Mamba layer; BSM scans patches forward and backward, while CSM scans row- and column-based sequences in four directions, then merges the reshaped features.Source: §3.2.3 · pp. 5–6
• In the VSS-MLP path, scanned visual features are combined with original patch features through a skip connection, then normalized and passed through an MLP.Source: §3.2.3 · p. 6

3.2.4 Mamba LLM

• The pretrained Mamba LLM receives the visual vector concatenated with text embeddings produced by tokenizing and embedding the natural-language query, and generates the response.Source: §3.2.4 · p. 6

4.1 Experimental setup

• The section first introduces implementation details and MLLM benchmarks.Source: §4 · p. 7

4.2–4.3 Results

• It then presents quantitative comparisons and qualitative results.Source: §4 · p. 7

4.4 Ablation studies

• It concludes with ablation studies.Source: §4 · p. 7

4.1.1 Implementation details

• Training aligns vision and language with the vision encoder and Mamba LLM frozen while updating the connector, then instruction-tunes the connector and Mamba LLM on 8 NVIDIA Tesla A800 GPUs.Source: §4.1.1 · p. 7

4.1.2 MLLM Benchmarks

• Evaluation covers eight benchmarks: VQA-v2, GQA, ScienceQA-IMG, TextVQA, POPE, MME, MMBench, and MM-Vet.Source: §4.1.2 · p. 7
• The benchmark suite spans image-question reasoning, OCR, object-hallucination detection, perceptual and cognitive abilities, and 16 emergent visual-language tasks.MMBench includes 3,000 single-choice questions across 20 dimensions; MM-Vet covers capabilities including recognition, knowledge, OCR, spatial awareness, language generation, and math.Source: §4.1.2 · p. 7

4.2 Quantitative Evaluation

• VL-Mamba scores 65.4 on SQAI, 48.9 on VQAT, and 1369.6 on MME, surpassing MobileVLM-3B’s respective 61.2, 47.5, and 1288.9.The models have similar parameter scale and use the same amount of multimodal training data; VL-Mamba uses 627B pretrained tokens versus MobileLLaMA’s 1.3T.Source: §4.2 · pp. 7–8
• VL-Mamba scores 76.6 on VQA-v2, 1369 on MME, and 32.6 on MM-Vet, exceeding LLaVA-Phi’s respective 71.4, 1335.1, and 28.9.LLaVA-Phi uses the Phi-2-2.7B language model with 1.4T pretrained tokens.Source: §4.2 · p. 8
• VL-Mamba performs similarly to LLaVA-1.5 on POPE despite LLaVA-1.5’s 13B LLM, approximately 4.6 times larger than the Mamba LLM.The authors also note that VL-Mamba has fewer parameters and limited training data compared with some larger models.Source: §4.2 · p. 8

4.3 Qualitative Result

• Figure 4 presents examples in which VL-Mamba understands the user’s question.The authors report that it responds accurately in these examples.Source: §4.3 · p. 8

4.4.1 The Effect of Variants of Language Model

• Mamba-2.8B-Slimpj outperforms Mamba-1.4B and Mamba-2.8B-Pile on all benchmarks as model scale and training tokens increase.The ablation uses CLIP-ViT and a vanilla MLP connector across variants, then selects Mamba-2.8B-Slimpj for subsequent experiments.Source: §4.4.1 · p. 9, Table 2 · p. 9

4.4.2 The Effect of Different Vision Encoders

• SigLIP-SO leads CLIP-ViT-L on most benchmarks, while CLIP-ViT-L performs substantially better on MME; the authors select SigLIP-SO for the final model.The comparison uses Mamba-2.8B-Slimpj and vanilla MLP connectors.Source: §4.4.2 · p. 9, Table 3 · p. 9

4.4.3 Ablation on Different MMC Architectures

• VSS-L2 performs relatively better than MLP and VSS-MLP on most benchmarks, especially VQAT, MME, MMB, and MM-Vet.These variants use SigLIP-SO, Mamba-2.8B-Slimpj, and bidirectional selective scanning.Source: §4.4.3 · p. 9, Table 4 · p. 9

4.4.4 Ablation on Different Scan Mechanisms

• Bidirectional scanning outperforms cross scanning on most benchmarks, though both score 76.6 on VQAv2.On MMB, bidirectional scanning scores 1369.6, 5.6 points higher than cross scanning.Source: §4.4.4 · p. 10, Table 5 · p. 10

5 Limitation

• Training data was not explored, although the authors note it could significantly affect benchmark performance.They identify higher-quality training data as a direction for future study.Source: §5 · p. 10

• Across eight multimodal benchmarks, VL-Mamba achieves performance comparable with some SoTA MLLMs.Source: §6 · p. 10
• VL-Mamba combines a pretrained Mamba LLM, a vision encoder, and an MMC with a VSS module bridging 2D non-causal image information and SSM causal modeling.Source: §6 · p. 10
• VSS uses Bidirectional Scanning Mechanism (BSM) and Cross Scanning Mechanism (CSM), while ablations evaluate language variants, vision encoders, connector architectures, and scan mechanisms.Source: §6 · p. 10