PaperlayerView exact revision on arXiv

Source-linked AI summary

LoMo: Local Modality Substitution for Deeper Vision-Language Fusion

Feng Han, Zhixiong Zhang, Zheming Liang, Yibin Wang, Jiaqi Wang

arXiv:2605.30265v1cs.CVcs.CL

TL;DR

VLMs often lose performance when equivalent textual content is rendered as an image, reflecting limited cross-carrier representational invariance. LoMo addresses this gap by locally substituting selected text spans with rendered visual carriers in interleaved sequences, improving multimodal reasoning and cross-modal fusion across 13 benchmarks. The method improves over standard SFT by 2.67 points on LLaVA-OneVision-1.5-8B and 2.82 points on Qwen3.5-9B.

  • Problem

    VLMs exhibit carrier sensitivity because standard multimodal corpora assign text and images distinct, asymmetric roles, limiting alignment between equivalent textual and visual representations.

  • Method

    LoMo reformulates text-only prompts by selecting text spans, rendering them as images, and reinserting them into text–visual–text sequences for implicit cross-modal alignment supervision.

  • Results

    LoMo delivers consistent gains across 13 multimodal benchmarks, improving over standard SFT by 2.67 points on LLaVA-OneVision-1.5-8B and 2.82 points on Qwen3.5-9B.

  • Takeaways & Limitations

    LoMo offers a lightweight, architecture-agnostic data-side recipe for improving cross-carrier consistency and multimodal task performance.

  • Takeaways & Limitations

    LoMo is evaluated only during SFT and on two 8B–9B backbones, leaving pre-training, RL-based post-training, and substantially larger models for future work.

Abstract

from arXiv · show

Vision-Language Models (VLMs) have achieved substantial progress across a wide range of understanding and reasoning tasks, driven by large-scale image-text training aimed at multimodal fusion. Ideally, replacing a textual question with its rendered-image counterpart should leave model performance essentially unaffected. In practice, however, such modality substitution induces dramatic performance degradation. We attribute this "carrier sensitivity" issue to an inherent bias in current training corpora. Across prevalent datasets such as image captioning, VQA, OCR, and web-sourced interleaved data, text and images are typically organized into distinct and asymmetric roles, with text serving as linguistic queries and images as visual references. Such data bias leads VLMs to exhibit distinct preferences for information acquisition across different modalities. Consequently, VLMs fail to align representations of semantically equivalent content across textual and visual carriers, making model reasoning fragile under modality substitution. To address this, we propose Local Modality Substitution (LoMo), a lightweight, architecture-agnostic data curation paradigm designed to provide supervision for cross-modal representational invariance between semantically equivalent text and image carriers. LoMo achieves this by reformulating single-modality prompts into seamlessly interleaved multimodal sequences. It dynamically selects target text spans and recasts them as rendered images, thereby preserving the same semantics across "text, visual, text" carriers. Extensive experiments across 13 diverse multimodal benchmarks demonstrate that LoMo significantly improves overall multimodal reasoning and yields deeper cross-modal fusion. Specifically, it delivers consistent gains across foundational models, improving over standard SFT by 2.67 points on LLaVA-OneVision-1.5-8B and 2.82 points on Qwen3.5-9B.

1 Introduction

Current VLMs are sensitive to whether equivalent content is presented as text or an image, reflecting a cross-carrier modality gap. LoMo addresses this through local text-to-image substitution and improves cross-modal alignment and multimodal task performance.

  • Replacing textual content with its rendered-image counterpart causes consistent, significant accuracy drops across mainstream VLMs.
  • Accuracy degradation increases monotonically with the representational distance between semantically equivalent text and image carriers.The average accuracy drop rises from 7.75% in the closest distance bin to 21.23% in the farthest.
  • LoMo reformulates text-only prompts into interleaved text–visual–text sequences by selecting and rendering semantically coherent spans.Its pipeline uses structure-aware span localization, visual rendering, and perceptual distortion while preserving the original supervision target.
  • LoMo provides cross-modal alignment supervision without architectural modifications, inference overhead, or additional annotations.
  • 14.2% lower pairwise cross-modal distance than Standard SFT indicates tighter alignment between textual and visual carriers.
  • LoMo improves over standard multimodal SFT by 2.67 points on LLaVA-OneVision-1.5-8B and 2.82 points on Qwen3.5-9B across 13 benchmarks.

2 Related Work

Related work covers VLM architectures and training, text-as-pixels modeling, and efforts to understand and reduce modality gaps through cross-modal alignment.

  • Vision-Language Models: VLM research combines pretrained vision encoders with language-model backbones, while newer families emphasize data curation and post-training.
  • Text-as-Pixels Modeling: Text-as-pixels methods model rendered text visually, including OCR-free document understanding and vision-encoder training on image–text documents.
  • Modality Gap and Cross-Modal Alignment: Modality-gap research studies disjoint visual and textual representations and links reduced gaps with improved downstream performance.

3 Methodology

LoMo converts text-only prompts into interleaved text-image-text instances by selecting and rendering a local span, while preserving the original answer target. Its substituted objective introduces implicit alignment between semantically equivalent textual and visual carriers.

  • Overview and Formulation: LoMo applies structure-aware span localization, visual rendering, and perceptual distortion to transform text-only instances into interleaved multimodal inputs.The method selects a semantically coherent span, renders it as an image, and optionally applies semantics-preserving distortions.
  • Overview and Formulation: The transformed sequence retains surrounding text around the rendered span, forming a text-image-text skeleton for joint carrier comprehension.The original supervision target is preserved while the model must recover semantics from both textual context and the embedded visual carrier.
  • Implementation of LoMo: LoMo localizes a semantically coherent span using block-level structure, extracting the middle one-third while preventing truncation inside equations.Short inputs may be selected entirely, whereas longer inputs are chunked with formulas treated as indivisible units.
  • Implementation of LoMo: LoMo improves over Standard SFT by +2.68 on LLaVA-OV1.5-8B and +2.82 on Qwen3.5-9B across 13 benchmarks.Figure 3 reports consistent improvements across the two backbones.
  • Implementation of LoMo: The rendering pipeline routes mathematical spans to LaTeX rendering and other spans to standard text rendering before applying sampled perceptual distortions.Distortions include rotation, blur, shadows or stains, and wave-like geometric deformation, while preserving rendered semantics.
  • Implicit Cross-Modal Alignment Supervision of LoMo: Optimizing the carrier-substituted sequence adds an implicit cross-modal alignment term to standard SFT, encouraging agreement between predictions for equivalent carriers.The derivation identifies standard SFT as the first term and the alignment contribution through a KL-divergence formulation.

4 Experiments

Experiments evaluate LoMo across two VLM backbones, 13 multimodal benchmarks, two input-modality protocols, and representation-alignment metrics. LoMo consistently improves accuracy and cross-modal alignment, with especially large gains when questions are rendered as pixels.

  • Experimental setup: LoMo is evaluated on two open-source VLM backbones across 13 benchmarks spanning reasoning, VQA, OCR, document understanding, and visual perception.Standard and rendered evaluations use identical linguistic content while changing only the question’s input modality.
  • Main results: +2.68 average points on LLaVA-OneVision-8B and +2.82 on Qwen3.5-9B under Standard Evaluation, with gains in 23 of 26 comparisons.The strongest improvements include instruction following, visual perception, factuality, document and OCR understanding, and math reasoning.
  • Main results: +18.86 points on LLaVA-OneVision-8B and +11.92 on Qwen3.5-9B under Rendered Evaluation, substantially exceeding the corresponding Standard-Evaluation gains.LoMo compresses Qwen3.5-9B’s Standard-to-Rendered drop from 11.17 points under Standard SFT to 2.07 points.
  • Representation analysis: Interleaving rendered spans between textual prefix and suffix turns cross-carrier integration into a task-level requirement during training.This format provides correspondence between text-as-tokens and text-as-pixels while preserving the original supervision target.
  • Ablations: Structure-Aware Span Localization contributes more than naive full-question rendering, while Perceptual Distortion raises the average gain from +2.22 to +2.68.Full-Text Rendering yields only a +1.19 average gain over Standard SFT.
  • Ablations: At 4M training examples, LoMo’s accuracy gain reaches +2.68, MIR is 0.122 lower, and paired distance is 0.49.Across data scales, accuracy and both alignment metrics improve together relative to Standard SFT.

5 Conclusion

The paper identifies carrier sensitivity in VLMs and attributes it to asymmetric text-image roles in standard training corpora. LoMo addresses this gap through local modality substitution and improves both multimodal accuracy and cross-modal alignment.

  • 5 Conclusion: Carrier sensitivity causes substantial performance drops when identical textual content is rendered as visual input, with the drop correlated with cross-modal representational distance.The paper attributes this gap to asymmetric roles for text and images in standard multimodal training corpora.
  • 5 Conclusion: LoMo augments standard SFT with implicit cross-modal alignment supervision through local modality substitution without architectural modifications or inference overhead.Its interleaved training format preserves semantics while linking text and rendered-image carriers.
  • 5 Conclusion: Experiments across 13 multimodal benchmarks show consistent accuracy improvements alongside stronger cross-modal alignment.The conclusion summarizes LoMo’s effectiveness across downstream performance and representation alignment.

A Pure-Text Capability Analysis

LoMo preserves or slightly improves pure-text capabilities relative to Standard SFT across five benchmarks on both evaluated backbones. On Qwen3.5-9B, the pure-text IFEval gain follows the same direction as the multimodal MM-IFEval gain.

  • LoMo matches or slightly exceeds Standard SFT on all five evaluated pure-text benchmarks.The benchmarks cover general knowledge, mathematical reasoning, code generation, and instruction following.
  • +0.28 and +0.58 are the average pure-text gains for the two evaluated backbones.
  • +2.59 on Qwen3.5-9B IFEval aligns directionally with the +5.49 multimodal MM-IFEval gain.

B.1 Compute Resources

LoMo data construction and model experiments used substantial dedicated compute resources. Rendering ran on two CPU servers, while training and evaluation used a single 8-GPU node.

  • 128 cores were available on each of the two CPU servers used for LoMo image rendering.
  • Approximately 20 hours were required for the image-rendering stage in total.
  • All model training and evaluation experiments ran on a single node equipped with 8 NVIDIA H200 GPUs.

B.2 Training Data Construction

The training corpus combines original LLaVA-OneVision 1.5 instruct data with LoMo-augmented rendered-text data. It uses equal amounts of multimodal and text-only instances, with half of the text-only data rendered and a matched non-rendered Standard SFT baseline.

  • The corpus samples 2M multimodal and 2M text-only instances.
  • LoMo renders 50% of the text-only instances into images using text question prompts from the pure-text corpus.
  • Standard SFT uses the same 4M instances as LoMo but without LoMo rendering for a fair comparison.

B.3 Training Setup

The models are fine-tuned under a standard supervised fine-tuning regime and evaluated deterministically under standardized context and rendering settings. The setup specifies long sequences, high image resolution, and throughput-oriented training components.

  • LLaVA-OneVision-1.5-8B-Base and Qwen3.5-9B-Base are fine-tuned with LLaMA-Factory under standard supervised fine-tuning.
  • Training uses a maximum sequence length of 32,768 tokens and maximum image resolution of 2,560,000 pixels.
  • Evaluation uses EvalScope with temperature 0 and a maximum context length of 32K tokens.
  • Rendered evaluation replaces each textual question with a rendered image while keeping all other evaluation settings identical.

C Limitations

LoMo’s evaluation is limited to SFT-stage integration, heuristic span selection, and two 8B–9B backbones. It also inherits the biases and failure modes of its pretrained VLM foundations, requiring established safety practices in high-stakes deployment.

  • LoMo is applied only during supervised fine-tuning, leaving pre-training and reinforcement-learning-based post-training integration for future work.
  • Its block-level middle-span localization heuristic leaves finer-grained, difficulty-aware, and curriculum-style selection strategies unexplored.
  • Validation covers only two 8B–9B backbones because of compute constraints, so substantially larger models remain untested.
  • LoMo inherits the biases and failure modes of pretrained VLMs, motivating established VLM safety practices for high-stakes applications.
Loading 2605.30265v1…