PaperlayerView exact revision on arXiv

Source-linked AI summary

Unified Vision-Language Modeling via Concept Space Alignment

Yifu Qiu, Paul-Ambroise Duquenne, Holger Schwenk

arXiv:2603.01096v1cs.CVcs.AIcs.CLcs.LG

TL;DR

Vision-language modeling lacks broadly multilingual, modality-agnostic visual embeddings connected to latent language modeling. The paper aligns vision representations to SONAR, demonstrates zero-shot visual understanding with LCM, and introduces V-LCM for instruction-tuned multimodal modeling. V-LCM matches state-of-the-art vision-language models on captioning and question answering while outperforming them across 61 of 62 tested languages.

  • Problem

    The integration of visual embeddings into universal multilingual text spaces remains underexplored.

  • Method

    The paper maps a vision encoder into SONAR through a three-stage coarse-to-fine alignment and extends LCM with multimodal instruction tuning in the shared SONAR and V-SONAR latent space.

  • Results

    V-LCM matches state-of-the-art vision-language models on captioning and question answering while outperforming them across 61 of 62 tested languages.

  • Takeaways & Limitations

    English-only LCM can process V-SONAR visual embeddings zero-shot, while V-LCM supports multilingual and multimodal vision-language tasks.

  • Takeaways & Limitations

    The zero-shot LCM experiments use an LCM trained only on English text and without video or captioning fine-tuning, while the vision alignment relies on Perception Encoder.

Abstract

from arXiv · show

We introduce V-SONAR, a vision-language embedding space extended from the text-only embedding space SONAR (Omnilingual Embeddings Team et al., 2026), which supports 1500 text languages and 177 speech languages. To construct V-SONAR, we propose a post-hoc alignment pipeline that maps the representations of an existing vision encoder into the SONAR space. We thoroughly evaluate V-SONAR and show that its embeddings achieve competitive performance on text-to-video retrieval. Equipped with the OMNISONAR text decoder, V-SONAR further surpasses state-of-the-art vision-language models on video captioning tasks, including DREAM-1K (BLEU 23.9 vs. 19.6) and PE-VIDEO (BLEU 39.0 vs. 30.0). Leveraging V-SONAR, we first demonstrate that the Large Concept Model (LCM; LCM team et al. 2024) operating in SONAR and trained with English text only, can perform both single- and multi-visual concept understanding in a zero-shot manner. Finally, we introduce V-LCM, which extends the LCM with vision-language instruction tuning. V-LCM encodes vision and language inputs into an unified sequence of latent embeddings via V-SONAR and SONAR, and it is trained with the same latent diffusion objective for next-embedding prediction as in LCM's text-only pre-training. Experiments on a large-scale multilingual and -modal instruction-tuning data mixture highlight the potential of V-LCM: V-LCM matches state-of-the-art vision-language models on tasks covering image/video captioning and question answering, while significantly outperforming them across 61 rich- to low-resource languages out of all 62 tested languages.

1 Introduction

The paper extends SONAR into a shared vision-language embedding space and uses it to connect visual inputs with latent diffusion language modeling. V-SONAR and V-LCM achieve strong retrieval, captioning, zero-shot visual understanding, and multilingual vision-language performance.

  • V-SONAR: V-SONAR embeddings achieve competitive text-to-video retrieval and state-of-the-art zero-shot video retrieval and captioning performance.On PE-Video retrieval, V-SONAR reaches Recall@1 of 73.03 versus 63.91 for SigLIP2-g-opt.
  • Zero-shot LCM: LCM, trained only on English text, can process V-SONAR visual embeddings zero-shot for single-concept captioning and multi-concept video summarization.LCM remains competitive with vision-language models across PE-Video, Dream-1K, VATEX, and VideoXum without video or captioning fine-tuning.
  • V-LCM: V-LCM unifies image, video, and text inputs in SONAR and V-SONAR latent space and trains with LCM’s latent diffusion objective for next-embedding prediction.It is evaluated on M3IT, which spans eight task categories, both image and video modalities, and 80 languages.
  • V-SONAR: V-SONAR extends the language- and modality-agnostic SONAR space to image and video through post-hoc coarse-to-fine alignment.The alignment uses image-caption pairs for coarse grounding, synthetic video-caption pairs for temporal adaptation, and human-annotated video captions for fine-grained alignment.
  • V-LCM: V-LCM matches state-of-the-art models on captioning and question answering while outperforming them in 61 of 62 tested languages.The multilingual evaluation ranges from high-resource to low-resource languages.

2 v-Sonar

v-Sonar is constructed by post-hoc aligning a vision encoder with Sonar’s multilingual semantic space, producing unified image/video embeddings. The section also describes the alignment objective, curriculum, and how this space supports multimodal LCM extensions.

  • Alignment setup: v-Sonar post-hoc aligns a state-of-the-art vision encoder with Sonar’s multilingual textual embedding space.The trainable vision mapping is learned against frozen Sonar text representations from paired visual inputs and captions.
  • Architecture: Images and videos are encoded frame by frame, temporally adapted, and aggregated into one video-level representation.The projector injects positional embeddings, enables frame interactions through temporal attention, and applies attention-based aggregation.
  • Alignment objective: The alignment minimizes mean squared error between visual and textual embeddings assumed to share the same semantic meaning.Sonar remains frozen while the projector and vision encoder are updated; an additional contrastive loss produced no significant gains.
  • Training curriculum: The three-stage curriculum progresses from 12M image–caption pairs toward increasingly complex visual semantics.The supplied passage identifies the initial 12M-pair stage and the subsequent introduction of 2M additional pairs.
  • Sonar variants: OmniSONAR substantially outperforms Sonar1 on multilingual similarity search over 200 Flores languages.The comparison uses xsim++, which includes hard negatives, and is also examined for vision captioning in an appendix.
  • LCM extension: v-LCM extends LCM by encoding visual and textual inputs into a shared latent space and training with latent diffusion for next-embedding prediction.The two-tower LCM separates contextualization from denoising, while v-LCM applies vision-language instruction fine-tuning to multimodal inputs.

3 Experiments

The experiments evaluate v-Sonar for text-to-video retrieval and video captioning, then test zero-shot LCM processing and instruction-tuned v-LCM across multimodal and multilingual tasks.

  • Text-video Retrieval: 9.12, 1.8 and 13.23 points are v-Sonar’s Recall@1 improvements over SigLIP2 on PE-Video, Dream-1k and Vatex, respectively.Against Perception Encoder, it improves by 9.12 and 21.85 on PE-Video and Vatex but loses 8.8 on Dream-1k.
  • Zero-shot LCM: LCM processes v-Sonar visual embeddings zero-shot, lagging the strongest baseline by 1.15, 4.44 and 4.76 BLEU on PE-Video, Dream-1k and Vatex.It also achieves 22.1 BERTScore-F1 on VideoXum, compared with 33.7 for PLM-8B and 20.5 for InternVL-2.5-8B.
  • Zero-shot LCM: Across all video-length categories, LCM with v-Sonar consistently outperforms LCM with Sonar on Rouge-L.Sonar performance declines as video length increases, whereas v-Sonar retains stronger performance.
  • v-LCM: 63.9 R-L on IVQA and 63.6 R-L on ActivityNetQA exceed zero-shot LCM scores of 48.9 and 51.7, respectively.v-LCM achieves state-of-the-art results on IVQA, ActivityNetQA and MSRVTT-QA, although it trails leading models on some other benchmarks.
  • v-LCM: 61 of 62 tested languages show v-LCM outperforming Qwen2.5-VL-7B and PLM-8B, with Dutch the only exception.Gains are substantial in mid- and low-resource languages; v-LCM produces meaningful outputs in Urdu, modern Arabic and Tamil where competing models fail entirely.

4 Related Works

Prior work established shared vision-language and multilingual embedding spaces, but large-scale alignment of visual embeddings into universal text spaces remains underexplored.

  • Vision-language Embeddings: CLIP and ALIGN pioneered large-scale contrastive alignment of paired images and captions for zero-shot transfer.Later work extended shared-representation pretraining to video-language applications.
  • Multilingual Embeddings: Multilingual embedding models such as LASER, LaBSE and Sonar demonstrate the effectiveness of language-agnostic spaces across hundreds of languages.Related research also explored speech-to-text alignment.
  • Research Gap: Large-scale alignment of visual embeddings into universal text spaces remains underexplored.

5 Conclusion

The paper extends Sonar to image and video through v-Sonar, uses it to process visual embeddings with LCM, and introduces instruction-tuned v-LCM for multimodal tasks.

  • Conclusion: v-Sonar extends Sonar to image and video, creating a shared embedding space spanning text, speech, image and video.
  • Conclusion: A three-stage training approach maps pooled Perception Encoder representations to semantic Sonar representations.
  • Conclusion: v-Sonar achieves competitive text-to-video retrieval and video-captioning results.
  • Conclusion: LCM can process image or video embeddings zero-shot through v-Sonar without training data in those modalities.
  • Conclusion: v-LCM uses multimodal instruction fine-tuning and matches state-of-the-art vision-language models while outperforming them across 61 rich- to low-resource languages.

A Contrastive Loss for Aligning Perception Encoder and Sonar

The paper evaluates adding a contrastive loss to MSE-based alignment between visual and textual representations in SONAR. Contrastive training improves retrieval-related behavior but does not significantly improve captioning, so the MSE-only objective is selected for downstream generation.

  • Contrastive objective: Contrastive training minimizes distances for matched vision–text pairs while pushing mismatched pairs apart.The loss combines cosine-similarity comparisons within a mini-batch using a temperature parameter.
  • Performance trade-off: Contrastive loss improves retrieval performance but does not significantly improve captioning over MSE-only alignment.The comparison is reported on PE-Video using a single MLP connector.
  • Objective selection: MSE-only alignment is chosen because v-Sonar is used downstream with v-LCM for generation tasks.The authors prioritize captioning-oriented downstream usage despite retrieval gains from contrastive training.
  • Embedding-space analysis: The authors hypothesize that contrastive training can shift visual embeddings away from the SONAR manifold and degrade generative reconstruction even when retrieval improves.The proposed explanation concerns altered norms or local covariance and relative rather than absolute similarity constraints.
  • Embedding-space analysis: Contrastive-trained embeddings have higher norm, covariance trace, and volume than MSE-only embeddings, indicating a more expanded distribution.The reported alignment consistencies in cosine similarity and MSE are also poorer for contrastive training.

B Dataset Statistics

The alignment curriculum uses large-scale image–caption data, synthetic video–caption data, and manually curated video–text data. These datasets supply grounding, linguistic richness, temporal dynamics, and verified multimodal supervision.

  • Dataset composition: The three-stage alignment pipeline uses image–caption pairs, synthetic video–caption pairs, and manually verified video-caption data.The dataset table reports sample counts, average video duration where applicable, and caption lengths.
  • Image data: SA1B and OpenImages provide large-scale image–caption pairs for grounding and linguistic richness.These datasets are grouped as PLM-Image datasets.
  • Video data: PLM-Video-Auto-YT1B contributes video–text pairs averaging 22.75 seconds, supporting temporal-dynamics learning.The dataset is used in the synthetic video-captioning stage.
  • Video data: PE-Video provides carefully curated human-annotated video-caption data for the alignment curriculum.It forms the manually verified video-captioning stage.

C Implementations

The implementation builds v-Sonar on a large Perception Encoder vision transformer and trains the alignment connector through a staged procedure. The v-LCM uses a diffusion-based next-sentence objective with AdamW optimization and cosine scheduling.

  • v-Sonar architecture: The vision tower uses PE-Core-G14-448 with 1.9B parameters and processes 448×448 RGB images into 1024 patches per frame.For video inputs, the model uniformly samples frames.
  • v-Sonar optimization: Training initially freezes the Perception Encoder for 2,000 steps while optimizing the projector.The projector uses a small-variance initialization to mitigate gradient explosion when mapping high-dimensional features.
  • v-Sonar optimization: The alignment model is trained with a three-stage curriculum spanning image captioning, synthetic video captioning, and manually verified video captioning.Stage 1 runs for 15 epochs, while Stages 2 and 3 use the same stated settings.
  • v-Sonar optimization: Optimization uses AdamW, cosine learning-rate decay, bfloat16 precision, gradient accumulation, and FSDP across 64 Nvidia A100-80G GPUs.Early stopping uses a patience of three epochs and checkpoints the best validation model.
  • v-LCM training: v-LCM adopts the LCM two-tower architecture with a diffusion-based next-sentence fine-tuning objective.Training uses AdamW, gradient clipping, cosine decay, warmup, and annealing to a final learning rate of 10^-6.

D Sonar vs OmniSONAR

The paper compares SONAR and OmniSONAR embedding spaces for text-to-video retrieval across three datasets. OmniSONAR generally has an advantage, while SONAR1 remains highly competitive on PVD-Bench.

  • Retrieval comparison: OmniSONAR generally outperforms SONAR1 in the reported text-to-video retrieval comparison.The comparison covers three retrieval datasets.
  • Retrieval comparison: 64.9 R@1 is achieved by SONAR1 on PVD-Bench, where it remains highly competitive compared with PECoreG.This is the representative retrieval result highlighted in the comparison.

E Detailed Multilingual Evaluation

The multilingual evaluation tests supported languages across image captioning, video captioning, and visual question answering. The model consistently outperforms baselines across tested languages, with limited exceptions, while alignment progressively reduces the modality gap.

  • Multilingual evaluation: The evaluation covers supported languages across ImageNet, MSR-VTT, MSRVTT-QA, OKVQA, VIST, and VQA-v2.Figures 5–14 report multilingual results across image captioning, video captioning, and question-answering tasks.
  • Multilingual evaluation: ImageNet is the main multilingual-evaluation exception, likely reflecting its widespread use and extensive coverage in existing vision-language models.
  • Multilingual evaluation: The model consistently outperforms baselines across all tested languages, except Thai in VQA-v2.
  • Embedding alignment: After each alignment stage, paired video and SONAR embeddings cluster more closely in t-SNE visualizations.The visualization uses 200 PE-Video test samples and explicitly connects 50 paired video-caption samples.

G Analysis in Cross-modal Drift for v-Sonar and v-LCM

The analysis examines whether decoding introduces cross-modal semantic drift. SONAR decoding remains nearly faithful to ground-truth semantics, while v-LCM shows slightly greater deviation attributed to stylistic paraphrasing from instruction tuning.

  • Drift analysis: The study evaluates semantic drift by comparing generated-caption embeddings with ground-truth caption embeddings and by testing round-trip video retrieval.It analyzes embeddings from SONAR decoding and v-LCM decoding on the PE-Video test set.
  • Embedding fidelity: SONAR-decoded captions have nearly identical cosine similarity or distance to ground truth, indicating negligible cross-modal drift.
  • Embedding fidelity: v-LCM captions show slightly larger deviation from ground truth, attributed to stylistic paraphrasing introduced by instruction-following training rather than semantic drift.
  • Round-trip retrieval: SONAR- and LCM-decoded captions retrieve the correct source video with extremely high accuracy, with LCM within 0.2% of SONAR on R@1.

H.4 Qualitative Cases for Video Captioning by v-Sonar

The qualitative cases compare v-Sonar and v-LCM with other models on captioning, question answering, and visual commonsense reasoning. They illustrate stronger factual, fine-grained, spatial, and motion-related understanding in selected examples.

  • Visual reasoning: The VCR example indicates that semantic-level alignment can still preserve layout grounding and spatial reasoning ability.
  • Video captioning: For one video, v-LCM produces a concise description of a man riding a red motorcycle on a mountain instead of LCM’s lengthy erroneous narration.
  • Video captioning: In another video, v-LCM describes a large building, windows, a grey pole, and people walking, while LCM generates repeated and implausible motion descriptions.
  • Question answering: V-LCM answers that a ‘V’ tile is worth 4 points, whereas LCM states that the information is insufficient.
  • Question answering: V-LCM selects the parachute-stability option involving a 1 m air leak at 200 km per hour, while LCM gives a less faithful wording of the option.
  • Question answering: V-LCM identifies the translator as Rebecca Herting, while LCM says the translator’s name is unavailable.
  • Fine-grained motion understanding: On fine-grained motion understanding, SONAR captures the mixer mechanics and blue-cylinder attachment that other VLMs miss or hallucinate.
Loading 2603.01096v1…