Seeing Beyond Words: Self-Supervised Visual Learning for Multimodal Large Language Models

Davide Caffagni; Sara Sarto; Marcella Cornia; Lorenzo Baraldi; Pier Luigi Dovesi; Shaghayegh Roohi; Mark Granroth-Wilding; Rita Cucchiara

Source-linked AI summary

Seeing Beyond Words: Self-Supervised Visual Learning for Multimodal Large Language Models

Davide Caffagni, Sara Sarto, Marcella Cornia, Lorenzo Baraldi, Pier Luigi Dovesi, Shaghayegh Roohi, Mark Granroth-Wilding, Rita Cucchiara

arXiv:2512.15885v1Dec 17, 2025cs.CVcs.AIcs.CLcs.MM

TL;DR

MLLMs remain limited on basic visual reasoning because captions provide incomplete visual supervision. JARVIS integrates I-JEPA-style masked latent prediction into LLaVA alignment, and experiments show consistent vision-centric gains across LLM families without degrading broader multimodal abilities.

Problem
MLLMs struggle with fundamental visual reasoning because caption-based training provides subjective and incomplete supervision for image understanding.
Method
JARVIS adds I-JEPA-inspired masked latent prediction to LLaVA alignment, using frozen context and target vision encoders with early LLM layers as the predictor.
Results
JARVIS consistently improves vision-centric benchmark performance across LLM families without degrading general cognition or multimodal reasoning abilities.
Takeaways & Limitations
Self-supervised visual learning can strengthen MLLM visual perception beyond language-only alignment while preserving broader task performance.
Takeaways & Limitations
JARVIS still falls short of human-level spatial reasoning, and the best predictor layer is selected empirically rather than by a principled criterion.

Abstract

from arXiv · show

Multimodal Large Language Models (MLLMs) have recently demonstrated impressive capabilities in connecting vision and language, yet their proficiency in fundamental visual reasoning tasks remains limited. This limitation can be attributed to the fact that MLLMs learn visual understanding primarily from textual descriptions, which constitute a subjective and inherently incomplete supervisory signal. Furthermore, the modest scale of multimodal instruction tuning compared to massive text-only pre-training leads MLLMs to overfit language priors while overlooking visual details. To address these issues, we introduce JARVIS, a JEPA-inspired framework for self-supervised visual enhancement in MLLMs. Specifically, we integrate the I-JEPA learning paradigm into the standard vision-language alignment pipeline of MLLMs training. Our approach leverages frozen vision foundation models as context and target encoders, while training the predictor, implemented as the early layers of an LLM, to learn structural and semantic regularities from images without relying exclusively on language supervision. Extensive experiments on standard MLLM benchmarks show that JARVIS consistently improves performance on vision-centric benchmarks across different LLM families, without degrading multimodal reasoning abilities. Our source code is publicly available at: https://github.com/aimagelab/JARVIS.

1. Introduction

MLLMs connect vision and language effectively but remain weak at basic visual reasoning because caption-based supervision omits image details. JARVIS adds self-supervised visual prediction to the LLaVA alignment pipeline and improves vision-centric benchmarks without sacrificing general cognition.

Current MLLMs struggle with object presence, counting, spatial relationships, and relative distance despite broad task success.
Caption supervision is subjective, incomplete, and unable to represent all information contained in an image.
JARVIS integrates I-JEPA-style self-supervision into visual instruction tuning so MLLMs learn from image structure beyond textual descriptions.
JARVIS outperforms the original LLaVA training paradigm on standard benchmarks, with especially notable gains on Cambrian vision-centric tasks across LLM families and without degrading general cognition.

2. Related Work

Prior work improves MLLM perception through vision encoders, projectors, vision experts, attention analysis, or self-supervised reconstruction. JARVIS differs by adding I-JEPA-like masked prediction during LLaVA pre-training, using frozen vision encoders and an LLM predictor.

Vision-centric strategies modify image-to-text projectors, combine CLIP with vision experts, or exploit attention patterns in the language backbone.
Self-supervised approaches recover visual embeddings with denoisers or align intermediate LLM activations to external vision foundation models.
JARVIS adds I-JEPA-like masked-image representation prediction during LLaVA pre-training while the LLM learns to process visual tokens.
JEPA predicts compatible latent representations; I-JEPA infers masked image-crop representations from nearby context, while V-JEPA extends the principle to video.
JARVIS uses frozen vision foundation models as context and target encoders while assigning prediction to the LLM, unlike methods that discard the predictor at inference.

3. Proposed Method

JARVIS augments LLaVA alignment with a masked latent-prediction objective: an LLM predicts target visual embeddings from visible context while also generating captions. Attention masking and loss dropout combine visual and textual supervision efficiently.

3.1. Background: The baseline MLLM maps image embeddings into the LLM space with a trainable projector and trains first on captions, then on visual dialogue.
3.2. Bringing JEPA into MLLMs: A single context block predicts multiple possibly overlapping target blocks, with target positions represented by a learnable embedding plus positional encoding.
3.2. Bringing JEPA into MLLMs: JARVIS masks contiguous blocks of visual embeddings, feeds visible context and target-position latents to the LLM, and predicts target embeddings from a frozen target encoder.
3.2. Bringing JEPA into MLLMs: The combined losses run in one LLM forward pass through an attention mask that prevents textual supervision from influencing target prediction.
3.2. Bringing JEPA into MLLMs: The JEPA loss compares projected LLM activations with target-encoder embeddings, while the model simultaneously retains caption generation through a modified next-token objective.
3.2. Bringing JEPA into MLLMs: JARVIS intermittently skips the JEPA loss to balance masked-image self-supervision with whole-image language alignment, then removes the target encoder and target projector during instruction tuning.

4. Experiments

Experiments evaluate JARVIS’s training choices and compare it with LLaVA, VIRAL, and baselines across visual and general MLLM benchmarks. Results show consistent gains in vision-centric performance across LLM families while preserving or improving other task categories.

Scaling the Target Projector: A two-layer MLP projector improves Vision-Centric performance by +1.7 points, whereas a linear projector lowers Blink from 46.8 to 45.3.Smooth L1 distance yields +2.6 average visual points but severely degrades General and Vision-Centric performance relative to baseline, so JARVIS uses negative cosine similarity.
Comparison with Related Methods and Baselines: +0.8 and +0.7 average Vision-Centric accuracy points over LLaVA are achieved with Vicuna-7B and Qwen2-7B, respectively, including +6.2 on Qwen2-7B CVBench3D.JARVIS maintains competitive or better General, Knowledge, and OCR performance.
Generalization Across Different LLMs: +1.4 Vision-Centric accuracy points raises LLaMA-3.2-3B from 52.1 to 53.5, while Gemma2-2B gains +0.8 points from 50.0 to 50.8.These results extend JARVIS’s visual improvements to smaller, resource-constrained MLLMs.
Scaling the Target Encoder: Using a stronger context encoder, JARVIS exceeds LLaVA by +1.6 General, +0.8 Knowledge, +4.8 OCR, and +1.8 Vision-Centric points.This setting switches from CLIP ViT-L/13@336 to SigLIP2-So400M L/14@384.

5. Conclusion

JARVIS enhances MLLM visual perception by adding an I-JEPA-inspired self-supervised objective that predicts missing image content in latent space. Across diverse LLM families, this consistently improves vision-centric benchmark performance and visual reasoning.

JARVIS integrates I-JEPA-style latent prediction into LLaVA alignment, guiding the LLM to learn structural and semantic regularities beyond caption supervision.The method predicts missing image regions in latent space rather than relying solely on textual descriptions.
JARVIS consistently improves performance across diverse LLM families and a wide range of vision-centric benchmarks.The reported gains emphasize improved perception and reasoning about complex visual information.

A. Additional Implementation Details

The implementation follows the LLaVA-1.5 training setup while adding frozen target-encoder supervision and masked latent prediction. JARVIS masks visual embeddings rather than pixels and uses sampled context and target patch blocks.

Each sample uses four possibly overlapping target blocks, packed with context embeddings into one LLM forward pass while blocking attention between target blocks.This efficient design replicates I-JEPA-style target isolation through the attention mask.
Target-block scale is sampled from (0.15, 0.20) with aspect ratio (0.75, 1.5), while the larger context block uses scale (0.85, 1.0).Context patches overlapping any target block are removed to prevent trivial predictions.
The experiments retain LLaVA-1.5 hyperparameters and use instruction-tuned open-source LLMs, with configurations documented in Tables 4 and 5.Table 4 covers both alignment and visual instruction tuning; Table 5 maps each LLM to its Hugging Face reference.
The context encoder processes the whole image, but only embeddings from the visible context region enter the LLM, while target positions receive learnable latent variables.Masking is applied to encoder output embeddings rather than at the pixel level.

B.1. Additional Ablation Studies

Additional ablations show that overlapping targets and the proposed attention configuration support performance, while disabling target-to-text attention sharply harms visual perception. The study also evaluates masking and predictor-design choices across Vicuna-7B and Qwen2-7B.

+1.6 Vision-Centric average points, from 50.1 to 51.7, are obtained when target regions may overlap rather than being forced apart.The authors associate limited overlap with better use of spatial context and cross-region consistency.
The masking implementation predicts four target blocks from a larger visible context block after removing context patches shared with targets.The target and context regions are represented as patch-index sets and converted into visual embeddings for the LLM.
Allowing target blocks to attend to one another leaves performance stable, whereas blocking their interaction reproduces the I-JEPA isolation strategy.The comparison evaluates alternative attention configurations between visual and textual tokens.
Vision-Centric average accuracy drops from 51.7 to 49.2 when target tokens cannot attend to textual tokens.The proposed configuration blocks attention between target blocks but allows text to attend to predicted targets, achieving the best reported General, OCR, and Vision-Centric scores.
Intermediate LLM layers are preferred for LJEPA prediction, and a nonlinear target projector remains important for improving visual perception beyond the baseline.The layer analysis and projector ablations include results with Qwen2-7B as well as Vicuna-7B.

B.2. Detailed Results on CVBench

JARVIS consistently outperforms LLaVA and other methods on individual CVBench tasks across Vicuna-7B and Qwen2-7B. The strongest reported gains concern fine-grained spatial and 3D visual understanding.

With Vicuna-7B, JARVIS scores 64.6 on Relation2D versus 59.8 for LLaVA and achieves the highest overall scores on CVBench2D and CVBench3D.The results indicate gains on individual fine-grained vision-centric tasks.
Qwen2-7B JARVIS reaches 73.0 on CVBench3D, a +6.2-point gain over LLaVA, with improvements in depth and relative-distance understanding.JARVIS achieves the best results on all reported Qwen2-7B CVBench tasks.

B.3. Complete Results on MLLM Tasks

Table 9 organizes Cambrian results across General, Knowledge, OCR, and vision-centric benchmarks. JARVIS preserves competitive performance on language- and knowledge-oriented tasks while strengthening visual reasoning.

The Cambrian evaluation groups datasets into General, Knowledge, and OCR categories alongside vision-centric benchmarks.General covers perception and scene understanding; Knowledge covers science, mathematics, and diagrams; OCR covers embedded text and numerical information.
JARVIS remains competitive on General, Knowledge, and OCR tasks while improving visual reasoning across Cambrian benchmarks.With Vicuna-7B, it is comparable to LLaVA on General and Knowledge tasks and preserves strong OCR accuracy; with Qwen2-7B, the evaluation uses CLIP ViT-L or SigLIP2 ViT- backbones.
A baseline exceeds JARVIS by only +0.2 on GQA, while JARVIS exactly matches the baseline on OCRBench.

C. Additional Qualitative Results

Qualitative comparisons show JARVIS handling visual grounding and geometric reasoning across diverse Cambrian categories. The examples emphasize more reliable recognition and chart and graph reading than prior MLLMs and competing methods.

Additional Qualitative Results: JARVIS correctly handles reflectance differences, depth ordering, counting, and occlusions in qualitative examples.
Additional Qualitative Results: Figure 7 compares LLaVA, VIRAL, and JARVIS using samples from General, Knowledge, OCR, and Vision-Centric categories.
Additional Qualitative Results: Across Cambrian categories, JARVIS shows more reliable recognition and more accurate reading of charts and graphs than baselines and competitors.

D. Limitations and Impact

JARVIS improves visual perception but does not yet achieve human-level spatial reasoning. The reported failures also motivate a more principled way to select the LLM layer used for self-supervised learning.

Limitations: JARVIS still falls short of human-level spatial reasoning, with Fig. 8 documenting failure cases on Vision-Centric tasks.The figure includes examples from MMVP, CVBench2D, and CVBench3D.
Limitations: In one example, all MLLMs incorrectly confirm that a spider has eight legs although the image shows six.
Limitations: The best-performing LJEPA layer is empirically one fourth of the LLM depth, but selecting this layer lacks a principled theoretical criterion.The authors report that this choice transfers across models and identify principled layer selection as future work.