PaperlayerView exact revision on arXiv

Source-linked AI summary

Boosting Visual Instruction Tuning with Self-Supervised Guidance

Sophia Sirko-Galouchenko, Monika Wysoczanska, Andrei Bursuc, Nicolas Thome, Spyros Gidaris

arXiv:2604.12966v1cs.CV

TL;DR

MLLMs can underuse visual information during instruction tuning, limiting fine-grained vision-centric reasoning even when visual representations are rich. V-GIFT adds a small number of visually grounded self-supervised instructions to standard tuning, and consistently improves vision-centric performance across models, training regimes, and benchmarks.

  • Problem

    MLLMs often under-utilize rich visual representations during instruction tuning because many instruction-following tasks can be partially solved using language priors, weakening vision-centric reasoning.

  • Method

    V-GIFT reformulates rotation, colorization, and correspondence pretext tasks as automatically supervised natural-language instruction triplets integrated into standard visual instruction tuning.

  • Results

    3–10% visually grounded SSL instructions consistently improve vision-centric reasoning across multiple benchmarks, MLLM backbones, and training regimes, with gains confirmed beyond additional compute.

  • Takeaways & Limitations

    Adjusting the instruction-tuning data distribution can improve visual grounding without architectural modifications, auxiliary objectives, or additional training stages.

Abstract

from arXiv · show

Multimodal large language models (MLLMs) perform well on many vision-language tasks but often struggle with vision-centric problems that require fine-grained visual reasoning. Recent evidence suggests that this limitation arises not from weak visual representations, but from under-utilization of visual information during instruction tuning, where many tasks can be partially solved using language priors alone. We propose a simple and lightweight approach that augments visual instruction tuning with a small number of visually grounded self-supervised tasks expressed as natural language instructions. By reformulating classical self-supervised pretext tasks, such as rotation prediction, color matching, and cross-view correspondence, as image-instruction-response triplets, we introduce supervision that cannot be solved without relying on visual evidence. Our approach requires no human annotations, no architectural modifications, and no additional training stages. Across multiple models, training regimes, and benchmarks, injecting only a small fraction (3-10%) of such visually grounded instructions consistently improves performance on vision-centric evaluations. Our findings highlight instruction tuning with visually grounded SSL tasks as a powerful lever for improving visual reasoning in MLLMs through simple adjustments to the training data distribution. Code available at: https://github.com/sirkosophia/V-GIFT

1 Introduction

MLLMs often underuse rich visual representations during instruction tuning, weakening fine-grained vision-centric reasoning. V-GIFT addresses this by adding automatically supervised, visually grounded tasks to standard instruction tuning without architectural or optimization changes.

  • MLLMs struggle with fine-grained vision-centric tasks because visual information is often under-utilized during decoding despite rich encoder representations.
  • V-GIFT reformulates rotation prediction, color matching, and cross-view correspondence as image–instruction–response triplets that cannot be solved from language priors alone.
  • The method derives supervision automatically from image transformations or feature-based correspondences, requiring no human annotation.
  • V-GIFT retains standard autoregressive cross-entropy training while adjusting the instruction distribution to include vision-only problems.
  • 3–10% visually grounded SSL instructions consistently improve vision-centric reasoning across models, training regimes, and benchmarks with minimal additional compute.
  • The framework integrates into existing instruction-following pipelines without architectural modifications, auxiliary losses, or additional training stages.

2 Related Work

Prior work has addressed MLLM visual limitations through architecture changes, auxiliary objectives, and richer supervision. V-GIFT instead adjusts instruction-tuning supervision by embedding self-supervised visual tasks directly into the existing instruction format.

  • MLLMs and vision-centric strategies: MLLM research includes cross-attention and projection-based architectural strategies for improving visual perception and multimodal processing.
  • MLLMs and vision-centric strategies: Other approaches supervise visual tokens using reconstruction losses or distillation from external vision foundation models.
  • V-GIFT's positioning: V-GIFT differs by incorporating self-supervised tasks directly into instruction tuning without auxiliary losses, additional stages, or costly RLVR pipelines.
  • Self-supervised learning: SSL research spans annotation-free pretext tasks from rotation prediction and colorization to contrastive learning, clustering, self-distillation, and masked image modeling.
  • V-GIFT's task design: The reformulated tasks require orientation recognition, fine-grained color matching, and cross-view correspondence, compelling reliance on visual evidence rather than language priors.

3 Method

V-GIFT augments visual instruction tuning with automatically generated, visually grounded self-supervised instructions while preserving the standard MLLM pipeline and training objective. The tasks include rotation prediction, point-wise color matching, and cross-view point correspondence, requiring visual evidence for their answers.

  • Training pipeline: The standard LLaVA-style pipeline uses a pretrained vision encoder, projection module, and LLM decoder, followed by vision–language alignment and visual instruction tuning.V-GIFT operates exclusively during the instruction tuning phase.
  • Integration: V-GIFT adds automatically generated visually grounded SSL tasks to visual instruction tuning without changing the architecture or optimization procedure.The tasks use the same image–instruction–response format and autoregressive cross-entropy loss as standard multimodal instruction data.
  • Rotation prediction: Rotation prediction asks the model to identify whether an image was rotated by 0°, 90°, 180°, or 270°.Solving the task requires recognizing object orientation relative to canonical natural-image orientation.
  • Point-wise colorization: Point-wise color matching converts an image to grayscale, labels sampled locations, and asks the model to match each location with its original color.Distinct local RGB colors are randomly permuted and presented as candidate answers, requiring fine-grained visual discrimination and spatial grounding.
  • Point correspondence: Point correspondence asks the model to identify the matching point across two views of the same object instance.Pseudo-segmentation masks and dense DINOv2 features generate object-region correspondences, while distractors are sampled and randomly labeled.
  • Task coverage: Together, the SSL tasks train sensitivity to geometry, orientation, fine-grained appearance, spatial grounding, and cross-view correspondence.Their answers are deterministic and automatically derived from image structure, encouraging reliance on visual evidence rather than language priors.

4 Experiments

Experiments evaluate V-GIFT across models, training regimes, benchmarks, SSL-task combinations, injection ratios, and training controls. Visually grounded SSL instructions consistently improve vision-centric performance, with gains tied to the supervision rather than additional compute and strongest when injected during instruction tuning.

  • Experimental setup: Experiments span LLaVA-1.5 Vicuna and Qwen models, LLaVA-OneVision-1.5, full fine-tuning, LoRA adaptation, and vision-centric and general benchmarks.The evaluated vision-centric benchmarks are CVB-2D, POPE, MMStar, and BLINK; general benchmarks include MathVista, OCRBench, and RealWorldQA.
  • Main results: Across almost all evaluation settings, incorporating SSL instruction tuning consistently improves performance over baseline models, including the distinct LLaVA-OneVision-1.5 architecture.The result extends beyond a single decoder, implementation, training pipeline, or dataset.
  • Main results: Substantial performance improvements persist with LoRA fine-tuning, and V-GIFT achieves higher overall performance than VIRAL in that parameter-efficient setting.The comparison is made without additional objectives or architectural modifications.
  • General benchmarks: Improvements on MathVista, RealWorldQA, and OCRBench, with on-par results for LLaVA-1.5-Vicuna, indicate that vision-centric SSL tasks do not hurt general reasoning skills.The reported improvements differ by model: MathVista and RealWorldQA improve for LLaVA-1.5-Qwen, while OCRBench improves for LLaVA-OneVision-1.5.
  • SSL-task analysis: Each SSL task independently improves average baseline performance, while combining rotation, colorization, and correspondence produces stronger and more consistent gains across benchmarks.The three tasks provide complementary supervision signals when used jointly.
  • Injection ratio and controls: Performance improves at ρ = 1%, peaks at ρ = 10% for LLaVA-1.5-Qwen and ρ = 3% for LLaVA-OneVision-1.5, and gains arise from SSL supervision rather than extra compute.Matched-iteration controls show that additional training compute alone does not improve performance.
  • Training-stage analysis: SSL improves performance only when injected during instruction tuning; applying it before tuning matches baseline performance, while applying it afterward causes degradation from catastrophic forgetting.Reducing the post-tuning ratio from 10% to 1% does not remove the reported problem.
  • Training-data analysis: Multiple data sources, including multiple views of a single image, yield gains, suggesting visually grounded objectives matter more than dataset scale or diversity.These tasks force reliance on visual input because language priors alone are insufficient.

5 Conclusion

V-GIFT integrates self-supervised pretext tasks into visual instruction tuning without architectural or recipe modifications. Across vision-centric benchmarks, it consistently improves performance with minimal computational overhead and supports finer-grained visual grounding.

  • 5 Conclusion: V-GIFT reformulates rotation, colorization, and point-correspondence pretext tasks as instruction-following examples within visual instruction tuning.The framework requires no architectural modifications or changes to the training recipe.
  • 5 Conclusion: V-GIFT yields consistent gains across vision-centric MLLM benchmarks while maintaining competitive generic reasoning capabilities and requiring minimal additional computational overhead.Qualitative analyses indicate improved grounding in fine-grained visual information.

Colorization task.

The colorization task asks the model to recover colors at labeled locations from a grayscale image, using automatically constructed, visually grounded examples.

  • Five points are sampled from each non-grayscale COCO 2017 image, with each color computed as the mean RGB value over a 5 × 5 neighborhood.Sampled points are constrained to lie at least 20 pixels from the image boundary.
  • Sampled point colors must be pairwise distinct by at least δ = 40 in RGB space, enforced through rejection sampling.Each RGB value is also assigned a human-readable XKCD color name by nearest-neighbor retrieval.
  • The grayscale image receives labeled markers, while the five ground-truth colors are shuffled and presented as numbered RGB-and-name candidates.The model outputs the correspondence between each point label and its true color index.

Point correspondence task.

The point correspondence task identifies a matching point across paired images by restricting candidates to a shared pseudo-object region and comparing dense visual features.

  • The task uses paired COCO 2017 images with precomputed self-supervised segmentation masks and a fixed list of image pairs.Pseudo-segmentation mask extraction is taken from prior work.
  • The pseudo-class with the largest union of pixel regions across both masks is selected as the object of interest.The selected class k^* defines the relevant region for correspondence.
  • DINOv2-ViT-B/14 patch features are extracted for both images, and a query point is sampled from the selected region in the first image.The query feature is matched against candidate patch features in the second image using cosine similarity.
  • The corresponding point is the patch within the selected region of the second image having the highest feature similarity to the query.This selected point is then used to form the correspondence target.
  • Two distractor points are sampled from the same region, and the three candidates are shuffled and labeled 0, 1, and 2.Single-image models receive a side-by-side composite, whereas multi-image models receive the two images separately.
  • Rotation prediction uses one of four clockwise angles—0°, 90°, 180°, or 270°—and requires the model to output the corresponding degree value.The task is presented as a direct natural-language prediction instruction.

A.2 Single-image training data construction

Single-image training data is expanded by generating many augmented views from one high-resolution image through stochastic cropping and appearance transformations.

  • Each augmented sample applies a random resized crop with area sampled from 0.1% to 8% of the original image and aspect ratio sampled from 3/4 to 4/3.The crop is resized to 224 × 224 using bilinear interpolation.

A.3 Evaluation protocols

Evaluation uses standardized toolkit procedures across vision-centric and general multimodal benchmarks, while attention maps inspect visual-token focus in a representative model.

  • Evaluation protocols: VLMEvalKit supplies answer extraction and matching procedures, without relying on external API-based models.The evaluation reports benchmark-specific accuracy or normalized scores.
  • Vision-centric benchmarks: CVBench-2D accuracy covers 1,438 Count and Relation questions, while MMStar overall accuracy covers 1,500 questions across six visual reasoning dimensions.POPE mean accuracy averages the random, popular, and adversarial COCO-POPE splits.
  • Task coverage: The visually grounded training tasks include colorization point matching, point correspondence, and rotation prediction.These task types are illustrated in Figure 6.
  • Injection-ratio analysis: Table 9 evaluates how the SSL injection ratio ρ affects vision-centric instruction-following performance for LLaVA-OneVision-1.5.
  • General benchmarks: BLINK overall accuracy spans 14 visual perception subtasks; MathVision accuracy spans 3,040 questions across 16 mathematical sub-fields.OCRBench is reported using its normalized final score.
  • Attention analysis: Attention maps visualize the last instruction token’s attention to visual tokens, selecting the layer with the lowest spatial entropy.The comparison is between baseline and V-GIFT in LLaVA-1.5 Vicuna 7B on random CV-Bench2D samples.

B Additional Results

This section presents detailed results from the ρ sensitivity study, complementing Figure 3 of the main paper.

  • Table 9 reports detailed results for the ρ sensitivity study.
  • The study examines the ρ parameter.
  • These results complement Figure 3 of the main paper.
Loading 2604.12966v1…