PaperlayerView exact revision on arXiv

Source-linked AI summary

Reconstructive Visual Instruction Tuning

Haochen Wang, Anlin Zheng, Yucheng Zhao, Tiancai Wang, Zheng Ge, Xiangyu Zhang, Zhaoxiang Zhang

arXiv:2410.09575v2cs.CVcs.AIcs.CLcs.LG

TL;DR

Text-only supervision leaves visual outputs unused, motivating ROSS to supervise them through image reconstruction. ROSS denoises latent image representations conditioned on visual outputs, and achieves strong comprehension and hallucination results with a single SigLIP encoder. Its scope is multimodal comprehension rather than photorealistic image generation.

  • Problem

    Conventional LMM instruction tuning supervises text outputs while leaving rich visual outputs unused, limiting vision-centric supervision.

  • Method

    ROSS trains a denoiser to reconstruct clean latent visual representations from input images, conditioned on the LMM’s visual outputs.

  • Results

    ROSS improves multimodal comprehension and reduces hallucinations, outperforming state-of-the-art alternatives under most cases with only a single SigLIP visual encoder.

  • Takeaways & Limitations

    Vision-centric reconstructive supervision can provide effective intrinsic activation without aggregating multiple visual experts.

  • Takeaways & Limitations

    ROSS does not provide photorealistic image-generation capabilities and is designed for enhanced multimodal comprehension instead.

Abstract

from arXiv · show

This paper introduces reconstructive visual instruction tuning (ROSS), a family of Large Multimodal Models (LMMs) that exploit vision-centric supervision signals. In contrast to conventional visual instruction tuning approaches that exclusively supervise text outputs, ROSS prompts LMMs to supervise visual outputs via reconstructing input images. By doing so, it capitalizes on the inherent richness and detail present within input images themselves, which are often lost in pure text supervision. However, producing meaningful feedback from natural images is challenging due to the heavy spatial redundancy of visual signals. To address this issue, ROSS employs a denoising objective to reconstruct latent representations of input images, avoiding directly regressing exact raw RGB values. This intrinsic activation design inherently encourages LMMs to maintain image detail, thereby enhancing their fine-grained comprehension capabilities and reducing hallucinations. Empirically, ROSS consistently brings significant improvements across different visual encoders and language models. In comparison with extrinsic assistance state-of-the-art alternatives that aggregate multiple visual experts, ROSS delivers competitive performance with a single SigLIP visual encoder, demonstrating the efficacy of our vision-centric supervision tailored for visual outputs.

1 INTRODUCTION

ROSS addresses visual shortcomings in text-supervised multimodal instruction tuning by adding intrinsic reconstructive supervision for visual outputs. Using latent denoising with a single visual encoder, it improves fine-grained comprehension and reduces hallucinations while outperforming comparable multi-expert alternatives.

  • Conventional visual instruction tuning is LLM-centric: visual encoders provide prefix tokens, but supervision derives exclusively from text outputs.
  • ROSS supervises visual outputs by reconstructing input images, using their rich details to encourage low-level preservation, finer-grained comprehension, and fewer hallucinations.
  • ROSS replaces redundant raw-image feedback with latent visual-token reconstruction and denoising to provide more meaningful vision-centric supervision.
  • ROSS uses intrinsic activation with one visual encoder instead of aggregating several external visual experts, simplifying adaptation to new visual domains.
  • ROSS achieves strong results across multimodal benchmarks, including fine-grained vision and hallucination evaluations, and significantly outperforms similar-sized multi-expert alternatives.

2 RELATED WORK

Related work includes conventional plug-in visual instruction tuning, stronger or aggregated visual encoders, and generative objectives. ROSS differs by targeting multimodal comprehension through reconstructive supervision rather than image generation.

  • Most visual instruction tuning LMMs use a plug-in architecture with language-supervised visual encoders, connectors, and LLMs.
  • Some approaches strengthen LMM vision using improved backbones or extrinsic assistance from multiple vision-only experts.
  • Generative LMMs incorporate text-to-image diffusion models for joint comprehension and generation, whereas ROSS uses reconstruction to catalyze multimodal comprehension.

3 PRELIMINARIES

LMMs extend causal language modeling by converting images into visual prefix tokens before text generation. Standard training supervises only the text continuation, leaving visual outputs unsupervised.

  • An LMM projects an input image into visual tokens with a visual encoder and multimodal projector before conditioning the LLM.
  • The standard LMM training recipe includes pre-training or alignment followed by supervised instruction tuning from language answers such as VQA responses.
  • Only text outputs after the visual-token prefix are supervised, while the corresponding visual outputs remain outside the likelihood objective.

4 ROSS: RECONSTRUCTIVE VISUAL INSTRUCTION TUNING

ROSS adds reconstructive supervision to ordinary next-token prediction, exploring targets and objectives that handle spatial redundancy. Its final design denoises latent tokens conditioned on visual outputs through a learned denoiser.

  • 4 ROSS: RECONSTRUCTIVE VISUAL INSTRUCTION TUNING: ROSS combines ordinary text prediction with a reconstructive loss applied to visual outputs and image-derived reconstruction targets.
  • 4 ROSS: RECONSTRUCTIVE VISUAL INSTRUCTION TUNING: The reconstruction target is controlled by a teacher tokenizer or visual model, while the measurement may use pixel regression, latent similarity, or denoising.
  • 4.1 ROSSR: REGRESSING AS RECONSTRUCTIVE VISUAL INSTRUCTION: ROSSR explores pixel, latent, and latent-to-pixel regression, but direct RGB regression can yield weak supervision because natural images contain heavy spatial redundancy.
  • 4.1 ROSSR: REGRESSING AS RECONSTRUCTIVE VISUAL INSTRUCTION: Latent reconstruction uses fine-grained tokens from models such as VAE, VQGAN, DINOv2, or DEiT-III, with cosine similarity for latent regression.
  • 4.2 ROSSD: DENOISING AS RECONSTRUCTIVE VISUAL INSTRUCTION: ROSSD replaces vanilla regression with denoising, whose noise injection acts as implicit augmentation and regularization while encouraging focus on the underlying data manifold.
  • 4.2 ROSSD: DENOISING AS RECONSTRUCTIVE VISUAL INSTRUCTION: ROSSD conditions on high-level visual outputs to recover clean latent tokens from noisy tokens, using a VAE-derived tokenizer by default.
  • 4.2 ROSSD: DENOISING AS RECONSTRUCTIVE VISUAL INSTRUCTION: The denoiser uses Transformer self-attention and projections for visual conditions, noisy tokens, and timesteps; alternative tokenizers remain possible, with KL-16 selected as best.

5 EXPERIMENTS

ROSS consistently improves multimodal comprehension through vision-centric reconstructive supervision, with the strongest formulation using KL-16 latent denoising. It improves attention to relevant image content, transfers across models and encoders, and surpasses strong alternatives with a single SigLIP encoder in most settings.

  • Ablation study: ROSSD, combining KL-16 latent targets with denoising, significantly improves over vanilla regression and becomes the final ROSS formulation.Denoising avoids overfitting exact latent token values, while KL-16 is selected for preserving image details.
  • In-depth analysis: ROSS attention values are significantly higher than LLaVA’s, indicating stronger focus on image regions relevant to the text query.Both mean and median attention scores are higher, and qualitative attention maps show more targeted visual focus.
  • In-depth analysis: Reconstructive objectives improve comprehension, whereas generative alternatives do not under the compared setup.The generative alternative requires additional creation data, while reconstruction can use the original supervised fine-tuning data.
  • Ablation study: ROSS yields significant improvements across different LLMs and visual encoders, especially on fine-grained comprehension benchmarks such as MMVP and ChartQA.The experiments use varied model configurations and report consistent gains in most cases.
  • In-depth analysis: ROSS features can be projected back into pixel space, with reasonable ImageNet-1K reconstructions demonstrating that high-level features retain image details.The reconstruction results use ROSS-7B on the ImageNet-1K validation set.
  • Comparison with state-of-the-arts: With a single SigLIP encoder, ROSS surpasses Cambrian-1 in most settings and even exceeds GPT-4V-1106 and Gemini-1.5 Pro on several benchmarks.ROSS also uses 7M instruction-tuning examples and maintains a lightweight inference procedure.

6 CONCLUSION

ROSS supervises visual outputs through reconstructive denoising, recovering clean latent image representations from high-level visual outputs. It improves comprehension and reduces hallucinations, while showing that comprehension models can support image generation without dedicated generation experts.

  • 6 CONCLUSION: ROSS uses a vision-centric reconstructive objective that denoises latent visual representations rather than directly regressing redundant RGB values.The denoiser conditions on high-level visual outputs to recover clean latent tokens.
  • 6 CONCLUSION: The objective enhances multimodal comprehension and reduces hallucinations.
  • 6 CONCLUSION: ROSS outperforms state-of-the-art alternatives in most cases using only a single SigLIP visual encoder.
  • 6 CONCLUSION: ROSS-7B high-level features contain sufficient detail for low-level image reconstruction without generation experts such as Stable Diffusion.

B IMPLEMENTATION DETAILS

The implementation largely follows established LLaVA-v1.5 configurations while using SigLIP, Qwen2 or Vicuna, mixed instruction data, and latent-token denoising targets. The added denoising process increases training time by approximately 10%.

  • B IMPLEMENTATION DETAILS: SigLIP represents a 384 × 384 image with 729 tokens, which are used as fine-grained denoising targets.
  • B IMPLEMENTATION DETAILS: The evaluation setup reformats MMVP questions into multiple-choice prompts and computes accuracy by exact matching.
  • B IMPLEMENTATION DETAILS: Denoising adds approximately 10% training time compared with the baseline, while its benefits outweigh the additional cost.
  • B IMPLEMENTATION DETAILS: ROSS remains robust across alternative denoising schedules.

C.1 MORE ABLATIONS

The ablations show that latent denoising, tokenizer choice, self-attention, and reconstructive supervision materially affect ROSS performance. Reconstructive supervision also avoids the weak input-target correspondence and specialized data requirements associated with generative alternatives.

  • C.1 MORE ABLATIONS: KL-16 outperforms VQ-16, achieving 0.87 rFID versus 2.19 on ImageNet 256×256 validation data.
  • C.1 MORE ABLATIONS: Self-attention is crucial because causal visual outputs require modeling inter-token discrepancies.
  • C.1 MORE ABLATIONS: ROSS consistently improves the baseline across denoising schedules, demonstrating robustness to the choice of β schedule.
  • C.1 MORE ABLATIONS: Generative objectives require specialized creation data and exhibit weaker correspondence between inputs and supervision targets.
  • C.1 MORE ABLATIONS: Reconstructive methods use original input images as direct auxiliary supervision, which supports accurate multimodal comprehension and interpretation.
  • C.1 MORE ABLATIONS: Vision-centric supervision yields especially significant improvements on HallusionBench, MMVP, ChartQA, and OCRBench.

C.2 COMPARISON ON HIGH-RESOLUTION BENCHMARKS

ROSS extends to high-resolution inputs by reconstructing each image crop with its denoising objective. The supplied passages also show focused gradient behavior on relevant regions and improvements across model and data scales.

  • C.2 COMPARISON ON HIGH-RESOLUTION BENCHMARKS: ROSS-anyres reconstructs each 384×384 crop and uses up to 3,645 visual tokens for high-resolution images.
  • C.2 COMPARISON ON HIGH-RESOLUTION BENCHMARKS: ROSS-7Banyres surpasses LLaVA-v1.6-7B and Cambrian-1-8B on high-resolution benchmarks.
  • C.2 COMPARISON ON HIGH-RESOLUTION BENCHMARKS: ROSS improves over the baseline across different model sizes in most cases.
  • C.2 COMPARISON ON HIGH-RESOLUTION BENCHMARKS: ROSS consistently brings significant improvements as training-data scale increases.
  • C.2 COMPARISON ON HIGH-RESOLUTION BENCHMARKS: GradCAM visualizations show increasingly focused gradients on relevant regions such as the spider web.

C.3 MORE ANALYSIS

ROSS improves multimodal performance across language, scaling, and fine-grained evaluation settings while preserving language capabilities. Its vision-centric supervision also produces gradients that focus more appropriately on relevant image regions.

  • Qualitative Analysis: Qualitative comparisons cover HallusionBench, MMBench, MMVP, and SpatialBench, including settings with depth-map understanding.
  • Language Capabilities: ROSS does not harm language capabilities, bringing improvements in most cases across multimodal knowledge, general understanding, and instruction-following benchmarks.
  • Scaling Properties: ROSS consistently improves performance as the training data scale increases and improves over the LLaVA baseline across Qwen2.5 model sizes.
  • Qualitative Analysis: On SpatialBench, adding the MiDaS-3.0 depth expert sometimes harms comprehension rather than improving it.
  • Gradient Analysis: ROSS produces more reasonable gradients that increasingly focus on relevant image regions, such as the spider web, during VQA.

C.4 QUALITATIVE COMPARISONS

ROSS shows stronger qualitative comprehension than instruction-tuning and extrinsic-assistance baselines across several visual benchmarks. The comparisons highlight reduced hallucinations, broader reasoning and spatial abilities, and improved visual-pattern recognition, while ROSS remains limited to comprehension rather than image generation.

  • Qualitative Comparisons: ROSS correctly answers some questions with reduced hallucinations even when GPT-4V fails, while Cambrian-1 can fail to follow instructions.
  • Qualitative Comparisons: Across MMBench examples, ROSS demonstrates enhanced reasoning, low-level comprehension, and spatial understanding skills.
  • Qualitative Comparisons: ROSS recognizes varied visual patterns on MMVP, suggesting its reconstructive objective addresses shortcomings of the original visual encoder.
  • Qualitative Comparisons: On SpatialBench, the MiDaS depth expert fails to improve depth-map understanding both quantitatively and qualitatively.
  • Discussion: ROSS does not provide photorealistic image-generation capabilities because it is designed for enhanced multimodal comprehension.
Loading 2410.09575v2…