PaperlayerView exact revision on arXiv

Source-linked AI summary

LLaVA-UHD v4: What Makes Efficient Visual Encoding in MLLMs?

Kechen Fang, Yihua Qin, Chongyi Wang, Wenshuo Ma, Tianyu Yu, Yuan Yao

arXiv:2605.08985v1cs.CV

TL;DR

High-resolution MLLMs incur large visual-token and ViT-computation costs because global encoding is followed by compression only after the encoder. LLaVA-UHD v4 combines slice-based encoding with early intra-ViT compression, reducing visual-encoding FLOPs by 55.75% while matching or surpassing post-ViT baseline performance.

  • Problem

    Global encoding creates image-area-scaled token sequences, while post-ViT compression leaves the ViT’s heavy internal computation unchanged.

  • Method

    LLaVA-UHD v4 combines slice-based encoding with a parameter-reusing intra-ViT compressor inserted in shallow ViT layers.

  • Results

    55.75% visual-encoding FLOPs reduction is achieved while LLaVA-UHD v4 matches or surpasses the post-ViT baseline across eight benchmarks at 16× compression.

  • Takeaways & Limitations

    The results support performing aggressive token reduction inside the vision encoder without sacrificing fine-grained downstream perception.

  • Takeaways & Limitations

    The compressor uses a fixed uniform spatial downsampling rate and has an insertion depth empirically selected for the SigLIP 2 backbone.

Abstract

from arXiv · show

Visual encoding constitutes a major computational bottleneck in Multimodal Large Language Models (MLLMs), especially for high-resolution image inputs. The prevailing practice typically adopts global encoding followed by post-ViT compression. Global encoding produces massive token sequences, while post-ViT compression incurs the full quadratic attention cost of the ViT before any token reduction takes place. In this work, we revisit this convention along two dimensions: the encoding strategy and visual token compression. First, controlled experiments show that slice-based encoding outperforms global encoding across benchmarks, suggesting that preserving local details through sliced views can be more beneficial than applying global attention for fine-grained perception. Second, we introduce intra-ViT early compression, which reduces tokens in shallow ViT layers and substantially lowers visual-encoding FLOPs while preserving downstream performance. By integrating intra-ViT compression into the slice-based encoding framework, we present LLaVA-UHD v4, an efficient and compute-controllable visual encoding scheme tailored for high-resolution inputs. Across a diverse set of benchmarks covering document understanding, OCR, and general VQA, LLaVA-UHD v4 reduces visual-encoding FLOPs by 55.8% while matching or even surpassing baseline performance. These results suggest that visual-encoding efficiency can be substantially improved without sacrificing downstream performance, providing a practical design direction for efficient high-resolution MLLMs. All model weights and code will be publicly released to support further research.

1 Introduction

High-resolution MLLMs face exploding visual-token sequences and expensive ViT computation under global encoding with post-ViT compression. LLaVA-UHD v4 instead combines slice-based encoding with early intra-ViT compression, preserving accuracy while reducing visual-encoding cost.

  • Motivation: Global encoding scales visual tokens with image area, while post-ViT compression leaves the ViT’s full computation unchanged.The prevailing pipeline reduces tokens only after the vision encoder has processed the complete high-resolution image.
  • Encoding paradigm: Slice-based encoding consistently outperforms global encoding across diverse benchmarks and avoids global encoding’s quadratic attention growth.The authors report that partitioning large images into slices provides sufficiently informative representations while reducing per-forward attention cost.
  • Method: The proposed early compressor combines window attention with a downsampling MLP inserted into shallow ViT layers.It reuses adjacent pretrained-layer weights and compresses tokens by 4× early enough for most later layers to process fewer tokens.
  • Results: At a shared 16× compression ratio, LLaVA-UHD v4 matches or surpasses the post-ViT baseline across eight standard benchmarks.The comparison uses the same overall downstream token budget.
  • Results: 55.75% acceleration in visual-encoding FLOPs is achieved while maintaining competitive downstream performance.The result follows from integrating slice-based encoding with the intra-ViT early compressor.

2 Rethinking High-Resolution Visual Encoding

The paper evaluates high-resolution encoding and connector compression under matched conditions. Slice-based encoding consistently beats global encoding, while pixel-unshuffle MLP connectors provide a stronger post-ViT baseline than query-based resamplers, motivating compression inside the ViT.

  • Experimental setup: The controlled study fixes the ViT, LLM, training data, and visual-token budget to isolate encoding and compression choices.The experiments use SigLIP 2 and Qwen3 as default backbones.
  • Slice-based Encoding: Slice-based encoding outperforms global encoding by 0.5–1.7 average-accuracy points across compression rates and data scales.Its advantage increases with training data scale in the reported SigLIP-2 sweep.
  • Slice-based Encoding: Slice-based encoding retains an approximately +1.5-point average margin with MoonViT across 8M and 16M data scales.This robustness test also evaluates an alternative higher-resolution slicing schedule at 16× compression.
  • Slice-based Encoding: The reported slice-based advantage holds across compression rates, vision backbones, and image resolutions.The analysis attributes this pattern to preserving locality and focusing encoder capacity on fine-grained patterns within each slice.
  • Connector Compression: Post-ViT compression reduces downstream LLM load but leaves the ViT’s internal computation unchanged, motivating intra-ViT compression.The section therefore treats slice-based encoding plus an MLP connector as the baseline for shifting token reduction into the encoder.
  • Connector Compression: Pixel-unshuffle MLP connectors outperform query-based resamplers in every tested configuration.The MLP lead ranges from +3.3 to +6.7 points at 4× compression and narrows to +0.4 points at 16× compression with 16M training data.

3 LLaVA-UHD v4

LLaVA-UHD v4 combines slice-based inputs, an intra-ViT compressor, and an MLP connector in a two-stage compression pipeline. The compressor aggregates local context, reduces tokens early, and lets most ViT layers operate at lower resolution.

  • 3.1 Overview: LLaVA-UHD v4 adds an intra-ViT compressor D to the slice-based encoding and MLP-connector baseline.The architecture is designed for efficient, compute-controllable high-resolution visual encoding.
  • 3.1 Overview: D applies local window attention followed by lightweight MLP-based downsampling before the remaining ViT layers.The input pipeline uses a thumbnail and aspect-ratio-aware high-resolution slices processed with per-view attention locality.
  • 3.1 Overview: The intra-ViT and post-ViT stages jointly reduce raw visual patches to the final LLM token sequence.The early stage reduces encoder-side tokens, while the MLP connector projects compressed visual features into language-model space.
  • 3.2 Design Principles: Early placement makes subsequent ViT layers process only 25% of the original tokens.The design targets the encoder’s computation rather than only reducing the LLM’s input burden.
  • 3.2 Design Principles: The compressor’s insertion depth balances compute savings against representational depth, with k=6 identified as the best efficiency-quality trade-off.Earlier insertion increases savings, while deeper placement retains more full-resolution pretrained processing.
  • 3.2.2 Parameter-Reuse Initialization: Parameter reuse initializes D from the preceding pretrained ViT layer to keep fine-tuning near the original representation manifold.The copied components include attention projections, head configuration, and LN1; the attention mask is changed to 2×2 local windows.
  • 3.2.1 Compressor Structure: Window attention exchanges information within non-overlapping 2×2 neighborhoods before PixelUnshuffle reshapes four tokens into one.The resulting 4d channels are fused back to dimension d by an MLP.

4 Experiment

Experiments compare LLaVA-UHD v4 with post-ViT compression under matched training and token-budget settings, finding substantial compute savings with comparable accuracy. Ablations identify the compressor design and insertion depth that best preserve performance.

  • Main results: 55.75% lower visual-encoding FLOPs, from 3555G to 1573G, while accuracy stays within ±0.8 points of the post-ViT baseline across five training scales.Both systems use identical training settings and an end-to-end 16× compression ratio.
  • Main results: LLaVA-UHD v4 preserves average scaling behavior from 4M to 64M training samples, with the average gap remaining within ±0.8 points.The post-ViT baseline increases from 68.2 to 76.2 average points, while LLaVA-UHD v4 increases from 67.4 to 75.6.
  • Compressor ablations: Naive in-ViT merging reduces FLOPs from 3555.1G to 1401.2G but lowers average accuracy below the post-ViT baseline.Average pooling scores 69.6 versus 70.6 for the baseline, while pixel-unshuffle MLP reaches 69.8.
  • Compressor ablations: 70.7 average accuracy results when window attention and reuse initialization are combined, exceeding either modification alone and slightly surpassing the post-ViT baseline.Reuse alone reaches 69.9 and window attention alone 70.1.
  • Compressor ablations: Direct cross-attention is query-sensitive: top-left queries reach 70.5 average accuracy, while window-mean queries fall to 69.8 under the same FLOPs.Window attention followed by a reuse-initialized MLP achieves 70.7 and remains strongest across tested scales.
  • Insertion depth: Insertion after layer 6 is Pareto-favorable, whereas layer 3 collapses average accuracy to 38.76 and deeper insertion increases FLOPs without improving accuracy.Insertion at layers 9 and 15 raises FLOPs to 1901G and 2557G, respectively.

5 Conclusion

LLaVA-UHD v4 combines slice-based encoding with parameter-reusing intra-ViT early compression to address high-resolution visual-encoding cost. The paper reports a large FLOPs reduction while maintaining or exceeding strong post-ViT baselines, with dynamic compression left for future work.

  • Conclusion: LLaVA-UHD v4 combines slice-based encoding with a parameter-reusing intra-ViT early compressor for efficient high-resolution visual encoding.The compressor reduces tokens inside the vision encoder rather than only after it.
  • Conclusion: 55.75% lower visual-encoding FLOPs under a 16× compression ratio, while fine-grained downstream performance matches or surpasses strong post-ViT baselines.This is the principal efficiency-quality result reported in the conclusion.
  • Conclusion: The current module uses a fixed compression rate, leaving dynamic, content-aware token reduction within the encoder for future research.This is the stated scope boundary in the conclusion.
  • Conclusion: The results suggest that aggressive token reduction can occur inside the vision encoder without sacrificing fine-grained perception.The paper presents this as a practical path toward more scalable multimodal foundation models.

B.1 Detailed Analysis of Encoding Strategies

Detailed analyses attribute slice-based encoding’s advantage primarily to local attention context rather than the compression ratio itself. Local views better support fine-grained patterns, while the tested MLP connector remains a strong baseline across compression settings.

  • Encoding strategies: Slice-based encoding improves average score over global encoding across evaluated backbones and slicing schedules, although individual benchmark outcomes remain mixed.The advantage persists with MoonViT and a higher-resolution slicing variant.
  • Encoding strategies: The key distinction is attention context: global encoding processes all patches jointly, whereas slice-based encoding preserves spatially coherent local views before compression.Both settings use a locality-preserving pixel-unshuffle MLP compressor.
  • Encoding strategies: Slice-based encoding encourages extraction of text, chart marks, and dense document patterns within local neighborhoods before spatial compression.Global attention preserves unrestricted interactions but may dilute the inductive bias toward local structure.
  • Connector baseline: The MLP connector is used as the post-ViT baseline because its spatially structured local merging preserves coarse layout, especially at 4× compression.The comparison covers multiple connector designs and downsampling rates.

B.2 Detailed Results of Connector Designs

Connector analyses compare MLP downsampling with resampling under slice-based encoding and varying compression rates. The results favor the simpler locality-preserving MLP, although the gap narrows as compression becomes more aggressive.

  • Connector comparison: The MLP connector’s largest gains appear at 4× compression, where each output token remains tied to a fixed local image neighborhood.This built-in spatial structure preserves a relatively rich coarse layout.
  • Connector comparison: At 16× compression, the connector gap narrows but does not reverse, with MLP remaining slightly ahead even for the resampler’s most favorable 16M-data setting.More aggressive compression forces both connectors to discard additional spatial detail.

B.3 Additional Ablations on the Open-Source LLaVA-OneVision Setting

Additional ablations in the open-source LLaVA-OneVision setting broadly confirm that local interaction before token reduction is more robust than naive MLP merging. The strongest average score comes from combining window attention with reused MLP parameters, although benchmark-level results are somewhat noisy.

  • The plain MLP merger lowers the baseline average from 62.1 to 60.4, showing that learnable in-ViT merging alone is insufficient.
  • Designs with local interaction before token reduction are substantially more robust than the plain MLP variant.The evaluated alternatives include cross-attention and window-attention designs.
  • Win-Attn w/ Reused MLP achieves the best average score, improving the baseline from 62.1 to 63.4.
  • Parameter reuse improves the average score from 63.2 to 63.4 compared with Win-Attn w/ MLP.The gain is modest but consistent with complementary local contextualization and parameter-reuse initialization.
  • The mixed per-benchmark pattern indicates that the open-source setting is somewhat noisier despite the best average result favoring reused window attention.

C Hyperparameters

The four-stage training recipes use a progression from vision-language alignment to high-quality image training and supervised instruction tuning. Detailed settings cover optimization schedules, training duration, warmup, trainable modules, and effective per-GPU batch size for two data settings.

  • Both training recipes begin with a warmup stage for vision-language alignment.
  • The recipes continue with high-quality image training and end with supervised instruction tuning.
  • The hyperparameter tables report learning-rate schedules, training length, warmup steps, trainable modules, and packing-equivalent per-GPU batch size.These settings are provided for the in-house data and LLaVA-OneVision training settings, respectively.

D Limitations

The paper identifies three limitations: fixed spatial downsampling, backbone-specific compressor depth, and fragmented context across slice boundaries. These constraints motivate adaptive compression, backbone-specific tuning, and improved global interaction.

  • The intra-ViT compressor applies a fixed, uniform spatial downsampling rate that does not adapt to varying information density across an image.The paper suggests allocating more tokens to dense text and fewer to plain backgrounds as a future direction.
  • The optimal compressor insertion depth, empirically set to k = 6 for SigLIP 2, may require reevaluation for architecturally distinct or substantially deeper vision encoders.
  • Slice-based encoding fragments high-resolution context across slice boundaries and relies primarily on the low-resolution thumbnail to bridge global interactions.
Loading 2605.08985v1…