PaperlayerView exact revision on arXiv

Source-linked AI summary

ViCO: A Training Strategy towards Semantic Aware Dynamic High-Resolution

Long Cui, Weiyun Wang, Jie Shao, Zichen Wen, Gen Luo, Linfeng Zhang, Yanting Zhang, Yu Qiao, Wenhai Wang

arXiv:2510.12793v2cs.CV

TL;DR

Image inputs increase MLLM inference costs through added visual tokens. ViCO trains consistency across compression rates and routes patches according to semantic complexity; it reports reducing visual tokens by up to 50% while maintaining perception, reasoning, and OCR capabilities.

  • Problem

    Image inputs add visual tokens that increase MLLM inference costs, especially when multiple images are processed.

    Source: §1 · p. 1

  • Method

    ViCO minimizes response-distribution gaps across compression rates and uses a Visual Resolution Router to allocate tokens to patches according to semantic complexity.

    Source: §1 · pp. 1–2

  • Results

    ViCO reduces visual tokens by up to 50% while maintaining perception, reasoning, and OCR capabilities.

    Source: §1 · p. 2, §5 · p. 9

  • Takeaways & Limitations

    ViCO allocates more visual tokens to informative regions and fewer to simpler ones, reducing inference-time computational cost.

    Source: §5 · p. 9

Abstract

from arXiv · show

Existing Multimodal Large Language Models (MLLMs) suffer from increased inference costs due to the additional vision tokens introduced by image inputs. In this work, we propose Visual Consistency Learning (ViCO), a novel training algorithm that enables the model to represent images of varying semantic complexities using different numbers of vision tokens. The key idea behind our method is to employ multiple MLP connectors, each with a different image compression ratio, to downsample the vision tokens based on the semantic complexity of the image. During training, we minimize the KL divergence between the responses conditioned on different MLP connectors. At inference time, we introduce an image router, termed Visual Resolution Router (ViR), that automatically selects the appropriate compression rate for each image patch. Compared with existing dynamic high-resolution strategies, which adjust the number of visual tokens based on image resolutions, our method dynamically adapts the number of visual tokens according to semantic complexity. Experimental results demonstrate that our method can reduce the number of vision tokens by up to 50% while maintaining the model's perception, reasoning, and OCR capabilities. We hope this work will contribute to the development of more efficient MLLMs. The code and models will be released to facilitate future research.

1. Introduction

Image inputs make visual tokens a major source of MLLM inference cost, especially when processing multiple images. ViCO allocates tokens according to patch semantics and reports substantial token reduction while maintaining performance across perception, reasoning, and OCR tasks.

1. Introduction

  • Image inputs increase inference costs, and InternVL3.5 can use up to 3,328 visual tokens per image, making vision tokens the primary inference-cost source in multi-image settings.The stated maximum comes from up to 13 patches, each represented by 256 visual tokens.Source: §1 · p. 1
  • ViCO routes patches with complex semantics to 256-token representations and simpler patches to 64-token representations, unlike resolution-based allocation strategies.Consistency Training minimizes the KL divergence between responses under different compression rates; Router Training selects a compression rate per patch.Source: §1 · pp. 1–2
  • Up to 50% fewer visual tokens and nearly doubled inference throughput are reported while preserving perception, reasoning, and OCR capabilities.The reported throughput improvement is relative to the original InternVL3.5 series across model scales.Source: Figure 1 · p. 2, §1 · p. 2
  • Visual Consistency Learning minimizes response-distribution gaps across compression rates to support highly compressed representations without significant performance loss.The Visual Resolution Router allocates tokens by patch semantic complexity to trade off efficiency and accuracy.Source: §1 · p. 2

2. Related Work

Prior MLLMs use varied architectures and often rely on resolution-based image patching, which increases visual-token counts. Related efficiency work compresses or removes visual tokens, but preserving fine-grained details remains important for vision-sensitive tasks such as OCR.

2.1. Multimodal Large Language Models.

  • MLLMs use connectors, fusion layers, or unified Transformers, while many dynamic high-resolution designs segment images by resolution and incur high visual-token costs.ViCO is presented as compatible with this dynamic-resolution paradigm while adding semantic-level adaptivity.Source: §2.1 · pp. 2–3

2.2. Efficient Vision Language Models.

  • Token-compression methods reduce visual redundancy through context tokens, adaptive pooling, cross-modal feature alignment, pruning, merging, or selection.FastV, SparseVLM, and ToMe are described as training-free approaches.Source: §2.2 · p. 3
  • Aggressive compression can lose fine-grained spatial and textual information, limiting these strategies on vision-sensitive tasks such as OCR.The related work frames balancing efficiency with preservation of detailed visual representations as a need.Source: §2.2 · p. 3

3. Visual Consistency Learning

ViCO represents patches with different numbers of visual tokens according to their semantic complexity. It trains the model for consistency across compression rates, then trains a router to choose patch resolutions based on compression impact.

3. Visual Consistency Learning

  • ViCO enables images with different semantic complexities to be represented using different numbers of vision tokens.The method is presented as a dynamic resolution strategy compatible with existing MLLMs.Source: §3 · p. 3

3.1. Consistency Training

  • Consistency Training minimizes KL divergence between the trained model's response distribution and a frozen reference model's distribution under sampled patch-compression rates.The reference uses uncompressed visual tokens; compressed patches use 64 tokens and the others use 256.Source: §3.1 · p. 3

3.2. Router Training

  • ViR uses each patch's relative loss increase under compression to label low-impact patches for aggressive compression and high-impact patches for conservative compression.The consistency loss is the basis for the loss ratio, which measures patch sensitivity to compression.Source: §3.2 · p. 3
  • A sliding-window percentile threshold targets roughly half the patches for compression, and training estimates each sample's label from a randomly selected patch.ViR is a binary classifier trained with cross-entropy while the MLLM backbone remains frozen; patch-level features are pooled and passed through an MLP.Source: §3.2 · pp. 3–4

4. Experiments

ViCO is evaluated across general, OCR, reasoning, multi-image, and video benchmarks, with performance largely retained across model scales and tasks. The experiments also examine whether patch-wise semantic compression preserves performance while reducing visual-token use.

4. Experiments

  • Evaluation spans general, OCR, reasoning, multi-image, and video benchmarks; training proceeds through consistency learning and then router learning.The consistency-learning stage uses a maximum text length of 32K; router learning trains only the lightweight router.Source: §4 · p. 4, §4.1 · pp. 4–5

4.2.1. Results on General Benchmarks

  • Across benchmarks, ViCO models retain over 99.6% of original performance on average, consistently from 4B to 241B-MoE.The reported average retention across all model sizes is about 99.7%.Source: §4.2.1 · p. 5

4.2.2. Results on OCR-related Benchmarks

  • 71% patch compression on OCRBench retains virtually the same 8B-model performance (84.0 vs. 83.9).The routing strategy more aggressively compresses simpler patches while preserving complex patches with critical semantic information.Source: §4.2.2 · p. 5

4.2.3. Results on Multi-Image and Video Understanding Benchmarks

  • On Video-MME with subtitles, the 241B-MoE model compresses approximately 70% of tokens and scores 76.6, slightly surpassing its original model.The 38B model compresses 63% of Video-MME tokens while maintaining almost 100% of its performance.Source: §4.2.3 · p. 5

4.3. Comparison with Existing Methods

The comparison study finds that fixed token-reduction approaches can falter on vision-sensitive benchmarks, while ViCO adapts compression to semantic importance. Ablations report benefits from consistency training and patch-level routing over indiscriminate, random, or image-level choices.

4.3. Comparison with Existing Methods

  • At comparable average compression, FastV and SparseVLM show substantial drops on vision-sensitive benchmarks such as OCR, while ViCO retains more tokens for critical tasks.The comparison describes FastV and SparseVLM as using manually predefined hyperparameters, unlike ViCO’s dynamic semantic assessment.Source: §4.3 · p. 5, Table 3 · p. 7

4.4. Ablation Study

  • OCRBench falls from 84.0 to 75.1 when the original model receives interleaved compression rates, while disabling dynamic resolution also degrades vision-sensitive benchmarks.The ablation describes consistency training as necessary for handling interleaved tokens at varying downsampling rates while maintaining performance.Source: §4.4 · p. 6, Table 3 · p. 7
  • 95.1% retention with indiscriminate compression and 97.6% with random routing trail the full ViCO router’s 99.6% retention.The random-routing variant keeps the overall compression ratio identical to ViCO with its Visual Resolution Router.Source: §4.4 · p. 7, Table 3 · p. 7
  • Image-level routing retains 98.8% versus 99.6% for patch-level routing, while compressing 12% of InfoVQA patches and 88% of BLINK patches.The study characterizes this dataset-dependent variation as unstable compression behavior and patch-level routing as more balanced.Source: §4.4 · p. 7, Table 3 · p. 7

4.5. Visualization of Image Routing

ViR routes patches according to semantic content, preserving salient regions at lower compression and compressing less critical areas more strongly. In deployment tests, 50% visual-token compression yields over 1.75× speedup, with larger gains at higher compression rates.

4.5. Visualization of Image Routing

  • Salient objects and semantically rich text are largely preserved, while homogeneous backgrounds are compressed more; scores above 0.5 trigger 64-token patches.White-shaded patches use 64 tokens and unshaded patches retain 256 tokens; higher confidence scores indicate a stronger tendency for compression.Source: §4.5 · p. 7, Figure 3 · p. 8

4.6. Throughput Analysis

  • Over 1.75× speedup is achieved when 50% of visual tokens are compressed, with larger gains at higher compression rates.Smaller text inputs benefit more from high compression than longer inputs.Source: §4.6 · p. 8, Table 4 · p. 8

5. Conclusion

ViCO represents images with different numbers of vision tokens according to semantic complexity rather than image resolution. It can reduce vision tokens by up to 50% while maintaining perception, reasoning, and OCR capabilities.

5. Conclusion

  • ViCO adapts the number of vision tokens to image semantic complexity, unlike dynamic high-resolution strategies that adapt token counts to image resolution.The conclusion frames this as semantic-level allocation of computation to informative image regions.Source: §5 · pp. 8–9
  • ViCO reduces vision tokens by up to 50% while maintaining the model’s perception, reasoning, and OCR capabilities.The paper presents this as an inference-time efficiency contribution.Source: §5 · p. 9
  • The authors state that semantic-based decisions let the model focus computation on the most informative image regions and offer insights for adaptive visual representations.They frame this as a contribution toward reducing inference-time computational cost in MLLMs.Source: §5 · p. 9

A. Performance of ViCO on small-scale models

On smaller InternVL3.5 models, ViCO retains nearly all original performance across general multimodal, OCR-related, multi-image, and video benchmarks. Tables 5 and 6 report the small-scale model results across these benchmark groups.

A. Performance of ViCO on small-scale models

  • InternVL3.5 1B and 2B models retain nearly all original performance across the evaluated benchmark range.Tables 5 and 6 cover general benchmarks as well as multi-image and video benchmarks.Source: Table 5 · p. 11, Table 6 · p. 11, A. Performance of ViCO on small-scale… · p. 11
  • Table 5 covers general benchmarks, including OCRBench, ChartQA, TextVQA, MMBench, RealWorldQA, and normalized MME scores.It also reports the ratio of compressed patches.Source: Table 5 · p. 11
  • Table 6 covers multi-image and video benchmarks, with Video-MME evaluated using 64-frame inputs in both subtitle and non-subtitle settings.The table caption identifies MMT-Bench and the compressed-patch ratio as well.Source: Table 6 · p. 11

B. Discussion on Additional Inference Latency

The added router and attention-pooling structures are described as having minimal theoretical computational overhead. The authors also report notable practical acceleration without significant extra computational cost or inference latency.

B. Discussion on Additional Inference Latency

  • The router has approximately 50M parameters for 1B–30B models and around 490M for 38B and 241B models, small relative to overall model size.The paper characterizes the resulting theoretical computational overhead as minimal.Source: B. Discussion on Additional Inference… · p. 11
  • Attention Pooling scales linearly with sequence length in computational complexity.The paper includes it among the added structures whose theoretical overhead is minimal.Source: B. Discussion on Additional Inference… · p. 11
  • The added components do not introduce significant extra computational cost or inference latency, while experiments show notable practical acceleration.The passage summarizes the deployment impact of the additional structures.Source: B. Discussion on Additional Inference… · p. 11

C. Visualization of Router

Figures 4 and 5 show ViCO’s Visual Resolution Router routing images divided into different patch layouts. Their annotations distinguish compressed patches, retained higher-token patches, and the router’s compression confidence.

C. Visualization of Router

Loading 2510.12793v2…