PaperlayerView exact revision on arXiv

Source-linked AI summary

VisionWeave: Weaving Elastic Visual Representations as a Native Capability of MLLMs

Yuan Feng, Qize Yang, Ruizhe Chen, Sibo Song, Haolin He, Muzhi Zhu, Zihan Liu, Yunfei Chu, Xize Cheng, Yuxuan Wang, Jin Xu, Xike Xie

arXiv:2610.07987v1cs.CVcs.AIcs.CL

TL;DR

Dense fixed-size visual tokens are costly, while existing approaches have limits in content-adaptive granularity and integration with modern MLLMs. VisionWeave learns to combine fine- and coarse-grained representations and allocate them by visual content. On Qwen3.8-27B, it saves 43.0% of visual tokens on average while preserving 98.9% native performance across eight benchmarks.

  • Problem

    Dense fixed-size visual-token sequences raise computational and memory demands, while downsampling, pruning, and adaptive methods have limitations in detail, task generalization, or integration with modern MLLMs and serving engines.

    Source: §1 · p. 2

  • Method

    VisionWeave combines native fine-grained tokens and learned coarse representations, with a granularity router allocating them according to visual content.

    Source: §1 · p. 3

  • Results

    43.0% average visual-token savings preserved 98.9% native performance across eight benchmarks, while FastV† and VisionZip retained 88.2% and 87.5% at a fixed 50% savings target.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The findings establish representational granularity as a learnable degree of freedom and validate content-adaptive computational effort as a feasible architectural principle.

    Source: §5 · p. 13

    Training cost limits comprehensive architectural exploration; this work focuses on two granularities, leaving richer multi-scale representations for future work.

    Source: A LIMITATIONS AND FUTURE WORKS · p. 18

Abstract

from arXiv · show

Multimodal large language models have become the dominant paradigm for visual understanding, but incur substantial costs by encoding inputs into dense, fixed-size patch tokens. However, visual information is unevenly distributed: some regions require fine-grained detail, while others admit compact representations. Downsampling sacrifices this detail, while existing token pruning and adaptive approaches remain limited in content-adaptive granularity, task generalization, and integration with modern MLLMs and serving infrastructure. Overcoming these limitations calls for foundation models that learn, end to end, where-and at what granularity-to allocate visual representations, a native capability we term elastic visual representation weaving. We introduce VisionWeave, establishing this capability in frontier-level MLLMs through large-scale training. It combines two components: a gated spatial pooler constructs coarse-grained representations alongside native fine-grained representations within a shared MRoPE coordinate, while a granularity router learns their content-adaptive allocation. Through self-distillation alone, we validate this capability on Qwen3.5-4B and scale to Qwen3.8-27B with over 30K A100 GPU-hours. Based on Qwen3.8-27B, VisionWeave adaptively adjusts token savings to visual content, saving 43.0% tokens on average while retaining 98.9% native performance across eight benchmarks, versus only 88% performance preserved for token pruning baselines with a fixed 50% savings target. Extensive evaluations confirm robust efficiency-quality trade-offs across diverse tasks, resolutions and video frames. When deployed on SGLang serving engine, our method achieves a 2.3x throughput gain while reducing mean TTFT by 54.4% and mean TPOT by 60.6%. Together, we believe these results position elastic visual weaving as a promising capability for next-generation multimodal models.

1 INTRODUCTION

The introduction argues that fixed-size visual tokens and existing compression approaches fall short of content-adaptive, broadly compatible processing. VisionWeave pairs learned coarse and native fine representations with adaptive allocation, reporting token savings while preserving benchmark performance.

1 INTRODUCTION

  • Dense fixed-size visual-token sequences raise computational and memory demands as MLLMs process higher-resolution images and longer video sequences.Downsampling sacrifices fine-grained detail, while pruning and merging methods can discard task-specific information and have preset compression ratios.Source: §1 · p. 2
  • Existing adaptive methods target standalone vision tasks or assign one resolution per tile, limiting fit with diverse MLLM objectives and finer spatial allocation.The introduction notes that detail-rich regions and low-information backgrounds can be interleaved within tiles.Source: §1 · p. 2
  • The proposed capability should adapt token savings to content, retain quality across visual tasks, outperform downsampling trade-offs, and work with frontier MLLMs and serving infrastructure.Source: §1 · p. 3
  • VisionWeave combines a gated spatial pooler for coarse representations with native fine tokens and a router that selects granularity for spatial blocks.The representations share an MRoPE coordinate; the router probes ViT features to allocate granularity.Source: §1 · p. 3, Figure 3 · p. 3
  • Self-distillation alone establishes the capability on Qwen3.5-4B and scales it to Qwen3.8-27B with over 30K A100 GPU-hours.Source: §1 · p. 3
  • 43.0% average visual-token savings preserved 98.9% native performance across eight benchmarks, while FastV† and VisionZip retained 88.2% and 87.5% at a fixed 50% savings target.At 512 tokens per image, savings ranged from 16.5% to 55.1%; evaluations across resolutions and video-frame budgets also found more favorable efficiency–quality trade-offs than input downsampling.Source: §1 · p. 3

2 METHOD

VisionWeave combines a gated spatial pooler that creates coarse representations with a granularity router that allocates coarse and native fine-grained tokens by visual content. Three-stage self-distillation trains this capability, which the authors integrate with modern serving infrastructure.

2.2.1 GATED SPATIAL POOLER

  • Each 2×2 patch neighborhood is pooled into a coarse feature and passed through the native merger, complementing four fine-grained tokens with one coarse token.Coarse and fine-grained tokens share an embedding space; coarse tokens are positioned at the geometric center of their corresponding fine tokens.Source: §2.2.1 · p. 4

2.2.2 GRANULARITY ROUTER

  • The router probes ViT features at multiple depths and selects one coarse token or four native fine-grained tokens per block using a probability threshold.Selected tokens form a mixed-granularity sequence in the shared MRoPE frame.Source: §2.2.2 · pp. 4–5

2.3.1 STAGE 1: COARSE-GRAINED SELF-DISTILLATION

  • Stage 1 trains the pooler on all-coarse sequences, using the model’s native forward pass as the teacher for self-distillation.Other components remain frozen, and the distillation objective uses the teacher’s top-512 vocabulary entries at each response position.Source: §2.3.1 · p. 6

2.3.2 STAGE 2: SOFT-MIXING SELF-DISTILLATION

  • Stage 2 trains only the router with a differentiable soft-mixing surrogate and a per-sample balance loss targeting a coarse-routing fraction of 0.8.Fine and coarse feature directions and magnitudes are interpolated separately; the balance penalty encourages more or less coarse routing according to each sample’s routing fraction.Source: §2.3.2 · p. 6

2.3.3 STAGE 3: MIXED-GRANULARITY SELF-DISTILLATION

  • Stage 3 distills on hard-routed mixed-granularity sequences, using the same token selection and RoPE coordinates as inference.The frozen teacher’s native forward pass provides supervision; this stage improved diverse tasks, especially grounding.Source: §2.3.3 · p. 6

2.4 TRAINING DETAILS AND DYNAMICS

  • Training uses a router sampling ViT features at depths [0, 13, 27], and Stage 3 updates the pooler and LLM while keeping the ViT, native merger, and router frozen.All three stages use the same data mixture; training settings include BF16, Adam, and cosine learning-rate decay.Source: §2.4 · p. 7
  • Qwen3.8-27B training uses about 0.78 million image, video, and GUI-grounding samples; Stage 2 router entropy fell from 0.658 to 0.427 nats.The validation share of blocks with pIJ in [0.4, 0.6) fell from 64.8% to 14.6%, while the share with pIJ ≥ 0.9 reached 28.3%.Source: §2.4 · p. 7, Figure 4 · p. 8

2.5 DEPLOY VISIONWEAVE ON MODERN SERVING INFRASTRUCTURE

  • The SGLang integration rescales coordinates for block-center RoPE and attaches layout metadata to visual responses so mixed-granularity sequences can be finalized before scheduling.The serving changes preserve dynamic batching, KV-cache management, and CUDA-graph decoding.Source: §2.5 · pp. 7–8

3 EXPERIMENTS

The experiments evaluate whether VisionWeave preserves quality while adapting token savings across tasks and input conditions, and whether it improves serving efficiency and supports inference-time control.

3.1 EXPERIMENTAL SETUP

  • The evaluation compares Qwen3.5-4B and Qwen3.8-27B with VisionZip and FastV† across eight benchmarks spanning image, document, GUI, and video tasks.FastV† selects tokens after the vision encoder for evaluation within SGLang; VisionWeave's default routing threshold is 0.5.Source: §3.1 · pp. 8–9

3.2 CONTENT-ADAPTIVE TOKEN SAVINGS WITH PRESERVED QUALITY

  • 1.08% average performance loss on Qwen3.8-27B contrasts with 11.82% for FastV† and 12.46% for VisionZip at a fixed 50% savings target.VisionWeave's savings vary by task: 27.3% on DocVQA versus 49.6–54.2% across three video benchmarks, where losses remain at or below 0.30%.Source: Table 4 · p. 9, §3.2 · p. 9

3.3 ROBUSTNESS ACROSS INPUT RESOLUTIONS AND FRAME BUDGETS

  • 1.20 points higher on LongVideoBench than the native model capped at 128 frames, VisionWeave used fewer visual tokens with a 256-frame cap.Across eight benchmarks and evaluated resolutions, it maintained favorable trade-offs and outperformed input downsampling in the evaluated tasks and resolutions.Source: §3.3 · pp. 9–10, Figure 5 · p. 10, Figure 6 · p. 11

3.4 END-TO-END SERVING EFFICIENCY ON SGLANG

  • 2.30× end-to-end throughput speedup on SGLang was accompanied by 57.1% lower P95 TTFT and 59.6% lower P95 TPOT.The benchmark used LongVideoBench with up to 256 frames per video.Source: Table 5 · p. 11, §3.4 · p. 11

3.5 INFERENCE-TIME CONTROL OF TOKEN SAVINGS

  • On Video-MME, lowering the threshold from 0.5 to 0.25 increased Qwen3.8-27B savings from 54.2% to 66.0%, with a 0.37-point performance loss.Threshold changes provide inference-time control without retraining, and the full-retention endpoint uses fine-grained representations without token savings.Source: Figure 8 · p. 11, §3.5 · p. 12

4 ANALYSIS OF ELASTIC VISUAL WEAVING

The analysis tests the effects of the gated spatial pooler and learned routing, then examines how mixed granularity represents task-relevant regions.

4.1 ABLATION STUDIES

  • At 75% token savings, the gated spatial pooler scored higher than pixel-unshuffle on all eight benchmarks, including 2.64-point gains on DocVQA and 2.77 on VideoOCR.The comparison used Qwen3.5-4B after Stage 1.Source: §4.1 · p. 12
  • Shuffling routing scores while preserving each input's number of coarse selections lowered mean scores by 2.31 points on Qwen3.5-4B and 5.36 on Qwen3.8-27B.The mean drops remained 1.30 and 1.53 points, respectively, even when ScreenSpotV2 was excluded.Source: Table 6 · p. 12, §4.1 · p. 12

4.2 VISUAL ANALYSIS OF GRANULARITY ALLOCATION

  • In GUI grounding examples, VisionWeave represented every region while FastV† dropped click-target tokens; on a document, it saved 45.2% of tokens while representing larger text coarsely.FastV† also removed informative text tokens, leaving some characters only partially represented; the examples illustrate spatial coverage without gaps.Source: §4.2 · pp. 12–13, Figure 9 · p. 13

5 CONCLUSION

VisionWeave establishes elastic visual representation weaving as a native MLLM capability through self-distillation alone. It combines adaptive token savings with efficiency–quality trade-offs across tasks, resolutions, and frame budgets, and translates token savings into serving gains.

5 CONCLUSION

  • VisionWeave establishes elastic visual representation weaving as a native MLLM capability through self-distillation alone.Source: §5 · p. 13
  • The method combines content-adaptive token savings with robust efficiency–quality trade-offs across tasks, resolutions, and frame budgets.Source: §5 · p. 13
  • On SGLang, the integration translates token savings into higher throughput and lower latency on modern serving infrastructure.Source: §5 · p. 13
  • The findings validate representational granularity as a learnable degree of freedom and content-adaptive computational effort as a feasible architectural principle.Source: §5 · p. 13

A LIMITATIONS AND FUTURE WORKS

Training cost constrains how comprehensively VisionWeave’s architectural choices have been explored. The current approach uses two granularities and introduces the capability through post-training self-distillation.

A LIMITATIONS AND FUTURE WORKS

B.1 CASE STUDY IN GUI GROUNDING TASKS

In GUI examples, VisionWeave interleaves fine- and coarse-grained representations while preserving region coverage in a shared spatial frame. FastV† drops tokens over critical click targets, leaving gaps that can hinder accurate grounding.

B.1 CASE STUDY IN GUI GROUNDING TASKS

  • VisionWeave interleaves fine- and coarse-grained representations while keeping every GUI region represented within a shared spatial frame.Source: §B.1 · p. 18
  • FastV† drops tokens covering critical click targets, such as the web-interface star button, leaving gaps in token coverage.Source: §B.1 · p. 18
  • The spatially coherent coverage helps explain VisionWeave’s substantial advantage over FastV† and VisionZip on grounding tasks.Source: §B.1 · p. 18

B.2 CASE STUDY IN DOCVQA TASKS

On two DocVQA examples, VisionWeave adapts its token allocation to visual content, using coarse representations for larger text while preserving full spatial coverage. Additional visualizations show fine- and coarse-grained representations interleaved according to visual content across two model scales.

B.2 CASE STUDY IN DOCVQA TASKS

  • At threshold t = 0.5, VisionWeave saves 45.2% of tokens on the form while retaining all native fine-grained tokens for the dense table.Source: §B.2 · p. 18
  • For larger text, VisionWeave uses coarse-grained representations while maintaining full spatial coverage; FastV† removes 50% of tokens in both examples, including tokens covering text.Source: §B.2 · p. 18

C MORE VISUALIZATIONS OF ELASTIC VISUAL WEAVING

D RELATED WORK

Prior pruning and adaptive-visual methods leave gaps in task-adaptive granularity and practical MLLM deployment. VisionWeave uses block-wise fine/coarse allocation and reports favorable efficiency–quality trade-offs with SGLang.

D RELATED WORK

  • Pruning and merging can degrade grounding and information-dense inputs, often use preset compression ratios, and rarely validate gains in modern serving engines.Some approaches also require attention weights or in-LLM pruning, complicating engine integration and dynamic batching.Source: D RELATED WORK · p. 18
  • SGLang evaluations across tasks, resolutions, and video-frame budgets show favorable efficiency–quality trade-offs for VisionWeave.Source: D RELATED WORK · p. 19
  • Unlike externally budgeted or tile-based adaptive methods, VisionWeave learns block-wise fine- and coarse-grained allocation within native-resolution processing.The paper argues that interleaved detail-rich regions and low-information backgrounds call for finer allocation than one resolution per tile.Source: D RELATED WORK · pp. 19–20
  • Document examples contrast content-adaptive granularity with fixed token savings.Source: Figure 11 · p. 20
  • KV-cache compression compacts stored key–value states without shortening the input sequence, making it complementary to visual token reduction.The paper describes combining the two mechanisms as a promising direction for reducing computation and cache memory.Source: D RELATED WORK · p. 21

E.1 ABLATION OF MIXED-GRANULARITY SELF-DISTILLATION

Stage 3 self-distillation on hard-routed mixed-granularity sequences improves Qwen3.8-27B on most benchmarks. The reported gains persist on average even when ScreenSpotV2 is excluded.

E.1 ABLATION OF MIXED-GRANULARITY SELF-DISTILLATION

  • Excluding ScreenSpotV2, the average still improves by 0.54 points, despite a 1.18-point decrease on RealWorldQA.Source: §E.1 · p. 21
  • Seven of eight benchmarks improve after Stage 3, raising the Qwen3.8-27B average from 73.15 to 76.73.ScreenSpotV2 rises from 66.59 to 91.43; excluding it, the average still improves by 0.54 points despite a 1.18-point decrease on RealWorldQA.Source: Table 7 · p. 21, §E.1 · p. 21
  • Additional visualizations cover Qwen3.5-4B and Qwen3.8-27B representations at a 2048-token input budget.Source: Figure 14 · p. 22, Figure 15 · p. 23

E.2 ABLATION OF DISTILLATION POLICY

At matched dataset-level token savings, VisionWeave retains higher mean scores than pruning baselines, including when ScreenSpotV2 is excluded. On Qwen3.5-4B, on- and off-policy distillation perform similarly; the SGLang serving tests use a specified two-GPU configuration.

E.2 ABLATION OF DISTILLATION POLICY

  • On- and off-policy Stage 3 self-distillation achieve similar mean scores across eight Qwen3.5-4B benchmarks, each leading on four.Off-policy self-distillation is used for training efficiency.Source: §E.2 · p. 23, Figure 17 · p. 24

E.3 PERFORMANCE AT MATCHED TOKEN SAVINGS

  • 74.63 versus 73.86/73.85 are the mean scores for VisionWeave versus FastV†/VisionZip when ScreenSpotV2 is excluded from the matched-savings comparison.The comparison matches dataset-level savings at t = 0.5 for each backbone and dataset.Source: §E.3 · pp. 24–25, Table 8 · p. 25

E.4 SGLANG SERVING CONFIGURATION

  • SGLang serving tests run each model on two A100-SXM4-80GB GPUs, with client concurrency capped at four outstanding requests.The configuration uses a 147,456-token context limit and 4,096-token prefill chunks.Source: §E.4 · p. 25

Related papers

Loading 2610.07987v1…