PaperlayerView exact revision on arXiv

Source-linked AI summary

Resolution as a First-Class Decision: Task-Conditioned Routing for Efficient Multimodal Large Language Models

Zhiqiang Xia, Yang Li, Xinyuan Zhang, Yuchen Liu, Haoyu Lu, Jiaming Xu, Runyu Shi, Ying Huang

arXiv:2609.34942v1cs.CV

TL;DR

High-resolution visual encoding makes MLLM inference costly, and existing approaches leave input resolution task-agnostic. TCRR uses a lightweight router to choose a query-conditioned resolution, reducing visual FLOPs and latency on Qwen3-VL-8B with little capability loss. The paper identifies uniform whole-image scaling as a boundary for sparse, localized-detail queries.

  • Problem

    High-resolution inputs create long visual-token sequences with quadratic attention cost, while existing compression often follows full-resolution encoding and input resolution remains task-agnostic.

    Source: §1 · p. 1

  • Method

    TCRR uses a lightweight cross-modal router to select minimal sufficient resolution from image content and textual semantics before the frozen MLLM backbone.

    Source: §1 · p. 2, §3 · p. 3

  • Results

    On Qwen3-VL-8B, TCRR reduces visual FLOPs by 40.9% and system latency by 53.7%, with negligible capability loss (∼0.6%).

    Source: §1 · p. 2

  • Takeaways & Limitations

    The reported results support treating resolution as a query-conditioned decision while leaving the MLLM backbone unmodified.

    Source: §1 · p. 2

    Because TCRR applies uniform resolution across an image, sparse localized-detail queries may require high global resolution and yield minimal upfront FLOPs reduction.

    Source: §7 · p. 9

Abstract

from arXiv · show

The inference efficiency of Multimodal Large Language Models (MLLMs) is severely constrained by massive visual token sequences induced by high-resolution inputs, with computational cost scaling quadratically. Existing approaches primarily focus on downstream token compression, while overlooking a fundamental upstream inefficiency: input resolution is treated as a static, task-agnostic hyperparameter. We propose Task-Conditioned Resolution Routing (TCRR), which formulates visual compression as a task-conditioned decision and employs a lightweight cross-modal router that conditions backbone visual representations on textual semantics via feature-wise modulation and cross-attention to predict the minimal sufficient compression level per query. To support this, we curate a dataset of 500k samples across 12 task categories, labeled via a teacher-oracle pipeline to approximate Pareto-optimal compression scales. Extensive experiments across diverse architectures show that TCRR achieves a superior efficiency frontier, specifically reducing visual FLOPs by 40.9% and latency by 53.7% on Qwen3-VL-8B while preserving competitive performance. Further analysis of scaling behavior confirms that dynamically routing visual compression enables optimal resource allocation without modifying the MLLM backbone.

1 Introduction

High-resolution visual encoding creates a major MLLM inference bottleneck, while existing resolution choices are static and task-agnostic. TCRR routes each image–text query to a minimal sufficient resolution; the paper reports substantial compute and latency reductions with little capability loss.

1 Introduction

  • High-resolution visual encoding produces long token sequences with quadratic attention cost, while downstream compression occurs after that expensive encoding [3] [4] [5].The paper identifies static, task-agnostic input resolution as an upstream inefficiency because spatial-detail needs vary with the prompt [6] [7].Source: §1 · p. 1
  • TCRR uses a lightweight cross-modal router to select minimal sufficient resolution from image content and textual semantics before the MLLM backbone.The router integrates feature-wise modulation and text-to-image attention.Source: §1 · p. 2
  • Res-500k contains 500k image–text pairs across 12 task categories, labeled by a teacher–oracle pipeline to preserve semantic equivalence at the highest compression ratio [8] [9].The paper reports negligible performance degradation on held-out test sets [10, 11].Source: §1 · p. 2
  • On Qwen3-VL-8B, TCRR reduces visual FLOPs by 40.9% and system latency by 53.7%, with negligible capability loss (∼0.6%).The reported latency reduction is 43% on the 235B-A22B architecture.Source: §1 · p. 2

2 Related Work

Prior work reduces visual computation either after full-resolution encoding or through task-agnostic input-resolution heuristics. TCRR instead conditions resolution selection on the prompt and places routing before a frozen MLLM backbone.

Visual Compression vs. Upstream Resolution

  • Downstream token-reduction methods [4] [5] [14] act after full-resolution encoding, leaving the ViT's peak memory and upfront compute costs intact.Upstream resolution strategies [15] address that bottleneck but rely on image heuristics rather than the prompt.Source: §2 · p. 3
  • TCRR conditions the input-resolution decision on the textual prompt, aligning spatial resource allocation with query demands.This bridges downstream compression and task-agnostic upstream scaling.Source: §2 · p. 3

Conditional Computation & Routing

  • Unlike intra-model Mixture-of-Experts [16] [17] and layer-skipping approaches [18] [19], TCRR routes at the input level as a plug-and-play pre-filter with the backbone frozen.The cited intra-model approaches assume fixed-fidelity inputs and often require architectural changes or joint pre-training.Source: §2 · p. 3

3 Problem Formulation

The formulation treats resolution as a task-dependent choice because visual-encoder cost grows quadratically with resolution and prompts impose different spatial demands. It seeks a low-cost scale that preserves the high-resolution reference response, approximating that choice with a lightweight router.

3 Problem Formulation

  • Visual-encoder attention cost scales quadratically with resolution, while image-conditioned dynamic-resolution methods ignore prompt-specific spatial demands.The formulation expresses cost as C(r) ∝ r^2, with P denoting patch size.Source: §3 · p. 3
  • For each image–text instance, the objective is the minimal candidate resolution that reduces cost without degrading the high-resolution reference response.The candidate resolutions come from a discrete set.Source: §3 · p. 3
  • Because exhaustive oracle inference is intractable, a lightweight policy predicts a scale from a fixed-resolution proxy and prompt; the frozen backbone then processes the rescaled image.Router overhead is typically less than 5% of backbone cost, and resolution reductions translate directly to wall-clock acceleration.Source: §3 · p. 3

4 Dataset Construction: Res-500k

Res-500k supplies task-conditioned supervision for learning minimal sufficient image scales. Its teacher–oracle labels assess semantic equivalence, with quality checks and a stratified test set supporting evaluation across varied resolution demands.

4 Dataset Construction: Res-500k

  • Res-500k is constructed to supervise minimal sufficient resolutions using semantic equivalence rather than next-token prediction loss as the sufficiency criterion.The passage notes that teacher-forcing loss may reflect language priors or benign phrasing differences instead of visual clarity.Source: §4 · p. 3

4.1 The Teacher–Oracle Pipeline & Quality Assurance

  • The teacher–oracle pipeline evaluates 10 candidate scales from 0.1 to 1.0 against a native-resolution reference response, using Qwen3-VL-8B as teacher.GPT-5.1 was selected as the semantic-equivalence judge after pilot studies; human evaluation of 1,200 samples aligned 100% with its judgments.Source: §4.1 · pp. 3–4
  • The oracle selection strategy preserved 100% of uncompressed performance across 12 evaluation benchmarks, establishing the reported Oracle Bound.The passage also reports that human verification found no systemic bias and prevented evaluation-set data leakage.Source: §4.1 · p. 4

4.2 Dataset Statistics and Diversity

  • Res-500k contains 500,000 instances balanced across 12 task categories, with 42% general vision-language, 37% document extraction, and 21% GUI and agent navigation tasks.A stratified hold-out test set contains 12,000 samples; task demands range from mostly low/mixed to high and very high resolution.Source: §4.2 · p. 4, Table 1 · p. 4

5 Task-Conditioned Resolution Routing

TCRR makes resolution an instance-level decision, predicting the minimal sufficient scale from visual content and textual intent before the image reaches an unmodified MLLM backbone.

5 Task-Conditioned Resolution Routing

  • TCRR predicts a minimal sufficient resolution scale from visual semantics and textual intent, making resolution an adaptive decision upstream of the unmodified MLLM backbone.Source: §5 · p. 4

5.1 Inference Pipeline Overview

  • The router processes the image–text pair, predicts a continuous scale, and resamples the image before routing it with the prompt to the backbone.Source: §5.1 · pp. 4–5, Figure 2 · p. 5

5.2 The Architecture of TCRR

  • A dual-encoder architecture uses progressive fusion to aggregate task-conditioned features through sequential feature-wise modulation and spatial cross-attention.Source: §5.2 · p. 5

5.2.1 Unimodal Feature Extraction

  • MobileNetV4 extracts a feature pyramid from a fixed 384×384 image, while BERT-Tiny attention-pools prompt tokens into a task-specific global embedding.The prompt representation can attend to resolution-critical cues such as requests to read small text or count objects.Source: §5.2.1 · p. 5

5.2.2 Task-Conditioned Cross-Modal Interaction

  • FiLM uses the global text embedding to channel-modulate visual features, acting as a semantic filter that amplifies task-relevant channels and attenuates background noise.Source: §5.2.2 · p. 6
  • Cross-attention uses global text as a query over modulated visual features, producing a spatially aware descriptor of regions critical to the instruction.Source: §5.2.2 · p. 6

5.2.3 Decision Head and Training Objective

  • The decision head fuses pooled visual context, cross-attended context, and global text into a positive scale; end-to-end MSE trains smooth scale regression against oracle targets.The paper places its computational-complexity analysis of routing overhead in Appendix B.Source: §5.2.3 · p. 6

6 Experiments

The experiments evaluate task-conditioned resolution routing across MLLM backbones, multimodal benchmarks, and specialized domains. They examine efficiency, routing components, tokenization, scaling, and proximity to oracle compression.

6.2 Main Results & Baselines (RQ1)

  • On Qwen3-VL-8B, TCRR reduces FLOPs by 40.9% and System Latency by 53.7%, with average capability dropping by 0.61%.Table 2 covers the Qwen3-VL family and summarizes reductions in FLOPs and latency while preserving average accuracy across 12 benchmarks.Source: Table 2 · p. 7, §6.2 · p. 7

6.1 Experimental Setup

  • Experiments cover 12 multimodal benchmarks, two specialized-domain datasets, Qwen3-VL and InternVL3.5 backbones, and accuracy, scale-prediction MAE, FLOPs, token reduction, and latency.Backbones remain unmodified; latency includes the visual encoder, projector, LLM prefill, and router overhead, but excludes autoregressive decoding variance.Source: §6.1 · p. 7

6.2 Main Results & Baselines (RQ1)

  • At a ∼45−50% token reduction, TCRR retains 99.2% of capability versus 96.6% for VisionZip on Qwen3-VL-8B.The comparison is at equivalent compression regimes.Source: §6.2 · p. 7, Table 3 · p. 8

6.4 Generalization and Tokenization (RQ3)

  • On OmniMedVQA, TCRR reduces tokens by 34.2% while increasing accuracy by 0.8%; on LRS-VQA, it prunes 71.2% of visual payload while preserving 95.4% capability.These are zero-shot specialized-domain results without domain-specific fine-tuning.Source: Table 5 · p. 8, §6.4 · p. 8
  • Qwen3-VL reduces FLOPs by 42% versus 16% for InternVL3.5; the paper attributes the gap to continuous patch-based tokenization versus fixed ≥448 × 448 tiling.The cross-family analysis identifies flexible tokenization as enabling greater savings than rigid tiling.Source: Table 6 · p. 8, §6.4 · p. 8

6.3 Ablation Studies (RQ2)

  • Text conditioning adds 4.0% accuracy over the vision-only baseline; Cross-Attention and FiLM add 0.8% and 0.5%, respectively, with lowest MAE of 1.14.The component ablation is on Qwen3-VL-8B; its caption identifies text conditioning as the main driver and FiLM and Cross-Attention as refinements.Source: §6.3 · p. 8, Table 4 · p. 8

6.5 Scaling Laws and Granularity (RQ4)

  • Router MAE improves log-linearly with training data and saturates at 500k samples; MobileNet-Medium with BERT-Tiny is identified as the efficiency-accuracy sweet spot.The paper notes that smaller MobileNet initialization limits feature granularity, while larger backbones increase inference costs.Source: §6.5 · p. 8, Table 7 · p. 9, Table 8 · p. 9
  • Moving from K = 2 to K = 10 nearly doubles average token savings, from 11.9M to 20.9M.Figure 3 concerns resolution granularity and token reduction across 12 benchmarks.Source: Figure 3 · p. 9, §6.5 · p. 9

7 Limitations

TCRR applies compression uniformly across each image, limiting upfront FLOPs reductions when a query requires preserving a small, localized detail.

7 Limitations

  • For localized-detail queries such as finding a distant license plate in a 4K panorama, uniform global resolution may require conservative high resolution and yield minimal upfront FLOPs reduction.The paper describes downstream spatial token pruning as complementary for discarding irrelevant background tokens in these sparse scenarios.Source: §7 · p. 9

8 Conclusion

TCRR makes resolution a query-aware decision and reports a shifted efficiency–accuracy frontier with near-zero capability degradation. The supplementary analyses explain semantic-equivalence supervision and estimate the router’s computational cost.

8 Conclusion

  • TCRR shifts the efficiency–accuracy Pareto frontier, cutting FLOPs by up to 41.6% and latency by 48.4% on Qwen3-VL with near-zero capability degradation.The conclusion frames resolution as a dynamic, first-class decision variable rather than a rigid preprocessing hyperparameter.Source: §8 · p. 10

A.1 GPT Judge Prompt

  • The oracle prompt accepts answers with the same correct final result as the full-resolution answer, then selects the smallest acceptable scale.If none of the tested scales from 0.1 to 0.9 is acceptable, it selects 1.0.Source: §A.1 · p. 13

A.2 Analysis of Supervision Signals: Why Loss is an Imperfect Proxy

  • Teacher-forcing loss can mislabel resolution sufficiency: language priors can hide lost visual evidence, while equivalent wording can spuriously demand higher resolution.The semantic-equivalence judge is proposed to filter lexical drift and check factual grounding.Source: §A.2 · p. 14

B.1 Input Specifications

  • TCRR’s router uses fixed-size image and text inputs, including a 384 × 384 image proxy and a 192-token prompt limit.Long prompts retain the first 30% and last 70% of tokens.Source: §B.1 · pp. 14–15

B.2 Theoretical FLOPs Decomposition

  • The router’s compute comprises visual encoding, text encoding, and cross-modal interaction, with MobileNetV4-Medium as its primary computational load.The visual backbone remains orders of magnitude smaller than standard MLLM visual encoders such as CLIP-ViT-L/14, which typically exceeds 80 GFLOPs.Source: §B.2 · p. 15
  • TCRR consumes approximately 0.8% of the compute budget required by a typical Qwen3-VL-8B visual encoder (approximately 100 GFLOPs), satisfying the stated design constraint.BERT-Tiny’s minimal footprint ensures text processing incurs virtually no latency penalty.Source: §B.2 · p. 15

B.3 Empirical Latency Analysis

Latency measurements show that TCRR adds modest overhead, while its fixed-proxy router can yield substantially larger backbone savings. The supplementary implementation details specify preprocessing and training choices.

B.3 Empirical Latency Analysis

  • 18.1 ms is the measured router latency; this is less than 7% of Qwen3-VL-2B execution time and less than 0.8% for Qwen3-VL-32B.Measurements used an NVIDIA H20 GPU, FP16 precision, and batch size 1.Source: §B.3 · p. 15, Table 10 · p. 16

B.4 Relative Overhead vs. Backbone Savings

  • On Qwen3-VL-8B, savings often exceed 500 GFLOPs per image, making the sub-1-GFLOP router overhead less than 0.2% of those savings.The router operates on a fixed 384 × 384 proxy, independent of the original high-resolution input size.Source: §B.4 · p. 16

C.1 Input Preprocessing

  • For router preprocessing, images are padded or scaled and centered on a 384 × 384 canvas to preserve their geometric structure.Images smaller than 384 pixels in both dimensions are placed at the top-left of a zero-padded canvas without scaling.Source: §C.1 · p. 16

C.2 Training Protocol

  • TCRR is trained end-to-end with MSE loss and AdamW, using cosine annealing, 5% linear warmup, and 20 epochs on Res-500k.Training uses Distributed Data Parallel with SyncBatchNorm.Source: §C.2 · p. 17

D Extended Ablation on Fusion Architectures

The single-stage fusion design balances routing accuracy and computational overhead better than the tested multi-stage alternative.

D Extended Ablation on Fusion Architectures

  • The multi-stage alternative interleaves multiple cross-attention and feed-forward blocks, unlike the single-pass FiLM and Cross-Attention design.The comparison is measured on the Qwen3-VL backbone.Source: D Extended Ablation on Fusion… · p. 17
  • The single-stage design records 18.1 ms router latency, lower than the multi-stage alternative’s 22.8 ms, while achieving 72.8% downstream task accuracy and 1.14 MAE.The multi-stage variant has an almost identical token reduction rate: 45.6% versus 45.8%.Source: D Extended Ablation on Fusion… · p. 17, Table 12 · p. 17
  • The reported latency penalty matters because router delay directly inflates end-to-end latency when the router is an unavoidable pre-filter.This is the paper’s stated rationale for preferring shallow fusion despite similar token reduction.Source: D Extended Ablation on Fusion… · p. 17

E Additional Statistical Results

The supplementary statistics describe benchmark resolution and compression-ratio distributions, while case studies illustrate prompt-sensitive routing and differing compression for dense-information versus high-level queries.

E Additional Statistical Results

  • Table 13 bins benchmark sample counts into eight pixel-area intervals, spanning below 256² through above 4096² pixels.The accompanying statistical analysis presents these resolution characteristics as context for task requirements.Source: Table 13 · p. 18, E Additional Statistical Results · p. 18

F TCRR: Qualitative Case Studies

F.2 Category 2: High-Resolution Retention for Dense Information

  • For dense text or complex structures, TCRR conservatively retains pixels; examples include 20% compute savings for ChartQA and 90% resolution retention for dense DocVQA.The DocVQA example extracts “79.23%” from a financially dense table.Source: §F.2 · p. 20, §F.2.1 · p. 20, §F.2.2 · p. 20

F.3 Category 3: Extreme Redundancy Pruning

  • For high-level semantic queries such as mood perception or basic object recognition, TCRR prunes 80%–90% of pixel area; examples scale images to 20% for mood and 10% for Shiba Inu recognition.Source: §F.3 · p. 21, §F.3.1 · p. 21, §F.3.2 · p. 21

Related papers

Loading 2609.34942v1…