PaperlayerView exact revision on arXiv

Source-linked AI summary

AdaptVision: Efficient Vision-Language Models via Adaptive Visual Acquisition

Zichuan Lin, Yicheng Liu, Yang Yang, Lvfang Tao, Deheng Ye

arXiv:2512.03794v3cs.CVcs.AIcs.CLcs.LG

TL;DR

VLMs incur substantial overhead from visual tokens, and existing efficient approaches largely rely on fixed-ratio compression rather than adapting to each sample. AdaptVision acquires visual information coarse-to-fine and trains with DTPO; across VQA benchmarks it outperforms efficient VLM methods while using fewer visual tokens.

  • Problem

    Existing efficient VLMs largely reduce visual tokens at predefined ratios, leaving open whether models can adaptively determine the minimum needed per sample despite the overhead of large visual-token counts.

    Source: §1 · p. 1

  • Method

    AdaptVision begins with compressed low-resolution visual tokens, crops high-resolution regions when needed, and uses DTPO to separate tool-learning and answer-accuracy objectives.

    Source: §1 · p. 2

  • Results

    Across VQA benchmarks, AdaptVision outperforms prior compression methods, and versus down-sample improves accuracy by 5.8% (92.1% →97.9%) with 7% (25% →33%) more visual tokens.

    Source: §5.2 · p. 7, §6 · p. 8

  • Takeaways & Limitations

    The results advance the development of computationally efficient and biologically inspired VLMs.

    Source: §6 · p. 8

    AdaptVision currently relies on one visual tool, fixed 1/4-resolution initialization, and at most two turns, which may limit deep visual reasoning on complex tasks.

    Source: §6 · p. 8

Abstract

from arXiv · show

Vision-Language Models (VLMs) have achieved remarkable success in visual question answering tasks, but their reliance on large numbers of visual tokens introduces significant computational overhead. While existing efficient VLM approaches reduce visual tokens through fixed-ratio compression, they operate passively and lack the ability to adapt to varying task requirements. This motivates a fundamental question: Can VLMs autonomously determine the minimum number of visual tokens required for each sample? Inspired by human active vision mechanisms, we introduce AdaptVision, an efficient VLM paradigm that enables adaptive visual token acquisition through a coarse-to-fine approach. Our model initially processes compressed visual tokens from low-resolution images and selectively acquires additional visual information by invoking a bounding box tool to crop key regions when necessary. We train AdaptVision using a reinforcement learning framework that carefully balances accuracy and efficiency. Central to our approach is Decoupled Turn Policy Optimization (DTPO), which decouples the learning objective into two components: (1) tool learning, which optimizes correct tool utilization, and (2) accuracy improvement, which refines the generated responses to improve answer correctness. Based on this formulation, we further decouple advantage estimation by computing separate advantages for tokens associated with each objective. This formulation enables more effective optimization for AdaptVision compared to vanilla GRPO. Comprehensive experiments across multiple VQA benchmarks demonstrate that AdaptVision achieves superior performance while consuming substantially fewer visual tokens than state-of-the-art efficient VLM methods.

1. Introduction

Visual-token overhead makes VLMs costly, while existing compression is largely fixed-ratio rather than sample-adaptive. AdaptVision uses coarse-to-fine visual acquisition and DTPO to balance tool use, answer accuracy, and efficiency.

1. Introduction

  • VLMs rely on many visual tokens, creating substantial memory and computational overhead, especially for high-resolution images.A 2048 × 1024 image yields 2,678 vision tokens in Qwen2.5-VL.Source: §1 · p. 1
  • Existing efficient VLM methods reduce tokens by predefined ratios, motivating the question of whether models can select the minimum needed for each sample.The paper frames this as adaptation to different scenarios.Source: §1 · p. 2
  • Human active vision first captures coarse scene information, then directs attention to salient regions for detailed analysis.The paper proposes this coarse-to-fine strategy as inspiration for adaptive visual acquisition.Source: §1 · p. 2
  • AdaptVision starts from compressed low-resolution visual tokens and requests high-resolution crops of key regions when needed, using reinforcement learning to balance accuracy and efficiency.The framework aims to determine minimum visual-token use while maintaining high accuracy.Source: §1 · p. 2
  • Vanilla GRPO gives every generated token one sequence-level reward and uniformly normalizes tokens, leaving tool-invoking sequences under-optimized relative to direct answers.The paper identifies ambiguous credit assignment and imbalanced optimization as challenges for its dual-objective policy.Source: §1 · p. 2
  • DTPO separates tool learning from answer-accuracy improvement, normalizes each objective separately, and estimates distinct advantages for tokens associated with each objective.The paper presents this decoupling as a way to balance learning signals and improve tool exploration.Source: §1 · p. 2
  • Across multiple VQA benchmarks, AdaptVision outperforms efficient VLM methods while using fewer visual tokens.The introduction summarizes this result and Figure 1 depicts the performance and efficiency comparison.Source: Figure 1 · p. 1, §1 · p. 2

2. Related work

Prior work studies both visual-token compression and active visual reasoning, but the paper identifies limited exploration of using visual tools specifically for token efficiency. AdaptVision combines coarse-to-fine acquisition with task-adaptive token use.

2. Related work

  • Prior active-vision systems use operations such as zoom and crop to improve detailed visual understanding, while applying this paradigm to token-compression efficiency remains less explored.The paper distinguishes its efficiency goal from prior work focused on answer accuracy.Source: §2 · p. 2
  • Many compression methods use predefined rules or ratios, while VisionThink chooses between low-resolution and original images but remains limited to coarse-grained decisions.Examples of fixed-rule methods include FastV, PyramidDrop, SparseVLM, and VisionZip.Source: §2 · pp. 2–3
  • AdaptVision instead learns coarse-to-fine behavior to determine the minimum visual tokens needed for each task.Figure 2 shows an initial 1/4-resolution image followed by a direct-answer or crop decision.Source: §2 · p. 3, Figure 2 · p. 3

3. Preliminary

The preliminary section introduces GRPO's response-generation and token-level optimization setup, then describes VLM components and why visual-token count matters for efficiency.

3.1. Reinforcement Learning for LLMs

  • GRPO generates a group of responses for each question and optimizes the policy with a clipped token-level objective, a group-based advantage estimate, and a KL penalty.The passage identifies KL divergence and the clipping and penalty coefficients as hyperparameterized parts of the objective.Source: §3.1 · p. 3

3.2. Vision Language Models

  • A VLM combines a visual encoder, modality projector, and LLM; its sequence length includes system, image, and question tokens, with image tokens often dominating.The passage notes that image-token count can be about 20 times the other two components, making its reduction important for efficiency.Source: §3.2 · p. 3

4. Methodology

AdaptVision begins with low-resolution visual tokens and requests crops of high-resolution regions when needed, aiming to limit token use while answering accurately. Its reward design and DTPO train the model to balance answer quality, tool use, and efficiency.

4.1. Framework

  • The model processes a low-resolution image using 25% of the original visual tokens, then answers directly or crops high-resolution regions when more detail is needed.The policy aims to request additional visual information only when necessary and to acquire the minimum needed to answer correctly.Source: §4.1 · p. 4

4.2. Reward Design

  • The reward combines answer correctness, format adherence, and tool-call balance with a tool reward for informative, small crops.Correct answers that use a tool are penalized, while direct answers are penalized when low-resolution confidence is low; the tool reward balances crop relevance against area.Source: §4.2 · p. 4
  • An area penalty encourages selecting the smallest crop that still yields a correct answer, reducing visual-token use while maintaining performance.The penalty uses the mean crop-area ratio among responses with correct answers and correct crops as its reference.Source: §4.2 · pp. 4–5

4.3. Efficient Learning via Decoupled Turn Policy Optimization

  • Vanilla GRPO gives every token one sequence-level reward and can under-optimize tool tokens in two-turn responses, blurring credit for tool requests and answers.In tool-invoking sequences, normalization suppresses tool-token gradient contributions relative to answer tokens.Source: §4.3 · p. 5
  • DTPO separates and independently normalizes the policy losses for tool and answer tokens to address tool-token under-optimization.The two-turn loss is divided into tool-token and answer-token terms with separate normalizing token counts.Source: §4.3 · p. 5
  • DTPO computes distinct advantages from tool and outcome rewards, combining them with a weighting parameter and applying tool advantage to tool tokens in tool-call responses.The paper sets λ = 0.3.Source: §4.3 · p. 5

5. Experiment

AdaptVision is evaluated on multiple VQA benchmarks against fixed- and adaptive-token baselines, with analyses of latency, reward design, policy training, and tool use. The results show a coarse-to-fine approach that uses additional visual information selectively.

5.1. Evaluation Setup

  • The evaluation uses ChartQA, OCRBench, DocVQA, MME, MMVet, RealWorldQA, POPE, MathVista, and MathVerse with Qwen2.5-VL-7B-Instruct.Training data includes samples answerable from low-resolution images and samples requiring high-resolution images for accurate answers.Source: §5.1 · p. 6

5.2. Main Results

  • AdaptVision improves accuracy by 5.8% (92.1% →97.9%) with only 7% (25% →33%) more visual tokens than the down-sample model, and achieves superior average performance with fewer tokens than prior compression methods.The comparison includes FastV, SparseVLM, VisionZip, and VisionThink; static methods use 50% token retention.Source: §5.2 · pp. 6–7
  • AdaptVision achieves 1.67x overall speedup over the vanilla model and VisionThink†, while its added reasoning and tool-call tokens increase inference time relative to down-sampling within acceptable bounds.The speedup is attributed primarily to more efficient visual-token use.Source: Figure 4 · p. 6, §5.2 · p. 7

5.3. Analysis

  • Without the balance reward, tool use quickly becomes excessive; without the tool reward, the model collapses to direct answering and stops invoking the tool after 10 training steps.With the balance reward, tool use adapts to the input; with the tool reward, the model explores and uses the tool to enhance performance.Source: §5.3 · p. 7
  • GRPO training becomes unstable and reaches a 100% tool-call ratio, whereas DTPO learns a stable policy that calls tools selectively according to task difficulty.The authors attribute GRPO's instability to ambiguous credit assignment and imbalanced optimization.Source: Figure 6 · p. 7, §5.3 · p. 8
  • AdaptVision frequently invokes the tool for fine-grained MathVerse and ChartQA tasks but rarely does so for general-understanding POPE tasks.The model answers directly when tools are unnecessary and uses them when beneficial.Source: §5.3 · p. 8

5.4. Case Study

  • In the case study, AdaptVision detects insufficient low-resolution detail, crops a relevant region from the high-resolution image, and answers correctly while minimizing visual-token use.The down-sample model answers incorrectly, while the vanilla model answers correctly at a high visual-token cost.Source: §5.4 · p. 8, Figure 7 · p. 8

6. Conclusion

The conclusion presents AdaptVision as a coarse-to-fine VLM that adaptively determines visual-token use, and reports superior performance with fewer tokens than previous efficient VLM methods. It also identifies limitations in its toolset, initial resolution, and reasoning depth.

6. Conclusion

  • AdaptVision combines adaptive coarse-to-fine visual reasoning with DTPO, which decouples policy objectives and advantage estimation, and outperforms prior efficient VLM methods while using significantly fewer visual tokens.The paper frames the results as advancing computationally efficient and biologically inspired VLMs.Source: §6 · p. 8
  • The framework uses one visual tool, a fixed 1/4-resolution starting point, and at most two reasoning turns, which may limit adaptability and deep visual reasoning for complex tasks.Source: §6 · p. 8

A.1. Prompt Details

  • AdaptVision uses separate prompts for bounding-box tool use, GPT-4o binary answer-correctness judgments, and GPT-4o binary cropped-region relevance rewards.The tool prompt follows the Qwen2.5-VL cookbook [3], while the judging prompt follows Yang et al. [46].Source: §A.1 · p. 11

A.2. Training and Evaluation Details

The appendix specifies the model, reinforcement-learning configuration, and inference settings used for training and evaluation.

A.2. Training and Evaluation Details

  • Training uses Qwen2.5-VL-7B-Instruct with veRL, 16 sampled responses per prompt, 80 training steps, and 4 nodes of 8 H20 GPUs each; inference uses vLLM at temperature 0.The training configuration also specifies mixed-precision FP16, batch size 512, mini-batch size 32, and learning rate 1e−6.Source: §A.2 · p. 11

A.3. Additional Results

Additional comparisons across token-retention ratios find that AdaptVision maintains higher performance with fewer visual tokens than the static compression methods.

A.3. Additional Results

  • Across different token-retention ratios, AdaptVision maintains superior performance with significantly fewer visual tokens, while FastV, SparseVLM, and VisionZip degrade as their ratios are reduced.Source: §A.3 · p. 11

B. Qualitative Results

Qualitative cases show AdaptVision answering directly when low-resolution input suffices and calling a crop tool when more detail is needed. DTPO separates tool and answer-token optimization to improve credit assignment and balance across turns.

B. Qualitative Results

  • AdaptVision answers directly when low-resolution input suffices, but invokes a bounding-box tool for detailed regions, producing correct answers with only a marginal token increase.The direct-answer and tool-call cases are shown in Figures 8 and 9, respectively.Source: Figure 8 · p. 11, Figure 9 · p. 11, B. Qualitative Results · p. 11

C. More Discussion on DTPO

  • DTPO computes distinct advantages for tool and outcome rewards and separately normalizes tool- and answer-token contributions across turns.These choices target more precise credit assignment and balanced optimization across tokens with different functions.Source: C. More Discussion on DTPO · p. 12

D. Robustness of DTPO

The reported experiments find DTPO robust to the tested hyperparameters and show competitive performance with a smaller open-source VLM replacing GPT-4o as the reward judge.

D. Robustness of DTPO

E. Reward Models for AdaptVision

  • Using Qwen3-VL-4B-Instruct instead of GPT-4o to compute the Rcrop reward still yields competitive performance and outperforms VisionThink.The comparison is reported in Table 3.Source: E. Reward Models for AdaptVision · p. 12

F. Generalize with Other VLM Architectures

AdaptVision changes interaction logic and training rather than the VLM backbone, and the authors limit interaction turns to reduce latency while identifying multi-round tool use as a possible accuracy extension.

F. Generalize with Other VLM Architectures

  • AdaptVision modifies interaction logic and training rather than the backbone, and its modular design is described as applicable to architectures such as Qwen3-VL or InternVL.The authors encourage extending AdaptVision to alternative VLM architectures.Source: F. Generalize with Other VLM… · p. 12

G. Extension to Multi-round Tool Calls

  • Interaction turns are limited to reduce latency, although multi-round tool calls could improve accuracy by obtaining finer-grained visual information at increased inference time.The authors suggest a deblurring module after zooming in as one possible extension for improving clarity on small objects.Source: G. Extension to Multi-round Tool Calls · p. 12

Related papers

Loading 2512.03794v3…