Adaptive Chain-of-Focus Reasoning via Dynamic Visual Search and Zooming for Efficient VLMs

• Constant zooming wastes visual tokens through unnecessary operations and can impair models in long contexts, motivating decisions about when evidence warrants another zoom.Source: §1 · pp. 2–3
• Adaptive-CoF answers from a global view when sufficient, otherwise searching and zooming into key regions to gather more visual cues.It appends the resulting visual tokens to the context for further analysis.Source: §1 · p. 3
• The SFT stage uses 5K reasoning trajectories constructed from SAM images and generated by a visual agent to cold-start Qwen2.5-VL-7B.Source: §1 · pp. 3–4
• In RL, adaptive group-aware rewards based on outcome accuracy and format refine decisions about answering directly or zooming, without human priors.Source: §1 · p. 4
• 18.9% gain on V⋆Bench and 8.6% boost on MME-RealWorld-Lite over Qwen2.5-VL-7B.The reported gains are from experiments on multiple challenging benchmarks.Source: §1 · p. 4
• On HR-Bench 4K, Adaptive-CoF uses about 5.4% of DeepEyes’ visual tokens, with zoom-in operations decreasing by up to 77.8%.The paper describes this as an adaptive reduction in zooming that cuts computational costs.Source: §1 · p. 4

2.1. Vision Language Models

• Adaptive-CoF searches and zooms key regions rather than processing high-resolution images wholesale, aiming to reduce irrelevant-region costs while improving reasoning.High-resolution images can provide more visual detail, while the approach selects regions in a chain of focus.Source: §2.1 · p. 5

2.2. Multimodal Reasoning

• Prior multimodal exploration methods can use frequent, redundant zoom-ins that raise computational and token costs; Adaptive-CoF decides whether more visual operations are necessary.It can instead rely on the current input when that is sufficient.Source: §2.2 · p. 6

2.3. LLM Adaptive Reasoning

• Where adaptive-reasoning approaches may depend on hand-crafted rules, Adaptive-CoF uses reinforcement learning to learn a context-aware policy without external routers or predefined prompts.The learned policy decides when deeper visual exploration is necessary.Source: §2.3 · p. 6

3.1. Formulation

• At each step, the VLM uses the image, query, and prior history to generate key regions and a textual response, which may be an answer or intermediate reasoning.The history contains prior regions, responses, and visual tokens; the framework illustration distinguishes textual, visual, bounding-box, and answer tokens by color.Source: §3.1 · p. 7, Figure 2 · p. 7

3.2. Architecture

• The model crops selected bounding boxes, enlarges them by a factor of two, encodes the regions, and appends their visual tokens to context for joint reasoning.Adaptive-CoF also learns when and how to re-examine regions through zoom-in operations.Source: §3.2 · p. 8

4. MM-Adaptive-CoF Dataset

• MM-Adaptive-CoF is built on images sampled from SA_1B, which predominantly features high-resolution images, and has distinct splits for supervised fine-tuning (SFT) and reinforcement learning (RL).Source: §4 · p. 8

4.1. Query-Response Generation

• The same query can have dynamic zoom-in needs across resolutions, while different queries have varying zoom requirements; QA pairs with consistent judgments are retained after five repetitions per resolution.Source: §4.1 · pp. 9–10

4.2. MM-Adaptive-CoF Supervised Fine-tuning (SFT) Dataset

• The SFT dataset selects 2,201 zoom-requiring and 2,294 directly answerable QA pairs, then adds step-by-step agent traces for zoom cases and GPT-4.1 reasoning chains for direct-answer cases.The resulting dataset D supplies answers and explanatory reasoning for Chain-of-Thought training.Source: §4.2 · p. 10

4.2.1. Visual Search Agent

• The agent iteratively locates and refines bounding boxes, generates grounded answers, and uses verifier feedback to correct its search until the answer is found or a step limit is reached.The dataset records these grounding and adjustment steps, providing supervision for the intermediate process as well as the final answer.Source: §4.2.1 · p. 12
• The visual search agent uses iterative tool-based reasoning and bounding-box adjustment rather than direct detection, recording intermediate grounding steps for interpretable supervision.This approach can ground small or ambiguous objects, including text regions, by combining visual cues, tool operations, and commonsense reasoning.Source: §4.2.1 · pp. 12–13

4.2.2. Dataset Analysis

• SFT sample counts are generally balanced across image resolutions, with slightly more high-resolution samples to reflect the focus on small-object understanding and the source data’s native resolutions.The sample-count distribution is shown in Fig. 7(a).Source: §4.2.2 · p. 14
• Zoom-in requirements decrease as image resolution increases, consistent with higher-resolution images providing richer visual detail.The authors report that the dataset is balanced across resolutions and reflects realistic zoom-in needs.Source: §4.2.2 · p. 14
• Single-turn and multi-turn samples average about 2.2k each, while sample frequency drops sharply with more turns and very long trajectories are rare.The authors characterize the dataset as covering diverse reasoning depths that can support adaptive choices between single-step answers and multi-step reasoning with zoom-in operations.Source: §4.2.2 · p. 14
• The dataset statistics are intended to encourage adaptive choices between single-step answers and multi-step reasoning with zoom-in operations.The authors state that these capabilities are intended to improve efficiency and interpretability in multimodal reasoning tasks.Source: §4.2.2 · pp. 14–15

4.3. MM-Adaptive-CoF Reinforcement Learning (RL) Dataset

• The RL stage uses approximately 10,000 diverse query-response instances, including 1,500 QA pairs from proprietary MM-Adaptive-CoF, augmented with VisDrone [59], ArxivQA, and ThinkLite-VL [42].These external datasets are included to enhance diversity and robustness of RL training.Source: §4.3 · p. 15
• The combined RL dataset is denoted U and is used for policy optimization to learn more generalized and robust adaptive reasoning capabilities.Its external data sources provide diverse reasoning challenges.Source: §4.3 · p. 15

5. Training

• Training proceeds from supervised fine-tuning on MM-Adaptive-CoF data to reinforcement learning on its RL dataset using Qwen2.5-VL-7B.Source: §5 · p. 15, §5.2 · p. 16

5.1. Cold-Start Tuning

• Cold-start tuning teaches textual and multimodal reasoning, including bounding-box predictions and zoom-in operations, while freezing the ViT and applying LoRA to remaining components.The model is trained on the MM-Adaptive-CoF SFT dataset with cross-entropy to prepare coherent traces for reward-based optimization.Source: §5.1 · pp. 15–16

5.2. Reinforcement Learning

• GRPO samples groups of trajectories and uses their relative reward rankings to update the policy, computing policy gradients only on textual reasoning tokens.Source: §5.2 · pp. 16–17

5.2.1. Adaptive Group-Aware Reward

• Adaptive group-aware reward adjusts the zoom-in path’s reward using whether any sampled response answers directly and correctly, promoting efficiency on simple tasks and deeper reasoning when needed.The method combines correctness and format signals, and the zoom-in path receives a reduced reward when an efficient direct solution exists.Source: §5.2.1 · pp. 17–18

6.1.1. Datasets and Benchmarks

• Benchmarks span high-resolution fine-grained QA, MME-RealWorld perception and reasoning, spatial reasoning, counting, and hallucination reliability.The suite includes V⋆Bench and HR-Bench, SpatialScore, TallyQA, and POPE.Source: §6.1.1 · pp. 18–19

6.1.2. Implementation Details

• The model uses Qwen2.5-VL-7B as its base and is trained in two stages: supervised fine-tuning followed by reinforcement learning.Source: §6.1.2 · p. 19, Table 1 · p. 19

6.1.3. Baselines

• Baselines include tool-based pipelines, end-to-end zooming models, and general VLMs including Qwen2.5-VL; unlike proactive zoomers, Adaptive-CoF explores only when needed.Source: §6.1.3 · pp. 19–20

6.2.1. Results on General Models

• Adaptive-CoF establishes open-source SOTA performance in the 7B–8B class across a wide range of benchmarks and substantially improves over its baseline.Source: §6.2.1 · p. 20, Table 2 · p. 21
• Adaptive-CoF improves over Qwen2.5-VL-7B by 18.9% on V⋆Bench and reports a 90.1 V⋆Bench score against visual-grounded reasoning models.It reports overall SOTA on HR-Bench 4K and remains highly competitive on HR-Bench 8K.Source: §6.2.1 · p. 20, Table 2 · p. 21, §6.2.2 · p. 22
• Adaptive-CoF reaches an overall MME-RealWorld-Lite score of 50.9, the highest among listed models, while leading in several sub-categories.Source: §6.2.1 · p. 20, Table 3 · p. 21, §6.2.2 · p. 22
• Adaptive-CoF reports a 3.4% POPE gain over its base model, a TallyQA score of 75.0 with a 6.4% improvement, and a SpatialScore of 20.6 with a 5.4% gain.Source: §6.2.1 · p. 20, Table 4 · p. 21

6.2.2. Results on Visual Grounded Reasoning Models

• On HR-Bench 4K, Adaptive-CoF uses about 5.4% of DeepEyes’ visual tokens and reduces zoom-ins by 77.8%; on V⋆Bench, zoom-ins fall by 75%.The paper reports comparable or SOTA performance alongside these efficiency gains.Source: §6.2.2 · pp. 22–23, Table 7 · p. 23
• Adaptive-CoF reports SOTA on SpatialScore and a POPE score of 89.3, alongside a highly competitive TallyQA result.Source: Table 8 · p. 23, §6.2.2 · p. 23

6.3.1. Statistics and Results

• Average zoom-in calls fall as image resolution rises, from about 2.5 calls at 224px to about 0.5 at 2560px.The analysis reports fewer, more effective zoom-ins at very high resolutions and a 10.47% accuracy gain at 2560px.Source: §6.3.1 · pp. 23–24
• Adaptive-CoF’s zooming strategy is associated with gains across resolutions: it zooms to address low-resolution limitations and refrains when high-resolution input is sufficient.Source: §6.3.1 · p. 24

6.3.2. Efficiency Analysis

• MITE, accuracy gain per 100 additional visual tokens relative to baseline, peaks at 30.8 for images above 1024px.The paper reports modest MITE at lower resolutions and substantially higher efficiency at high resolutions.Source: §6.3.2 · p. 25
• Adaptive-CoF surpasses the baseline’s approximately 80% peak accuracy with just over 2,000 visual tokens, versus over 4,000 for Qwen2.5-VL-7B—nearly 50% fewer tokens.Source: §6.3.2 · p. 25

6.4.1. Visualized Reasoning Cases

• Adaptive-CoF answers simple queries directly, uses one zoom for moderately challenging ones, and performs iterative search for difficult fine-grained queries.The examples include identifying a visible bus, recognizing a distant jacket’s color, and locating a small lock after correcting an initial failure.Source: §6.4.1 · p. 25, Figure 9 · p. 26

6.4.2. Adaptive Case Study

• At 448 pixels, the bucket is initially invisible; two successive zooms locate it and identify it as white, while at 1024 pixels one zoom suffices.At 1024 pixels the bucket is visible, but its color remains unclear before zooming.Source: §6.4.2 · p. 27
• For the bag-color query, the model searches iteratively at 672 pixels, adds one zoom at 1024 pixels, and answers directly at 1920 pixels.The bag is initially difficult to locate at 672 pixels, its color remains ambiguous at 1024, and it is clearly visible at 1920.Source: §6.4.2 · pp. 27–28
• The case studies report that reasoning depth tracks visual clarity, moving from multi-step exploration at low resolutions to direct answers at high resolutions.The authors state that this maintains accuracy while reducing computational cost.Source: §6.4.2 · pp. 27–28

6.5. Ablation Study

• SFT yields substantial improvements, raising MME-RealWorld-Lite from 42.3 to 50.1 and V⋆Bench from 71.2 to 89.5, highlighting the SFT stage’s importance.The authors associate the SFT stage with establishing foundational Adaptive-CoF reasoning patterns and tool-calling comprehension.Source: §6.5 · p. 28
• Correctness-and-format rewards alone lead the model to avoid zoom-in reasoning, degrading performance and weakening visual grounding.The variant combines 0.9 correctness and 0.1 format scores.Source: §6.5 · p. 28
• Adding adaptive group-aware RL refines decisions about when and how to zoom, whereas optimizing only correctness and format collapses adaptive interaction behavior.The authors describe the adaptive reward as better aligned with the intended visual reasoning objectives.Source: §6.5 · pp. 28–29
• The RL variant with text-only chain-of-thought and no zoom-in reasoning shows a clear performance drop, indicating textual reasoning cannot fully substitute for adaptive visual search.The ablation is presented as a test of explicit visual interaction.Source: §6.5 · pp. 28–29

7. Conclusion

• Adaptive-CoF enables VLMs to perform adaptive fine-grained visual search and zooming through a two-stage SFT-and-RL training pipeline.The framework is described as balancing detailed perception with computational efficiency rather than static viewing or exhaustive zooming.Source: §7 · p. 29
• Experiments demonstrate state-of-the-art performance on challenging benchmarks with significantly reduced computational cost.The conclusion characterizes this as balancing detailed perception and computational efficiency.Source: §7 · p. 29
• The current work focuses on single-image query–response tasks; visual dialogue and multi-image reasoning across scenes are future extensions.Source: §7 · p. 29