PaperlayerView exact revision on arXiv

Source-linked AI summary

Look Where It Matters: High-Resolution Crops Retrieval for Efficient VLMs

Nimrod Shabtay, Moshe Kimhi, Artem Spector, Sivan Haray, Ehud Rivlin, Chaim Baskin, Raja Giryes, Eli Schwartz

arXiv:2603.16932v1cs.CVcs.AI

TL;DR

High-resolution VLM inputs capture fine details but impose substantial computational costs, while low-resolution inputs can miss critical information. AwaRes uses a low-resolution global view and tool-calling to retrieve query-specific high-resolution crops, with automatically curated supervision and staged SFT plus GRPO training. Across six benchmarks, it matches full-resolution performance on average while using substantially fewer visual tokens and improves efficiency relative to global resolution escalation.

  • Problem

    High-resolution VLM inference is computationally expensive, while low-resolution inputs may miss critical fine-grained visual information.

  • Method

    AwaRes combines a low-resolution global view with tool-called high-resolution crops, automatically curated crop supervision, cold-start SFT, and multi-turn GRPO with crop-cost penalties.

  • Results

    80.3% versus 80.46% average performance across six benchmarks is achieved with 36% of the pixels/tokens, while improving efficiency relative to global resolution-escalation baselines.

  • Takeaways & Limitations

    Spatial-on-demand crop acquisition provides a practical path toward high-detail multimodal reasoning under tight compute and latency budgets.

  • Takeaways & Limitations

    Failures arise when question-relevant detail occupies a small or ambiguous image region, causing crops to capture nearby but insufficient context.

Abstract

from arXiv · show

Vision-language models (VLMs) typically process images at a native high-resolution, forcing a trade-off between accuracy and computational efficiency: high-resolution inputs capture fine details but incur significant computational costs, while low-resolution inputs advocate for efficiency, they potentially miss critical visual information, like small text. We present AwaRes, a spatial-on-demand framework that resolves this accuracy-efficiency trade-off by operating on a low-resolution global view and using tool-calling to retrieve only high-resolution segments needed for a given query. We construct supervised data automatically: a judge compares low- vs.\ high-resolution answers to label whether cropping is needed, and an oracle grounding model localizes the evidence for the correct answer, which we map to a discrete crop set to form multi-turn tool-use trajectories. We train our framework with cold-start SFT followed by multi-turn GRPO with a composite reward that combines semantic answer correctness with explicit crop-cost penalties. Project page: https://nimrodshabtay.github.io/AwaRes

1 Introduction

AwaRes addresses the cost of high-resolution VLM inference by combining a low-resolution global view with targeted high-resolution crops selected through tool-calling. It automatically curates tool-use supervision and trains a policy to balance answer accuracy with crop efficiency.

  • High-resolution VLM inference is costly because visual-token counts grow rapidly with image resolution, despite detail-sensitive tasks requiring fine information.
  • Fine-grained information is often spatially sparse, so retrieving targeted regions can avoid the cost of processing the full image at native resolution.
  • AwaRes first processes a low-resolution global view, then retrieves only query-relevant high-resolution sub-regions through a tool call when additional detail is needed.The multi-turn structure supports KV-cache reuse without architectural changes.
  • The coupled-decision policy jointly determines whether more resolution is needed and which crop subset to acquire, combining escalation with localization in the first-turn action.
  • Automatic supervision compares low- and high-resolution outputs to identify when cropping is necessary and uses oracle grounding to localize supporting evidence without manual spatial annotations.
  • 80.3% versus 80.46% average performance across six benchmarks is achieved with 36% of the pixels/tokens, while slightly improving over full-resolution baselines on ChartQA, DocVQA, and OCRBench.

3 Method

AwaRes uses a low-resolution global view by default and retrieves targeted high-resolution crops only when additional detail is needed. Automatically curated trajectories teach this coupled decision before GRPO optimizes answer quality against crop costs.

  • Problem setup: AwaRes first processes a low-resolution global view, then requests selected high-resolution crops when needed before producing the final answer.The crop turn retains the low-resolution context and appends retrieved crops to the dialogue.
  • Problem setup: A single policy couples whether to escalate with which crop subset to request, treating an empty crop set as a direct answer.The value of escalation depends on localization because inaccurate regions can waste computation without improving correctness.
  • Automatic supervision: Automatic supervision compares low- and full-resolution predictions with an LLM judge, labeling examples as low-resolution-sufficient or high-resolution-required.The judge uses semantic comparison to ground truth rather than relying on exact string matching.
  • Automatic supervision: For high-resolution-required examples, an oracle grounding model localizes answer evidence and maps its bounding box to a discrete crop set using an IoU threshold.The candidate set contains quadrants, half-image regions, a center crop, and the full image; the target includes candidates with IoU at least 0.5.
  • Training: The resulting direct-answer and tool-call-then-answer transcripts are used for SFT, which teaches the protocol and initializes a KL-regularized GRPO reference policy.The tool-call turn is upweighted because it controls both efficiency and downstream answer quality; GRPO then addresses SFT’s tendency to over-request crops.
  • Training: GRPO optimizes completed multi-turn trajectories with a semantic answer reward minus an asymmetric tool-use cost.The cost penalizes missed tool calls on high-resolution-required examples and penalizes crop usage according to the requested set and its cost weight.

4 Experimental Results

Across six benchmarks, AwaRes maintains near-full-resolution accuracy while substantially reducing visual-token use and wall-clock latency. Ablations show that GRPO and crop-cost rewards refine SFT’s tendency to overuse crops.

  • Main results: 80.30 average score matches the 80.46 full-resolution baseline while using 0.36× the visual tokens.The policy answers from the low-resolution overview when possible and requests targeted crops when necessary.
  • Main results: 80.30 vs. 79.23 improves accuracy over VisionThink while RTR falls from 0.61 to 0.36.On ChartQA, AwaRes exceeds the full-resolution baseline by +0.84 percent at RTR 0.32, while VisionThink reaches RTR 1.15.
  • Ablations: GRPO shifts crop decisions toward selective tool use, increasing LR decisions from SFT’s 46.3% to 72.2% and reducing “All” from 16.6% to 4.9%.The explicit tool-use cost encourages efficient strategies that may differ from oracle annotations.
  • Ablations: Joint trajectory-level SFT improves average performance and efficiency over split training, reaching 77.90 versus 75.15 accuracy and 0.43 versus 0.66 RTR.Increasing the tool-turn weight to 5 reaches 79.70 accuracy at RTR 0.49, whereas GRPO-only training barely uses the tool and has lower accuracy.
  • Ablations: Removing crop-area cost raises RTR from 0.36 to 0.42, while removing crop cost entirely raises it to 0.51.SFT initialization and KL regularization limit the increase when tool cost is removed entirely.

5 Conclusion

AwaRes uses low-resolution global views and selectively retrieves high-resolution crops through tool calls, trained with automatic supervision and multi-turn optimization. Across six benchmarks, it matches average full-resolution performance with substantially fewer visual tokens, while future work targets finer crop control and video.

  • 5 Conclusion: AwaRes preserves a low-resolution global view and retrieves only query-relevant high-resolution crops through a tool-calling interface.Multi-turn KV-cache reuse supports deployment-friendly inference.
  • 5 Conclusion: Automatic supervision labels resolution needs, grounds supporting evidence, and creates multi-turn crop-request trajectories without manual spatial annotations.Cold-start SFT is followed by multi-turn GRPO with semantic correctness and crop-usage penalties.
  • 5 Conclusion: Across six benchmarks, AwaRes matches full-resolution performance on average while using substantially fewer visual tokens and improving efficiency over global resolution-escalation baselines.The evaluation spans document understanding and general visual question answering.
  • 5 Conclusion: Future work may extend crop selection from a discrete set to continuous bounding boxes and generalize spatial-on-demand perception to video understanding.These directions target finer spatial control and potential temporal-sparsity benefits.

Supplementary Materials

The supplementary materials add analyses and visual examples covering data curation, latency, response length, and cold-start training. They include both successful annotations and failure cases from the automatic pipeline.

  • Supplementary Materials: The supplementary document provides additional details, analyses, and visual examples that complement the main paper.Its topics include data curation, latency measurements, response length, and cold-start SFT analysis.
  • Supplementary Materials: Section 3.2 presents supplementary visual examples of successful annotations and failure cases from the automatic data-curation process.The examples cover the pipeline’s annotation behavior.
  • Supplementary Materials: The latency analysis compares AwaRes and VisionThink using hardware setup and wall-clock measurements, alongside response-length analysis of efficiency gains.The supplementary document organizes these analyses as dedicated sections.

1 Data Annotation

The supplementary data-annotation section illustrates automatic-process failure cases and characterizes the resolution distribution of the training data. Resolution varies across datasets and is capped at 2000×2000.

  • 1 Data Annotation: Figure 2 illustrates failure cases where the automatic data-curation process did not produce satisfactory results.The section pairs these failures with supplementary visual examples from the pipeline.
  • 1 Data Annotation: DocVQA has the highest training-data resolutions, while LLaVA-Multi has the lowest; VisionThink-Smart and ChartQA span broader resolution ranges.TextVQA concentrations cluster around 1000 pixels along one axis.
  • 1 Data Annotation: All resolutions are capped at 2000×2000 because prior work found negligible performance impact from truncating DocVQA resolution.The cap standardizes the maximum image resolution used in the data.

2 Benchmark Qualitative Examples

The qualitative examples show AwaRes selecting high-resolution regions containing task-relevant details across charts, documents, and natural images. Failure cases occur when cropping or contextual evidence remains insufficient.

  • 2 Benchmark Qualitative Examples: Figures 4–9 show successful crops whose predicted answers match ground truth across charts, documents, and natural images.Each conversation includes the question, model tool call, answer, low-resolution input, and retrieved crop.
  • 2 Benchmark Qualitative Examples: Figure 1 contrasts chart and natural-image curation examples, showing oracle boxes for question and answer regions alongside the selected training crop.The natural-image example requires OCR of a brand banner.
  • 2 Benchmark Qualitative Examples: The retrieved crops isolate relevant details such as axis labels, table cells, brand logos, and foreground subjects.These details are fine-grained and unresolvable in the low-resolution overview alone.
  • 2 Benchmark Qualitative Examples: Figure 10 shows failures involving incorrect regions or correct crops followed by wrong answers.These failures typically occur when relevant detail is small or ambiguous and nearby context is insufficient.

3 Latency Analysis

AwaRes was evaluated against VisionThink using wall-clock latency on an Nvidia H100-80GB GPU. Across six benchmarks, it maintained competitive accuracy while substantially reducing latency and response verbosity.

  • Latency comparison: 4.4× lower average wall-clock time: AwaRes reduced latency from 2.71s to 0.61s versus VisionThink while improving the average metric score from 79.23 to 80.47.The comparison used native HuggingFace implementations and measured generation from start to final answer.
  • Benchmark-specific latency: 7.7× faster on ChartQA and 5.3× faster on OCRBench, where VisionThink’s extended reasoning traces created substantial overhead.
  • Response verbosity: 5.8×–28.8× more characters were generated by VisionThink than AwaRes across benchmarks.The response-length difference reflects efficiency beyond visual-token savings.
  • Predictability: AwaRes produced substantially lower response-length variance across all benchmarks, making completion times and latency more predictable.VisionThink’s standard deviations often exceeded its means, complicating deployment in latency-sensitive applications.

4 Additional Cold-Start (SFT) Analysis

The cold-start analysis treats crop invocation and crop localization as coupled policy decisions. Trajectory-level SFT improves escalation reliability, localization quality, tool-call validity, and downstream efficiency, while objective and data choices trade off accuracy against crop usage.

  • CDP formulation: The coupled-decision policy jointly determines whether to request additional resolution and which crop subset to retrieve.No-call actions use C=∅, while nonempty C represents escalation with localization.
  • Policy diagnostics: Call precision, recall, F1, and LR false-positive rate evaluate when the tool is invoked, while exact match, relaxed IoU match, and average area evaluate where it looks.Region metrics are conditioned on calling and compare requested crops with oracle targets.
  • Call decision: Trajectory-level SFT increased HR-call recall from 15.34 to 41.02 and reduced LR-call FPR from 79.69 to 63.33 relative to baseline SFT.Baseline SFT had moderate precision but unreliable escalation on HR cases and frequent calls on LR cases.
  • Region decision: Trajectory-level SFT increased relaxed region matching from 32.6 to 48.85 and reduced average requested area from 0.59 to 0.402.Upweighting the tool-call turn further raised exact match to 41.3 and relaxed match to 75.5, with area increasing to 0.463.
  • Sensitivity analysis: Trajectory-level SFT improved average accuracy from 75.15 to 77.90 while increasing call rate from 22.55% to 25.21% and reducing average requested area from 0.59 to 0.402.These changes indicate more frequent but more localized crop requests.

5 ANLS-Based Data Curation Analysis

The analysis compares LaaJ and ANLS for labeling resolution sufficiency during cold-start data curation. LaaJ-based labels produce higher accuracy, especially on text- and detail-sensitive benchmarks, despite higher crop usage.

  • Labeling strategy: ANLS is a string-oriented OCR similarity measure that is ill-suited to resolution-sufficiency labels when cropped inputs produce semantically correct but perturbed answers.
  • Overall comparison: 79.70 versus 75.9 average accuracy favored LaaJ-based labels over ANLS-based labels.ANLS labeling also reduced RTR from 0.49 to 0.38.
  • Benchmark comparison: 77.0 versus 70.5 on ChartQA and 78.8 versus 70.0 on OCRBench favored LaaJ over ANLS.The accuracy drops were pronounced on text- and detail-sensitive benchmarks.

6 Training Details

The supplementary material provides the training details for AwaRes’s two training stages and reports the labeling-strategy comparison used in the data-curation analysis.

  • Training stages: Cold-Start SFT and Tool Optimization via GRPO are the two training stages detailed for AwaRes.
  • Labeling evaluation: Table 6 compares LaaJ using LLaMA-3.3-70B with ANLS across benchmark Accuracy and RTR.
  • Hyperparameters: Table 7 lists the training hyperparameters for the Cold-Start SFT and Tool Optimization stages.

7 Prompts

Section 7 specifies prompts for judging answer quality, grounding question and answer regions, and controlling adaptive crop requests. It also illustrates successful and failed cropping behaviors across charts, documents, labels, and outdoor scenes.

  • 7.1 LLM-as-a-Judge Prompt: The LLM-as-a-Judge prompt compares two responses with a ground-truth answer and returns whether Response 1, Response 2, or neither is better.The output is restricted to the labels 0, 1, or 2.
  • 7.2 Oracle Grounding Prompt: The oracle grounding prompt asks the model to identify both question regions and answer regions relevant to the visual question.Each region is represented by a bounding box in normalized image coordinates.
  • 7.2 Oracle Grounding Prompt: Oracle boxes use coordinates from 0 to 1000, with tight top-left and bottom-right bounds around relevant text or visual elements.Multiple boxes may be supplied, and question and answer regions may share coordinates when they overlap.
  • 7.2 Oracle Grounding Prompt: The oracle output is constrained to a Python dictionary containing only question and answer bounding-box lists, without explanations or other text.The required format explicitly lists one or more normalized boxes for each key.
  • 7.3 SFT / GRPO / Inference Prompt: The inference prompt requests GET_CROPS with crop numbers when the low-resolution image is insufficient for accurate analysis.The available crop identifiers correspond to spatial areas such as corners, center, top, bottom, left, right, or the full image.
  • 7.3 SFT / GRPO / Inference Prompt: Adaptive crops isolate chart segments, axis labels, trend points, and categories to recover precise numerical values that are unclear in the low-resolution overview.Examples include finding the smallest value, computing a median, and reading a specific percentage.
  • 7.3 SFT / GRPO / Inference Prompt: Targeted crops resolve fine-grained text and attributes in scanned documents, product labels, cluttered displays, and partially occluded objects.The examples include extracting a weight, reading a wine variety or brand, identifying colors, and recognizing a small object.
  • 7.3 SFT / GRPO / Inference Prompt: Other examples use crops for spatial counting and direction tasks, while failures arise from insufficient question context or incorrect reading after localization.Reported errors include using 2012 instead of 2013 in an average and reading “Aspartame” from the wrong package region.
Loading 2603.16932v1…