Learning to Look Again: Loss-Gap Supervision for Free-form Crop Routing in Vision-Language Models

Jinchang Zhu; Rong Fu; Yi Ding; Chenghao Wu; Ying Liu; Menglin Yang

Source-linked AI summary

Learning to Look Again: Loss-Gap Supervision for Free-form Crop Routing in Vision-Language Models

Jinchang Zhu, Rong Fu, Yi Ding, Chenghao Wu, Ying Liu, Menglin Yang

arXiv:2608.21762v1Aug 22, 2026cs.CVcs.CL

TL;DR

Detail-centric VLM questions can fail when answer-critical evidence disappears in a low-resolution global view, while uniformly adding visual tokens wastes computation and can disturb global-context tasks. GapSight uses target-model answer changes under candidate crops to train a one-shot router that decides whether and where to re-read. Across three VLM backbones and six benchmarks, it improves Base no-zoom performance; on InternVL2.5-8B, Six-Bench Avg reaches 64.29 from 52.25.

Problem
Detail-centric VLM answers can fail when evidence visible in the image is lost in a low-resolution global view, while uniform token increases spend computation indiscriminately.
Method
GapSight compares target-model answer loss or correct-option margin under global-only and candidate crop-augmented views, then distills the gaps into a router predicting review action, utility, and a continuous crop box.
Results
64.29 Six-Bench Avg is reached by InternVL2.5-8B versus 52.25 for Base no-zoom, while GapSight improves corresponding Base no-zoom settings across six benchmarks and three backbones.
Takeaways & Limitations
Loss-gap supervision provides a practical way to teach VLMs when and where to look again, with task-adaptive and model-specific visual behavior.
Takeaways & Limitations
Crop labels are mined separately for each backbone because crop utility depends on model-specific visual and language behavior.

Abstract

from arXiv · show

Vision-language models (VLMs) fail many detail-centric questions for a concrete reason: the answer is visible in the image, yet lost after the image is compressed into a low-resolution global view. Allocating more visual tokens to every query improves some OCR and document cases, but it spends computation indiscriminately and can disturb tasks that rely on global context. We propose GapSight, a framework for learning visual re-reading: a VLM first takes a global glance, then selectively returns to a free-form region when the question calls for local evidence. The supervision comes from the target model's own failure signal. Offline, we compare answer loss or multiple-choice option margin under a global-only view and candidate crop-augmented views; crops that improve the target answer become model-specific review labels. A lightweight free-form crop router distills these labels into a one-shot inference policy that predicts whether to review, expected utility, and a continuous crop box from the global state. Across LLaVA-1.5-7B, InternVL2.5-8B, and Qwen2-VL-2B-Instruct, GapSight improves the Base no-zoom baseline on six benchmarks spanning OCR, documents, charts, infographics, VStarBench, and MME-RealWorld-Lite. On InternVL2.5-8B, GapSight raises the six-benchmark average from 52.25 to 64.29, above CropVLM (57.16), ViCrop (55.84), and ZoomRefine (54.43). Mechanism analyses show that the router rescues concrete wrong answers, adapts its action rate by task, and forms a favorable token-performance profile. These results position loss-gap supervision as a practical route to teaching VLMs when and where to look again.

Introduction

GapSight learns when and where a VLM should re-read an image by converting target-model answer improvements from candidate crops into a one-shot free-form routing policy. Across three backbones and six benchmarks, it improves no-zoom baselines while adapting review behavior and visual-token use.

Loss-gap supervision: GapSight mines model-specific supervision by comparing answer loss or correct-option margin under global-only and candidate crop-augmented views.Positive gaps identify crops that improve the target VLM’s answer behavior and supervise both review decisions and crop locations.
Practical design: GapSight aligns supervision to each target backbone, separates re-reading from answer generation, and exposes thresholds that control the visual budget.The same crop can have different utility across models, while gate and utility thresholds adjust review frequency.
Evaluation: Across LLaVA-1.5-7B, InternVL2.5-8B, and Qwen2-VL-2B-Instruct, GapSight improves the corresponding Base no-zoom settings across six benchmarks.The benchmarks cover OCR, documents, charts, infographics, small-object reasoning, and high-resolution real-world multiple-choice understanding.
Evaluation: 64.29 Six-Bench Avg is achieved by InternVL2.5-8B, rising from 52.25 and exceeding CropVLM (57.16), ViCrop (55.84), and ZoomRefine (54.43).The reported average is obtained with 391.0 average visual tokens, below the token cost of the external crop baselines.
Mechanism analysis: 107 error repairs versus 27 regressions show that the InternVL2.5-8B policy rescues more wrong answers than it introduces across four VQA benchmarks.The gate reviews more often on text- and infographic-heavy tasks and abstains more when global scene context matters.
Free-form routing: The lightweight router predicts from the global image state whether to review, expected review utility, and a continuous crop box before crop tokens are available.At inference, it preserves the preview or injects one crop into the global-plus-crop context before final answering.

Related Work

Prior work improves VLM perception through higher resolution, dynamic token allocation, visual search, and learning from model behavior. GapSight differs by using target-model answer changes to train a one-shot policy for deciding when and where to crop.

High-resolution evidence: High-resolution VLM research couples visual encoders with language models for broad multimodal transfer and improved multimodal perception.Examples include CLIP, Flamingo, BLIP-2, PaLI, and recent open VLM families.
Dynamic resolution and visual-token allocation: Dynamic-resolution methods change how pixels become tokens through variable resolution, image partitioning, aspect-ratio handling, or token pruning and merging.NaViT, LLaVA-UHD, and Monkey address flexible or high-resolution visual inputs, while other methods target visual efficiency.
Visual search, cropping, and zooming: Visual search, cropping, and zooming methods explicitly reinspect image regions to resolve visual challenges before or during VLM answering.V*, ViCrop, ZoomEye, and Zoom-Refine represent this inference-oriented line of work.
Visual search, cropping, and zooming: GapSight makes the choice between local magnification and global context trainable by distilling crop improvements into a one-shot router.Offline probing records answer-loss or correct-option-margin gains, which supervise action, utility, and continuous-box prediction.
Learning supervision from model behavior: Learning-from-behavior methods use comparisons, feedback, critiques, or reasoning traces as training signals; GapSight applies this view to visual action learning.Its loss gap compares global-only and crop-augmented answer behavior to decide whether to preserve or review a region.
Text-rich and high-resolution evaluation: Text-rich benchmarks test whether limited visual tokens preserve text, layout, charts, small objects, and high-resolution real-world evidence.OCRBench, MMBench, MMMU, and MathVista broaden evaluation pressure beyond generic image understanding.

Method

GapSight learns a one-shot policy that decides from a global preview whether to preserve context or review one free-form crop, using target-model answer improvements as supervision.

Policy outputs: The router predicts preserve versus review, expected utility, and a normalized continuous crop box from hidden states of the global preview.It commits before rendering review-crop tokens, matching the deployment-time allocation decision.
Candidate probing: Offline candidate probing compares target-model answer behavior between global-only and crop-augmented views.Candidate banks contain diverse centers, scales, aspect ratios, compact regions, and context-expanded regions.
Utility supervision: Generated-answer crop utility is the reduction in answer negative log-likelihood, while multiple-choice utility is the change in correct-option margin.A positive loss gap means the crop makes the target answer easier for the target VLM.
Label mining: Clearly positive best utilities produce review labels for action, utility, and box; low or negative utilities produce preserve labels without positive box targets.Ambiguous middle cases are filtered or down-weighted, so the three router targets derive from one answer-consequence signal.
Box prediction: The free-form box combines a candidate-bank prior with bounded residual refinement, allowing crops to follow text blocks, chart regions, document fields, or localized objects.Rendering may context-expand the crop when neighboring layout is needed.
Training and inference: Training uses dual-path preserve and review examples, with gate, utility, and box losses optimized while the VLM backbone remains fixed.Inference encodes the global preview, optionally injects one crop, and generates the answer in the standard format.

Experiments

GapSight is evaluated across three VLM backbones and six complementary benchmarks, using model-specific mined labels and comparisons against external crop methods. Experiments also assess supervision sources, token cost, and the score–cost frontier.

Evaluation Setup: Six benchmarks cover scene text, documents, charts, infographics, fine-grained spatial reasoning, and high-resolution multiple-choice understanding.The evaluation uses fixed held-out subsets, with scores averaged over three seeds.
Evaluation Setup: Three backbones—LLaVA-1.5-7B, InternVL2.5-8B, and Qwen2-VL-2B-Instruct—are evaluated with free-form and multiple-choice supervision formats.Free-form datasets use answer-likelihood labels, while multiple-choice settings use option-margin labels.
Main Results: GapSight improves Six-Bench Avg over Base no-zoom on every reported model row, reaching 64.29 from 52.25 on InternVL2.5-8B.The same comparison reports 9.81 points of improvement for LLaVA-1.5-7B and an increase from 44.65 to 56.02 for Qwen2-VL-2B-Instruct.
Main Results: On InternVL2.5-8B, GapSight reaches 64.29 Six-Bench Avg, exceeding ZoomRefine at 54.43, ViCrop at 55.84, and CropVLM at 57.16.These comparisons use the same six-benchmark aggregate metric.
Efficiency Analysis: GapSight occupies a favorable score–cost frontier, achieving the highest mean score across backbones while using fewer visual tokens than external crop and zoom baselines.Figure 2 averages Six-Bench Avg and visual-token usage across the three reported backbones.
Supervision Ablation: Loss-gap supervision improves the InternVL2.5-8B VQA average by 4.69 points over CLIP relevance and by 7.57 points over random-label supervision.The ablation holds the backbone, router, candidate bank, schedule, and evaluation protocol fixed while changing the label teacher.

Mechanism Analysis

GapSight’s analyses show that crop utility is model-specific and that selective, task-adaptive review repairs errors while controlling visual-token cost. Its continuous boxes preserve surrounding context rather than selecting generic salient regions.

Useful Crops Are Model-Specific: 0.44 average off-diagonal retention shows that crops mined for one backbone preserve only part of their utility on another.Transfer is higher on DocVQA and VStarBench but drops sharply on ChartQA and MME-Lite.
When Review Helps and Hurts: 107 errors were repaired against 27 regressions on four InternVL2.5-8B VQA benchmarks, producing a net correction of +80.TextVQA and DocVQA show the strongest rescue patterns, while ChartQA has fewer rescues and a lower action rate.
When Review Helps and Hurts: 76.5% review on InfographicVQA versus 25.6% on MME-Lite demonstrates task-adaptive gating rather than a fixed crop budget.The router reviews more frequently when local text matters and abstains more often when global context is important.
Crop Geometry and Context: 0.20 median positive-crop area and 0.315 mean predicted area indicate boxes that retain local evidence with surrounding context.The geometry preserves units, axis labels, and field labels needed to interpret text-rich evidence.

Conclusion

GapSight learns visual re-reading from model-specific loss gaps and distills the signal into a one-shot crop policy. Across multiple backbones and six benchmarks, it improves Base no-zoom inference while producing task-adaptive, model-specific behavior.

Conclusion: GapSight converts answer-loss or option-margin improvements from candidate crops into supervision for a one-shot action, utility score, and continuous crop box.The router controls when and where the VLM looks again from the global state.
Conclusion: Across multiple VLM backbones and six benchmarks, GapSight improves Base no-zoom inference against recent crop and zoom baselines.Mechanism analyses report error rescues, task-adaptive action rates, context-preserving boxes, and improved token-performance profiles.