PaperlayerView exact revision on arXiv

Source-linked AI summary

LOCI: A Locator-Critic with Refinement Loop

Walid Bousselham, Mathilde Caron, Arsha Nagrani, Cordelia Schmid

arXiv:2608.30959v1cs.CVcs.AI

TL;DR

Complex visual question answering is limited primarily by failures to locate critical image details rather than by high-level reasoning. LOCI addresses this with a training-free Locator-Critic pipeline that separates visual search from verification and iteratively refines evidence, achieving state-of-the-art results with substantial gains across open-weight and proprietary models. Its main scope boundary is higher token and sequential-call cost, along with vulnerability to visually plausible distractors.

  • Problem

    VLMs struggle with complex visual analysis because they may locate irrelevant or ambiguous regions and build plausible reasoning on flawed visual evidence.

  • Method

    LOCI is a training-free pipeline in which a Locator proposes image crops and a separate Critic evaluates their sufficiency, providing feedback for iterative refinement.

  • Results

    LOCI achieves state-of-the-art performance across complex visual benchmarks, improving Qwen3-VL by +12.1 on V*, +5.8 on HR-Bench, and +11.2 on VisualProbe.

  • Takeaways & Limitations

    Decoupled visual search and verification improves VLM reliability across both open-weight and proprietary models.

  • Takeaways & Limitations

    The loop requires more tokens and sequential VLM calls, and the Critic can be deceived by visually plausible distractors.

Abstract

from arXiv · show

Vision-Language Models (VLMs) still struggle on tasks requiring complex visual understanding. We argue that the core issue is not high-level reasoning, but instead failing to locate critical details in the image. Due to this shortcoming, VLMs generate often plausible but incorrect reasoning based on flawed perceptual grounding. To address this, we propose Locator-Critic (LOCI), a training-free framework that decouples visual search from evidence verification. LOCI employs a Locator agent to propose candidate visual evidence and a separate Critic agent to evaluate its relevance and sufficiency. These agents engage in an iterative refinement loop, progressively improving the evidence until it is adequate to answer the given question. This decoupled, self-correcting process yields substantial performance gains, achieving state-of-the-art results on multiple complex visual benchmarks. LOCI improves accuracy for both open-weight models like Qwen3-VL (+12.1 on V*, +5.8 on HR-Bench and +11.2 on VisualProbe-Hard) and proprietary models like Gemini 2.5 Pro (+8.9 on V*, +4.3 on HR-Bench, +4.8 on VisualProbe-Hard).

1 Introduction

LOCI argues that complex visual question answering is bottlenecked primarily by locating relevant visual cues, not by high-level reasoning. It addresses this by separating evidence localization from verification and iteratively refining the evidence.

  • VLMs can produce sophisticated reasoning, but focusing on irrelevant or ambiguous regions leads to plausible reasoning grounded in flawed visual evidence.
  • A single inference call that proposes and verifies evidence can bias the model toward confirming its own initial hypothesis.
  • When given ground-truth visual regions, models answer almost all V* questions correctly, supporting localization as the primary bottleneck.
  • LOCI uses specialized Locator and Critic agents to decouple finding visual evidence from independently verifying its relevance and sufficiency.
  • The self-correcting loop rejects insufficient evidence and guides the Locator to search again until relevant objects are grounded.
  • LOCI achieves state-of-the-art results across complex visual benchmarks, improving Qwen3-VL by +12.1 on V*, +5.8 on HR-Bench, and +11.2 on VisualProbe.
  • The framework improves both open-weight and proprietary models through decoupled verification and iterative refinement.

2 Related Works

Prior work equips multimodal models with tools, visual search, or critic modules, but LOCI focuses verification on intermediate visual evidence. Its contribution is a training-free, decoupled search-and-verification framework that differs from final-answer judging and single-model self-correction.

  • Tool-augmented MLLMs use operations such as cropping, rotation, contrast enhancement, and image manipulation, often with substantial training overhead.
  • Active-perception methods select zoomed or high-resolution regions, while some rely on Monte Carlo Tree Search or multi-stage training pipelines.
  • Self-correction methods can suffer cognitive fixation because the critic and generator share underlying biases.
  • Actor-critic approaches use a separate model or process to verify reasoning outputs and provide feedback.
  • LOCI evaluates the quality of intermediate visual crops rather than the final textual answer, while decoupling the Critic from the Locator’s reasoning.

3 Method

LOCI combines a crop-generating Locator with a visual-evidence Critic in an iterative loop. The Critic determines whether evidence is sufficient and guides further search, while parallel Locators expand exploration when a single search may become stuck.

  • 3.1 Locator-Critic Framework: A Locator generates crops containing evidence needed for the question, while a Critic decides whether the evidence is sufficient and reliable.
  • 3.1 Locator-Critic Framework: The Locator generates bounding boxes and executable Python code, receives execution outputs, and adaptively refines crop extraction.
  • 3.1 Locator-Critic Framework: At each turn, the Locator uses cumulative history to output up to m evidence crops and answers only at the final turn.
  • 3.1 Locator-Critic Framework: The Locator can modify previous code using execution traces and targeted Critic feedback to improve subsequent searches.
  • 3.1 Locator-Critic Framework: The Critic receives the question, original image, and crops, then returns an ongoing-or-final decision with feedback when more search is needed.
  • 3.1 Locator-Critic Framework: Restricting the Critic to visual crops prevents Locator explanations from deceiving it into approving insufficient evidence.
  • 3.1 Locator-Critic Framework: Off-the-shelf detectors produce non-refinable crops, limiting iterative improvement and yielding results comparable to the first turn.
  • 3.2 Multi-Locator Ensembling: Parallel Locators search different regions simultaneously, while a centralized Critic evaluates their combined crops and broadcasts unified feedback.

4 Experiments

Experiments show that LOCI’s gains arise from accurate, independently verified visual evidence and iterative search, improving performance across benchmarks and model families while incurring extra compute and retaining distractor-related failures.

  • Oracle experiment: Oracle crops raise Gemini2.5-Pro accuracy on V* from 83.0% to 98.2%, identifying visual localization as the primary bottleneck.The remaining 1.8% failures involve ambiguous questions with multiple viable answers.
  • Ablations: The Critic adds +3.5 points beyond Locator-only, producing LOCI’s +8.9-point improvement over direct prompting.Providing Locator reasoning to the Critic severely degrades performance, so verification must rely on image crops alone.
  • Same-budget baselines: Best-of-10 reaches 89.0 despite nearly 10× LOCI’s token budget, while LOCI reaches 92.7; matching the workflow without decoupled verification reaches only 85.9.These comparisons distinguish LOCI’s architecture from simply spending more inference-time compute.
  • Locator choice: The Critic improves the off-the-shelf locator by only +0.7 points but improves the VLM-based Locator by +3.5 points because only the latter can refine missed evidence.A fixed detector limits Critic effectiveness to selecting from its initial crop set.
  • Iterative refinement: From Tmax = 1 to Tmax = 10, crop recall rises from 0.60 to 0.69 and VQA accuracy from 0.89 to over 0.92.The recall plateau reflects misalignment between detection metrics and functionally useful VQA crops.
  • Parallel locators: Parallel locators improve V* accuracy from 92.7% at k = 1 to 96.0% at k = 8, with no further gains beyond k = 8.Parallel diverse exploration mitigates cases where one Locator remains stuck on an incorrect hypothesis.
  • Benchmark results: LOCI achieves state-of-the-art results across V*, HR-Bench, and VisualProbe, including +11.5 points over DyFo on V* and +8.2 over Mini-o3 on HR-Bench.Using the same Gemini2.5-Pro backbone, LOCI beats DyFo by +3.2 points with 2× fewer tokens and zero external tool calls.
  • Limitations: LOCI requires more tokens and sequential VLM calls, and the Critic can be deceived by visually plausible distractors.The authors suggest adaptive stopping, difficulty-aware budgets, and distillation as future mitigations.

5 Conclusion

LOCI is a training-free framework for complex visual question answering that separates visual evidence discovery from verification. The paper reports state-of-the-art performance and consistent improvements across open-weight and proprietary models.

  • 5 Conclusion: LOCI introduces a training-free Locator-Critic framework for complex visual question answering.The Locator proposes visual evidence, while an independent Critic verifies it through a self-correcting loop.
  • 5 Conclusion: LOCI achieves state-of-the-art performance across multiple complex visual benchmarks.The framework improves both open-weight and proprietary models over direct prompting and prior methods.
  • 5 Conclusion: The framework decouples visual detection from verification to produce answers grounded in reliable evidence.The authors identify the interaction between an explorative Locator and a Critic as a broadly applicable principle.
  • 5 Conclusion: Increasing crops per turn improves accuracy until performance saturates at m = 4 while reducing the iterations needed.The paper adopts m = 4 as the default setting to balance performance and token efficiency.

A.1 Ablation on Number of Crops per Turn

Increasing the number of crops generated per turn rapidly improves V* accuracy up to m = 4, after which performance plateaus. More crops also reduce refinement iterations, while incorrect answers require longer searches.

  • A.1 Ablation on Number of Crops per Turn: Accuracy on V* improves rapidly as crops per turn increase from 1 to 4, then reaches a plateau.The experiment varies m from 1 to 7 with Tmax = 10 using Gemini 2.5 Pro.
  • A.1 Ablation on Number of Crops per Turn: m = 4 is selected as the default setting to balance performance and token efficiency.The choice follows the observed accuracy plateau.
  • A.1 Ablation on Number of Crops per Turn: Increasing crops per turn reduces the average number of iterations required to reach a final answer.Generating more crops performs a self-refinement step within a single turn, increasing the chance of satisfying the Critic early.
  • A.1 Ablation on Number of Crops per Turn: Incorrect answers consume about 5.7 turns on average, compared with about 3 turns for correct answers.The authors associate longer searches with difficult or ambiguous queries and insufficiently isolated visual evidence.

A.2 Ablation on Maximum Number of Turns with Single Crop

With a single crop per turn, increasing the maximum number of turns improves accuracy but saturates below the default multi-crop setting. The results distinguish sequential feedback from parallel crop diversity.

  • A.2 Ablation on Maximum Number of Turns with Single Crop: With m = 1, accuracy reaches 89.9% at Tmax = 10 and saturates at 90.1% at Tmax = 15.The experiment varies Tmax from 1 to 15 on V*.
  • A.2 Ablation on Maximum Number of Turns with Single Crop: The single-crop setting remains below the default m = 4 accuracy of 92.7% despite the extended turn budget.This comparison indicates that additional sequential turns do not compensate for reduced parallel exploration.
  • A.2 Ablation on Maximum Number of Turns with Single Crop: Per-turn crop diversity enables immediate self-correction, whereas Critic feedback provides higher-level guidance away from dead-end search paths.The paper assigns distinct roles to parallel crop generation and iterative verification.

A.3 Evaluation of Critic Selection Capability

The Critic is evaluated on selecting relevant crops from distractors, with accuracy and selection metrics reported under constrained experimental conditions. The results assess selection capability when valid evidence is guaranteed to be present.

  • A.3 Evaluation of Critic Selection Capability: The synthetic task mixes ground-truth crops with three randomly generated, non-overlapping distractor crops.Distractors use standard aspect ratios, while the ground-truth regions come from V*.
  • A.3 Evaluation of Critic Selection Capability: The Critic is prompted to select at least one crop so selection performance is evaluated separately from the decision to continue searching.Without this constraint, the Critic rejects all provided crops in 34% of samples.
  • A.3 Evaluation of Critic Selection Capability: 99.48% Precision and 97.38% Recall show near-perfect Critic selection when high-quality evidence is included among distractors.The constrained setting guarantees that valid crops are present for selection.
  • A.3 Evaluation of Critic Selection Capability: Selected crops yield 95.29% downstream VQA accuracy.Downstream accuracy is measured by passing the Critic-selected crops to the final answering agent.

A.4 Ablation on Additional Datasets

Additional-dataset experiments show that LOCI’s Locator and Critic improve performance beyond V*, while parallel exploration and broader model coverage extend its gains.

  • Dataset construction: The additional HR-Bench and MME-RealWorld analyses isolate Hard subsets to evaluate tasks requiring visual search.Samples are retained when Gemini 2.5 Pro fails under direct prompting at 768 × 768 resolution, excluding tasks solvable without explicit visual exploration.
  • Component analysis: +9.2 points: Qwen3-VL gains this improvement on MME-RealWorld (Hard) from adding only the Locator.Adding the Critic further boosts performance and produces the highest accuracy for both Qwen3-VL and Gemini 2.5 Pro.
  • Locator ensembling: Multi-Locator Ensembling improves performance consistently as parallel locators increase from k = 1 to k = 4 across V*, HR-Bench (Hard), and MME-RealWorld (Hard).On MME-RealWorld (Hard), performance rises by +5.9 points, from 39.1% to 45.0%.
  • Generalization across VLMs: +25.35 points: Grok4.1-Fast improves from 45.55% to 70.90% on V* with LOCI.LOCI improves all four evaluated models on V*, with the largest gains for the weakest baseline.
  • Baselines: The ablation compares direct prompting, Self-Refine, Self-Consistency@10, Best-of-10, Explicit Locate-Verify-Answer, and DyFo against LOCI.These baselines include single-agent refinement, sampling-based selection, same-agent verification, and training-free crop search.

B.1 Oracle Experiment Failure Cases

In the oracle setting, most remaining failures arise from annotation ambiguity rather than visual misunderstanding, with a smaller set involving genuine spatial reasoning errors.

  • Failure sources: Most oracle-experiment failures stem from ambiguous dataset annotations rather than model reasoning failures.Examples include color labels for a pale-pink versus white handbag and a broom whose black bristles determine the annotated answer despite its gray handle.
  • Failure sources: A genuine exception occurs when the model correctly identifies spatial relations but draws the wrong conclusion about the motorcycle’s side of the street.The model recognizes that the motorcycle is behind the person walking a dog and that both are on the right side, then incorrectly assigns the motorcycle’s position.

B.2 Qualitative Analysis: Consequence of Exposing Critic to Locator Reasoning

Exposing the Critic to the Locator’s textual reasoning can induce confirmation bias, whereas visual-only verification rejects invalid crops and forces continued search.

  • Text-exposed Critic: The Locator searches irrelevant regions in Crops #1–4, and the Critic later accepts an incorrect pavement-only Crop #9 after reading the Locator’s confident claim.The resulting confirmation bias causes the Critic to override its own visual perception.
  • Visual-only Critic: Standard LOCI restricts the Critic to visual input, so it rejects empty crops and forces the Locator to continue until it isolates the broom and chair.This contrasts directly with the text-exposed configuration, where the Critic accepts invalid evidence.

B.3 Qualitative Analysis: Resilience against Hallucination

The qualitative comparison shows that textual confidence can make the Critic accept hallucinated visual evidence, while visual-only feedback supports correction and successful localization.

  • Critic with Access to Text: When exposed to the Locator’s reasoning, the Critic becomes susceptible to persuasive but hallucinated descriptions and prematurely validates an incorrect crop.The Locator misidentifies a green background object as the target, and the system terminates with the prediction “green.”
  • Standard LOCI: In standard LOCI, the Critic receives only crops and rejects attempts based on irrelevant green doors or repeated crops of a nearby person.Its feedback escalates toward correcting the Locator’s spatial focus.
  • Standard LOCI: Visual-only feedback leads the Locator to isolate the correct object at Turn 9, identify it as primarily black or dark gray, and answer correctly.The Critic provides spatial guidance toward the far right of the image and a path into the trees.

B.4 Metric-Task Misalignment

LOCI’s crop recall can understate visual usefulness because VQA questions often require broader context than tight ground-truth boxes provide. The framework therefore favors crops that preserve sufficient task-relevant information, even when IoU-based metrics penalize them.

  • 0.69 crop recall plateaued while VQA accuracy exceeded 92%, revealing misalignment between detection metrics and task-required visual evidence.The discrepancy reflects tight IoU-based ground-truth matching versus the broader evidence needed for VQA.
  • LOCI can achieve recall 0 at IoU > 0.05 while still producing crops that are valid for answering questions.This occurs because crop utility is not equivalent to overlap with annotated boxes.
  • For spatial questions, LOCI’s wider crop includes both related objects, preserving the context needed to infer their relationship.Ground-truth annotations may isolate only one object, causing a useful relational crop to receive low IoU and recall.
  • For object-detail questions, LOCI may use a looser crop that contains all necessary visual details rather than tightly matching the annotated object box.The handbag example shows a wider vertical strip still preserves enough information to determine color.
Loading 2608.30959v1…