PaperlayerView exact revision on arXiv

Source-linked AI summary

PixelEyes: Decoupling Perception and Reasoning for Pinpoint Visual Evidence Seeking

Dengxian Gong, Yuanzheng Wu, Haobo Yuan, Zhengdong Hu, Tao Zhang, Yikang Zhou, Shihao Chen, Quanzhu Niu, Kai Wang, Jason Li, Haochen Wang, Lu Qi, Shunping Ji, Ming-Hsuan Yang

arXiv:2607.00115v1cs.CV

TL;DR

MLLMs often entangle visual localization with reasoning, making visual evidence seeking difficult to evaluate and prone to inefficient search. PixelEyes decouples these roles with mask-guided perception and semantic-region BFS, and outperforms prior active-perception agents across six benchmarks while using fewer turns.

  • Problem

    Existing methods couple fine-grained perception and general reasoning, while benchmarks often cannot cleanly distinguish localization failures from reasoning failures.

  • Method

    PixelEyes decouples perception from reasoning through mask-guided search, Semantic-Region BFS, switchable tool use, and PixelEyes-6K trajectory training.

  • Results

    PixelEyes outperforms prior active-perception agents across six benchmarks while using fewer turns.

  • Takeaways & Limitations

    The results suggest that explicitly optimizing the structure and logic of visual evidence seeking is a promising path toward more reliable active visual reasoning agents.

  • Takeaways & Limitations

    The decoupled implementation introduces greater architectural complexity than fully end-to-end systems, leaving computational efficiency open for optimization.

Abstract

from arXiv · show

This paper explores multi-turn visual reasoning and observes that MLLMs repeatedly fail to localize the target, leading to long, redundant trajectories. We attribute this failure to the entanglement of reasoning and perception within a single model, the MLLM reasons and localizes simultaneously, and inaccurate localization triggers additional reasoning turns that bloat the trajectory. To solve this problem, we propose PixelEyes, a multi-turn visual reasoning agent that explicitly decouples reasoning from perception, i.e., the reasoner decides what to look for, while a specialized perception tool answers where it is. Specifically, PixelEyes introduces 1) Mask-guided Visual Search. A referring segmentation model is invoked to provide mask-precise localization, freeing the reasoner from the need to compensate for imprecise grounding. 2) Semantic-region Breadth-first Search (BFS). To eliminate redundant loops caused by repeatedly cropping incorrect sub-regions, we organize exploration as a breadth-first search over semantic regions. To internalize these capabilities, we construct the PixelEyes-6K dataset by resynthesizing expert trajectories from existing data. This explicitly embeds our mask-guided search and BFS logic into the model. We further introduce Pinpoint-Bench, a zero-hint visual search benchmark, i.e., no location cues are provided in the question, with instance-level masks and bounding boxes that separate localization failures from reasoning failures, enabling fine-grained analysis of failure modes such as inattentional blindness. Recent state-of-the-art MLLMs and visual reasoning agents leave large headroom on Pinpoint-Bench, demonstrating its quality and difficulty. Code and models are open-sourced.

1 Introduction

PixelEyes addresses visual-search failures caused by entangling fine-grained perception with reasoning, especially when decisive targets occupy tiny image regions. It decouples these roles with mask-guided localization and structured search, and introduces PixelEyes-6K and Pinpoint-Bench to train and diagnose the resulting behavior.

  • Motivation: Decisive evidence can occupy less than 1% of a high-resolution image, forcing agents to localize targets and reason about them within limited turns.Active visual reasoners crop, zoom, and re-examine images to gather evidence.
  • Problem: Single-model approaches jointly perform region-level perception and reasoning, causing weak grounding to produce long trajectories and redundant exploration.The paper states that these models are weaker at grounding than perception specialists and weaker at reasoning than strong general-purpose VLMs.
  • PixelEyes: PixelEyes decouples roles: a general-purpose VLM decides what to seek, while SAMTok returns a pixel-level mask specifying where it is.Semantic-Region BFS anchors coordinates in the original image and proposes a new low-IoU region when grounding fails.
  • Training: PixelEyes-6K contains 5.8K correct expert trajectories generated with a mask-based crop tool, then used for Qwen-3-VL supervised fine-tuning and vanilla GRPO reinforcement learning.The trajectories augment Gemini-3-Flash on existing image-question pairs and retain only interactions reaching a correct answer.
  • Evaluation: Pinpoint-Bench provides 433 human-annotated ultra-high-resolution samples with target masks averaging 0.07% of image area and both mask and bounding-box annotations.Its zero-hint design enables localization failures to be separated from reasoning failures.
  • Contributions: The contributions comprise PixelEyes, PixelEyes-6K, and Pinpoint-Bench, with diagnostic LSR/TAE metrics and evaluations across six visual benchmarks.The paper reports that PixelEyes outperforms prior active-perception agents at both 4B and 8B.

2 Related Work

Related work spans general-purpose vision-language models with coarse grounding and dedicated perception modules, alongside agents that iteratively crop, zoom, or re-observe images. These agents range from heuristic and tree-based search to reinforcement-learned interaction, aggressive multi-turn scaling, and single-pass zoom distillation.

  • Vision-Language Models and Specialized Perception: General-purpose VLMs align visual and textual representations and achieve strong multimodal reasoning, but their coarse bounding-box grounding degrades on small or cluttered targets.Examples include Flamingo, LLaVA, GPT-4o, Gemini, InternVL, and Qwen-VL.
  • Vision-Language Models and Specialized Perception: A parallel line of work equips MLLMs with dedicated perception modules.
  • Multi-turn Visual Reasoning Agents: Active-perception agents iteratively crop, zoom, or re-observe, using heuristic or tree-based search in early work and reinforcement learning in more recent methods.
  • Multi-turn Visual Reasoning Agents: Mini-o3 aggressively scales interaction turns with dozens of bounding-box crops per trajectory, whereas ZwZ distills zooming into a single forward pass.

3 Method: PixelEyes

PixelEyes formulates visual evidence seeking as a multi-turn decision process that decouples reasoning from fine-grained perception through switchable cropping tools and semantic-region BFS. It also introduces PixelEyes-6K for training and Pinpoint-Bench with dense annotations and diagnostics for separating localization from reasoning failures.

  • Agentic visual search: PixelEyes models visual evidence seeking as an iterative loop in which a VLM policy generates reasoning and planning thoughts before selecting actions from accumulated text-and-image context.Each crop returns a local image observation appended to the trajectory for the next turn.
  • Switchable tool use: The agent switches among mask-based crop, BBox-based crop, and Answer operations according to the query and whether instance-level segmentation applies.Mask-based cropping uses a coarse bounding box and referring expression for precise grounding, while BBox-based cropping handles regions such as charts or maps.
  • Mask-guided visual search: Mask-guided search augments the base VLM with SAMTok to localize referring expressions precisely, produce target-centric crops, and filter background noise.The system enlarges the mask-derived tight box by a factor of (1 + α) to preserve context and falls back to direct cropping when grounding fails.
  • Semantic-region BFS: Semantic-region BFS avoids DFS-style backtracking by shifting to unexamined regions when the perception tool fails to ground a target, while region captions provide semantic footprints for global planning.The strategy relies on the prior that a failed grounding attempt within a proposed region makes the target highly likely to be absent there.
  • Training data: PixelEyes-6K is created by resynthesizing trajectories with mask-guided search, switchable tools, and semantic-region BFS logic.The dataset internalizes the proposed search capabilities without requiring additional data.
  • Pinpoint-Bench: Pinpoint-Bench contains 433 human-annotated high-resolution samples in cluttered scenes, with bounding boxes and instance masks enabling trajectory-level diagnosis through TAE and LSR.LSR counts visual discovery when any crop covers a target mask, and the LSR–Accuracy gap diagnoses inattentional blindness by distinguishing perception from reasoning bottlenecks.

4 Experiments

PixelEyes is evaluated on active-perception, complex-search, and general reasoning benchmarks, with Pinpoint-Bench metrics separating localization from answering. It outperforms baselines and ablations show that PixelEyes-6K, RL, mask-guided search, BFS, switchable tools, and robust grounding each contribute to performance.

  • Benchmarks and metrics: Experiments cover active perception, complex visual search, and general or structural reasoning, using standard accuracy plus LSR and TAE on Pinpoint-Bench.The benchmark groups include V*, HR-Bench, VisualProbe, Pinpoint-Bench, MME-RealWorld-Lite, and Tree-Bench.
  • Comparison with baselines: At 8B, PixelEyes reaches 94.24% on V*, 85.00% on HR-Bench-4K, and 83.15% on HR-Bench-8K, exceeding Qwen-3-VL-235B on all three.At 4B, it improves over Qwen-3-VL-4B by +11.5 on V*, +7.0 on HR-Bench-8K, and +19.8/+14.9/+12.0 on VisualProbe Hard/Medium/Easy.
  • Pinpoint-Bench analysis: On zero-hint Pinpoint-Bench, Mini-o3 achieves LSR = 78.52% but only 44.34% accuracy and TAE = 8.38, exposing a 34-point LSR −Acc gap linked to inattentional blindness.PixelEyes-4B trades −1.61 LSR for +10.39 Acc. and +17.75 TAE, while PixelEyes-8B further improves Acc. and TAE.
  • Ablations: training: Fine-tuning on PixelEyes-6K reaches 50.94 on VisualProbe-Hard, whereas Mini-o3 data reduces it from 34.91 to 24.52 (−10.4) through redundant DFS loops.The comparison supports trajectory structure, rather than data volume alone, as the relevant training factor.
  • Ablations: training: Vanilla GRPO raises VisualProbe-Hard to 54.72 and Pinpoint-Bench Acc. to 54.73 when added to the PixelEyes supervised-fine-tuned model.The ablation uses Qwen-3-VL-4B as the default base and reports results after RL unless stated otherwise.
  • Ablations: search and grounding: Mask-guided search consistently outperforms pure bbox search, while semantic-region BFS enables exclusion of searched regions; bbox fallback adds +1.75 on HR-Bench-4K and +0.38 on HR-Bench-8K.SAMTok achieves 99.17% ISR (476/480 calls), whereas replacing it with Sa2VA degrades Acc., TAE, and LSR, especially for small high-resolution targets.

5 Conclusion

PixelEyes presents a perception-reasoning decoupled visual reasoning agent that delegates fine-grained localization to SAMTok while retaining high-level reasoning in a general-purpose VLM. It also contributes PixelEyes-6K and Pinpoint-Bench for training and diagnostic evaluation of precise visual evidence acquisition.

  • 5 Conclusion: PixelEyes decouples perception and reasoning by delegating fine-grained localization to SAMTok while preserving high-level reasoning within a strong general-purpose VLM.The paradigm is intended to alleviate limitations of coupled perception and reasoning.
  • 5 Conclusion: Mask-guided visual search, Semantic-Region BFS, and Switchable Tool Use enable precise, efficient, and broadly applicable evidence acquisition across diverse visual scenarios.These mechanisms support the agent’s search process under the proposed decoupling paradigm.
  • 5 Conclusion: PixelEyes-6K is produced by a high-fidelity trajectory synthesis engine that augments Gemini-3-Flash with the mask_based_crop perception tool.The dataset trains models under the perception-reasoning decoupling paradigm.
  • 5 Conclusion: Pinpoint-Bench is a challenging ultra-high-resolution benchmark using a strict zero-hint protocol and dense mask and bounding-box annotations for diagnostic evaluation.It evaluates Location Success Rate (LSR) and Turn-To-Answer Efficiency (TAE).

Appendix · A More Experiment Results

The appendix reports training and evaluation settings, then shows that PixelEyes’ mask-guided search transfers to Gemini as a training-free, plug-and-play augmentation with gains across all benchmarks.

  • A More Experiment Results: SFT fine-tunes the base model for 1 epoch with AdamW, a 2e-5 learning rate, 0.05 weight decay, and batch size 32.The implementation also uses (β1, β2) = (0.9, 0.999), a 0.03 warmup ratio, and gradient clipping at maximum norm 1.0.
  • A More Experiment Results: RL uses GRPO with a 1e-6 learning rate, group size 16, and seq-mean-token-mean loss.Training stabilization applies clipping thresholds of 0.2 and 0.3.
  • A More Experiment Results: Testing runs on 4 NVIDIA H100 GPUs with a maximum of 6 interaction rounds per query.These settings define the reported evaluation environment.
  • A More Experiment Results: All baseline methods are evaluated using their officially released implementations without modifications.This keeps the comparison tied to each baseline’s official implementation.
  • A More Experiment Results: Table 7 reports training-free results for integrating PixelEyes’ tools into Gemini’s multi-turn reasoning process.MME-R-L denotes the MME-RealWorld-Lite benchmark.
  • A More Experiment Results: The Gemini-3-Flash Tool variant integrates mask-guided search directly into the Gemini API as a plug-and-play component.The integration requires no additional fine-tuning or parameter updates and yields substantial gains across all benchmarks.

B Deeper analysis on Pinpoint-Bench

Pinpoint-Bench analysis shows that PixelEyes gains from precise mask-guided localization, especially on fine-grained attribute and OCR tasks, while reducing interaction turns. Failure decomposition also exposes inattentional blindness in models that produce overly coarse bounding boxes, and spatial-relation tasks remain challenging.

  • Failure decomposition: Mini-o3 reaches 78.52% LSR but produces overly coarse bounding boxes, yielding a 38.34% inattentional blindness rate.The failure decomposition separates Pinpoint-Bench outcomes into three mutually exclusive categories.
  • Task-specific analysis: PixelEyes-4B-RL achieves 56.52% in Attribute Recognition and 54.62% in OCR, outperforming Mini-o3 by +12.25% and +11.54%, respectively.It also outperforms Qwen-3-VL-4B by +12.65% and +9.24% on these two task types.
  • Task-specific analysis: Qwen-3-VL-4B leads Spatial Relation tasks at 60.00%, indicating that complex relative-position reasoning still depends heavily on global layout understanding.PixelEyes’s strongest gains occur in tasks requiring fine-grained local visual features and precise text-region localization.
  • Turn-level efficiency: PixelEyes-4B-RL requires only 2.09 average turns per sample, compared with Mini-o3’s 5.29 average turns.These turn-level statistics are reported as primitive components of the composite TAE metric on Pinpoint-Bench.

C More Details of Pinpoint-Bench

Pinpoint-Bench is constructed through raw data collection, collaborative annotation, and rigorous quality inspection, and covers diverse visual-search tasks. Its ultrahigh-resolution images contain exceptionally small targets, creating a needle-in-a-haystack localization challenge.

  • Construction: Pinpoint-Bench construction proceeds through raw data collection, collaborative annotation, and rigorous quality inspection.The collection stage crawled 1884 raw images, while annotators worked concurrently on a dedicated online platform.
  • Task composition: The benchmark includes single-target attribute recognition, spatial relationship reasoning among two or three objects, and OCR-centric queries.Attribute recognition is the primary focus among these task types.
  • Image statistics: Images typically exceed 10 megapixels and have an average resolution of 5500 × 3516 pixels.The benchmark features ultrahigh-definition samples.
  • Localization difficulty: All targets occupy less than 1% of the total image area, emphasizing the benchmark’s needle-in-a-haystack localization difficulty.The average target footprint is described as minuscule, making targets exceptionally challenging to locate.

D Limitations

PixelEyes’ decoupled design improves performance but introduces architectural complexity compared with fully end-to-end systems, leaving computational efficiency open for further optimization.

  • D Limitations: PixelEyes introduces architectural complexity compared with fully end-to-end systems, leaving room to optimize computational efficiency.This complexity is a limitation of the current implementation despite its superior performance.

E Visualizations

Visualizations show that PixelEyes can precisely localize difficult microscopic targets while conducting multi-turn searches, but remains limited by the base model’s macro-search deficiencies and hallucinations. Additional cases illustrate Pinpoint-Bench’s difficulty through distractor-driven and counting failures.

  • Successful cases: PixelEyes precisely localized microscopic targets in two successful Pinpoint-Bench cases while supporting multi-turn tool-use search.It quickly grounded an umbrella to answer its color and progressively localized a masked woman using coarse boxes and mask grounding to determine crutch use.
  • Failure cases: PixelEyes localized a clock target but answered incorrectly, reflecting the base model’s inherent reasoning limitations.The visualization attributes the failure to the base model rather than localization.
  • Challenging cases: Pinpoint-Bench’s high-difficulty cases show multi-turn cropping causing Mini-o3 to mistake a parking-garage car for the rooftop car and to answer 1 for the train count.The examples are presented as representative challenging cases.
  • Failure cases: PixelEyes repeatedly grounded the wrong person in an earphone-color question, then confidently answered incorrectly while hallucinating earphones on the grounded man.Lighting variation, multiple earphone-wearing distractors, and an ignored clothing constraint contributed to the failure.
Loading 2607.00115v1…