PaperlayerView exact revision on arXiv

Source-linked AI summary

From Web to Pixels: Bringing Agentic Search into Visual Perception

Bokang Yang, Xinyi Sun, Kaituo Feng, Xingping Dong, Dongming Wu, Xiangyu Yue

arXiv:2605.12497v1cs.CV

TL;DR

Open-world visual perception requires resolving a visible object from external evidence before grounding it to pixels. The paper introduces WebEyes and Pixel-Searcher for this setting, with Pixel-Searcher achieving the strongest open-source performance across grounding, segmentation, and VQA.

  • Problem

    Perception Deep Research addresses open-world queries where external evidence is needed to resolve a hidden target identity and bind it to a visual instance.

  • Method

    The paper introduces WebEyes and Pixel-Searcher, an object-anchored benchmark and agentic search-to-pixel workflow for resolving identities and producing boxes, masks, or grounded answers.

  • Results

    Pixel-Searcher achieves the strongest open-source performance across Search-based Grounding, Search-based Segmentation, and Search-based VQA.

  • Takeaways & Limitations

    Diagnostic results identify evidence acquisition, identity resolution, and visual instance binding as the main bottlenecks in this setting.

Abstract

from arXiv · show

Visual perception connects high-level semantic understanding to pixel-level perception, but most existing settings assume that the decisive evidence for identifying a target is already in the image or frozen model knowledge. We study a more practical yet harder open-world case where a visible object must first be resolved from external facts, recent events, long-tail entities, or multi-hop relations before it can be localized. We formalize this challenge as Perception Deep Research and introduce WebEye, an object-anchored benchmark with verifiable evidence, knowledge-intensive queries, precise box/mask annotations, and three task views: Search-based Grounding, Search-based Segmentation, and Search-based VQA. WebEyes contains 120 images, 473 annotated object instances, 645 unique QA pairs, and 1,927 task samples. We further propose Pixel-Searcher, an agentic search-to-pixel workflow that resolves hidden target identities and binds them to boxes, masks, or grounded answers. Experiments show that Pixel-Searcher achieves the strongest open-source performance across all three task views, while failures mainly arise from evidence acquisition, identity resolution, and visual instance binding.

1 Introduction

Perception Deep Research extends visual perception to open-world queries requiring external evidence, hidden-identity resolution, and binding targets to concrete visual outputs. The paper introduces WebEyes and Pixel-Searcher to benchmark and perform this search-to-pixel process across grounding, segmentation, and VQA.

  • Problem formulation: Perception Deep Research requires models to resolve a target identity from indirect external clues and bind it to a concrete visual instance.Queries may reference roles, creators, brands, release histories, recent events, or relations to other entities.
  • Benchmark: WebEyes is an object-anchored benchmark with verifiable evidence, knowledge-intensive queries, precise box/mask annotations, and three complementary task views.Its views are Search-based Grounding, Search-based Segmentation, and Search-based VQA.
  • Method: Pixel-Searcher decomposes queries, gathers external evidence, resolves identities, matches visual candidates, and produces the required box, mask, or answer.It is designed as an agentic search-to-pixel workflow for Perception Deep Research.
  • Motivation: Direct perception models struggle when decisive clues are absent from images and frozen knowledge is insufficient.The benchmark therefore tests active external evidence gathering rather than only visual recognition or internal-knowledge reasoning.
  • Findings: Diagnostic experiments identify evidence acquisition, identity resolution, and visual instance binding as key bottlenecks.These bottlenecks span the transition from web knowledge to grounded visual outputs.

2 Related Work

Related work spans language-guided visual perception, which grounds natural-language expressions or text spans to visual regions, and agentic multimodal search, which retrieves and synthesizes external evidence for knowledge-intensive questions. WebEyes combines these directions through search-based segmentation, grounding, and VQA task views.

  • Visual perception with language: Language-guided visual perception includes referring expression comprehension, phrase grounding, and segmentation, with contextual object relations serving as key cues.RefCOCO-style settings localize objects from natural-language expressions, while MDETR aligns sentence-level text spans with image regions using an end-to-end detector.
  • Agentic multimodal search: Agentic multimodal search addresses questions requiring knowledge beyond the image by decomposing queries, issuing searches, and synthesizing retrieved evidence.OK-VQA exposes the need for external factual knowledge; MMSearch evaluates search-engine behavior, and WebWatcher extends this toward browsing-centric visual deep research.
  • WebEyes task views: WebEyes task views produce a mask for search-based segmentation, a grounded region for search-based grounding, and a selected description for search-based VQA.The figure illustrates the three output formats for highlighted targets.

3 WebEyes Benchmark

WebEyes benchmarks Perception Deep Research by requiring models to resolve hidden real-world entities through external evidence and bind them to precise visual outputs. It provides three object-level task views and a traceable, quality-controlled construction process spanning annotation, evidence retrieval, knowledge-based QA, and grounded evaluation.

  • Benchmark formulation: WebEyes requires identifying the real-world entity referred to by a knowledge-intensive query, locating its matching image instance, and returning a box, mask, or answer choice.The benchmark preserves the chain from annotated objects to web evidence, queries, and task-specific grounded outputs.
  • Data format: The three task views are SearchGround for bounding-box prediction, SearchSeg for mask prediction, and SearchVQA for selecting the correct knowledge-rich description.They respectively test entity localization, pixel-level shape recovery, and matching a grounded region to an external-knowledge description.
  • Scale and categories: 1,927 task samples derive from 120 images, 473 annotated object instances, and 645 unique QA pairs.The dataset includes 645 SearchGround samples, 645 SearchSeg samples, and 637 valid multiple-choice SearchVQA samples.
  • Construction pipeline: WebEyes uses an object-first workflow that expands each annotated object into evidence paths, questions, and task instances.Objects receive refined masks, boxes, names, categories, and visual feature summaries that support evaluation, retrieval, and question generation.
  • Construction pipeline: Each object undergoes three-round chained evidence retrieval, resolving a searchable entity before retrieving public, non-visual evidence from the six-month pre-annotation window.Knowledge-based questions hide the target entity name and direct visual attributes; multi-hop questions require at least two facts before resolving the visible target.
  • Quality control: 38.2% of automatically generated candidates are rejected by shortcut filtering, followed by another 49.2% rejected during manual review.Reviewers verify evidence, target uniqueness, leakage, mask/box quality, and consistency across SearchGround, SearchSeg, and SearchVQA.

4 Pixel-Searcher: An Agentic Search-to-Pixel Workflow

Pixel-Searcher reframes knowledge-intensive visual queries as a two-phase agentic search-to-pixel workflow: it resolves hidden target identities from external evidence, then binds them to visible instances and produces task-specific visual outputs.

  • 4 Pixel-Searcher: An Agentic Search-to-Pixel Workflow: Pixel-Searcher has two phases: Agentic Search & Target Resolution, followed by Agentic Grounding & Tool Use.The first phase finds missing identity evidence and summarizes the hidden target; the second binds it to a visible instance and invokes visual tools.
  • Agentic Search & Target Resolution: The first phase converts each query into a structured hypothesis containing the resolved entity, visual category, and image-checkable cues distilled from external evidence.These cues retain object type, appearance, identity clues, and reference evidence while removing irrelevant reasoning paths.
  • Agentic Grounding & Tool Use: For forward tasks, the resolved hypothesis binds the target to a visible region, with segmentation adding a promptable tool to produce the final mask.Search-based Grounding returns the verified region directly, while Search-based Segmentation obtains the mask.
  • Agentic Grounding & Tool Use: For Search-based VQA, Pixel-Searcher reverses the process by resolving answer options into evidence-aware cues and selecting the option best supported by grounded visual evidence.This mode begins with a highlighted image region rather than a target query requiring forward localization.
  • Agentic Search & Target Resolution: Pixel-Searcher adaptively alternates Search, Reason, and Resolve actions to connect retrieved facts and determine whether evidence sufficiently identifies the target.The agent first plans the query and decomposes it into searchable sub-goals when needed.

K Key Cues Input: Multi-Hop Question & Image

Pixel-Searcher uses an adaptive, round-bounded evidence-resolution loop to identify a visible target and then binds the resolved hypothesis to grounded outputs. Its inspectable workflow supports box grounding, segmentation, and reverse evidence-based VQA, with failures traceable to distinct stages.

  • Evidence resolution: The resolution loop is adaptive within a maximum number of rounds, using one lookup for simple queries and multiple connected evidence pieces for harder ones.Evidence collected within at most T rounds is denoted E1:T.
  • Evidence resolution: The resolution agent produces a visually grounded hypothesis containing the final visible entity, coarse category, and image-checkable cues, while rejecting intermediate, generic, unsupported, or inconsistent candidates.The hypothesis is designed for visual grounding rather than free-form textual answering.
  • Visual grounding: Pixel-Searcher grounds the resolved hypothesis by invoking region tools and verifying candidate regions against both the image and external evidence.This makes grounding a tool-assisted decision process rather than one-shot text-to-box prediction.
  • Task outputs: For segmentation, the verified region is passed to SAM3 for boundary refinement, while Pixel-Searcher resolves and locates the correct instance.For Search-based Grounding, the verified region is returned directly as the final answer.
  • Task outputs: For Search-based VQA, the system resolves each answer option into an entity-level summary and selects the option whose identity and visual cues best match the highlighted region.The workflow applies evidence integration in reverse and yields an inspectable process whose failures can be attributed to search planning, evidence integration, instance binding, or mask refinement.

5 Experiments

Experiments show that WebEyes remains challenging, while Pixel-Searcher improves open-source models across grounding, segmentation, and VQA by resolving hidden entities before visual prediction. Ablations and failure analysis indicate that evidence acquisition, entity resolution, and instance binding—not box-to-mask transfer alone—are the main bottlenecks.

  • Experimental setup: All methods use identical WebEyes inputs, splits, and task interfaces without task-specific finetuning, while Pixel-Searcher adds hidden-entity search before grounding and mask refinement.Direct baselines predict boxes from image and query; segmentation uses SAM3, and Search-based VQA uses the image, target box, and answer options.
  • Search-based Grounding: 34.17 IoU and 41.30 R@0.5: Pixel-Searcher improves Qwen3-VL-8B from 26.81 and 32.61, respectively, and leads open-source methods in Search-based Grounding.Gains are clearest in ambiguity-heavy Anime and ICON categories, though precise box prediction remains difficult.
  • Search-based Segmentation: 39.17 gIoU and 32.41 cIoU: Pixel-Searcher improves Qwen3-VL-8B from 35.78 and 25.94, respectively, and ranks first among open-source methods in Search-based Segmentation.The strongest category-level gains occur in Vehicles, Anime, and Product, consistent with improved grounding transferring to SAM3 refinement.
  • Search-based VQA: 42.24 overall accuracy: Pixel-Searcher improves Qwen3-VL-8B from 36.34 and performs best among open-source methods in Search-based VQA, with gains in Icons and Product.The smaller margin to closed-source models indicates that fine-grained semantic comparison also matters.
  • Ablation and failure analysis: 304 of 389 segmentation failures are search/entity errors, versus 75 entity-correct region errors and 10 box-to-mask transfer errors, while removing direct candidates sharply reduces all reported metrics.Without direct candidates, IoU drops from 34.17 to 20.14 and R@0.5 from 41.30 to 19.72; gIoU/cIoU drop from 39.17/32.41 to 20.14/15.71, whereas direct-only reaches 22.28 IoU and 26.49 gIoU.

6 Conclusion

The paper introduces Perception Deep Research, requiring web-derived evidence to become box- and pixel-level predictions, and instantiates it with WebEyes and Pixel-Searcher. Agentic search consistently improves Search-based Grounding, Search-based Segmentation, and Search-based VQA when visual appearance alone is insufficient.

  • Perception Deep Research converts web-derived evidence into box-level and pixel-level predictions.
  • WebEyes provides a benchmark for the proposed setting, while Pixel-Searcher instantiates the agentic search workflow.
  • Agentic search consistently improves Search-based Grounding, Search-based Segmentation, and Search-based VQA when visual appearance alone is insufficient.
  • The dominant failure-analysis bottleneck occurs before mask refinement after a correct box.

A Dataset Samples

Table 6 illustrates WebEyes samples by pairing five source images with every annotated object, target mask, object name, and knowledge-intensive query. The queries require identifying film or character entities through distinctive plot events, references, voice-actor links, ownership facts, or role credits.

  • Representative samples: Table 6 presents five WebEyes source images with all annotated objects, overlaid target masks, object names, and corresponding knowledge-intensive queries.Each selected image lists its complete set of annotated objects alongside the visual and textual annotations.
  • Knowledge-intensive queries: The queries identify characters through distinctive film events, including knocking out a snake with a frying pan, surviving via an antivenom pen, or revealing a hidden map.These examples connect visible targets to specific narrative actions or objects in films.
  • Knowledge-intensive queries: Other queries use character descriptions, voice-actor relationships, ownership facts, and musical performance credits to resolve the target in an image.Examples include the “Zootopia know-it-all,” a voice actor linked to Pokemon Horizons, a dog named Chiffon, and Miisha Shimizu’s role in Skip and Loafer.

B Prompt Templates

Pixel-Searcher’s prompt templates enforce structured, multi-round entity resolution before visual grounding. They use external appearance evidence and candidate-image comparisons to identify the final visible target and return precise grounding outputs in strict JSON.

  • B Prompt Templates: The templates fill placeholders with sample-specific questions, evidence, entity hypotheses, visual candidates, or answer options at inference time.This appendix presents the main prompt templates used by Pixel-Searcher.
  • B.1 Hidden-Entity Search: Hidden-entity search decomposes multi-hop questions into 1–3 dependency-ordered sub-questions while preserving the final visible target.Intermediate years, events, or historical clues must not replace the item or person to be grounded.
  • B.1 Hidden-Entity Search: The search agent alternates SEARCH, THINK, and ANSWER actions, preferring new targeted searches when evidence is missing or ambiguous.At most one THINK may occur before another SEARCH or ANSWER, and only SEARCH or ANSWER consumes an interaction round.
  • B.1 Hidden-Entity Search: Entity-resolution outputs include the entity name, visual category, entity type, key cues, confidence, and unresolved ambiguities.The templates require strict JSON and permit an empty ambiguity list when the evidence already resolves the entity.
  • B.1 Hidden-Entity Search: A consistency check rejects entities that are generic, unsupported, incorrect, or intermediate clues, and requests 1–2 targeted follow-up queries when needed.A consistency score of 5 denotes an entity that is well supported and specific.
  • B.2 Visual Grounding: Visual grounding extracts appearance descriptions covering shape, color, size, logos, and distinguishing physical features from search results.The resulting JSON contains a concise visual description, shape, color, and distinctive features.
  • B.2 Visual Grounding: Bounding-box prompts ground the target in the FIRST image using absolute pixel coordinates, reference images, tight single-instance boxes, or null when no plausible match exists.When similar instances appear, the prompt selects the one best matching the supplied cues and avoids broad boxes spanning multiple objects.
  • B.2 Visual Grounding: Candidate-selection prompts compare labeled boxes and ordered crops in the FIRST image with web reference images, returning the best candidate, runner-up, confidence, and reason.The input explicitly supplies candidate ordering and candidate lines for the comparison.
Loading 2605.12497v1…