From Web to Pixels: Bringing Agentic Search into Visual Perception

Introduction

• Open-world perception extends beyond visible-category recognition and internal-model knowledge because decisive evidence may lie outside the image, motivating active multi-hop web search for grounded perception.The paper frames this challenge in light of Deep Research progress on knowledge-intensive tasks [15] [16] [17].Source: §1 · p. 1
• Perception Deep Research requires resolving indirect factual clues into a target identity and verifying that identity against a concrete image region, rather than relying on knowledge or appearance alone.Queries may involve roles, creators, affiliations, release history, recent events, or relations to other entities; images can contain plausible distractors or similar-looking instances.Source: §1 · pp. 1–2
• WebEyes is an object-anchored benchmark with verifiable evidence, knowledge-intensive queries, precise spatial annotations, and three task views: Search-based Grounding, Search-based Segmentation, and Search-based VQA.The task views evaluate box prediction, mask prediction, and region-level answer selection, respectively.Source: §1 · p. 2
• Pixel-Searcher decomposes queries, gathers external evidence, resolves identities, matches visual candidates, and produces boxes, masks, or answers; diagnostics identify evidence acquisition, identity resolution, and instance binding as bottlenecks.Direct perception models struggle when decisive clues are absent from the image and frozen knowledge is insufficient; Pixel-Searcher improves performance through external evidence and step-wise reasoning.Source: §1 · p. 2

Visual perception with language

• RefCOCO-style methods use object relations for localization [20] [22], MDETR aligns text spans with regions, and LISA produces masks from reasoning prompts [9], but generally assume the target is already identifiable.Other methods address open-set detection, mask prediction, video grounding, and region-level multimodal grounding [8] [10] [23] [24] [26] [27] [28].Source: §2 · p. 3

Agentic multimodal search

• OK-VQA studies factual knowledge beyond images [12], while MMSearch and WebWatcher examine search-based answer synthesis and browsing [13] [19]; this work evaluates grounding web evidence in object-level outputs.Shared annotations span grounding, segmentation, and VQA, and the method resolves the hidden entity before binding it to a visual instance.Source: §2 · p. 3

3.1 Benchmark Format and Statistics

• WebEyes evaluates knowledge-guided perception through SearchGround boxes, SearchSeg masks, and SearchVQA descriptions matched to highlighted instances.The benchmark connects annotated objects, web evidence, queries, and grounded targets; its task views test localization, pixel-level shape recovery, and matching regions to external-knowledge descriptions.Source: §3 · p. 4, §3.1 · p. 4
• The benchmark contains 120 images, 473 annotated instances, 645 unique QA pairs, and 1,927 total task samples.The QA pairs yield 645 samples each for SearchGround and SearchSeg; 637 also have valid SearchVQA options.Source: §3.1 · p. 4
• Unlike RefCOCO-style alignment [20], ReasonSeg [9], and search benchmarks MMSearch and BrowseComp-VL [13] [19], WebEyes grounds searched evidence in boxes, masks, and region-level verification.The cited benchmark families mainly evaluate language-to-region alignment, reasoning-based segmentation without web identity resolution, or browsing with textual or image-level outputs.Source: §3.1 · p. 4

3.2 Annotation Pipeline

• WebEyes uses an object-first pipeline to create traceable links from image annotations through evidence and questions to grounded task instances.Images are selected for multi-instance complexity, objects receive masks, boxes, names, categories, and visual descriptions, and three-round chained searches retrieve recent public facts and build multi-hop evidence paths. Questions hide target names and direct visual attributes while retaining factual clues.Source: §3.2 · p. 5

3.3 Quality Control

• Quality control rejects shortcut-solvable or ambiguous candidates through automatic filtering and manual checks, preserving a verified chain from source image to grounded answer.Automatic filtering rejects 38.2% of generated candidates, and reviewers reject another 49.2% of those that pass it while checking evidence, target uniqueness, leakage, annotation quality, and task consistency.Source: §3.3 · p. 5

4 Pixel-Searcher: An Agentic Search-to-Pixel Workflow

• Pixel-Searcher has two phases: Agentic Search & Target Resolution finds and summarizes missing identity evidence, then Agentic Grounding & Tool Use binds the target to a visible instance.The second phase invokes visual tools to produce task-specific outputs.Source: §4 · p. 6

4.1 Overview

• Given an image and query, Pixel-Searcher forms a structured hypothesis with a resolved entity, visual category, and image-checkable cues distilled from external evidence.The hypothesis filters irrelevant reasoning and retains information needed for grounding; grounding and segmentation return a verified region or mask, while VQA matches answer options to a highlighted region.Source: §4.1 · p. 6

4.2 Agentic Search and Target Resolution

• Because queries may identify targets through events, creators, brands, roles, release history, or recent news, the agent plans searchable sub-goals and alternates among Search, Reason, and Resolve.This adaptive loop determines what the system must find rather than relying on the original query alone.Source: §4.2 · p. 6

K Key Cues

• The bounded search–reason loop can use one lookup or connect multiple evidence pieces, and it checks that the final hypothesis is visible, specific, supported, and visually consistent.Unsupported, overly generic, intermediate-clue, or context-inconsistent hypotheses are repaired.Source: K Key Cues · p. 7

4.3 Agentic Grounding and Tool Use

• Grounding tools propose regions that are verified against both image and resolved evidence; segmentation passes the verified region to SAM3 for boundary refinement, and failures can be traced to search, evidence integration, instance binding, or mask refinement.For Search-based Grounding, the verified region is the final answer.Source: §4.3 · pp. 7–8

5.1 Experimental Setup

• All methods use identical WebEyes inputs, splits, and task-specific interfaces without task-specific finetuning; Pixel-Searcher adds hidden-entity search before grounding and mask refinement.Grounding reports IoU and Recall@0.5, segmentation reports gIoU and cIoU, and Search-based VQA reports exact-match accuracy; Qwen3-VL-8B-Instruct is the general Qwen baseline.Source: §5.1 · p. 8

5.2 Main Results on WebEyes

• Pixel-Searcher raises Qwen3-VL-8B from 26.81 to 34.17 IoU and from 32.61 to 41.30 R@0.5 in Search-based Grounding.Gains are clearest in Anime and ICON, although translating external evidence into precise boxes remains difficult.Source: §5.2 · p. 8
• Across all three task views, Pixel-Searcher ranks first among open-source methods and improves Qwen3-VL-8B, while WebEyes remains challenging.Its gains are clearest in ambiguity-heavy categories, and the remaining gap to closed-source systems reflects evidence selection, entity resolution, and visual-region matching limitations.Source: §5.2 · pp. 8–9, Table 3 · p. 9, Table 4 · p. 9
• Pixel-Searcher raises Qwen3-VL-8B from 35.78 to 39.17 gIoU and from 25.94 to 32.41 cIoU in Search-based Segmentation.Gains are strongest in Vehicles, Anime, and Product, consistent with better hidden-entity grounding transferring to box-prompted SAM3 refinement.Source: §5.2 · p. 9, Table 3 · p. 9
• Pixel-Searcher raises Qwen3-VL-8B from 36.34 to 42.24 overall accuracy in Search-based VQA, with clear gains in Icons and Product.The smaller margin to closed-source models suggests fine-grained semantic comparison also matters.Source: §5.2 · p. 9, Table 4 · p. 9
• WebEyes often requires instance-level verification: models must reject plausible regions whose identities conflict with retrieved external evidence.The benchmark stresses recovering the hidden target identity and disambiguating similar visible objects, unlike standard referring perception where visual attributes often identify the target directly.Source: §5.2 · pp. 9–10

5.3 Ablation and Failure Analysis

• Removing direct localization cues lowers IoU from 34.17 to 20.14 and R@0.5 from 41.30 to 19.72, while gIoU/cIoU fall from 39.17/32.41 to 20.14/15.71.The direct-only variant reaches 22.28 IoU and 26.49 gIoU; reference matching and contradiction checking add smaller, consistent gains, so direct grounding must be combined with resolved entity evidence and visual verification.Source: §5.3 · p. 10, Table 5 · p. 10
• Among 389 failed segmentation samples, 304 are search/entity errors, 75 are entity-correct region errors, and 10 are box-to-mask transfer errors.About three quarters of errors occur before selecting the correct entity, while visual matching remains nontrivial even when external evidence is correct.Source: §5.3 · p. 10

6 Conclusion

• Agentic search consistently improves Search-based Grounding, Search-based Segmentation, and Search-based VQA when visual appearance alone is insufficient.The paper identifies perception deep research as converting web-derived evidence into box-level and pixel-level predictions.Source: §6 · p. 10
• The dominant bottleneck lies in the path from search planning to entity resolution and visual instance verification, not mask refinement after a correct box.Source: §6 · p. 10
• WebEyes provides benchmark infrastructure for studying perception deep research, while Pixel-Searcher offers a starting workflow for identifying entities and binding them to visual instances.Source: §6 · p. 10

• Table 6 pairs five source images with all annotated objects, overlaid target masks, object names, and corresponding queries.Source: A Dataset Samples · p. 14, Table 6 · p. 14
• One query asks for the film character who knocked a snake unconscious with a frying pan.Source: A Dataset Samples · p. 15
• Other queries identify characters through a heart-stabbed antivenom pen rescue or a map hidden in the Lynxley Journal.Source: A Dataset Samples · p. 15

B Prompt Templates

• The templates use braces for sample-specific questions, evidence, entity hypotheses, visual candidates, or answer options.Placeholders are filled at inference time.Source: B Prompt Templates · p. 16

B.1 Hidden-Entity Search

• Multi-hop questions are decomposed into 1–3 dependency-ordered sub-questions while preserving the final visible target rather than an intermediate clue.If a question is already simple, it remains the only sub-question.Source: §B.1 · p. 16
• The search agent alternates SEARCH, limited THINK, and ANSWER actions, returning a resolved entity with category, type, cues, and confidence.It prefers further targeted searches when evidence is ambiguous and must answer on the final interaction round.Source: §B.1 · p. 16
• Entity checks reject unsupported, overly generic, or intermediate-clue identities and require explicit evidence for exact model-level resolutions.When a proposed entity is inconsistent, prompts request targeted follow-up queries or re-resolution toward the best-supported visible target.Source: §B.1 · pp. 17–18

B.2 Visual Grounding

• Visual grounding extracts appearance cues such as shape, color, size, logos, and distinctive features from search results, then uses them to locate the target.Prompts require a tight box around one concrete instance in the first image, or a null box when no plausible match exists.Source: §B.2 · p. 18
• Candidate matching compares labeled boxes and crops with reference images, favors exact instance-level and tighter matches, and scores visual support against contradictions.The prompts constrain reasons to visible or explicit evidence and lower support for broad, multi-object, or mostly-background boxes.Source: §B.2 · p. 19

B.3 Grounding-Based VQA

• Grounding-based VQA identifies an entity from a description or selects the best answer option using the highlighted object’s visible evidence and searched entity information.Option selection returns a zero-based index, confidence, and a short reason.Source: §B.3 · p. 20

B.4 Fallback Candidate Generation

• Fallback candidate generation detects salient foreground instances with tight pixel-coordinate boxes, then ranks detected objects by visual saliency.Detection is capped by a configurable maximum number of boxes.Source: §B.4 · pp. 20–21
• The template requires JSON-only output, exactly one occurrence of each candidate ID, and no reasoning.These are stated as rules for fallback candidate generation.Source: §B.4 · p. 21