PaperlayerView exact revision on arXiv

Source-linked AI summary

EviRover: Reinforcing Agentic Perception Beyond a Glance

Kaixuan Fan, Kaituo Feng, Tianshuo Peng, Yilei Jiang, Manyuan Zhang, Junke Wang, Xiangyu Yue

arXiv:2609.40230v1cs.CVcs.AI

TL;DR

One-shot visual perception can fall short when queries require fine-grained inspection or knowledge beyond the image. EviRover is trained to seek missing evidence for perceptual queries, and improves across EviLens and other benchmarks, including by 30 points on average over its backbone on EviLens.

  • Problem

    One-shot perception assumes the image and a model’s parametric knowledge suffice, although some queries require finer visual inspection or information beyond the image.

    Source: §1 · pp. 1–2

  • Method

    EviRover learns to seek missing evidence within or beyond the image, using dedicated training data and supervised fine-tuning followed by agentic reinforcement learning.

    Source: §1 · pp. 2–3, §5 · p. 9

  • Results

    EviRover improves across EviLens, WebEyes, conventional perception, and general multimodal benchmarks, with a 30-point average gain over its backbone on EviLens.

    Source: §1 · p. 3

  • Takeaways & Limitations

    Accurate perception underpins downstream applications such as embodied manipulation and image editing.

    Source: §1 · p. 2

Abstract

from arXiv · show

Visual perception is conventionally formulated as a one-shot prediction from a single glance at the image, under the assumption that the image content and the model's parametric knowledge suffice to resolve the query. This assumption often fails in real-world scenarios that hinge on fine-grained visual details or require knowledge-intensive and up-to-date information. We term such cases \textit{perception under insufficient evidence} and formulate perception as an agentic process that can obtain information beyond a single glance. To address the absence of data for this setting, we design two dedicated data generation pipelines, yielding EviRover-SFT-5K and EviRover-RL-12K for training. We further construct EviLens, a human-verified benchmark comprising 688 instances across five perception categories. Building on these data, we present EviRover, to our knowledge the first perception agent explicitly trained to resolve perceptual queries through interaction, using supervised fine-tuning followed by agentic reinforcement learning. Experiments show that the 4B EviRover outperforms its backbone by 30 points on average on EviLens, reaching performance comparable to advanced proprietary models. The gains transfer beyond EviLens to WebEyes, conventional perception benchmarks, and general multimodal benchmarks, including a 15-point improvement on BrowseComp-VL. All code, models, and data are released.

1 INTRODUCTION

The introduction frames one-shot perception as inadequate when a query needs finer visual inspection or information beyond the image. It proposes learning to seek evidence for perceptual outputs and reports gains on EviLens and beyond.

1 INTRODUCTION

  • Perception under insufficient evidence arises when a single glance cannot resolve fine visual details or identify a target using image content and parametric knowledge alone.The proposed evidence-seeking formulation makes the perceptual output itself the goal: acquired evidence must be translated into a location, boundary, or count on the input image.Source: §1 · pp. 1–2
  • 30 points on average across EviLens’s five categories: EviRover outperforms its backbone, reaches performance comparable to advanced proprietary models, and generalizes across other benchmarks.Reported transfer includes WebEyes, conventional perception benchmarks, and general multimodal benchmarks, with a 15-point gain on BrowseComp-VL.Source: §1 · p. 3
  • EviRover is presented as the first perception agent trained to decide whether evidence is missing and how to acquire it within or beyond the image.The introduction contrasts this learned search with prior prompting strategies and approaches limited to external knowledge or segmentation.Source: §1 · pp. 2–3
  • Two data-generation pipelines yield EviRover-SFT-5K, EviRover-RL-12K, and EviLens, a human-verified benchmark of 688 instances across five perception categories.The pipelines target evidence missed at a glance and evidence requiring external information; EviLens covers segmentation, counting, localization, recognition, and spot-the-difference.Source: §1 · pp. 2–3

2 RELATED WORK

Prior visual perception work commonly predicts from the image in one shot, while evidence-seeking agents generally gather information for textual answers. EviRover instead treats perception as the goal and makes choosing what evidence to acquire part of the task.

Visual Perception

  • Conventional perception methods, including fine-grained small-object work, retain a one-shot assumption that needed evidence is present in the image.EviRover lifts that assumption by allowing perceptual predictions to draw on evidence acquired beyond a single glance.Source: §2 · p. 3

Evidence-Seeking Multimodal Agents

  • Image-inspection and web-search agents acquire evidence for textual answers, treating perception as a means rather than an end.Recent search-based grounding and segmentation methods either depend on manually designed prompting or restrict evidence seeking to external search for a specific task.Source: §2 · pp. 3–4
  • EviRover makes the perceptual prediction the goal, requiring acquired evidence to be resolved into an output on the input image.Because queries lack different kinds of evidence, deciding what to acquire and how is integral to the task.Source: §2 · p. 4

3 METHOD

EviRover’s method builds data and a benchmark for perception queries that require evidence beyond an initial glance, then trains an agent to acquire that evidence through interaction. The datasets target both hard-to-see image content and information outside the image; EviLens evaluates the resulting capabilities across five categories.

3.1 DATASET CONSTRUCTION

  • The two data-generation pipelines target evidence that is visually hard to resolve at a glance and evidence found beyond the image.The first uses tiny targets and multi-region exploration scenes; the second uses group photographs and verified identities to construct queries.Source: §3.1 · pp. 4–5
  • Queries over group photographs use verified identities, and iterative entity replacement makes them indirect until a model cannot resolve them without tools, up to 20 hops.Grounding and segmentation target at least one verified identity; counting requires all depicted individuals to be verified.Source: §3.1 · p. 5
  • SFT trajectories are retained for correct, logically consistent action sequences, while easy queries are downsampled so training includes both direct answers and evidence seeking.The retained trajectories form EviRover-SFT-5K alongside EviRover-RL-12K.Source: §3.1 · pp. 5–6

3.2 EVILENS

  • EviLens is a manually verified 688-instance benchmark spanning localization, recognition, spot-the-difference, segmentation, and counting.Its categories distinguish tiny or hidden targets, indirectly identified targets, cross-panel differences, and recognition-setting outputs as masks or counts.Source: §3.2 · p. 6

3.3 MODEL TRAINING

  • EviRover trains first with supervised fine-tuning for interaction and tool use, then with reinforcement learning to acquire missing evidence using final-output correctness as feedback.Its shared ten-tool framework combines external search, fine-grained visual inspection, and task-specific prediction tools.Source: §3.3 · pp. 6–7
  • Training uses task-specific outcome rewards in [0, 1], including mask IoU for segmentation and a soft set-level reward for spot-the-difference.For segmentation, predictions provide a box and positive and negative points; the spot-the-difference reward matches predicted boxes to annotated differences.Source: §3.3 · p. 7

4 EXPERIMENTS

EviRover improves over its backbone across EviLens categories and extends gains to WebEyes, conventional perception, and broader multimodal benchmarks. Evaluation compares direct and agentic systems under shared prompts and output interfaces, with agent trajectories limited to 35 tool calls.

4.1 TRAINING AND EVALUATION SETTINGS

  • Direct baselines predict in one pass, while agentic methods have the shared tool framework and a 35-call trajectory limit; all methods use the same prompt and task-specific output interfaces.Training uses a single node of 8 NVIDIA A800-80GB GPUs with FSDP.Source: §4.1 · p. 7

4.2 PERFORMANCE ANALYSIS ON EVILENS

  • 30.2 points on average over its backbone across EviLens categories; EviRover leads localization and recognition and ranks second on segmentation.It reaches a level comparable with the strongest proprietary systems, and comes within 1.3 points of GPT-5.6-Sol on counting.Source: Table 2 · p. 8, §4.2 · p. 8

4.3 PERFORMANCE ON WEBEYES

  • 40.0 IoU and 48.5 R@0.5 on grounding; EviRover leads evaluated models, beats Pixel-Searcher-8B by 10.4 gIoU on segmentation, and is comparable on VQA.VQA was absent from EviRover’s training data.Source: Table 3 · p. 8, §4.3 · p. 9

4.4 GENERALIZATION BEYOND EVILENS

  • EviRover improves over its backbone on every ReasonSeg and RefCOCOg split and metric, and outperforms task-specific models on most metrics.The comparisons include Seg-Zero-7B on segmentation and Perception-R1 on grounding.Source: Table 4 · p. 9, §4.4 · p. 9
  • EviRover improves over its backbone on all four general multimodal benchmarks, with gains up to 4 points on reasoning and BrowseComp-VL accuracy rising from 9.5 to 24.8.The benchmarks cover multidisciplinary reasoning, mathematical reasoning, and agentic visual search.Source: Table 5 · p. 9, §4.4 · p. 9

5 CONCLUSION

The paper frames perception under insufficient evidence as evidence seeking and trains EviRover to acquire missing evidence through supervised fine-tuning and agentic reinforcement learning. Experiments show gains over its backbone across EviLens, WebEyes, conventional perception, and general multimodal benchmarks.

5 CONCLUSION

  • EviRover substantially improves over its backbone on EviLens and WebEyes, with gains extending to conventional perception and general multimodal benchmarks.The paper concludes that learning to identify and acquire missing evidence improves perceptual accuracy and yields evidence-seeking capabilities that generalize to broader multimodal reasoning.Source: §5 · p. 9
  • The paper studies perception under insufficient evidence, where one glance at the image does not suffice to resolve the query.It reframes this setting as an evidence-seeking process.Source: §5 · pp. 9–10
  • EviRover actively seeks query-specific missing evidence through supervised fine-tuning followed by agentic reinforcement learning.The paper supports this approach with two dedicated data-generation pipelines and the human-verified EviLens benchmark.Source: §5 · p. 9

A.1 DATASET CONSTRUCTION

The dataset construction section describes filtering supervised fine-tuning trajectories and constructing description hops for anime and real-person queries in the RL data. The hop counts differ substantially between those query types.

A.1 DATASET CONSTRUCTION

  • Gemini-3.0-Pro was used for localization annotation because it was the only model then able to complete the task.Source: §A.1 · p. 15
  • Trajectories where the annotation model answers directly or verifies with one tool call are downsampled to discourage unsupported answers while retaining direct answering when the model’s knowledge suffices.The authors say imitation of these trajectories could encourage hallucination in Qwen3VL-4B-Instruct.Source: §A.1 · p. 15
  • Anime queries contain substantially fewer description hops than real-person queries in the RL data.The authors attribute this to limited coverage of anime characters in the model’s parametric knowledge; construction stops when Seed-1.8 can no longer resolve a query without tools.Source: §A.1 · p. 15, Figure 5 · p. 15

A.2.1 REWARD DETAILS

The reward details describe fallback scoring for segmentation tool failures and a spot-the-difference reward that provides partial credit for approximate matches while discouraging weak or redundant predictions. Matching is greedy rather than optimal.

A.2.1 REWARD DETAILS

  • When SAM3 fails to return a mask, the predicted box is scored as a rectangular mask against ground truth, preserving rollout ordering as a strict lower bound on SAM3’s result.These trajectories are flagged in training metadata so the fallback rate can be monitored.Source: §A.2.1 · p. 15
  • Setting the candidate IoU threshold to 0.1, below the evaluation threshold of 0.5, gives partially localized predictions intermediate reward during training.Using the evaluation threshold for reward computation would give such predictions zero credit.Source: §A.2.1 · p. 15
  • Weighting matched pairs by IoU^2 keeps weak overlaps informative but gives them substantially less credit than well-localized matches.For example, IoU values of 0.15 and 0.9 contribute 0.0225 and 0.81, respectively.Source: §A.2.1 · pp. 15–16
  • An additional box helps only when its recovered match is strong enough relative to current set-level performance; unmatched or duplicate boxes strictly reduce reward.The low threshold and exponent together provide dense approximate-localization feedback while discouraging indiscriminate or overly coarse proposals.Source: §A.2.1 · p. 16
  • Candidate pairs are greedily matched by descending IoU in O(nm log nm); optimal matching would change reward only in rare configurations.The method is described as stable in practice.Source: §A.2.1 · p. 16

A.3 EVALUATION DETAILS

The evaluation details standardize box-coordinate formats and specify the search and page-retrieval services. Spot-the-difference is evaluated using greedy one-to-one matching, with micro F1 as the primary metric.

A.3 EVALUATION DETAILS

  • Gemini predictions use yxyx coordinates while other models use xyxy, and all boxes are converted to a common format before scoring.Source: §A.3 · p. 16
  • All models are decoded at temperature 0.7; Serper serves text, image, and text-to-image search, while Jina retrieves web-page content.Source: §A.3 · p. 16
  • Spot-the-difference uses greedy one-to-one matching at IoU 0.5, and reports micro and macro F1 averaged across eight rollouts.Micro F1 is primary because it is more stable across 15 images containing 79 annotated differences.Source: §A.3 · p. 16

A.4 WHY TOKEN-MEAN REDUCTION IS ILL-SUITED FOR AGENTIC RL

Token-mean reduction changes how trajectories contribute to the policy update, favoring longer rollouts rather than merely changing a normalization constant. The resulting length bias can disrupt the balance of centered advantages and concentrate optimization influence, especially when trajectory lengths are heterogeneous.

A.4 WHY TOKEN-MEAN REDUCTION IS ILL-SUITED FOR AGENTIC RL

  • Token-mean gives longer trajectories greater effective weight, systematically overrepresenting them in the policy update.The section contrasts this with sequence-level averaging, which preserves equal trajectory weighting.Source: §A.4 · p. 17
  • When trajectory length correlates with advantage, token-mean can unbalance positive and negative update coefficients despite centered trajectory-level advantages.The imbalance can arise when negative advantages have greater magnitude at larger lengths, even without more frequent negative advantages.Source: §A.4 · pp. 16, 18
  • With heterogeneous or heavy-tailed lengths, token-mean reduces effective sample size, allowing a few unusually long rollouts to dominate an update.The section contrasts this concentration with sequence-level averaging, which assigns each trajectory equal weight.Source: §A.4 · pp. 18–19
  • Negative-advantage updates suppress the sampled action, and token-mean magnifies that suppression when long trajectories carry more negative advantage mass.The argument does not require PPO clipping or importance ratios; the same reasoning is stated to apply to the clipped surrogate gradient.Source: §A.4 · pp. 17, 19

A.5 PROMPT AND TOOL SPECIFICATIONS

This section specifies EviRover’s prompts and tools for gathering and verifying visual evidence, analyzing images, and producing task-specific answers. It also presents training-set localization and segmentation trajectories generated by EviRover.

A.5.2 TOOL SPECIFICATIONS

  • The shared tool interface uses coordinates normalized to [0, 1000], xyxy boxes, and numbered tool-returned images that can be reused in later calls.The query image is ORIGINAL; returned images are numbered IMG 001, IMG 002, and so on.Source: §A.5 · p. 19, §A.5.2 · p. 21

A.5.1 SYSTEM PROMPT

  • The system prompt requires one tool call per turn at most, normalized coordinates, and task-specific answer formats for grounding, segmentation, counting, and spot-the-difference tasks.Answers must be enclosed in <answer> tags; bounding boxes use xyxy coordinates and points use [x, y].Source: §A.5.1 · p. 20
  • The prompt directs the agent to inspect small or subtle targets and use crop, verify, and verify_part strategically to reduce uncertainty and check localization.verify checks placement and scale in the full image, while verify_part checks local content and tightness; both can be repeated.Source: §A.5.1 · p. 20, §A.5.2 · p. 22
  • For spot-the-difference tasks, the prompt requires comparing candidate regions against the right panel and reporting only confirmed differences on the left panel.Boxes must stay within the left panel, with x coordinates from 0 to 500; compare_lr checks corresponding regions side by side.Source: §A.5.1 · pp. 20–21, §A.5.2 · p. 22

A.5.2 TOOL SPECIFICATIONS

  • Web and image evidence tools support text search, text-based image search, reverse image search with optional cropping, and webpage browsing for query-relevant details.The browse tool uses Qwen3.5-27B as its summary model; image search accepts an optional box to isolate a region.Source: §A.5.2 · pp. 22–24
  • Visual and task-specific tools let the agent inspect and confirm regions, verify SAM3 mask overlays, compare panels, and perform image-based numerical computation.crop examines enlarged details, verify overlays boxes on the original, and verify_part crops a candidate for final checking; these tools can be called repeatedly.Source: §A.5.2 · pp. 22, 24, 25
  • compare_lr returns corresponding left- and right-panel regions side by side at identical size, enabling direct comparison of spot-the-difference images.The tool uses one box on the left panel; corresponding right-panel coordinates are offset by 500 pixels, making a single crop impractical.Source: §A.5.2 · p. 25
  • compare_lr takes an image_id and a four-number bounding box normalized to 0–1000 on the left panel, with x_max no greater than 500.The schema identifies ORIGINAL as the usual image_id.Source: §A.5.2 · pp. 25–26
  • The Python tool supports calculations, coordinate transformations, and image analysis, and returns visual results by assigning a PIL Image to _result_image.It also permits printed text output; code producing neither text nor an image is disallowed.Source: §A.5.2 · p. 26
  • Python execution preloads the original image, images from prior tool results, and an image count, with libraries including NumPy, PIL, OpenCV, and SciPy.The required input is a code string.Source: §A.5.2 · p. 26

A.6 AGENT TRAJECTORIES ON TRAINING AND BENCHMARK EXAMPLES

  • Figure 6 presents an example localization trajectory generated by EviRover on the training set.Source: Figure 6 · p. 26
  • Figure 7 presents an example segmentation trajectory generated by EviRover on the training set.Source: Figure 7 · p. 27

Related papers

Loading 2609.40230v1…