PaperlayerView exact revision on arXiv

Source-linked AI summary

HyperEyes: Dual-Grained Efficiency-Aware Reinforcement Learning for Parallel Multimodal Search Agents

Guankai Li, Jiabin Chen, Yi Xu, Xichen Zhang, Yuan Lu

arXiv:2605.07177v2cs.LGcs.AI

TL;DR

Multimodal search agents often process independent entities sequentially, incurring redundant interaction rounds and overlooking inference cost. HyperEyes enables concurrent grounded search and trains for efficiency, surpassing the strongest comparable open-source agent by 9.9% in accuracy with 5.3× fewer tool-call rounds on average.

  • Problem

    Existing multimodal search agents often invoke tools sequentially, creating redundant interaction rounds for decomposable multi-entity queries while evaluation omits inference cost.

  • Method

    HyperEyes fuses visual grounding and retrieval into one atomic action, combining parallel-amenable data synthesis with dual-grained reinforcement learning using trajectory- and token-level efficiency signals.

  • Results

    9.9% higher accuracy and 5.3× fewer tool-call rounds on average than the strongest comparable open-source agent across six benchmarks.

  • Takeaways & Limitations

    IMEB makes operational efficiency a first-class metric alongside answer accuracy for multi-entity visual search evaluation.

  • Takeaways & Limitations

    The framework is limited to static image and text environments, lacking grounding mechanisms for dynamic modalities such as video or audio.

Abstract

from arXiv · show

Existing multimodal search agents process target entities sequentially, issuing one tool call per entity and accumulating redundant interaction rounds whenever a query decomposes into independent sub-retrievals. We argue that effective multimodal agents should search wider rather than longer: dispatching multiple grounded queries concurrently within a round. To this end, we present HyperEyes, a parallel multimodal search agent that fuses visual grounding and retrieval into a single atomic action, enabling concurrent search across multiple entities while treating inference efficiency as a first-class training objective. HyperEyes is trained in two stages. For cold-start supervision, we develop a Parallel-Amenable Data Synthesis Pipeline covering visual multi-entity and textual multi-constraint queries, curating efficiency-oriented trajectories via Progressive Rejection Sampling. Building on this, our central contribution, a Dual-Grained Efficiency-Aware Reinforcement Learning framework, operates at two levels. At the macro level, we propose TRACE (Tool-use Reference-Adaptive Cost Efficiency), a trajectory-level reward whose reference is monotonically tightened during training to suppress superfluous tool calls without restricting genuine multi-hop search. At the micro level, we adapt On-Policy Distillation to inject dense token-level corrective signals from an external teacher on failed rollouts, mitigating the credit-assignment deficiency of sparse outcome rewards. Since existing benchmarks evaluate accuracy as the sole metric, omitting inference cost, we introduce IMEB, a human-curated benchmark of 300 instances that jointly evaluates search capability and efficiency. Across six benchmarks, HyperEyes-30B surpasses the strongest comparable open-source agent by 9.9% in accuracy with 5.3x fewer tool-call rounds on average.

1 Introduction

HyperEyes addresses redundant sequential search by grounding and retrieving multiple entities concurrently, while explicitly optimizing search efficiency alongside accuracy. It introduces IMEB for joint evaluation and reports higher accuracy with substantially fewer tool-call rounds than the strongest open-source agent.

  • Motivation: Multimodal search agents often invoke tools sequentially, creating redundant interaction rounds when queries involve multiple entities.External retrieval tools address the training-cutoff limits of LLM and MLLM parametric knowledge, but prevailing multimodal agents deepen reasoning through sequential calls.
  • Motivation: Parallel capability alone does not ensure efficiency because accuracy-only rewards provide no incentive to prefer compact trajectories over verbose brute-force search.The passage identifies explicit efficiency objectives as necessary for effective parallel search behavior.
  • HyperEyes: HyperEyes fuses visual grounding and retrieval in a Unified Grounded Search action space to search multiple entities concurrently in one turn.Its training combines Parallel-Amenable Data Synthesis with Dual-Grained Efficiency-Aware RL, including trajectory-level constraints and token-level On-Policy Distillation.
  • Evaluation: IMEB jointly evaluates answer accuracy and search efficiency on multi-entity visual scenarios, making operational efficiency a first-class metric.The benchmark is human-curated and uses image-question instances requiring concurrent localization.
  • Results: 9.9% in accuracy: HyperEyes-30B surpasses the strongest open-source agent while requiring 5.3× fewer tool-call rounds on average across six benchmarks.The paper characterizes this result as Pareto dominance over existing models.

2 Related Work

Related work has shifted from static single-hop RAG toward agentic systems that perform long-horizon reasoning and multi-turn tool use for complex queries. However, multimodal search agents remain inefficient in multi-entity settings, motivating efficient, parallelized, and redundancy-aware search.

  • Agentic Deep Research: Agentic Deep Research extends static single-hop RAG through iterative prompting, supervised fine-tuning, long-horizon search, and robust multi-turn tool calling.Examples include ReAct and Self-Ask, which address complex multi-hop and open-domain information-seeking challenges.
  • Multimodal Search Agents: MLLM research increasingly targets active environmental interaction, long-horizon search, and multi-tool orchestration for knowledge-intensive multimodal queries.Recent multi-hop benchmarks include FVQA, MMSearch-Plus, and BrowseComp-VL.
  • Efficiency Bottlenecks: Multi-entity multimodal search agents incur prohibitive latency from sequential tool invocations and code-execution sandbox initialization overhead.Decoupled “manipulate-then-search” paradigms create additional inefficiencies.
  • Open Challenge: Efficient, parallelized, and redundancy-aware multimodal search remains a critical unresolved challenge.This challenge follows the compounded inefficiencies of existing long-horizon multimodal search agents.

3 HyperEyes

HyperEyes is an iterative multimodal search agent that combines grounded image and text retrieval with parallel-oriented data synthesis and dual-grained efficiency-aware training. Its framework targets redundant tool use through efficient trajectory curation, trajectory-level rewards, and token-level correction on failed rollouts.

  • Agent architecture: HyperEyes follows an iterative reasoning-and-acting process in which each turn generates reasoning, selects a tool call, receives an observation, and continues until answering or reaching turn limit T.The agent trajectory is conditioned on accumulated context and uses policy πθ to select actions.
  • Agent architecture: HyperEyes provides image search for visually grounded queries and text search for natural-language evidence retrieval.Unified Grounded Search replaces separate crop-then-search processing, addressing localization brittleness and enabling parallel multi-entity retrieval.
  • Data synthesis: 271,000 genuinely tool-dependent tasks are built from public, annotated, visual multi-entity, and textual multi-constraint sources after filtering tasks solvable without external tools.The synthesis includes 25,000 novel queries, including 5,000 textual multi-constraint queries.
  • Data synthesis: 30,000 high-fidelity trajectories are retained for SFT through Progressive Rejection Sampling and quality filtering, favoring shortest successful, non-redundant parallel dispatch behaviors.Filtering removes format-invalid, zero-information-gain, and ungrounded trajectories.
  • Dual-grained reinforcement learning: Dual-Grained Efficiency-Aware RL supplements efficiency-filtered SFT with macro-level sequence optimization and micro-level token correction for failed rollouts.TRACE addresses redundant tool invocations at the trajectory level, while OPD recovers supervision for valid intermediate reasoning and tool calls preceding final errors.

4 Construction of IMEB Benchmark

IMEB addresses the omission of tool-call efficiency in multimodal search evaluation by testing concurrent localization and retrieval across multiple entities. It contains 300 human-verified instances and introduces CAS to jointly measure correctness and search efficiency.

  • Benchmark motivation: IMEB targets a benchmark gap by making search efficiency a primary evaluation axis for queries requiring concurrent localization and retrieval across multiple entities.Existing benchmarks emphasize reasoning accuracy while neglecting tool-call efficiency, encouraging sequential trajectories that increase latency and noisy retrievals.
  • Benchmark construction: 300 rigorously verified instances span diverse domains, averaging 4.6 entities per image and requiring concurrent external tool invocation.PhD-level annotators used multiple rounds of double-blind cross-validation, human peer review, and automated filtering to ensure questions are unambiguously solvable.
  • Evaluation metric: IMEB introduces the Cost-Aware Score (CAS) to jointly quantify reasoning correctness and search efficiency.CAS is designed to evaluate both dimensions rather than accuracy alone.
  • Evaluation metric: CAS preserves correctness as the primary objective while penalizing token consumption and sequential tool-call rounds using latency-oriented weights.The denominator uses Ntok, measured in thousands, and Ntool; the weights approximate one-second overheads for generation and tool execution.

5 Experiment

HyperEyes is evaluated against open-source agents and proprietary frontier models across six multimodal search benchmarks, including IMEB for efficiency-aware multi-entity retrieval. Results show improved accuracy-efficiency trade-offs, while ablations validate strict data filtering, adaptive TRACE references, and an efficiency-aligned distillation teacher.

  • Baselines: HyperEyes is compared with native Qwen3-VL backbones, five representative multimodal search agents, and three leading commercial models.The evaluated commercial models are Kimi-K2.5, Claude-Opus-4.6, and Gemini-3.1-Pro.
  • Benchmarks and Metrics: Evaluation spans six benchmarks covering shallow-hop visual search, complex multi-hop visual reasoning, and multi-entity grounded retrieval through IMEB.The benchmarks are MMSearch, FVQA, LiveVQA, BrowseComp-VL, MMSearch-Plus, and IMEB.
  • Main Results: 66.6% average accuracy is achieved by HyperEyes-235B after RL while reducing tool-call turns on complex benchmarks such as BCVL.The passage attributes this simultaneous accuracy-efficiency improvement to the RL stage and TRACE’s adaptive guidance.
  • Main Results: 4.3× and more are the CAS advantages reported for HyperEyes-30B over second-best open-source competitors on BCVL and IMEB, respectively.CAS jointly evaluates accuracy and inference cost, addressing verbose trajectories that traditional accuracy metrics do not penalize.
  • Data Curation Ablation: 7.2 absolute points of average accuracy are gained after strict filtering reduces a 121,000-trajectory training set to one quarter.The ablation supports quality filtering over raw data volume for training efficiency.
  • RL Ablations: 1.6 points of accuracy are added by adaptive TRACE beyond static tool-call references, while aligned HyperEyes-235B distillation adds 1.3 points without increasing the tool-call budget.Outcome-only rewards inflate tool calls, and replacing the aligned teacher with vanilla Qwen3-VL-235B causes a severe accuracy drop.

6 Conclusion

HyperEyes is a parallel multimodal search agent designed to search wider rather than longer by combining visual grounding and retrieval in one atomic action. Its two-stage training pipeline synthesizes parallel-amenable queries and distills 30K efficient cold-start trajectories.

  • Agent Design: HyperEyes fuses visual grounding and retrieval into one atomic action, dispatching grounded queries concurrently within each round.This enables parallel multimodal search across visual multi-entity and textual multi-constraint queries.
  • Two-Stage Training: Training begins with a Parallel-Amenable Data Synthesis Pipeline that produces visual multi-entity and textual multi-constraint queries.The pipeline targets queries whose retrieval components can be handled in parallel.
  • Two-Stage Training: 30K efficient cold-start trajectories are distilled using Progressive Rejection Sampling and quality filtering.These trajectories form the output of the first training stage.

A Limitations

HyperEyes has three stated limitations: its On-Policy Distillation depends on a stronger same-family teacher, its reasoning capability is therefore bounded, and its framework does not support dynamic video or audio environments.

  • A Limitations: On-Policy Distillation requires a stronger same-family teacher, inherently bounding the student’s reasoning capabilities and preventing direct application at frontier scale.This dependency limits how the method can be extended to frontier-scale models.
  • A Limitations: The parallel framework focuses exclusively on static image and text environments.It does not address dynamic modalities requiring spatial-temporal grounding, such as video or audio.

B Broader Impacts … C.1.3 QA Filtering

HyperEyes combines efficiency, reproducibility, and grounded applicability with a multi-stage pipeline for synthesizing and filtering visual and textual retrieval-oriented QA data. The curation process builds structured multi-entity and multi-constraint questions, then removes samples that are unreliable or answerable without tools.

  • B Broader Impacts: 5× lower per-query tool cost supports shorter latency, lower energy consumption, and grounded multimodal QA on resource-constrained devices.This reduction occurs at comparable or better accuracy.
  • B Broader Impacts: Released IMEB, synthesis data, training recipes, and HyperEyes-30B checkpoints establish a reproducible foundation for efficiency-aware multimodal-agent research.The releases lower entry barriers for academic and resource-constrained groups.
  • B Broader Impacts: Unified Grounded Search and Dual-Grained Efficiency-Aware RL provide a general recipe for source-grounded assistants in education, scientific exploration, accessibility, and visual question answering.The stated applications require reliable, verifiable answers.
  • C.1.1 Details of Visual Multi-Entity Synthesis: The visual multi-entity synthesis pipeline comprises source selection, per-class knowledge-base and QA-pool construction, and mosaic-based question composition.Its base corpus spans five fine-grained datasets covering birds, flowers, cars, aircraft, and pets.
  • C.1.1 Details of Visual Multi-Entity Synthesis: Gemini-3.0-Flash enriches each class with web-derived knowledge and combines sampled evidence with entity locations to generate composite-image QA.Images are drawn from 2 to 8 classes and assembled into regular-grid mosaics such as 2×1, 1×4, and 2×4.
  • C.1.2 Details of Textual Multi-Constraint Synthesis: The textual multi-constraint pipeline discovers pivots, filters attributes, constructs constraint chains, and generates natural-language questions.Pivot discovery uses a 2 to 3 hop Wikidata walk, while candidate answers are first-order neighbors of the pivot.
  • C.1.2 Details of Textual Multi-Constraint Synthesis: Whitelist-based predicates and diversity-aware greedy filtering shrink candidates to |B∗| ∈[1, 8], after which Gemini-3.0-Flash paraphrases constraints without exposing the underlying schema.The whitelist emphasizes discriminative domains, while high-bias attributes are forbidden.
  • C.1.3 QA Filtering: A unified two-stage filter discards QA pairs failing any of six Gemini-3.0-Flash checks, then removes items answerable by Qwen3-VL-235B under tool-free pass@1.The six checks cover factual consistency, uniqueness, clarity, temporal stability, naturalness, and answer non-leakage.

C.2 SFT data · C.3 RL data · C.4 IMEB data

The data pipeline builds efficiency-oriented SFT and RL datasets through difficulty filtering, progressive trajectory selection, and failure-based sampling, then introduces IMEB as a manually curated testbed requiring parallel multimodal search. Together, these procedures target robust, tool-dependent, and efficiency-aware agent evaluation.

  • C.2 SFT data: 271,000 initial QA pairs combine public multi-hop and visual-recognition benchmarks with self-synthesized parallel-amenable corpora and a specialized source.This pool is processed by the progressive rejection sampling pipeline to construct supervised fine-tuning data.
  • C.2 SFT data: Qwen3-VL-235B filters out queries it solves on the first attempt with full tool access, retaining harder instances for learning parallel search.The filtering uses pass@1 evaluation and discards samples considered too easy.
  • C.2 SFT data: Progressive rejection sampling tests budgets B = {2, 4, 8} with five trajectories per budget, retaining the shortest successful rollout found under the tightest feasible budget.The policy model is Gemini-3.0-Flash, and sampling uses pass@5 at each budget level.
  • C.2 SFT data: Quality filtering removes malformed or noncompliant trajectories, unimodal or sequential shortcuts, and searches producing duplicate or off-topic information.The constraints enforce think-before-act formatting, multimodal synergy, concurrent retrieval, and nonzero information gain.
  • C.3 RL data: The RL dataset retains queries where the SFT-trained policy fails at pass@1 but succeeds within five attempts at pass@5, yielding moderate-difficulty samples for initial TRACE references.These samples come from queries that previously failed during progressive rejection sampling.
  • C.4 IMEB data: Five PhD annotators source diverse multi-entity images and vary entity counts, domains, question types, and visual complexities when authoring IMEB question-answer pairs.IMEB curation combines manual annotation with automated filtering to assess parallel multimodal search capabilities.
  • C.4 IMEB data: Double-blind cross-validation verifies that each IMEB instance is solvable from correct context, unambiguous, and strictly requires parallel search; Qwen3-VL-235B removes tool-unnecessary questions.The final verifier discards instances answered correctly without invoking search tools.

D Evaluation Details … E.6 Hyperparameter Summary

The paper details evaluation controls and a two-stage HyperEyes training setup for 30B and 235B variants, including distinct infrastructure, search backends, reinforcement-learning recipes, and distillation. It also documents environment mismatches affecting baseline comparisons and summarizes implementation hyperparameters in dedicated tables.

  • D Evaluation Details: Baseline comparisons are complicated by differing search APIs, parsing modules, summary models, system prompts, and checkpoint availability across agents.WebWatcher, MMSearch-R1, and VDR use Jina with different summary models, while DeepEyes-V2 restricts observations to search-result snippets; such differences can affect performance independently of reasoning capability.
  • E Training Details: HyperEyes training uses SFT followed by GRPO-based RL, with separate 30B and 235B runs sharing the same approximately 30,000 curated trajectories.SFT uses LoRA fine-tuning, while RL uses TRACE on medium-difficulty subsets containing 6,056 samples for 30B and 9,337 for 235B.
  • E.1 Computing Infrastructure: All training runs use eight nodes with 8× NVIDIA H20 141 GB GPUs, while the OPD teacher runs on a dedicated inference cluster.SFT uses msswift, RL uses Relax with Megatron-LM and SGLang, and both stages run on PyTorch 2.9.1, CUDA 12.9, and SGLang 0.5.9.
  • E.2 Supervised Fine-Tuning Stage: The 30B and 235B SFT variants initialize from their respective Qwen3-VL instruct checkpoints and share the same LoRA and optimization recipe.Only Megatron parallelism is adjusted for the larger backbone.
  • E.3 Reinforcement Learning Stage: The RL variants share GRPO and TRACE, but HyperEyes-30B additionally enables OPD, whereas HyperEyes-235B uses TRACE alone and becomes the teacher.TRACE uses format penalty −0.5, γ = 1.5, λred = −0.1, and eight rollouts per update; OPD uses λkd = 0.05.
  • E.4 Search Backends: HyperEyes uses unified asynchronous text and reverse-image search adapters supporting parallel batched queries, with each rollout capped at 8 tool invocations and 9 model turns.Text search uses SerpAPI Google Web Search, image search uses Google Reverse Image Search, and retrieval concurrency is limited to 64 in-flight requests.
  • E.5 On-Policy Distillation Teacher: The OPD teacher is served through a separate SGLang HTTP endpoint and is the converged HyperEyes-235B (RL) model.This decoupling permits a larger RL-aligned teacher without increasing training-side memory.
  • E.6 Hyperparameter Summary: Tables 9 and 10 provide the complete SFT- and RL-stage hyperparameters, distinguishing shared values from per-variant settings.A single listed value applies to both 30B and 235B; separate columns specify variant-specific settings.

E.7 Robustness to Random Seed · F Further Analysis · F.1 More Tool Calls Do Not Imply Higher Accuracy

HyperEyes-30B remains stable across three independent RL seeds, with minimal variation in accuracy and tool-call efficiency. Further analysis motivates progressive rejection sampling by testing whether larger tool-call budgets improve accuracy.

  • E.7 Robustness to Random Seed: HyperEyes-30B RL was re-run with N = 3 independent seeds while holding the SFT checkpoint, data, hyperparameters, and infrastructure fixed.The seeds were 42, 1234, and 2026, with evaluation across all six benchmarks.
  • E.7 Robustness to Random Seed: 1.22 points: accuracy standard deviation was at most this value across six benchmarks, while the six-benchmark average varied by only 0.19 points.The maximum occurred on MMSearch.
  • E.7 Robustness to Random Seed: 0.10: the maximum per-benchmark standard deviation of average tool turns across seeds, compared with only 0.02 on the six-benchmark average.These results support stability of HyperEyes’s efficiency claims under different random seeds.
  • E.7 Robustness to Random Seed: 63.7–64.1: the six-benchmark average accuracy stayed within this band across seeds, while average tool turns stayed within 2.17–2.20.The reported Table 2 numbers were therefore not attributed to a favorable seed.
  • E.7 Robustness to Random Seed: Table 11 compares mean ± std across three RL runs with the single-seed Table 2 results, reporting benchmark accuracy (%) and average tool calls per query.The comparison uses the same SFT checkpoint and seeds {42, 1234, 2026}.
  • F.1 More Tool Calls Do Not Imply Higher Accuracy: Progressive rejection sampling was motivated by the observation that blindly increasing tool calls fails to improve, and often degrades, final-answer accuracy.The phenomenon was evaluated with Qwen3-VL-235B on FVQA and BCVL across five tool-call budgets.

F.2 Validity Analysis of the Unified Grounded Search Paradigm

The validity analysis compares Unified Grounded Search with LLM Crop and Code Crop under controlled data and scale conditions. Across six benchmarks and both model scales, Unified Grounded Search achieves higher average accuracy with fewer tool-call turns.

  • Compared paradigms: Unified Grounded Search jointly emits bounding boxes and search actions for all target entities in one decision, collapsing cropping and retrieval into a parallel tool-call round.LLM Crop uses external Qwen3-VL-235B grounding after a natural-language region description, whereas Code Crop executes generated Python in a sandbox.
  • Controlled comparison: The comparison holds training data, trajectory scale, and other variables constant, using 3k QA pairs synthesized by the Visual Multi-Entity Synthesis pipeline.Unified Grounded Search and Code Crop trajectories are mutually converted from the same underlying trajectory data, differing only in grounding expression.
  • Results: 58.2 average accuracy on the 235B backbone exceeds 53.0 for LLM Crop and 55.3 for Code Crop across the six benchmarks.The advantage persists across both the 30B and 235B scales.
  • Results: 2.04 average tool-call turns on the 235B backbone is lower than 2.7 for LLM Crop and 3.07 for Code Crop.Unified Grounded Search therefore combines the highest average accuracy with the fewest average tool-call turns.

F.3 Robustness to Distractor Evidence · F.4 Case Study: DeepEyes-V2 vs. HyperEyes · G Prompt Templates

HyperEyes remains more robust than Qwen3-VL under shuffled in-domain distractors and uses parallel grounded search to answer a six-person visual question with roughly 4× fewer tool-use rounds. The prompt templates specify unified region-level retrieval, batched independent queries, and efficiency-oriented trajectory construction.

  • F.3 Robustness to Distractor Evidence: F.3 Robustness to Distractor Evidence: 1.9%–5.8% accuracy separates HyperEyes-235B (SFT) from the Qwen3-VL-235B backbone across all tested noise levels.The result is attributed to sharper sensitivity to answer-relevant evidence from the parallel-amenable SFT corpus.
  • F.4 Case Study: DeepEyes-V2 vs. HyperEyes: F.4 Case Study: DeepEyes-V2 vs. HyperEyes: DeepEyes-V2 serially crops and searches each person, whereas HyperEyes grounds all six individuals in one unified search round.HyperEyes then performs a focused text search on the verified candidate’s election record.
  • F.4 Case Study: DeepEyes-V2 vs. HyperEyes: F.4 Case Study: DeepEyes-V2 vs. HyperEyes: 4× fewer tool-use rounds are achieved while substantially improving answer accuracy on the multi-entity visual question.The case study’s correct answer is Symbiosis New Party.
  • G Prompt Templates: G Prompt Templates: Unified grounded search accepts normalized image regions through image_search and multiple text queries through text_search.Coordinates use a 0.0~1.0 normalized range, and examples cover full-image, region-level, and batched text searches.
  • G Prompt Templates: G Prompt Templates: Baseline prompt variants include LLM-Crop, which uses crop_image before image_search, and Code-Crop, which generates Python cropping code before searching sub-images.The appendix also specifies an LLM-as-a-judge prompt for benchmark accuracy evaluation.
  • G Prompt Templates: G Prompt Templates: Trajectory construction requires batching all target regions and independent sub-queries, then stopping tool calls once evidence is sufficient.The trajectory prompt therefore tightens the unified grounded-search action guidelines toward parallelism and efficiency.
Loading 2605.07177v2…