PaperlayerView exact revision on arXiv

Source-linked AI summary

WeAgent-MMSearch: Native Text-Vision Interaction for Multimodal Search Agents

Zongkai Liu, Hui Zhang, Liqiang Niu, Zhen Cao, Han Li, Juntao Liu, Wenchao Chen, Chengduo Zhao, Chao Yu, Fandong Meng

arXiv:2608.28062v1cs.AI

TL;DR

Multimodal search agents need current visual and textual web evidence, but existing environments may drop images and long-horizon failures can waste or corrupt training. This paper introduces a harness, data-and-training system, and diagnostic benchmark; its final RL model improves the eight-benchmark average by 19.22 points over the base checkpoint. A scope boundary is that cached responses trade some freshness for predictable latency on repeated requests.

  • Problem

    Existing multimodal search environments may omit retrieved images from later context, while long-horizon execution failures waste computation and contaminate training signals.

  • Method

    WeAgent-MMSearch combines WeAgent-Harness for persistent text–vision interaction and runtime recovery with harness-native task construction, agentic SFT, and Failure-Aware GSPO.

  • Results

    19.22 points: WeAgent-MMSearch-RL improves the eight-benchmark average over the base checkpoint to 55.97 and competes with much larger models.

  • Takeaways & Limitations

    VisTarget-Bench separates target-image retrieval from visual interpretation, while persistent multimodal interaction and runtime recovery form complementary components of the reported system.

  • Takeaways & Limitations

    Caching trades a small amount of freshness for predictable latency when an identical request has already been observed.

Abstract

from arXiv · show

Multimodal search agents extend parametric knowledge with newly emerging and long-tail evidence from the open web. Yet many existing agentic search environments often expose retrieved evidence only as text and omit tool-returned images from subsequent context, reducing visually grounded trajectories to text-only reasoning. Long-horizon interaction also compounds tool-call, response-length, timeout, and budget failures, which can discard salvageable trajectories, waste rollout computation, and disturb policy updates. To address these issues, we introduce WeAgent-Harness, a multimodal agentic harness that supports native text-vision interaction and runtime recovery. Retrieved images receive persistent disk references, allowing the model to inspect, process, and cite them throughout the trajectory. Based on this harness, we develop WeAgent-MMSearch, an integrated system spanning data construction, agentic post-training, and multimodal rollout. For data construction, a strong MLLM uses WeAgent-Harness to discover, synthesize, and verify MMSearch-style tasks and collect expert trajectories. During post-training, our Failure-Aware GSPO (FA-GSPO) recovers salvageable abnormal rollouts and filters invalid ones to improve bounded multimodal planning and search.We also introduce VisTarget-Bench, a 150-task human-verified benchmark that pairs each question with a held-out target image, distinguishing image-retrieval failures from visual-perception failures. Evaluation on VisTarget-Bench and seven public benchmarks shows that agentic post-training improves the average score by 19.22 points, enabling our model to outperform similarly sized open-source models and rival models with roughly ten times its parameter count.

1 Introduction

The paper addresses incomplete visual feedback, unreliable long-horizon execution, and scarce high-quality MMSearch supervision with a multimodal harness, data pipeline, and failure-aware post-training recipe. Evaluation shows substantial gains across VisTarget-Bench and seven public benchmarks.

  • Motivation: Existing search environments often omit retrieved images from later policy context, causing visually grounded trajectories to collapse into text-only reasoning.Visual facts that cannot be faithfully verbalized may disappear from the model-visible state.
  • Motivation: Long-horizon interactions amplify tool-call errors, oversized outputs, timeouts, and budget exhaustion, wasting rollout computation and contaminating optimization signals.These abnormalities can interrupt reasoning, reduce throughput, and produce failures unrelated to search-policy quality.
  • Motivation: High-quality MMSearch supervision is scarce because tasks must preserve visual dependence across multi-hop retrieval while remaining executable under live tool contracts.Task synthesis, validity checking, and expert rollout collection jointly constrain available supervision.
  • Contributions: WeAgent-Harness preserves retrieved images through persistent references with provenance, supports native text–vision interaction, and recovers or bounds abnormal executions.It integrates search, webpage, image, reverse-image, and code tools while caching requests and returning structured errors for self-correction.
  • Contributions: WeAgent-MMSearch uses the harness for task discovery, synthesis, verification, expert-trajectory collection, agentic SFT, and Failure-Aware GSPO.FA-GSPO retains salvageable abnormal rollouts and filters invalid samples from the training signal.
  • Results: 19.22 points: agentic post-training improves the average score across VisTarget-Bench and seven public benchmarks, reaching 55.97% for WeAgent-MMSearch-RL.The final model outperforms similarly sized open-source models and rivals models with roughly ten times its parameter count.

2 Related work

Prior multimodal search systems range from modular planner–reasoner designs to end-to-end reinforcement-learning agents, while datasets and benchmarks increasingly target long-horizon, provenance-aware visual search. The paper positions VisTarget-Bench as a distinct diagnostic for target-image retrieval and post-retrieval visual correctness.

  • Agentic search: Early systems use modular designs, including planner–reasoner coordination with web and image tools or vision-language models coupled to web agents.AVIS and Vision Search Assistant exemplify these complementary architectures.
  • Agentic search: More recent systems adapt end-to-end reinforcement learning to train a single vision-language model to interleave reasoning and search.This line differs from explicitly modular tool orchestration.
  • Data and optimization: Long-horizon search research addresses task scarcity and trajectory cost through synthetic difficulty control, source-anchored visual grounding, diverse tool environments, and failure-aware RL.REDSearcher and OpenSearch-VL combine data construction with optimization strategies for reducing interaction burden.
  • Benchmarks: VisTarget-Bench differs from existing multimodal search benchmarks by pairing each question with a held-out target image and evaluating target-image retrieval separately from answer correctness.This diagnostic targets visual dependence rather than only end-to-end answer quality.

3 WeAgent-Harness: Multimodal Agentic Harness

WeAgent-Harness provides a shared multimodal state and evidence contract for asynchronous rollout, tool interaction, training, and inference. It keeps visual evidence addressable across turns while using caching, recovery, and budgets to make executions bounded and replayable.

  • Persistent multimodal state: WeAgent-Harness is the shared interaction substrate for task construction, trajectory collection, post-training, and inference.It exposes web, image, reverse-image, and code tools through one multimodal state and evidence contract.
  • Persistent multimodal state: Textual history and registered images are jointly serialized into each subsequent policy context, allowing retrieved pixels to condition later reasoning and tool calls.Images retain stable model-visible references and source metadata for later revisiting, reverse-search, and comparison.
  • Persistent multimodal state: Figure 3 separates concurrent rollout, reward, and training services from the harness-managed state, action, and text–image observation cycle.The harness serializes aligned multimodal interactions as trainable trajectories.
  • Reliable and bounded execution: Exact-match caching returns previously observed search responses immediately while unseen or time-sensitive requests still reach the live web.Periodic refreshes limit the freshness cost of cache reuse.
  • Reliable and bounded execution: Runtime recovery repairs unambiguous call errors, returns structured observations for unclear errors, suppresses repeated invalid calls, and bounds overlong episodes.These mechanisms preserve recoverable trajectories while preventing stalled or indefinitely long episodes from dominating rollouts.
  • Harness-native data pipeline: Figure 4 links seed-based, backward multi-hop task construction with trajectory sampling, assigning successful executions to SFT and mixed outcomes to RL.The same harness-native contract supports both stages.

4 Harness-Native MMSearch Data Pipeline

The harness-native pipeline converts visually grounded seeds into verified multi-hop tasks and executable trajectories, then normalizes and filters in-house and open-source data into final SFT and RL mixtures.

  • Pipeline overview: The four in-house stages are seed generation, backward multi-hop expansion, task validation, and trajectory sampling under the same contract used for learning and inference.The expert agent uses the shared tools, observations, and evidence registry throughout construction and execution.
  • Stage 1: Seed generation: Seeds combine time-sensitive web events and stable visually grounded facts, retaining authoritative sources, timestamps, target images, provenance, and pixel-dependent answers.A seed is accepted only when its image is reachable, visually necessary, and auditable.
  • Stage 2: Backward multi-hop expansion: Backward expansion constructs 2–4-hop dependency chains by generating one predecessor QA pair at a time from the verified visual endpoint.Solving a composed task requires following bridge relations, retrieving the target image, and reading its pixels.
  • Stage 3: Task validation: Validation removes candidates with missing images, non-unique bridges, answer leakage, inconsistent chains, or answers that do not genuinely depend on target-image pixels.The process requires complete anchors, relations, endpoints, and provenance.
  • Stage 4: Trajectory sampling: Successful and complete trajectories form SFT data, while failed or incomplete executions remain useful for RL after stratifying solved and unsolved tasks.The RL query set balances demonstrated feasibility with headroom for policy improvement and holds out benchmark candidates before optimization.
  • Mixture construction: External SFT and RL corpora are converted into a common contract because their tool names, action schemas, and answer conventions have source-dependent semantics.Source-specific namespaces prevent collisions while preserving a shared trajectory grammar.
  • Mixture construction: 8-rollout groups: RL filtering discards uniformly successful or uniformly failed queries and removes ambiguous or non-verifiable targets.Retained prompts provide outcome variance and concrete, judgeable targets for group-relative optimization.
  • Mixture construction: 59.5K trajectories: the final SFT mixture contains 3.3K in-house and 56.2K open-source trajectories, while the RL mixture contains 13.3K prompts.The RL mixture includes 4.9K in-house and 8.3K open-source prompts.

5 VisTarget-Bench: Diagnosing Visual Evidence Use

VisTarget-Bench evaluates whether agents retrieve and use intended visual evidence, not merely produce correct final answers. Its held-out target-image design separates retrieval failures from failures to apply retrieved visual information.

  • Benchmark Design: VisTarget-Bench pairs each task with a held-out target image and requires the agent to return its supporting image index.The benchmark contains 150 human-verified tasks, constructed from seeds held out before SFT and RL splitting.
  • Diagnostic Metrics: Comparing the cited image with the target separates final-answer accuracy, target-image retrieval, and answer correctness after target acquisition.A VLM judge assesses target-image correspondence, while invalid image citations receive zero correspondence.
  • Benchmark Design: The benchmark contains 80 tasks with input images and 70 tasks beginning with text only.These counts correspond to 53.3% and 46.7% of the 150-task benchmark, respectively.
  • Diagnostic Metrics: Accuracy measures final-answer correctness, Image Cite measures visual-evidence citation, Target Match measures target correspondence, and Match+Acc measures correctness after target acquisition.Conditional rates are defined as zero when their conditioning set is empty.

6 Agentic Post-Training

Agentic post-training combines supervised learning from successful demonstrations with FA-GSPO reinforcement learning that recovers and filters long-horizon rollouts. The resulting procedure teaches tool-use behavior while excluding invalid trajectories from optimization.

  • Supervised Fine-Tuning: SFT learns basic reasoning and tool-use behavior from correct, complete demonstration trajectories.The model is trained with a next-token objective while tool observations and intervention prompts are masked from the loss.
  • Failure-Aware Reinforcement Learning: FA-GSPO separates runtime recoverability from optimization eligibility by recovering candidate rollouts before retaining them for policy updates.It addresses failures such as tool-call timeouts and overlong responses in long-horizon search.
  • Failure-Aware Reinforcement Learning: Overlong turns are regenerated once, reducing wasted computation from discarded trajectories.The recovery step precedes filtering of trajectories that remain unusable after runtime checks.
  • Eligible-Trajectory Filtering: Only trajectories that remain trainable after runtime checks enter the dynamically constructed eligible set E_g.The effective number of trajectories is data-dependent, and eligible trajectories alone participate in group normalization.
  • Reward and Optimization: The reward combines binary answer correctness and response-format compliance as R_gi = 0.9 C_gi + 0.1 F_gi.C_gi comes from an external answer judge, while F_gi indicates whether the final assistant turn obeys the required format.
  • Eligible-Trajectory Filtering: Invalid trajectories contribute neither to group reward statistics nor to the policy gradient, and groups with fewer than two eligible trajectories are discarded.This preserves a within-prompt preference signal for policy updates.

7 Experiments

Experiments evaluate WeAgent-MMSearch across eight benchmarks, VisTarget-Bench diagnostics, harness ablations, data and optimization ablations, and runtime recovery. Results show gains from persistent image re-feeding, curated harness-native data, and FA-GSPO recovery and filtering.

  • Main Results: Evaluation covers seven public benchmarks and VisTarget-Bench, with reported results generally averaged over three runs and baselines shown as point estimates.The eight-benchmark average includes all seven public benchmarks plus VisTarget-Bench.
  • Main Results: WeAgent-MMSearch-RL outperforms or remains competitive with similarly scaled open-source search agents, although several frontier models retain an overall advantage.Post-training narrows the frontier-model gap and reaches comparable performance on several benchmarks.
  • VisTarget-Bench: Post-training raises VisTarget-Bench final-answer accuracy from 8.00% for Base to 24.00% after SFT and 30.22% after RL, while image citation rises from 1.33% to 20.67%.Among image-citing responses, target matching reaches 47.62% for SFT and 87.10% for RL; after target matching, answer accuracy reaches 70.00% and 77.78%.
  • Harness Ablation: Across compared models, WeAgent-Harness improves aggregate scores over Hermes, with gains attributed to richer multimodal observations and re-feeding retrieved images.Removing image re-feeding deprives the policy of pixel-level evidence that text search alone cannot recover.
  • Data and Optimization Ablations: 48.06% average score follows trajectory-format normalization from 44.07%, and harness-native trajectories raise it further to 51.04%, including VisTarget-Bench growth from 14.45% to 24.00%.The ablation isolates unified formatting and the addition of harness-native data.
  • Data and Optimization Ablations: 55.97% average score and 30.22% VisTarget-Bench accuracy follow complete FA-GSPO, versus 51.10% and 9.78% without its proposed recovery and filtering mechanisms.Recovery retains locally abnormal executions, while context-limit and wall-clock-timeout trajectories are excluded from optimization.

8 Conclusion

The conclusion presents WeAgent-MMSearch as an end-to-end multimodal search system centered on persistent visual state and runtime recovery. Its final model improves over the base, competes strongly with open-source and much larger models, while target-image retrieval remains distinct from visual interpretation.

  • System: WeAgent-MMSearch preserves retrieved images as addressable policy state, re-feeds visual observations across turns, and applies runtime recovery before local errors invalidate trajectories.The same interaction contract supports task construction, expert sampling, SFT, FA-GSPO, and inference.
  • Results: 55.97 is the final model’s eight-benchmark average, a 19.22-point improvement over the base checkpoint.The reported model outperforms similarly sized open-source models and is competitive with models having roughly ten times its parameter count.
  • Interpretation: On VisTarget-Bench, matching the supporting image does not guarantee full Match+Acc, and RL improves target-image-matched answer accuracy over SFT.The result separates evidence retrieval from successful visual interpretation.
  • Analysis: Persistent multimodal interaction, environment-aligned learning, and runtime recovery are reported as complementary components of the system’s gains.The analysis links text-only harness weakness, targeted recovery interventions, and exclusion of context-limit and wall-clock-timeout trajectories to these components.

A Implementation Details

Implementation details specify distinct SFT and RL optimization settings. SFT updates language and visual modules, whereas RL freezes the vision tower and projector and trains only the language-model policy.

  • SFT: SFT uses a global batch size of 32 trajectories, with learning rates of 1.0 × 10−5 for the language model and 1.0 × 10−6 for the vision tower.
  • RL: RL initializes from the SFT checkpoint, freezes the vision tower and projector, and trains only the language-model policy.The RL recipe uses a prompt batch of 64 with 8 samples per prompt, for a global trajectory batch of 512.
  • RL: RL uses a constant learning rate of 5.0 × 10−7 and rollout temperature 1.0.

B Tool Interfaces and Runtime Configuration

The runtime configuration exposes five native tools and makes returned images persistent, addressable, and available to the next model turn. Tables define the model-visible interfaces, active limits, recovery rules, and persistence policy.

  • Tool Interfaces: A tool-returned image is saved under a stable relative filename, registered in episode evidence state, and attached as visual input to the next model turn.This persistence policy supports continued multimodal interaction across the trajectory.
  • Runtime Configuration: Table 4 defines the model-visible tool interfaces, while Table 5 specifies episode budgets, recovery rules, and persistence policy.

C RL Prompts

The RL rollout and evaluation prompts use runtime-instantiated time information and tool schemas, with system-level settings documented separately.

  • CURRENT_TIME is instantiated at hour granularity during runtime.
  • TOOL_LIST is populated with the available function schemas.
  • Tables 5 and 6 document harness budgets, recovery behavior, tool settings, and prompts for RL rollouts and evaluations.

D Final-Response Comparison

The qualitative comparisons show WeAgent-MMSearch-RL answering several multimodal retrieval questions correctly where Kimi K2.6 with Hermes answered incorrectly or produced no final answer.

  • Case 1: Membrane color: WeAgent-MMSearch-RL correctly identifies the membrane color in a Class II cytokine-receptor schematic.The membrane-embedded receptor portions are shown in green or greenish/teal.
  • The comparison cases connect visual answers with retrieved context about the relevant receptor schematic, Shui-script chart, and Canada Post stamp.
  • Case 2: Shui-script chart: Kimi K2.6 with Hermes incorrectly answers the Shui-script chart question, while WeAgent-MMSearch-RL gives quan (‘dog’) after ma (‘horse’).The chart sequence places quan after ma in the third column.
  • Case 3: Canada Picture Postage: Kimi K2.6 with Hermes produces no final answer for the 2015 Canada Picture Postage question.The model hypothesizes CANADA and a Permanent P mark, then stops after another search call without boxing a response.
  • Case 3: Canada Picture Postage: WeAgent-MMSearch-RL correctly identifies the 2005 Canada–China issue, the 2015 cougar Picture Postage, and the fixed text and postage symbol.The vertical text is CANADA, and the lower-right symbol is a maple leaf containing P for the Permanent domestic rate.

E Qualitative RL Response Cases

The qualitative rollouts illustrate multimodal search trajectories that identify entities, retrieve supporting evidence, inspect images, and answer visual questions.

  • Case 1: Cross-event athlete retrieval and suit-color reading: The racing suit in the selected celebration photograph is described as light blue and dark purple.The reference answer gives the two main colors as light blue and dark purple.
  • Case 1: Cross-event athlete retrieval and suit-color reading: A reverse-image search identifies Max Langenhan and the Milano Cortina 2026 men’s singles luge event.
  • Case 1: Cross-event athlete retrieval and suit-color reading: The rollout uses image inspection to identify visual details that search-result text does not specifically mention.The suit colors are inferred from the retrieved images rather than from textual search results.
  • Case 1: Cross-event athlete retrieval and suit-color reading: The rollout then identifies Italy’s best-performing men’s singles luger as Dominik Fischnaller, who won bronze.
  • Other qualitative cases retrieve a scholar and receptor schematic, while the appendix presents tool-return previews with held-out target images.The examples include figure identification and visual interpretation across multimodal search tasks.
Loading 2608.28062v1…