PaperlayerView exact revision on arXiv

Source-linked AI summary

Recon-Act: A Self-Evolving Multi-Agent Browser-Use System via Web Reconnaissance, Tool Generation, and Task Execution

Kaiwen He, Zhiwei Wang, Chenyi Zhuang, Jinjie Gu

arXiv:2509.21072v1cs.AI

TL;DR

Long-horizon browser tasks remain challenging because agents can sequence actions unreliably and rely on trial and error. Recon-Act uses reconnaissance and action teams to turn contrasting trajectories into generalized tools, and under Level 3 achieves state-of-the-art VisualWebArena performance while retaining human intervention for some roles.

  • Problem

    On real-world, multi-turn browser tasks, current agents still face brittle tool orchestration and trial-and-error, while dynamic planning can require substantial simulation.

    Source: §1 · p. 2

  • Method

    Recon-Act uses reconnaissance and action teams to analyze successful and failed trajectories, generate generalized tools, and iteratively feed tool and action outcomes into a closed-loop process.

    Source: §1 · p. 2, §3.1 · p. 5

  • Results

    36.48% overall success on VisualWebArena, surpassing the previous best by 2.74%, and setting a new state of the art on that benchmark.

    Source: §4.2 · p. 6

  • Takeaways & Limitations

    The paper concludes that Recon-Act can provide targeted assistance in unfamiliar environments and help agents complete general tasks more effectively.

    Source: §1 · p. 2

    Present learning depends heavily on constructed training data, particularly successful trajectories, making the process similar to supervised training.

    Source: §6 · p. 8

Abstract

from arXiv · show

Recent years, multimodal models have made remarkable strides and pave the way for intelligent browser use agents. However, when solving tasks on real world webpages in multi-turn, long-horizon trajectories, current agents still suffer from disordered action sequencing and excessive trial and error during execution. This paper introduces Recon-Act, a self-evolving multi-agent framework grounded in Reconnaissance-Action behavioral paradigm. The system comprises a Reconnaissance Team and an Action Team: the former conducts comparative analysis and tool generation, while the latter handles intent decomposition, tool orchestration, and execution. By contrasting the erroneous trajectories with successful ones, the Reconnaissance Team infers remedies, and abstracts them into a unified notion of generalized tools, either expressed as hints or as rule-based codes, and register to the tool archive in real time. The Action Team reinference the process empowered with these targeting tools, thus establishing a closed-loop training pipeline of data-tools-action-feedback. Following the 6 level implementation roadmap proposed in this work, we have currently reached Level 3 (with limited human-in-the-loop intervention). Leveraging generalized tools obtained through reconnaissance, Recon-Act substantially improves adaptability to unseen websites and solvability on long-horizon tasks, and achieves state-of-the-art performance on the challenging VisualWebArena dataset.

1 Introduction

Recon-Act addresses brittle tool orchestration and trial-and-error in long browser tasks with a self-evolving, two-team framework that turns trajectory feedback into generalized tools. Its Level 3 configuration achieves state-of-the-art performance on VisualWebArena, while retaining human intervention in two roles.

1 Introduction

  • The paper identifies brittle orchestration and trial-and-error in unfamiliar environments as persistent challenges for multi-turn, long-trajectory browser tasks.The introduction also notes that dynamic planning methods can require extensive simulation and produce long trajectories.Source: §1 · p. 2
  • Recon-Act formalizes reconnaissance as information exploration and distillation through a limited number of actions to guide execution on unfamiliar pages.The process targets task-relevant observations in information-dense browser environments.Source: §1 · p. 2
  • The framework contrasts positive and negative trajectories to derive feedback and evolve tools in a closed loop of data, tools, actions, and feedback.The Reconnaissance Team provides hints or dedicated tools to the Action Team.Source: §1 · p. 2
  • The system is organized into Reconnaissance and Action teams, with implementation staged across six levels; Level 3 uses human-operated Analyst and Tool Manager components.At Level 3, the Master, Execution Agent, and Coder are model-powered; Level 6 is described as an end-to-end model.Source: §1 · pp. 2–3
  • Recon-Act achieves state-of-the-art performance on the VisualWebArena dataset under its Level 3 configuration.The implementation retains human-in-the-loop intervention for the Analyst and Tool Manager.Source: §1 · p. 3

2 Related Work

Related work spans general GUI agents, dynamic planning, and tool learning or generation. Recon-Act is distinguished in the discussion by browser-focused reconnaissance and a tool-centric feedback loop.

2.1 GUI-Agents with browser-use ability

  • GUI-agent research covers browser use alongside desktop control, mobile workflows, and visual grounding, while Recon-Act emphasizes browser-specific observation spaces and observation-generation tools.The discussion describes this focus as a departure from broader GUI-agent paradigms.Source: §2.1 · p. 3

2.2 Dynamic Planning Methods

  • Dynamic planning methods generate and score candidate actions or plans, select or backtrack, and execute iteratively; Recon-Act instead uses targeted reconnaissance and generalized tools refined through contrastive analysis.The passage characterizes Recon-Act as complementary to advances in agentic reinforcement learning.Source: §2.2 · p. 3

2.3 Agent and Tool

  • Tool research distinguishes learning to select and use tools from generating them, while Figure 2 depicts Recon-Act's two-team workflow for registering tools and feeding trajectory assessments back into training.Prior work discussed includes tool-data construction, tool verification, tool invocation, and code generation.Source: §2.3 · pp. 3–4, Figure 2 · p. 4

3 Methodology

Recon-Act trains by using trajectory feedback to improve a shared toolset, then runs browser tasks with an Action Team that routes between hints, decision tools, and a fallback execution agent.

3 Methodology

  • During training, failed trajectories, successful examples, and browser context inform reconnaissance; generated or updated tools trigger further training-set inference until updates stop improving accuracy or the process otherwise terminates.The Reconnaissance Team is active during training, while inference uses the Action Team and its pretrained tools.Source: §3 · pp. 4–5

3.1 Reconnaissance Team

  • The Reconnaissance Team is trained on cold-start queries that the system cannot yet solve, using paired successful and failed trajectories to establish a feedback loop.The queries derive from browser-use needs and target common websites with some generalizability.Source: §3.1 · p. 5
  • An Analyst compares successful and erroneous trajectories step by step, uses reconnaissance tools to diagnose failures, and proposes remedies that a Coder implements as registered tools.Tools share a parameter schema and string output format to avoid per-task parameter customization.Source: §3.1 · p. 5

3.2 Action Team

  • The Action Team's Master selects tools from the query and browser context, while the Tool Manager adds or updates registered tools and the Execution Agent supplies fallback actions.Tool updates occur during training; inference uses hard-coded routing, with Hint tools supplying signals and Decision tools emitting authoritative actions.Source: §3.2 · p. 5
  • At inference, a Hint-mode tool is followed by the Execution Agent, whereas a Decision-mode tool returns its action directly before the browser state updates.The Master first interprets the initial query and browser context, then the router dispatches the selected tool.Source: §3.2 · p. 5

4 Experiments

The experiments evaluate Recon-Act on VisualWebArena and report the benchmark’s overall and subdomain results. The implementation uses a Level 3, human-in-the-loop configuration with five agents and eleven tools.

4.2 Main Results

  • 36.48% overall success on VisualWebArena surpasses Yu et al. (2025) by 2.74%, setting a new state of the art on this benchmark.Shopping reaches 39.27%, above the prior best of 32.30% by 6.97%; on Classifieds and Reddit, Recon-Act trails current baselines by 1.68% and 1.56%.Source: Table 1 · p. 6, §4.2 · p. 6

4.1 Datasets and Evaluation Metrics

  • VisualWebArena covers about 910 realistic tasks across classifieds, shopping, and Reddit, with evaluations including exact match, semantic equivalence, prohibited-content checks, and visual goal completion.The benchmark also uses mustinclude checks and SSIM-based fuzzy image matching.Source: §4.1 · p. 6

4.3 Implementation Details

  • The Level 3 implementation has five agents across two teams and eleven tools; the coder, master, and execution agent use large models, while the analyst and tool manager are human-driven.Tables 2 and 3 summarize the agents and tools; training uses fewer than ten manually authored examples per domain and omits random-walk exploration.Source: §4.3 · p. 6, Table 2 · p. 7, Table 3 · p. 7
  • The reconnaissance analyst is prompted to ground reasoning in solution procedures and favor direct “goto” navigation, while tools are tailored to each website’s characteristics.The coder receives a fixed input-output interface and basic code structure to improve the feasibility of generated tool code.Source: §4.3 · p. 7
  • Tool registration remains a human task because generated tools can be narrow, ambiguously named, and fragmented, requiring feature adjustments and merges.For example, a cheapest-item sorter may lack a branch for finding the most expensive item.Source: §4.3 · p. 7

5 Conclusion

Recon-Act combines reconnaissance and action teams in a tool-centric, self-evolving browser-interaction system. At Level 3, it achieves state-of-the-art VisualWebArena performance while retaining human–AI collaboration for the analyst and tool manager.

5 Conclusion

  • Recon-Act uses contrastive analysis of positive and negative trajectories to generate feedback in a data–tool–action–feedback evolutionary pipeline.Its reconnaissance operations distill salient information from dense pages using a small number of exploratory actions.Source: §5 · p. 8
  • The system currently reaches Level 3, retaining human–AI collaboration for the analyst and tool manager while other components use vision–language models.Source: §5 · p. 8
  • Under that configuration, Recon-Act sets a new state of the art on VisualWebArena for complex multi-turn tasks in unfamiliar environments.Source: §5 · p. 8

6 Limitations and Future Work

The paper identifies dependence on constructed training data, limited analyst and tool-manager capabilities, and narrow website coverage as remaining constraints. Future work targets more autonomous training, stronger reasoning and coding, and broader reconnaissance.

6 Limitations and Future Work

  • Current learning depends heavily on constructed training data, particularly successful trajectories, making the process similar to supervised training.The authors plan random-walk-style self-exploration to generate additional successful trajectories and make training-set construction more autonomous.Source: §6 · p. 8
  • Progress beyond Level 3 requires stronger analyst reasoning and tool-manager coding, which currently constrain reductions in human involvement.The authors propose targeted analytical-data training and improvements to tool registration and code branching.Source: §6 · p. 8
  • The reconnaissance module currently succeeds only on a fixed set of websites and has not generalized to a broader, more heterogeneous web environment.Source: §6 · p. 8

Related papers

Loading 2509.21072v1…