PaperlayerView exact revision on arXiv

Source-linked AI summary

Boosting Virtual Agent Learning and Reasoning: A Step-Wise, Multi-Dimensional, and Generalist Reward Model with Benchmark

Bingchen Miao, Yang Wu, Minghe Gao, Qifan Yu, Wendong Bu, Wenqiao Zhang, Yunfei Li, Siliang Tang, Tat-Seng Chua, Juncheng Li

arXiv:2503.18665v2cs.CV

TL;DR

Outcome-based rewards from human-annotated trajectories provide limited fine-grained process supervision for Generalist Virtual Agents. Similar combines five-dimensional step assessment, automatic step-wise annotation, and Triple-M training; experiments report SRMEval gains and applications in training and inference-time scaling.

  • Problem

    GVA training commonly relies on outcome-based rewards from human-annotated trajectories, leaving intermediate execution steps without fine-grained process supervision.

    Source: §1 · p. 1

  • Method

    Similar defines five action-assessment dimensions, uses MCTS-P to collect step-wise annotations, and trains a reward model with the Triple-M strategy.

    Source: §1 · pp. 1–2

  • Results

    On SRMEval, Similar-TM-Llama achieves an Avg score of 61.2 versus Similar-RL-Llama (53.9, a 13.5% improvement); Similar-RL-Llama improves 13.2% over baseline Llama-3.2-11B-Vision.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Similar provides fine-grained rewards for GVA training and can filter candidate actions at inference, where selecting more likely task-completing actions enhances accuracy and reduces time.

    Source: §1 · p. 2

    The framework depends on MCTS-P annotation quality, which may introduce biases or inaccuracies in complex or ambiguous tasks; scalability across environments and tasks remains incompletely explored.

    Source: Limitations and Future Investigation · p. 9

Abstract

from arXiv · show

The development of Generalist Virtual Agents (GVAs) has shown significant promise in autonomous task execution. However, current training paradigms face critical limitations, including reliance on outcome supervision and labor-intensive human annotations. To address these challenges, we propose Similar, a Step-Wise Multi-Dimensional Generalist Reward Model, which offers fine-grained signals for agent training and can choose better action for inference-time scaling. Specifically, we begin by systematically defining five dimensions for evaluating agent actions. Building on this framework, we design an MCTS-P algorithm to automatically collect and annotate step-wise, five-dimensional agent execution data. Using this data, we train Similar with the Triple-M strategy. Furthermore, we introduce the first benchmark in the virtual agent domain for step-wise, multi-dimensional reward model training and evaluation, named SRM. This benchmark consists of two components: SRMTrain, which serves as the training set for Similar, and SRMEval, a manually selected test set for evaluating the reward model. Experimental results demonstrate that Similar, through its step-wise, multi-dimensional assessment and synergistic gain, provides GVAs with effective intermediate signals during both training and inference-time scaling. The project is available at https://github.com/antgroup/Similar.

1. Introduction

The introduction frames outcome-only, human-annotated GVA training as lacking fine-grained process supervision and difficult to scale. It presents Similar as an automated step-wise, multi-dimensional reward model, alongside SRM and reported training and inference benefits.

1. Introduction

  • Existing outcome-based rewards focus on task success or final state, while step-wise process supervision and human annotation of trajectories are labor-intensive and difficult to scale.The introduction also identifies inference-time action selection as difficult to scale under result-based training with extensive human annotation.Source: §1 · p. 1
  • The approach defines five dimensions for assessing agent steps, uses MCTS-P to collect step-wise annotations, and trains Similar with the Triple-M strategy.The dimensions are Helpfulness, Odds of Success, Efficiency, Task Relevance, and Coherence.Source: §1 · pp. 2–3
  • SRM is introduced as the first GVA-domain benchmark for step-wise, multi-dimensional reward-model training and evaluation, with SRMTrain (78k) and curated SRMEval (32k).The benchmark comprises 110k automatically annotated data points.Source: §1 · p. 2
  • Similar provides fine-grained rewards during GVA training and can guide inference-time scaling by filtering candidate actions with search algorithms such as MCTS.The paper says selecting actions more likely to complete tasks enhances accuracy and reduces time.Source: §1 · pp. 2–3
  • Similar-TM-Llama achieves an Avg score of 61.2 on SRMEval versus Similar-RL-Llama (53.9, a 13.5% improvement); Similar-RL-Llama improves 13.2% over baseline Llama-3.2-11B-Vision.The paper attributes the improvement to the Triple-M strategy and reports the baseline comparison using Similar-RL-Llama.Source: §1 · p. 2

2. Related Work

Prior virtual-agent fine-tuning methods face challenges from human-annotation dependence and limited process supervision. Reward-model research contrasts outcome and process feedback, while this work uses MCTS to collect fine-grained annotations.

2.1. Fine-Tuning Virtual Agent

  • Human-annotated data and missing process supervision constrain the scalability and adaptability of virtual-agent fine-tuning methods.Imitation learning and reinforcement learning use curated expert trajectories or outcome rewards; RFT and DPO also face these challenges.Source: §2.1 · p. 3

2.2. Reward Models for Virtual Agent

  • Process Reward Models assess intermediate steps, and studies report they outperform Outcome Reward Models on tasks such as math reasoning.Outcome Reward Models focus on task success; generating high-quality process supervision remains challenging because human annotation is expensive.Source: §2.2 · p. 3
  • Prior work has used MCTS for automated data collection, and this work applies it to gather fine-grained, multi-dimensional agent-action annotations.The cited approaches include step-level Q-value and ReST-MCTS∗.Source: §2.2 · p. 3

3. Method

Similar’s method defines five dimensions for assessing agent steps, uses MCTS-P to collect step-wise annotations, and trains a reward model with Triple-M.

3.1. Five-Dimensional Process Supervision Framework

  • The framework assesses steps through Helpfulness, Odds of Success, Efficiency, Task Relevance, and Coherence; the first three are automatic and the last two use MLLMs.Helpfulness reflects contribution to completion, Odds of Success measures successful-path likelihood, and Efficiency reflects resource use through step count.Source: §3.1 · pp. 3–5

3.2. Automatic Generalist Dataset Collecting

  • MCTS-P uses the five-dimensional scores to select and update search-tree nodes while collecting annotated trajectories across Web, Android, Linux, and Windows.Its data-collection procedure simulates possible outcomes, computes three dimensions automatically, and uses an MLLM to assess Task Relevance and Coherence.Source: §3.2 · p. 5

3.3. Triple-M Strategy for RM Training

  • Triple-M predicts five-dimensional rewards from the final hidden state, then uses prompt-aware gating to combine them into a scalar score.The gating network is trained with Bradley-Terry loss while the backbone and regression layer remain frozen.Source: §3.3 · pp. 5–6

4. SRM Benchmark

SRM is a multimodal, multi-dimensional, multi-platform benchmark for virtual-agent reward models, with separate training and manually annotated evaluation sets. It evaluates whether a model can select the better action in a pair for a specified dimension.

Dataset construction

  • SRM comprises 110k preference pairs collected across WebArena, VisualWebArena, Android World, and OSWorld, with human experts checking sampled pairs.GPT-4o-1120 generated 10k trajectories; 70% of benchmark task data was used for collection and 30% reserved for evaluation to avoid overlap.Source: §4 · p. 6
  • SRMTrain contains 78k training points, while SRMEval contains 32k manually annotated test points with tasks distinct from training.SRMEval includes step-level dimensions, an integrated total score, and a trajectory-level score averaging step totals.Source: §4 · p. 6

Evaluation task

  • The benchmark defines action-pair selection at step i for a specified dimension, measured by accuracy under each evaluation type.SRM is presented as a benchmark built from multimodal, multi-dimensional, and multi-platform annotated data.Source: §4 · p. 6

5. Experiments

The experiments assess Similar as a reward model on SRMEval and as guidance for agent training and inference-time search. Similar-TM generally outperforms Similar-RL, while inference-time scaling gains level off beyond eight simulated child nodes.

5.1. Experimental Setup

  • The setup compares prompted reward-model baselines and SRMTrain-trained Similar-RL and Similar-TM models, evaluating preference alignment and agent performance in training and inference settings.Training evaluations use WebArena and Android World; inference evaluations use Android World and OSWorld, with 30% of benchmark examples reserved for evaluation.Source: §5.1 · pp. 6–7

5.2. Effective Alignment of Preference

  • 61.2 Avg for Similar-TM-Llama versus 53.9 for Similar-RL-Llama (↑13.5%) on SRMEval; Similar-TM-Qwen showed similar performance.Similar-RL-Llama scored 53.9 Avg versus 47.6 for Llama-3.2-11B-Vision-Instruct (↑13.2%), and exceeded GPT-4o (51.4) and GPT-4-Turbo (46.6).Source: §5.2 · p. 7

5.3. Similar for RL Training

  • With OS-Atlas, Similar-TM-Llama improved task success over Similar-RL-Llama by 3.8% on Android World (33.6 →34.9) and 29.9% on WebArena (27.4 →35.6).Similar-RL-Llama also exceeded the original OS-Atlas and Llama-3.2V reward-model settings on both environments.Source: §5.3 · pp. 7–8, Table 2 · p. 7

5.4. Similar for Inference-Time Scaling

  • With GPT-4o, Similar-TM-Llama improved task success over Similar-RL-Llama by 16.8% on Android World (29.6 →34.6) and 25.9% on OSWorld (13.1 →16.5).Similar-RL-Llama also outperformed the original agent and MLLM-reward setting; scaling helped when N ≤8, but performance plateaued or declined for N > 8.Source: Table 3 · p. 7, §5.4 · p. 8, Figure 6 · p. 8

5.5. Indepth Analysis

  • On Android World, excluding Helpfulness, Odds of Success, Efficiency, Task Relevance, or Coherence reduced performance by 6.9%, 4.4%, 3.5%, 2.0%, or 0.8%, respectively.The dimension correlations were all below 0.47; the authors interpret this as indicating independence despite some correlation.Source: §5.5 · pp. 8–9

6. Conclusion

The paper presents Similar as a step-wise, multi-dimensional reward model for guiding virtual-agent training and inference, alongside SRM as a reward-model benchmark.

6. Conclusion

  • Similar provides step-wise, multi-dimensional feedback during GVA training and inference, and the authors report superior SRMEval performance and effectiveness across diverse tasks.The paper also introduces SRM as its reward-model evaluation benchmark.Source: §6 · p. 9

Impact Statement

The impact discussion describes Similar’s potential uses in agent training, inference, and data cleaning, while noting limits in annotation reliability and evaluation coverage. The accompanying materials detail the benchmark prompts and model pipeline.

Limitations and Future Investigation

  • The framework depends on MCTS-P annotation quality, which may introduce bias or inaccuracies in complex or ambiguous tasks; niche and specialized domains remain untested.Experiments cover Web, Android, Linux, and Windows; broader evaluation on less structured or dynamic interfaces is left for future work.Source: Limitations and Future Investigation · p. 9

Impact on RL Training

  • The paper describes Similar’s fine-grained feedback as helping agents identify and correct intermediate errors, supporting faster convergence and improved task performance.It also attributes more robust policies and less reliance on labor-intensive annotation to the Triple-M strategy’s integration of multiple assessment dimensions and experts.Source: Impact on RL Training · p. 9

Impact on Inference-Time Scaling

  • The inference discussion says Similar supplies multi-dimensional rewards to evaluate and filter candidate actions in MCTS, with experiments reporting gains that plateau beyond a threshold.The reported plateau is attributed to agent-model limitations when simulating additional actions.Source: Impact on Inference-Time Scaling · p. 9

Impact on Data Cleaning

  • The paper describes using Similar’s feedback to identify and remove low-quality or noisy data points, including irrelevant or incoherent actions, with less human intervention.The discussion presents this as useful where datasets are large and manual inspection is impractical.Source: Impact on Data Cleaning · p. 9

Appendix; Overview

  • The appendix overview lists prompts, human evaluation details, case studies, ablations, and additional SRMEval visualizations.It also points to pseudocode for the Similar pipeline.Source: Overview · p. 14

A. Detailed Prompt Design; A.1. Prompt for testing general MLLMs on SRMEval

  • The SRMEval prompt defines Helpfulness by contribution to task completion, Odds of Success by completion potential, and Efficiency by change in remaining steps relative to total task steps.It also defines Task Relevance and Coherence as scores for relation to the instruction and compactness with the previous step, respectively.Source: §A.1 · pp. 15–16

B. Similar Pipeline Pseudocode; B.1. Five-Dimensional Process Supervision Framework

  • The Similar pipeline combines five-dimensional step supervision, automatic generalist dataset collection, and Triple-M reward-model training.The framework computes Helpfulness, Odds of Success, and Efficiency automatically, while MLLMs assess Task Relevance and Coherence.Source: B. Similar Pipeline Pseudocode · p. 16

B.2. Automatic Generalist Dataset Collecting

The dataset-collection and training sections describe MCTS-P annotation across platforms and a Triple-M strategy combining multiple kinds of data.

B.2. Automatic Generalist Dataset Collecting

  • MCTS-P collects annotated step-wise data across Web, Android, Linux, and Windows.Source: §B.2 · p. 17

B.3. Triple-M Strategy for Reward Model Training

  • Triple-M integrates multi-step, multi-dimensional, and multi-modal data for reward-model training.Source: §B.3 · p. 17
  • Triple-M training has two stages: regression-layer optimization followed by dynamic-gating-network adjustment.Source: §B.3 · p. 17

C. Human Evaluation Details and Human Acceptance of SRM

Human acceptance exceeds 78.8% across the five dimensions, while the examples illustrate how Helpfulness, Odds of Success, and Efficiency distinguish candidate actions.

C. Human Evaluation Details and Human Acceptance of SRM

D.1. Helpfulness

  • For Helpfulness, deleting an incorrect timer input is scored above entering another incorrect value.Source: §D.1 · p. 18

D.2. Odds of Success

  • For Odds of Success, navigating back to search for an unavailable delete option outranks importing a calendar file.Source: §D.2 · p. 18

D.3. Efficiency

  • For Efficiency, clicking a displayed Calendar search-history item scores higher than re-entering the app name, though both actions are valid.Source: §D.3 · p. 19

D.4. Task Relevance

The examples distinguish Task Relevance from Coherence, and the extended ablation examines the five dimensions across Android World, WebArena, and OSWorld.

D.4. Task Relevance

  • In the recipe task, clicking Cook is more task-relevant than navigating back after recipe creation.Source: §D.4 · p. 20

D.5. Coherence

  • In the product-search trajectory, following the Herbs, Spices & Seasonings link coherently advances the task, unlike repeating the earlier menu action.Source: §D.5 · p. 20

E. Comprehensive Ablation Experiments

E.1. Impact of Individual Dimensions

Combining dimensions, especially Helpfulness, Odds of Success, and Efficiency, yields higher reported success rates than using individual dimensions; Helpfulness alone also has the strongest individual impact.

E.1. Impact of Individual Dimensions

  • Adding H alone raises success rates from 30.7% to 32.5% on Android World, 20.6% to 26.1% on WebArena, and 14.6% to 15.8% on OSWorld.Source: §E.1 · p. 20

E.2. Combined Impact of Multiple Dimensions

  • 34.3%, 35.9%, and 17.2% success rates are reported for H+OS+E on Android World, WebArena, and OSWorld, respectively, exceeding the H+OS combination on each benchmark.The H+OS rates are 33.4%, 31.4%, and 16.7%, respectively.Source: Table 6 · p. 21, §E.2 · p. 21

E.1. Impact of Individual Dimensions

  • Odds of Success (OS) improves by 31.9%, 24.7%, and 15.4% on the respective benchmarks, indicating its importance in guiding the agent; E contributes positively to a lesser extent, while TR and C show more modest improvements.Source: §E.1 · p. 21

F. More Visualizations of SRMEval

Table 7 presents SRMEval cases, illustrating a benchmark designed to evaluate reward models’ assessment of agent-action quality and preference alignment.

F. More Visualizations of SRMEval

Loading 2503.18665v2…