PaperlayerView exact revision on arXiv

Source-linked AI summary

Adaptive-Group Process Rewards for GUI Agents

Zirui Shang, Yang Liu, Xin Shu, Zhi Gao, Xinxiao Wu, Lifeng Fan

arXiv:2610.17710v1cs.AI

TL;DR

Sparse outcome rewards make it difficult to assign credit to intermediate GUI actions whose value may depend on evidence beyond the immediate observation. The paper dynamically groups actions by subgoal and combines action- and group-level assessments into process rewards for GRPO. It reports consistent improvements in process-reward accuracy and task success rate across benchmarks and downstream RL training.

  • Problem

    Sparse outcome rewards assign a trajectory-level signal to intermediate actions, even though judging an action’s semantic purpose may require evidence from later observations and actions.

    Source: §1 · p. 2

  • Method

    The framework dynamically partitions trajectories into semantically coherent subgoal groups and combines action-level evaluations with refined group-level evaluations as process rewards for GRPO.

    Source: §1 · p. 3

  • Results

    The framework shows consistent improvements in process-reward accuracy and task success rate on process-reward benchmarks and downstream RL training.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The process rewards provide finer-grained credit assignment during GUI policy optimization.

    Source: §1 · p. 3

    The scoring reference is not a strict prior, and trajectories need not follow its route; alternative paths remain valid.

    Source: §3.3 · p. 5

Abstract

from arXiv · show

Training graphical user interface (GUI) agents with reinforcement learning suffers from sparse outcome rewards, leading to inefficient learning. Process rewards offer a promising direction for credit assignment to intermediate actions; however, accurately evaluating an action requires identifying evidence across long-horizon GUI trajectories, as its value may depend on information beyond its immediate state and observation. In this paper, we propose an adaptive-group process reward framework that dynamically partitions a GUI trajectory into semantically coherent subgoal groups, enabling process reward estimation at both the action and group levels. At the action level, a critic evaluates each action based on the visual changes and subsequent actions within the group for action correctness and contribution to the subgoal. At the group level, the framework aggregates action-level evaluations to assess subgoal completion and contribution to the overall task, and further refines these assessments by leveraging the context of all groups in the trajectory. The process reward of each action is obtained by combining its action-level evaluation with the refined group-level evaluation. We further augment GRPO's outcome advantage with the proposed process rewards to provide finer-grained guidance for GUI agent training. Experiments on process reward benchmarks and downstream RL training demonstrate the effectiveness of the proposed framework.

1 INTRODUCTION

Sparse outcome rewards make it difficult to assign credit across long GUI trajectories, while action value can depend on evidence beyond the immediate observation. The paper introduces adaptive subgoal groups and two-level process rewards, reporting consistent improvements on process-reward benchmarks and downstream RL training.

1 INTRODUCTION

  • Sparse outcome rewards assign the same trajectory-level signal to intermediate actions, while an action’s semantic purpose may only become clear from later observations and actions.This broader evidence need makes reliable process-reward construction challenging.Source: §1 · p. 2
  • Fixed windows can miss long-range evidence or include irrelevant context, while predefined milestones can under-credit valid paths that differ from the preset route.The paper argues process rewards should adapt their evidence scope and credit assignment to the executed trajectory.Source: §1 · p. 2
  • The framework consistently improves process-reward accuracy and task success rate on process-reward benchmarks and in downstream RL training.The downstream evaluation is on OSWorld.Source: §1 · p. 3
  • Adaptive grouping partitions executed GUI trajectories into semantically coherent subgoal groups, rather than using a fixed partition rule.The task instruction and agent responses guide the grouping.Source: §1 · pp. 2–3
  • The critic evaluates action execution and subgoal contribution, then combines action-level assessments with refined group-level assessments to produce each action’s reward.Group assessments account for subgoal completion, contribution to the task, and the context of all groups.Source: §1 · pp. 2–3

2 RELATED WORK

GUI agents have advanced through stronger models and reinforcement learning, but assigning credit to intermediate actions remains difficult with sparse outcome rewards. Existing GUI process-reward approaches use fixed local contexts or predefined task structures, while this work groups executed actions by subgoal and evaluates them at action and group levels.

2.1 GUI AGENTS

  • Reinforcement learning has become a central direction for GUI agents because it optimizes long-horizon interactions from environment feedback, but pipelines still rely heavily on sparse outcome rewards.The related work names DART, UI-Genie, Orcust, and ClawGUI as examples of GUI RL systems.Source: §2.1 · p. 3

2.2 PROCESS REWARD MODELING

  • Unlike reasoning and coding tasks with execution checks, GUI action value depends on visual changes, task requirements, and subsequent operations, so its semantic role must be inferred from trajectory context.The section groups existing GUI process-reward methods into fixed-window step rewards and milestone rewards.Source: §2.2 · p. 3
  • Prior methods judge actions within preset local contexts or task structures, whereas adaptive groups follow executed subgoals and combine execution and contribution assessments at action and group levels.The related work contrasts fixed-window methods and milestone methods with the proposed approach.Source: §2.2 · p. 4

3 METHOD

The method partitions executed GUI trajectories into subgoal groups, evaluates actions and groups with complementary evidence, and uses the resulting per-action process rewards to refine GRPO’s step-level learning signal.

3.1 PROBLEM FORMULATION

  • Each GUI task is represented as an instruction-conditioned trajectory, and process reward generation assigns a reward to every executed action for use in step-level GRPO optimization.The trajectory records observations, textual responses, and actions; the process-reward function partitions it before generating action- and group-level assessments.Source: §3.1 · p. 4

3.2 ADAPTIVE-GROUP TRAJECTORY PARTITIONING

  • The grouping critic uses the instruction and executed responses to form a contiguous, non-overlapping partition into hypothesized subgoals and their action-step indices.Groups are recovered from the executed trajectory rather than imposed as fixed milestones, so valid executions may produce different groups.Source: §3.2 · p. 4

3.3 ADAPTIVE-GROUP PROCESS REWARD GENERATION

  • An instruction-conditioned reference guides scoring, but is not a strict prior: the trajectory need not follow the preferred process or attain the target state if an alternative path satisfies the instruction.The reference includes the target state implied by the instruction and the preferred process for the group.Source: §3.3 · p. 5
  • The action critic scores a group’s actions together for execution and subgoal contribution, using later actions as context and mapping the judgments to scalar scores.For long, image-heavy groups, screenshots can be processed in successive windows while the group’s responses remain available as text.Source: §3.3 · pp. 5–6
  • The group critic assesses subgoal completion and task contribution, revisits those judgments using all groups’ context, and multiplies the refined group score by each action score.Both assessments are jointly necessary: an action’s process reward is high only when it is correctly executed and its group contributes to the task.Source: §3.3 · p. 6

3.4 POLICY OPTIMIZATION WITH PROCESS REWARDS

  • GRPO retains its standardized outcome advantage and adds a process advantage standardized over sampled steps, replacing the same outcome signal previously applied across every step.The outcome term is constant across a trajectory’s steps, so by itself it gives useful and redundant actions the same learning signal.Source: §3.4 · p. 6

4 EXPERIMENTS

Across process-reward benchmarks and OSWorld, the proposed framework outperforms comparison methods; ablations also favor adaptive grouping and the complete scoring and aggregation design.

4.1 EXPERIMENTAL SETUP

  • Evaluation spans CUARewardBench, CUAStepBench, and OSWorld’s Chrome, LibreOffice Impress, and Calc; OSWorld results include single-run and three-run mean success rates.The benchmarks assess step judgments against human labels, while OSWorld success is determined by execution-based scripts on the final desktop state.Source: §4.1 · p. 7

4.2 PROCESS REWARD EVALUATION

  • 81.3 OA leads CUARewardBench under both templates (77.7 GPT-5/OpenCUA; 74.9 GPT-5/SE-WSM), while CUAStepBench F1 is 39.9 versus 38.1 for SeekJudge-9B.On CUAStepBench, the framework also has the highest recall (57.2), at lower precision (30.6 vs. 44.0).Source: Table 1 · p. 7, Table 2 · p. 7, §4.2 · p. 8

4.3 REINFORCEMENT LEARNING ON OSWORLD

  • 40.6/34.5 OA (1-run/3-run mean) exceeds UI-TARS by 15.0/11.9 points and Outcome GRPO by 4.9/4.1, with highest success on all three applications.Ours also exceeds SeekJudge by 3.5/3.1 OA points (40.6/34.5 vs. 37.1/31.4).Source: Table 4 · p. 8, §4.3 · p. 9

4.4 ABLATION STUDY

  • 81.3 OA for adaptive grouping falls to 75.7 with all actions in one group, 73.7 with fixed windows, and 80.4 when screenshots guide grouping.The authors attribute the fixed-window drop to missed or irrelevant context, and the screenshot-guided drop to interference between visually similar screenshots from different subgoals.Source: §4.4 · p. 9, Table 5 · p. 9
  • 81.3 OA is highest; removing any scoring stage or replacing product aggregation with an equal sum lowers accuracy.The ablation compares removal of the reference, refinement, group-scoring, or action-scoring stage, as well as equal-sum aggregation.Source: §4.4 · p. 9, Table 6 · p. 9

5 CONCLUSION

The conclusion presents an adaptive-group process reward framework for GUI agents and reports its effectiveness in process-reward benchmarks and downstream RL training.

5 CONCLUSION

  • The framework dynamically partitions GUI trajectories into semantically coherent subgoal groups for process reward estimation at action and group levels.These process rewards are used to shape GRPO’s step-level advantage during policy optimization.Source: §5 · p. 10
  • Each action’s process reward combines its action-level evaluation with the refined group-level evaluation.Source: §5 · p. 10
  • Experiments on process reward benchmarks and downstream RL training demonstrate the framework’s effectiveness.Source: §5 · p. 10

A ADDITIONAL IMPLEMENTATION DETAILS

The appendix lists the score mapping and critic prompts used in the framework.

A ADDITIONAL IMPLEMENTATION DETAILS

A.1 SCORE MAPPING

This appendix describes critics and a shared label-to-scalar mapping for evaluating GUI actions and subgoal groups, including refinement of group-level status and task contribution using evidence across the trajectory while preserving step-level execution results.

A.1 SCORE MAPPING

  • Table 7 maps two labels to scalars in {0, 0.2, 0.4, 0.6, 0.8, 1.0} and is shared across action- and group-level scoring.The labels represent execution or subgoal status and subgoal or task contribution, respectively.Source: §A.1 · p. 13, Table 7 · p. 13

A.2 GROUPING CRITIC PROMPT

  • The grouping critic partitions the trajectory into contiguous subgoal groups that cover every step exactly once, in order, including a final DONE or FAIL step when present.It reads the user instruction and agent responses to produce the groups.Source: §A.2 · p. 13

A.3 REFERENCE CRITIC PROMPT

  • The reference critic uses task instructions and group history, current-group, and future-group context to specify target state, rules, and preferred process.It treats the group name as a hypothesis, grounds requirements in task rules, and does not accept an agent-proposed answer without verification.Source: §A.3 · p. 14

A.4 ACTION-LEVEL SCORING CRITIC PROMPT

  • The action critic scores target steps in sliding windows, using screenshots for the scored window and text for other group actions to judge execution and subgoal contribution.It uses later steps to assess an action’s usefulness and checks whether a finished() action appropriately claims completion.Source: §A.4 · pp. 14–15

A.5 GROUP-LEVEL SCORING CRITIC PROMPT

  • The group critic evaluates each group from action judgments, its reference, and prior summaries, separately assessing subgoal status and global-task contribution.It distinguishes whether the group realized its intended effect from whether its route usefully advances the task; wrong routes can be completed yet harmful.Source: §A.5 · pp. 16–17

A.6 REFINEMENT CRITIC PROMPT

  • After all groups are initially scored, the refinement critic revisits every group-level judgment using the full sequence of group conclusions.Source: §A.6 · pp. 17–18
  • Refinement uses earlier and later group results to assess dependencies and reuse, invalidation, repair, abandonment, or dead ends without blaming a correct group for an unrelated later failure.It refines group-level conclusions only; step-level execution results remain unchanged.Source: §A.6 · p. 18
  • Groups receive a refined subgoal status of Completed, Partial, Failed, or Uncertain according to achievement, meaningful but incomplete progress, invalid results, or insufficient cross-group evidence.Source: §A.6 · p. 18
  • Goal contributions are classified as Milestone, UsefulEfficient, UsefulInefficient, or Exploratory based on decisive task requirements, route efficiency, or diagnostic information that helps select a better route.Source: §A.6 · p. 18
  • A group is Redundant if harmless but not meaningfully used or needed, and Harmful if it follows a wrong route, corrupts required state, blocks progress, or leads later groups to build on invalid state.Source: §A.6 · p. 18

B.1 EFFICIENCY ANALYSIS

Asynchronous scoring limits the sampling-throughput reduction from adding the framework during OSWorld RL, compared with synchronous scoring.

B.1 EFFICIENCY ANALYSIS

  • 87.9 trajectories per hour with asynchronous scoring compares with 92.3 without the framework, while synchronous scoring with the framework reaches 60.3.Without the framework, throughput is 92.3 trajectories per hour under both settings.Source: §B.1 · p. 18, Table 8 · p. 19
  • Asynchronous scoring starts the next trajectory immediately and scores the completed one in the background.Source: §B.1 · p. 18, Table 8 · p. 19

C.1 ERROR ANALYSIS

Among 63 critic error steps on CUARewardBench, insufficient application knowledge was the largest disagreement category, alongside visual or interaction misreads, missed constraints, and overestimated group contribution.

C.1 ERROR ANALYSIS

  • 41.3% of the 63 CUARewardBench critic error steps involved insufficient application knowledge.The critic lacked the correct menu, setting, syntax, or operator knowledge.Source: §C.1 · p. 19, Table 9 · p. 19
  • 22.2% were visual or interaction misreads, while 19.0% involved missed instruction constraints.Examples include misreading a date or filter state, and overlooking requirements already specified in the task.Source: §C.1 · p. 19, Table 9 · p. 19
  • 17.5% of the errors overestimated a group's contribution to the task.One example is scoring a group that resized one image layer when the task required resizing the whole image.Source: §C.1 · p. 19, Table 9 · p. 19

C.2 CASE STUDIES

The case studies illustrate how adaptive groups score actions in relation to subgoals and trajectory context, including both useful partial progress and critic error. Figures 3–5 show action-level scores alongside group-level assessments.

C.2 CASE STUDIES

  • Figures 3–5 pair each screenshot and action with its action-level score, while colored boxes mark same-subgoal steps and show refined group scores and contribution.Source: §C.2 · p. 19
  • In the failed Do Not Track trajectory, opening Settings and Privacy and security remains useful, while the premature DONE scores 0.After Group 3 reveals the setting is not on the explored page, Group 4 is lowered to redundant; returning to Site settings is useful.Source: §C.2 · p. 19, Figure 3 · p. 20
  • In Calc, the failed first auto-fill attempt remains distinct from the successful retry, which becomes useful after “No. #” appears.The Activities misclick and missed fill handle are harmful; Esc is Useful-efficient, and the failed group remains exploratory while the column is empty.Source: §C.2 · p. 19, Figure 4 · p. 20
  • In the Bills-email failure case, the critic misreads 3/1/2024 as newer than 12/1/2024 and scores opening the AWS invoice 1.0.The authors identify this as a visual-judgment error involving a small UI detail that the critic model does not read reliably.Source: §C.2 · pp. 19–21, Figure 5 · p. 21

Related papers

Loading 2610.17710v1…