PaperlayerView exact revision on arXiv

Source-linked AI summary

SCA: Spatial Credit Assignment for Reinforcement Learning of GUI Agents

Shengtian Yang, Ziyu Xiong, Kaibing Yang, Guangfeng Cai, Yewen Li, Peng Jiang, Gai Kun, Qingpeng Cai, Lei Feng

arXiv:2609.36939v1cs.AI

TL;DR

Binary group rewards tie failed GUI clicks at different locations and offer no relative signal when every sampled click misses. SCA uses held-out coordinate–reward predictions for mixed-hit groups and target proximity for all-miss groups. It improves results across the reported GUI benchmarks, with strongest listed reinforcement-fine-tuning performance on ScreenSpot-Pro and most low-level metrics.

  • Problem

    Binary group credit treats spatially different failed clicks alike and gives all-miss groups no relative signal, despite their different distances from the target.

    Source: §1 · p. 1

  • Method

    SCA adjusts group-relative credit with held-out coordinate–reward residuals for mixed-hit groups and target proximity for all-miss groups.

    Source: §1 · p. 1, §4.1 · p. 4

  • Results

    SCA has the best listed reinforcement-fine-tuning results on ScreenSpot-Pro and ten of eleven low-level metrics; GUI-Act-Web GR is 89.0 ± 0.42, 1.58 percentage points above GUI-R1.

    Source: §1 · p. 2, §5.2 · p. 8

  • Takeaways & Limitations

    In the reported offline evaluation, SCA’s three-seed means exceed listed GUI-R1 point estimates on grounding and action-prediction suites, while the synthetic study measures update direction and error.

    Source: §7 · p. 10

    Evaluation uses one 3B backbone at N = 5 on fixed GUI states, and does not test online recovery from actions that change later observations.

    Source: §6 · p. 9

Abstract

from arXiv · show

GUI agents automate tasks on digital devices by grounding language instructions in visual interfaces. Existing group-relative reinforcement learning improves GUI action prediction by comparing the rewards of multiple responses sampled from the same GUI state. However, binary evaluation treats spatially different failed clicks as identical and provides no relative signal when all sampled clicks fail. To address these limitations, we propose Spatial Credit Assignment (SCA), which uses the screen coordinates of sampled clicks to refine group-relative credit. Specifically, SCA predicts each held-out response's reward from the other responses in groups containing both successes and failures, then uses the prediction residual to adjust credit. When all sampled clicks fail, SCA instead orders them by distance to the annotated target. These spatial references are used only to construct the training update; the deployed policy remains unchanged. We evaluate whether this correction improves the policy update itself by comparing its error and directional alignment with the exact return gradient in a controlled synthetic study. Across GUI grounding and offline action-prediction benchmarks, SCA improves grounding across professional domains and achieves the strongest results among reinforcement-fine-tuned models on most action-prediction metrics, with consistent gains across the reported GUI suites.

1 Introduction

Reward-only group credit cannot distinguish spatially different failed clicks and provides no relative signal when every sampled click misses. SCA uses within-group coordinates to refine credit in those cases and improves results across the reported GUI benchmarks.

1 Introduction

  • Binary reward-only credit ties failed clicks at different distances and directions, and gives all-miss groups zero advantage despite their spatial differences.This discards useful spatial evidence when successful clicks are rare.Source: §1 · p. 1
  • SCA-Residual uses mixed-hit coordinate–reward trends, while SCA-Prox ranks clicks by target proximity in all-miss groups; all-hit groups retain ordinary group credit.The spatial references are fitted within the current group and affect training only; inference uses the trained policy alone.Source: §1 · pp. 1–2
  • SCA achieves the best listed reinforcement-fine-tuning results on ScreenSpot-Pro and ten of eleven low-level metrics, with three-seed means of 26.1 across twelve ScreenSpot-Pro subcolumns, 80.5 ± 0.41 OmniAct-Desktop grounding, and 66.0 ± 0.52 GUI-Odyssey.Figure 1 summarizes the benchmark comparisons; the reported comparisons use SCA three-seed means and listed baseline scores.Source: §1 · p. 2, Figure 1 · p. 2
  • The paper formulates distinct mixed-hit and all-miss failure regimes and develops corresponding spatial-credit branches for grouped GUI reinforcement learning.It also validates the update through exact-gradient analysis and a three-seed evaluation across grounding and offline action-prediction benchmarks.Source: §1 · p. 2

2 Related Work

Prior work provides group-relative reward comparisons, spatial supervision, and benchmarks for GUI grounding and action prediction. SCA addresses the remaining tie between spatially distinct clicks with equal binary outcomes by using within-group coordinate–reward references.

2 Related Work

  • Group-relative methods construct credit from sampled rewards, while related approaches also use critics, analytic control variates, or factorized baselines.For GUI clicks, binary group comparisons still tie spatially distinct actions whenever their outcomes agree.Source: §2 · p. 2
  • SCA’s leave-one-out reference excludes the scored response’s reward from its fit and gate, and is evaluated at the sampled coordinate to retain action conditioning.The construction draws on the sample-splitting principle.Source: §2 · pp. 2–3
  • GUI spatial-supervision work includes Gaussian reward modeling and self-evolutionary reinforcement learning, while SCA uses target proximity for all-miss groups and coordinate–reward trends for mixed-hit groups.Distance alone cannot capture reward variation by direction among sampled actions.Source: §2 · p. 3
  • ScreenSpot and ScreenSpot-Pro evaluate grounding across interface types and professional domains, while GUI-Act, OmniAct, GUI-Odyssey, and AndroidControl cover action prediction and interaction settings.The evaluation focuses on fixed offline GUI states, using click coordinates and targets for grounding and action-type and step-success metrics for action prediction.Source: §2 · p. 3

3 Preliminaries

The paper frames GUI actions as structured predictions from a screenshot and instruction, evaluated deterministically on fixed states. Grouped rollouts compare candidate responses for one state, providing the setting in which spatial credit can refine reward-based training.

3 Preliminaries

  • SCA assigns spatial credit after action evaluation: Residual refines mixed-hit groups, Prox uses annotated-target distance for all-miss groups, and all-hit groups retain standard group advantage.The spatial references are discarded after the update, so deployment uses the trained policy alone.Source: §3 · p. 3
  • SCA returns detached scalar response credit for the clipped actor loss, while the token mask, likelihood ratio, clipping, and KL term follow the base objective.The KL term regularizes the policy against the reference model.Source: §3 · p. 3
  • The primary evaluator uses binary success indicators, while a dense-reward study combines Gaussian click reward and format validity as y_i = 0.8R_Gaussian,i + 0.2R_format,i.Binary indicators still route groups by regime; the denser response signal is used inside the mixed-hit residual fit.Source: §3 · p. 3
  • A GUI action combines an action type with arguments, such as click coordinates, text, or a scroll or navigation direction or target.The policy couples visual grounding with action-type and argument prediction.Source: §3 · p. 4
  • Each example contains a fixed GUI state, instruction, and annotated target, and a deterministic evaluator checks parsed action fields and click coordinates.The evaluator supplies binary success while preserving coordinates for spatial credit estimation.Source: §3 · p. 4
  • For one state, the policy samples N responses under the same screenshot and instruction, and relative credit compares candidate actions within that group rather than across trajectory time steps.Responses can share a binary outcome while pointing to different locations, and all-miss groups can still be ordered by target distance.Source: §3 · p. 4

4 Method

SCA routes grouped responses by reward regime, using held-out spatial reward predictions for mixed-hit groups and target proximity for all-miss groups. The resulting response-level credit refines ordinary group credit without changing the deployed policy or the base actor objective.

4 Method

  • The method assigns credit across responses sampled for one state, not across trajectory time steps, and then uses each response’s credit to weight its token sequence.Binary GRPO gives all misses in a mixed group the same advantage and gives an all-miss group zero advantage.Source: §4 · p. 4

4.1 Spatial Credit Assignment

  • Residual predicts reward at each held-out click from coordinate–reward trends fitted to the other responses, using four projections and Theil–Sen lines.At N = 5, each outer fit uses four points; holding out the scored response prevents its reward from determining its own reference value.Source: §4.1 · p. 4
  • Directions contribute to the mixed-hit trend when held-out predictions correlate positively with reward and reduce squared error versus the fold-local reward mean.Weights reflect error reduction, and predictive quality controls a spatial-weight increase from zero at q_i = 0.80 to one at q_i = 0.99.Source: §4.1 · p. 4
  • Residual credit is the standardized difference between observed reward and the reference predicted from other rollouts, blended with ordinary group advantage.Invalid predictions receive zero residual and weight; when all weights are zero, Residual returns the original group advantage.Source: §4.1 · p. 5
  • Prox supplies the all-miss branch by assigning higher credit to closer valid clicks, with near-equidistant misses receiving a small update.Its credit is scaled by the spread of proximity scores.Source: §4.1 · p. 5
  • Prox handles all-miss groups, Residual handles mixed-hit groups, and all-hit groups keep the original group advantage; invalid or degenerate spatial estimates fall back to that advantage.With dense reward, routing still uses binary indicators while the denser response signal supplies the Residual fit.Source: §4.1 · pp. 5–6
  • The scored response is excluded from its fit and gate, confining the added computation to small regressions over the rollout group rather than a learned critic or inference-time network.At N = 5, each fit uses only four training points.Source: §4.1 · p. 6
  • After routing, SCA standardizes one scalar credit per response and detaches it before weighting response tokens in the clipped objective.The evaluator, sampler, clipping rule, and KL term remain unchanged, allowing experiments to test the credit estimator itself.Source: §4.1 · p. 6

5 Experiments

Across GUI grounding and offline action-prediction benchmarks, SCA improves on listed GUI-R1 results in the reported comparisons. A controlled exact-gradient analysis finds lower update error with positive gradient alignment, while showing that error alone does not establish an unbiased update.

5.1 Implementation Details

  • Training starts from Qwen2.5-VL-3B-Instruct and uses five sampled responses per GUI state; evaluation covers professional grounding, web and desktop action prediction, and recorded cross-app mobile states.Benchmarks report action-type accuracy, click-grounding accuracy, and step success rate; SCA results are means and sample standard deviations across three seeds.Source: §5.1 · p. 6

5.2 Experimental Results

  • SCA exceeds GUI-R1 in all 16 grounding subcolumns and by 1.58, 1.90, and 2.13 percentage points on GUI-Act-Web, OmniAct-Web, and OmniAct-Desktop GR, respectively.SCA leads listed RFT scores in ten of eleven low-level metrics; UI-R1 has the highest RFT value on GUI-Odyssey.Source: §5.2 · pp. 6, 8, Table 1 · p. 7, Table 2 · p. 7

5.3 Analysis of the Main Results

  • SCA scores higher in all six ScreenSpot-Pro domains, with gains of 0.70–1.10 points; averaged across domains, text scores are 42.00 versus 40.77 and icon scores are 10.17 versus 9.68.The text and icon gains over GUI-R1 are 1.23 and 0.48 points, respectively, and icon grounding remains harder.Source: §5.3 · p. 8, Table 3 · p. 8

5.4 Exact-Gradient Analysis

  • On the fixed four-context mixture, SCA-Residual has gradient MSE 0.14825 and cosine 0.99985 to the exact mean gradient; Full SCA has 0.14754 and 0.99997.Residual reduces MSE by about 8.5% relative to GRPO while retaining positive alignment.Source: §5.4 · pp. 8–9, Table 4 · p. 9
  • SCA’s lower MSE accompanies positive alignment but does not imply an unbiased estimator: its norm ratios remain above one, while OLS-LOO has negative cosine despite lower MSE.The target-distance LOO control’s mean horizontal update is −0.01560, compared with the exact value 0.08143.Source: §5.4 · pp. 8–9
  • The spatial correction lowers MSE only when its interaction with the original estimation error offsets the correction’s squared magnitude; reward-prediction accuracy alone does not determine that balance.The exact-gradient comparison holds sampled actions fixed to test the resulting update rather than reward prediction alone.Source: §5.4 · p. 9

6 Limitations

The evaluation is limited to one 3B backbone, five responses per fixed GUI state, and target annotations for Prox. The reported gains do not isolate the two SCA branches, and online interaction and broader settings remain untested.

6 Limitations

  • The evaluation uses one 3B backbone with N = 5 on fixed GUI states, and Prox has access to target annotations.These settings bound the reported evaluation scope.Source: §6 · p. 9
  • The benchmark results measure the combined SCA rule, so gains cannot be attributed to Residual or Prox without matched branch and distance-control experiments.Gate sensitivity, reward-shaping interactions, and scaling across model sizes also remain open empirical questions.Source: §6 · p. 9
  • Gate calibration uses synthetic data, and the recorded-state evaluation does not test recovery from actions that change later observations in online environments.OSWorld, WebArena, and MiniWoB are named as examples of such environments.Source: §6 · p. 9

7 Conclusion

SCA assigns GUI click credit using neighboring rollout geometry, with separate signals for mixed-hit and all-miss groups. In offline evaluations it exceeds listed GUI-R1 point estimates, while branch controls, online interaction, and broader backbones remain future work.

7 Conclusion

  • SCA uses a cross-fitted residual for mixed-hit groups and target proximity for all-miss groups to assign credit from neighboring rollout geometry.The synthetic study measures update direction and error.Source: §7 · p. 10
  • Across the reported offline grounding and action-prediction suites, SCA’s three-seed means are above the listed GUI-R1 point estimates.Branch controls, online interaction, and broader backbones are left for future evaluation.Source: §7 · p. 10
  • The reported evaluation leaves the separate contributions of Residual and Prox untested through branch controls.The evaluation also does not cover online interaction or broader backbones.Source: §7 · p. 10

A.1 Training configuration and comparison sources

The reported SCA runs use a shared training configuration across three seeds, while baseline comparisons are descriptive. The appendices specify metric aggregation and the deterministic acceptance rules used for illustrated cases.

A.1 Training configuration and comparison sources

  • SCA headline runs use Qwen2.5-VL-3B-Instruct with five responses per GUI state and the same objective, prompt, evaluator, and optimizer family across seeds.The runtime correlation blend uses [ρlo, ρhi] = [0.3, 0.7]. Comparisons to published baseline point estimates are descriptive, without established paired test-set uncertainty or exact evaluator equivalence for every row.Source: §A.1 · p. 13

A.2 Metrics and aggregation

  • Type denotes action-type accuracy, GR click-grounding accuracy, and SR step success; GUI-Odyssey scores concern recorded-state predictions, not closed-loop task completion.ScreenSpot-Pro’s text mean is the unweighted mean of twelve domain–UI-type scores, while Figure 1’s ScreenSpot mean averages four displayed Web/Desktop Text/Icon scores.Source: §A.2 · p. 13
  • Text describes improvements as absolute percentage-point differences, and SCA sample standard deviations measure variation across training seeds rather than significance against baselines.Baseline rows are comparison point estimates; the reported differences are descriptive rather than paired significance tests.Source: §A.2 · p. 13

A.3 Recorded-case evaluator

  • Illustrated grounding cases count as successful when the normalized coordinate distance to the reference is below 0.142, while text uses token-set F1 of at least 0.5.Other actions use type match; these tests specify the acceptance rule for the illustrated cases.Source: §A.3 · p. 13

B Gate Calibration

The appendix describes synthetic calibration of a soft gate and a nested directional prediction procedure, then states properties and examples of the resulting spatial credit. The gate band is selected under constraints and passes a one-shot holdout on a new random draw from the same synthetic family.

B Gate Calibration

  • The N = 5 gate calibration compares candidate bands on 10,000 synthetic groups per scenario, spanning nulls, clean signals, and coordinate, label, and combined noise.N ∈ {8, 16} are retained as out-of-distribution stress settings.Source: B Gate Calibration · p. 13, Table 5 · p. 14
  • The constrained selection rule chooses [0.80, 0.99], and the selected band passes all blocking constraints in a one-shot holdout using a new random draw from the same synthetic family.The holdout was evaluated without reselection.Source: B Gate Calibration · pp. 13–14, Table 5 · p. 14

C.1 Directional prediction and nested selection

  • Directional candidates are fit on rollouts excluding the held-out action, with nested fits used to assess prediction skill; positive skills determine ensemble weights and the gate.When no candidate has positive skill, the rollout receives zero spatial weight.Source: §C.1 · p. 14

C.2 Properties and worked example

  • Under the fixed mixed-hit route, the held-out reward does not affect its own cross-fitted prediction, candidate weights, ensemble skill, or gate.The held-out reward is used later to form residual and group-relative credit.Source: §C.2 · p. 14
  • The spatial correction is action-conditioned because its prediction is evaluated at the sampled action, so excluding that action’s reward does not make the correction’s expected score term zero.The paper measures resulting direction, scale, and error in the synthetic setting.Source: §C.2 · p. 14
  • Residual uses sampled actions and response or routing channels without reading target geometry, whereas Prox and distance-based rules additionally use target geometry.The task evaluator supplies scalar rewards; spatial rules use additional listed quantities.Source: §C.2 · pp. 14–15, Table 6 · p. 15
  • In the five-click example, Residual assigns the first miss positive final credit because its observed reward of 0 exceeds the clipped linear reference of −0.423.The example contrasts with GRPO, which gives all three misses the same credit; the reference is not a success probability.Source: §C.2 · p. 15, Table 7 · p. 15

D Algorithm

  • The actor objective uses a KL term with λKL = 10−2, whose implemented low-variance estimator is a masked mean of a clipped expression.The KL term regularizes the policy against the reference model.Source: D Algorithm · p. 15

D.1 Numerical implementation

The implementation uses guarded mixed-hit residualization and a separate proximity path for all-miss groups. The synthetic audit compares sampled-group estimators with exact return gradients across a uniform contextual mixture.

D.1 Numerical implementation

  • Invalid coordinates, degenerate fits, or failed quality checks trigger zero skill or a fallback to ordinary group-relative credit.The mixed-hit Residual path also falls back when any coordinate in the calibrated N = 5 group is invalid; all-miss Prox uses its nonempty valid-coordinate subset.Source: §D.1 · p. 15
  • With D = 4 and N = 5, Residual performs 20 four-point outer fits and 80 three-point inner fits per mixed-hit group.Each outer fit considers at most six coordinate pairs, and each inner fit at most three.Source: §D.1 · p. 15

E.1 Synthetic setting

  • The synthetic audit uses a diagonal Gaussian policy and exact analytic gradients of binary target-box hit probability as its reference.It samples 100,000 groups per context at σ = 1 and N = 5, then compares a 400,000-group uniform contextual mixture with the arithmetic mean of four exact context gradients.Source: §E.1 · p. 16

E.2 Gradient error

The section relates SCA's correction to gradient error and tests estimator behavior under controlled reward and coordinate perturbations. The reported preference between fits varies with the perturbation setting.

E.2 Gradient error

  • Residual and GRPO updates on the same sampled group differ by a correction term X.The update is expressed using each response's policy score and credit; the synthetic mixture evaluates it against the exact return gradient.Source: §E.2 · pp. 16–17
  • Gradient MSE decreases only when the correction offsets its squared magnitude by opposing the baseline error strongly enough.This condition concerns the score-weighted, normalized update, not reward-fitting error alone.Source: §E.2 · p. 17

E.3 Controlled corruption

  • The corruption study perturbs one reward or coordinate in 50,000 groups per scenario and measures effects on the other rollouts.Table 9 describes estimated mean gradients and component-wise Monte Carlo standard errors on the uniform contextual mixture; its exact row has no sampling error.Source: §E.3 · p. 17, Table 9 · p. 17
  • For the balanced coordinate spike, OLS is stronger, while the median ensemble has a slightly smaller MSE increase for the sparse coordinate spike.The two estimators are close in reward-flip settings, and the preferred fit depends on the perturbation in these small-group settings.Source: Table 10 · p. 17, §E.3 · p. 17

F Prompt Templates and Recorded Cases

The appendix specifies shared GUI prompt templates, action parsing and verification, and recorded examples from grounding and action-prediction evaluations.

F.1 Prompt templates

  • The grounding and action-prediction benchmarks share a GUI-R1 instruction format, with the admissible action set varying by benchmark.The ScreenSpot and ScreenSpot-Pro user message includes the instruction and action history, with None for single-step cases.Source: F Prompt Templates and Recorded Cases · p. 17, §F.1 · p. 18
  • The templates request an action, a cursor point when clicking, and any required input text.The grounding template names click as its action; OmniAct uses the same message skeleton with a benchmark-specific action set.Source: §F.1 · p. 18
  • The listed action vocabularies differ by benchmark, including click for ScreenSpot, several web actions, and desktop actions.OmniAct-Web lists press_tab, moveto, rightclick, press_enter, scroll, click, press_down, hotkey, press_space, and doubleclick; OmniAct-Desktop lists click, moveto, and press_pgdn.Source: §F.1 · p. 18
  • The grounding script uses RUN1-R1 and OmniAct scripts use GUI-R1; non-coordinate actions receive sentinel point [-100,-100].Pointer actions use the benchmark screenshot coordinate frame, and SCA receives parsed coordinates and scalar rewards through the interface.Source: §F.1 · p. 18

F.2 Verifiable action fields

  • Evaluation checks an action's type and applicable argument, using accepted coordinates for pointer actions, token-set F1 ≥0.5 for text, and type match otherwise.Malformed outputs receive zero under the verifiable action-field definition.Source: §F.2 · p. 18

F.3 Recorded evaluation cases

  • The web-shopping example predicts a click at (1276, 114) against ground truth (1287, 115), and the evaluator accepts the coordinate.The orange ellipse marks the target and the blue cross the parsed SCA coordinate.Source: §F.3 · p. 19
  • The desktop map example predicts a satellite-view click at (2465, 37) against ground truth (2413, 39), which is accepted.The orange ellipse marks the target and the blue cross the parsed coordinate.Source: §F.3 · p. 20
  • The web double-click example predicts (1205, 14) against ground truth (1221, 14), and the evaluator accepts it.The cases span a web click, a high-resolution desktop click, and a double-click; rollout-level credit is illustrated separately.Source: §F.3 · p. 21

Related papers

Loading 2609.36939v1…