PaperlayerView exact revision on arXiv

Source-linked AI summary

Guides That Cause Actions: An Offline Study of Guide-Action Mutual Reinforcement in Multimodal Web Agents

Chengguang Gan, Yunhao Liang, QingHao Zhang, Shiwen Ni

arXiv:2609.27353v1cs.CL

TL;DR

Live environment and judge drift make controlled studies of web-agent training difficult. WebMRE provides an audited offline benchmark and deterministic protocol for measuring guide–action interaction. Joint guide–action training improves element selection across both model sizes and decoding orders, with gains growing with scale.

  • Problem

    Live environment state and judge-model drift make repeated scores unstable, limiting controlled comparisons of web-agent training choices.

    Source: §1 · p. 1

  • Method

    WebMRE pairs human-oriented guide sentences with grounded actions in an audited corpus and uses a deterministic offline protocol to study their interaction.

    Source: §1 · pp. 1–2, §6 · p. 8

  • Results

    0.9 points at 4B and 1.7 points at 9B in action-first decoding; joint guide–action training improves element selection in both models and decoding orders, with gains growing by scale.

    Source: §1 · p. 2

  • Takeaways & Limitations

    An offline reward can credit guides according to the actions they induce, but optimizing it brings no gain yet.

    Source: §6 · p. 8

    WebMRE derives from one benchmark family, so site and layout diversity is bounded by WebArena, and its offline step metrics complement rather than replace live end-to-end evaluation.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Web agents are usually evaluated in live environments, where environment state and judge models drift between runs, so the same checkpoint rarely reproduces the same score, making controlled studies of training phenomena impractical. We present WebMRE, an offline benchmark of 541 tasks and 5,293 steps derived from successful WebArena trajectories, with fully audited test labels and a deterministic protocol that scores a checkpoint identically on every run without any environment. Each step pairs a human oriented guide sentence with a grounded action, enabling the first study of the mutual reinforcement effect between them in web agents. Averaged over three seeds the effect holds for both models in both decoding orders and grows with scale: jointly decoding a guide lifts element selection over an action only reference by 0.9 and 0.2 points for Qwen3.5-4B and by 1.7 and 2.2 points for Qwen3.5-9B. A mediation analysis shows that the guide is a causal channel rather than commentary: forcing the gold guide as a decoding prefix lifts action accuracy from .422 to .684, another step's guide collapses it to .055, and a paraphrase that renames the target still recovers half of the gain, so the channel carries instruction meaning and not only the label string. The same channel yields an offline reward that only a replayable protocol makes computable, though optimizing it from a strong checkpoint brings no gain yet. Our fine tuned models outperform GPT-5.5, Claude Opus 4.8, and Gemini 3.5 Flash, run zero shot, on every offline metric.

1 Introduction

Live environment drift makes controlled studies of web-agent training difficult. This paper introduces WebMRE and uses it to measure guide–action reinforcement, test how guides influence actions, and construct an offline reward.

1 Introduction

  • Live evaluation is useful for end-to-end competence but unstable for training comparisons: environment and judge drift produced up to a twenty-point score difference across two deployments.The authors argue that this noise obstructs controlled comparisons when single-digit differences matter.Source: §1 · p. 1
  • Each web-agent step takes a screenshot, task intent, and interaction history, then produces a human-oriented guide and an action grounded in an element or pixel coordinates.The task is studied on a deliberately offline footing.Source: §1 · p. 1
  • 0.9 and 1.7 points in action-first decoding for Qwen3.5-4B and Qwen3.5-9B, respectively; joint guide–action training improves element selection in both models and both decoding orders, with gains growing by scale.Guide-first decoding adds 0.2 points at 4B and 2.2 points at 9B over the action-only reference.Source: §1 · p. 2
  • Forcing the gold guide raises action accuracy to .684, while another step’s guide lowers it to .055; a paraphrase control recovers half the gold gain.The paraphrase renames the target by function, indicating that the channel carries instruction meaning, not only surface-label overlap.Source: §1 · p. 2
  • WebMRE pairs guide sentences with grounded actions in 541 tasks and 5,293 steps, and uses audited labels and deterministic scoring without an environment.The protocol uses greedy decoding and rule-based matching, so the same checkpoint yields the same score across runs.Source: §1 · p. 2
  • Mediated-Guide GRPO rewards a guide according to the action it induces, but optimizing it from a strong supervised checkpoint brings no gain yet.The same offline protocol also finds that fine-tuned models outperform zero-shot GPT-5.5, Claude Opus 4.8, and Gemini 3.5 Flash on every offline metric.Source: §1 · p. 2

2 Related Work

Prior work offers live web benchmarks, offline action corpora, text-augmented GUI training, and action-focused reinforcement learning. This paper distinguishes its contribution by pairing human-oriented guides with actions and measuring how the two forms of supervision interact.

2 Related Work

  • WebMRE pairs every action with a human-oriented guide and treats label reliability as an object of study, addressing omissions in existing offline action corpora.The related-work discussion contrasts live benchmarks with offline corpora and tutorial-derived trajectories.Source: §2 · pp. 2–3
  • Prior GUI pipelines add instruction-like text as auxiliary supervision, while this work measures whether it helps, in which direction, and under which decoding order across seeds.The comparison also separates effects that survive seed variation from those that do not.Source: §2 · p. 3
  • Unlike prior reinforcement-learning approaches that reward actions or trajectories, Mediated-Guide GRPO prices a guide by the action it causes, without an environment or learned judge.The paper positions this distinction alongside live environment feedback and rule-based action rewards.Source: §2 · p. 3
  • The paper brings the mutual reinforcement effect, previously studied in information extraction, to web guide agents and examines its mechanism with an offline, repeatable evaluation.Its stated object of study differs from the benchmark and training lines it reviews.Source: §2 · p. 3

3 The WebMRE Dataset and Offline Protocol

The section builds WebMRE by unifying WebArena trajectories, auditing noisy element labels, and defining a deterministic offline protocol for scoring web-agent outputs.

3.1 Provenance and Unification

  • WebMRE unifies successful WebArena trajectories into 541 tasks and 5,293 steps, restoring 859 coordinate-only steps and flagging recaptured observations as stale in 59% of steps.Each record pairs task intent and a guide sentence with an action grounded by element id and pixel coordinates; the two views share tasks, splits, screenshots, and guides but differ in action grounding and element-id availability.Source: §3.1 · pp. 3–4, Figure 3 · p. 4, Table 1 · p. 4

3.2 Three Judge Label Audit

  • Three-judge audits remapped 225 of 465 audited steps, marked 128 unmappable, confirmed 54, and disputed 58; 51 disputed test steps are excluded from element-level scoring.The audit covers flagged test steps and the first-round pool, while flagged training steps remain unaudited, leaving shared residual label noise across training conditions.Source: §3.2 · p. 4

3.3 Task Suite and Offline Metrics

  • Experiments compare four output contracts—action only, guide only, action then guide, and guide then action—using the SoM view.All conditions receive the same screenshot, task intent, previous-step guides, and candidate menu; only the required output keys vary.Source: §3.3 · pp. 4–5
  • Scoring combines a format gate, action-type and target matching, typed-text token F1, and guide BLEU, ROUGE, and judge-rated usefulness.Element-level terms exclude disputed labels; target matching depends on the grounding view, and usefulness is the majority share of guides rated useful by three judges.Source: §3.3 · p. 5
  • Every score is a pure function of greedy model output, so repeated evaluations of the same checkpoint return the same score vector without an environment or judge in the action metrics.A full test pass costs under one GPU hour and scoring is immediate.Source: §3.3 · p. 5

4 Mutual Reinforcement Effect in the Web Guide Agent Task

Across three-seed fine-tuning of Qwen3.5-4B and Qwen3.5-9B under action-only, guide-only, and joint output contracts, joint guide-action training improves element selection in both decoding orders. Robustness checks show where the gains hold across sites, action types, stale screenshots, and unseen intent templates.

4.1 Setup

  • Table 2 reports SoM element selection, BLEU-1 guide scores, and three-seed means for fine-tuned rows across action-only, guide-only, and joint contracts.The 4B and 9B models use the same four output contracts; frontier models run zero-shot under J.Source: Table 2 · p. 5, §4.1 · p. 5

4.2 Results

  • At 9B, joint contracts remain close to guide-only G on guide metrics: J and JG score .551 and .558 BLEU-1 against .563 for G.Order effects differ by scale: JG has the better guide score at 9B, while J is better on both guide and action at 4B.Source: §4.2 · pp. 5–6
  • Fine-tuned models beat the best zero-shot row by 17 action points and 20 BLEU-1 points on guides.Format compliance is not the differentiator: OFCR is .999 for action-only and .984 to .991 for joint contracts.Source: §4.2 · p. 6
  • +0.9 and +0.2 points at 4B, and +1.7 and +2.2 points at 9B: joint contracts J and JG raise SoM element-selection SAA over action-only A.The values are mean gains over three training seeds; J is action first and JG guide first.Source: §4.2 · p. 5, Table 3 · p. 7

4.3 Robustness

  • A joint contract is the best row on five of six sites for each model; the exceptions are reddit at 4B and shopping at 9B.The administrative shop and gitlab also improve at 9B, from .262 to .292 and .482 to .523, respectively.Source: §4.3 · p. 6, Table 4 · p. 7
  • At 9B, the best joint contract adds 3.3 points on clicks and 2.2 on typing, while finish accuracy remains .363 across all contracts.The gains concentrate on actions requiring target identification; scrolling gains 1.8 points as well.Source: §4.3 · p. 7
  • Both joint contracts gain on stale-screenshot steps at both sizes, and unflagged steps stay at or above action-only except for 4B JG.At 9B, action-only accuracy is .372 on flagged steps versus .472 on unflagged steps.Source: §4.3 · p. 7, Table 5 · p. 7
  • The joint gain persists on 15 test tasks with intent templates absent from training, rather than only on familiar guide-action pairs.Source: §4.3 · p. 7

5 Analysis of Guides as a Causal Channel

Forced-guide interventions show that the guide acts as an instruction channel for action prediction, not merely accompanying commentary. Gold guides substantially improve action accuracy, while shuffled guides impair it; paraphrases retain part of the gain.

5 Analysis of Guides as a Causal Channel

  • Forcing the gold guide raises action accuracy from .422 to .684, while a different step’s guide drops it to .055; the renamed-target paraphrase reaches .554.The intervention uses all 1,220 SoM test steps with the Qwen3.5-9B guide-first model; forcing its own guide reproduces free decoding at .422.Source: §5 · pp. 7–8, Table 6 · p. 8
  • Gold guides receive 81.5% usefulness ratings, while generated guides score 51.0% for the fine-tuned model and 41.4% for GPT-5.5.The judges’ pairwise agreement is .93 to .97; half of the fine-tuned model’s generated guides still fail the usefulness rating.Source: §5 · p. 8
  • The gap between .422 free decoding and .684 with gold guides points to guide generation, rather than grounding, as the binding constraint.The paraphrase result indicates that the headroom concerns instruction content, not only surface form.Source: §5 · pp. 7–8

6 Conclusion

An offline deterministic protocol makes guide-action mutual reinforcement measurable in web agents, and joint guide-action training improves element selection across both model sizes and decoding orders. Forced-prefix results identify the guide as a causal channel, and the same channel supports an offline reward that has not improved an already strong supervised checkpoint.

6 Conclusion

  • Joint guide-action training improves element selection for Qwen3.5-4B and Qwen3.5-9B in both decoding orders, with larger gains at 9B.Source: §6 · p. 8
  • A forced-prefix intervention with a paraphrase control shows that the model executes the guide it receives, making the guide a causal channel rather than commentary.Source: §6 · p. 8
  • Mediated guide rewards are computable offline, but optimizing them has not yet improved an already strong supervised checkpoint.Source: §6 · p. 8

Limitations

WebMRE has bounded site and layout diversity because it derives from one benchmark family, and offline step metrics complement rather than replace live end-to-end evaluation. The appendix also documents a two-round reinforcement-learning study and the prompts used for model and judge runs.

Limitations

  • WebMRE's single benchmark family bounds site and layout diversity, and offline step metrics complement rather than replace live end-to-end evaluation.The reinforcement-learning study used a deliberately small budget of two rounds; extending the corpus and protocol is left to future work.Source: Limitations · p. 9

A Prompt Templates

  • Figures A1–A3 document the locked prompts, contract and view differences, and user-message template used in the study.The templates are reproduced verbatim, with the output-contract paragraph and grounding key as the only degrees of freedom.Source: A Prompt Templates · p. 10

B Audit and Judge Prompts

  • Figures A4–A7 reproduce the label-audit, guide-rewrite, blind acceptance, and guide-usefulness judge prompts.Each judge sees the screenshot, answers in strict JSON, and receives the same prompt across the three vendors.Source: B Audit and Judge Prompts · p. 10

C Worked Examples

The worked examples show six test steps, one from each site, pairing the model’s SoM screenshot and input text with its output and audited gold action. Four examples are solved and two fail.

C Worked Examples

  • Four of six worked test steps are solved; the two failures pair a wrong guide with a wrong action.The examples show the SoM screenshot, text input, and fine-tuned 4B JG output against the audited gold action.Source: C Worked Examples · p. 10

D Dataset Statistics

The appendix reports corpus composition and distributions for the unified WebMRE dataset. Guides and candidate menus are short and large, respectively, while task lengths vary substantially.

D Dataset Statistics

  • Tables A1–A4 provide corpus composition, and Figure A14 shows distributions behind corpus-level means.Table A1 breaks down train and test task counts by site; each task contributes steps to both views.Source: D Dataset Statistics · p. 10, Table A1 · p. 10
  • Table A2 reports gold action-type distributions by view and split, with test-split percentages in parentheses.Source: Table A2 · p. 10
  • Table A3 reports text and structure statistics for 5,293 steps and 541 tasks, with guide and candidate statistics per step and intent and step counts per task.P90 denotes the 90th percentile.Source: Table A3 · p. 10
  • Guides have a median length of 13 words, candidate menus a median of 55 elements, and tasks a median of 6 steps with a maximum of 71.The passage notes that later steps consume guide history rather than raw action history.Source: D Dataset Statistics · p. 10

E Full Results

The appendix supplies per-seed results and supporting materials for the reported means. It also documents the prompt templates, judge prompts, and training configurations used in the study.

E Full Results

  • Table A5 gives per-seed values and changes from the action-only reference; element-level scoring excludes 51 disputed steps.Source: E Full Results · p. 10
  • Figures A1–A3 show the locked SoM and coordinate prompts, contract and view differences, final instruction lines, and user-message template.Source: Figure A1 · p. 11, Figure A2 · p. 11, Figure A3 · p. 12
  • Figures A4–A7 reproduce the label-audit, guide-rewrite, blind pairwise acceptance, and guide-usefulness judge prompts.Source: Figure A4 · p. 12, Figure A5 · p. 12, Figure A6 · p. 13
  • Table A4 reports training configurations by family, including SFT runs across four output contracts and three seeds and two MG-GRPO reward arms.MG-GRPO uses 1,500 SoM train prompts, one clipped policy update per 16-prompt chunk per rank, and no KL term.Source: Table A4 · p. 13

F Mediated-Guide GRPO

Mediated-Guide GRPO uses an induced-action reward computable under the offline protocol, but it does not improve on the supervised starting point.

F Mediated-Guide GRPO

  • MG-GRPO pays a guide according to the action it induces by forcing the sampled guide as a prefix on a frozen observation, without reference text, a learned reward model, or an environment.The mediated reward is computable offline by re-querying the policy under a forced prefix on a frozen observation.Source: F Mediated-Guide GRPO · pp. 13–14
  • .433 versus .429: the mediated arm ends above the action-only control, but no checkpoint beats the .436 supervised starting point.Every checkpoint lands within about one point below the starting point.Source: Table A6 · p. 14, F Mediated-Guide GRPO · p. 14
Loading 2609.27353v1…