PaperlayerView exact revision on arXiv

Source-linked AI summary

Matched-Context Group Policy Optimization for LLM Agent Training

Chenxu Zhang, Yaorui Shi, Wenlong Zhang, Jianchao Tan, Zihan Wang, Yiming Cheng

arXiv:2610.13755v1cs.LGcs.AI

TL;DR

Long-horizon agent training must assign credit to intermediate actions from sparse terminal feedback, while useful comparisons need context and evidence for both alternatives. MCGPO builds weighted action preferences from existing rollouts, and improves mean success over matched GraphGPO across six tested settings.

  • Problem

    Sparse terminal rewards must guide intermediate actions in long-horizon tasks, but comparisons can mix prior progress with current-action contributions and richer contexts can leave one alternative poorly supported.

    Source: §1 · p. 2

  • Method

    MCGPO matches pre-action state and rubric progress, uses opposite-fold terminal outcomes to orient supported action pairs, and adds an evidence-weighted pairwise loss to the shared task objective.

    Source: RESEARCH QUESTION · p. 2

  • Results

    MCGPO improves three-seed mean success over matched GraphGPO across six settings, including gains of 3.96 and 5.16 points on WebShop with Qwen3-4B and Qwen3-8B.

    Source: RESEARCH QUESTION · p. 2, §5 · p. 9

  • Takeaways & Limitations

    The results support extracting useful action-level supervision from existing rollout groups, with joint state–rubric matching achieving the highest mean in all four representation-control settings.

    Source: RESEARCH QUESTION · p. 2, §5 · p. 9

    Main comparisons use three training seeds and repeated evaluation on a fixed validation set, so small gains retain statistical uncertainty.

    Source: E Limitations · p. 26

Abstract

from arXiv · show

Training long-horizon language model agents requires assigning credit to intermediate decisions from sparse terminal feedback. Within a finite group of sibling rollouts, useful action comparisons depend on their context, evidence for both alternatives, and how that evidence enters learning. We propose Matched-Context Group Policy Optimization (MCGPO), a procedure for constructing weighted action preferences from already collected trajectories. MCGPO defines comparison contexts with pre-action state summaries and observed rubric progress, annotated by task rules. Two-action support selects among ordered exact and coarse matching rules. Fixed folds separate candidate decisions from the terminal outcomes used to orient their preferences, and support-dependent evidence weights scale an auxiliary pairwise objective alongside shared GraphGPO task learning. MCGPO reuses rollout data and each response's original history, preserving critic-free training. Across six environment-backbone settings with three training seeds each, MCGPO improves mean success over matched GraphGPO: by 0.47-1.46 percentage points on ALFWorld and WebShop with Qwen2.5-1.5B and 7B, and by 3.96 and 5.16 points on WebShop with Qwen3-4B and 8B. The ALFWorld gains build on baseline success above 92%, where remaining headroom is limited. Integrated success improves in three of four Qwen2.5 settings. Three-seed factorial controls support opposite-fold evidence and evidence weighting; joint state-rubric matching achieves the highest mean in all four representation-control settings.

1 Introduction

MCGPO addresses how to make local action comparisons meaningful when prior task progress and limited evidence for both alternatives complicate credit assignment. It constructs weighted preferences from existing rollouts and improves mean success across the tested settings.

1 Introduction

  • Comparisons at the same summarized state can mix prior progress with an action’s contribution, while stricter contexts reduce repeated evidence for comparing different actions.The introduction illustrates this with two empty-handed visits to the same countertop after different prior task progress.Source: §1 · p. 2

RESEARCH QUESTION

  • MCGPO matches pre-action state summaries and rubric progress, uses opposite-fold terminal outcomes to orient preferences, and weights a pairwise loss alongside the shared GraphGPO objective.Its labels are observational outcome contrasts under the rollout policy, and the method reuses rollout data and original response histories.Source: RESEARCH QUESTION · p. 2
  • MCGPO improves three-seed mean success over matched GraphGPO by 0.47–1.46 percentage points in four Qwen2.5 settings and by 3.96 and 5.16 points on Qwen3 WebShop.Integrated success improves in three of four Qwen2.5 settings; joint state–rubric matching has the highest mean in all four representation-control settings.Source: RESEARCH QUESTION · p. 2
  • The paper evaluates construction utility and component effects with matched comparisons, supervision controls, and representation diagnostics.Three-seed factorial controls support opposite-fold evidence and evidence weighting.Source: RESEARCH QUESTION · pp. 2–3

2 Related Work

Related work addresses credit assignment through group-based comparisons, task-structured supervision, and process or preference signals. MCGPO instead forms supported action-pair supervision from matched context and collected rollouts.

Agent training and group-based credit

  • Group-credit methods use anchor states, histories, transitions, or trees, whereas MCGPO constructs action-pair supervision using pre-action state, rubric progress, two-action support, and opposite-fold outcomes.The passage also describes graph-based methods that aggregate credit through transitions or shared-state trees.Source: §2 · p. 3

Task-structured supervision

  • Task-structured methods use rubrics, stages, criterion dependencies, or milestones; MCGPO uses task-specific progress extractors to condition comparisons and terminal outcomes to label preferences.The passage distinguishes MCGPO’s comparison structure from methods that derive it from histories or observed graphs.Source: §2 · p. 3

Process, preference, and value supervision

  • Process and preference methods draw on terminal feedback, step preferences, search, extra continuations, or value models; MCGPO reuses sibling rollouts and separates candidate decisions from label contributors with fixed folds.The passage notes that these designs are compared in Appendix Table 4.Source: §2 · p. 3

3 Methodology

MCGPO turns sibling rollouts into local preferences through matching, cross-rollout labeling, and evidence-weighted policy learning. These steps preserve the shared task update and critic-free training.

3 Methodology

  • Eight sibling trajectories are split into fixed folds so one fold supplies candidate decisions and the other supplies outcome evidence, with roles swapped to use both halves.Figure 2 summarizes the Verify block’s use of opposite-fold terminal outcomes and the pairwise loss alongside the shared task objective.Source: §3 · p. 3, Figure 2 · p. 4

3.1 Task-conditioned matching (Match)

  • Matching combines pre-action state summaries with rubric progress annotated by task rules, then selects the first ordered exact or coarse rule with support for both actions.Unsupported pairs are omitted; a supported exact match is retained even if its later outcome contrast falls below threshold.Source: §3.1 · p. 4

3.2 Cross-rollout preferences (Label)

  • Opposite-fold mean terminal successes orient retained pairs, and contrast magnitude, support shrinkage, and resolution determine their evidence weights.The method keeps the earliest qualified comparison per sibling-trajectory pair and uses a disjoint allocator to bound preferences and reuse of decision rows.Source: §3.2 · p. 4

3.3 Evidence-weighted policy learning (Learn)

  • The evidence-weighted pairwise objective complements the shared GraphGPO task loss, using original response histories and scaling pair contributions with available supervision.Tasks without preferences still receive the task update, and the construction reuses sampled trajectories and policy log probabilities for critic-free training.Source: §3.3 · p. 5

4 Experiments

The experiments test MCGPO against matched GraphGPO across ALFWorld and WebShop, then examine its supervision components, preference construction, and transfer to Qwen3. Across the reported settings, mean success improves, with results and controls varying in scope and seed count.

4.1.1 Benchmarks and Models

  • The evaluation covers ALFWorld and WebShop with Qwen2.5-1.5B/7B, three seeds per setting, and matched GraphGPO task-credit learning; validation uses the same 128 tasks every five updates.Qwen2.5 responses use the two most recent interaction steps and up to 512 generated tokens; runs train for 250 updates, and the primary metric is mean success over the last five scheduled evaluations.Source: §4.1.1 · p. 5, §4.1.2 · pp. 5–6

4.2 RQ1: Does MCGPO Improve Task Completion and Learning Progress?

  • MCGPO improves mean success over matched GraphGPO in all four Qwen2.5 settings by 1.04/1.46 points on ALFWorld and 1.09/0.47 points on WebShop at 1.5B/7B.ALFWorld success rises from 92.60%/94.84% to 93.65%/96.30%; on WebShop at 1.5B, success rises while graded score falls from 88.79 to 86.83.Source: §4.2 · p. 6, Table 1 · p. 6
  • The first three Qwen2.5 settings improve integrated success over updates 0–250, by 1.25/1.46 points on ALFWorld and 2.98/−0.02 on WebShop at 1.5B/7B.The integrated-success comparison measures learning progress along the policy-update axis; at update 100, the smoothed 1.5B advantage is 7.60 points on WebShop and 3.54 on ALFWorld.Source: §4.2 · pp. 6–7, Figure 3 · p. 7

4.3 RQ2: Which Parts of the Supervision Design Matter?

  • The candidate-progress alternative reduces success by 1.41 points on WebShop and 3.13 on ALFWorld, while shuffled rubric labels reduce it by 2.34 and 0.63 points, respectively.The candidate-progress control jointly changes evidence source, label content, and available support; it favors the complete terminal-outcome preference design over that alternative.Source: §4.3 · p. 7, Table 2 · p. 8
  • Joint state–rubric conditioning has the highest three-seed mean success in all four representation-control settings, exceeding State-only by 3.96–9.90 points and Rubric-only by 5.15–9.01 points.The gains persist at terminal evaluation; WebShop State-only retains task-match score bins in its coarse state.Source: §4.3 · p. 8

4.4 RQ3: What Preferences and Weights Does the Construction Supply?

  • Of 1,990 accepted preferences, 1,805 (90.70%) use coarse matching; the weaker-supported action has one evidence trajectory in 1,547 pairs (77.74%).Average supply declines from 3.99 pairs per update in updates 26–50 to 2.08 in updates 226–250.Source: Figure 4 · p. 8, §4.4 · p. 9
  • Preference directions agree with candidate outcomes at 74.35% pooled agreement when ties receive half credit; mean absolute outcome contrast is 0.781 and mean evidence weight is 0.208.Agreement measures association with candidate terminal outcomes, which also reflect subsequent actions.Source: Figure 4 · p. 8, §4.4 · p. 9

4.5 RQ4: Do the WebShop Gains Extend to Qwen3 Backbones?

  • 3.96 and 5.16 percentage points: MCGPO raises WebShop mean success over GraphGPO from 78.18% to 82.14% at Qwen3-4B and from 79.69% to 84.84% at 8B.All six paired runs improve; these Qwen3 runs use action-only responses.Source: §4.5 · p. 9, Table 3 · p. 9

5 Conclusion

MCGPO constructs weighted action preferences from sibling rollouts using matched contexts, opposite-fold terminal outcomes, and an auxiliary pairwise loss alongside GraphGPO task learning. Its conclusion reports improved mean success across six settings and supports extracting action-level supervision from existing rollout groups.

5 Conclusion

  • Across six environment–backbone settings, MCGPO improves three-seed mean success over matched GraphGPO, including Qwen3 WebShop gains of 3.96 and 5.16 points.Representation and factorial controls support joint context matching, opposite-fold evidence, and evidence weighting; scorer-matched comparisons favor MCGPO over reward shaping.Source: §5 · p. 9

AI Use Statement

  • Generative AI tools assisted with code development, experiment-log organization and analysis, manuscript drafting, and language editing; the authors say they independently verified the final content.Source: AI Use Statement · p. 9

A Algorithm and comparison construction

  • Match–Label–Learn selects comparison contexts supported by both actions, orients preferences with opposite-fold terminal outcomes, and adds a weighted pairwise loss to the GraphGPO task objective.The same eight sibling trajectories supply the task objective and auxiliary preferences.Source: §5 · p. 9, A Algorithm and comparison construction · p. 14

A.1 Comparison keys and first visits

  • Comparison keys combine pre-action state summaries, active rubric criteria, and a rule-derived frontier of eligible inactive criteria whose prerequisites are met and which are not blocked.Ordered exact-to-coarse matching retains the rubric signature; each trajectory contributes its first eligible visit, and candidates have different canonical actions.Source: §A.1 · pp. 14–15, Table 5 · p. 15
  • Fixed rollout folds separate candidate decisions from outcome evidence; support counts and empirical action-success rates determine the support gate, with shrinkage adjusting evidence weights.The four-trajectory evidence fold permits balanced action support of one or two; invalid groups retain their task update with zero auxiliary weight.Source: A Algorithm and comparison construction · p. 14, §A.1 · p. 15

A.2 Outcome provenance and pairwise update

  • With batch inputs fixed, changing the candidate fold’s outcomes leaves that direction’s labels and weights unchanged; the reverse direction uses those outcomes as evidence.The stated property is conditional: whole-group outcome-based batch admission lies outside it, and finite-support contrasts remain observational estimates.Source: §A.2 · p. 15

A.3 Shared task objective

The shared task objective combines graph-based returns with token-level task scores, then applies normalized, clipped updates. The experimental setup keeps key training and evaluation conditions aligned across matched methods.

A.3 Shared task objective

  • The GraphGPO task branch uses directed distance to an observed successful goal for its graph return and combines normalized graph returns with normalized token-level task scores.Unreachable states receive one more than the largest finite distance; both task-credit coefficients are one, and each row advantage is broadcast to valid response tokens.Source: §A.3 · p. 16
  • The task objective uses clipped token-ratio losses with a separate rule for negative advantages, and normalization uses group means and sample standard deviations.Singleton step groups receive centered score zero; singleton task groups use mean zero and unit scale.Source: §A.3 · p. 16

B Experimental setup

  • Within a backbone, GraphGPO and MCGPO use the same task data, response protocol, optimizer, and evaluation grid.The shared training recipe is used by both methods.Source: B Experimental setup · p. 16

B.1 Training and evaluation configuration

The appendix records shared training and evaluation settings, including the retained checkpoint lineage for main runs and mechanism controls. Supplementary results provide seed-level readouts, uncertainty, and control definitions.

B.1 Training and evaluation configuration

  • Each update uses eight sibling rollouts per task, with 250 policy updates, responses capped at 512 tokens, rollout temperature 1.0, learning rate 10−6, and weight decay 0.01.Validation samples one rollout on the same 128 tasks every five updates at temperature 0.4.Source: §B.1 · pp. 16–17, Table 7 · p. 16
  • Main runs retain their archived 0→150→250 protocol, while mechanism controls target 250 updates from initialization and use the corresponding Full reference seed.Scheduled evaluations follow the retained checkpoint lineage; reload evaluations and rolled-back branches are excluded.Source: §B.1 · p. 17

C Supplementary policy results and ablations

  • Supplementary policy results complement the main questions with seed-level readouts, uncertainty, and control definitions; success is reported in percent and paired gains in percentage points.Source: C Supplementary policy results and… · p. 17

C.1 Readouts and seed-level results

The section defines late-training success and integrated success as complementary readouts, then reports how uncertainty is estimated across training seeds.

C.1 Readouts and seed-level results

  • L5 averages success over the final five scheduled evaluations of the shared 250-update budget.Success is first averaged within each seed; means and sample SDs are then computed across seeds.Source: §C.1 · p. 17, Table 8 · p. 17
  • Integrated success summarizes learning progress over the complete raw evaluation grid, while E250 retains the final evaluation.Table 10 reports success and integrated success on a 0–100 scale, with paired 95% Student-t intervals.Source: Table 8 · p. 17, §C.1 · p. 18, Table 10 · p. 18
  • Paired intervals for L5 and integrated success span zero in all four Qwen2.5 settings, quantifying uncertainty around the observed mean gains.The three training seeds determine uncertainty; evaluations within a training trajectory are dependent.Source: §C.1 · p. 18, Table 10 · p. 18

C.2 Matched training-transition budgets at 1.5B

The appendix examines budget-matched results, supervision and representation controls, and diagnostics of learning and preference construction.

C.2 Matched training-transition budgets at 1.5B

  • 2.60 points on ALFWorld and 3.39 on WebShop are the mean gains at nearly matched training-transition budgets.The selected checkpoint pairs differ in budget by less than 1%; selection maximizes the smaller budget rather than success.Source: Table 11 · p. 19, Figure 5 · p. 19, §C.2 · p. 19

C.3 Mechanism controls

  • Single-run controls separately examine candidate-progress evidence, removal of pairwise learning, and rubric-based reward shaping.The candidate-progress control jointly changes evidence source, labels, support, and weights; replication would be needed to estimate variability.Source: §C.3 · p. 19
  • Opposite-fold terminal evidence and evidence-based weighting each improve Qwen3-4B WebShop success across both settings of the other factor.All four contrasts are positive in every seed, and paired 95% intervals lie above zero.Source: Table 12 · p. 20, §C.3 · p. 20

C.4 Representation controls

  • Full joint state-rubric matching has the highest mean in all four settings, exceeding State-only by 3.96–9.90 points on the primary readout.It also exceeds Rubric-only by 5.15–9.01 points; State-only can merge progress stages, while Rubric-only can merge environmental situations.Source: §C.4 · p. 20, Table 14 · p. 21

C.3 Mechanism controls

  • 2.29 points at 4B and 2.66 at 8B are MCGPO’s mean WebShop gains over scorer-matched reward shaping.Gains are positive in all six paired runs.Source: §C.3 · p. 20, Table 13 · p. 21

C.5 Learning curves and supervision dynamics

  • ALFWorld training success rises from 60.3% to 95.3% across sampled windows as rollout length falls from 29.34 to 10.17 steps.Mean anchor-state group size rises from 4.10 to 6.62; these are joint training trends, not component-effect estimates.Source: §C.5 · p. 21, Figure 7 · p. 22
  • Coarse matching supplies 1,805 of 1,990 accepted WebShop pairs, or 90.70%; pooled agreement with candidate outcomes is 74.35%.The agreement pools all accepted pairs, while the per-seed/window outcome ratios are aggregated separately.Source: §C.5 · p. 21
  • Evidence-weighted pairwise loss is nonzero in all 511 updates with accepted pairs and zero in all 239 empty-pair updates.The pooled evidence-contrast and weight means are 0.781 and 0.208, respectively.Source: §C.5 · pp. 21–22

D Rubric, matching, and preference diagnostics

This appendix specifies environment-specific rubric signals and matching rules, then diagnoses comparison availability and selected WebShop preferences.

D.1 Environment rubric rules

  • ALFWorld rubric progress tracks target-object acquisition, processing, and placement; successful actions update the corresponding sets and required counts.The required count is two for Pick2 and one otherwise; entity matching preserves object-instance numbers.Source: §D.1 · p. 23, Table 15 · p. 23
  • WebShop rubric channels use browser-session fields and scorer-derived product matches, including pre-purchase preview information.Absent components and caught scoring errors retain initialized zeros; terminal evidence marks scores of at least 0.999 as success.Source: §D.1 · p. 23, Table 16 · p. 23
  • Task-rule graphs define prerequisite tests and action keys; missing required trace fields disable auxiliary supervision but retain the task update.Progress-measurement errors can change groups and action support, and extractor precision and recall against independent annotations remain unmeasured.Source: §D.1 · p. 23, Table 17 · p. 24

D.2 Matching support and a captured example

  • The matching census covers ten archived update-60 captures, with four-trajectory candidate folds and first eligible visits counted per trajectory and key.The captures contain 160 task instances and 1,280 trajectories and are separate from the training runs used in Table 14.Source: §D.2 · p. 24, Figure 8 · p. 24
  • On the archived snapshots, Full replay reproduces stored pairs and directions; State-only selects the same pairs but has no accepted WebShop pairs after outcome filtering.Exact and coarse supported-pair counts can overlap, and the counts precede outcome-contrast filtering and the final quota.Source: §D.2 · p. 24, Table 18 · p. 25
  • A changed progress signature keeps a later ALFWorld visit distinct even when the countertop, empty hands, and target placements match.The example concerns the task “put two toiletpaper in drawer.”Source: §D.2 · p. 24, Figure 9 · p. 25

D.3 WebShop agreement with candidate outcomes

  • Archived WebShop diagnostics pool preferred-candidate wins, ties, and reversals; agreement measures their association with candidate terminal outcomes.These samples contain repeated training observations; complete 250-update diagnostics appear separately in Appendix C.5.Source: Table 19 · p. 25, §D.3 · p. 25

E Limitations

The authors limit the evidence to two environments and Qwen backbones, with transfer requiring suitable task-specific signals and rules. The appendices also document the interaction prompts used for ALFWorld and WebShop.

E Limitations

  • Transfer beyond ALFWorld and WebShop with Qwen backbones requires suitable state summaries, rubric channels, and task rules.WebShop also uses structured goal-scorer information during training; broader model families, more seeds, and independent test tasks remain to be evaluated.Source: E Limitations · pp. 25–26

F Environment Prompts

  • Prompts combine the current task and observation with admissible actions and, on later decisions, recent interaction history.For Qwen2.5, history is limited to the two most recent observation–action pairs; long WebShop prompts can fall back to no-history context.Source: F Environment Prompts · p. 26, §F.2 · p. 26

F.1 ALFWorld

  • ALFWorld prompts provide the task observation and admissible actions, with reasoning and the selected action placed in separate tags.The task is inserted explicitly in subsequent decisions.Source: §F.1 · p. 26

F.2 WebShop

  • WebShop prompts expose the shopping goal, rendered page, and search or click actions; Qwen2.5 uses tagged reasoning and action responses.For Qwen3, the response instruction instead requires exactly one action block and no other text.Source: §F.2 · pp. 26–27

G Illustrative Task and Rollout Trace

This appendix traces one successful MCGPO ALFWorld rollout for a two-object drawer task, showing its prompt, feedback, and terminal reward. The selected trace illustrates the interaction protocol, not performance or a matched preference pair.

G Illustrative Task and Rollout Trace

Loading 2610.13755v1…