PaperlayerView exact revision on arXiv

Source-linked AI summary

Agent-G$^2$: Gaussian Guidance for Agentic Reinforcement Learning

Zixuan Wang, Yanrui Miao, Zhengxi Lu, Teng Pan, Yiwen Qiu, Hongxing Li, Peng Qiu, Ruiqing Zhang, Yongliang Shen

arXiv:2608.23318v1cs.AIcs.CL

TL;DR

Hint-based RL must choose guidance depth under sparse rewards and heterogeneous task difficulty, while existing scalar schedules or per-sample probing are limited. Agent-G² samples per-task Gaussian depths whose parameters are estimated online from policy-optimization rollouts, and it outperforms major baselines on ALFWorld at under one-third of probing cost.

  • Problem

    Existing guidance-depth methods use deterministic scalars: schedules ignore per-task heterogeneity, while per-sample probing requires extra rollouts.

  • Method

    Agent-G² estimates each task’s Gaussian guidance-depth center and spread online from global and per-cluster rollout statistics, without probe rollouts or a learned depth predictor.

  • Results

    Agent-G² is strongest on ALFWorld and the strongest non-probing WebShop method at both model scales, outperforming the strongest hint-free baseline by 3.9 points on ALFWorld.

  • Takeaways & Limitations

    Gaussian schedule design provides per-task depth variation and can substitute for backbone scaling within the evaluated ALFWorld and WebShop settings.

  • Takeaways & Limitations

    Agent-G² requires one expert trajectory per training task and cannot be applied directly when such trajectories are unavailable or costly to obtain.

Abstract

from arXiv · show

Hint-based reinforcement learning addresses reward sparsity in long-horizon agentic tasks by retaining a prefix of an expert trajectory before each rollout, letting the policy explore from a state closer to success. Its effectiveness hinges on the guidance depth: how much of the trajectory to keep. Existing methods treat this depth as a deterministic scalar. Scheduled approaches share one value across samples and ignore per-task heterogeneity; per-sample probing estimates it separately at the cost of extra rollouts. We find that useful guidance occupies a band of depths whose informativeness profile is approximately Gaussian around the band center, rather than concentrating at a single optimal point. We propose Agent-G$^2$, a Gaussian guidance framework that draws the depth per task from a Gaussian whose center and spread are estimated online from rollouts already collected for policy optimization, requiring no probe rollouts or learned depth predictor. The center combines a global baseline with per-cluster difficulty, and the spread tracks within-cluster variance. We evaluate Agent-G$^2$ on ALFWorld and WebShop on Qwen2.5-1.5B / 7B-Instruct. Agent-G$^2$ outperforms the strongest hint-based, hint-free, and Aux-RL baselines on ALFWorld by 2.3 / 3.9 / 7.4 points at under one-third the rollout cost of per-sample probing.

1 Introduction

Long-horizon agentic RL suffers from sparse terminal rewards, and hint-based RL depends on choosing guidance depth despite heterogeneous task difficulty. Agent-G² models useful depths as per-task Gaussian distributions estimated from existing rollouts, improving benchmark performance without probe-rollout overhead.

  • Motivation: Sparse terminal rewards make on-policy exploration from initial states unlikely to reach successful outcomes in long-horizon tasks.Hint-based RL addresses this by retaining an expert-trajectory prefix before each rollout.
  • Motivation: Guidance depth is critical: too little guidance yields few successful rollouts, while too much removes contrast needed for advantage estimation.Existing methods treat depth as a deterministic scalar despite this sensitivity.
  • Prior methods: Schedule-based methods share one depth across samples, whereas per-sample methods incur O(log n) or N× extra rollouts and still address heterogeneous agentic tasks inefficiently.Agentic batches can contain tasks ranging from two-step Pick to twenty-step Pick Two examples.
  • Approach: Agent-G² estimates a per-task Gaussian guidance distribution online using a global baseline and per-cluster difficulty and variance statistics from existing rollouts.The framework requires neither probe rollouts nor a learned depth predictor.
  • Results: 95.3% and 98.4% overall success are achieved on ALFWorld with Qwen2.5-1.5B and 7B, respectively.Agent-G² also achieves 92.3 reward on WebShop at both scales, with 78.9% and 84.4% final-purchase success.

2 Preliminary Analysis

The analysis finds that useful guidance spans a neighborhood with an approximately Gaussian informativeness profile, exposing a trade-off between scalar scheduling and costly per-sample probing.

  • Shared-depth scheduling: Shared-depth schedulers place only 15%–23% of rollout groups in range, while even Target-acc leaves 38% outside the informative band.Most assignments from the tested step-based schedulers are over-guided.
  • Per-sample probing: Enumeration reaches 47% mismatch with Mprobe=4 at 2× the GRPO rollout budget, while near-zero mismatch requires Mprobe=32 at 20× the budget.Binary Search still leaves 75% of assignments mismatched with Mprobe=2 per candidate depth.
  • Useful guidance: The in-range guidance regime spans multiple neighboring depths rather than a single optimal value.This indicates that several depths can keep rollouts in the just-learnable regime.
  • Useful guidance: The aligned informativeness profile is unimodal, approximately symmetric, and well fit by a Gaussian with σ=0.22 and R2=0.92.The fitted center and spread provide the parameters used by the later sampler.

3 Method: Agent-G2

Agent-G² adapts guidance distributions from clustered task statistics and shared policy-optimization rollouts. It samples task-specific prefix depths, updates the policy, and refreshes schedule parameters without additional probe rollouts.

  • Adaptive Gaussian Schedule: Agent-G² converts the preliminary Gaussian-like depth profile into an adaptive Gaussian schedule derived from existing rollout statistics.The same rollouts support policy updates and schedule adaptation.
  • Adaptive Gaussian Schedule: Tasks are clustered offline by expert-trajectory length as a proxy for difficulty, with cluster success means and variances updated by EMA.Ak estimates cluster success, while Vk measures response dispersion within the cluster.
  • Adaptive Gaussian Schedule: A lower cluster success rate shifts the Gaussian center toward deeper guidance, while higher within-cluster variance widens its spread.The per-task parameters use µi = clip(µglobal + λ(ptarget −Ak), 0, 1) and σi = max(γVk, σmin).
  • Sampling and rollout: One guidance ratio is sampled per task, converted to a prefix length, and shared by all R rollouts from that task’s post-prefix state.Independent sampling creates task-level depth variation even when tasks share cluster parameters.
  • Training: The training loop combines GRPO with teacher-forced loss on sampled expert prefixes, while terminal rewards refresh µglobal, Ak, and Vk for the next batch.Schedule adaptation introduces no probe rollouts or learned depth predictor.

4 Experiments

Agent-G² is evaluated on long-horizon ALFWorld and WebShop tasks across two Qwen2.5-Instruct scales, where it achieves strong performance and training efficiency. Analyses show that heterogeneous, rollout-derived Gaussian depth sampling—and its cluster-aware center and adaptive spread—drives the gains beyond imitation.

  • Setup: Agent-G² is evaluated on ALFWorld and WebShop using Qwen2.5-1.5B/7B-Instruct, with ALFWorld grouped into Short, Medium, and Long horizons.The benchmarks cover embodied household tasks and web navigation with rewards issued at task completion.
  • Main results: Agent-G² reaches 95.3% and 98.4% ALFWorld success at 1.5B and 7B, respectively, outperforming RLVMR and Enumeration without auxiliary networks or probe rollouts.The margins over RLVMR are +7.4 / +6.6, while the margins over Enumeration are +9.3 / +2.3.
  • Main results: On WebShop, Agent-G² is the strongest non-probing method at both scales, achieving 92.3 reward score with 78.9% / 84.4% purchase success.At 1.5B, it also exceeds Enumeration by +2.2 reward-score points.
  • Cross-scale robustness: Agent-G² remains effective across scales: its ALFWorld margin over the strongest hint-based baseline grows from +1.5 to +2.3, while WebShop purchase success rises from 78.9% to 84.4%.The 1.5B model also surpasses all 7B non-probing baselines.
  • Beyond imitation: Full SFT and Sampled-Prefix SFT reach 56.3% and 26.6%, respectively, versus Agent-G²’s 95.3%, isolating the contribution of post-prefix RL.Removing LGRPO reproduces Sampled-Prefix SFT, whereas removing Laux lowers success by 8.6 points.
  • Training dynamics: The largest convergence gap appears when per-task depth distributions are widest, because scalar depth is too deep for easy samples and too shallow for hard ones.Agent-G² samples a neighborhood of depths so both ends of the difficulty spectrum receive usable guidance.
  • Training dynamics: Agent-G²’s schedule shifts from deep prefixes at t=5, widens around t=25–50, and concentrates near zero by t=200 as cluster success improves.The mean follows global and cluster success, while the spread follows rollout-derived within-cluster variance.
  • Training efficiency: Agent-G² reaches scheduled-baseline accuracy in roughly half the gradient steps and costs 88 seconds per step versus 285–425 seconds for probing methods.Scheduled methods cost 57–80 seconds per step, so Agent-G² trades modest per-step overhead for faster convergence and much lower probing cost.

5 Related Work

Prior hint-based RL methods choose expert-prefix depth either globally or through costly per-sample probing, while auxiliary approaches convert expert trajectories into separate supervision signals.

  • Hint-based RL injects expert trajectory prefixes into rollouts to address reward sparsity, with methods differing mainly in prefix-depth selection.
  • Schedule-based methods share one depth across samples, whereas per-sample methods estimate instance-specific depths using extra rollouts.Per-sample probing can require O(log n) extra rollouts per sample or N× the rollout budget.
  • Both hint-based lines treat depth as a deterministic scalar and largely overlook per-task heterogeneity within a training context.
  • Auxiliary-supervision methods instead derive signals such as SFT targets or trajectory-level preference pairs from expert trajectories.

6 Conclusion

Agent-G2 introduces Gaussian guidance for long-horizon agentic tasks, sampling task-specific depths from rollout-estimated distributions rather than using deterministic depths. It achieves leading performance without probe rollouts or a learned depth predictor.

  • Agent-G2 models guidance depth as a per-task Gaussian with online-estimated center and spread from policy-optimization rollouts.The framework avoids both a learned depth predictor and extra probe rollouts.
  • Agent-G2 is strongest on ALFWorld and the strongest non-probing method on WebShop at both 1.5B and 7B scales.At 1.5B, Agent-G2 surpasses 7B BEACON, indicating schedule design can substitute for backbone scaling within the reported comparison.
  • 1.5B Agent-G2 surpasses 7B BEACON on the reported benchmark comparison.

Limitations

Agent-G2 depends on expert trajectories and uses a Gaussian parameterization and fixed offline difficulty clusters whose suitability may vary across tasks and training progress.

  • Agent-G2 requires one expert trajectory per training task, limiting direct application when such trajectories are unavailable or costly.Extending the framework to suboptimal demonstrations or language hints remains an open direction.
  • Gaussian guidance fits the informativeness profiles on the two benchmarks, but multimodal or heavily skewed profiles may require richer distributions.A uniform-distribution ablation suggests stochastic coverage of the informative band may matter more than the exact distributional shape.
  • Difficulty clusters are fixed offline using expert-trajectory length as a proxy for horizon and difficulty.This partitioning does not adapt as policy competence changes task-relative difficulty; online difficulty estimation is suggested as a future improvement.

Ethics Statement

The authors report adherence to ethics and conduct guidelines, use of public benchmarks and open-source models, and planned MIT-licensed code release.

  • The work adheres to the ACL Code of Ethics and Code of Conduct.
  • The experiments use publicly available benchmarks and pretrained open-source models.
  • Code and training scripts are planned for release under the MIT License upon publication.
  • Claude was used for grammatical refinement, while the authors state that the scientific content and conclusions are their own.

A Diagnostic Protocol Details

The diagnostic protocol measures whether assigned guidance depths fall within each task’s just-learnable band and compares shared-depth, probing, and Gaussian schedules by mismatch and rollout cost. It also tests whether informativeness follows a Gaussian profile across training.

  • In-range set and mismatch ratio: The in-range set contains depths with estimated success rates between 0.4 and 0.6, the just-learnable band used for diagnosis.Depths below 0.4 are under-guided, while depths above 0.6 are over-guided.
  • In-range set and mismatch ratio: The mismatch ratio is the fraction of tasks assigned depths outside their in-range sets, with ρ = 0 meaning all assignments are in-range.ρ = 1 means every assignment is under- or over-guided.
  • Shared-depth schedulers: Shared-depth schedulers apply one depth to every task in a batch, and the protocol categorizes assignments as under-guided, in-range, or over-guided.The diagnostic repeats this comparison across additional training checkpoints to test whether mismatch is checkpoint-specific.
  • Probing schedulers: Per-sample probing evaluates candidate depths using additional rollouts, with Enumeration using the full grid and Binary Search using visited subsets.The protocol varies M_probe from 2 to 32 and reports cost relative to the GRPO rollout budget.
  • Rollout-cost comparison: Agent-G2 is plotted at 1× rollout cost because its schedule is refreshed from the R = 8 GRPO rollouts without probing.Its parameters are read from rollouts already collected for the GRPO update.
  • Gaussian fit across training: Across three training windows, Gaussian fits retain high quality with R^2 ∈[0.89, 0.94], while the fitted width changes only mildly.The same Gaussian form is refit after pooling aligned depth-offset and Bernoulli-variance pairs within each window.
  • Choice of Gaussian for the sampler: The Gaussian sampler uses mean and variance because batch rollouts naturally provide per-cluster success-rate statistics for its two-moment parameterization.This design avoids requiring additional shape parameters that are not directly available from rollout aggregates.

B Implementation Details

The implementation builds on shared training infrastructure, uses identical expert-data access for relevant comparisons, and estimates Gaussian guidance per horizon cluster. Ablations examine schedule dynamics and the number and construction of clusters.

  • Training setup: Agent-G2 uses the GiGPO codebase, shared hyperparameters, and inherited train/validation/test splits on both benchmarks.The full configuration is listed in Table 5.
  • Baselines: Baseline implementations use the authors’ public defaults for GiGPO, BEACON, StepHint, and TraPO, while prompting baselines run at inference time.Table 7 summarizes the baseline configurations.
  • Expert trajectory source: Agent-G2 and RLVMR use the same underlying expert-data pool, so their comparison differs in algorithmic usage rather than expert-data access.Agent-G2 uses expert prefixes as a token-level imitation target within the joint objective without an auxiliary network or additional supervision signal.
  • Hardware and framework: Training runs use the verl framework on a single node with 8 NVIDIA H800 GPUs; a 1.5B ALFWorld run takes approximately 10 hours.The reported run uses 200 gradient steps.
  • Per-cluster training dynamics: At t=5, cluster means remain near 1; at t=30 they separate by horizon, and at t=75 wider spreads sample a neighborhood around shallow centers.By t=200, the means collapse toward zero.
  • Per-cluster training dynamics: The schedule trajectories track μ_t and σ_t separately for Short, Medium, and Long horizon clusters across training steps.Figure 8 organizes one horizon cluster per row and the two parameters by columns.
  • Clustering design: Length-quantile clustering is stable for K=3–5, whereas K=10 reduces accuracy by 9.1 points as clusters become too small for stable EMA estimation.Random assignment at K=3 comes within 0.5 points of K=1, indicating that clustering gains rely on aligned grouping.
  • Clustering design: Length is used because it correlates with unguided terminal-reward difficulty and requires neither task-type annotations nor semantic embeddings.The same alignment rule therefore applies to both benchmarks.
Loading 2608.23318v1…