Agent-G$^2$: Gaussian Guidance for Agentic Reinforcement Learning

Motivation

• Sparse terminal rewards make on-policy exploration from initial states unlikely to reach successful outcomes in long-horizon tasks.Hint-based RL addresses this by retaining an expert-trajectory prefix before each rollout.Source: §1 · p. 1
• Guidance depth is critical: too little guidance yields few successful rollouts, while too much removes contrast needed for advantage estimation.Existing methods treat depth as a deterministic scalar despite this sensitivity.Source: §1 · p. 1

Prior methods

• Schedule-based methods share one depth across samples, whereas per-sample methods incur O(log n) or N× extra rollouts and still address heterogeneous agentic tasks inefficiently.Agentic batches can contain tasks ranging from two-step Pick to twenty-step Pick Two examples.Source: §1 · p. 2

Approach

• Agent-G² estimates a per-task Gaussian guidance distribution online using a global baseline and per-cluster difficulty and variance statistics from existing rollouts.The framework requires neither probe rollouts nor a learned depth predictor.Source: §1 · p. 2

Results

• 95.3% and 98.4% overall success are achieved on ALFWorld with Qwen2.5-1.5B and 7B, respectively.Agent-G² also achieves 92.3 reward on WebShop at both scales, with 78.9% and 84.4% final-purchase success.Source: §1 · p. 2

Shared-depth scheduling

• Shared-depth schedulers place only 15%–23% of rollout groups in range, while even Target-acc leaves 38% outside the informative band.Most assignments from the tested step-based schedulers are over-guided.Source: §2 · p. 3

Per-sample probing

• Enumeration reaches 47% mismatch with Mprobe=4 at 2× the GRPO rollout budget, while near-zero mismatch requires Mprobe=32 at 20× the budget.Binary Search still leaves 75% of assignments mismatched with Mprobe=2 per candidate depth.Source: §2 · p. 3

Useful guidance

• The in-range guidance regime spans multiple neighboring depths rather than a single optimal value.This indicates that several depths can keep rollouts in the just-learnable regime.Source: §2 · p. 3
• The aligned informativeness profile is unimodal, approximately symmetric, and well fit by a Gaussian with σ=0.22 and R2=0.92.The fitted center and spread provide the parameters used by the later sampler.Source: §2 · p. 3

Adaptive Gaussian Schedule

• Agent-G² converts the preliminary Gaussian-like depth profile into an adaptive Gaussian schedule derived from existing rollout statistics.The same rollouts support policy updates and schedule adaptation.Source: §3 · p. 3, §3.4 · p. 5
• Tasks are clustered offline by expert-trajectory length as a proxy for difficulty, with cluster success means and variances updated by EMA.Ak estimates cluster success, while Vk measures response dispersion within the cluster.Source: §3.2 · p. 4
• A lower cluster success rate shifts the Gaussian center toward deeper guidance, while higher within-cluster variance widens its spread.The per-task parameters use µi = clip(µglobal + λ(ptarget −Ak), 0, 1) and σi = max(γVk, σmin).Source: §3.2 · p. 4

Sampling and rollout

• One guidance ratio is sampled per task, converted to a prefix length, and shared by all R rollouts from that task’s post-prefix state.Independent sampling creates task-level depth variation even when tasks share cluster parameters.Source: §3.3 · p. 5

Training

• The training loop combines GRPO with teacher-forced loss on sampled expert prefixes, while terminal rewards refresh µglobal, Ak, and Vk for the next batch.Schedule adaptation introduces no probe rollouts or learned depth predictor.Source: §3.4 · p. 5

Setup

• Agent-G² is evaluated on ALFWorld and WebShop using Qwen2.5-1.5B/7B-Instruct, with ALFWorld grouped into Short, Medium, and Long horizons.The benchmarks cover embodied household tasks and web navigation with rewards issued at task completion.Source: §4.1 · p. 5

Main results

• Agent-G² reaches 95.3% and 98.4% ALFWorld success at 1.5B and 7B, respectively, outperforming RLVMR and Enumeration without auxiliary networks or probe rollouts.The margins over RLVMR are +7.4 / +6.6, while the margins over Enumeration are +9.3 / +2.3.Source: §4.2 · p. 6
• On WebShop, Agent-G² is the strongest non-probing method at both scales, achieving 92.3 reward score with 78.9% / 84.4% purchase success.At 1.5B, it also exceeds Enumeration by +2.2 reward-score points.Source: §4.2 · p. 6

Cross-scale robustness

• Agent-G² remains effective across scales: its ALFWorld margin over the strongest hint-based baseline grows from +1.5 to +2.3, while WebShop purchase success rises from 78.9% to 84.4%.The 1.5B model also surpasses all 7B non-probing baselines.Source: §4.2 · p. 6

Beyond imitation

• Full SFT and Sampled-Prefix SFT reach 56.3% and 26.6%, respectively, versus Agent-G²’s 95.3%, isolating the contribution of post-prefix RL.Removing LGRPO reproduces Sampled-Prefix SFT, whereas removing Laux lowers success by 8.6 points.Source: §4.3 · p. 7, §4.4 · p. 8

Training dynamics

• The largest convergence gap appears when per-task depth distributions are widest, because scalar depth is too deep for easy samples and too shallow for hard ones.Agent-G² samples a neighborhood of depths so both ends of the difficulty spectrum receive usable guidance.Source: §4.3 · p. 7
• Agent-G²’s schedule shifts from deep prefixes at t=5, widens around t=25–50, and concentrates near zero by t=200 as cluster success improves.The mean follows global and cluster success, while the spread follows rollout-derived within-cluster variance.Source: §4.3 · p. 7

Training efficiency

• Agent-G² reaches scheduled-baseline accuracy in roughly half the gradient steps and costs 88 seconds per step versus 285–425 seconds for probing methods.Scheduled methods cost 57–80 seconds per step, so Agent-G² trades modest per-step overhead for faster convergence and much lower probing cost.Source: §4.3 · p. 7

• Hint-based RL injects expert trajectory prefixes into rollouts to address reward sparsity, with methods differing mainly in prefix-depth selection.Source: §5 · p. 8
• Schedule-based methods share one depth across samples, whereas per-sample methods estimate instance-specific depths using extra rollouts.Per-sample probing can require O(log n) extra rollouts per sample or N× the rollout budget.Source: §5 · p. 8
• Both hint-based lines treat depth as a deterministic scalar and largely overlook per-task heterogeneity within a training context.Source: §5 · p. 8
• Auxiliary-supervision methods instead derive signals such as SFT targets or trajectory-level preference pairs from expert trajectories.Source: §5 · p. 8

• Agent-G2 models guidance depth as a per-task Gaussian with online-estimated center and spread from policy-optimization rollouts.The framework avoids both a learned depth predictor and extra probe rollouts.Source: §6 · p. 8
• Agent-G2 is strongest on ALFWorld and the strongest non-probing method on WebShop at both 1.5B and 7B scales.At 1.5B, Agent-G2 surpasses 7B BEACON, indicating schedule design can substitute for backbone scaling within the reported comparison.Source: §6 · p. 9
• 1.5B Agent-G2 surpasses 7B BEACON on the reported benchmark comparison.Source: §6 · p. 9

• Agent-G2 requires one expert trajectory per training task, limiting direct application when such trajectories are unavailable or costly.Extending the framework to suboptimal demonstrations or language hints remains an open direction.Source: Limitations · p. 9
• Gaussian guidance fits the informativeness profiles on the two benchmarks, but multimodal or heavily skewed profiles may require richer distributions.A uniform-distribution ablation suggests stochastic coverage of the informative band may matter more than the exact distributional shape.Source: Limitations · p. 9
• Difficulty clusters are fixed offline using expert-trajectory length as a proxy for horizon and difficulty.This partitioning does not adapt as policy competence changes task-relative difficulty; online difficulty estimation is suggested as a future improvement.Source: Limitations · p. 9

• The work adheres to the ACL Code of Ethics and Code of Conduct.Source: Ethics Statement · p. 9
• The experiments use publicly available benchmarks and pretrained open-source models.Source: Ethics Statement · p. 9
• Code and training scripts are planned for release under the MIT License upon publication.Source: Ethics Statement · p. 9
• Claude was used for grammatical refinement, while the authors state that the scientific content and conclusions are their own.Source: Ethics Statement · p. 9

In-range set and mismatch ratio

• The in-range set contains depths with estimated success rates between 0.4 and 0.6, the just-learnable band used for diagnosis.Depths below 0.4 are under-guided, while depths above 0.6 are over-guided.Source: A Diagnostic Protocol Details · p. 13
• The mismatch ratio is the fraction of tasks assigned depths outside their in-range sets, with ρ = 0 meaning all assignments are in-range.ρ = 1 means every assignment is under- or over-guided.Source: A Diagnostic Protocol Details · p. 13

Shared-depth schedulers

• Shared-depth schedulers apply one depth to every task in a batch, and the protocol categorizes assignments as under-guided, in-range, or over-guided.The diagnostic repeats this comparison across additional training checkpoints to test whether mismatch is checkpoint-specific.Source: A Diagnostic Protocol Details · p. 13

Probing schedulers

• Per-sample probing evaluates candidate depths using additional rollouts, with Enumeration using the full grid and Binary Search using visited subsets.The protocol varies M_probe from 2 to 32 and reports cost relative to the GRPO rollout budget.Source: A Diagnostic Protocol Details · p. 13

Rollout-cost comparison

• Agent-G2 is plotted at 1× rollout cost because its schedule is refreshed from the R = 8 GRPO rollouts without probing.Its parameters are read from rollouts already collected for the GRPO update.Source: A Diagnostic Protocol Details · p. 13

Gaussian fit across training

• Across three training windows, Gaussian fits retain high quality with R^2 ∈[0.89, 0.94], while the fitted width changes only mildly.The same Gaussian form is refit after pooling aligned depth-offset and Bernoulli-variance pairs within each window.Source: A Diagnostic Protocol Details · pp. 13–14, Table 6 · p. 14

Choice of Gaussian for the sampler

• The Gaussian sampler uses mean and variance because batch rollouts naturally provide per-cluster success-rate statistics for its two-moment parameterization.This design avoids requiring additional shape parameters that are not directly available from rollout aggregates.Source: A Diagnostic Protocol Details · p. 14

Training setup

• Agent-G2 uses the GiGPO codebase, shared hyperparameters, and inherited train/validation/test splits on both benchmarks.The full configuration is listed in Table 5.Source: B Implementation Details · p. 14

Baselines

• Baseline implementations use the authors’ public defaults for GiGPO, BEACON, StepHint, and TraPO, while prompting baselines run at inference time.Table 7 summarizes the baseline configurations.Source: B Implementation Details · p. 14, Table 7 · p. 15

Expert trajectory source

• Agent-G2 and RLVMR use the same underlying expert-data pool, so their comparison differs in algorithmic usage rather than expert-data access.Agent-G2 uses expert prefixes as a token-level imitation target within the joint objective without an auxiliary network or additional supervision signal.Source: B Implementation Details · pp. 14–15

Hardware and framework

• Training runs use the verl framework on a single node with 8 NVIDIA H800 GPUs; a 1.5B ALFWorld run takes approximately 10 hours.The reported run uses 200 gradient steps.Source: B Implementation Details · p. 15

Per-cluster training dynamics

• At t=5, cluster means remain near 1; at t=30 they separate by horizon, and at t=75 wider spreads sample a neighborhood around shallow centers.By t=200, the means collapse toward zero.Source: B Implementation Details · p. 15
• The schedule trajectories track μ_t and σ_t separately for Short, Medium, and Long horizon clusters across training steps.Figure 8 organizes one horizon cluster per row and the two parameters by columns.Source: Figure 8 · p. 15

Clustering design

• Length-quantile clustering is stable for K=3–5, whereas K=10 reduces accuracy by 9.1 points as clusters become too small for stable EMA estimation.Random assignment at K=3 comes within 0.5 points of K=1, indicating that clustering gains rely on aligned grouping.Source: B Implementation Details · p. 16, Table 8 · p. 16
• Length is used because it correlates with unguided terminal-reward difficulty and requires neither task-type annotations nor semantic embeddings.The same alignment rule therefore applies to both benchmarks.Source: B Implementation Details · p. 16