PlanPO: Group Planning-Aware Policy Optimization for Multi-Turn Agentic LLMs

Dayang Liang; Liyuan He; Xuan Feng; Shuxin Li; Bo An; Yunlong Liu

Source-linked AI summary

PlanPO: Group Planning-Aware Policy Optimization for Multi-Turn Agentic LLMs

Dayang Liang, Liyuan He, Xuan Feng, Shuxin Li, Bo An, Yunlong Liu

arXiv:2608.17289v1Aug 18, 2026cs.AI

TL;DR

Group-relative methods often treat successful but inefficient trajectories alike, weakening their training signal for multi-turn agentic tasks. PlanPO uses success-conditioned, coarse-to-fine length-normalized advantages and improves over GRPO by 27.2% on average across ALFWorld, WebShop, and SciWorld.

Problem
Existing group-relative methods do not distinguish successful trajectories with substantially different interaction efficiency, limiting enrichment of rollout-driven training signals.
Method
PlanPO computes separate relative advantages from success-conditioned trajectory-level and response-level length normalization, then combines them to preserve multiscale distinctions.
Results
27.2% average improvement over GRPO across ALFWorld, WebShop, and SciWorld accompanies consistent gains over strong baselines at negligible additional training cost.
Takeaways & Limitations
Successful-rollout heterogeneity provides a simple, scalable signal for learning planning-aware and generalizable behaviors in multi-turn agentic LLMs.
Takeaways & Limitations
The task setting assigns each trajectory only a terminal reward of 10 when the task goal is reached.

Abstract

from arXiv · show

Group-relative policy optimization has emerged as a key paradigm for training agentic large language models (LLMs) on multi-turn interactive tasks. However, most existing variants fail to distinguish advantages among successful trajectories even when these trajectories differ substantially in their interaction efficiency. For instance, circuitous successes are often assigned the identical outcome reward, causing advantage collapse and severe performance bottlenecks. To this end, we propose Group Planning-aware Policy Optimization (PlanPO), a simple yet effective RL method for learning generalizable planning abilities beyond task-specific high-quality behavior patterns. Specifically, PlanPO introduces coarse-to-fine advantage signals, which capture the relative differences in trajectory-level lengths and turn-level response lengths conditioned on successful trajectories sampled for the same task. Within the group-relative optimization structure, this enables agents to actively learn generalizable and deliberate behaviors spanning interaction planning and textual generation from high-quality rollouts, without degenerating into vanilla length minimization. Experimentally, PlanPO improves over GRPO by 27.2\% on average across the challenging multi-turn benchmarks ALFWorld, WebShop, and SciWorld, outperforming recent powerful baselines while incurring negligible additional training cost.

Introduction

PlanPO addresses successful-rollout heterogeneity in group-relative agentic RL by using success-conditioned length profiles to create coarse-to-fine advantages. Across ALFWorld, WebShop, and SciWorld, it improves over GRPO by 27.2% on average while adding negligible training cost.

Method: PlanPO normalizes outcome rewards by turn-level trajectory lengths and token-level response lengths among successful trajectories sampled for the same task.These length profiles provide coarse-to-fine dense reward signals without requiring value models.
Motivation: Successful rollouts can reach the same goal yet differ substantially in length, directness, and reasoning quality, limiting group-relative policy learning.Redundant turns and reasoning can mask rollout quality and make successful trajectories equally informative despite meaningful differences.
Method: PlanPO computes separate relative advantages for trajectory-level and response-level signals, then combines them through a weighted formulation.The method avoids directly summing dense rewards while preserving discriminative information among successful rollouts.
Results: 27.2%: PlanPO improves over GRPO on average across ALFWorld, WebShop, and SciWorld while incurring negligible additional training cost.The evaluation uses Qwen2.5-1.5B-Instruct and Qwen2.5-7B-Instruct, and PlanPO consistently outperforms recent strong baselines.
Results: 24.3%: PlanPO gains on out-of-distribution tasks in ALFWorld.This result is reported alongside the average improvement over GRPO.

Preliminaries

Multi-turn agentic RL models LLM agents as finite-horizon MDPs in which policies generate textual actions across sequential environment interactions. Group-relative methods sample complete trajectory groups and estimate advantages from normalized rewards, avoiding PPO’s critic-network overhead.

Multi-Turn Agentic RL: Multi-turn agentic RL formulates LLM-agent interaction as a finite-horizon MDP with states, textual actions, transitions, and scalar rewards over T turns.A trajectory records the interaction sequence across turns.
Group-Relative Policy Optimization: Group-relative policy optimization samples N complete candidate trajectories for each task and assigns each trajectory a scalar reward reflecting outcome quality or success.The group is generated by the old policy πθold.
GRPO: GRPO normalizes each trajectory reward using the mean and variance of group rewards to estimate its advantage.This replaces critic-based advantage learning with a sampling-based estimator.
GRPO: The sampling-based GRPO estimator reduces the memory and computational overhead of PPO’s critic architecture.The reduction follows from avoiding the conventional critic network.

Group Planning-Aware Policy Optimization

PlanPO learns planning quality from successful same-task rollouts by combining trajectory-level and turn-level length signals in a coarse-to-fine group-relative advantage. Its design preserves planning as the dominant signal while refining response-level credit without relying on costly post-hoc or verifier-based signals.

Trajectory-level advantage: PlanPO uses multiscale rollout length to distinguish successful trajectories that differ in planning efficiency, while avoiding unconditional length minimization.Trajectory length provides a coarse signal only among successful rollouts with the same outcome reward.
Trajectory-level advantage: Successful trajectories receive length-normalized episode rewards, whose group-relative advantages are broadcast across their turns to discriminate process quality.The task setting assigns terminal outcome reward R(τ_i) = 10 when the goal is reached.
Turn-level advantage: PlanPO computes turn-level advantages among successful responses for the same task, assigning each response’s fine-grained signal to all its tokens.This conditioning makes planful responses desirable only when their corresponding actions remain correct.
Multiscale integration: The final advantage combines trajectory- and turn-level signals with a decaying coefficient, keeping trajectory-level planning quality dominant and response length a refinement.The coefficient satisfies 0 ≤ α_final ≤ α(k) < α_init ≤ 1, while excessive response-length weighting can hurt training.
Policy optimization: PlanPO retains the clipped group-relative objective backbone used by GRPO-style methods, with KL regularization controlled by β.The policy uses the importance sampling ratio and clipping coefficient ε within this objective.

Experiments

Experiments evaluate PlanPO on ALFWorld, WebShop, and SciWorld, showing strong benchmark performance, component-dependent gains, generalization to unseen tasks, and planning-aware efficiency. The results attribute improvements to success-conditioned, coarse-to-fine advantage signals rather than unconditional length minimization.

Benchmarks: PlanPO is evaluated on three challenging multi-turn benchmarks: ALFWorld, WebShop, and SciWorld.ALFWorld measures long-horizon textual reasoning and decision-making in an embodied household environment.
Performance: 91.3% overall success on ALFWorld is achieved by PlanPO with Qwen2.5-1.5B-Instruct, outperforming existing powerful baselines including GiGPO and EMPG.The passage reports that PlanPO also substantially outperforms advanced closed-source models on ALFWorld and WebShop.
Ablation Study: PlanPO’s gains primarily stem from the trajectory-level advantage AE, while a properly weighted, decaying turn-level advantage AS further improves performance.Ablating either level signal significantly degrades performance, and αinit = 0.1 performs best among the tested schedules.
Generalization Verification: 87.1% success on out-of-distribution ALFWorld tasks improves over GRPO by +17.0 points and over GiGPO by +4.7 points.The in-distribution to out-of-distribution gap decreases from 91.3% to 87.1%, indicating generalization to unseen task configurations.
Planning-Aware Strategies: 13.8 mean interaction turns and 56.3 response tokens are achieved by PlanPO, compared with 26.1 turns and 95.1 tokens for GRPO.PlanPO prefers successful rollouts that are interaction-efficient and response-concise, while unconditional length shaping hurts task success.

Related Work

Reinforcement learning has become central to improving LLM reasoning and interactive agents. PPO-based RLHF uses a value model, while critic-free methods such as ReMax, RLOO, and GRPO improve scalability through sampled baselines or group-relative rewards.

RL has become a central recipe for improving LLM reasoning and interactive agents.
PPO-based RLHF trains a value model to reduce variance.
ReMax, RLOO, and GRPO replace the learned critic with sampled baselines or group-relative rewards, improving scalability for LLMs.

Conclusion

PlanPO addresses successful-rollout heterogeneity in group-relative reinforcement learning for multi-turn agentic LLMs by constructing success-conditioned length-induced advantages. Across multiple benchmarks, it achieves strong task performance and generalization with negligible overhead.

Conclusion: PlanPO targets successful-rollout heterogeneity in group-relative reinforcement learning for multi-turn agentic LLMs.The method is designed for multi-turn agentic settings.
Conclusion: PlanPO constructs success-conditioned length-induced advantages from trajectory-level interaction and token-level response generation.Its advantages combine interaction-level and response-generation-level signals.
Conclusion: Across multiple benchmarks, PlanPO consistently outperforms existing methods with the best task performance, stronger generalization, planful strategies, and negligible overhead.The conclusion reports consistent gains across multiple benchmarks while emphasizing low additional cost.