PaperlayerView exact revision on arXiv

Source-linked AI summary

The Physics of Multi-Turn Long-Horizon Planning: From Pre-training to Post-training via Single- and Multi-Teacher On-Policy Agentic Distillation

Tianyi Men, Zhuoran Jin, Kang Liu, Jun Zhao

arXiv:2607.24720v1cs.CLcs.AIcs.LG

TL;DR

Long-horizon planning remains difficult to analyze and improve because foundation models learn from uncontrollable, opaque Internet data. This paper studies planning across pre-training, RL-based post-training, and multi-teacher consolidation in a unified controlled environment, finding that world-model internalization, high-quality data, broader OPD applicability, and shared-pattern convergence are central to robust improvement and integration.

  • Problem

    Foundation models still struggle with long-horizon planning, while opaque and uncontrollable training data obscures how this ability is acquired, strengthened, and integrated.

  • Method

    The paper uses a unified controlled multi-turn environment to study planning across pre-training, GRPO and OPD post-training, and multi-teacher on-policy distillation.

  • Results

    Explicit world-model internalization, limited long-horizon data, and high-quality trajectories improve planning, while OPD has a broader effective region than GRPO and MOPD integrates compatible planning patterns.

  • Takeaways & Limitations

    Planning improvement depends on the training stage: construct world models with suitable data, apply post-training selectively, and integrate teachers through compatible shared planning patterns.

  • Takeaways & Limitations

    Suboptimal trajectories can severely degrade long-horizon planning, and simply increasing sampling is ineffective and computationally impractical.

Abstract

from arXiv · show

Multi-turn long-horizon planning is critical for foundation model agents, yet how to fundamentally improve it remains unclear. Existing models are trained on uncontrollable and opaque Internet data, making it difficult to identify how planning ability is acquired, shaped, and integrated. To address this challenge, we introduce a unified and controlled multi-turn environment that enables precise control. It allows systematically study long-horizon planning across three stages. (1) Planning ability acquisition during pre-training. We study data format, distribution, and quality. Explicit world model construction through CoT state transition modeling yields stronger long-horizon generalization. Atomic skills alone are insufficient for compositional generalization, whereas a litte long-horizon data works. Moreover, suboptimal trajectories severely impair performance because errors amplify over long horizons. (2) Planning ability shaping via GRPO and OPD post-training. Through mutual information, we distinguish general planning patterns from task-specific planning knowledge. For planning patterns, we identify three application regions of post-training: unnecessary, effective, and unsupported. OPD has a broader effective region than GRPO under low-quality and long-horizon settings, as it provides more consistent update directions. For planning knowledge, distilling unseen procedures from a teacher with different knowledge may impair student's prior world modeling without fully establishing new knowledge. (3) Planning ability integration through MOPD post-training. We show that multi-teacher on-policy distillation (MOPD) integrates capabilities by converging to shared planning-pattern across environments. Compatible patterns enable cross-environment generalization, partially shared patterns support continual learning, while completely conflicting patterns cause severe interference.

1. Introduction

The paper studies how to fundamentally improve multi-turn long-horizon planning across pre-training, RL-based post-training, and multi-teacher consolidation. It addresses this question with a unified controlled environment that exposes planning acquisition, shaping, and integration mechanisms.

  • Motivation: Foundation models show basic short-horizon planning but still struggle significantly with multi-turn long-horizon tasks.Existing benchmarks motivate long-horizon planning as a critical capability for real-world general intelligent agents.
  • Core challenge: Uncontrollable, opaque Internet training data obscures where planning ability originates and how it can be strengthened.This leaves unresolved whether long-horizon planning is acquired during pre-training or newly developed later.
  • Methodological framework: A unified controlled multi-turn environment enables precise control over environment structure, task length, data quality, planning knowledge, and planning patterns.The environment is designed to study long-horizon planning across the paper’s three training stages.
  • Planning ability acquisition during pre-training: Pre-training analysis examines planning acquisition through data format, distribution, and quality, including world-model internalization, atomic-skill composition, and trajectory quality.Explicit chain-of-thought state-transition modeling produces stronger long-horizon generalization than direct action prediction.
  • Planning ability shaping via GRPO and OPD post-training: Post-training analysis studies planning-pattern shaping with GRPO and OPD, and planning-knowledge shaping, while identifying unnecessary, effective, and unsupported RL regions.Under low-quality pre-training data and long-horizon tasks, OPD has a broader effective boundary than GRPO for general planning patterns.
  • Planning ability integration through MOPD post-training: MOPD integrates multiple teachers by converging toward shared planning-pattern distributions, supporting cross-environment generalization, continual learning, and conflict analysis.The integration study distinguishes shared or compatible planning patterns from cross-environment conflicts.

2. Preliminaries

The paper formulates planning as sequential decision-making toward a goal and defines skills as ordered sub-step mappings, with a world model representing state-transition rules. It then formalizes multi-turn on-policy agentic distillation and distinguishes OPD variants by how teacher feedback and vocabulary distributions are used.

  • Planning Definition: Planning generates executable actions that transition an initial state to a final state satisfying the target goal.The formulation specifies an initial state s0, goal g, action sequence A, and resulting state sT.
  • Agent Skill/Experience/Procedural Knowledge/Script Knowledge: A skill maps a natural-language task description to an ordered sequence of sub-steps.Skills, experiences, procedural knowledge, and script knowledge are represented as tuples (d, P).
  • World Model: The world model learns state-transition rules and internalizes procedural skills within the transition function.It models T(st, at) → st+1 to predict the next state from the current state and action.
  • On-Policy Agentic Distillation: Agent-OPD trains the student directly on self-explored K-turn trajectories, using teacher and student next-token distributions at each turn and token.The trajectory includes the goal, initial state, accumulated context, generated sequence, and resulting states.
  • PG-Style OPD vs GKD-Style OPD: PG-style OPD uses sampled token log-ratios as dense rewards, whereas GKD-style OPD directly minimizes local KL divergence between full predictive distributions.PG-style reweights trajectory sampling probabilities; GKD-style reshapes token-level distributions on already visited prefixes.
  • OPD Vocabulary Variants: Sampled-Token OPD evaluates sampled tokens, Top-k OPD restricts divergence to the student’s highest-probability tokens, and Full-Vocabulary OPD aligns distributions over the entire vocabulary.Sampled-token methods avoid full-vocabulary computation; Top-k uses truncated, renormalized distributions; Full-Vocabulary OPD applies exact KL without masking.

3. Task Formulation

The task formulation builds a controllable multi-domain planning gym from hierarchical skill graphs with logical synthesis rules, then maps these graphs to diverse concrete environments. Agents must synthesize targets from inventories under controlled difficulty, with held-out configurations and repeated evaluation measuring final success.

  • 3.1. Controllable Planning Gym: Hierarchical skill graphs span three domains, with each domain organized into H layers, W categories per layer, and N unique concrete items per category.An LLM generates category trees and concrete item entities to provide rich semantic relationships.
  • 3.1. Controllable Planning Gym: Graph transitions use OR over independent recipes, while each recipe requires an AND combination of its constituent prerequisite items.For target node v, valid pathways follow the form v ⇐⇒ (u_1,1 ∧ · · · ∧ u_1,k_1) ∨ · · · ∨ (u_m,1 ∧ · · · ∧ u_m,k_m).
  • 3.1. Controllable Planning Gym: Greedy minimization limits high-step-count explosion, while disjoint abstract graphs use fixed-probability prerequisite variation to produce non-overlapping rule sets.The algorithm prioritizes combining items with fewer preceding synthesis steps, and each domain contains graphs such as Graph A and Graph B.
  • 3.1. Controllable Planning Gym: Abstract graphs become concrete gyms through index-mapped items, combining fixed and randomized configurations while reserving K item-index combinations per category for testing.The remaining indices are randomly sampled for diverse training environments to prevent data leakage.
  • 3.1. Controllable Planning Gym: Task difficulty equals the minimum synthesis actions needed from the initial inventory to the target, with tasks categorized as Short, Middle, or Long.Targets are sampled from higher layers and starting materials from lower layers, while distractor items inject realistic planning noise.
  • 3.1. Controllable Planning Gym: Pre-training, post-training, and testing use separate configuration splits, with novel familiar-item combinations reserved for testing and high-quality trajectories reserved for post-training.Most training configurations support exploratory pre-training across trajectories, while a smaller subset supports post-training.
  • 3.2. Task Setup: The agent sequentially synthesizes a target within limited steps from prerequisites and distractors, using natural-language category descriptions that hide underlying item IDs.At each step, it selects the correct sequence of materials from the inventory to execute valid synthesis actions.
  • 3.3. Evaluation: Evaluation uniformly samples M tasks across step counts on reserved Graph A test configurations, reporting final success rate over K independent runs.Success requires synthesizing the target while strictly adhering to unseen rules across Short, Middle, and Long difficulty levels.

4. Multi-Turn Long-Horizon Planning Ability Acquisition during Pre-training

Pre-training planning ability depends on explicit internal world-model construction, exposure to long-horizon trajectories, and high-quality demonstrations. Direct answering and atomic skills alone generalize poorly, while small amounts of long-horizon data help connect skills and suboptimal trajectories cause errors to compound over time.

  • World-model internalization: Explicitly modeling intermediate state transitions through CoT enables internal simulation and search over future states rather than relying only on action-sequence imitation.The internal world model represents state transitions and supports depth-first exploration of skill sequences without external interaction.
  • World-model internalization: World-model training consistently outperforms direct answering across difficulty levels, yielding stronger multi-turn long-horizon planning generalization despite slower convergence and higher token use.Short-horizon improvements are +9.4% for avg@8 and +6.9% for pass@8.
  • Atomic compositional generalization: Training exclusively on atomic short-horizon skills produces strong short-task performance but fails to compose them into longer-horizon plans.The short-only condition reaches pass@8 of 93.12% on short tasks, versus 0.83% on middle and 0.00% on long-horizon subsets.
  • Atomic compositional generalization: Introducing a small fraction of long-horizon trajectories activates planning generalization by teaching the model to connect atomic skills sequentially.The passage specifically reports that merely 5% of middle-horizon trajectories causes a sudden improvement on longer levels.
  • Suboptimal trajectories: Multiple optimal reasoning patterns preserve similar generalization, whereas mixing 4 optimal : 8 suboptimal trajectories severely damages long-horizon planning.Suboptimal decisions create errors that accumulate and amplify step by step, driving middle- and long-horizon performance near zero; increasing sampling does not overcome these flaws.

5. Planning Ability Shaping via GRPO and On-Policy Agentic Distillation

The section separates reusable planning patterns from task-specific planning knowledge and identifies when GRPO and OPD can shape each effectively. It finds OPD more stable than GRPO in long-horizon, low-quality settings, while mismatched procedural knowledge can make distillation harmful.

  • Planning patterns and knowledge: Planning patterns are reusable strategies such as reflection or backtracking, whereas planning knowledge contains task-specific skills and procedures tightly bound to the current state.This distinction motivates using mutual information to characterize how strongly task inputs constrain each output.
  • RL applicability regions: RL is unnecessary when planning patterns perform similarly, effective when optimization can discover better patterns, and unsupported when better patterns exist but RL cannot reliably discover them.The three regions are defined by performance gaps among patterns and RL’s ability to find improved patterns.
  • GRPO and OPD stability: GRPO becomes increasingly noisy as task horizons lengthen, while OPD maintains more stable update directions under long-horizon and low-quality pre-training conditions.The instability is attributed to coarse credit assignment in GRPO; OPD shows consistently high subspace alignment under ideal teacher conditions.
  • Knowledge mismatch: Knowledge mismatch blocks OPD: a perfect teacher can reduce the student’s overall pass@8 to 36.60, below the base instruct model’s 44.03.The failure occurs when teacher and student internalize different valid procedural paths, indicating that successful distillation requires aligned internal world modeling.
  • Distillation limitations: Gold-standard data alone cannot teach recovery from errors, and small RL updates are insufficient for point-by-point distillation of large, sample-specific planning knowledge.Error-state prefixes and coverage of multiple correct reasoning paths are needed to avoid treating valid alternatives as incorrect.

6. Planning Ability Integration through Multi-Teacher On-Policy Agentic Distillation

MOPD integrates multiple teachers through sequential on-policy distillation, with outcomes determined by whether planning patterns are shared and compatible across environments. Shared compatible patterns enable cross-environment generalization, shared conflicting patterns can support continual learning, and entirely conflicting patterns cause severe forgetting and interference.

  • Unified analysis framework: MOPD analyzes generalization, continual learning, and conflict modes using shared-pattern and compatibility dimensions across environments.The framework classifies planning-pattern relationships according to whether patterns are shared and whether they are compatible with different environments.
  • Generalization mode: Shared and compatible planning patterns enable cross-environment generalization, even when each teacher is effective only in its own environment.Sequential MOPD transfers capabilities across domains by seeking a subset of the shared teacher distribution rather than covering every teacher preference.
  • Continual learning mode: Shared but conflicting planning patterns can support continual learning when the student preserves earlier domain skills while learning from new teachers.Under the Form c setting, final Low difficulty avg@8 scores remain 91.88 for Fantasy Alchemy and 88.98 for Livestock Farming, while Electronic Assembly reaches 89.77.
  • Conflict mode: When environments have no shared planning patterns and their patterns conflict, sequential MOPD causes severe forgetting as the student learns one expert after another.The results characterize this relationship as incompatible with retaining previously acquired domain capabilities.
  • Update dynamics: MOPD preserves the base model through small, stable parameter updates while learning task signals, remaining close to the instruct model throughout sequential training.The distance between Instruct and MOPD 1 is 4.82 × 10^-1, whereas distances from Instruct to teacher models are around 7.10 × 10^1 to 7.36 × 10^1.

7. Related Work

Related work spans long-horizon agentic planning benchmarks, applications, training methods, OPD, multi-teacher distillation, and the roles of world modeling and compositional generalization. Existing studies emphasize post-training and single-turn OPD, whereas this work provides a unified analysis from pre-training through single-expert and multi-teacher distillation.

  • Long-Horizon Agentic Planning: Long-horizon agentic planning research covers benchmarks, applications, trajectory synthesis, reinforcement-learning algorithms, and agent process reward modeling.Examples include OSWorld 2.0, EdgeBench, DeepPlanning, Agent-A1, Openresearcher, SAO, and TurnOPD.
  • Single-Turn and Multi-Turn OPD: OPD provides denser supervision than standard GRPO, with existing work primarily studying single-turn rather than multi-turn OPD.Early single-turn OPD was proposed as a knowledge-distillation method, while multi-turn OPD remains rarely studied.
  • Single-Turn and Multi-Turn OPD: Existing studies mainly focus on post-training, while opaque pre-training data limits analysis of underlying mechanisms.This work studies pre-training, single-expert distillation, and multi-teacher distillation across data, algorithms, and parameter mechanisms.
  • Multi-Teacher OPD: Multi-teacher OPD is studied as a model-consolidation or continual-learning technique for cross-domain capability integration and algorithmic optimization.Related applications include integration across different foundation models, with MiMo-V2-Flash and DeepSeek-V4 cited as examples.
  • World Modeling and Compositional Generalization: World modeling and compositional generalization are identified as especially important abilities for long-horizon planning.Prior work examines internalizing world models into language models and studying compositional generalization for planning.

8. Conclusion

The work uses a unified, controlled environment to study how multi-turn long-horizon planning is acquired and improved across three training stages. It finds that pre-training requires explicit world-model internalization, some long-horizon data, and high-quality trajectories, while post-training distinguishes planning patterns from planning knowledge.

  • A unified, controlled environment enables systematic study of multi-turn long-horizon planning across three training stages.
  • Pre-training: At pre-training, explicit world-model internalization, limited long-horizon data, and high-quality trajectories are critical for robust long-horizon planning.
  • RL-based post-training: RL-based post-training distinguishes general planning patterns from task-specific planning knowledge, with OPD having a broader effective region than GRP.

Appendix Table of Contents · A Pre-training Details

The appendix includes a table of contents for the pre-training details, covering model, schema, training, and corpus configurations. These subsections span pages 37–39.

  • Appendix Table of Contents: The appendix table of contents introduces the pre-training details section and its organization.It lists four configuration subsections under A Pre-training Details.
  • A Pre-training Details: A.1 documents the pre-trained model configuration on page 37.The entry identifies the model configuration as the first pre-training-details subsection.
  • A Pre-training Details: The listed pre-training configuration topics progress from model and schema settings to training and corpus settings.The table of contents orders these topics across pages 37, 38, and 39.
  • A Pre-training Details: A.4 covers the pre-training corpus configuration on page 39.This is the final listed subsection in the pre-training details appendix.

A.1. Pre-trained Model Configuration

The study uses a compact, randomly initialized language model based on the Qwen2ForCausalLM architecture, with approximately 100 million parameters and a byte-level BPE tokenizer.

  • A.1. Pre-trained Model Configuration: The randomly initialized model follows Qwen2ForCausalLM specifications and contains approximately 100 million (100M) parameters.The model is named Qwen2.5-100M.
  • A.1. Pre-trained Model Configuration: The accompanying tokenizer uses byte-level BPE tokenization.

A.2. Pre-training Schema Configuration

The pre-training schema defines a controlled, multi-domain benchmark with hierarchical compositional item generation and structured dependency graphs. Increasing compositional depth and injected noise make long-horizon planning progressively harder while discouraging trivial pattern matching.

  • Schema Coverage: The corpus spans Fantasy Alchemy, Livestock Farming, and Electronic Assembly, with 200 item nodes and 4000 item names per domain across 5 hierarchical levels.Each level contains 40 item categories and 20 instances per category.
  • Compositional Generation: Long-level items are recursively generated from shorter-level components using compositional schemas based on AND and OR operators.This organizes item categories around compositional construction rather than isolated atomic skills.
  • Dependency Structure: The dependency structure uses 30% one-input dependencies and 70% two-input compositions, producing a sparse but non-trivial item graph.The fixed degree distribution imposes a controlled structural prior over composition difficulty.
  • Hierarchical Depth: Compositional depth increases construction complexity, from an average of 1.82 atomic components at Level 2 to up to 5.78 for Level 5 items traced to Level 1 sources.The increasing number of required atomic components creates progressively longer construction paths from base materials.
  • Noise and Robustness: Each instance mixes target-relevant items with noise items, requiring recovery of the correct compositional structure under partial observability.This design prevents trivial pattern matching and supports controlled evaluation of long-horizon compositional planning.

A.3. Pre-training Configuration

The pre-training configuration builds controlled planning datasets from three synthetic domains, with paired recipe-graph splits sharing item names but differing in structure. It systematically constructs inventories and partitions instantiations into pre-training, post-training, and held-out test pools by plan length.

  • Synthetic domains and recipe splits: Datasets span electronic assembly, livestock farming, and fantasy alchemy, with split A and split B using shared item names but different recipe graphs.Each domain is generated from shared item configurations, concrete instantiations, and two recipe schemas.
  • Sample construction: Each sample’s initial inventory combines recursively required materials with two to four distractor items sampled from the same starting layer.The builder enumerates non-base targets and feasible starting layers for both recipe graphs before constructing the inventory.
  • Dataset partitioning: The first 20 instantiations form the test pool, indices 20–219 form the post-training pool, and all remaining instantiations form the pre-training pool.Post-training and test samples are additionally grouped into short plans of one to five steps, mid plans of six to eight steps, and long plans of at least nine steps.
  • Dataset metadata: Dataset filenames encode the synthetic domain, recipe split, subset, trajectory source, and whether samples belong to the full teacher_all pretraining set.The naming convention distinguishes pretrain, posttrain, and test subsets and identifies whether recipe graph A or B produces the gold trajectory.

A.4. Pre-training Corpus Configuration

The pre-training corpus uses distinct expert teacher datasets and broad multi-domain student datasets, with all experiments sharing a fixed full-parameter SFT setup on Qwen2.5-100M.

  • Corpus composition: Each teacher dataset contains exactly 622,960 samples focused on one of three fields across three difficulty levels, while student datasets span all three domains.The teacher fields are fantasy alchemy, livestock farming, and electronic assembly.
  • Training configuration: Experiments use 2 GPUs, per-device batch size 128, 2 gradient accumulation steps, learning rate 1.0e-4, 9000 training steps, 0.1 warmup ratio, and bf16 precision.Training uses a cosine scheduler and full-parameter SFT on the same Qwen2.5-100M base model.
Loading 2607.24720v1…