PaperlayerView exact revision on arXiv

Source-linked AI summary

From Trainee to Trainer: LLM-Designed Training Environment for RL with Multi-Agent Reasoning

Chao Chen, Chengzu Li, Zhiwei Li, Yinhong Liu, Zhijiang Guo

arXiv:2606.17682v1cs.CL

TL;DR

RL training still depends on manually redesigned environments, so this paper lets the current policy analyze failures and redesign its next training distribution. On MAPF-FrozenLake, the framework achieves the strongest aggregate performance, outperforming larger proprietary LLMs and fixed-environment baselines.

  • Problem

    RL training environments are still largely redesigned manually, while it remains unclear whether policy models can proactively redesign the generators defining their future training distributions.

  • Method

    A closed-loop environment engineer uses structured training, validation, and environment statistics to modify generator parameters for the next RL stage.

  • Results

    Across 3-, 4-, and 5-agent benchmarks, the framework achieves the highest aggregate valid and optimal rates, outperforming commercial and fixed-configuration baselines.

  • Takeaways & Limitations

    Successful redesign relies on failure evidence and preserving effective configurations, while RL training improves the checkpoint’s ability to diagnose its remaining weaknesses.

  • Takeaways & Limitations

    The study evaluates one self-contained task family and a specific RL pipeline, so redesign strategies may not transfer directly to qualitatively different domains or training paradigms.

Abstract

from arXiv · show

Reinforcement learning pipelines for Large Language Model (LLM) training often rely on manually redesigned environments between stages, requiring practitioners to heuristically infer which configuration will best improve the current policy. To automate this process, we propose the LLM-as-Environment-Engineer framework in which the current policy model analyzes failure trajectories together with contextual information and proposes modifications to the next-stage training environment configuration. We also introduce MAPF-FrozenLake, a controllable testbed whose generator exposes multi-dimensional environment configurations, making it suitable for studying and benchmarking environment redesign. On this testbed, we condition the environment engineer on structured summaries of policy behavior, failure cases, and environment statistics, from which it produces the configuration for the next training stage. With Qwen3-4B as the backbone, our framework achieves the strongest aggregate performance on our benchmarks, outperforming larger proprietary LLMs (e.g., GPT, Gemini) and fixed-environment training baselines. We further analyze which forms of context are most effective, finding that successful environment updates rely on failure evidence and preserve configurations that already work. Interestingly, the current RL checkpoint serves as a better environment engineer than the original base model, suggesting that policy learning improves the model's ability to diagnose its remaining weaknesses.

1. Introduction

The paper frames environment design as a major but largely manual determinant of RL training outcomes and asks whether policy models can redesign their future training environments. It proposes a closed-loop LLM-as-Environment-Engineer framework and the controllable MAPF-FrozenLake testbed to study this process.

  • Motivation: Environment design affects the behaviors, failures, exploration signals, optimization efficiency, generalization, and capabilities encountered during LLM RL training, yet remains largely manual.Existing adaptation methods mainly select examples, schedule difficulty, or synthesize data within fixed environment families.
  • Framework: The LLM-as-Environment-Engineer framework has the current policy analyze structured behavior, validation-failure, and environment-statistics summaries before proposing the next-stage environment configuration.The model modifies generator parameters rather than directly synthesizing or selecting individual training examples.
  • Challenges: Effective redesign must distinguish sparse, configuration-dependent failure modes because overly difficult environments can collapse learning signals while overly easy ones cause saturation and weak exploration.Aggregated reward may hide distinct failures across training-distribution regions, making evidence-based redesign necessary.
  • Testbed: MAPF-FrozenLake provides a controllable multi-agent grid testbed with deterministic signals for path validity, optimality, and total trajectory length.The testbed supports controlled analysis of how environment redesign affects downstream RL learning dynamics.
  • Contributions: A 4B Qwen3 model under the framework outperforms carefully designed curricula and much larger proprietary GPT and Gemini models as environment designers.The paper investigates how feedback signals, self-diagnosis, and redesign behaviors relate to successful downstream adaptation.

2. Related Work

Prior work connects this framework to curriculum learning and model self-improvement. Curriculum methods organize training difficulty, while self-improvement methods let models generate training signals with multi-model or single-model paradigms.

  • Curriculum Learning: Curriculum learning improves training efficiency by controlling the difficulty or ordering of training experience across domains.In reinforcement learning, approaches often assume predefined task sets and learn or specify a curriculum over them.
  • Curriculum Learning: When predefined task sets are unavailable, curricula can be generated automatically by identifying tasks near the current policy’s capability boundary.
  • Self-Improvement: Self-improvement methods reduce reliance on external supervision by allowing models to generate training signals such as tasks, responses, critiques, rewards, or opponents.
  • Self-Improvement: Existing self-improvement approaches mainly use multi-model methods, which train models jointly, or single-model methods.In multi-model methods, one model provides challenges or feedback for another.

3. Method

The method closes the RL training loop by having the current learner checkpoint redesign future environments from validation evidence and contextual modules. MAPF-FrozenLake exposes controllable configuration dimensions for studying this process, while six context variants culminate in the adopted V6 framework.

  • Testbed and configuration: MAPF-FrozenLake generates multi-agent grid instances with configurable map-size distributions, conflict ratios, and hole densities, and evaluates validity and total steps.Agents must reach goals without collisions or falling into holes, with wait actions available to resolve conflicts.
  • Testbed and configuration: Each map size specifies data, hole, and wait ratios, producing the round-specific training set from configuration C_n; the initial configuration is randomly sampled.The data ratios sum to 1, while hole and wait ratios control map difficulty and conflict resolution requirements.
  • Closed-loop training: Each round follows train → eval → design: the current learner checkpoint reads validation results, proposes C_n+1, and the generator produces the next training data.The loop validates the updated learner, records configuration–evaluation history, composes context, projects the proposed configuration to enforce constraints, and repeats.
  • Context design: The design context combines failure breakdowns, task guidelines, configuration–failure history, model-generated summaries, and RL training details across six evaluated variants.Failure breakdowns include aggregate and per-map-size validity outcomes and failure types; V4 removes the random round-0 configuration, while V5 and V6 add summaries and training details.
  • Context design: V6 is adopted as the final framework after comparing the per-setting results.V6 adds the model-generated summary and training-details modules on top of V4.

4. Experiments and Analysis

Experiments evaluate staged environment redesign on 2-agent training data and separate 3- to 5-agent generalization benchmarks using valid and optimal rates. The proposed Qwen3-4B framework achieves the strongest aggregate performance, while analyses show that failure evidence, stage-aware training context, and independent self-correction improve environment updates.

  • Data Construction and Evaluation: Training uses 4000 two-agent instances per round, generated across 3×3–10×10 grids with configurable data, hole, and wait ratios.Round 0 uses a randomly sampled configuration to produce the initial training data.
  • Data Construction and Evaluation: Generalization is evaluated on 3-, 4-, and 5-agent instances across wait-ratio subsets wr_025, wr_050, and wr_075, using valid rate and optimal rate.The benchmark samples five hole ratios across map sizes; valid rate measures legal parsing, while optimal rate measures finding an optimal solution.
  • Main Results: Across all three agent counts, Qwen3-4B + GRPO + Ours achieves the highest aggregate Sum on both metrics, surpassing closed- and open-source baselines.Against Kimi-K2.5, the framework improves valid rate by +5.20 to +6.19 points and optimal rate by +2… points across the 3- to 5-agent benchmarks.
  • Behavioral Analysis: Effective context settings condition environment updates on observed failure patterns rather than surface map-size templates, while training details help V6 identify useful learning signals.V6 plateaus at 7×7–9×9 and drops at 10×10 on hole and wait ratios, unlike monotone size-driven variants.
  • Behavioral Analysis: Self-correction requires evidence independent of the model’s previous narration, because self-summaries can override raw failure breakdowns and preserve ineffective templates.V4 R2 and V6 R2 edit directly from raw failures, whereas V5 R2 relies on its own R1 self-summary.
  • Ablation and Checkpoint Analysis: Bookkeeping-only training context outperforms full RL-detail context on every agent count for both valid and optimal rates, and the current RL checkpoint outperforms the base model on all benchmarks.The bookkeeping-only setting supplies round and epoch information without detailed optimization parameters; with other components fixed, the checkpoint wins across 3-, 4-, and 5-agent evaluations.

5. Conclusion

The paper introduces a closed-loop framework in which an LLM redesigns its own RL training environment, enabled by the controllable MAPF-FrozenLake testbed. A 4B policy model guided by structured feedback outperforms larger proprietary LLMs, while successful redesign relies on evidence-driven adaptation and improved self-diagnosis from RL training.

  • Framework and testbed: The framework uses an LLM as an environment engineer that proactively redesigns training configurations for its own RL learning.This creates a closed-loop process for policy-conditioned environment engineering.
  • Framework and testbed: MAPF-FrozenLake provides a controllable testbed, where a 4B policy model guided by structured feedback iteratively proposes configurations that consistently outperform larger proprietary LLMs on multi-agent path-finding tasks.The comparison includes larger proprietary LLMs, while the benchmark covers Multi-Agent Path Finding tasks.
  • Mechanistic findings: Successful environment redesign depends on evidence-driven adaptation rather than naive difficulty maximization, while RL training improves the model’s ability to diagnose its own weaknesses.These findings support studying self-improving learning systems through policy-conditioned environment engineering.

A. Validity checks for the accuracy reward

The accuracy reward R_acc is binary: a response succeeds only if it passes all eight validity checks; failing any one sets R_acc = 0.

  • Validity criterion: R_acc equals 0 whenever any of the eight validity checks fails.A response must pass every check to count as a success.
  • Eight checks: The checks require parsable paths, legal moves, conflict freedom, goal-reaching, correct starts, matching agent counts, hole-free paths, and in-bounds paths.Legal moves have Manhattan distance ≤1; paths must avoid vertex or edge conflicts, holes, and grid boundaries.

B. Per-setting valid rate of the six context variants

Across the three evaluation benchmarks, V6 achieves the highest total valid rate among all six context settings, aggregated across 3-, 4-, and 5-agent instances and all map sizes.

  • Per-setting valid rate: V6 achieves the highest valid rate on every benchmark among the six context settings.The rates aggregate across 3-, 4-, and 5-agent instances and all map sizes.

C. Training details

Training uses GRPO on Qwen3-4B with large trajectory batches across four H100 GPUs, followed by multi-round optimization with fixed actor, reference-policy, and systems configurations.

  • Batching and hardware: Each prompt yields 𝑛=8 trajectories, producing 1024 samples per mini-batch with 8 gradient-accumulation steps across 4× H100 80 GB GPUs.Sampling uses vLLM with tensor_parallel=2, gpu_memory_utilization=0.55, max_prompt_length=1536, and max_response_length=5120; the per-GPU micro-batch is 4.
  • Optimization setup: The actor uses AdamW with a constant learning rate of 2×10−6 and a β=10−3 KL penalty in the actor loss, without reward KL or entropy regularization.GRPO requires no critic.
  • Training schedule and systems: Each round runs three RL epochs over 4000 training samples, and the pipeline runs for three rounds using VERL v0.6.1, PyTorch 2.8, CUDA 12.4, and vLLM 0.10.3.The reference policy is frozen from the start of each round with FSDP parameter offload; the actor uses FSDP without offload and gradient checkpointing.

D. Prompts used by the environment engineer

At each training round, the environment engineer analyzes evaluation results and training history to propose a next-round YAML configuration, while a separate summary prompt records observations, rationale, and monitoring targets. The prompts constrain redesign to 2-agent data and emphasize failure evidence, per-size performance, continual learning, and the fixed 4000-sample budget.

  • Prompt workflow: The two prompts are used at every round: the analysis prompt emits the next configuration, and the summary prompt feeds a concise round note back as S-module context.Both prompts operate within a two-agent training setup, even though evaluation tests generalization to tasks with more agents.
  • Analysis prompt: The analysis prompt uses overall and per-map-size evaluation breakdowns, training history, and current data distribution to identify lagging map sizes and failure types.It distinguishes conflict, parse, illegal-move, goal, hole-collision, and out-of-bounds failures.
  • Analysis prompt: The prompt preserves progressive easy-to-hard learning and requires all eight map sizes to remain included while their ratios sum to 1.0 under a fixed 4000-sample budget.Increasing one map-size ratio implicitly reduces another, so curriculum changes reallocate rather than expand the dataset.
  • Analysis prompt: The engineer may change ratio, hole_ratio, and wait_ratio for any map size, outputting only desired changes as a fenced YAML block for the next training round.The configuration controls map-size allocation, obstacle density, and WAIT-action coverage.
  • Summary prompt: The summary prompt condenses each completed round into 3–5 sentences covering current evaluation observations, the new configuration’s rationale, and what to monitor next.It asks for specificity about map sizes, failure types, and parameter changes.

E. Limitations

The study is limited by its evaluation on a single, self-contained task family and a specific RL training pipeline. These constraints may limit transferability and leave interactions with other training approaches unresolved.

  • Task and pipeline scope: MAPF-FrozenLake represents a single, self-contained task family, so learned redesign strategies may not directly transfer to domains with qualitatively different failure modes or evaluation signals.Although the testbed offers fine-grained control over training distributions, its task-family scope constrains conclusions about broader domains.
  • Task and pipeline scope: The experiments focus on a specific RL training pipeline, limiting what the study establishes about environment redesign beyond that setting.The passage identifies this focus as a limitation but is truncated before specifying the other training approaches considered.
Loading 2606.17682v1…