PaperlayerView exact revision on arXiv

Source-linked AI summary

Verifiable Hidden Dynamics Play: Generating Agentic RL Environments from Solved Mechanisms

Xinjie Shen, Wei Fan, Xudong Guo, Jianhong Tu, Yang Su, Chuqiao Kuang, Yinger Zhang, Dayiheng Liu

arXiv:2609.27321v1cs.AIcs.CL

TL;DR

Language-model agents need diverse, long-horizon environments with dependable outcome signals, but environment generation often leaves dynamics and evaluation to be aligned afterward. VHD-Play solves a mathematical model before rendering it as a corpus-grounded stateful environment, deriving dynamics and trajectory scoring from that shared model. Training improves agentic performance across generated and external tasks, while the reported evidence has scope limits.

  • Problem

    Generating diverse agentic environments with dependable outcome signals at low extension cost remains difficult because pipelines commonly define evaluation after constructing the environment.

    Source: §3 · p. 3

  • Method

    VHD-Play samples and solves a mathematical model, then uses a corpus-grounded setter to realize its dynamics and interface while deriving trajectory evaluation from the same model.

    Source: §3 · p. 3, §3.3 · p. 5

  • Results

    Training raises mean agentic score across five optimization families from 0.204 to 0.815 and transfers to held-out and unseen families, tool use, travel planning, and long-horizon e-commerce.

    Source: §1 · p. 2, §5.1 · pp. 6–7

  • Takeaways & Limitations

    The results support using solved mechanisms to extend verifiable reinforcement learning from final answers toward stateful trajectories across the tested mechanisms and benchmarks.

    Source: §6 · p. 8

    Generated-family results use one training run and one evaluation seed, and the reported implementation trains on terminal outcomes without intermediate supervision.

    Source: K Limitations and scope · p. 28

Abstract

from arXiv · show

Language-model agents increasingly face long-horizon tasks with evolving state, interdependent decisions, and delayed outcomes. Scaling their training requires diverse agentic environments, dependable outcome signals, and low extension cost. Existing generation pipelines commonly construct an environment before defining its outcome rule or annotating its trajectories, leaving dynamics and evaluation to be aligned post hoc. VHD-Play reverses this dependency by sampling and solving a mathematical model before a corpus-grounded setter renders its decision process as stateful tools. The executable dynamics and trajectory-scoring reference are inherited from the same solved model. The pipeline produces 3,300 diverse agentic environments at a cost of a few cents each. Training Qwen3.6-35B-A3B on three families raises its mean agentic score from 0.204 to 0.815 in a five-family diagnostic. Gains also appear on held-out instances from all three training families and eight unseen mechanism families, then extend beyond the generated substrate to external benchmarks for general function calling, travel planning, and 365-day e-commerce. On E-Commerce Bench, the trained checkpoint completes every run without bankruptcy and exceeds Qwen3.7-Max. We compare written-out problems with stateful versions that reveal or hide their parameters. The comparison shows that most of the learnable gap lies in stateful interaction rather than underlying problem solving. A frozen 35B setter realizes larger environments, and scale-matched training retains gains as mechanism size and horizon grow, indicating the potential for an evolving training substrate.

1 Introduction

Language-model agents increasingly need to act through evolving, long-horizon workflows, but existing approaches address environment construction and dependable outcome signals separately. VHD-Play solves a mathematical model first, then uses it to generate stateful environments and outcome signals; training improves performance across agentic tasks and external benchmarks.

1 Introduction

  • Long-horizon workflows require agents to act within evolving processes, where information gathered earlier and commitments made can shape later choices.The introduction notes that reliability declines as expert time and interaction horizons grow, and that written-out and agentic forms show a performance gap.Source: §1 · p. 1
  • Existing approaches provide exact answer-based signals, custom interactive environments, human feedback, learned judges, or generated tools, but do not jointly solve diversity, dependable outcomes, and low extension cost.Generative pipelines commonly build the environment before fixing its reward or evaluation rule, leaving dynamics and evaluation to be aligned afterward.Source: §1 · pp. 1–2
  • VHD-Play solves a sampled mathematical problem first, then uses a corpus-grounded setter to render its decisions as stateful tools and its solution as the outcome signal.The player does not see the model parameters or solution and must recover relevant information through interaction.Source: §1 · p. 2
  • Training Qwen3.6-35B-A3B raises mean agentic score across five optimization families from 0.204 to 0.815, with gains also reported on held-out, unseen, and external tasks.The reported external results include a 3.4× mean ending balance on a 365-day storefront, ten interaction-focused BFCL V4 cells improving by 2.84 points, and preserved written-out problem solving.Source: §1 · p. 2

2 Related Work

Prior work offers verifiable answers or plans, formal interactive environments, and generated tasks with varied feedback, but these approaches leave different parts of agentic environment construction unresolved. VHD-Play combines sampled mechanisms and corpus grounding with a computable reference outcome.

Formal and verifiable reasoning

  • Answer-based QA and search provide inexpensive exact signals but do not require agents to operate within changing state; formal environments provide exact dynamics but retain domain specifications.SATLM and LLM+P delegate inference to formal solvers, while text games, PDDL environments, and planning domains supply formal interactive settings.Source: §2 · p. 2

Generated agentic environments

  • Generated tools and environments broaden task coverage and reduce authoring effort, while learned or generated graders require validation and dynamics–evaluation agreement is often established afterward.Work on partial observability and long-horizon execution sharpens behavioral targets but often assumes fixed environments.Source: §2 · pp. 2–3
  • VHD-Play bridges these approaches by pre-solving each sampled mechanism and retaining its reference outcome for diverse agentic instances with hidden dynamics and graded scoring.This combines computable outcome references with corpus-grounded generation rather than requiring a fixed domain specification for each instance.Source: §2 · pp. 2–3

3 Formulation

VHD-Play treats an agentic environment and its outcome signal as artifacts derived from one solved mechanism. The formulation separates the model, executable dynamics, and policy-facing interface, making information exposure and trajectory scoring explicit.

3 Formulation

  • The construction problem is joint: the environment determines possible trajectories, while its evaluator determines their value, so separate generation leaves their agreement to be established afterward.VHD-Play instead begins with a mathematical model and renders its transitions, information structure, constraints, and objective as an environment.Source: §3 · p. 3

3.1 Construction order

  • Environment-first construction fixes the interactive process before defining an outcome rule or annotating trajectories; VHD-Play solves M(θ) first to fix zθ and Rθ before realizing D(θ) and wrapping it as E(θ).The process and evaluation in VHD-Play both descend from the sampled model.Source: §3.1 · pp. 3–4

3.2 Model, dynamics, and interface

  • A mechanism draw specifies a mathematical model M(θ), executable dynamics D(θ), and wrapped environment E(θ), separating decisions and constraints from transitions, utility, observations, actions, and turn budget.D(θ) contains state space, transition, and utility; E(θ) adds an observation map, action set, and budget of T turns.Source: §3.2 · p. 4
  • The interaction process links policy actions to state transitions, observations, and trajectory utility over a finite turn horizon.The interface may limit observations to views of hidden state or parameters, while probe and decision actions both consume the shared budget.Source: §3.2 · p. 4
  • The builder receives the complete draw, while the policy sees only interface-published observations; separating model from interface allows the same problem to be presented in written-out or agentic form.A frozen setter can realize and wrap the model rather than inventing the governing structure and evaluation jointly; inventory control illustrates hidden demand, shared capacity, and persistent commitments.Source: §3.2 · pp. 4–5

3.3 Outcome evaluation

  • A family solver computes optimal and default-policy reference values before interaction, and the episode-level reward scores realized utility against those fixed references on a clipped 0-to-1 scale.The optimum is an upper reference and need not be attainable by an online policy under partial information; no learned evaluator estimates the endpoints.Source: §3.3 · p. 5
  • The construction combines stateful interaction, a verifiable optimum, a reference-grounded score, and an explicit information boundary, with instances diversified by resampling rather than reauthoring.The following sections test new draws, mechanism families, larger horizons, and the difference between written-out and agentic forms.Source: §3.3 · p. 5

4 Producing environments at scale

The generation pipeline turns solved model instances into corpus-grounded environments, then admits candidates only after execution and reference checks. Its construction supports varied scenarios and mechanisms without reauthoring each instance.

Sampling and solving

  • Each instance samples model parameters and solves the model without a language model, then maps decisions, constraints, and objectives into executable state updates and terminal utility.Closed-form procedures, enumeration, dynamic programming, or numerical optimization compute the reference outcomes; the corpus seed varies setting and interface.Source: §4 · p. 5

Corpus-grounded realization

  • Table 1 summarizes the generated substrate’s scale and structure, distinguishing scenario seeds that diversify setting and language from mechanism families that determine dynamics and outcome semantics.The table caption points to appendices for family definitions, verification coverage, cost reconstruction, and full distributions.Source: Table 1 · p. 6
  • A frozen setter receives the complete model draw and a sampled real-world corpus passage, then generates the scenario, database, tools, and player instructions while the model determines decisions and evaluation.The starting Qwen3.6-35B-A3B checkpoint can already produce valid environments in the setter role.Source: §4 · p. 6

Admission and scale

  • Admission executes generated candidates and checks agreement with the precomputed construction; across the held-out audit, sampled environments executed successfully and realized outcomes remained within stored references.Scenario seeds span 28 topical domains with normalized entropy 0.997, and episodes average 59.7 assistant turns.Source: §4 · p. 6

Policy optimization

  • Admitted environments provide trajectories and fixed episode-level signals for GRPO training, while the mechanism-first construction can also be paired with other policy optimizers.The implementation optimizes groups of rollouts from the same instance.Source: §4 · p. 6

5 Experiments

The experiments test transfer across generated and external tasks, isolate the learnable gap between written-out and stateful problem solving, and examine whether training gains persist as environments scale. Results show gains across held-out and unseen mechanisms, external benchmarks, and larger knapsack configurations.

5.1 Training gains and transfer

  • Held-out families gain +0.56 on average, near-OOD families +0.63, and far-OOD groups +0.24 and +0.16; E-Commerce mean ending balance reaches 3.4× Base and exceeds Qwen3.7-Max.The trained checkpoint completes all five storefront runs without bankruptcy.Source: §5.1 · pp. 6–7, Figure 4 · p. 7

5.2 A learnable gap under stateful interaction

  • Base scores 0.962 in F but 0.231 in I and 0.204 in A; training gains are (+0.030, +0.644, +0.611), closing 77% of the F-to-A gap.I reveals parameters upfront, whereas A requires acquiring them through interaction; both use the same multi-step environment and outcome scale.Source: §5.2 · p. 7, Table 2 · p. 7
  • Paired trajectories illustrate myopic cross-period allocation and premature action under incomplete observation, while the trained policy plans across the horizon or reveals all items before planning.In the seven-period allocation example, the base policy spends 27 of 29 shared-budget units in the first three periods and scores 0; the trained policy scores 1.Source: §5.2 · p. 8
  • Fixed-stratum analysis attributes +0.030 of the +0.443 agentic gain to increased tool-use adoption, leaving +0.413 within fixed strata.The gain coincides with a held-out-state policy shift of DKL(πtrained∥πBase) = 0.089 nats per assistant token.Source: §5.2 · p. 8

5.3 Co-scaling environment and policy frontiers

  • A frozen Qwen3.6-35B-A3B setter realizes admitted environments while a second copy trains from Base and is evaluated on fresh environments at matched scales.The sweep scales knapsack tasks from 6 items over 5 periods to 11 items over 11 periods.Source: §5.3 · p. 8
  • Across four knapsack scales, gains rise +0.412, +0.551, +0.675, and +0.861; the two larger scales average +0.768 versus +0.482 for the smaller pair.At 6 × 5, Base and trained scores are 0.432 and 0.843; at 11 × 11, they are 0.085 and 0.946.Source: Figure 5 · p. 8, §5.3 · p. 8

6 Conclusion

VHD-Play derives stateful environments and outcome signals from solved mechanisms. Experiments show training gains across tested mechanisms and external tasks, while setter and scale experiments indicate potential for expanding the training substrate.

6 Conclusion

  • Mechanism-first construction turns solved models into diverse stateful environments with solver-derived outcome signals.Corpus grounding supplies settings, while asymmetric interfaces require information acquisition and consequential action.Source: §6 · p. 8
  • Across tested mechanisms and external benchmarks, training improves policy performance under changing state while preserving written-out problem solving.A fixed setter realizes new draws, and larger mechanisms and horizons continue to provide training headroom under the same evaluator.Source: §6 · p. 8
  • By deriving dynamics and outcome rules from a shared solved mechanism, VHD-Play extends verifiable reinforcement learning from answers toward trajectories.The paper presents this as a practical framework for expanding agentic training environments as policy capabilities develop across broader task settings.Source: §6 · p. 8

A Discussion

The discussion frames VHD-Play as a way to align environment dynamics and outcome signals through a shared formal mechanism. It describes replay-based admission, model-derived references, and corpus-grounded setter construction, while limiting its claims to feasibility and learnability over the evaluated families.

A Discussion

  • VHD-Play assigns corpus seeds to setting and language, mechanisms to state transitions and outcome semantics, and interfaces to what policies can observe before acting.A precomputed reference anchors the outcome signal.Source: A Discussion · p. 13
  • The three-presentation comparison finds a sharp gap between written-out mathematical competence and stateful policy performance, with much of the gap remaining even when parameters are revealed.The discussion connects training gains to information acquisition, cross-period decisions, and exploration cost, and notes that fixed-stratum analysis separates gains from increased code-tool use.Source: A Discussion · p. 13
  • The authors limit their evidence to feasibility and learnability over evaluated families, not proof that policies reconstruct underlying models or that operations research suffices for agentic competence.They identify parameterized variation, executable realization, controlled information asymmetry, and a pre-interaction outcome standard as properties that other formal mechanisms could share.Source: A Discussion · p. 13

B Admission and verification

  • Admission checks executability, a family-appropriate default outcome, and agreement between replayed and precomputed optimum outcomes where replay is supported.In a 480-execution audit, no run raised an exception, all 240 repeated terminal outcomes agreed, and no replayed raw utility exceeded its precomputed optimum.Source: B Admission and verification · pp. 13–14

C.1 Reference computation

  • Reference procedures span closed-form methods, enumeration, dynamic programming, and numerical optimization; the reported families compute references without calling a language model.Family registration reuses a parameter sampler, reference procedure, and realization or replay adapter across parameter and corpus draws.Source: §C.1 · p. 14

C.2 Setter passes

  • The setter receives a separately sampled corpus passage and the complete parameter vector, then emits a scenario, database, tools, instructions, default policy, and output format.The source corpus spans 28 topical domains, and candidates failing execution or interface alignment are refined or regenerated.Source: §C.2 · p. 14

C.3 Setter capacity

The section tests setter feasibility, describes isolation and training choices, and traces how solved mechanisms become representative stateful environments. It also locates the method within a framework separating environment construction, feedback, and optimization.

C.3 Setter capacity

  • 21 of 22 emitted records preserved the reference under compatible replay, yielding 70% admission; Qwen3.7-Max passed all 30, with fewer attempts and no interface refinements.Both setters cost roughly $0.01–$0.03 per admitted environment in the available reconstruction.Source: §C.3 · pp. 14–15, Table 3 · p. 15

C.4 Environment and interpreter isolation

  • The player interface exposes declared operations but withholds latent values and references; written-out and agentic forms differ in parameter access, not computation access.Each environment retains its sampled draw and reference pair, while the shared computation utility cannot access environment internals.Source: §C.4 · p. 15

C.5 Training implementation

  • Training runs GRPO for 34 optimizer steps, consuming 2,176 environments and producing 34,816 scored rollouts.Each step uses 64 prompts, 16 retained rollouts per prompt, and 18 attempts through oversampling.Source: §C.5 · p. 15

D Representative generated environments

  • Three admitted records trace how inventory control, routing, and knapsack become operational tasks in hardware, education, and adaptive sports.Figure 6 highlights each task’s information-acquisition and commitment pattern, persistent state, and solver-backed advantage over the fixed default.Source: D Representative generated environments · pp. 15–16, Figure 6 · p. 16

D.1 Inventory control as hardware replenishment

  • The inventory task turns joint replenishment into six days of ordering with shared capacity, setup costs, hidden demand, and irreversible lost sales.Its sampler maps orders to actions, inventory to persistent state, constraints to transitions, and objective terms to utility; compatible replay reproduces the solver-derived reference.Source: §D.1 · pp. 16–17, Table 5 · p. 17

D.2 Routing as education logistics

  • The education-routing task combines probing hidden site demand, dispatching capacity-limited routes, and managing backlog across a three-day horizon.Held–Karp routing plus dynamic programming computes the optimum; the candidate’s reference policy reproduces it through the generated tools.Source: §D.2 · p. 17

D.3 Knapsack allocation as adaptive-sports planning

  • The adaptive-sports task allocates equipment across nine days under shelf capacity and a shared budget, with hidden daily revenue revealed by probing.An exact two-level dynamic program computes the reference, which the candidate reproduces through the generated tools.Source: §D.3 · p. 17

E Construction, feedback, and optimization

  • The framework separates environment construction, outcome provenance, feedback localization, and policy optimization; these choices need not move together.Its four axes identify where dynamics and interfaces originate, what defines success, where assessment attaches, and how feedback updates a model.Source: E Construction, feedback, and… · p. 17, Table 6 · p. 18

E.1 Construction order and method coverage

VHD-Play’s construction order fixes the outcome standard before realizing the environment, unlike environment-first pipelines. A matched comparison with EnvScaler and a discussion of downstream feedback and optimization distinguish construction provenance from later training choices.

E.1 Construction order and method coverage

  • VHD-Play reverses environment-first construction: solving M(θ) first fixes zθ and Rθ, then realization produces D(θ) and E(θ).The mechanism guides realization, and the precomputed standard provides a reference for checking it.Source: §E.1 · p. 18

E.2 A released environment-first comparison

  • Under the matched 35B starting checkpoint and update budget, VHD-Play improves all three external aggregates, while EnvScaler’s transfer is less uniform.EnvScaler raises TravelBench plan quality from 0.700 to 0.753, but its BFCL ten-cell mean is 58.56 against Base’s 61.25 and mean storefront balance is 52,661 versus 54,294.Source: §E.2 · p. 19, Figure 7 · p. 20, Table 8 · p. 20

E.3 Feedback and optimization

  • Mechanism-first construction fixes environment and evaluation provenance, but leaves feedback placement and policy optimization to downstream choices.The reported realization uses a normalized scalar terminal reward and GRPO; other options include process signals and PPO.Source: §E.3 · p. 21

F Evaluation settings

The evaluation uses a 3,300-environment corpus split across training and held-out families, and describes the mechanisms and reference procedures used across eleven families.

F Evaluation settings

  • The 3,300 environments comprise 2,200 training instances, 300 held-out instances from three training families, and 800 instances across eight unseen families.Each unseen evaluation family contributes 100 environments.Source: F Evaluation settings · p. 21, Table 9 · p. 21
  • Generated-family episodes have a median of 31 assistant turns, a mean of 59.7, and a 95th percentile of 211.Episodes are capped at 1,000 turns, with per-turn generation capped at 16,384 tokens and cumulative generation at 120,000 tokens.Source: F Evaluation settings · p. 21

G Mechanism families

  • Table 10 catalogs eleven mechanism families and their sampled mathematics, decision structures, reference procedures, and evaluation bands.Sequential stopping and search use hindsight bounds unattainable by online policies, so those rows compare the two arms rather than distance to an attainable optimum.Source: G Mechanism families · p. 21, Table 10 · p. 22

H.1 Training dynamics

The reported checkpoint improves on external benchmarks, while diagnostics and paired trajectories describe gains in stateful interaction, horizon-wide planning, and information use. Training curves document optimization, not transfer.

H.1 Training dynamics

  • Training rewards rise for Inventory DP from 0.408 to 0.904 and routing from 0.150 to 0.898, while negotiation reaches 0.470 from near zero.These training-instance curves show optimization progress, but cannot distinguish stateful-policy learning from fitting training instances.Source: §H.1 · p. 22

H.2 Generated-family results

  • Training reduces the mean written-out-to-agentic shortfall from 0.758 to 0.177 while retaining near-ceiling written-out performance.The reported forward KL between trained and Base policies is 0.089 nats per token on held-out trajectory states.Source: Table 11 · p. 23, §H.2 · pp. 23–24

H.3.1 Horizon-wide planning

  • The trained policy plans all seven period-level selections under the shared budget before committing, unlike Base’s period-by-period decomposition.In this paired case, Base scores 0 and the trained policy scores 1.Source: §H.3.1 · p. 24

H.3.2 Information acquisition before commitment

  • The trained policy completes observation before solving a full-horizon allocation, whereas Base commits early and exhausts the budget before a valuable late-period item peaks.The paired case is illustrative, not an estimate of failure prevalence.Source: §H.3.2 · p. 24, Table 13 · p. 24

H.3.3 Stopping costly information acquisition

  • After identifying the same leading option, the trained policy stops after eight inspections costing 27 and earns net value 66.29; Base inspects all 14, spending 134 for net value −40.71.The policies ultimately select the same option.Source: §H.3.3 · pp. 24–25

H.4 External-benchmark details

  • Table 14 presents three external benchmarks using their native metrics; its BFCL row reports the unweighted mean across the ten interaction-focused cells.Source: Table 14 · p. 25

H.5 Code-tool adoption control

  • Across eleven evaluation families, pooled code-tool adoption rises from 50.4 to 79.2 percent, yet most of the observed score difference comes from improvement within fixed tool-use strata, not increased calling.Source: §H.5 · p. 26

I Admitted-environment statistics

The admitted training pool spans topical domains with near-uniform entropy, and the scale sweep uses separate checkpoints evaluated on fresh held-out environments at each scale.

I Admitted-environment statistics

J Environment-complexity sweep

K Limitations and scope

The reported scope uses computable reference outcomes and terminal-only supervision, with training on three families and generated-family evaluation on eleven. Results also rely on one training run and one evaluation seed.

K Limitations and scope

  • Sequential stopping and sequential search use hindsight ceilings as upper anchors, which need not be attainable by an online policy.Across reported families, reference outcomes are computable; replay checks compare generated dynamics with precomputed outcomes over oracle, default, and idle paths on eight families.Source: K Limitations and scope · p. 27
  • The reported implementation supplies one terminal outcome and no intermediate supervision to the policy optimizer.Training covers three families, while generated-family evaluation covers eleven, mostly established mathematics and operations-research models.Source: K Limitations and scope · p. 28
  • Generated-family results use one training run and one evaluation seed, and external results follow each benchmark’s native protocol.The ablations describe presentation, tool use, and scale; training does not separately attribute gains to the outcome rule and interface.Source: K Limitations and scope · p. 28

Related papers

Loading 2609.27321v1…