Iterative Policy Refinement through Semantic Rollout Analysis

1 Introduction

• Existing LLM-generated structured policies can encode assumptions that conflict with expert demonstrations, while prior approaches rely heavily on experts to provide or refine domain knowledge.For example, an assumed steering-damping relationship can conflict with expert steering during sharp turns and produce unwanted oscillations.Source: §1 · p. 1
• The framework alternates demonstration-based parameter fitting with LLM-generated rollout analysis and structural revisions, reducing reliance on expert refinement.The LLM generates analysis code, runs it on tabular rollouts, and uses the results to revise the structure before fitting parameters again.Source: §1 · pp. 1–2
• Analysis code tests expected policy behavior against complete tabular rollouts outside the LLM, including trajectories beyond its context limit or data that are difficult to visualize.The analysis can target specific policy computations, such as whether an angular-velocity term damps rotation as intended.Source: §1 · p. 2

2.1 Structured policies and learning from demonstrations

• Structured policy work includes executable policies, symbolic plans, intermediate components, and interpretable computation graphs such as GaP [3].Other approaches use spatial value maps and language-guided state abstractions [10] [25] [26].Source: §2.1 · p. 2
• Demonstration-guided methods use expert knowledge, narrated actions, or interactive corrections, while this work uses a sparse semantic structure whose continuous parameters are learned from demonstrations.It extends that separation by using tabular-data analysis code to identify misalignment and guide structural updates, reducing the need for human corrective instructions.Source: §2.1 · p. 2
• Reinforcement learning from demonstrations can shape rewards, augment replay data, or initialize policies and dynamics models; here, demonstrations fit parameters within successively refined structures.Consequently, subsequent reinforcement learning starts with both a learned policy and a demonstration-aligned structure.Source: §2.1 · p. 2

2.2 LLM-guided program revision and execution feedback

• Graph-based refinement and rollout-feedback systems also improve control programs, skills, or learning workflows, with GaP using simulation rehearsal to refine graph structure and parameters [3].LLM-guided program revision is also studied in self-debugging, algorithm evolution, and coding-agent improvement [5] [22] [29].Source: §2.2 · pp. 2–3
• Unlike approaches that search over complete executable policies or optimize reward functions, this pipeline analyzes intermediate policy values to guide structural refinement.One generated program defines the trainable policy structure; another analyzes tabular rollouts, connecting numerical feedback to computations that can be revised.Source: §2.2 · p. 3

3 Semantic policy structure

• A structured policy maps observations to actions through semantic latents, separating task-specific graph structure from demonstration-fitted continuous parameters.The symbolic component is a bipartite graph of variable and operator nodes, while the neural component contains learnable latent values; intermediate computations expose how actions are generated.Source: §3 · p. 3

4.1 Generating the initial policy

• The initial policy structure is generated by querying the LLM with only the task description, so its simple structure may misalign with the demonstrations.Refinement aims to close this structure-demonstration gap.Source: §4.1 · p. 4

4.2 Fitting policy parameters from demonstrations

• Imitation learning fits the structure’s weights to demonstration state-action pairs, leaving the structure unchanged during parameter fitting.In the racing example, it learns centering_weight to determine how quickly the policy steers toward the track center.Source: §4.2 · p. 4

4.3 Generating analysis code from the policy structure

• The LLM generates Python tests from expected behaviors and the current structure, linking each hypothesis to relevant computations and variables.Tests can inspect specific semantic contributions, such as the percentage of off-track steps with center-directed steering, rather than only aggregate actions.Source: §4.3 · p. 4

4.4 Collecting tabular trajectories from policy rollouts

• Rollout tables record states, intermediate latent values, parameters, actions, and next states, making the internal computations available for analysis.The figure traces lateral offset and steering across a 1,000-step rollout, illustrating the physical meaning available in semantic latent records.Source: §4.4 · p. 5, Figure 3 · p. 5

4.5 Executing the analysis code and revising the policy structure

• Analysis found center-directed steering in only 14.9% of off-track steps, motivating a structural update that adds stronger steering when the car is off-track.The new branch computes an off-track indicator, scales it by a trainable nonnegative recovery strength, and exposes the indicator for later analysis; the cycle repeats after revision.Source: §4.5 · pp. 5–6

4.6 Reinforcement learning on the demonstration-aligned structure

• After structural convergence, finite-length GSPO reinforcement learning further optimizes the continuous parameters.Trajectories are divided into length-l tracklets and grouped by similar starting positions.Source: §4.6 · p. 6

5.1 Environment setup and choice of baselines

• The evaluation uses one demonstration per task: joystick-provided for long-horizon Car Racing and motion-planner-provided for Door Opening's approach-and-pull stages.Baselines include an unstructured MLP, a zero-shot LLM-generated structure, and MLES program evolution.Source: §5.1 · p. 6, §5 · p. 6

5.2 Iterative refinement and overall performance comparison

• Under imitation learning, refinement outperforms the zero-shot structure by 15% (p ≈0.007), with performance similar to human demonstrations.Source: §5.2 · p. 6, Table 1 · p. 7
• Approximately 80% lower minimum handle-to-target distance (p < 10−4) accompanies Door Opening revisions that progress from linear approach to reach-and-pull behavior.The refined policy generalizes from one demonstration to unseen configurations.Source: §5.2 · p. 7, Figure 5 · p. 7
• The refined structure exceeds the MLP in mean return under each matched learning condition; at high RL budget, its 906.5 versus 850.2 lead is not significant (p ≈0.136).With RL, the refined structure also exceeds human demonstrations significantly (p ≈0.002).Source: Table 1 · p. 7, §5.2 · p. 8
• With 16 queries, MLES reaches 240.0 return versus 900.5 for the refined policy; matching 847.1 takes MLES 2,016 queries, 224x as many, and 8,064 resets.Source: Table 1 · p. 7, §5.2 · p. 8

5.3 Ablation on feedback type

• Generated analysis keeps Car Racing returns above 839 across four revisions, while direct table feedback drops to 415.4 at the second revision.The no-execution-feedback condition increases misalignment and causes return to collapse.Source: Figure 4 · p. 7, §5.3 · p. 8

• Tests derive from the current policy structure and rollout-table column names, so they may not anticipate or expose every failure mode.Source: §6 · p. 8
• Current evaluations cover only low-dimensional tasks following the MDP assumption; larger feature spaces and POMDP settings remain future directions.Source: §6 · p. 8
• The method may benefit from a fine-tuned LLM or a higher-level domain-specific language, such as Scenic, that may be easier to update than PyTorch.Source: §6 · p. 8

• The method alternates demonstration-based parameter fitting with LLM-generated diagnostics and structural edits, linking execution and revision through semantic latents.Source: §7 · p. 8
• Sparse semantic policy structure lets the LLM test specific policy relationships and use diagnostic results to guide structural revision.Source: §7 · p. 8
• With one demonstration per task, refinement improves imitation learning, generalizes to unseen configurations, and reduces subsequent reinforcement-learning resets.Source: §7 · p. 8

A.1 Policy strategy coverage

• Refinement retains the initial ten Car Racing strategy categories and adds steering toward curve exteriors and braking during sharp turns.The categories summarize broad control strategies rather than every operator or structural edit.Source: §A.1 · p. 12

A.2 Diagnostic categories and coverage

• Generated diagnostics cover summary statistics, reconstruction consistency, correlations and regressions, and temporal behavior or action limits.These checks connect computed values and definitions to behavior and changing control signals.Source: §A.2 · pp. 12–13
• Across three refinement rounds, generated tests number 13, 10, and 10, with later rounds adding checks for angular velocity, off-track behavior, and sharp-turn steering.Diagnostics identify a policy quantity, behavioral expectation, and editable computation, guiding feature, constraint, and interaction updates.Source: §A.2 · p. 13, Table 4 · p. 13

B.1 Off-track recovery behavior

• 74 off-track steps yielded center-directed steering in only 14.9%, exposing a recovery failure not located by aggregate return.The analysis filters steps where absolute lateral offset exceeds half_track_width.Source: §B.1 · pp. 13–14
• Figure 6 records slip and target-speed values before refinement and after three revisions, using shared axis limits.Source: Figure 6 · p. 14
• The revision adds an off-track indicator and trainable nonnegative recovery strength to increase centering beyond the track boundary.Other steering contributions remain in place.Source: §B.1 · p. 14

B.2 Car Racing programs, diagnostics, and rollouts

• Three revisions move the slip-to-target-speed branch from an unconstrained wheel-mean feature to a compressed positive residual with a nonpositive target-speed contribution.Table 5 summarizes successive revisions aligning the slip proxy’s learned contribution with its intended speed-reducing role.Source: §B.2 · p. 14
• Figure 6 compares 1,000-step rollouts of the initial and third-round policies, showing slip-proxy scale and resulting target-speed trajectories on shared axis limits.Source: §B.2 · p. 14
• The target-speed calculation passes reconstruction, but the positive slip coefficient fails the expected sign check.The slip example combines a generated sign check with an arithmetic reconstruction check from the same round.Source: §B.2 · pp. 14–15

C.1 Refinement procedure and execution

• The refinement loop fits the current structure to demonstrations, generates diagnostic code, executes it on rollout logs, and uses the results to revise the program or stop.The algorithm permits additional diagnostics before revision and optionally applies post-refinement RL.Source: §C.1 · p. 15
• The loop defaults to at most six refinement rounds, allows early stopping, and permits up to five regeneration attempts if fitting a revised program fails.Source: §C.1 · p. 15

C.2 Post-refinement reinforcement learning

• Post-refinement RL optimizes a clipped policy-gradient objective with group-relative reward-to-go and windowed likelihood ratios.Trajectories share an initial condition and are grouped by time step for advantage estimation; ratios multiply action likelihoods over a window of L valid steps.Source: §C.2 · pp. 15–16
• For each time step, reward-to-go is normalized using the population mean and standard deviation over trajectories valid at that step.The objective is clipped over valid window starts and includes an entropy bonus whose coefficient decays linearly during training.Source: §C.2 · pp. 15–16
• The Car Racing likelihood window defaults to 150 steps; optimizer defaults include γ = 0.98, clipping threshold ϵ = 0.2, and Adam learning rate 0.007.The setup uses 20 sampled episodes per update, three optimization epochs, and initial entropy coefficient 0.01.Source: §C.2 · p. 16

D.1 Shared generation specification

• The shared generation specification asks for meaningful intermediate computations, trainable coefficients, instrumentation, and initialization based on expected signs and scales.Source: §D.1 · p. 16
• The policy should encode task-relevant semantic relationships, while continuous actions are represented by multivariate-normal parameters including an action mean and lower-triangular Cholesky matrix.The specification also asks for a semantically named info_dict containing intermediate steps and weights, and favors simple differentiable features and meaningful initialization.Source: §D.1 · p. 16

D Policy-generation prompts and interfaces

• The prompt interfaces describe Car Racing and Door Opening as continuous-control tasks and provide environment descriptions and observation schemas to guide policy generation.The two environments are shown together in Figure 7.Source: Figure 7 · p. 17, §D.2 · p. 17

D.2 Environment descriptions and observation schemas

• Car Racing observations provide 12 upcoming tiles with lateral position, longitudinal position, relative heading, and a sharp-corner indicator, alongside speed, heading, angular velocity, and wheel sensors.The environment description identifies continuous steering, acceleration, and braking outputs.Source: §D.2 · p. 17
• Door Opening inputs include current and previous gripper and door observations, the door orientation, and a target position; outputs specify gripper motion and an open/close command.The task objective is to reach the handle and open the door to a desired angle.Source: §D.2 · p. 18

E Diagnostic and revision prompts

• Diagnostic prompts ask the LLM to propose analyses from the policy code and schema, then use the resulting test outcomes to assess expected behavior and possible anomalies.The prompt assumes the model has converged under imitation learning, with weights maximizing similarity to expert actions within the current structure.Source: E Diagnostic and revision prompts · p. 18, §E.2 · p. 20

E.2 Revision, additional tests, and termination

• The revision interface lets the LLM propose an updated program, request additional tests, or indicate that no further action is needed.Updates must preserve the model interface and logging outputs; the prompt asks for semantic latents and trainable parameters in the returned information dictionary.Source: §E.2 · pp. 20–21

E.1 Hypothesis generation and analysis execution

• Generated analysis functions process rollout data through a dataframe and write test results to a common output file rather than receiving the full numerical rollout in the initial diagnostic context.The prompt requests functions that take df, collect tests in run_all(df), and write results to test_results.txt.Source: §E.1 · pp. 18–19
• The prompt asks the LLM to analyze the task domain and policy structure, identify nominal behaviors and anomalies, and formulate hypotheses testable from rollout data.It asks that the tests cover potential issues and check whether the policy exhibits anticipated suboptimal behaviors.Source: §E.1 · p. 19
• The prompt directs analysis results to motivate policy-structure updates, such as adding a latent variable, changing an operation, or bounding learned parameters, before another imitation-learning session.It asks scripts to process data and test hypotheses, with defensive error handling; action sampling occurs outside the model and sampled actions are clipped to desired ranges.Source: §E.1 · p. 19
• The per-policy context includes an environment description, policy code, dataframe columns, and a sparse task-specific model trained from one demonstration.The dataframe records model intermediate steps alongside environment observations from policy rollouts.Source: §E.1 · pp. 18, 20