Code to Control: Synthesizing Parameterized Reactive Controllers

1 INTRODUCTION

• Direct LLM policies require inference at every decision, while theory-based and programmatic world-model approaches require planning and can be difficult to model and debug.Small synthesis errors in complex world models can compound during rollouts.Source: §1 · p. 1
• The framework separates LLM-synthesized controller structure from numerical parameters fitted by derivative-free search using environment returns.In motor-control domains, the LLM supplies interpretable feature terms and search fits the matrix mapping them to actuator torques.Source: §1 · p. 1
• The LLM receives neither environment dynamics nor a demonstration; it revises its program using gameplay trajectories and rewards.The resulting reactive-style policies are especially suited to domains where complete world models would be laborious and offer minimal performance benefits.Source: §1 · pp. 1–2
• Code to Control outperforms planning-based program synthesis on all six Atari games and nine of ten MuJoCo tasks, and exceeds PPO trained for 20 million steps on three Atari games using fewer than 60k steps on most games.On six MuJoCo tasks, it also scores above an oracle planner given the true simulator.Source: §1 · p. 2
• The evaluation spans six Atari games, Flappy Bird, and ten MuJoCo tasks, including transfer and adaptation under changes in environment dynamics.For Flappy Bird, the program stays fixed during parameter-refitting adaptation.Source: §1 · p. 2
• The paper measures action-selection cost and reports that Code to Control executes faster than PPO, planning-based methods, and LLM-based baselines.Its executable controllers require neither LLM inference nor planning at decision time.Source: §1 · p. 2

2.1 PROBLEM FORMULATION

• A finite-horizon MDP specifies observations, actions, transitions, rewards, and a horizon; the objective is to maximize expected undiscounted episode return.The trajectory shown to the LLM consists of at most H transitions and records the rewards collected during the episode.Source: §2.1 · pp. 2–3

2.2 CODE TO CONTROL

• A policy is synthesized as Python source code mapping observations to actions, with timestep also supplied in continuous domains.For continuous control, the policy factors into an LLM-synthesized feature map of interpretable observation- and timestep-derived programs and a fitted matrix W mapping features to actions.Source: §2.2 · p. 3, §2.1 · p. 3
• Feature normalization makes equal-sized weight perturbations more comparable across differently scaled features, while clipping keeps outputs within the admissible torque range.The feature scales can differ substantially, so normalization supports the optimizer’s fixed-size perturbations.Source: §2.2 · p. 3

2.2.1 CONTROLLER SYNTHESIS AND PARAMETER OPTIMIZATION

• In discrete domains, the LLM synthesizes an executable policy, then revises the latest policy using its trajectory and return while retaining the best controller encountered.The prompt context contains a state schema, action space, object API, generic mission statement, and initial observation; it supplies neither dynamics nor a demonstration.Source: §2.2.1 · p. 4
• In continuous domains, random-action transitions inform independently synthesized feature maps, and derivative-free search fits weights for each map against episode returns.The reported experiments use Augmented Random Search; return feedback fits numerical parameters without revising the feature map, and the best fitted controller is retained.Source: §2.2.1 · p. 4

3.1 EXPERIMENTAL SETUP

• The evaluation spans Atari games, Flappy Bird, and MuJoCo tasks, with comparisons to direct LLM action selection, planning-based program synthesis, and PPO.Atari observations use object lists parsed from image frames, and Flappy Bird uses the same object-level schema.Source: §3 · p. 4, §3.1 · p. 4, §3.2 · p. 4

3.2 LEARNING PROGRAMMATIC POLICIES IN ATARI

• Action selection takes 11.4 µs for Code to Control, versus 76.5 µs for PPO, 1.29 s for ReAct, and 46 ms per action for WorldCoder planning.Across six Atari games, Code to Control uses 120 LLM calls and 487k tokens; on Pong, the reported controller is found after 7 calls and 25,312 environment steps.Source: §3.2 · p. 5, Figure 2 · p. 6
• In Pong, the LLM revised paddle targeting from the ball’s current position to a five-tick velocity-based prediction, improving the score from −8 to +18; revisions can nevertheless reduce performance.Across Atari episodes, 72% score below the run’s best so far, and 28% fall by more than 25% from the preceding episode; the champion is retained separately.Source: §3.2 · pp. 5–6
• Code to Control scores higher than WorldCoder, ReAct, and 100k-step PPO on all six Atari games, and remains ahead of 20-million-step PPO on three.On Pong, it scores +18 versus −17 for ReAct and PoE-World and −19 for WorldCoder.Source: §3.2 · p. 5, Figure 2 · p. 6

3.3 ROBUSTNESS TO CHANGES IN ENVIRONMENT DYNAMICS

• Across eight Flappy Bird dynamics changes, controllers transfer zero-shot in several settings, while parameter-only refitting restores full survival in three of five frozen-controller failures.Reversing gravity is the exception: refitting cannot recover successful play when a structural program change is required.Source: §3.3 · pp. 6–7, Figure 3 · p. 7

3.4 SCALING TO CONTINUOUS CONTROL

• Code to Control scores higher than planning-based program-synthesis baselines on nine of ten MuJoCo environments and exceeds published CQL results on all ten.It also exceeds the oracle planner on six of ten environments; that comparison retains the planning procedure while replacing the learned model with the true simulator.Source: §3.4 · p. 7, Figure 4 · p. 8
• HalfCheetah and Swimmer controllers rely strongly on current physical state rather than fixed periodic timing, while return alone can overstate locomotion in tasks with alive rewards.The reported Humanoid controller has median displacement −0.44 m despite a return of 472.92, including 512 in accumulated alive reward.Source: §3.4 · p. 8

Theory-based RL and code world models

• Unlike world-model methods that plan through learned dynamics, Code to Control selects actions without model simulation or a planner at decision time [Tsividis et al., 2026; Ahmed et al., 2025; 2026].PoE-World additionally uses a gameplay demonstration and assumes access to the task goal; Code to Control collects interaction data and learns the goal while synthesizing its policy.Source: §4 · p. 8

Programmatic policies

• Code to Control’s related-work distinction is direct execution of Python policies without translating them into a predefined DSL.PIRL, PROPEL, LEAPS, and LLM-GS constrain policies to a DSL or symbolic program space, whereas Code to Control learns through interaction in Atari and continuous-control domains.Source: §4 · pp. 8–9

LLMs as policies and as code generators for control

• Code to Control learns from interaction and reward, rather than natural-language commands or a predefined task-specific control-primitive library [Wang et al., 2023].Its controller acts in the primitive action space and has tunable parameters for adaptation to new dynamics.Source: §4 · p. 9

Derivative-free optimization for continuous control

• ARS and evolution strategies search policy parameters through perturbations without backpropagating through the environment [Salimans et al., 2017].Code to Control uses ARS to fit the action-mapping matrix after the LLM synthesizes the feature representation.Source: §4 · p. 9

5 DISCUSSION

• The learned Python controller executes directly, shifting computation from decision time to synthesis while retaining a structured, interpretable program.Once learning is complete, action selection requires neither LLM inference nor planning.Source: §5 · p. 9
• The experiments focus on reactive tasks where behavior can follow from the current observation; explicit prediction or long-horizon lookahead may favor planning-based world models.After learning, actions require only controller execution, with no model rollout, planning procedure, or LLM call.Source: §5 · p. 9
• The current formulation uses structured state rather than learning directly from pixels, and the effective balance between program revision and fixed structure remains open.Excessive revision can destroy useful structure, while fixing structure can limit adaptation; performance is also shaped by the environment’s objective.Source: §5 · p. 9
• The authors report using generative AI for literature discovery, drafting, research ideation, and execution, with review and verification by the authors.Code is available; the appendix provides the algorithm, prompts, and example synthesized controllers, and specifies the language models used in each domain.Source: AI USE STATEMENT · p. 10, REPRODUCIBILITY STATEMENT · p. 10

Discrete controller synthesis and revision

• For Atari and Flappy Bird, the LLM synthesizes and revises the complete executable controller, including numerical constants; parameter fitting is unnecessary for the initial discrete-action setup.The controller maps the current state directly to a primitive action, without search, simulation, or lookahead.Source: §A.2 · pp. 11–12

Continuous feature-map synthesis

• For continuous control, the LLM proposes feature-map terms and derivative-free search fits the numerical weights mapping those features to actions.The prompt requests candidate terms rather than a finished controller; fitting can adapt the numerical parameters while keeping the synthesized structure fixed.Source: §A.2 · pp. 11, 13, 14

Discrete controller synthesis and revision

• Discrete-controller revisions use gameplay trajectories and rewards, with the algorithm retaining the best controller encountered separately from the latest revision.The LLM revises the latest policy using its own play and reward feedback.Source: §A.2 · pp. 11–12
• The prompts ground controller revisions in observed transitions and encourage anticipating motion from velocity or changes between successive positions.For discrete control, the prompt favors relative geometry and a small number of clear cases over elaborate expressions.Source: §A.2 · p. 13

Continuous feature-map synthesis

• The continuous prompt asks for diverse, finite feature terms that remain O(1) in magnitude, so fitted weights can compose the building blocks.Examples include state fields, velocities, periodic terms, products or differences, bias, and growing or saturating terms.Source: §A.2 · p. 14

• The Pong controller predicts ball height five ticks ahead and compares it with player center using a 4.0 deadzone to choose up, down, or no-op.Source: §A.3 · pp. 14–15
• 311 at the episode cap for Flappy Bird; +18 median score across three runs for Pong.Source: §A.3 · pp. 14–15
• The Flappy Bird controller identifies the next pipe pair and steers toward its gap, using different targeting behavior at close, moderate, and far ranges.Source: §A.3 · pp. 15–16
• The continuous-control map returns 14 terms, and search fits weights unseen by the model, with one weight row per actuator.Its features include posture, joint angles and velocities, forward velocity, gait clocks, and front-back leg asymmetry.Source: §A.3 · pp. 16–17