PaperlayerView exact revision on arXiv

Source-linked AI summary

CARL: Chat Assisted Reinforcement Learning

Sanjiv Das, Harshad Khadilkar, Sukrit Mittal, Daniel Ostrov, Jacob Sisk

arXiv:2610.31014v1cs.HC

TL;DR

Applied RL can miss human intent in its specification, and experts may also need to verify results and adjust parameters. CARL uses a structured, human–AI workflow to specify and implement RL systems, then supports post-training review and iteration. Across two examples, the paper presents CARL as a way to produce runnable RL pipelines through collaborative design, while noting that specialized search can outperform it at substantially higher computation time.

  • Problem

    Applied RL can fail when human intent is incompletely translated into objectives, constraints, and rewards, or when experts cannot validate or tune the resulting solution.

    Source: §1 · pp. 2–3

  • Method

    CARL guides experts through interactive RL specification before autonomous pipeline generation, then supports post-training verification and parameter changes through iteration.

    Source: §1 · p. 1, §6.4 · p. 24

  • Results

    The paper reports that a structured conversational workflow can turn expert intent into a runnable RL pipeline, with two examples illustrating the approach.

    Source: §1 · p. 4, §8 · p. 35

  • Takeaways & Limitations

    CARL’s approach keeps experts involved in design and review; for the Goddard problem, MC-Local-Greedy often achieved higher apogee but took at least 100 times longer than CARL.

    Source: §8 · p. 35, §8.3 · p. 36

    CARL is not a replacement for specialized search: MC-Local-Greedy can achieve higher apogee when per-instance computation is acceptable, at a wall-clock cost at least 100 times longer.

    Source: §8.3 · p. 36

Abstract

from arXiv · show

Failure of applied Reinforcement Learning (RL) to solve a given problem predominantly stems from unaddressed specification gaps, where the intent of human experts in the problem's domain is inaccurately or incompletely translated by AI into mathematical objectives, constraints, and reward functions. Failure can also stem from unaddressed performance gaps, where human domain experts are unable to determine the validity of the RL based solution or where parameters (i.e., hyperparameters or weights on the reward terms comprising the reward) are poorly chosen. Chat Assisted Reinforcement Learning (CARL) bridges these gaps through a human-AI co-design interface architecture that enforces interactive specification prior to code generation and then enables subsequent verification and parameter alterations via iteration. More specifically, CARL leads domain experts through a structured ten-step workflow for each iteration using step-specific skill files, domain-aware suggestions via an assistant persona (called Trillian), and compact decision summaries that maintain context across iterations without context overload. Once the domain expert confirms the Markov decision process (MDP) specified by this procedure, CARL's autonomous tool-use loop generates, tests, executes, and debugs an end-to-end RL training pipeline, followed by an interactive post-training diagnosis and revisit step that allows for verification and parameter alteration before running a new iteration if desired.

1 Introduction

Applied RL can fail when human intent is incompletely specified or when experts cannot validate and tune the resulting solution. CARL makes specification a human–AI collaboration before implementation, then supports verification and iteration.

1 Introduction

  • CARL structures expert–AI interaction to translate an optimization question into a formal RL specification before generating, evaluating, and iteratively improving a complete system.Trillian offers domain-informed suggestions that the expert can accept, change, or reject.Source: §1 · pp. 1–2
  • RL can address high-dimensional and path-dependent optimal-control problems, but its solutions are not guaranteed to be even locally optimal and require expert verification.The paper also notes that bespoke RL systems involve many sensitive hyperparameters and can take months to develop.Source: §1 · p. 2
  • Misspecified rewards, observations, or constraints can produce reward hacking, state aliasing, or unsafe constraint violations.Examples include an agent hiding spilled food rather than cleaning it and an agent unable to distinguish states that require different actions.Source: §1 · p. 3
  • The paper presents CARL as RL specification with human feedback, alongside its methodology, software architecture, and illustrative examples.It connects this approach to RLHF while focusing on expert feedback during RL specification.Source: §1 · p. 3
  • The finance and rocket examples test CARL’s ability to turn high-level objectives into constraint-aware RL implementations across different domains.The finance case is path-dependent; the rocket case adds a hard dynamic-pressure constraint.Source: §1 · pp. 3–4

2 Context: Background and Work Related to CARL

CARL is positioned at the problem-formalization stage of RL, where domain knowledge and iterative human oversight shape objectives before implementation. The paper connects this design to specification engineering and human–AI interaction, then illustrates it in finance and rocket-control problems.

Related applications and expertise

  • The paper places these examples alongside RL applications across many domains, including finance, robotics, healthcare, energy, and logistics.It also reviews work on AI assistance and domain expertise, including reports that expert involvement can shape outcomes and adoption.Source: §2 · pp. 4–5

Illustrative domains

  • CARL’s two exemplars span dynamic investment and constrained rocket-thrust control, with simplified cases allowing comparison against exact solutions.The paper identifies scenarios where exact methods fail and RL is used to find solutions.Source: §2 · p. 4

Specification engineering

  • The paper motivates CARL by noting that translating strategic goals into tractable problems involves uncertain, consequential formulation choices.It cites prior work describing problem formulation as discretionary and not self-evident.Source: §2 · p. 5

Human oversight

  • Prior work cited in the paper emphasizes iteration, human evaluation, and control because AI agents may not effectively assess the systems they build.The paper also notes that multi-stage generation-verification pipelines can propagate errors (47).Source: §2 · p. 5

Human–AI interaction

  • CARL frames specification as mixed-initiative HCI: users contribute domain knowledge and judgments while the system elicits inspectable objectives, states, actions, rewards, and constraints.The design emphasizes user control, correction, and understandable system behavior.Source: §2 · p. 5

Context and positioning

  • CARL focuses on formalizing the RL problem, decomposing specification into discrete conversational steps rather than treating code generation as the whole task.It aims to manage both specification and performance gaps while producing a prototype solution.Source: §2 · pp. 5–6

3 A Tour of CARL from the User’s Perspective

CARL’s user-facing design separates interactive problem specification from autonomous implementation and makes post-training review available for another iteration. The tour demonstrates this workflow on an optimal-stopping task and describes additional modes that support continuity and user control.

3.1 Ten-Step Workflow

  • Steps 1–7 interactively specify the MDP, Steps 8–9 implement and evaluate it, and Step 10 can revisit the design for another iteration.The ordered workflow makes later implementation depend on recorded decisions about the problem and its evaluation.Source: Table 1 · p. 6, §3.1 · pp. 6–8

3.2 Demonstration on a Simple Optimal Stopping Problem

  • In the demonstration, CARL turns the game description into a finite-horizon stopping problem, with the user narrowing success to near-optimal expected payoff.The user also specified that the analytical solution should be used for evaluation, not training.Source: §3.2 · pp. 8–9
  • A dry run corrected CARL’s estimated optimum from 30.87 to 43.55 by computing it with backward induction before training proceeded.The full run then produced a policy near the analytical performance benchmark.Source: §3.2 · p. 10
  • 43.52 versus 43.55: the learned policy’s expected value was close to the analytical optimum, which remained an evaluation benchmark rather than a training signal.CARL reported no need for another iteration after this result.Source: §3.2 · p. 10

3.3 Auxiliary Features

  • Auxiliary features include brainstorming formulations, resuming interrupted sessions, interactive post-training revisits, and user interjections during autonomous steps.CARL can also update Trillian’s persona conservatively from evidence about accepted or edited suggestions and generate a paper-style report.Source: §3.3 · pp. 10–11

4 Logical Design Choices Behind CARL

CARL separates domain clarification and MDP specification from implementation, using a staged workflow that preserves decisions and assigns authority between the expert and the agent. Its design also offers a reusable pattern for other tasks that produce durable artifacts from ambiguous objectives.

4.1 Design Principles

  • CARL separates problem understanding, MDP formalization, and implementation, making specification logically prior to code generation.Steps 1–2 elicit the objective, controlled system, episode structure, and constraints; Steps 3–7 develop MDP choices sequentially.Source: §4.1 · p. 11
  • Trillian offers domain-knowledge-based suggestions that experts can accept, change, or reject, allowing them to trade interaction speed against careful involvement.CARL carries forward decisions rather than full transcripts, and Step 10 supports result analysis and revised-code generation for another iteration.Source: §4.1 · p. 12

4.2 Skill Files and Decision Memory

  • Step-specific skill files narrow each conversation to its local design objective and checklist, while compact decision summaries preserve an auditable specification for later steps and autonomous implementation.Table 2 is a condensed excerpt of the Step 6 reward-definition skill, with a local objective, design hints, and checklist items.Source: §4.2 · p. 12, Table 2 · p. 12

4.3 Human Control and Agent Autonomy

  • The domain expert controls domain meaning through Steps 1–7, then CARL implements the approved specification autonomously because a correct implementation of the wrong MDP still optimizes the wrong problem.CARL may propose formulations and refinements, but the expert approves them; Step 7 confirms training choices in one turn.Source: §4.3 · p. 13

4.4 Generalizing the CARL Approach to More Than Reinforcement Learning

  • CARL applies its step-wise conversation, skill-file, decision-memory, and human/agent boundary pattern to tasks beyond reinforcement learning when they benefit from these structures.The pattern is intended for producing reusable artifacts such as code repositories, analysis pipelines, reports, simulators, or prototypes.Source: §4.4 · p. 14

5 Technical Design Choices Behind CARL

CARL is a model-agnostic orchestration layer that implements its staged workflow with persistent session artifacts, step-specific skills, and distinct interactive and autonomous phases. It also supports session recovery, post-run diagnosis, literature grounding, and optional report generation.

5.1 Technology Stack

  • A Python orchestration layer supports Flask and terminal interfaces, with separate modules for conversation, session persistence, tools, and reporting.Each session stores messages, tool calls, summaries, code, logs, plots, and reports in a project directory.Source: §5.1 · p. 15

5.2 Models Used

  • The system assigns different models to the main workflow, Trillian suggestions, and report writing, while allowing users to choose and switch models.The named model choices are those used at the time of writing, not the only applicable options.Source: §5.2 · p. 15

5.3 Model-Agnostic Conversational Orchestration

  • CARL controls model interaction through step-specific prompts, memory, tools, and completion criteria, with mechanisms for continuing truncated responses and accepting user guidance during autonomous execution.Tool access varies by step, including code-writing and command-running tools in Steps 8 and 9.Source: §5.3 · p. 15

5.4 Skill Definitions

  • Step-specific skills encode procedural MDP knowledge, from delaying formalization until domain details are understood to validating environments, reward routing, and training behavior against baselines.Parameter choices are to follow problem structure rather than repeated user queries; Figure 5 illustrates autonomous execution and a ship-or-iterate recommendation.Source: §5.4 · p. 16, Figure 5 · p. 16

5.5 Trillian Persona

  • Trillian is a separate suggestion model for early specification steps; users may accept, edit, or ignore its proposed replies, and it deactivates before autonomous implementation.CARL records whether suggestions were accepted or edited as session evidence.Source: §5.5 · p. 17

5.6 Resume and Revisit Modes

  • Resume mode restores the first unfinished step and prior artifacts, while Step 10 diagnoses completed runs and prepares revised training code without retraining during diagnosis.Revisit can inspect files and plots and requires user preflight guidance before artifact inspection.Source: §5.6 · p. 17

5.7 Literature Search and Grounding

  • CARL filters and saves literature-search results, summarizes papers from PDF pages when available, and restricts report citations to saved bibliography data to reduce hallucinated references.Abstracts serve as a fallback when PDF-page summaries are unavailable.Source: §5.7 · pp. 17–18

5.8 Report Generation

  • The reporting layer gathers session memory, literature, code, logs, plots, and conversations, then generates and assembles LaTeX sections and compiles them when the toolchain is available.Reporting documents workflow outputs rather than steering RL specification.Source: §5.8 · p. 18

6 Example 1: Goals Based Wealth Management (GBWM)

The GBWM example examines how CARL uses staged dialogue to turn domain descriptions into implementable RL problems, then applies policy diagnosis and iteration to address specification and performance gaps. Across seven evaluation scenarios, it highlights the value of interactive specification when initial choices are open and of refining specifications during implementation.

6.1 Our Example GBWM Problem with Conditional Utility Bonuses

  • The example uses annual stock and bond allocation plus binary goal-taking over a finite horizon, with utilities increasing by 25% of base utility per previously attained goal.This path dependence makes early goals potentially valuable as bonuses for later goals, while early spending can also deplete wealth; the authors note that standard dynamic programming cannot solve this chronological dependence.Source: §6.1 · pp. 19–20

6.2 Evaluation Scenarios and Algorithms

  • The evaluation compares six methods across seven increasing- and decreasing-utility scenarios, crossing full versus partial specification with CARL, custom-skill Claude Code, and skill-free Claude Code.Partial specification is a coherent domain brief that leaves important MDP choices unresolved; CARL can ask the user to resolve them before coding, unlike the non-interactive alternatives.Source: §6.2 · pp. 20–21

6.3.1 Initial run

  • CARL translated user clarifications into masked unaffordable goal-taking actions, annual decision-step dynamics, additive conditional-utility accounting, and separate goal-taking and allocation agents with current-plus-future versus future-only utility credit.The allocation action is bounded to [0, 1] by a beta-distribution policy; the objective uses γ = 1, with no shaping rewards.Source: §6.3.1 · pp. 21–23

6.3.2 Post-training policy diagnosis

  • The post-training diagnosis frames the clarifications and modeling choices as enabling a learnable reward optimizer rather than a technically valid but weaker MDP, helping minimize the specification gap.The authors emphasize that designing an RL algorithm requires many nontrivial choices, and CARL provides interactive scaffolding for making them.Source: §6.3.2 · p. 23

6.4 Results

  • CARL-FullSpec, CARL-PartSpec, and CC-Custom-Skill-FullSpec cluster among the top methods, while partially specified non-CARL methods fare worse as the number of goals increases.The comparison averages mean utility across 1,000,000 Monte Carlo paths for each method-scenario combination, using identical market-return sequences.Source: Figure 6 · p. 23, §6.4 · pp. 23–24
  • 7.94 vs 7.36 vs 4.19 mean utility in the increasing four-goal case, and 5.78 vs 5.27 vs 5.45 in the decreasing case, for CARL-PartSpec, CC-Custom-Skill-PartSpec, and CC-Only-PartSpec.The results show CARL-PartSpec leading the two partial-specification alternatives in both four-goal scenarios.Source: §6.4 · p. 24
  • After its first run, CARL-PartSpec’s blurry, stock-heavy policy prompted a revisit; a lower beta floor and annealed terminal-wealth-variance penalty produced quicker transitions and raised mean utility from 7.80 to 7.94.CARL diagnosed that a shallow objective let the policy stay near its prior mean with little effect on expected utility; the second-run changes encouraged more extreme stock allocations.Source: §6.4 · p. 24

6.5 Interpretation

  • CARL’s staged dialogue can turn a domain-level description into an implementable MDP without requiring a complete mathematical specification upfront.Source: §6.5 · p. 25
  • The largest relative gains appear when initial specifications are incomplete, rather than among fully specified systems.Source: §6.5 · p. 25
  • Poor performance can reflect earlier MDP specification choices, making problem-specification quality difficult to separate from RL implementation quality.Source: §6.5 · p. 25
  • The example supports starting with a reasonably good specification and fleshing it out during implementation as observations emerge; its inspectable policy surfaces make underspecification visible, and CARL is presented as a critical RL-development component, not a cosmetic front end.Source: §6.5 · p. 25

7 Example 2: Goddard Rocket Problem

The Goddard rocket task maximizes apogee subject to a hard dynamic-pressure limit, which can require throttling beyond the unconstrained analytic solution. The section specifies the rocket model and evaluation scenarios, then compares heuristic, analytical, and learning-based methods.

7.1 Dynamics

  • The model tracks altitude, velocity, and mass under thrust, gravity, and quadratic drag, with exponential atmospheric density and RK4 integration at 0.1-second steps.Episodes end at apogee, and fuel depletion within a step is handled by splitting integration at the fuel-out instant.Source: §7.1 · pp. 26–27

7.2 Dynamic-Pressure Constraint

  • At each step, the environment limits thrust when needed to keep next-step dynamic pressure within Qmax, using binary search to find the maximum feasible thrust.Constraint-limited steps incur a 5% drag surcharge, applied identically across methods.Source: §7.2 · p. 27

7.3 Objective

  • The objective is maximum peak altitude, subject to the rocket dynamics, throttle bounds, pressure limit, and nonnegative fuel.Source: §7.3 · p. 27, §7.2 · p. 27

7.4 Evaluation Scenarios

  • Each scenario varies fuel load, maximum thrust, drag coefficient, and pressure limit; the evaluation grid crosses three fuel loads with six pressure limits.The 18 evaluation scenarios are held out from training, which uses domain randomization over substantially wider parameter ranges.Source: §7.4 · p. 27, Table 5 · p. 27

7.5 Methods

  • All six methods use a shared deterministic evaluation environment, so peak-altitude differences reflect policy differences rather than integration drift or implementation mismatches.The comparison includes three heuristic or analytical methods and three independently trained learning-based methods.Source: §7.5 · p. 28

7.5.1 Heuristic and Analytical Baseline Methods

  • MC-Local-Greedy searches 30-segment throttle schedules, refines the eight strongest of 500 candidates with coordinate descent, and is the most expensive method by wall-clock time.It comes close to the unconstrained analytic solution when Qmax is not binding and beats it when the constraint is more severe, though it is not guaranteed globally optimal.Source: §7.5.1 · p. 28
  • The unconstrained bang-arc-bang solution is optimal only without Qmax, so it becomes progressively worse as the pressure constraint tightens.For binding constraints, a method that optimizes around Qmax may outperform one that merely conforms to it.Source: §7 · p. 26, §7.5.1 · p. 29

7.5.2 Learning-Based Methods

  • The learning-based policies use beta-distributed throttle actions, which stay within [0, 1], while the distribution shape controls whether exploration favors interior or boundary throttle values.CARL uses a boundary-mode beta head and a 14-dimensional observation with two burnout-shaping features; the other two implementations use unimodal heads.Source: §7.5.2 · pp. 29–30

7.6 CARL Clarification Dialogue

CARL’s dialogue turns a complete domain brief into an explicit MDP by surfacing choices about state, action, reward, and training. Post-training inspection and revision improved its rocket policies, which remained close to the best-performing method while using far less computation than MC-Local-Greedy.

7.6 CARL Clarification Dialogue

  • Even with the rocket physics specified, the dialogue had to resolve state design, action safety, reward normalization, and hyperparameters before training.The authors compare an initial run (Steps 1–9) with two Step 10 revisits; methods were rolled out once per scenario in deterministic dynamics.Source: §7.6 · p. 30, §7.7.1 · p. 33

7.6.1 Initial Run

  • The initial dialogue removed unnecessary altitude state, retained pressure proximity, and added headroom as a direct feature for learning.The expert noted altitude and density were monotonic in this atmosphere model, while the headroom feature gives the network a shortcut for learning pressure-related physics.Source: §7.6.1 · pp. 30–31
  • The expert replaced sparse terminal reward with stepwise altitude increments and set γ = 1; boundary-mode beta policies addressed endpoint exploration collapse.A unimodal beta policy collapsed to near-zero entropy after the first PPO update without policy improvement; CARL switched to a boundary-mode beta head.Source: §7.6.1 · p. 31

7.6.2 Post-Training Diagnosis

  • After early burnout appeared in throttle plots, the expert guided CARL to add a one-sided burnout penalty and observation features for a target burnout time.Revisit 2 replaced the linearly growing target with the fitted quadratic ttarget = 75x−8x2, where x = mfuel,0/1000.Source: §7.6.2 · p. 32

7.6.3 Use of heuristics for self-improvement with CARL

  • The revisit phase used training evidence and expert input to revise the MDP, including a later change to the burnout target’s functional form.Neither CARL nor CC-Custom-Skill used the suggested rule-based supervised warm-up; CARL could still revise its formulation through human-guided revisits.Source: §7.6.3 · p. 32

7.7.2 Comparison of CARL Revisits

  • Revisit 2 was CARL’s strongest internal run in 16 of 18 scenarios; average apogee improved 11.4% over the initial run.Revisit 1 improved on the initial run by 4.1%, and Revisit 2 further improved on Revisit 1 by 6.9%.Source: Table 6 · p. 33, §7.7.2 · p. 33

7.7.3 Comparison Across Methods

  • CARL was within 3.1% of the best-performing method per scenario and exceeded CC-Only by 19.9%, while using far less computational time than MC-Local-Greedy.MC-Local-Greedy performed best when the pressure constraint bound, at 100–253 times CARL’s wall-clock cost; CARL’s learned policies evaluate quickly after training.Source: Table 7 · p. 34, §7.7.3 · p. 34

7.7.4 Profiles Over Time

  • Across time profiles, CARL followed the unconstrained baseline more closely than the CC policies, while constraint-active intervals were marked in the altitude and velocity traces.MC-Local-Greedy tracked the unconstrained solution closely when pressure was not binding, though its 30 time segments made the trajectory jittery.Source: §7.7.4 · p. 35

8 Conclusions

CARL treats applied RL’s central challenge as translating domain intent into a complete MDP, then uses human-led specification and post-training revision to address that challenge. The two examples show why domain briefs leave important modeling choices open, while the rocket comparison also shows that specialized search can outperform CARL at substantially greater computational cost.

8.1 Addressing the Specification Gap

  • CARL decomposes specification into ten sequential steps, with a skill file and checklist for each, and discusses every MDP component before code generation.The paper’s premise is that specification failure, rather than an algorithmic problem, is the principal barrier to applied RL adoption.Source: §8.1 · pp. 35–36

8.2 The Importance of Humans-in-the-Loop versus Automation in Reducing the Specification Gap and the Performance Gap

  • Domain experts control substantive modeling decisions in Steps 1–7, while CARL implements, tests, and trains; Step 10 reopens design using evidence from the trained policy.The authors distinguish implementation errors in a correct MDP from a well-trained policy optimizing the wrong MDP.Source: §8.2 · p. 36

8.3 Takeaways from our Two Examples

  • The examples show that precise domain briefs still leave MDP choices unresolved, including observations, action boundaries, reward scale, safety handling, and training schedules.The GBWM brief also left feasibility rules, terminal-goal treatment, utility arithmetic, reward timing, and state normalization open.Source: §8.3 · p. 36
  • CARL’s interactive workflow matters most when an incomplete brief leaves modeling choices to surface and resolve; policy inspection can reveal MDP-related defects requiring revision.In the GBWM comparison, CARL-PartSpec remained close to CARL-FullSpec; in Goddard, throttle traces revealed early burnout and prompted burnout-aware changes.Source: §8.3 · p. 36

8.4 Implications of our Work for Future Deployment and Evaluation

  • MC-Local-Greedy often achieved higher rocket apogees than CARL but required at least 100 times longer wall-clock time; CARL is not a replacement for specialized optimization.The paper suggests CARL where experts can judge behavior but cannot be expected to write a complete RL formulation.Source: §8.3 · p. 36, §8.4 · pp. 36–37

9 Future Work

Future work proposes testing CARL on broader RL problems and user populations, adapting its interaction policy, improving evaluation and explainability, and exploring CARL-like workflows beyond RL.

Expanding to More RL Problems

  • The two examples have compact state descriptions, low-dimensional actions, and inspectable evaluations; broader problems, including partially observable systems, remain to be tested.Potential areas include robotics, multi-agent coordination, healthcare, logistics, and recommendation systems; inverse RL, preference learning, and trajectory comparisons could help propose rewards.Source: §9 · p. 37

Expanding the Testing User Base

  • Broader usability evidence should include users with varied RL knowledge, programming experience, and domain expertise, and measure workflow burden as well as policy quality.Proposed measures include specification time, clarification turns, edits to suggestions, perceived control, confidence, and diagnostic ability.Source: §9 · p. 37

Adaptation Beyond Trillian

  • Future CARL versions could adapt when to ask, propose defaults, surface uncertainty, or offer alternatives, with interaction modes tailored to different user roles.One proposal is an active-learning controller that estimates decision confidence and queries users when clarification has high expected value.Source: §9 · p. 37

Responsible AI

  • Responsible deployment in broader applications calls for stress tests, constraint audits, reproducibility checks, uncertainty reporting, baselines, and policy explanations.The proposed explanations include natural-language policy summaries, feature-importance or sensitivity analyses, counterfactuals, and guidance on when not to trust a policy.Source: §9 · p. 37

CARL as a General Concept

  • The authors propose treating CARL as a pattern for structured human-AI workflows beyond RL, potentially using meta-workflows to design domain-specific assistants.The proposed core discipline includes staged specification, human authority over meaning, autonomous artifact construction, and evidence-driven revision.Source: §9 · p. 38

A.1 Full Specification

The appendix specification describes a goals-based wealth-management task with annual investment and goal-taking decisions, sampled problem cases, and prescribed evaluation examples and plots.

A.1 Full Specification

  • The agent chooses whether to take or skip a present goal and allocates investment between bonds and stocks at each time step.Goal-taking is binary; investment allocation is continuous from all bonds at 0 to all stock at 1.Source: §A.1 · p. 41
  • The policy is intended to generalize across one-, two-, and three-goal cases with time horizons of 10–20 years and specified goal-amount ranges.The ranges are $100–$500 for one goal, $50–$250 per goal for two goals, and $40–$200 per goal for three goals.Source: §A.1 · p. 41
  • A four-goal case uses horizons of 10–20 years and goal amounts of $25–$125; goal utilities are integers from 1 to 8 and sum to 8.Final goals must be taken greedily when enough wealth is available at the final time step.Source: §A.1 · p. 41
  • The task assumes initial wealth of 100, annual decisions, no cashflows, and full investment of available wealth.Stock and bond evolution may use daily-frequency parameters, which must be simulated over each one-year step while accounting for correlation.Source: §A.1 · p. 41
  • Evaluation uses four canonical 16-year examples, from one to four goals, with goal amounts, timing, and utilities specified for each.The specification also calls for policy heatmaps over wealth and time, with log-scaled wealth axes and color indicating the stock allocation.Source: §A.1 · pp. 41–42
  • Episodes continue to time T even after bankruptcy or goal attainment, and utility is evaluated only at T; future goals gain a 25% bonus per previously taken goal.The conditional bonus compounds across earlier taken goals as described in the specification.Source: §A.1 · p. 42

A.2 Partial Specification

The specification defines a single policy for feasible wealth-planning problems with one to four goals over 10–20 years. It sets the canonical evaluation cases and plotting requirements, with future-goal utility boosted by previously attained goals.

Actions

  • Each year offers a goal take-or-skip decision when a goal is due and an investment allocation from all bonds to all stocks.The time step is one year.Source: §A.2 · p. 42

Problem scope

  • The policy must work across 1–4 goals, at most one per year, and 10–20-year horizons, for reasonably feasible goal amounts.The specification assumes an initial wealth of 100 and uses one policy across this problem range.Source: §A.2 · p. 42

Canonical examples

  • Evaluation uses four 16-year canonical cases with one to four goals, differing in goal amounts, timing, and utility.The four-goal case uses four $100 goals at years 0, 4, 8, and 12, with utilities 1, 1, 2, and 4.Source: §A.2 · p. 42

Evaluation and plotting

  • Policy heatmaps plot wealth against time with color for stock allocation, and evaluation wealth is shown on a log axis.The stock and bond evolution uses the standard suggested parameter values; the specification’s “above” refers to Subsection 6.1.Source: §A.2 · pp. 42–43

Conditional utility

  • Each previously taken goal adds 25% to the utility of every future goal that is subsequently taken.For example, taking two earlier goals raises a future goal’s utility by 50%.Source: §A.2 · p. 43

B GBWM Implementation Comparison

The implementation comparison organizes six AI approaches by policy state dimension and action representation, and explains the DQN, Double DQN, PPO, and MLP abbreviations.

Implementation comparison

  • The table compares six AI approaches by state dimension and policy actions, with CARL’s state dimension reported as the larger of its two views.It defines DQN, Double DQN, PPO, and MLP and distinguishes training-time action selection from evaluation where described.Source: Table 8 · p. 44

B.1 State Variables for the Six Methods

The six-method comparison groups state inputs by shared feature meaning and normalization, while accompanying text explains how the goal-planning and wealth features inform policy decisions.

Table conventions

  • The table groups related normalizations together and may represent a feature family with repeated entries for each goal slot.Some methods expose both raw goals-taken count and its bonus multiplier.Source: Table 9 · p. 45

Wealth and time features

  • Log wealth normalizes wealth by initial wealth before taking logs, matching the multiplicative nature of investment returns.Methods differ in time normalization: some use the maximum horizon, while some non-interactive runs use the sampled horizon.Source: §B.1 · p. 45

Goal representation

  • State inputs encode future goal calendars, padding unused slots with zeros and using a presence mask to distinguish padding from real goals.The calendar includes each future goal’s timing, amount, and utility.Source: Table 9 · p. 45, §B.1 · p. 46

Planning summaries

  • Aggregate present-value wealth slack and remaining utility summarize affordability relative to future goals and the reward still available.Time to the next goal supplies a compact urgency signal.Source: §B.1 · p. 46

Goal progress and decisions

  • Goals-taken fraction distinguishes progress such as 1 of 2 goals from 1 of 4, while goal count or bonus multiplier captures path-dependent utility increases.Current-goal features indicate whether a goal is due, affordable, and what wealth cost and utility taking it entails.Source: §B.1 · pp. 45–46

C State Variables for the Three Methods in the Goddard Problem Implementation

The Goddard implementation compares state inputs across three trained methods, including CARL-specific ascent-progress and burnout features. The accompanying descriptions connect the inputs to rocket dynamics and constraint handling.

State-variable comparison

  • CARL alone observes energy-based ascent progress and two burnout-shaping features; unlike the other methods, it omits direct altitude and uses atmospheric density instead.CC-Custom-Skill and CC-Only have no explicit burnout timing and rely on fuel fraction and thrust-to-weight alone.Source: Table 10 · p. 47, C State Variables for the Three… · pp. 47–48

Progress and fuel state

  • Current mass, remaining fuel, altitude, and velocity inform acceleration, burn budget, and further climb potential after burnout.The energy-based ascent-progress feature combines altitude, kinetic energy, and remaining propellant energy to distinguish trajectories with different climbing capacity.Source: C State Variables for the Three… · pp. 47–48

Constraint and flight state

  • Dynamic-pressure fraction and headroom indicate proximity to the Qmax limit, while velocity and atmospheric density jointly determine dynamic pressure.Density also affects drag, and lower density permits greater speed before reaching the same Qmax.Source: C State Variables for the Three… · pp. 47–48

Episode parameters

  • Max thrust, Qmax, drag coefficient, and initial fuel mass describe episode-specific engine scale, structural limit, atmospheric cost, and energy budget.Thrust-to-weight ratio indicates whether full thrust can overcome gravity, and improves as mass falls.Source: C State Variables for the Three… · p. 48

Policy visualizations

  • Figures 7 and 8 display investment and goal-taking policies for CARL, CC-Custom-Skill, and CC-Only; Figure 9 shows CARL-PartSpec’s initial-run policies before revisit.Figure 7 covers fully specified methods, while Figure 8 covers partially specified methods.Source: Figure 7 · p. 49, Figure 8 · p. 50, Figure 9 · p. 51

Rocket dynamics and example profiles

  • Figure 10 depicts thrust, weight, and drag as the forces whose imbalance accelerates or decelerates the rocket.Figure 11 reports a sharp CC-Custom-Skill failure at 293 m, while Figure 12 describes strong performance by CARL and MC-Local-Greedy under a tight Qmax constraint.Source: Figure 10 · p. 51, Figure 11 · p. 52, Figure 12 · p. 53
Loading 2610.31014v1…