Source-linked AI summary
GARDEN: Generalist Autonomous Reward Design Through Evidence Navigation for Cooperative Multi-Agent Reinforcement Learning
Dogan Urgun
TL;DR
Sparse task rewards make cooperative credit assignment difficult, and LLM-designed rewards are usually tested in a single setting. GARDEN develops one reward across related scenarios by investigating evidence and probing drafts, then transfers it unchanged; held-out Overcooked and SMAX results improve over native rewards.
Problem
Task-completion rewards provide little guidance for assigning credit across cooperative agents, while LLM-designed rewards are typically evaluated in one setting and use predefined evidence.
Source: §I · p. 1Method
GARDEN lets an LLM investigate environment and training evidence and test draft rewards, then freezes one program developed across related scenarios for held-out evaluation.
Source: §I · p. 1, B. Autonomous Evidence Navigation · p. 3Results
Held-out returns rose by 58.2% and 52.5% in Overcooked, and mean held-out SMAX win rate rose from 51.0% to 89.9%.
Source: §VI · p. 7Takeaways & Limitations
The frozen rewards transferred successfully to held-out Overcooked and SMAX scenarios, with policies trained from scratch on scenarios concealed during design.
Source: §VI · p. 7The evaluation was pilot-informed rather than researcher-blind because one held-out SMAX scenario had been examined in pilots.
Source: D. Evaluation Protocol · p. 5
Abstract
from arXiv · showhide
Cooperative multi-agent reinforcement learning (MARL) enables teams of agents, such as robots and autonomous vehicles, to learn coordinated behavior, but what such teams learn is largely determined by their reward function. Because task success alone provides too little guidance for learning, intermediate rewards are usually designed by hand, and this effort must often be repeated whenever the environment or the team changes. Large language models (LLMs) can automate the effort by writing reward functions as code, yet the resulting rewards are typically designed and tested in a single setting. To enable their reuse across settings, GARDEN (Generalist Autonomous Reward Design through Evidence Navigation) is proposed, in which one reward program is developed for a family of related scenarios and transferred, unchanged, to scenarios that were not available during its design. During development, environment code, training records, and agent behavior are investigated by an LLM reward engineer, and draft rewards are tested in trial training runs, while the learning algorithm and the rule for selecting the final reward are held fixed. On unseen scenarios of the Overcooked and SMAX benchmarks, task returns were increased by 58.2% and 52.5% over the default rewards on two kitchen layouts, and the mean win rate in two combat scenarios was raised from 51.0% to 89.9%. Under matched candidate and probe budgets, this exceeded the 84.2% and 79.2% obtained with fixed diagnostic feedback and Eureka-inspired reward reflection. These results show that a single autonomously engineered reward can be reused effectively across related cooperative scenarios.
I. INTRODUCTION
Sparse task rewards make cooperative credit assignment difficult, while hand-designed shaping must be repeatedly revised as settings change. GARDEN develops one reward across related scenarios by letting an LLM investigate evidence and test drafts, then freezes it for held-out scenarios.
I. INTRODUCTION
- When only task completion is rewarded, assigning credit across agents and interdependent actions is difficult, and effective shaping requires expertise and repeated trial and error.A reward effective in one setting may induce unintended behavior in a new one, making redesign a recurring cost as environments or teams vary.Source: §I · p. 1
- Existing LLM reward-design methods typically evaluate rewards in one setting and refine them using evidence selected in advance, potentially missing coordination failures absent from summary statistics.GARDEN instead lets the reward engineer inspect source code, training artifacts, and behavior, and test draft rewards through learning probes while keeping the learner and selection rule fixed.Source: §I · pp. 1–2
- GARDEN frames reward design at the task-family level: one executable program is developed on several scenarios, frozen, and evaluated on concealed scenarios to separate reward from policy generalization.The LLM selects evidence and constructs analyses, while the learning algorithm and final selection rule remain fixed.Source: §I · pp. 1–2
II. RELATED WORK
- Related approaches span reward shaping, LLM reward engineering, and cooperative MARL learning signals, but generally fix the design evidence and assess rewards on their target task.The evidence available to representative LLM-based approaches is compared in Table I.Source: §II · p. 2
A. Reward Shaping and Transfer
Prior reward-transfer and LLM-design approaches rely on specified reward forms, fixed feedback, or task-specific evaluation. GARDEN instead synthesizes executable rewards from evidence and tests one unchanged program across related scenarios.
A. Reward Shaping and Transfer
- Potential-based shaping transfer depends on specific conditions, while other approaches use demonstrations, preferences, meta-learned potentials, or learner-side credit assignment.GARDEN leaves the learner unchanged and synthesizes an executable reward from source code and empirical evidence for reuse across scenarios with different geometry or team structure.Source: A. Reward Shaping and Transfer · p. 2
B. LLM-Based Reward Engineering
- LLM reward methods use fixed feedback forms and evaluate rewards on their design task; ARBO delegates evidence choice but limits records to scores, training curves, and error tracebacks.GARDEN additionally examines raw trajectories and checkpoints, tests drafts through private learning probes, selects jointly across development scenarios, and evaluates the unchanged program on withheld scenarios.Source: B. LLM-Based Reward Engineering · p. 2
C. Learning Signals for Cooperative MARL
- Cooperative MARL reward methods often alter other learning components and evaluate signals in their use scenarios; GARDEN varies only the frozen reward and measures it on held-out scenarios.The incentive-aware study [21] also synthesizes executable rewards, but returns fixed diagnostics and evaluates each reward in its design scenario.Source: C. Learning Signals for Cooperative MARL · p. 3
III. METHODOLOGY
The methodology formulates reward design over cooperative task families whose scenarios share task semantics and interaction mechanics but differ structurally. A reward program supplies per-agent shaping, and its transfer is assessed empirically rather than guaranteed.
A. Task-Family Reward Design
- Scenarios are modeled as decentralized partially observable Markov decision processes with multiple agents, a shared sparse task reward, and an episode limit.Performance is measured irrespective of the shaping used during training.Source: A. Task-Family Reward Design · p. 3
- A cooperative task family comprises scenarios with shared task semantics and interaction mechanics but differences such as layout, team size, or unit composition, partitioned into development and held-out sets.The stated problem seeks a program using development evidence, while held-out outcomes are unavailable during design and no transfer guarantee is implied.Source: A. Task-Family Reward Design · p. 3
- The reward program returns per-agent shaping values and bounded memory that resets at episode boundaries.The supplied passages describe the program as a deterministic executable function; scenario identity is not supplied, though it may be inferred from geometry or team composition.Source: A. Task-Family Reward Design · p. 3
- Training compares generated shaping with native shaping, which is the environment's built-in shaping, alongside the task reward.The native shaping can be inspected as source code but is not added to, imported by, or called from the generated program.Source: A. Task-Family Reward Design · p. 3
B. Autonomous Evidence Navigation
GARDEN delegates evidence investigation and draft-reward probes to a tool-using LLM while fixing the protocol and learner. Candidates work independently, are assessed on development scenarios, and a selected program is frozen before held-out evaluation.
B. Autonomous Evidence Navigation
- The LLM can inspect environment and learner source, reward logic, training records, trajectories, losses, curves, checkpoints, and prior design records without a prescribed diagnostic checklist.It may construct analyses and request a bounded number of learning probes on chosen development scenarios using the formal assessment learner, budget, and seed.Source: B. Autonomous Evidence Navigation · p. 3
C. Independent Candidates and Shared Evidence
- After G generations, a fixed rule selects and freezes one program, which is evaluated by training new policies from scratch on held-out scenarios against native policies.The procedure is summarized in Algorithm 1 and illustrated in Fig. 1.Source: C. Independent Candidates and Shared… · p. 4
- Candidates are developed in independent sessions with private probes and evidence, while completed earlier generations are visible and sibling candidates' work remains withheld.Each committed candidate is assessed on every development scenario before the evidence is updated for the next generation.Source: C. Independent Candidates and Shared… · pp. 3–4
D. Mechanical Selection and Transfer
After assessing the candidate programs, the method selects one using development sparse-return improvement and then freezes it for held-out training. Selection uses no LLM preference or held-out results.
D. Mechanical Selection and Transfer
- The selected reward program is chosen mechanically after all candidates are assessed, rather than by LLM preference or held-out performance.Ties are resolved by worst-scenario normalized improvement, then mean sparse return, then a deterministic identifier.Source: D. Mechanical Selection and Transfer · p. 4
- Candidate performance is scored by development mean sparse return relative to the native reference, using relative improvement or absolute improvement depending on the reference magnitude.Relative improvement is used when the native reference magnitude is at least 1; absolute improvement is used otherwise, including for win rates.Source: D. Mechanical Selection and Transfer · p. 4
- The chosen source is frozen and hashed, and native and generated-reward policies are trained from scratch on held-out scenarios under identical settings.Because policy parameters are not transferred, the evaluation measures generalization of the learning signal.Source: D. Mechanical Selection and Transfer · p. 4
IV. EXPERIMENTAL DESIGN
The experiments use Overcooked and SMAX task families, each with a separate reward program developed on some scenarios and evaluated on held-out scenarios with structural differences.
A. Environments and Rewards
- Experiments use Overcooked cooking and SMAX cooperative combat, whose scenarios vary in layout, team size, unit types, and numerical balance.Overcooked agents observe the full kitchen, while SMAX units observe nearby units.Source: A. Environments and Rewards · p. 4
- A separate reward program is developed for each family, and its four held-out scenarios differ from development scenarios in layout, team size, or opposing-team composition.The held-out scenarios are shown in Fig. 2.Source: A. Environments and Rewards · p. 4
- The native reward combines sparse task reward with built-in shaping, while generated shaping replaces the built-in shaping at constant weights during training.Performance is measured using task reward alone; the native baseline uses the same constant shaping schedule as the generated reward.Source: A. Environments and Rewards · p. 4
B. Policy Learning
Policies are trained with MAPPO using centralized training and decentralized execution, with architectures adapted to each benchmark’s observation structure. Training settings are held constant across reward conditions.
B. Policy Learning
- MAPPO uses shared actor parameters for decentralized action selection and a centralized critic that estimates per-agent values from joint information.The critic supports team credit assignment and is used only during training.Source: B. Policy Learning · p. 4
- Overcooked uses convolutional and dense layers for full-grid observations, whereas SMAX uses recurrent GRU networks for agents’ local observations.The Overcooked critic receives both agents’ concatenated observations; the SMAX critic receives global state and remaining time.Source: B. Policy Learning · pp. 4–5
- Only allied SMAX units are trained, with unit types and start positions randomized each episode in SMACv2 scenarios.Fig. 2 depicts the held-out Overcooked layouts and SMAX battles, including a larger opposing marine team and randomized SMAX unit types and positions.Source: B. Policy Learning · p. 5, Figure 2 · p. 5
- Every reward condition within a family uses the same architecture, optimization settings, and approximately five-million-step training budget, with policies initialized from scratch.Final policies are evaluated with sampled actions over 64 Overcooked episodes and 256 SMAX episodes; held-out results do not select checkpoints.Source: B. Policy Learning · p. 5
C. Reward Search and Controls
Reward searches use fixed candidate and probe budgets, while controls restrict the reward engineer’s evidence access. Evaluation uses matched policy seeds, but the pilot-informed design and one search per condition constrain interpretation.
C. Reward Search and Controls
- Each family’s reward search uses three generations of two candidates, with at most two complete-training probes per candidate session and fixed time and interaction allowances.Probes and development assessments use one training seed; the reward engineer is gpt-6.1-sol at maximum reasoning effort.Source: C. Reward Search and Controls · p. 5
- The SMAX controls match candidate and probe budgets, learner, and selection rule, but vary evidence access and reward history.Fixed diagnostics restrict training evidence to standardized reports; Eureka-inspired reflection refines the previous generation’s best reward using standardized task and component feedback.Source: C. Reward Search and Controls · p. 5
- Evidence access, history, and inference effort are compared jointly rather than as isolated factors, with one search per condition and rewards frozen before held-out training.Thus, the comparison does not isolate the effect of any one of those factors.Source: C. Reward Search and Controls · p. 5
D. Evaluation Protocol
- Each reward is developed once, then tested on held-out conditions with 10 matched policy seeds; uncertainty across reward searches is not captured by those seeds.The scenario split and training budget were fixed after pilot calibration, and one held-out SMAX scenario was examined in pilots, making the evaluation pilot-informed rather than researcher-blind.Source: D. Evaluation Protocol · p. 5
- Reported gains are point-estimate differences between GARDEN and the reference mean on each held-out scenario.Native-shaping gains are reported in Table IV, while control comparisons are reported in Section V-B.Source: D. Evaluation Protocol · p. 5
V. EXPERIMENTAL RESULTS
GARDEN improved performance in all four held-out scenarios, while the evaluation used matched training settings and reported returns and win rates on their natural scales. The architectures differed between Overcooked and SMAX.
A. Transfer to Held-Out Scenarios
- Overcooked returns are reported as percentages of native results, while SMAX gains are absolute win-rate differences in percentage points.The reporting follows the selection score: relative improvement for return-based tasks and absolute improvement for win rates.Source: A. Transfer to Held-Out Scenarios · pp. 5–6
- Overcooked uses feedforward convolutional networks on full-grid observations, whereas SMAX uses recurrent networks with action masking.Actor parameters are shared across agents, and critics are used only during training.Source: Figure 3 · p. 6
- Both families used 305 updates and 4 PPO epochs, with discount 0.99, GAE parameter 0.95, clipping 0.20, value coefficient 0.50, and linear learning-rate decay.Source: A. Transfer to Held-Out Scenarios · p. 6
- 58.2% and 52.5% higher task returns on two Overcooked layouts, and SMAX win-rate gains of 68.8 pp and 9.0 pp, raised mean held-out win rate from 51.0% to 89.9%.The same frozen program was used on both Overcooked layouts; held-out policies were trained from scratch.Source: A. Transfer to Held-Out Scenarios · p. 6
B. Comparison With Diagnostic and Reflection Controls
GARDEN achieved the highest mean win rate in both held-out SMAX scenarios and outperformed fixed-diagnostic and Eureka-inspired controls. Development examples show how training evidence informed reward revisions, but those drafts were not necessarily the programs selected for transfer.
B. Comparison With Diagnostic and Reflection Controls
- 89.9% mean held-out win rate exceeded fixed diagnostics at 84.2% and Eureka-inspired reflection at 79.2%, with GARDEN highest in both scenarios.It gained 4.9 pp on 10m vs 11m and 6.5 pp on SMACv2 10 units over fixed diagnostics; candidate and probe budgets, learner, and selection rule were held fixed.Source: B. Comparison With Diagnostic and… · p. 6, Table V · p. 7
C. Evidence-Guided Reward Development
- The two evidence-guided investigations concern development drafts rather than necessarily the programs selected for transfer.Each probe trained a new policy for 4 997 120 environment steps with one fixed training seed.Source: C. Evidence-Guided Reward Development · pp. 6–7
- In Overcooked, the candidate used both pots and increased mean deliveries from nine to twelve, while mean task return rose by 33.5%.Demand-aware shaping, wall-respecting routing, and partly shared production credits were combined, so the observations do not identify which component produced the change.Source: C. Evidence-Guided Reward Development · p. 6
- On 5m vs 6m, revising the SMAX draft after examining combat traces and missed firing opportunities raised wins from 22 to 238 of 256 episodes.The win rate increased from 8.6% to 93.0% under the same training budget and seed; each probe used one fixed training seed.Source: C. Evidence-Guided Reward Development · pp. 6–7
VI. CONCLUSION
GARDEN developed one reward program from several development scenarios, then froze it for new-policy training on held-out scenarios. The reported results support reward reuse across Overcooked and SMAX task families and motivate extending the approach to further task families and larger teams.
VI. CONCLUSION
- One reward program was developed from several scenarios, frozen, and used to train new policies on held-out scenarios.The learner and selection rule were held fixed.Source: §VI · p. 7
- Overcooked held-out task returns increased by 58.2% and 52.5%, while SMAX mean held-out win rate rose from 51.0% to 89.9%.Under matched candidate and probe budgets, GARDEN outperformed fixed-diagnostic and Eureka-inspired reward design by 5.7 and 10.7 pp.Source: §VI · p. 7
- Development cases show source inspection, custom analyses, and learning probes being used to diagnose coordination problems and revise reward programs.Source: §VI · p. 7
- The results support task-family reward engineering with autonomous evidence navigation as a route toward reusable rewards, with further task families and larger teams as extensions.Source: §VI · p. 7
- The implementation, prompts, and experimental configuration are available online, and other supporting data are available from the author on reasonable request.The Overcooked and SMAX environments are part of the publicly available JaxMARL library.Source: CODE AND DATA AVAILABILITY · p. 7