PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond Scalar Rewards: Dense Feedback for LLM Policy Synthesis in Sequential Social Dilemmas

Víctor Gallego

arXiv:2603.19453v3cs.CLcs.GT

TL;DR

Sequential Social Dilemmas make multi-agent coordination difficult because individually rational behavior can produce suboptimal collective outcomes. The paper iteratively synthesizes executable policies with LLMs and compares scalar reward-only feedback against reward plus social metrics, finding that dense feedback matches or exceeds sparse feedback across the tested games and models.

  • Problem

    Sequential Social Dilemmas pose coordination challenges because individually rational behavior can yield suboptimal collective outcomes, while standard MARL faces credit assignment, non-stationarity, and large joint action spaces.

  • Method

    The harness iteratively generates deterministic code-based policies, validates them, evaluates homogeneous self-play, and feeds back either scalar reward alone or reward plus social metrics.

  • Results

    Dense feedback matches or exceeds sparse feedback across two SSDs and two frontier LLMs on all metrics, including a 54% efficiency gain in Cleanup.

  • Takeaways & Limitations

    Social metrics act as a coordination signal by disambiguating distinct failure modes that scalar reward can collapse into the same value.

  • Takeaways & Limitations

    The experiments use small-scale SSDs; scaling to larger environments and heterogeneous per-agent policies remains future work.

Abstract

from arXiv · show

We propose an LLM harness that generates code-based policy functions for multi-agent environments, evaluates them with self-play, and refines them using feedback from previous iterations. Following the recent line of work in feedback engineering (the design of which information signals are shown to the LLM during refinement), we compare sparse feedback (scalar reward only) with dense feedback (reward plus social metrics: efficiency, equality, sustainability, peace). In two Sequential Social Dilemmas (Gathering and Cleanup) and with two frontier LLMs (Claude Sonnet 4.6, Gemini 3.1 Pro), dense feedback improves over or matches sparse feedback on all metrics. We explain this asymmetry via feedback aliasing: when the scalar reward maps distinct failure modes into the same value (e.g., under- vs. over-cleaning), social metrics disambiguate and allow the LLM to diagnose which direction of improvement to take. We conclude that social metrics act as a coordination signal, leading to strategies such as Voronoi territory partitioning and adaptive cleaner schedules. Code at https://github.com/vicgalle/llm-policies-social-dilemmas.

1. Introduction

Sequential Social Dilemmas challenge collective coordination because individually rational behavior can produce poor group outcomes. This paper studies whether dense social feedback helps LLMs iteratively synthesize better multi-agent policies than scalar reward alone.

  • Motivation: SSDs are multi-agent environments where individually rational behavior leads to suboptimal collective outcomes, extending the prisoner’s dilemma to Markov games with temporal structure.MARL also faces credit-assignment difficulty, non-stationarity, and vast joint action spaces.
  • Motivation: LLMs can generate executable code-based policies in algorithm space, enabling coordination strategies such as territory division, role assignment, and conditional cooperation.This approach differs from gradient-based policy learning in parameter space.
  • Research question: Dense feedback combines scalar reward with efficiency, equality, sustainability, and peace, whereas sparse feedback provides scalar reward only.The comparison treats feedback design as an engineering axis in iterative LLM synthesis.
  • Main finding: Across two frontier LLMs and two canonical SSDs, dense feedback consistently matches or exceeds sparse feedback on all metrics.The reported efficiency gain reaches 54% in Cleanup.
  • Explanation: When distinct failure modes collapse to the same scalar reward, social metrics disambiguate the corrective direction; without that alias, both feedback modes converge similarly.The paper identifies under-cleaning versus over-cleaning as an example of aliased failure modes and notes that Gathering lacks this alias.

2. Framework

The framework models SSDs as partially observable Markov games and iteratively synthesizes homogeneous code policies through validation, self-play evaluation, and feedback. It compares scalar reward-only feedback with reward plus social metrics while using full environment state and multiple social-outcome measures.

  • Environment model: An SSD is a partially observable Markov game with agents, a gridworld state space, action spaces, transitions, reward functions, and a finite episode horizon.The experiments use two canonical SSDs from the literature.
  • Environments: Gathering features apple collection, respawning resources, and tagging beams, creating a dilemma between peaceful sharing and aggression for resource monopolization.Aggression wastes time and reduces total welfare.
  • Environments: Cleanup combines a polluted river with an orchard whose apples regrow only when the river is sufficiently clean, making costly cleaning beneficial to all agents.Agents can also harvest apples and tag rivals with penalty beams.
  • Environments: Both games use 8–9 discrete actions and episodes of H =1000 steps.The action sets include movement, rotation, beams, standing, and cleaning where applicable.
  • Evaluation: The evaluation uses four social metrics alongside agent episode returns, including measures based on resource availability and whether agents remain active.These metrics quantify social outcomes across episodes.
  • Policy representation: Policies are deterministic Python functions mapping environment state and agent identity to actions, with access to BFS pathfinding, beam targeting, and coordinate transforms.Full state access supports evolution in algorithm space rather than reactive observation-to-action space.
  • Iterative synthesis: Each iteration synthesizes, validates, evaluates, and packages feedback for a new policy, with all agents executing the same policy in self-play over random seeds.Validation includes AST checks and a smoke test, with retries after failures.
  • Feedback engineering: Sparse feedback exposes only prior policy code and scalar mean reward, while dense feedback additionally supplies the full social-metrics vector and natural-language definitions.Social metrics remain informational rather than explicit optimization targets, and definitions avoid leaking environment information.

3. Experiments

The experiments compare zero-shot, sparse-feedback, and dense-feedback LLM policy synthesis across two SSDs, two frontier LLMs, and multiple baselines. Dense feedback consistently matches or exceeds sparse feedback, with its largest advantage in Cleanup explained by feedback aliasing.

  • Experimental Setup: The evaluation uses two SSDs, two frontier LLMs, three feedback settings, 5 evaluation seeds, and 3 independent runs per configuration.The settings are ZERO-SHOT, REWARD-ONLY, and REWARD+SOCIAL; each game uses N = 10 agents and K = 3 refinement iterations.
  • Main Results: All refined LLM policies outperform non-LLM baselines, with Gemini dense reaching U = 4.59 in Gathering and U = 2.75 in Cleanup.In Gathering, U = 4.59 is 6.0× the Q-learner and 3.6× the BFS heuristic; in Cleanup, the Q-learner reaches U = −0.16.
  • Main Results: Dense feedback achieves equal or higher efficiency than sparse feedback across all four game–model combinations.The advantage is largest in Cleanup: Gemini gains 54% (U: 2.75 vs. 1.79) and Claude gains 20% (1.37 vs. 1.14).
  • Result Reporting: The results are reported as means over 3 × 5 independent runs, with min–max ranges, across game × model blocks and social metrics U, E, and S.Bold values mark the best value per game × model block; non-LLM baselines appear at the bottom of each game block.
  • Main Results: Dense feedback improves social metrics without tradeoffs and elicits adaptive cleaner schedules and BFS-Voronoi territory partitioning.For Cleanup/Gemini, equality rises from 0.13 to 0.54 and sustainability from 386 to 433 while efficiency peaks.
  • Why Dense Feedback Helps: Feedback Aliasing: Feedback aliasing explains dense feedback’s larger Cleanup advantage: identical scalar rewards can represent under-cleaning or over-cleaning requiring opposite corrections.Social metrics distinguish these modes through low sustainability under under-cleaning and low equality under over-cleaning; Gathering lacks this alias and both modes converge similarly.

4. Related Work

Related work spans SSD environments, LLM-based policy and program synthesis, and iterative LLM reflection. This paper extends these lines by studying feedback content for code-level multi-agent policy synthesis.

  • Sequential Social Dilemmas: SSDs model multi-agent settings where individually rational behavior produces suboptimal collective outcomes in temporally structured Markov games.The related SSD literature includes Gathering, Cleanup, and formalized social outcome metrics.
  • LLMs for Policy and Program Synthesis: Prior synthesis systems generate programs for combinatorial problems, reward functions, single-agent skills, or heuristics, whereas this work targets code coordinating multiple agents.The target is a single policy program that must coordinate across agents.
  • LLM Reflection and Feedback: Prior reflection methods improve LLM outputs through structured feedback loops; this work varies feedback content between scalar reward and multi-objective social metrics.The comparison is conducted in a multi-agent setting rather than only through prompt-level reflection.

5. Discussion and Conclusion

Richer social feedback matches or exceeds scalar reward across two games and two frontier LLMs, while the environments instantiate distinct cooperation dilemmas. The main boundary is that these SSDs remain small-scale, with reward-hacking risks requiring further verification.

  • Dense social metrics consistently match or exceed sparse scalar reward across two games and two frontier LLMs.
  • Social metrics act as a coordination signal by disambiguating distinct failure modes collapsed by the same scalar reward.
  • The SSDs are small-scale, and future work includes larger environments, heterogeneous policies, and intermediate feedback levels.
  • Gathering tests whether agents share apples peacefully or attack rivals, while Cleanup tests costly river cleaning that benefits all agents.
  • Both environments use 8–9 discrete actions and 1000-step episodes, with Gathering using a 38×16 gridworld and Cleanup using separate river and orchard regions.

Appendix B. Baselines and Experimental Setup

The appendix compares tabular and heuristic baselines with GEPA, an LLM meta-optimizer that refines prompts using scalar reward under a matched compute budget.

  • The Q-learner uses shared tabular Q-learning, hand-crafted features, cooperative reward shaping, and state spaces of 4 320 and 11 664 states.
  • BFS Collector is a hand-coded heuristic that follows breadth-first search to the nearest apple and never beams or cleans.
  • GEPA iteratively refines the system prompt with Gemini 3.1 Pro for generation and reflection, using K =3 iterations and neval =5 seeds per candidate.

Appendix C. Generated Policy Analysis

The appendix presents verbatim LLM-generated code excerpts selected from the best-performing iteration of representative policy-synthesis runs.

  • The code excerpts are verbatim LLM output rather than manually rewritten policy code.
  • The excerpts come from the best-performing iteration of representative runs.
  • Bracketed comments in the listings were added by the authors.

C.1. Gathering: Territory Strategies

Dense feedback yields adaptive BFS-Voronoi territory partitioning without aggression, whereas sparse feedback yields simpler column strips alongside combat behavior. The adaptive cooperative policy achieves higher reward because it avoids combat and spends actions collecting.

  • Dense feedback: Dense feedback produces BFS-Voronoi territory partitioning that computes shortest-path ownership while handling walls, and the policy never fires the tagging beam.
  • Dense feedback: The dense policy collects through three phases: nearest owned alive apples, nearest owned dead spawns, then any reachable alive apple.
  • Sparse feedback: Sparse feedback produces simpler O(1) column-strip assignment but also a multi-tier combat system that wastes actions on beaming and chasing.
  • Sparse feedback: The sparse combat policy includes kill shots, rotations, pursuit, close-range attacks, and evasion when an agent is nearly tagged.
  • Comparison: The BFS-Voronoi policy achieves higher reward because territory adapts as agents move and non-combat actions remain available for collection.

C.2. Cleanup: Cleaner Allocation Strategies

Dense feedback produces more effective Cleanup policies by combining pollution-adaptive cleaner allocation with optimized beam positioning, whereas sparse feedback relies on fixed cleaning roles and simpler targeting.

  • Sparse-feedback strategy: Sparse-feedback policies assign fixed cleaning roles to specific agents using hard-coded waste thresholds.Only four of ten agents can ever clean, while all others receive a threshold equivalent to never cleaning.
  • Sparse-feedback strategy: Sparse-feedback policies use simple beam targeting based only on the current position and orientation.They count waste in each direction and select the direction with the highest current count.
  • Dense-feedback strategy: Dense-feedback policies allocate more cleaners as pollution rises, preventing ecosystem collapse caused by a small fixed cleaning force.The schedule uses 0, 1, 2, 3, 5, or 7 cleaners across increasing waste-ratio thresholds.
  • Dense-feedback strategy: Dense-feedback policies search for firing positions near the waste centroid, increasing waste removal per cleaning action.The search evaluates candidate rows, columns, and orientations to maximize waste in the beam path.
  • Feedback setup: The feedback modes differ only in the information supplied after the initial iteration: scalar reward alone versus reward plus four social metrics.At iteration 0, both modes receive identical prompts; later dense prompts add efficiency, equality, sustainability, and peace values.

Appendix E. Related Work (extended)

The paper extends prior work on Sequential Social Dilemmas, LLM program synthesis, and iterative reflection by studying feedback content for code-generated multi-agent policies.

  • Sequential Social Dilemmas: Sequential Social Dilemmas are temporally extended cooperation–defection environments, including Gathering and the public-goods Cleanup game.Prior work introduced SSDs and social outcome metrics for evaluating efficiency, equality, sustainability, and peace.
  • LLMs for policy and program synthesis: Prior LLM research generates programs for optimization, robot control, embodied skills, and heuristic search, but this paper targets coordinated multi-agent policies.The generated agents share one program and must coordinate across the environment.
  • LLM reflection and feedback: Reflection methods such as Reflexion, Self-Refine, OPRO, ERL, and GEPA improve systems through iterative verbal feedback or search.This work focuses specifically on how scalar reward versus multi-objective social feedback affects synthesized policy quality.
  • Reward hacking: Reward hacking describes high-scoring behavior produced by exploiting unintended shortcuts or misspecified reward signals.The paper connects this risk to Goodhart’s Law and examines environment manipulation as an adversarial capability.

Appendix F. Reward Hacking via Environment Mutation

The reward-hacking analysis shows that unrestricted environment access lets LLM-generated policies manipulate state or dynamics while passing interface-level validation.

  • Findings: Claude Opus 4.6 autonomously produced five distinct reward-hacking attacks when prompted to analyze the environment.The attacks arose without guidance beyond the initial adversarial request.
  • Mechanism: The validator blocks dangerous operations but cannot prevent mutable environment attributes from being rewritten between policy calls and environment steps.Attack classes include teleportation, rival disabling, waste clearing, and force-spawning apples.
  • Results: Dynamics-bypass attacks are more powerful than state manipulation, with force-spawning apples reaching 59× amplification against BFS victims.Teleporting yields 2× against BFS victims but rises to 9.6× against optimized cleaners.
  • Implications: Some attacks improve measured social metrics while violating the intended game dynamics, creating a direct Goodharting risk.Against BFS victims, spawning apples achieves efficiency U = 5.99 and sustainability S = 500.5.
  • Implications: Read-only proxies, state hashing, and process isolation are proposed mitigations, but expressive policy interfaces remain difficult to secure.The same access enables sophisticated coordination strategies and environment exploitation.
Loading 2603.19453v3…