PaperlayerView exact revision on arXiv

Source-linked AI summary

General Modular Harness for LLM Agents in Multi-Turn Gaming Environments

Yuxuan Zhang, Haoyang Yu, Lanxiang Hu, Haojian Jin, Hao Zhang

arXiv:2507.11633v1cs.AI

TL;DR

The paper addresses the lack of a principled way to study modular agent design in general-purpose multi-turn environments. It introduces a game-based harness with perception, memory, and reasoning modules, and reports that the full harness improves mean win rate over an unharnessed baseline across four games.

  • Problem

    The paper identifies a lack of a principled, systematic method for studying modular agent design in general-purpose multi-turn environments.

    Source: §1 · p. 1

  • Method

    The authors build a modular LLM/VLM harness with perception, memory, and reasoning components and use diverse games to study module contributions.

    Source: §1 · p. 2, §5 · p. 7

  • Results

    Across all four games, the full modular harness significantly improves mean win rate over the unharnessed LLM baseline (paired t-test, p < 0.05), with perception and memory showing complementary effects.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The framework uses games to enable systematic module analysis and provides a scalable, reproducible approach for studying modular agents across diverse environments.

    Source: §5 · pp. 7, 9

    The reported performance uses a small-scale setup of three evaluation runs per model per game, with models marked * based on a single run due to cost.

    Source: §3.1 · p. 4, §4.1 · p. 6

Abstract

from arXiv · show

We introduce a modular harness design for LLM agents that composes of perception, memory, and reasoning components, enabling a single LLM or VLM backbone to tackle a wide spectrum of multi turn gaming environments without domain-specific engineering. Using classic and modern game suites as low-barrier, high-diversity testbeds, our framework provides a unified workflow for analyzing how each module affects performance across dynamic interactive settings. Extensive experiments demonstrate that the harness lifts gameplay performance consistently over un-harnessed baselines and reveals distinct contribution patterns, for example, memory dominates in long-horizon puzzles while perception is critical in vision noisy arcades. These findings highlight the effectiveness of our modular harness design in advancing general-purpose agent, given the familiarity and ubiquity of games in everyday human experience.

1. Introduction

The introduction identifies a gap in systematic study of modular agent design and proposes games as a unified testbed for a three-module harness. It reports higher mean win rates than an unharnessed baseline across four games, with different contributions from perception and memory.

1. Introduction

  • The paper identifies a gap: existing workflows can be domain-specific and intricate, limiting transfer and making individual module contributions difficult to isolate.The authors propose interactive games as a testbed with well-defined rewards, diverse tasks, and challenging objectives.Source: §1 · p. 1
  • Gymnasium provides a shared interface for diverse multi-turn games, while Stable Retro extends it to classic console titles with minimal integration overhead.The paper describes the games as accessible to non-experts, with evaluations intended to reflect core cognitive abilities rather than domain-specific hacks.Source: §1 · pp. 1–2
  • Sokoban, Candy Crush, 2048, and Tetris are selected to cover a broad spectrum of multi-turn interaction patterns.The games are used as a low-barrier, scalable testbed, and the adapter can integrate additional Gym or retro environments with minimal engineering effort.Source: §1 · p. 2
  • Across all four games, the full modular harness significantly improves mean win rate over the unharnessed LLM baseline (paired t-test, p < 0.05).Ablations show complementary effects: perception-only configurations benefit Sokoban and Tetris, while memory-only setups tend to improve 2048 and Candy Crush, especially for non-reasoning models that repeat invalid moves.Source: §1 · p. 2
  • The harness decomposes one LLM/VLM agent into plug-and-play perception, memory, and reasoning modules, whose contributions can be examined through controlled ablations.Perception processes grid or detailed visual inputs; memory stores recent trajectories and synthesizes reflections; reasoning combines perceptual embeddings and memory traces to make sequential decisions.Source: §1 · p. 2
  • The experiments report consistent performance gains across diverse games and distinct contributions from the harness modules.The introduction frames this as evidence for the effectiveness of structured modular design.Source: §1 · p. 3

2. Related Works

Related work spans game benchmarks, harnesses for gaming agents, and multi-turn agents in non-game settings. The paper positions its approach alongside work using feedback, memory, perception, and reasoning across these areas.

2. Related Works

  • Game benchmarks cover grid maps, strategy reasoning, conversational play, human interaction, multi-task performance, and multimodal murder-mystery games.The cited examples include GameTraversalBenchmark, GameBench, GameEval, GameArena, LMGame-Bench, and WhodunitBench.Source: §2 · p. 3
  • Gaming-agent harnesses use mechanisms such as memory, feedback, reflection, and iterative synthesis, while LMAct emphasizes in-context observation-action trajectories.The passage contrasts LMAct's imitation-learning emphasis with prompting agents to reflect and improve from environment feedback.Source: §2 · p. 3
  • Non-game agent work integrates reflection, reasoning, perception, and interaction across multi-turn tasks in web, app, and other domains.The paper cites Reflexion, ReAct, AgentBench, WebVoyager, and AppAgent as examples.Source: §2 · p. 3

3. Methods

The methods use familiar games as a broad testbed, defining game-specific rewards and modular perception, memory, and reasoning components. They also standardize prompts through a two-stage optimization procedure.

3. Methods

  • The methods reuse familiar, popular games while preserving their original settings to evaluate a broad range of perception and generation abilities.Source: §3 · p. 3

3.1. Games

  • The suite includes Tetris, Sokoban, Candy Crush, and 2048, which exercise visual perception, spatial reasoning, and long-horizon planning.Source: §3.1 · pp. 3–4

3.2. Game Metric Design

  • Game performance is measured with progression rewards for incremental advances or long-horizon rewards for completing multi-step objectives.Source: §3.2 · pp. 4–5

3.3. Modular Harness Design

  • The harness converts game inputs into text, vision, or combined representations, stores recent trajectories for reflection, and uses a controller to integrate modules and select actions.Source: §3.3 · p. 5

3.4. Prompt Standardization

  • Prompt standardization first uses empirical prompt engineering, then applies DSPy with SIMBA to refine templates and select the best-performing prompt across target models.Source: §3.4 · p. 5

4. Experiments

Experiments evaluate 13 models across four classical video games, comparing harnessed, unharnessed, and random play and examining module ablations, prompt standardization, and benchmark correlations.

4.1. Modular Harness Performance: Quantitative Analysis

  • Evaluations were small-scale, with three runs per model per game because running the latest foundation models is costly.The authors characterize these results as preliminary.Source: §4.1 · p. 6

4.1.1. SEPARATION FROM RANDOM BASELINE: GLASS’S δ EFFECT SIZES

  • Harnessed Glass’s δ values were positive in 29/30 cases, versus 17/30 unharnessed, and the harness outperformed the unharnessed configuration in 23/30 cases (≈76.7%).The comparison excludes Sokoban because random play has zero variance; the evaluation used three runs per model per game due to foundation-model costs.Source: §4.1.1 · p. 6, §4.1 · p. 6

4.2. Modular Ablation

  • Perception boosts Sokoban and Tetris, with particularly notable gains for Gemini and O4-mini in Sokoban and O4-mini in Tetris.The authors report that structured spatial inputs appear to unlock planning behaviors otherwise latent under raw token-based input.Source: §4.2 · p. 6
  • Memory improves long-horizon 2048 and Candy Crush, especially for weak zero-shot models in 2048, and reduces Candy Crush score variance across episodes.The authors connect memory to preserving context and strategy over time.Source: §4.2 · p. 6
  • Combined perception and memory yield additive or even multiplicative improvements across all four games, and help distinguish top-performing models in Candy Crush.Dual-module evaluations may expose distinctions that zero-shot or single-module setups obscure; the modules also increase evaluation discriminative power.Source: §4.2 · pp. 6–7

4.1.2. DIRECT COMPARISON: PAIRED-SAMPLE T-TESTS

  • All four games improve significantly with the harness: Candy Crush (+217.50 points), Sokoban (+1.97), 2048 (+17.81), and Tetris (+5.60) (p < 0.05).Figure 2 plots paired Harness–No Harness score differences across ten models per game; Candy Crush has the highest median gain.Source: §4.1.2 · p. 6, Figure 2 · p. 7

4.3. Prompt Standardization

  • DSPy-based prompt optimization improves average performance and reduces variance across models and games, following empirical prompt design.The authors report that well-crafted empirical prompts can still vary by more than one standard deviation.Source: §4.3 · p. 7

4.4. Correlation Study

  • Across 20 benchmarks, games correlate positively with commonly evaluated benchmarks: Sokoban with math and coding, Tetris and 2048 with pattern recognition, and Candy Crush with coding.The study examines game performance as a possible proxy for broader model capabilities.Source: §4.4 · p. 7

5. Conclusion

The paper introduces a modular harness that consistently improves performance in multi-turn interactive environments. Its framework supports module ablations and scalable, reproducible study across diverse games.

  • The modular harness consistently improves model performance in multi-turn interactive environments.Source: §5 · p. 7
  • Table 2 covers game scores across four games and conditions, while Table 3 reports Glass’s δ by model, condition, and game.Source: Table 2 · p. 8, Table 3 · p. 8
  • Ablating perception, memory, and reasoning isolates their contributions to overall agent behavior.Source: §5 · pp. 7, 9
  • The framework supports diverse game environments and provides a scalable, reproducible solution for studying modular agents.Source: §5 · p. 9

A. Game Setting Details

The game-setting details define game-specific scores to quantify proficiency, using measures intended to reflect each model’s capability and permit consistent statistical analysis.

B.2. Qualitative Study

The qualitative study presents case studies of the latest models’ reasoning traces during 2048 gameplay.

  • The case studies examine the latest models’ reasoning traces during 2048 gameplay.Source: §B.2 · p. 13

B.2.1. MEMORY MODULE

The 2048 case study contrasts how GPT-4o and o3 reflect on moves and plan subsequent actions. A prompt template directs reflections toward state changes, action effectiveness, patterns, and future strategy.

B.2.1. MEMORY MODULE

  • o3’s reflections link detailed tile changes and merge ladders to next steps within a longer-term corner-dominance strategy.Source: §B.2.1 · p. 13
  • GPT-4o’s reflections track empty-space changes and corner preservation, but also flag moves that fail to create merges or advance tile positioning.Source: §B.2.1 · p. 14
  • o3’s examples evaluate moves by merges, empty-space changes, and preserving the 256 corner, advising against space-losing moves without progress.Source: §B.2.1 · p. 15

Memory Module Prompt Template

C. Benchmark List for Correlation Study

The study uses 20 public benchmarks across seven capability categories to examine general-purpose abilities, then uses model rankings to analyze correlations and latent capability axes.

C. Benchmark List for Correlation Study

  • The study covers 20 public benchmarks spanning factual knowledge, physics, mathematics, coding, visual reasoning, language understanding, and puzzle solving.The benchmarks support correlation and decomposition analyses.Source: C. Benchmark List for Correlation Study · p. 16
  • The rankings are used to compute Spearman correlations and identify latent capability axes through low-rank decomposition.Source: C. Benchmark List for Correlation Study · p. 16
  • Table 4 groups model rankings across the 20 benchmarks by capability category.The table caption specifies that rank 1 is best.Source: Table 4 · p. 17

D. Prompt Optimization

The prompt-optimization section contrasts empirically designed 2048 prompts with DSPy-optimized prompts and reports reduced performance discrepancy between candidate prompts across target models.

D.1. Empirically Optimized Baseline Prompts

  • The empirical 2048 templates give strategic guidance such as corner anchoring, maintaining merge paths, and planning several moves ahead, alongside game history and symbolic board features.Source: §D.1 · pp. 18–19
  • The empirical templates prescribe a thought-and-move response, with the selected action restricted to up, down, left, or right.Source: §D.1 · pp. 18–19

D.2. DSPy Optimized Prompts and Comparison

  • The DSPy prompt examples require a concise two-line thought-and-move response and prohibit additional text.Source: §D.2 · pp. 21–22
  • The DSPy-optimized templates retain 2048 strategy guidance, including corner anchoring, stable tile arrangements, merge paths, and forward planning.They also include previous game history and symbolic board features.Source: §D.2 · pp. 21–22

D.3. Performance Comparison

  • The DSPy pipeline searches for a prompt with the highest average reward across target models, using five optimizer models and 20 optimization steps.Source: §D.3 · p. 23
  • 33.8% to 63.5%: prompt optimization reduced performance discrepancy between two candidate prompts across three target models over three runs.Table 5 reports performance across prompt types for Game 2048 with the harness.Source: §D.3 · p. 23, Table 5 · p. 23

Related papers

Loading 2507.11633v1…