PaperlayerView exact revision on arXiv

Source-linked AI summary

S3Gym: Can LLMs Turn Self-Testing and Self-Judging into Self-Improvement?

Jiajun Shi, Siyuan Tao, Yuhao Wu, Zexuan Wang, Jingyuan Zhang, Jiaheng Liu, Xinping Lei, Xinrong Zhang, Siyuan Fang, Zhewen Tan, Tianle Cai, Junhao Fang, Jiameng Huang, Yueyang Wang, Jinkai Liu, Yuxuan Zhang, Jian Yang, Zhoujun Li, Shen Yan, Wenhao Huang, Ge Zhang

arXiv:2608.31100v1cs.CL

TL;DR

Existing agent benchmarks largely treat LLMs as fixed policies, leaving limited evidence about whether agents can improve from their own interactions. S3Gym evaluates this capability and finds that the most effective experience pathway depends on task structure, while training can be unstable.

  • Problem

    Existing benchmarks largely evaluate LLMs as fixed policies, offering limited insight into whether agents can use their past interactions to improve future behavior.

    Source: §1 · p. 1

  • Method

    S3Gym links Self-Testing, Self-Judging, and behavioral change across separate exploration and held-out evaluation phases, comparing History ICL, Summary Memory, and parameter Training.

    Source: §1 · p. 3

  • Results

    Summary memory helps when compact rules transfer across episodes, while direct history often performs better in reactive control games; training on self-generated trajectories remains unstable.

    Source: §7 · p. 19

  • Takeaways & Limitations

    Future self-improving agents need better judgment calibration, memory selection, and trajectory filtering, not just more interaction data.

    Source: §7 · p. 19

    Whether trajectories bootstrap an effective supervised signal requires trajectory-level analysis to verify.

    Source: §6.1 · p. 16

Abstract

from arXiv · show

Large language models (LLMs) increasingly interact with external environments and accumulate substantial behavioral experience, yet existing agent benchmarks largely evaluate them as fixed policies. It therefore remains unclear whether an agent can actively test its behavior, judge the resulting experience, and use that experience to improve future decisions. We introduce \textbf{S\textsuperscript{3}Gym}, an interactive benchmark for evaluating LLM self-improvement through three coupled capabilities: \textbf{Self-Testing}, \textbf{Self-Judging}, and \textbf{Self-Improvement}. S$^3$Gym separates permissive exploration from strict held-out evaluation and instantiates this protocol in seven text-based games with executable environment verifiers. We evaluate three pathways for incorporating interaction experience: direct History ICL, score-conditioned Summary Memory, and parameter Training. Our experiments reveal that self-improvement is neither automatic nor uniform. Context-level experience improves performance for several model--game pairs, but the most effective pathway depends strongly on the task structure: summaries are beneficial when experience can be compressed into reusable strategic rules, yet often underperform raw history when success depends on precise, state-contingent information. Parameter training produces substantial gains on some tasks, but also exhibits unstable improvement and severe negative transfer on others. These findings show that recognizing successful actions is insufficient; agents must also transform feedback into executable and transferable policies. S$^3$Gym provides a unified framework for diagnosing this process and identifying the bottlenecks that prevent agents from translating interaction experience into reliable self-improvement.

1 Introduction

S3Gym addresses the limits of fixed-policy benchmarks by evaluating whether LLMs can turn environmental experience into better behavior through Self-Testing, Self-Judging, and Self-Improvement.

Framework

  • The framework treats experience-driven learning as a cycle of gathering diagnostic evidence, interpreting success and failure, and changing future behavior.It defines these capabilities as Self-Testing, Self-Judging, and Self-Improvement.Source: §1 · pp. 2–3

Benchmark and protocol

  • S3Gym separates permissive exploration from stricter held-out evaluation to test whether experience transfers and whether self-judgment errors impede improvement.Self-judgments organize and reuse interaction experience, while environment-verifiable scores measure behavioral performance.Source: §1 · pp. 2–3

Environments

  • Seven text-based games provide verifiable testbeds spanning latent-rule induction, constraint satisfaction, numerical transformation, spatial planning, resource allocation, survival, and multi-agent strategy.Related exploration and evaluation configurations test transfer rather than memorization of isolated actions or seeds.Source: §1 · pp. 2–3

Experience-integration pathways

  • S3Gym compares History ICL, Summary Memory, and parameter Training under one protocol, spanning detailed in-context evidence, compressed external rules, and persistent parameter updates.Raw histories preserve detail but consume context; summaries depend on judgment quality, while training may amplify incorrectly judged experience.Source: §1 · pp. 2–3

Evaluation scope

  • The evaluation compares model-generated judgments with verifier-computed outcomes and analyzes judgment reliability, summary effectiveness, cross-game consistency, improvement dynamics, and failure modes.The benchmark assesses whether agents identify useful or harmful experience and translate judgments into actionable directions.Source: §1 · p. 3

2 Background

S3Gym isolates the feedback-and-improvement loop that conventional executable benchmarks presuppose, asking agents to generate evidence, judge it, and improve future behavior. Its motivation is the deployment gap between a capable model and a working system.

2 Background

  • Conventional agent benchmarks fix executable tasks, rewards or judges, and scaffolds, whereas deployment success depends on a system being used, continuing to work, and improving after failures.In deployment, adapting a general-purpose model to customer data formats, workflows, and operational constraints is part of the work.Source: §2 · p. 3
  • Forward-deployed engineers (FDEs), a title popularized by Palantir and adopted by frontier-model firms, adapt models to deployment environments; human engineers bridge capable models and working systems.FDEs are embedded in deployment environments, while related roles include solutions architects, applied engineers, and platform engineers.Source: §2 · pp. 3–4
  • From the model’s perspective, deployment requires translating vague needs into objectives, constructing reliable feedback, and building and maintaining usable execution systems.These systems include tools, context management, state, lifecycle logic, and verification interfaces that may need adaptation as requirements change.Source: §2 · p. 4
  • S3Gym withholds verifier outcomes during exploration, requiring agents to self-test, self-judge, and use judged experience to improve behavior, while retaining the verifier as external ground truth.This isolates whether self-judgments match actual outcomes and whether judged experience produces transferable improvement.Source: §2 · p. 4

3 Related Work

Prior work supplies interactive benchmarks and experience-based adaptation methods, while S3Gym shifts evaluation toward whether agents can judge and convert their experience into improved behavior.

3.1 Interactive Agent Benchmarks

  • Interactive benchmarks including AgentBench [15], AgentBoard [16], TextWorld [5], ALFWorld [27], and ScienceWorld assess multi-step agents but generally treat models as fixed policies.S3Gym builds on controllable, verifiable environments while shifting the target from static task competence to experience-driven behavioral improvement.Source: §3.1 · p. 4

3.2 Experience-Based Agent Improvement

  • Prior improvement work reuses experience through reflection [26], skill libraries [30], trajectory distillation [19], [7], [33], and parameter optimization, [11], [9], [36].Interactive approaches include RetroAgent [39] and test-time self-improvement [1], but typically study specific mechanisms or settings; S3Gym compares History ICL, Summary Memory, and parameter training under a unified exploration and held-out evaluation protocol.Source: §3.2 · pp. 4–5

3.3 Evaluating Adaptive and Self-Improving Agents

  • Unlike benchmarks targeting individual adaptation artifacts, S3Gym evaluates self-testing, explicit self-judging, and improvement under executable verification across matched strategies and stricter held-out configurations.Recording model judgments alongside executable outcomes helps distinguish inaccurate experience evaluation from failure to turn correctly identified feedback into transferable behavior.Source: §3.3 · p. 5

4 Method

S3Gym tests whether agents can use experience from permissive exploration to improve decisions evaluated under stricter, held-out conditions. It compares retaining interaction history, compressing experience into summaries, and training model parameters.

4.1 Overview

  • Each cycle moves from permissive exploration through self-judgment and experience reuse to evaluation on stricter, held-out configurations.Exploration trajectories are incorporated as raw history, summary memory, or training examples before the updated agent is evaluated.Source: §4.1 · p. 6

4.2 Interaction Protocol and Formalization

  • Agents produce actions and immediate self-scores, while verifier rewards remain benchmark-side signals unavailable during exploration.The interaction history records the agent’s self-score and observable state transition; verifier rewards assess self-judgment reliability, while final scores measure episode performance.Source: §4.2 · pp. 7–8

4.3 Self-Judging and Experience Consolidation

  • Self-judgment marks low-scoring segments as possible mistakes and high-scoring segments as candidate strategies, which are retained as raw trajectories or compressed summaries.Summaries capture strategies to retain, mistakes to avoid, and concrete directions for the next interaction cycle.Source: §4.3 · p. 8

4.4 Exploration and Evaluation Phases

  • Exploration uses more permissive conditions, while evaluation tests transfer under stricter rules with disjoint seeds and no evaluation data reused for improvement.Exploration can use smaller state spaces, weaker failure penalties, or extra trials; evaluation trajectories are excluded from history, memory, and training data.Source: §4.4 · p. 9

4.5 Self-Improving Pathways

  • The three pathways retain score-annotated trajectories in context, provide consolidated summaries, or fine-tune parameters using examples derived from exploration.Training keeps high-scoring actions as positive examples and filters or corrects low-scoring actions; all pathways share the same base model, seeds, interaction budget, and decoding settings.Source: §4.5 · pp. 9–10

5 Experiments

Across seven text-based games, S3Gym tests whether models can use interaction experience to improve under strict evaluation; results show that gains depend on the model, game, and experience pathway.

5.1 Benchmark Games

  • The seven games cover complementary interactive reasoning, with generally permissive exploration configurations and stricter or more difficult evaluation configurations.The tasks include latent-rule induction, constraint satisfaction, numerical transformation, spatial planning, long-horizon control, resource allocation, and multi-agent strategy.Source: §5.1 · p. 10

5.2 Experimental Setup

  • The main comparison covers seven proprietary LLMs with complete results across all seven games, excluding models with unfinished checkpoints.The evaluated models are GPT-4o, GPT-4.1, o3-mini, Gemini-2.5-Flash, Gemini-2.5-Pro, GPT-5.5, and Gemini-3.5-Flash.Source: §5.2 · p. 10
  • History ICL retains detailed interaction evidence but uses more context, whereas Summary Memory compresses trajectories into rules, mistakes, and future directions and depends on Self-Judging quality.Compression can discard state-specific evidence or preserve incorrect judgments, so summaries may help when long trajectories are hard to use directly but are not uniformly superior to raw history.Source: §5.2 · p. 10, §5.3 · p. 11

5.3 Main Results

  • Max. captures best observed capability, Avg. overall exploration performance, and AUC+ sustained improvement above the initial score; sparse-score games can show isolated peaks, unlike denser sequential-feedback tasks.Because game score scales differ, raw AUC+ should be compared within games; cross-game consistency is better assessed using within-game ranks or the number of games improved.Source: §5.2 · p. 11, §5.3 · p. 13
  • Leaders vary by model, game, and pathway: History ICL favors Gemini-3.5-Flash on Chess, Snake, and Trust Evolution and GPT-5.5 on brittle tasks, while Summary Memory shifts Chess, Nullify, and Tetris leadership to GPT-5.5.Under Summary Memory, GPT-5.5 also has the highest average and maximum scores on Snake, while Gemini-3.5-Flash remains strongest in average performance on PvZ and Trust Evolution.Source: §5.3 · p. 11, Table 4 · p. 12
  • Summary Memory is selective: Gemini-2.5-Flash raises Minesweeper AUC+ from 0.000 to 7.794 and PvZ AUC+ from 24.402 to 238.501, while GPT-5.5’s PvZ AUC+ falls from 548.499 to 33.219.GPT-5.5’s Chess AUC+ rises from 0.474 to 16.840, while Gemini-3.5-Flash’s Chess AUC+ falls from 12.280 to 0.000 and Trust Evolution AUC+ from 200.000 to 55.095.Source: §5.3 · pp. 11, 13

6 Analysis

Analysis finds that parameter training and score-conditioned summaries can improve behavior, but neither works uniformly across games; self-judging is only partially reliable and does not by itself predict subsequent improvement. Summaries work best when they turn feedback into precise, reusable action rules, while state-dependent tasks often require details that summaries discard.

6.1 RQ1: Can Parameter Training Enable Self-Improvement?

  • Parameter training improved Trust Evolution substantially but was inconsistent overall, with weak Chess gains, transient Snake success, no gains on three games, and severe Plants-vs-Zombies degradation.The results indicate that improvement depends on the quality, diversity, and correctness of the experience internalized through training.Source: Figure 7 · p. 15, §6.1 · pp. 15–16
  • Trust Evolution rose from 0 to a maximum of 30, with a post-training average of 8.684 and AUC+ of 163.5; Plants-vs-Zombies fell from 23 initially to 6 at every updated checkpoint.Chess reached 0.0667 only at the final checkpoint, while Snake scored 1 at epochs 8, 12, and 15 before returning to zero; Minesweeper, Nullify, and Tetris stayed at zero. Too few successful exploration trajectories are a possible explanation for the latter results, but require trajectory-level analysis.Source: §6.1 · pp. 15–16

6.2 RQ2: Is Agent Self-Judging Reliable?

  • Self-Judging was only partially reliable: high event agreement in five games was partly attributable to zero-reward transitions, and PvZ had the largest calibration error despite high agreement.Chess and Trust were the clearest failures, with weak agreement and over-confidence; reliability weakened where rewards depended on hidden dynamics, opponent strategy, or exact multi-object state prediction.Source: Table 5 · p. 16, §6.2 · pp. 16–17
  • Across runs, judgment–improvement coupling was near zero (ρ(A, g) = −0.010; ρ(−E, g) = −0.018), while event-agreement correlations with NABA were weakly negative (r = −0.23; ρ = −0.11).Accurate local scoring alone was insufficient; agents also needed to turn feedback into reusable state abstractions, decision rules, and exploration strategies.Source: Table 6 · p. 17, §6.2 · p. 17

6.3 RQ3: Do Score-Conditioned Summaries Produce Effective Iteration Directions?

  • Summary Memory improved average NABA on Nullify, Tetris, and Trust, whereas direct history performed better on average in Minesweeper, PvZ, and Snake.Summaries helped when experience could be compressed into stable policy rules, but often lost board configurations, timing, hazard positions, and partially observed risk needed for state-contingent decisions.Source: Table 7 · p. 18, §6.3 · p. 18

Successful summaries

  • Executable summaries yielded ∆NABA values of +66.89, +14.42, and +25.49 by turning payoff feedback, line-clearing experience, and local number constraints into action rules.The examples show how summaries can specify opponent-conditioned choices, look-ahead placement, or constraint-based moves.Source: Successful summaries · pp. 18–19

Failure summaries

  • Across Plants-vs-Zombies, Minesweeper, Chess, and Snake, coarse summaries yielded ∆NABA of −2.56, −22.09, −33.27, and −22.00, while a concrete arithmetic procedure achieved +8.50.The arithmetic case used terminal units −19 and 19; choosing 0 1 earned reward 1.0. Effective summaries require a reusable failure mode and an action rule precise enough to execute in a new state.Source: Failure summaries · p. 19

7 Conclusion

S3GYM makes self-improvement measurable by testing whether LLMs can turn interaction histories into future performance. Its findings are mixed: effective experience reuse depends on task structure, while training on self-generated trajectories remains unstable.

7 Conclusion

  • Summary memory helps when compact rules transfer across episodes, while direct history often works better in reactive control games where details matter.Training on self-generated trajectories remains unstable in the current implementation.Source: §7 · p. 19
  • S3GYM measures whether LLMs transform interaction history into future performance by separating relaxed exploration from stricter evaluation.It supports both context-level and training-level reuse of trajectories.Source: §7 · p. 19
  • Future self-improving agents need better judgment calibration, memory selection, and trajectory filtering—not simply more interaction data.Source: §7 · p. 19

Appendix · A Game Prompt Templates

The appendix documents canonical prompts for seven games and their shared interaction protocol, then presents game-specific rules, states, actions, and answer formats. It also specifies how raw-history and summary memory differ while preserving the same current-game information and self-judging contract.

A Game Prompt Templates

  • The canonical templates are transcribed from the game implementations, with whitespace normalized and episode-specific values replaced by placeholders.At inference, each game-specific system prompt is combined with the current-step prompt and, for memory-based agents, available hint messages.Source: A Game Prompt Templates · p. 23

A.1 Shared Interaction Protocol

  • Summary memory compresses truncated cross-episode experience into cached tips, while direct memory supplies retained raw steps; both retain current-episode history and the game prompt.Direct history is truncated at its tail if the context budget is exceeded; summary memory makes a separate compression call at episode step 0.Source: §A.1 · p. 23
  • Every game prompt requires an action and a self-judged immediate reward, with END assigned score 0 when the agent terminates the episode.The score is intended to reflect the reward under the game rules, not free-form confidence.Source: §A.1 · p. 24

A.2 State Inference and Deduction

  • State-inference prompts ask agents to predict piece locations from board sequences, while Minesweeper prompts require one uncover, flag, or unflag action on a partially observed board.The Minesweeper score is the ratio of correctly flagged mines to total mines.Source: §A.2 · p. 24

A.3 Symbolic and Spatial Planning

  • Symbolic and spatial prompts define unit-cancellation by combining indexed values toward zero, and Tetris play by choosing a drop column and rotation to maximize cleared rows.Complete cancellation scores 1; otherwise the terminal score is 0.Source: §A.3 · p. 25

A.4 Reactive Control and Strategic Interaction

  • Reactive-control prompts ask for one snake movement under collision and no-reversal rules, whereas Plants-vs-Zombies permits zero or more plant placements each turn.Snake episodes are capped at a maximum number of steps; Plants-vs-Zombies scores one point per survived turn.Source: §A.4 · p. 25
  • The repeated-interaction prompt frames collaborate-or-cheat decisions as inference about an opponent with a stable within-episode strategy and specified payoffs.The agent observes round and interaction history and chooses one action for the next round.Source: §A.4 · p. 25

Related papers

Loading 2608.31100v1…