Source-linked AI summary
S3Gym: Can LLMs Turn Self-Testing and Self-Judging into Self-Improvement?
Jiajun Shi, Siyuan Tao, Yuhao Wu, Zexuan Wang, Jingyuan Zhang, Jiaheng Liu, Xinping Lei, Xinrong Zhang, Siyuan Fang, Zhewen Tan, Tianle Cai, Junhao Fang, Jiameng Huang, Yueyang Wang, Jinkai Liu, Yuxuan Zhang, Jian Yang, Zhoujun Li, Shen Yan, Wenhao Huang, Ge Zhang
TL;DR
LLM benchmarks commonly evaluate fixed policies, leaving unclear whether agents can test, judge, and learn from their own environmental experience. S3Gym evaluates this loop across held-out text-game tasks and compares history, summary, and parameter-based reuse. Results are mixed: pathway effectiveness depends on task and model, while parameter training can be unstable and self-judgment affects what experience is retained.
Problem
Existing agent benchmarks usually evaluate models as fixed policies, providing limited insight into whether they can use past interactions to improve future behavior.
Method
S3Gym evaluates Self-Testing, Self-Judging, and Self-Improvement across separate exploration and held-out evaluation phases, comparing History ICL, Summary Memory, and parameter Training.
Results
Self-improvement is mixed and task-dependent: context-level pathways help some model–game pairs, while parameter training yields substantial gains on some tasks but unstable improvement and negative transfer on others.
Takeaways & Limitations
Reliable self-improvement requires agents to transform feedback into executable, transferable policies through calibrated judgment, memory selection, and trajectory filtering.
Takeaways & Limitations
Parameter-training conclusions remain constrained because noisy or incorrectly judged trajectories can be consolidated into the model and overwrite effective behavior.
Abstract
from arXiv · showhide
Large language models (LLMs) increasingly interact with external environments and accumulate substantial behavioral experience, yet existing agent benchmarks largely evaluate them as fixed policies. It therefore remains unclear whether an agent can actively test its behavior, judge the resulting experience, and use that experience to improve future decisions. We introduce \textbf{S\textsuperscript{3}Gym}, an interactive benchmark for evaluating LLM self-improvement through three coupled capabilities: \textbf{Self-Testing}, \textbf{Self-Judging}, and \textbf{Self-Improvement}. S$^3$Gym separates permissive exploration from strict held-out evaluation and instantiates this protocol in seven text-based games with executable environment verifiers. We evaluate three pathways for incorporating interaction experience: direct History ICL, score-conditioned Summary Memory, and parameter Training. Our experiments reveal that self-improvement is neither automatic nor uniform. Context-level experience improves performance for several model--game pairs, but the most effective pathway depends strongly on the task structure: summaries are beneficial when experience can be compressed into reusable strategic rules, yet often underperform raw history when success depends on precise, state-contingent information. Parameter training produces substantial gains on some tasks, but also exhibits unstable improvement and severe negative transfer on others. These findings show that recognizing successful actions is insufficient; agents must also transform feedback into executable and transferable policies. S$^3$Gym provides a unified framework for diagnosing this process and identifying the bottlenecks that prevent agents from translating interaction experience into reliable self-improvement.
1 Introduction
S3Gym frames self-improvement as a loop in which agents test behavior, judge experience, and incorporate it into future decisions. It evaluates this process across separate exploration and held-out evaluation phases using multiple experience-integration pathways.
- Motivation: Existing benchmarks usually evaluate agents as fixed policies, leaving their ability to improve from past interactions insufficiently measured.They primarily assess capability at evaluation time rather than experience-driven behavioral change.
- Evaluation protocol: The benchmark separates permissive exploration from stricter held-out evaluation, allowing agents to search for strategies before testing whether they generalize.Exploration includes successful, unsuccessful, and partially successful trajectories.
- Environments: The benchmark uses text-based games with multi-turn interaction, partial observability, delayed consequences, and programmatically verifiable scores.These properties provide rich trajectories while preserving reproducible evaluation.
- Framework: S3Gym formalizes self-improvement as the coupled process of Self-Testing, Self-Judging, and subsequent behavioral change.The framework connects experience generation, interpretation, and reuse in an iterative evaluation protocol.
- Experience integration: S3Gym compares History ICL, Summary Memory, and parameter Training as distinct mechanisms for incorporating interaction experience.These pathways operate at short-term context, external-memory, and long-term parameter levels.
- Empirical analysis: The study evaluates whether agents can identify useful and harmful experience and translate those judgments into actionable improvement directions across seven environments.It also analyzes improvement dynamics, judgment reliability, summary effectiveness, cross-game consistency, and failure modes.
2 Background
The background identifies a deployment gap that standard agent benchmarks usually presuppose: converting vague goals, unreliable feedback, and incomplete execution systems into workable environments. S3Gym focuses specifically on the feedback-and-improvement layer.
- Benchmark assumptions: Standard benchmarks typically assume that the task, reward or judge, and execution scaffold have already been made executable.This makes controlled comparison possible but leaves deployment-oriented engineering largely outside evaluation.
- Deployment gap: Forward-deployed engineers turn general-purpose models into systems that work with customer data, workflows, and operational constraints.Their success criterion includes continued real-world use and improvement after failures.
- Missing structure: Deployment work must translate vague targets into objectives, construct reliable feedback signals, and build usable tools, state, and verification interfaces.These are the structures benchmark designers commonly presuppose.
- S3Gym’s focus: S3Gym isolates the agent-facing feedback loop while retaining an executable verifier as external ground truth and hiding verifier outcomes during exploration.Agents must generate evidence through Self-Testing, evaluate it through Self-Judging, and use it for later behavior.
3 Related Work
Prior work studies interactive evaluation and several ways agents can reuse experience, but usually treats adaptation mechanisms or outcomes separately. S3Gym instead compares experience pathways within shared environments while diagnosing where improvement fails.
- Interactive benchmarks: Interactive benchmarks such as AgentBench, AgentBoard, TextWorld, ALFWorld, and ScienceWorld provide controlled settings for evaluating multi-step agent behavior.Their primary focus is decision-making, tool use, planning, or language-grounded interaction.
- Open gap: Existing benchmarks generally treat models as fixed policies and use trajectories mainly for task-completion measurement or failure diagnosis.They provide limited evidence about transforming experience into improved future behavior.
- Experience reuse: Experience-reuse research spans verbal feedback and contextual reflection, external skill libraries, and parameter updates from self-produced trajectories.Reflexion, Voyager, STaR, ReST, Re-ReST, and related work represent these levels.
- Judgment reliability: Model-generated supervision can be useful but remains vulnerable to bias, calibration errors, and weak identification of causally important steps.Prior audits report that several judgment signals perform no better than chance against executed replay for step-level credit assignment.
- Adaptive evaluation: Recent adaptive benchmarks examine post-training, long-term evolution, persistent harnesses, or retained experience, but emphasize evolutionary outcomes or individual mechanisms.S3Gym targets the process of converting interaction experience into future behavioral improvement.
- Benchmark distinction: S3Gym compares multiple strategies under shared environments and budgets while recording both model judgments and executable outcomes.This separates failures caused by inaccurate evaluation from failures to convert correct feedback into transferable behavior.
4 Method
S3Gym evaluates self-improvement as a cycle in which agents explore permissive environments, self-judge experience, consolidate it through one of three pathways, and transfer resulting behavior to stricter held-out configurations.
- Cycle design: Each self-improvement cycle explores permissive configurations, judges the resulting trajectories, consolidates experience, updates the pathway-specific agent, and evaluates it on stricter held-out configurations.Exploration and evaluation use disjoint random seeds, and evaluation trajectories are excluded from subsequent history, memory, and training data.
- Interaction protocol: The interaction protocol records observations, histories, actions, self-judged immediate scores, environment feedback, verifier rewards, and final episode scores.Verifier rewards and final scores remain benchmark-side signals during exploration, while the agent receives observable feedback for subsequent interaction.
- Self-Judging: Self-Judging supplies internal signals for retaining high-scoring strategies and identifying low-scoring mistakes when precise environment rewards are unavailable to the agent.The benchmark compares self-judged scores with verifier-computed rewards without exposing ground-truth rewards during exploration.
- Self-Improving Pathways: Summary Memory compresses judged trajectories into reusable strategies, mistakes, and future directions, creating an information bottleneck that depends on judgment and summarization quality.It tests whether lengthy interaction histories can be abstracted into transferable strategies.
- Self-Improving Pathways: History ICL appends score-annotated trajectories to later contexts, preserving detailed evidence but making improvement subject to context-length constraints.All pathways share the same base model, seeds, interaction budget, and decoding settings.
- Self-Improving Pathways: Parameter Training converts judged trajectories into supervised fine-tuning examples, retaining high-scoring actions and filtering or correcting low-scoring actions before parameter updates.Training is treated as an auxiliary pathway because it introduces optimization variables absent from context-level improvement.
5 Experiments
S3Gym evaluates self-improvement across seven text-based games by separating permissive exploration from stricter evaluation and comparing context-level experience pathways. Results show that improvement depends jointly on the model, pathway, and environment, with summary memory selectively helping or hurting relative to raw history.
- 5.1 Benchmark Games: S3Gym covers seven games with permissive exploration configurations and stricter evaluation configurations, enabling experience-driven improvement to be assessed under held-out conditions.The games span complementary interactive reasoning forms, while evaluation may impose stricter termination conditions, larger search spaces, or denser dynamics.
- 5.2 Experimental Setup: History ICL appends interaction histories, whereas Summary Memory compresses trajectories into rules, mistakes, and future directions.History preserves detailed evidence but requires more context; summaries impose an information bottleneck and depend more on judging and summarization quality.
- 5.2 Experimental Setup: Avg., Max., and AUC+ measure overall performance, best observed capability, and sustained improvement above the initial score, respectively.Because score scales differ across games, raw AUC+ values should be compared within the same game rather than directly across games.
- 5.3 Main Results: Self-improvement varies jointly with the base model, experience pathway, and environment rather than being determined by base capability alone.Under History ICL, Gemini-3.5-Flash leads Chess and has the largest AUC+ on Snake and Trust Evolution, while GPT-5.5 leads several brittle tasks; under Summary Memory, GPT-5.5 leads Chess, Nullify, and Tetris.
- 5.3 Main Results: Gemini-2.5-Flash increases Minesweeper AUC+ from 0.000 to 7.794 and PvZ AUC+ from 24.402 to 238.501 with Summary Memory, while GPT-5.5’s Chess AUC+ rises from 0.474 to 16.840.These gains contrast with declines for other model–game pairs, showing that summary memory is not uniformly superior to raw history.
- 5.3 Main Results: Chess, Minesweeper, and Nullify have sparse or discrete scores, whereas Tetris, Snake, PvZ, and Trust Evolution provide denser feedback for assessing sustained adaptation.The complementary metrics distinguish occasional breakthroughs, overall performance, and persistent improvement; cross-game consistency should use within-game ranks or games improved rather than raw AUC+ averages.
6 Analysis
Parameter training and context-level memory both support self-improvement only under specific task conditions. Training can internalize useful strategies, but self-judging reliability and the match between experience representation and task structure remain decisive.
- Parameter Training: Training raises Trust Evolution from 0 to a maximum of 30, with scores above baseline at 18 of 19 updated checkpoints.Post-training average reaches 8.684 and AUC+ reaches 163.5.
- Parameter Training: Training is inconsistent across tasks: it improves Trust Evolution, weakly improves Chess, intermittently improves Snake, and severely degrades Plants-vs-Zombies.Minesweeper, Nullify, and Tetris show no measurable improvement.
- Self-Judging: Accurate local self-judging does not reliably produce improvement at the next evaluation checkpoint.Coupling correlations are near zero: ρ(A, g) = −0.010 and ρ(−E, g) = −0.018; event agreement correlates weakly negatively with NABA under Pearson and Spearman measures.
- Context-Level Pathways: Summary Memory improves average NABA on Nullify, Tetris, and Trust, where experience can be compressed into compact, reusable strategic abstractions.Examples include exact cancellation, stable board surfaces, and adaptation to inferred opponent strategies.
- Context-Level Pathways: Direct history performs better on average in Minesweeper, Plants-vs-Zombies, and Snake because success depends on local configurations, timing, hazards, and partially observed risk.Summaries can preserve high-level objectives while discarding state-contingent execution details.
- Summary Memory: Summary-based improvement succeeds when scores identify reusable failure modes and summaries convert them into precise executable rules.Concrete arithmetic and opponent-conditioned rules improve performance, whereas coarse advice or generic heuristics can produce negative ΔNABA.
7 Conclusion
S3GYM makes self-improvement measurable by testing whether agents can transform interaction history into better future behavior. Its evidence is mixed: summary memory helps when compact rules transfer, direct history helps when state details matter, and parameter training remains unstable.
- Conclusion: S3GYM separates relaxed exploration from stricter evaluation and supports both context-level and training-level reuse of trajectories.The benchmark evaluates whether interaction history translates into future performance.
- Conclusion: Summary memory helps when experience admits compact transferable rules, while direct history is often better for reactive control tasks requiring detailed state information.Training on self-generated trajectories remains unstable in the current implementation.
- Conclusion: Future self-improving agents need better judgment calibration, memory selection, and trajectory filtering, not merely more interaction data.
8 Contributions
The supplied contribution passages identify the paper’s contributors and corresponding authors.
- Contributors: Core contributors are Jiajun Shi, Siyuan Tao, Yuhao Wu, Zexuan Wang, and Jingyuan Zhang.
- Contributors: Additional contributors include Jiaheng Liu, Xinping Lei, Xinrong Zhang, Siyuan Fang, Zhewen Tan, Tianle Cai, Junhao Fang, Jiameng Huang, Yueyang Wang, Jinkai Liu, and Yuxuan Zhang.
- Contributors: Corresponding authors are Jian Yang, Zhoujun Li, Shen Yan, Wenhao Huang, and Ge Zhang.
A Game Prompt Templates
The appendix specifies canonical game prompts, shared interaction contracts, and distinct representations for raw-history and summary memory. It also documents game-specific state, action, and scoring interfaces.
- A Game Prompt Templates: The canonical templates follow the implementation and define each game’s rules, action space, coordinate conventions, and answer syntax.Six games share a base instruction; Trust Evolution uses a game-specific system instruction.
- A.1 Shared Interaction Protocol: Direct memory preserves raw previous-episode steps as environment-state and agent-action pairs, while truncating the previous-history tail when context is exceeded.Current-episode history and the current game prompt remain available in full.
- A.1 Shared Interaction Protocol: Summary memory compresses truncated prior history into cached strategy tips, while retaining the current episode as raw state–action–score history.Only cross-episode experience is compressed.
- A.1 Shared Interaction Protocol: Every memory condition requires the model to output both an action and a self-judged immediate score, or END with score 0 after termination.The self-judged score is intended to represent reward under the game rules rather than free-form confidence.
- A.2 State Inference and Deduction: The prompt suite covers piece-position prediction, Minesweeper deduction, symbolic unit elimination, and Tetris placement.These interfaces specify board representations, legal actions, and task-specific objectives.
- A.4 Reactive Control and Strategic Interaction: The remaining prompts define Snake control, Plants-vs-Zombies placement, and repeated interaction against an opponent.They expose dynamic grids or interaction histories and require actions in the corresponding game syntax.