Source-linked AI summary
OptiArena: Can LLMs Improve Executable Algorithms under Fixed Resource Budgets?
Wenjun Peng, Xinyu Wang
TL;DR
Static benchmarks capture only part of LLMs’ role in iterative coding, leaving the question of bounded algorithm improvement under fixed budgets. OptiArena tests five-edit policy revision in games; models recover weak-start capability more consistently than they refine competent baselines, with substantial variation across models and games.
Problem
Static QA and code-generation benchmarks capture only part of LLMs’ roles as coding agents and research tools, while evaluator-guided discovery results can depend on substantial, uncontrolled budgets.
Source: §1 · p. 1Method
OptiArena has LLMs edit executable game strategies over five rounds in a fixed minimal scaffold, with bounded feedback, sealed references, fixed resources, and separate weak-start and competent-baseline regimes.
Source: §1 · p. 1, §3 · p. 3, §3.2 · p. 4Results
Weak-starter recovery is substantial but uneven, while competent-baseline refinement is less reliable and can be destructive across models, games, and reference signals.
Source: §6 · p. 9Takeaways & Limitations
OptiArena provides a reproducible, budget-explicit foundation for studying LLMs as research tools in bounded algorithmic workflows.
Source: §6 · p. 9The main matrices use one prompt condition, seed, and five-edit budget with designated starters, so they do not establish stability across prompts, seeds, longer searches, or alternative starter strengths.
Source: Limitations · p. 9
Abstract
from arXiv · showhide
Static QA and code-generation benchmarks only partially capture the role that large language models (LLMs) now play as coding agents and research tools. We introduce OptiArena, a budget-controlled testbed for studying whether LLMs can improve executable game-playing algorithms through five rounds of code edits within a fixed minimal scaffold and under bounded evaluator feedback and fixed resource budgets. The testbed uses two optimization regimes, surface obfuscation controls, calibrated references, held-out/stress splits, and diagnostics for degradation and exceptional failures, with LLM API cost reported separately from local evaluator wall-clock. The empirical study asks three questions: whether models can close the calibrated gap between a designated weak starter and an editable competent baseline, whether they can refine editable competent baselines without damaging them, and whether gains survive surface obfuscation controls. Across twelve frontier LLMs and five games, models improve designated weak starters more consistently than they refine editable competent baselines, with substantial variation across games and models. OptiArena provides a practical testbed for measuring bounded-resource algorithm optimization within the five-edit, fixed-scaffold setting studied here. Code is available at https://github.com/WJ-Peng/OptiArena.
1 Introduction
Static benchmarks capture only part of LLMs’ emerging role in iterative coding and research workflows, motivating a test of bounded algorithm improvement under fixed budgets. OptiArena uses five-edit game-policy optimization to distinguish recovery from weak starters from refinement of competent baselines, while testing surface familiarity and evaluator overfitting.
Motivation
- Static-input benchmarks capture only a narrow slice of LLMs’ growing role as coding agents that revise executable artifacts using feedback.The paper frames bounded improvement of existing algorithms as a distinct question from answering static prompts or generating code once.Source: §1 · p. 1
Testbed
- OptiArena tests whether LLMs can improve executable strategy code through five edits in a fixed scaffold with bounded evaluator feedback and resources.Games provide precise rules, replayable execution, and repeated measurement against hidden references.Source: §1 · pp. 1–2
Evaluation design
- The two regimes separate recovering useful behavior from designated weak starters from making local improvements to editable competent baselines.These comparisons use registered anchors and remaining headroom, rather than claiming global worst-case or optimal performance.Source: §1 · pp. 1–2
Robustness and scope
- The study tests whether gains persist after recognizable task cues are removed and uses independent capability signals, where available, to identify evaluator overfitting.It evaluates 12 frontier LLMs across five game families and reports exceptional failures and LLM API cost separately from local evaluator wall-clock.Source: §1 · p. 2
2 Related Work
Prior work extends evaluation from static code tasks toward agents and evaluator-guided optimization, while game-based work offers controlled execution settings. OptiArena focuses on bounded, feedback-driven revision of existing strategy code under fixed scaffolds and resource limits.
2.1 Code and Agent Evaluation
- Code evaluation has expanded from static tasks toward repository edits and agent interaction, while OptiArena measures algorithmic improvement and baseline degradation rather than code-generation quality, pass/fail correctness, or raw runtime speed.The paper places this outcome alongside earlier executable and agent evaluation approaches.Source: §2.1 · p. 2
2.2 LLM-Guided Optimization and Discovery
- Unlike discovery systems with broader search loops, OptiArena fixes a minimal scaffold, feedback channel, and resource limits to study revision of executable algorithms.Its focus is the model’s ability to revise algorithms under bounded feedback.Source: §2.2 · pp. 2–3
2.3 Game-Based Evaluation
- Games provide precise rules and replayable trials; OptiArena uses them to evaluate persistent policy-code revision, sealed references, and surface obfuscation controls.The task surface is obfuscated without changing game dynamics to test sensitivity to familiar cues.Source: §2.3 · p. 3
3 OptiArena
OptiArena runs bounded edits to executable game policies under sealed evaluation and registered resource limits. Its protocol separates weak-start recovery from competent-baseline refinement and measures held-out performance, surface sensitivity, and diagnostic outcomes.
3 OptiArena
- Each run binds a model to a predeclared condition and provides five edits to revise a runnable policy before candidate selection and sealed evaluation.The editable baseline is visible source code, while the hidden reference and final evaluation material remain unavailable to the model; exceeding the registered compute budget counts as failure.Source: §3 · p. 3
3.1 Task Suite and Policy Interface
- The five-game suite spans four two-player zero-sum games and stochastic single-player 2048, with the policy module as the model’s editable object.The harness handles execution and sandbox bookkeeping; the prompt-exposed strategy code is the only object the model can change.Source: §3.1 · pp. 3–4, Table 1 · p. 4
3.2 Optimization Regimes and Feedback Loop
- Weak-Start Optimization edits a designated weak policy to recover calibrated capability, whereas Strong-Baseline Refinement edits a competent policy for local gains while preserving resource discipline.“Strong” means competent relative to the designated weak starter, not globally optimal.Source: §3.2 · p. 4
3.3 Sealed Splits and Surface Controls
- Held-out evaluation uses disjoint openings or seed pools, stress evaluation shifts to a harder registered distribution, and WSO surface controls obfuscate prompts while preserving game dynamics.Candidates face the same registered resource policy across these evaluations.Source: §3.3 · p. 4
3.4 Metrics
- WSO reports per-game capability against registered references, using gap-closing ratio when well conditioned and interpreting values above 1 as surpassing the calibrated competent baseline.The per-game estimator is registered according to the strongest available reference signal; cross-game aggregates use normalized capability metrics.Source: §3.4 · p. 5
- SBR measures change from the editable competent baseline under matched evaluation material, while contamination sensitivity compares vanilla and obfuscated capability.Positive SBR values indicate improvement over the competent baseline; positive contamination sensitivity indicates an advantage for the familiar vanilla surface.Source: §3.4 · p. 5
- Validity, degradation, and cost are diagnostics: they track completed trials, cells scoring below the starter, and LLM optimization-loop cost separately from local evaluation time.Candidate-caused failures remain benchmark outcomes, and aggregate comparisons resample matched model-game cells rather than treating missing rows as neutral.Source: §3.4 · pp. 4–5
4 Experiments
The experiments test weak-start recovery, refinement of competent baselines, and robustness to surface obfuscation under a shared, bounded code-editing and evaluation protocol. Weak-start results vary by model and game, while competent-baseline refinement is more difficult and contamination sensitivity is heterogeneous.
4.1 Experimental Setup
- Each model–game–regime run uses at most five edit calls and fixed evaluation budgets, with development, held-out, and stress splits used for selected candidates.Final splits use n = 50 games or trials; API cost and token use are archived separately from local evaluator time.Source: §4.1 · p. 6
4.2 Optimization Results: Weak Starts and Competent Baselines
- Weak-start capability varies by model and game: mean Cap ranges from 0.31 to 0.67, with Connect 4 and 2048 highest across models and Hex agreement with MoHex low.Gemini 3.5 Flash leads at 0.67; mean Cap is 0.82 on Connect 4 and 0.80 on 2048, while mean Hex agreement is 0.04.Source: §4.2 · p. 6
- Refinement is harder and more game-dependent than weak-start repair: seven models have non-positive mean two-player change, while seven improve on 2048, four degrade, and one is unchanged.Breakthrough supplies most positive refinements; ten of twelve models degrade on Connect 4, and all are neutral on saturated Othello and Hex win-rate surfaces.Source: §4.2 · p. 6
- Selected SBR candidates complete all held-out trials in 58 of 60 cells, so negative and neutral refinement outcomes are retained rather than treated as missing failures.The table reports ∆SBR relative to the empirical bounded-resource competent baseline, with 2048 reported separately.Source: §4.2 · p. 7, Table 4 · p. 8
4.3 Contamination Sensitivity
- Across the matched WSO control, mean CSI-Cap is −0.043 (95% CI [−0.124, 0.043]), with game-level signs varying rather than a uniform familiar-surface advantage.The scale-normalized aggregate is 0.016 (95% CI [−0.105, 0.144]); the table summarizes 60 matched model–game pairs.Source: §4.3 · p. 7, Table 5 · p. 8
4.4 Diagnostic and Qualitative Analysis
- The selected edits include both algorithm insertions and destructive rewrites, while Connect 4 win-rate changes can diverge from solved-oracle move optimality.The extension to 10 or 15 edits shows no consistent improvement trend in WSO Cap or SBR improvement over the five-edit budget.Source: §4.4 · pp. 7–8
5 Discussion
OptiArena targets the middle ground between static code generation and open-ended discovery: bounded-budget improvement of executable algorithms using feedback. Its game-based results are diagnostic and bounded-resource claims, not claims of state-of-the-art play or autonomous discovery.
5 Discussion
- OptiArena evaluates whether models can improve runnable algorithms through feedback, rather than only generate code that passes tests or produce open-ended discoveries.Games supply exact rules, repeated execution, strong references, and interpretable failure modes for this diagnostic setting.Source: §5 · pp. 8–9
- Degradation is treated as an outcome: plausible edits can remove invariants, time out through expanded search, or replace a strong heuristic with a weaker familiar template.The benchmark distinguishes controlled local edits from recognizable but weaker rewrites.Source: §5 · p. 9
- The aggregate vanilla–obfuscated WSO comparison is small and neither aggregate interval excludes zero, while game-level variation cautions against a single model-wide robustness conclusion.The matched comparison is interpreted on the registered Cap scale and is not extended to SBR.Source: §5 · p. 9
- The empirical claims concern bounded-resource recovery from designated weak starters, local refinement over editable competent baselines, and sensitivity to surface obfuscation.Games are a compact diagnostic substrate; the policy interface and archive could be adapted to non-game executable tasks.Source: §5 · p. 9
6 Conclusion
OPTIARENA reframes LLM evaluation as bounded-resource optimization of executable strategy code, with empirical results showing more consistent gains from weak starters than from competent-baseline refinement.
6 Conclusion
- OPTIARENA evaluates whether models can edit executable strategy code over feedback rounds under sealed references, fixed budgets, and reproducibility gates.It provides a reproducible, budget-explicit foundation for studying LLMs in algorithmic workflows.Source: §6 · p. 9
- Models recover substantially but unevenly from designated weak starters, while competent-baseline refinement is less reliable and can be destructive.The contrast varies across models, games, and reference signals.Source: §6 · p. 9
- The differing outcomes across optimization regimes indicate that bounded-feedback optimization is a distinct capability, not a consequence of executable code alone.Source: §6 · p. 9
Limitations
The conclusions are limited to relative improvement under registered budgets and sealed references in a compact set of game environments.
Limitations
- OPTIARENA’s conclusions concern bounded, feedback-driven code optimization in a compact set of game environments, not all programming or open-ended scientific discovery.Source: Limitations · p. 9
- Generality depends on the selected games, references, splits, and model versions, despite controls including held-out and stress evaluation and contamination controls.Source: Limitations · p. 9
- The main matrices use one prompt condition, seed, five-edit budget, and designated starters; they do not establish stability across alternatives.The obfuscation control covers WSO surface familiarity rather than semantic transfer, and some two-player records allow only aggregate inference.Source: Limitations · p. 9
Ethical Considerations
The paper describes safeguards and release boundaries for benchmark artifacts, then organizes appendices to support auditability.
Ethical Considerations
- Generated code runs only in a sandbox with disabled network access, fixed resource limits, static validation, and sealed evaluators.The stated risk is overclaiming generated-code reliability or LLM research-system autonomy; candidate programs are benchmark artifacts, not trusted software.Source: Ethical Considerations · p. 9
- Artifact release must respect external engine and solver licenses, including components that cannot be redistributed directly.Source: Ethical Considerations · p. 9
- The authors conceived, implemented, and executed the experimental framework, benchmark design, and reported experiments; AI assistants helped polish prose and parts of the implementation code.Source: Ethical Considerations · p. 9
Appendix
- The appendices document the task registry, optimization loop, metrics, experimental reporting, artifact release, case selection, and dependency provenance to make the benchmark auditable.Source: Appendix · p. 12
A Task Registry and Environment Details
The appendix specifies game anchors and calibration, an intentionally restricted policy interface, and an archived, feedback-driven optimization process with fixed prompt constraints.
A.1 Policy Contract
- Each candidate policy receives state, legal actions, player identifier, and time budget, and must return one legal action under restrictions on imports and external access.Source: §A.1 · p. 13
A.2 Native Implementations and Conformance
- Where OpenSpiel lacks a needed game or parameterization, native implementations are admitted only after conformance checks against a reference implementation.Checks cover legal actions, transitions, terminal detection, and scoring where applicable; evaluation uses disjoint development, held-out, and stress pools.Source: §A.2 · p. 13
B Optimization Protocol and Prompting
- Across the edit loop, the harness initializes and archives a starter, selects a parent, prompts the LLM, validates and evaluates replacements, then selects a final candidate for held-out and stress testing.Archive records include source files, parent relations, evaluation payloads, token cost, latency, and diff summaries for audits and degradation analysis.Source: B Optimization Protocol and Prompting · p. 13
B.1 Feedback Schema
- Evaluator feedback summarizes validity, performance, efficiency, and optional traces, while withholding sealed-reference source, hidden openings, solver values, and final-evaluation material.Source: §B.1 · p. 13
B.2 Prompt Templates
- Prompt conditions share a fixed contract and user-message sequence, with only requested reasoning style varying; the template requires one complete policy source file.The schematic user fields include task information, parent source, compact feedback, recent history, and condition-specific instructions.Source: §B.2 · p. 13
A Task Registry and Environment Details
- Editable starters are separated from hidden strength references; Table 7 reports registered weak-starter and competent-baseline scores against the calibration opponent, using win rates for Breakthrough, Othello, Hex, and Connect 4 and robust score for 2048.For stochastic 2048, the naive score is the mean of twelve models’ own iteration-0 evaluations. Near-zero Othello and Hex gaps make win-rate GCR uninformative, not evidence that the competent baseline lacks headroom under the sealed oracle.Source: A Task Registry and Environment Details · p. 12, Table 7 · p. 14
B.2.1 Shared System Instructions
- The shared system instructions require the policy interface, allowed libraries, no evaluator or external-tool access, and compliance with move-time and memory budgets.Source: §B.2.1 · p. 14
B.2.2 User Message Fields
- The user message’s fixed field order lets prompt-condition ablations vary requested reasoning style without changing task information.Source: §B.2.2 · p. 14
C.1 Metric Details for Saturated Games
The appendix specifies capability measures for games where win-rate comparisons may saturate, explains the contamination-sensitivity comparison, and describes result provenance and diagnostic outputs.
C.1 Metric Details for Saturated Games
- When win rate saturates against strong references, TIARENA registers a per-game estimator in advance: move-optimal rate for Connect 4, move agreement for Hex and Othello, reference-Δ for 2048, and calibrated-gap GCR for Breakthrough.Source: §C.1 · p. 14
- The Cap/Δ table separates each game's selected capability estimator from the sign of its change, while validity is reported only when below 1.00.This keeps saturated games in the benchmark while making the estimator explicit.Source: §C.1 · p. 15
C.2 Metric Normalization and Diagnostics
- Across 60 matched model–game cells, mean CSI-Cap is −0.043 (95% CI [−0.124, 0.043]) and normalized CSI is 0.016 (95% CI [−0.105, 0.144]).Obfuscation preserves game dynamics, so CSI measures sensitivity to surface familiarity rather than task difficulty; the control applies to WSO, not SBR.Source: §C.2 · p. 15
- Development-to-held-out and held-out-to-stress gaps diagnose feedback-split overfitting and stability under the registered harder distribution, respectively.Cost uses LLM usage metadata, separately from local evaluator wall-clock; degradation counts selected candidates worse than their starters.Source: §C.2 · p. 15
C.3 Result Tables and Provenance
- The main-text WSO and SBR matrices are populated from archived SQLite databases and analysis JSON, tying each value to recorded evaluation metadata.Source: §C.3 · p. 15
C.3.1 Diagnostic Surfaces
- Released diagnostic surfaces include per-model, per-game WSO CSI, validity and degradation summaries, and cost-efficiency and edit-type analyses.The main text reports the WSO and SBR matrices and the matched WSO contamination summary in Table 5.Source: §C.3.1 · p. 15
D Experimental Design and Reporting Rules
The experimental design defines three confirmatory comparisons, a bounded run budget, and metadata and inclusion rules for reproducible aggregate reporting.
D.1 Reporting Scope
- The confirmatory design comprises vanilla WSO, matched SBR, and a WSO comparison with obfuscated tasks; pilots, partial controls, and supplementary edit-budget extensions are excluded from the primary matrices.Source: §D.1 · p. 15
D.2 Model Roster
- The model roster groups twelve systems for cross-model comparisons, with a finer grouping documented in Table 9.Closed models provide proprietary comparison points; open-weight models test coverage across cost and scale profiles.Source: §D.2 · pp. 15–16, Table 9 · p. 16
D.3 Run Budget
- Each primary run allows five edit calls after the starter evaluation, then evaluates the selected candidate on development, held-out, and stress splits with n = 50 games or trials per split.Two-player policies receive one second per move; 2048 receives 0.1 seconds, and candidates run in a one-CPU, 2048 MB sandbox.Source: §D.3 · p. 16
D.4 Inclusion and Rerun Rules
- A model-generated invalid program, destructive edit, low score, or candidate-caused timeout remains a valid outcome, while infrastructure failures may be rerun.A game excluded for an infrastructure defect is excluded for all models in that comparison and reported explicitly.Source: §D.4 · p. 16
D.5 Reproducibility Checks
- Aggregate inclusion requires recorded matching evaluation metadata and disjoint, replayable development, held-out, and stress inputs; native rules implementations must pass conformance tests.Source: §D.5 · p. 16
- Uncertainty intervals use paired openings and side assignments for two-player games where available, or seed pools for stochastic tasks; cross-model summaries aggregate by model–game cell.Source: §D.5 · p. 16
E.1 Qualitative Audit Samples
The qualitative audit uses criteria-selected cases, preserves their replayable records, and documents the released artifacts and external dependencies.
E.1 Qualitative Audit Samples
- Audit cases cover successful weak-start recovery, destructive refinement, invalid code, and excessive-search timeout, with run and evaluation identifiers retained for replay.The sampled edits include new search procedures, a from-scratch baseline rewrite, and an in-place evaluation-function change.Source: §E.1 · pp. 16–17, Table 10 · p. 17
E.2 Archived Artifacts
- The archive includes run databases, candidate sources, configurations, calibration records, analysis outputs, and experiment definitions needed to replay reported comparisons.Source: §E.2 · p. 17
E.3 External Dependency Notes
- Vanilla rules use OpenSpiel where available, with named external references for several games; license constraints prevent bundling some GPL or AGPL engines.Metadata records versions, engine identifiers, budgets, and license constraints, alongside reproducibility records for reported comparisons.Source: §E.3 · p. 17