Source-linked AI summary
LoopArena: Benchmarking Models as Runtime Controllers for Loop Engineering
Yi Wang, Haopeng Zhang, Chengxiang Huang, Rui Dai, Kaikui Liu, Piotr Koniusz, Xiangxiang Chu
TL;DR
LoopArena addresses the lack of direct evidence about whether a model can control a coding-agent loop separately from the Worker’s coding ability. It benchmarks a Controller guiding a fixed Worker across three execution scopes, finding limited full-task success while enabling substantially cheaper slice-based comparison.
Problem
End-to-end coding-agent outcomes do not isolate loop guidance from the Worker’s ability to carry out long-running tasks.
Method
LoopArena compares Controller models while holding the Worker and execution setup fixed, using Type I, Type II, and Type III evaluations of decisions, task slices, and full tasks.
Results
24.69% is the highest observed Type III Strict Success Rate, while paired Type II evaluation reduces estimated inference cost by 64.4% on average and yields ρII,III = 0.9747 under Core.
Takeaways & Limitations
Full-task loop control remains difficult, and useful control must adapt to the evolving run rather than merely restate a persistent goal.
Takeaways & Limitations
LoopArena currently covers repository-level coding tasks and a structured Controller–Worker organization, leaving other domains, Worker families, and multi-Worker settings for extension.
Abstract
from arXiv · showhide
Loop Engineering is emerging as a practice for organizing development work around coding agents. Instead of writing each prompt by hand, practitioners design loops that monitor progress, assign work, run checks, and decide what the agent should do next. Even with a capable coding agent, a loop may trust a stale progress note, skip needed verification, spend its budget in the wrong direction, or stop before the task is safe to submit. Yet the final outcome of one end-to-end run cannot tell whether success or failure reflects the loop's guidance or the coding agent's ability to carry out the task. We introduce LoopArena, a benchmark for evaluating how well one model can guide a separate coding agent through a long-running task. The model under evaluation is the \textbf{Controller}: after each coding round, it receives a structured summary of the run and instructs a separate, fixed coding agent, the \textbf{Worker}, on what to do or verify next, or decides whether to stop. LoopArena evaluates this ability in three complementary settings that differ in execution scope and cost. Type I scores next-step Loop Contract selection through execution-validated questions without running the Worker at evaluation time. Type II executes repeated control over a selected slice of a full task, while Type III evaluates the paired full task from its original state. On full tasks, the best observed Strict Success Rate is \textbf{24.69\%}, leaving substantial room for improvement in long-horizon loop control. Across Controllers, the paired reduction in estimated inference cost averages \textbf{64.4\%}, and Type II produces a similar ordering under the main Core criterion (Spearman's \(ρ=\textbf{0.9747}\)). We release the benchmark data and evaluation code at https://github.com/AMAP-ML/LoopArena .
1 Introduction
LoopArena treats runtime loop control as a distinct capability: a Controller guides a fixed Worker through long-horizon coding tasks while the benchmark separates Controller quality from Worker execution. It compares Controllers with controlled execution and evaluates decisions at three complementary scopes.
- Loop Engineering replaces manually written successive prompts with loops that monitor progress, judge evidence, and manage interaction rounds.
- The Controller reviews structured run summaries after each Worker round and can redirect implementation, request verification, or stop; baselines include no control and fixed goal restatement.
- LoopArena evaluates whether a Controller can decide what a separate coding agent should do next during long-horizon repository-level tasks.
- The benchmark holds the Worker, tools, environment, budget, evaluator, and control interface fixed, so comparisons target the Controller model.
- Type I scores individual Loop Contract decisions without evaluation-time Worker execution, while Type II and Type III assess paired task slices and complete tasks.
2 The LoopArena Benchmark
LoopArena organizes Controller evaluation around control cycles in which a read-only Evidence Packet informs a Loop Contract for the persistent Worker. Its three settings vary how much of the task is executed, from one decision to a complete run.
- Type I: Contract selection: Type I evaluates one control decision through a four-way Loop Contract choice, with execution completed during benchmark construction.New Controllers require no Worker execution for Type I evaluation.
- Type II: Condensed coding task: Type II starts from a prepared intermediate workspace and evaluates one coherent slice of a full task, reducing execution required per evaluation.
- Type III: Full coding task: Type III evaluates the corresponding complete task from its original state, covering investigation, implementation, verification, and stopping.
- Control cycle: The Worker alone uses coding tools within assigned segments, while the persistent conversation remains across control cycles.
- Control cycle: A temporary Reporter summarizes task context, completed work, verification evidence, and remaining issues into an Evidence Packet without modifying the workspace.
- Control cycle: The Controller receives the latest Packet and prior history without workspace access, then issues a Contract to advance work, request focused verification, or stop.
3 Benchmark Construction
LoopArena constructs paired full-task and task-slice evaluations from SCBench and BeyondSWE, alongside execution-validated Type I control questions. Construction fixes states and candidates, then validates outcomes and input isolation.
- Type II and Type III use full coding tasks from SCBench and BeyondSWE, while Type I uses restorable control points from Controller-guided trajectories.
- Type II and Type III share 27 source tasks, while Type I requires no Worker execution during evaluation.
- The source benchmarks provide complementary structures: SCBench targets long-horizon iterative coding, while BeyondSWE covers broader software-engineering tasks.
- Each Type II case selects a stage evaluable from a prepared workspace whose starting state fails an introduced requirement and completed state passes expected requirements.
- Type I freezes four candidate Loop Contracts before replay outcomes are observed and accepts an option only when it is the unique winner under two matched replay schedules.
- Automated checks, LLM-assisted review, and expert review verify consistency, stable Type I answers, candidate plausibility, and protection against leakage of later trajectory information.
4 Evaluation Metrics
LoopArena measures Controller performance at isolated decision and executable task levels, using success, cost, and ranking metrics under fixed evaluation protocols.
- Evaluation settings: Type I measures single Contract selection from a frozen state, while Types II and III measure repeated control over paired task slices and full tasks.The executable settings also compare inference cost and Controller orderings across execution scopes.
- Contract metric: Type I reports Contract Accuracy as 100 × AccI(π), and missing, multiple, out-of-range, or unparseable responses count as incorrect.Each Controller answers every question once, and invalid responses are additionally reported in Invalid Rate.
- Evaluation protocol: Each Type II or Type III policy is evaluated on the same task setup in three separate runs, with no-control and fixed-control references reused across Controller comparisons.The policy set includes evaluated Controllers plus no-control and fixed-control references.
- Success metric: Strict Success Rate counts a run only when it passes the setting-specific evaluator and follows the control protocol.SCBench requires every Core check in the frozen scored set to pass, while BeyondSWE requires an official Harbor reward of one.
- Resource metric: Estimated inference cost sums call-level costs using recorded token counts and a frozen standard-price schedule without prompt caching.Controller-guided totals include Worker, Reporter, and Controller calls; reference totals include Worker calls.
5 Experiments
Experiments compare fixed-setup Controllers across three settings: contract selection, paired task-slice control, and paired full-task control. Full-task success remains low, while Type II substantially reduces cost and preserves Controller ordering under the Core criterion.
- Experimental design: The same Controller models are evaluated in all three settings, with Qwen3.7-Plus serving as the shared Worker for Type II and Type III.Reporter configurations and no-control and fixed-control references are also shared across executable settings.
- Experimental design: The main analysis compares full-task Controllers, tests whether Type II retains that comparison more cheaply, and examines what Type I reveals about individual decisions.Model revisions, provider settings, and the run plan are fixed before scoring.
- Main results: 24.69% is the strongest Type III Strict Success Rate, with evaluated Controllers ranging from 16.05% to 24.69%.These results indicate that full-task control remains difficult under the benchmark’s success criteria.
- Main results: 64.4% is the average paired reduction in estimated inference cost for Type II relative to Type III, while the Core Controller ordering has Spearman’s ρII,III = 0.9747.Type II keeps the executable Controller–Worker loop and evaluator but uses a shorter execution scope.
- Main results: Type I Contract Accuracy ranges from 72.22% to 87.78%, with a 0% Invalid Rate; the strongest deterministic shortcut reaches 31.11%.Evaluating another Controller requires one response per question and no Worker execution after benchmark construction.
6 Related Work
LoopArena differs from benchmarks that score complete coding-agent systems or isolated process behaviors by evaluating a Controller’s repeated instructions to a separate coding Worker and their execution-grounded effects.
- Coding-agent benchmarks: Most coding-agent benchmarks score the complete coding-agent system by evaluating the final repository state, whereas LoopArena fixes the Worker and execution setup to compare Controllers.This changes the benchmark’s primary object of evaluation from the complete system to the model directing the Worker.
- Harnesses and Loop Engineering: Harness benchmarks compare complete model–harness configurations, while LoopArena evaluates the Controller within a fixed Controller–Worker organization.Related harness work varies tools, context policy, state representation, or control logic around models.
- Process and feedback evaluation: Process and feedback benchmarks score interactive tool use, individual trajectory steps, or execution-grounded supervision, while LoopArena links repeated control decisions to downstream effects under execution.The benchmark evaluates instructions from one model to a separate coding agent.
- Efficient benchmark evaluation: Efficient benchmark evaluation uses cheaper settings and ranking correlations to test whether reduced-cost evaluations recover full-benchmark comparisons; LoopArena applies this principle to paired task-slice and full-task control.Its executable settings compare Controller orderings across different execution scopes.
7 Limitations
LoopArena currently covers repository-level coding tasks and a structured Controller–Worker organization, leaving broader domains and loop configurations for future extension.
- Scope: LoopArena’s current scope is repository-level coding tasks with structured handoffs between one Controller and one Worker.Broader coverage would require additional software domains, Worker families, multi-Worker settings, and loop organizations.
- Scope: Applying the three-setting design beyond coding requires domain-specific task construction and executable evaluators.This is an explicit boundary on extending the benchmark beyond its current domain.
8 Conclusion
LoopArena measures long-running loop control across decision, slice, and full-task settings while holding the Worker fixed. Full-task success remains limited, while lower-cost Type II evaluation closely matches Type III Controller ordering.
- 24.69% is the highest observed Type III Strict Success Rate on full tasks.
- 64.4% is the average paired Type II cost reduction across Controllers.
- ρII,III = 0.9747 indicates similar Core-based Controller ordering between Type II and Type III.
- Type I uses execution-validated next-contract questions, whereas Type II and Type III evaluate selected slices and corresponding full tasks.The shortened Type I example concerns choosing the next instruction from four candidates.
- LoopArena contains 90 Type I questions and 27 paired Type II–Type III task instances.The Type III panel has 27 unique official tasks: 11 from SCBench and 16 from BeyondSWE.
B Benchmark Construction and Quality Control
Type I benchmark items are constructed by freezing plausible contract candidates before matched replays and retaining only stable, uniquely resolved winners. Automated checks and post-freeze audits then verify integrity without changing replay-defined answers.
- Four plausible Loop Contracts are frozen before downstream execution, with one recorded candidate and three alternatives.Candidates are schema-compliant and presented in neutral order.
- Two matched replay schedules execute all four candidates from the same restored state while varying only the initial Contract.The Worker, budget, evaluator, continuation policy, stopping rules, and random seed are matched within each schedule.
- Terminal task success is the primary outcome, with successful candidates further ranked by downstream Controller cycles and then Worker turns.Token counts and repository calls are diagnostic only and do not determine the winner.
- Items are retained only when both schedules identify the same unique winner.Items are rejected when all candidates fail, no unique winner exists, or the schedules disagree.
- Automated checks and post-freeze LLM-assisted or expert audits verify protocol, provenance, clarity, plausibility, leakage, and consistency.Review cannot override the confirmed replay winner.
C Harness and Evaluation Details
The harness separates reporting, control, and coding: the Reporter summarizes Worker progress, the Controller issues a structured contract, and only the Worker modifies the repository. Repeated execution continues until stopping or a terminal condition.
- Only the Worker edits the repository, runs commands, and uses coding tools.The Reporter inspects a static read-only workspace, while the Controller has no coding tools.
- The Evidence Packet combines task context, Worker history, verification evidence, open issues, cited Worker turns, and the remaining turn budget.Reporter interaction is kept outside the persistent Worker conversation.
- The Controller returns advance, verify, or stop; continuing decisions include bounded Worker instructions, invariants, and acceptance conditions.The harness validates the decision as the canonical Loop Contract.
- The harness renders a continuing Contract as the next Worker instruction without another model call.A Stop Contract sends the current workspace to the evaluator.
- Each Worker episode ends after 600 ReAct turns or 7,200 seconds, while Controller-guided runs allow at most 128 control cycles and 86,400 cumulative seconds.Reaching a Worker limit counts as task failure.
- Runs share Worker configuration and evaluators across reference policies and Controllers, and invalid responses or protocol violations count as failure.
D Experimental Details
Experiments compare a fixed panel of Controllers under standardized interfaces and paired evaluation plans. Outcomes include contract accuracy, strict task success, estimated model-inference cost, and Controller-order correlation across settings.
- Controller panel and inference interfaces: Five Controllers are evaluated: Qwen3.7-Plus, DeepSeek-V4-Flash-0731, GLM 5.2, GPT-5.5, and Claude Opus 4.8.Exact model identifiers and inference settings are preserved in the run manifest.
- Metrics: Contract Accuracy is the fraction of Type I questions where the selected candidate matches the construction-fixed correct option.Missing, multiple, out-of-range, and unparseable answers are incorrect and contribute to Invalid Rate.
- Metrics: Strict Success Rate averages binary evaluator outcomes over all tasks and K = 3 runs.No-control and fixed-control runs are shared across Controller comparisons.
- Resource accounting: Estimated run cost sums standard-priced input and output token usage for every model call, assuming no prompt caching.Model-control totals include Worker, Reporter, and Controller calls; infrastructure costs are excluded.
- Resource accounting: Type II cost reduction is reported as the paired percentage reduction for each Controller, alongside total tokens, Worker turns, and control cycles.
- Run plan: Spearman’s ρII,III compares Controller rankings across Type II and Type III while excluding reference policies.Tied scores receive average ranks, and strictly opposite pairwise orderings are counted as rank reversals.
F.1 Type I results
Type I provides low-cost diagnosis of individual Controller decisions, while diagnostic analyses characterize shortcut performance, cost accounting, and termination behavior across executable settings.
- Type I results: Type I reports Contract Accuracy and estimated evaluation cost for each Controller, with invalid responses counted as incorrect.The standardized marginal cost excludes one-time candidate execution during benchmark construction.
- Evaluation cost: Type I reuses candidate executions from benchmark construction, so evaluating another Controller requires one response per question and no Worker run.The benchmark construction records and frozen price schedule support standardized cost estimation.
- Diagnostic analyses: 162 of 1,614 Controller calls reached the fixed output limit, and 77 of 167 evaluations contained at least one such call.These bounded-protocol events contributed to protocol failures in the reported evaluations.
G Extended Comparison with Related Evaluation Settings
LoopArena differs from nearby evaluation paradigms by evaluating the model that guides a fixed coding Worker, and it releases data, harnesses, records, and analysis code for all settings.
- Evaluated object: LoopArena holds the coding Worker fixed and scores the model that guides it, unlike benchmarks that score the complete coding stack.Its three settings vary execution amount while targeting Controller loop-control ability.
- Released benchmark data: The release includes Type I questions, Type II workspaces and task slices, and 27 unique Type III task specifications.Construction records preserve selected trajectories, candidate replays, and the frozen Type II–Type III pairing.
- Released infrastructure: The released harness covers Worker, Reporter, and Controller runners, Evidence Packet formatting, Contract validation, evaluator adapters, and result aggregation.Run configurations and evaluation records include prompts, model revisions, seeds, transcripts, usage, costs, and evaluator receipts.
- Analysis and prompts: Analysis code implements Contract Accuracy, Strict Success Rate, inference-cost estimation, cost reduction, rank correlation, shortcut analyses, and table generation.Prompt templates and benchmark instances define the task, current state, reported evidence, and candidate decisions.
- Worker protocol: Controlled work limits the Worker to the latest assignment, which must serve the overall goal without adding or removing requirements.Repository observations and tool results take precedence over Controller-provided summaries for consequential details.
- Evidence and tool use: The Worker must verify consequential details with repository tools and may not claim a check passed without observing its result.The protocol also restricts tool use to tools attached to the current request and requires repository-contained changes.