Source-linked AI summary
Zero-Shot Self-Orchestration with Ledger-Based Control for Improved LLM Coding Performance
Victor Gao, Vida Khosrowshahi, Ali Khosrowshahi, Xihao Sun, Juhyun Lee, Simon, Lee
TL;DR
Prior multi-agent LLM evaluations often mix coordination with changes in compute, tools, prompts, and retrieval, leaving the source of gains unclear. The paper compares a training-free manager–worker scaffold over a shared filesystem with the same model answering in one pass. Across LiveCodeBench experiments, the scaffold improves some models substantially but has null or negative effects for others, with context management and problem decomposition recurring as proposed mechanisms.
Problem
Prior multi-agent LLM comparisons are mixed and confounded because pipelines change token budgets, tool calls, prompts, and retrieval alongside coordination.
Method
The paper evaluates zero-shot self-orchestration by comparing the same model and benchmark in single-pass and manager–worker conditions using a shared filesystem workspace, without training or per-benchmark tuning.
Results
+23.4 points for Qwen3.8-27B, +10.6 for GPT-5.6-Luna, and +8.0 for GPT-5.6-Terra show substantial scaffold gains, while Qwen3.6-35B has null or negative results depending on setting.
Takeaways & Limitations
The scaffold's benefit is conditional: recurring mechanisms include short worker calls and shared notes for context management, plus problem decomposition, with larger improvements for some reasoning-disabled or smaller models.
Takeaways & Limitations
Provider routing introduced a substantial, time-varying confound in the broader sweep, so the paper's headline claims rely on pinned-backend arms.
Abstract
from arXiv · showhide
Multi-agent large language model systems are widely reported to beat single-model baselines, but the evidence is mixed, and comparisons are usually confounded: pipelines change token budgets, tool calls, and prompts simultaneously, so an aggregate gain rarely reveals what actually helped. We investigate the effect of introducing the manager-worker scaffold over a shared filesystem workspace, with no training and no per-benchmark tuning, measured against the same model answering in a single pass. Across nine models -- five open-weight, spanning 9B to ~2.8T parameters, and four frontier closed models -- on the 100 latest hard LiveCodeBench problems, the scaffold's benefit is real but conditional: large and statistically significant for some (Qwen3.8-27B +23.4, GPT-5.6-Luna +10.6 and GPT-5.6-Terra +8.0, each over five paired passes; Kimi-K3 +30.4 and Minimax-M3 +11.0 over five paired passes with reasoning off, both at $p < 10^{-4}$, and +42 and +12 in a single pass at a 128k cap) and null or negative for others (Qwen3.6-35B -1 to -9 with reasoning off). With the manager, Opus-5 achieves the highest score in the study at 91% in one pass. Running a manager roughly triples the token bill, but it buys accuracy more cheaply than moving to a larger model does: GPT-5.6-Terra with a manager nearly matches Fable 5's single-call accuracy (85.0 against 87.4, $p = 0.59$) at a fifth of the price (\$11.71 against \$61.11 per 100-problem pass, $p < 10^{-4}$), and the Qwen-27B arm does it for \$51.75 on weights anyone can self-host. Our transcript analysis finds several mechanisms behind the gains, of which two recur: context management, in which short worker calls and shared notes organize state and reduce truncation, and problem decomposition. Improvements are modest for large models with reasoning enabled, but larger for some models with reasoning disabled and for smaller models with reasoning enabled.
1 Introduction
Multi-agent LLM systems promise gains through coordination, but prior comparisons often confound coordination with changes in compute, prompts, tools, and retrieval. This paper isolates a training-free manager–worker scaffold over a shared workspace against the same model answering once.
- Motivation: Multi-agent systems may outperform single-pass models by decomposing problems, critiquing attempts, and pooling partial results, but existing evidence is mixed and confounded.Prior pipelines commonly change token budgets, tool calls, prompts, and retrieval simultaneously.
- Experimental comparison: The study compares the same model and problem set in single-pass and manager–worker conditions, adding only dynamic coordination and test-time computation.The manager revises tasks, selects the next step, and stops when appropriate; every role uses the same model in a fresh context.
- Design distinction: Unlike fixed workflows, the manager re-curates the task list after each round and decides when to stop for each problem.This places the design between learned orchestrators, hand-designed pipelines, and shared-blackboard systems while retaining no training or learned search policy.
- Positioning: Zero-shot self-orchestration uses inference-time orchestration without training or task-specific decomposition demonstrations.The manager is a fixed-prompt, training-free counterpart to learned orchestration systems.
- Design: The scaffold combines a persistent shared filesystem ledger with an adaptive manager that curates tasks and delegates one next step at a time.The workspace stores the plan, task list, notes, and current best solution across fresh agent invocations.
2 Results
Across pinned-backend experiments, the manager-worker scaffold improves accuracy for the tested models, with larger gains for weaker single-call systems, but the improvement carries substantially higher token costs and varies by model and failure mode.
- Accuracy: +10.6 ± 5.1 and +8.0 are GPT-5.6-Luna’s and GPT-5.6-Terra’s manager gains, respectively, with all three tested models benefiting across five passes.Luna’s per-pass gain ranges from +4 to +17, while Terra’s narrower band is more reliable.
- Accuracy: +23.4 points is Qwen3.8-27B’s largest gain, reaching 86.4 ± 2.7 while narrowing its spread from 4.1 to 2.7 SD.Its manager arm is comparable to Fable 5 and Opus-5 single-call scores.
- Accuracy: The manager’s gain shrinks monotonically as single-call accuracy strengthens: +23.4, +10.6, then +8.0 points.The managed arms converge a few points below the study’s best single-call score rather than exceeding it.
- Cost: +153%, +266%, and +244% are the token-bill increases for Qwen3.8-27B, GPT-5.6-Luna, and GPT-5.6-Terra, respectively.Their per-pass costs rise from $20.44 to $51.75, $0.41 to $1.50, and $3.41 to $11.71.
- Cost versus accuracy: 85.0 versus 87.4 pass@1 lets GPT-5.6-Terra with a manager nearly match Fable 5 at $11.71 versus $61.11 per pass.The accuracy difference is within error (p = 0.59), while the savings are significant (p < 10^-4).
- Failure modes: 25/500 rescued no-code Qwen3.8-27B cells account for 5.0 points, about one-fifth of its +23.4-point total improvement.The manager passed 25 of the single arm’s 35 no-code cells and failed the remaining 10.
3 Method
The study compares a fixed model in a single call with the same model in a training-free, dynamically coordinated manager–worker scaffold using a shared filesystem workspace. The v2 loop adds bounded coordination, sample verification, and cut-off summarization, but evaluator defects and backend differences constrain interpretation.
- Control flow: V2 uses 10 manager→worker cycles, a sample-test verifier, and a cut-off summarizer, whereas the original uses 4 cycles without either addition.A no-progress guard is retained across versions.
- Controls: Manager-minus-single deltas are reported separately because the two scaffold versions differ in manager-arm features and are not strictly comparable.The single-call baseline remains one call under either version.
- Design: The workspace persists the plan, task list, notes, and current solution across fresh model invocations.These files are the only shared state among roles.
- Control flow: The v2 loop runs planning, brainstorming, dynamic task management, single-task implementation, sample verification, and finalization.The manager can stop when the problem is done, while failed public samples force continuation.
- Controls: The single-call baseline receives one call with no shared workspace or loop, while manager generative calls use higher temperatures than execution calls.The baseline and workers share the solver prompt, but workers additionally receive scaffold state and a structured subagent contract.
- Evaluation: The pinned-backend conditions use a 128k output cap with reasoning on and five passes, while providers and context limits differ across arms.The cap bounds generation rather than the context window; Qwen3.8-27B instead shares its 262,144-token context between prompt and output.
- Evaluation: A LiveCodeBench evaluator defect repeatedly returned the first binary input line, so all §2.1–§2.3 scores were corrected by rescoring stored generations.The correction replaced the binary mock with a position-advancing BytesIO; no model was rerun.
4 Discussion
The transcript evidence attributes gains mainly to explicit decomposition and context management, especially when models cannot organize reasoning or stop generation effectively. The scaffold can also regress performance when deliberation replaces a better initial plan, and provider reliability limits some comparisons.
- 4.1 Where the manager clearly helps: Written complexity warnings and range-query planning let workers avoid a naïve O(N · M) scan that timed out in a 9B single pass.The manager’s brainstorm prescribed a range-query structure before implementation and also used iterative DFS to avoid recursion-depth failures.
- 4.1 Where the manager clearly helps: Splitting coupled objectives across worker cycles turned one tangled captioning solution into separate forward-cost and lexicographic-reconstruction dynamic programs.The single-pass program printed nothing on a small case, while decomposition separated the two implementation responsibilities.
- 4.1 Where the manager clearly helps: The scaffold converted an implicit algorithmic leap into an explicit written plan before implementation.A worker then implemented the plan over two manager→worker cycles.
- 4.1 Where the manager clearly helps: A manager-produced standard segment-tree solution was 2.5× smaller than a failed single-pass Segment-Tree-Beats-shaped implementation.The manager-only win occurred in four of five passes.
- 4.1 Where the manager clearly helps: Bounding lemmas, reductions, and proofs were recorded before coding, producing manager-only wins across all five passes in the Qwen example.For Qwen3.8-27B, the manager committed a min-cut reduction after the single call exhausted 250,000 tokens without emitting a program.
- 4.2 Context management: Short worker calls and organized notes keep reasoning from overflowing one context and preserve completed work on disk.The analysis links this context management to cut-off prevention and to models that otherwise continue self-verifying without terminating.
- 4.3 A large effect on thinking-disabled models: The manager’s largest benefits occur when models lack internal planning or let reasoning run away.Reported gains include +42 and +12 at 128k with reasoning off for Kimi-K3 and Minimax-M3, and +18 for Qwen3.6-35B with reasoning on.
- 4.4 Regressions: when the manager hurts: The scaffold can backfire when deliberation replaces a correct initial algorithm with a slower, incorrect plan.Qwen3.6-35B’s reasoning-off deltas were −1.2 at 16k and −9 at 128k.
5 Conclusion
The manager–worker scaffold improves coding performance for several fixed-model comparisons, but its gains are conditional across models and settings. It also increases cost substantially, while sometimes offering a cheaper route to accuracy than switching to a larger model.
- Performance: +23.4, +10.6, and +8.0 points were the gains for Qwen3.8-27B, GPT-5.6-Luna, and GPT-5.6-Terra, respectively, over five paired passes.All three comparisons used the same underlying model, problem set, pinned backend, and verifier-gated v2 scaffold.
- Performance: 91% was Opus-5’s highest observed score with the scaffold, while Qwen3.6-35B showed no significant improvement at 16k and a −9-point change at 128k with reasoning off.The strongest gains were conditional rather than universal: with reasoning off, three of four open models improved by +3 to +42 points.
- Performance: 86.4 was Qwen3.8-27B’s managed score, compared with 87.4 ± 1.1 for Claude Fable 5’s best single-call result.The managed Qwen result was also above Opus-5’s single-call score of 85.
- Cost: A manager roughly triples the bill, yet GPT-5.6-Terra with management costs $11.71 per pass versus $61.11 for Fable 5’s single call.GPT-5.6-Terra reaches 85.0 against Fable 5’s 87.4, with the difference unresolved at p = 0.59.
- Cost: Qwen3.8-27B with management reaches within 1.0 point of Fable 5 using a much smaller parameter count suitable for local inference.The paper presents this as a lower-cost alternative to switching to a larger model.