Source-linked AI summary
SoL-Pi: Recursively Scaling Auto-Research Loops for Efficient Agent Harness
Haozhe Liu, Tian Ye, Sensen Gao, Qihang Cao, Yitong Li, Mingchen Zhuge, Duomin Wang, Ruihua Zhang, Ping Luo, Jiawang Bian, Lei Zhu, Ligeng Zhu, Enze Xie, Song Han
TL;DR
Long-horizon agent work makes token efficiency important, while harness optimization is difficult because its interacting components require costly trace inspection and tuning. SoL-Pi scales RSI-inspired auto-research across diverse environments to discover and combine reusable harness mechanisms. On EdgeBench, it achieves performance comparable to Pi across GPT-5.6 Sol and Opus 5 while substantially reducing token traffic and cost.
Problem
Long-horizon agents make task-level token efficiency a systems concern, but harness optimization is difficult to scale because coupled components require substantial human effort to inspect traces and translate recurring failures into changes.
Source: §1 · p. 2Method
SoL-Pi iteratively analyzes execution traces, tests harness changes in prepared environments, and retains candidates using capability and efficiency checks, with held-out validation separated from search.
Source: §1 · p. 2Results
44.7–49.0% lower recorded token traffic and about one-third lower token cost accompany comparable EdgeBench performance across GPT-5.6 Sol and Opus 5.
Source: §5 · p. 12Takeaways & Limitations
Scaling auto-research across many tasks and ideas provides preliminary evidence that it can yield reusable harness improvements beyond their development setting.
Source: §1 · p. 2The current harness was updated from trajectories generated by a single LLM, and its mechanisms trigger less often and less intensively on the other evaluated backend.
Source: §5.1 · p. 12
Abstract
from arXiv · showhide
As coding agents move from supervised code completion to unattended, around-the-clock exploration, their work expands from isolated predictions into long trajectories of reasoning, tool use, and feedback. Token efficiency therefore becomes important for scaling recursive self-improvement. We take an RSI-inspired approach at the harness layer, scaling auto-research loops across increasingly numerous and diverse environments for harness rollouts. At this scale, the process yields reusable improvements that transfer beyond their development setting, moving automated harness discovery toward production-level outcomes. Four mechanisms survive selection and form SoL-Pi, spanning action execution, context compaction, observation handling, and delegated reading. On the 51-task EdgeBench evaluation, SoL-Pi achieves performance comparable to Pi across GPT-5.6 Sol and Opus 5 while reducing recorded token traffic by 44.7-49.0% and API cost by about one third. In other words, estimated hourly savings are \$8.75-\$13.50 relative to native Codex and Claude Code harnesses, and \$4.36-\$5.71 relative to Pi.
1. Introduction
Long-horizon agents make token efficiency a systems concern, but harness changes are difficult to optimize because tool use, context, verification, and recovery are coupled. SoL-Pi uses an RSI-inspired search process with separate development and held-out validation, finding four reusable mechanisms that reduce token traffic while maintaining comparable EdgeBench performance across two models.
1. Introduction
- Harness optimization is difficult to scale because tool use, context management, verification, delegation, recovery, and termination are tightly coupled, while inspecting traces and turning failures into code changes demands substantial human effort.A locally beneficial change can cause downstream failures or shift token costs to later execution stages.Source: §1 · p. 2
- SoL-Pi has an AI optimizer analyze a separate base-harness agent’s traces, propose and test changes in prepared environments, and use capability and efficiency checks to retain candidates.Development results guide subsequent iterations.Source: §1 · p. 2
- Prior automated harness searches have shown feasibility, but evolved harnesses can overfit search tasks and yield only marginal gains on unseen tasks, motivating separate search feedback and final evaluation.SoL-Pi separates candidate development from held-out validation to seek improvements that transfer beyond search tasks.Source: §1 · p. 2
- The search design combines broad hypothesis coverage with repeated refinement, evaluates held-out evidence only after candidates are frozen, and uses isolated lineages to prevent failures from coupling across candidates.These principles guide scaling AI-led auto-research across diverse ideas and environments.Source: §1 · p. 2
- 44.7–49.0% lower recorded token traffic and about one-third lower API cost accompany performance comparable to Pi across GPT-5.6 Sol and Opus 5 on EdgeBench.The search covered roughly 150 proposed directions and 500 executable environments, with more than 3,000 runs and 60,000 agent–environment interactions.Source: §1 · p. 2
2. Method
The method searches for harness changes that improve efficiency across development environments while preserving task capability, then validates frozen candidates separately. Broad, isolated research loops use repository and verifier-driven tasks to discover and combine four mechanisms targeting different sources of interaction overhead.
2.1. Harness Auto-Research for Token Efficiency
- Candidates pass only if all capability metrics stay within predeclared tolerances and at least one declared efficiency metric improves; EdgeBench evaluation is held out until the harness and acceptance rule are frozen.The pipeline retains nondominated candidates, and held-out failures reject candidates without further optimization.Source: §2.1 · p. 3, Figure 2 · p. 3
2.2. Broad-to-Deep Harness Search
- The broad-to-deep search begins with 152 directions across six proposal families, then develops selected hypotheses through independent implementation, review, fixed experiments, and iterative refinement.Disposable instances of a shared skill template provide breadth through more isolated loops and depth through repeated refinement; these search counts do not establish a scaling law.Source: §2.2 · p. 3
2.3. Search Environments
- The search set contains 535 executable environments: 495 repository tasks and 40 synthetic verifier-driven tasks, while EdgeBench remains reserved for held-out evaluation.Repository tasks use pre-fix repositories and hidden regression tests; verifier-driven tasks allow multiple valid solution paths. Of EdgeBench’s 51 public tasks, 11 support one-way acceptance and 40 are reserved for final generalization evaluation.Source: §2.3 · pp. 3–4, Figure 3 · p. 4, §2.5 · p. 6
2.4. Discovered Harness Mechanisms
- Action Fusion combines a file mutation with its follow-up command in one request, while Online Context Compact checks at plan-step completion whether projected input savings exceed cache-rewrite costs.Commands requiring inspection of the mutation result remain separate; later compactions account for unrecovered rewrite costs.Source: §2.4 · p. 4, Figure 4 · p. 5
- ObservationPack replaces repeated large outputs with a handle and excerpt after two full requests, while Evidence-Preserving Reducer compacts eligible logs into verified receipts and falls back to originals when checks fail.Exact archived output remains retrievable; the reducer processes results before ObservationPack, which skips marked receipts to preserve verified evidence.Source: §2.4 · pp. 4–5, Figure 4 · p. 5
2.5. Implementation Details and Backend Setup
- The four mechanisms target complementary workflow stages, and all are implemented as Pi extensions; the combined harness is evaluated end to end for capability and efficiency.The implementation freezes mechanism source, configuration, metrics, tolerances, and acceptance rule before EdgeBench evaluation; its context-compaction gate does not separately price the summarization call.Source: §2.4 · p. 5, §2.5 · pp. 5–6
3. Experimental Evaluation
The evaluation tests SoL-Pi on EdgeBench, Terminal-Bench 4, IMO 2026, and kernel optimization, comparing task scores, token traffic, and API cost. It examines cross-backend transfer, agent swarms, mechanism effects, and the development of Action Fusion.
3.1. Overall Comparison
- On GPT-5.6 Sol, SoL-Pi [Efficiency] uses 1.10 B tokens (49.0% fewer than Pi) while retaining 93.7% of Pi’s average score (42.0 vs. 44.8); [Performance] raises average score 5.3%.The Efficiency point’s token cost is 33.2% lower than Pi’s; the Performance point also reduces token traffic by 6.1% and improves token efficiency by 9.8%.Source: §3.1 · p. 6, Table 1 · p. 6
- On Opus 5, the unadapted SoL-Pi stack retains 94.3% of Pi’s average score, with 44.7% less token traffic and 33.5% lower API cost.It was developed with GPT-5.6 Sol and applied to Opus 5 without further search or adaptation.Source: §3.1 · p. 6, Table 2 · p. 7
3.2. Evaluation on Terminal-Bench 4 and IMO 2026
- On Terminal-Bench 4, SoL-Pi solves 15 of 63 tasks versus 18 for Codex and Pi, while costing 26.3% less than Pi ($211.12 vs. $286.45); on IMO it passes 3 of 6 at $20.90 per pass.The IMO cost per passed problem is below Codex’s $22.89 and Pi’s $25.32.Source: §3.2 · p. 7, Table 3 · p. 7
3.3. Efficient Agent Swarms
- In one two-hour kernel run, the SoL-Pi swarm reaches 1,127 cycles at $60.11 versus Pi’s 1,366 cycles at $82.12; all final candidates pass the official correctness check.The SoL-Pi swarm’s API cost is 26.8% below the Pi baseline swarm, while the single agent remains least expensive.Source: §3.3 · pp. 7–8, Figure 5 · p. 8
3.4. Learned Mechanisms and Backend Behavior
- Every mechanism reduces total tokens under both backends; Opus 5 has lower activation, yet every configuration improves efficiency on triggered tasks and the full stack preserves a similar score–efficiency trade-off.ObservationPack has the highest average score under GPT-5.6 Sol, while Action Fusion does so under Opus 5; the full stack has the lowest total token count and token cost in both backend blocks.Source: §3.4 · p. 8, Table 4 · p. 9, Figure 6 · p. 9
- Full-stack gains exceed standalone gains on each mechanism’s own triggered-task subset, though these descriptive comparisons do not isolate interaction effects; total cost also falls despite reduced cache-read traffic.For GPT-5.6 Sol, cache-read traffic drops from 2.1326 B to 1.0605 B tokens, cache-write traffic rises from 0.0141 B to 0.0316 B, and total model cost falls from $1,339 to $894.Source: §3.4 · pp. 8–9, Figure 7 · p. 10
3.5. From Observation to a Retained Mechanism
- Action Fusion’s lineage spans 27 iterations across four stages; unreliable prompt-only triggering led to exposing the fused action in the tool schema, then selecting a frozen configuration using task score and trigger rate before held-out validation.Stage 03 recorded 18 prompt-optimization explorations, while the figure reports ten retained steps.Source: §3.5 · pp. 10–11, Figure 8 · p. 10
4. Related Work
Related work connects agent performance to harness and interface design, then studies automated harness and context optimization. SoL-Pi differs by searching across multiple harness components for reusable efficiency improvements while maintaining task performance.
4.1. Agent Harnesses and Automated Agent Design
- Harness research links interface and system design to agent behavior, while automated-design work searches prompts, code, workflows, and modular components with the model fixed.Recursive self-improvement work also explores editable agent code and meta-level procedures.Source: §4.1 · p. 11
4.2. Automated Harness Optimization
- Prior harness optimizers search code, prompts, configurations, or experience, but generalization remains an open question after reports of overfitting and marginal held-out gains.SoL-Pi searches diverse development environments and keeps final held-out evaluation separate from candidate development.Source: §4.2 · p. 11
4.3. Context and Token-Efficient Agents
- Context methods control context growth through code-based control flow, trajectory removal or compression, folding, explicit context maintenance, and reusable strategies.SoL-Pi instead automates discovery across harness components, combining selected mechanisms for action execution, context compaction, observation handling, and delegated reading.Source: §4.3 · pp. 11–12
5. Conclusion
SoL-Pi combines four harness mechanisms and, on EdgeBench, reduces token traffic and cost at comparable performance. The conclusion presents harness pretraining, multi-backend training, recursive efficiency, and search scaling as future directions or unresolved challenges.
5. Conclusion
- On EdgeBench, the complete stack reduces token traffic by 44.7–49.0% and token cost by about one third at comparable performance.The complete stack is consistent across GPT-5.6 Sol and Opus 5 within the evaluated setting.Source: §5 · p. 12
5.1. Limitations and Future Directions
- Sustained gains from exposing the harness to more tasks and diversifying research ideas remain a hypothesis, with current results providing preliminary evidence.The paper calls this long-term direction “pretraining the harness.”Source: §5.1 · p. 12
- The current harness was updated from one LLM’s trajectories; on the other evaluated backend, its mechanisms trigger less often and less intensively.The mechanisms still improve token efficiency when triggered; multi-backend training and validation may improve robustness while preserving task quality and efficiency.Source: §5.1 · p. 12
- Recursive efficient improvement is a long-term vision: compounding efficiency gains have not been demonstrated in this study.The proposed next cycle would use lower per-run costs to cover more environments, trajectories, and research ideas under a fixed budget.Source: §5.1 · p. 12
- Complete auto-research loops are computationally expensive, so controlled fixed-budget comparisons of search breadth and depth remain challenging.Systematic investigation of scaling laws along these dimensions is left to future work.Source: §5.1 · p. 12