Source-linked AI summary
RobustSGPO: Search-Space Control for Agent Harness Evolution
Zibo Zhao, Jijun Shi, Mo Zhou, Zhongyuan Wang, Shifu Bie, Yunfei Zhang, Xuanting Zhou, Xiangyu Wu, Bin Liu, Ruiming Tang, Wenwu Ou, Kun Gai
TL;DR
RobustSGPO addresses unresolved edit-scope and operation choices in SGPO by controlling requested edits, patch construction and checking, and search starting points. Across held-out tasks and equal-token evaluations, it improves completion and quality over SGPO, while retention reduces source-task degradation but adds overhead.
Problem
SGPO leaves edit scope and operation selection unresolved, while restricting edits can limit repair of cross-agent failures and broad permissions may not yield valid edits.
Method
RobustSGPO specifies requested edits, constructs and checks patches, and continues search from the incumbent or retained snapshots using explicit edit controls.
Results
RobustSGPO completes 24/30 held-out tasks versus SGPO’s 18/30 and reaches test quality 4.14 versus 3.77 within 20 million tokens.
Takeaways & Limitations
Periodic permissions improve final quality, executable prescribed edits improve structural validity, and retained starting points expand search coverage, with retention adding search overhead.
Takeaways & Limitations
The current program does not automatically update caller instructions when creating an evidence agent, so loading, references, invocation triggers, and task quality are checked separately.
Abstract
from arXiv · showhide
Semantic-gradient-based prompt optimization (SGPO) improves agent harnesses using execution feedback, but its local update rule leaves the choice of edit scope and operation unresolved. We introduce RobustSGPO, which specifies the requested edit, constructs and checks the patch, and continues search from either the incumbent or retained snapshots. We evaluate permission scheduling, cumulative controls, and task-family transfer in the AgentX brainstorming workflow using 120 tasks, 95 runs, and 7,350 candidate attempts. Periodic $1\to2\to3$ scheduling exceeds fixed maximum permission by 0.28 test-score points. RobustSGPO increases completion on 30 held-out tasks from 60.0% to 80.0% and improves test quality from 3.77 to 4.14 under a 20-million-token budget. Category retention reduces source-task degradation after a shift, whereas random retention reaches a higher destination endpoint. Search-space control benefits quality through executable edits and alternative starting points, with measurable retention overhead.
1 Introduction
The paper extends SGPO for agent-harness evolution by controlling which edits are selected, making them executable, and retaining alternative search starting points. It evaluates these controls against held-out quality, transfer, and token cost.
- SGPO diagnoses trace failures, edits an agent specification, and admits candidates through paired replay.
- Multi-agent behavior is distributed across instructions, input/output contracts, and agent connections, so editing one agent can limit direct repair of cross-agent failures.
- Opening the whole permitted space does not ensure exploration because generation can still concentrate on familiar instruction rewrites.
- RobustSGPO explicitly controls edit selection, execution, and retention, connecting these controls to held-out quality, transfer, and token cost.
2 Related Work
Related work spans harness evolution, structured search, prompt optimization, and candidate retention. RobustSGPO focuses specifically on selecting scope, operation, and targets before generation while enforcing prescribed edits, and its experiments do not rank it against those systems on shared benchmarks.
- Harness evolution: Prior harness-evolution systems search code, expose editable components, or combine failure mining with regression validation.
- Structured search and adaptation: Structured-search systems explore roles, topology, protocols, bounded configurations, typed compositions, harness trees, or joint solution–harness evolution.
- Prompt evolution and retention: Prompt-evolution methods optimize textual variables, while GEPA retains complementary candidates on a Pareto frontier.
- Prompt evolution and retention: RobustSGPO instead groups complete snapshots by edit scope and operation, and the experiments isolate cumulative controls within AgentX rather than ranking shared-benchmark performance.
3 System Context and Baseline SGPO
AgentX uses a closed-loop workflow in which brainstorming, development, and evaluation agents consume feedback and trajectory data while harness evolution improves them. The baseline SGPO-I loop samples traces, derives replay tasks and semantic gradients, refines one target agent, and accepts or rolls back updates through paired replay.
- AgentX workflow: In the evaluated brainstorming workflow, a Brainstorm Agent generates ideas, a Developing Agent changes code, and an Evaluation Agent analyzes experiments.
- AgentX workflow: AgentX replaces a manually handed-off recommendation pipeline with an agent-driven closed loop involving brainstorming, development, and evaluation agents.
- Baseline SGPO-I: SGPO-I samples traces, derives rubrics and standalone replay tasks, and computes a natural-language loss and semantic gradient for one target agent.
- Baseline SGPO-I: The gradient identifies missing requirements, sequencing errors, or broken contracts, after which a refinement agent generates an update.
- Baseline SGPO-I: The experiment agent replays identical tasks on both versions, admitting improvements and rolling back otherwise while recording failed patches, scores, and diagnoses.
4 The Search-Space Problem
The search-space problem is the gap between permitted edits and valid generated edits. The paper distinguishes three permission levels and tests scheduling, effective-category coverage, structural validity, and retained starting points after a task shift.
- The nominal space contains permitted edits, while the realized space contains edits actually generated; validity checks select the effective space.
- Effective edits are valid rather than merely admitted by replay, and broader permissions need not yield more valid edits or higher quality.
- α1 edits one designated agent, α2 edits any existing brainstorming agents while preserving their set, and α3 additionally permits agent addition, removal, and routing changes.
- E1 compares permission schedules, E2 measures valid-category coverage and structural validity, and E3 tests retained starting points after a task shift.
5 Robust SGPO
RobustSGPO makes search-space control explicit within AgentX’s existing diagnosis, proposal, replay, and accept/rollback loop. It selects requested edits, constructs and checks patches, and continues search from the incumbent or retained snapshots.
- 5 Robust SGPO: RobustSGPO adds three explicit decisions: which edit to attempt, how to construct and check it, and which version to continue from.It retains AgentX’s diagnosis, proposal, replay, and accept/rollback loop.
- 5.1 Choose What to Change This Round: The illustrative repair aligns matching fields across idea and validate rather than merely asking idea to provide more evidence.The request specifies a pair edit, field-alignment operation, and targets idea and validate.
- 5.1 Choose What to Change This Round: The controller selects feasible scope–operation categories and eligible targets before content generation, using feedback to guide the requested edit.Scope and operation need not be independent, and permissions bound allowed changes.
- 5.2 Construct and Check the Requested Patch: Exact-target checking rejects patches whose changed object set differs from the requested target set, but does not establish semantic correctness or task quality.A request naming idea and validate rejects changing only idea or additionally changing propose.
- 5.2 Construct and Check the Requested Patch: The Typed Compiler constructs prescribed addition and removal patches, while ordinary instruction edits remain model-generated.Predefined construction performs repeated structural editing work and helps requested changes pass validity checks.
- 5.3 Continue from Which Version: The archive retains up to ten category snapshots, alternates archive and incumbent parents, and rescoring after task shifts is charged to search cost.Better candidates replace earlier entries within retained categories, while descendants must beat the current incumbent.
6 Experimental Results
Across permission scheduling, cumulative search controls, task-family transfer, and equal-token evaluation, the experiments show that controlling edit scope, executable patch construction, and retained starting points improves quality and completion, with retention introducing measurable overhead.
- E1: Periodic Scheduling Improves Final Quality: Periodic 1 →2 →3 scheduling finishes 0.28 points above fixed α3, despite fixed α3 leading early.Final test scores are 4.34 for periodic 1 →2 →3 and 4.06 for fixed α3.
- E2: Cumulative Search-Space Controls: RobustSGPO raises round-30 held-out quality from SGPO’s 3.82 to 4.30 while expanding effective-category coverage to 68%.The cumulative configurations report quality scores of 3.82, 3.94, 4.04, 4.20, and 4.30, with corresponding coverage values of 48%, 46%, 48%, 54%, and 68%.
- E2: Cumulative Search-Space Controls: Structured Search reaches 77.8% structural validity versus 48.9% for Constrained Search, reducing unsuccessful structural proposals.Figure 5 also records ten occupied archive categories and 17 admitted archive-origin updates.
- E3: Adaptation and Retention After a Task Shift: After the task shift, category retention has the smallest source-task loss, whereas random retention reaches the highest destination endpoint.The archive policy therefore changes the adaptation–retention trade-off rather than optimizing both endpoints simultaneously.
- 6.5 Behavioral Outcomes at Equal Token Cost: RobustSGPO completes 24/30 held-out tasks versus SGPO’s 18/30, including cross-agent completion of 11/15 versus 7/15.Local completion rises from 11/15 to 13/15.
- 6.5 Behavioral Outcomes at Equal Token Cost: Within 20 million tokens, RobustSGPO reaches test quality 4.14 versus SGPO’s 3.77, exceeding SGPO by 0.37 points while incurring archive overhead.Its margin over Structured Search shrinks from 0.10 at equal rounds to 0.04 at equal token cost.
7 Conclusion
RobustSGPO controls harness evolution by structuring requested edits, checking patches, and retaining alternative search starting points. These controls improve final quality, structural validity, and transfer while introducing search overhead.
- RobustSGPO decomposes free-form updates into requested edits, patch construction and checking, and retained search starting points.
- Periodic permissions improve final quality, while predefined add/remove operations improve structural validity.
- The full method outperforms free search at equal token cost, while category retention reduces source-task degradation after transfer with search overhead.