PaperlayerView exact revision on arXiv

Source-linked AI summary

Harmony Harness: Multi-Objective Self-Evolution of Agent Harnesses

Weichen Yu, Lesly Miculicich, Bhavana Dalvi Mishra, Jiefeng Chen, Shuo Li, Matt Fredrikson, Tomas Pfister, Long T. Le

arXiv:2610.15150v1cs.AI

TL;DR

Self-evolving harnesses have largely optimized one objective and evolved as monolithic artifacts, leaving competing goals and transferable versus environment-specific design insufficiently separated. Harmony Harness uses decomposition-based multi-objective search and factorized, dynamically composed harness evolution. It consistently outperforms baselines across benchmarks, with the authors reporting gains from combining these approaches.

  • Problem

    Existing harness self-evolution optimizes a single objective and evolves monolithic artifacts, leaving safety as an independent objective and reusable versus environment-specific design insufficiently addressed.

    Source: §1 · p. 1

  • Method

    Harmony Harness uses decomposition-based search across objective preferences and separately evolves general-purpose and environment-specific harness components for dynamic composition at inference time.

    Source: §1 · p. 2, §4 · p. 4

  • Results

    Harmony Harness consistently outperforms baselines across benchmarks with identical dual objectives; it achieves 76.7 F&S on the SecRepoBench evolve split.

    Source: §1 · p. 3, §5.1 · p. 9, §6 · p. 11

  • Takeaways & Limitations

    In secure code generation, joint harness-level evolution improves functionality and security together and generalizes to unseen vulnerabilities, while factorization further improves in-domain and cross-benchmark generalization.

    Source: §6 · p. 11

    On SecRepoBench the Pareto front is narrow, and the framework's frontier-search advantage is expected to matter most where security and functionality genuinely trade off.

    Source: §I · p. 28

Abstract

from arXiv · show

Recent work on self-evolving LLM agents treats the agent harness as an optimization target that can be iteratively improved through execution feedback. Existing approaches, however, largely optimize a single objective and evolve the harness as a monolithic artifact, limiting their ability to handle competing goals and to separate transferable design from environment-specific adaptation. We introduce Harmony Harness, a framework for multi-objective, compositional harness evolution. Harmony Harness formulates harness evolution as a multi-objective optimization problem and uses decomposition-based evolutionary search to maintain solutions across different regions of the Pareto frontier, avoiding reliance on a hand-designed structural diversity metric for repository-scale harnesses. In parallel, we introduce factorized evolution operators that separately refine general-purpose harness mechanisms and benchmark-specific adaptations, together with a dynamic composition workflow that integrates them at inference time. Across SecRepoBench and BaxBench, Harmony Harness consistently outperforms strong harness-evolution baselines, with gains exceeding 10% on both benchmarks. Our results show that explicitly modeling both objective trade-offs and harness structure yields substantially stronger and more transferable agent evolution than existing single-objective, monolithic approaches.

1. Introduction

The introduction identifies two gaps in self-evolving agent harnesses: single-objective optimization and monolithic evolution that mixes reusable design with environment-specific adaptation. Harmony Harness addresses these with decomposition-based multi-objective search and factorized harness evolution, and the authors report gains over baselines from combining these mechanisms.

1. Introduction

  • Existing harness self-evolution optimizes a single scalar objective, leaving multi-objective evolution—especially safety as an independent axis—unaddressed.The introduction also describes monolithic evolution as mixing transferable mechanisms with environment-specific calibration under a single-environment reward signal.Source: §1 · pp. 1–2
  • Linear scalarization can miss non-convex Pareto regions, while constrained optimization reduces the problem to constrained single-objective search rather than mapping trade-offs.In the secure-coding benchmarks studied, functionality and security often improve together, so decomposition's value is objective-directed search pressure rather than recovering non-convex regions.Source: §1 · pp. 1–2
  • A harness combines general capabilities with environment-dependent calibration, but monolithic evolution optimizes and encodes both together against one environment's feedback.Examples include container execution versus benchmark-specific sandbox conventions, and output compression versus deciding what can safely be omitted from a particular tool's output.Source: §1 · p. 2
  • Decomposition assigns objective preferences to subproblems across Pareto-frontier regions, diversifying selection without a hand-designed structural diversity metric for repository-scale harnesses.Such metrics are difficult to define when harness variations span prompts, control flow, tools, verification logic, and orchestration; objective-based selection does not guarantee behavioral diversity.Source: §1 · p. 2
  • Harmony Harness consistently outperforms baselines and generalizes across benchmarks with identical functionality and security objectives; factorized evolution and dynamic orchestration outperform static workflows.The authors attribute the margin over naive formulations to combining decomposition, factorized evolution, and dynamic orchestration, rather than to the aggregation rule alone.Source: §1 · p. 3

2. Related Work

Prior agent-improvement work includes reflection, memory, skills, workflow updates, prompt evolution, and code-level agent evolution. The paper distinguishes Harmony Harness by evolving the whole harness across functionality and security objectives while separating reusable from environment-specific components.

2. Related Work

  • GEPA maintains a Pareto frontier of prompts, while ADAS treats the agent as editable code; Harmony Harness adapts frontier search to whole harnesses and a functionality-security objective space.Harmony Harness uses MOEA/D with LLM variation operators rather than ADAS's archive-based meta-agent search (Agrawal et al., 2026).Source: §2 · p. 3
  • The Darwin Gödel Machine and meta-harness evolve coding-agent code monolithically, without separating reusable and environment-specific components (Zhang et al., 2026b; Lee et al., 2026).The related work also contrasts safety-as-objective approaches with constrained formulations, while Harmony Harness keeps safety as an objective and searches the frontier by decomposition.Source: §2 · p. 3
  • Earlier self-improving systems use reflection, memory, skill libraries, or workflow updates, including Voyager, Reflexion, and AdaPlanner (Wang et al., 2024; Shinn et al., 2023; Sun et al., 2023).The paper characterizes these as conceptual predecessors and says it borrows ideas, not their algorithms or code.Source: §2 · p. 3

3. Problem Formulation

The problem formulation treats harness evaluation as execution-grounded functionality and security objectives over tasks with hidden security policies. It searches the objective marginals and selects a balanced harness using search-split results, keeping held-out splits out of selection.

3. Problem Formulation

  • Each harness is evaluated on functionality and security using task executions; the task specification is visible to the agent, while the corresponding security policy is hidden at solve time and enters through the evaluation oracle.The policy can include an exploit, and execution induces a trajectory distribution reflecting agent stochasticity.Source: §3 · p. 3
  • The search optimizes functionality and security marginals rather than their joint rate, which is bounded by the marginals but can differ between harnesses with identical marginal scores.Raising both marginals can raise the joint-rate ceiling and, when their sum exceeds 1, its lower bound; the paper reports the joint rate and the gap from the smaller marginal.Source: §3 · pp. 3–4
  • The proposer receives marginal objectives because they indicate which property failed on a task, whereas joint success alone does not.For balanced weights, a Tchebycheff subproblem emphasizes the currently worse objective; the deployed harness is selected as the most balanced child using search-split results, with held-out splits excluded.Source: §3 · p. 4

4. Multi-Objective and Factored Harness Evolution

Harmony Harness evolves repository-scale harnesses toward different Pareto-frontier regions and factorizes updates by component type and adaptation scope. Separate operators evolve general-purpose and environment-focused modifications, which are composed dynamically at inference time.

4. Multi-Objective and Factored Harness Evolution

  • The framework jointly maintains incumbent harnesses for objective-preference subproblems, with an LLM proposer generating candidates from execution-grounded feedback and selectively replacing nearby incumbents.The population evolves toward different regions of the Pareto frontier.Source: §4 · p. 4
  • Factorization localizes updates by component and separates reusable mechanisms from environment-specific adaptations, which are evolved through different operators and dynamically composed at inference time.The paper presents these decompositions as an inductive bias, not as unique or theoretically exact partitions.Source: §4 · p. 4
  • The evolvable genome is the entire agent harness except for a minimal evaluator interface, including prompts, tools, skills, control logic, verification mechanisms, and orchestration.Tools provide executable capabilities, skills encode reusable domain knowledge, and logic and orchestration govern how capabilities are used.Source: §4 · p. 4

1 Diagnose Execution feedback

Harmony Harness evolves harnesses across functionality–security trade-offs while separating reusable components from environment-specific adaptations. At inference, it composes selected components into a task-specific harness.

4.1. Multi-Objective Evolution by Decomposition

  • Each weighted objective subproblem maintains an incumbent harness, and together the subproblems approximate different trade-offs along the Pareto frontier.Unlike a single globally authoritative scalar reward, local scalarization preserves distinct objective preferences.Source: §4.1 · pp. 5–6
  • Preserving incumbents for distinct objective preferences avoids a hand-designed structural-diversity metric, but does not guarantee behavioral diversity.For repository-scale harnesses, structural distance can poorly reflect behavioral differences.Source: §4.1 · p. 6

4.2. Factorized Harness Evolution

  • The proposer diagnoses execution failures, forms an improvement hypothesis, then selects both a component class and whether its edit belongs in general or environment-specific scope.The component and scope choices can be revised during evolution.Source: §4.2 · p. 7
  • A dedicated generalize operator can reorganize components across scopes when their placement creates unnecessary specialization.Source: §4.2 · p. 7

4.3. Dynamic Composition at Inference

  • At inference, a composition agent selects components from general and environment-specific pools to build a task-specific harness conditioned on the environment and task.It selects components without modifying the evolved components, loading environment-specific mechanisms only when relevant.Source: §4.3 · pp. 7–8

5. Experiments

Harmony Harness leads the reported comparisons on both benchmarks, including held-out splits and zero-shot cross-benchmark transfer. Experiments compare evolution strategies under matched conditions and assess how the framework transfers to another benchmark and task format.

5. Experiments

  • Experiments evolve each method on the evolve split, then evaluate without further evolution on both held-out splits.SecRepoBench and BaxBench each use separate harness populations; the reported metric is secure-pass@1, the conjunction of functional and security success.Source: §5.1 · p. 8, §5 · p. 8

5.1. Main comparison against self-evolution baselines

  • 76.7 F&S on SecRepoBench evolve and 61.3 average F&S on BaxBench; Harmony Harness is best on every split of both benchmarks.On SecRepoBench held-out IN and OUT, it scores 86.7 and 66.7 F&S, respectively, versus 50.0 and 36.7 for baseline.Source: §5.1 · p. 9, Table 2 · p. 9
  • On SecRepoBench, Tchebycheff replacement reaches 76.7 F&S versus 70.0 for weighted-sum and sequential evolution, 60.0 for hard constraint, and 53.3 for separately evolved harnesses.The component ablation adds factored evolving and dynamic orchestration in turn, lifting F&S from 63.3 to 66.7 and then 76.7.Source: §5.1 · p. 9, Table 2 · p. 9
  • Dynamic orchestration on BaxBench adds +10.7 secure-pass and +3.6 functional-pass over a fixed pipeline, with an 8% token overhead.It averages 1.2 extra steps and spends more only where it helps.Source: §5.1 · p. 10

5.2. Cross-benchmark transfer

  • Zero-shot transfer lifts F&S over baseline to 57.1 on BaxBench (+14.2) and 70.0 on SecRepoBench (+23.3).The target benchmark is not seen during evolution; in-domain SecRepoBench F&S is 76.7.Source: §5.2 · p. 10
  • On CWEval, F&S improves in all five languages, including C and C++, which are absent from BaxBench.Per-language gains range from +10.1 for JavaScript to +19.3 for Go; functionality falls on three languages, with an overall trade-off of 2.0 functionality points for 15.5 security points and 14.3 F&S points.Source: §5.2 · pp. 10–11

6. Conclusion

Harmony Harness treats functionality and security as separate objectives and evolves harnesses toward different regions of their Pareto frontier. Its conclusion reports improved security and functionality together in secure code generation and generalization to unseen vulnerabilities, with further in-domain and cross-benchmark generalization from factorization.

6. Conclusion

  • In secure code generation, joint harness-level evolution improves security and functionality together and generalizes to unseen vulnerabilities when feedback is execution-grounded.The conclusion contrasts this with naive feedback-driven evolution, which falls back.Source: §6 · p. 11
  • Factorizing general and environment-specific harness components further improves in-domain and cross-benchmark generalization.Source: §6 · p. 11

AI use statement

  • The authors report using generative AI for editing, drafting, standard implementation, and related-work discovery, with human verification and testing of assisted content and code.Source: AI use statement · p. 11

B. More Equations of the Problem Formulation.

C. Implementation Details

The implementation evaluates Harmony Harness on two complementary secure-code benchmarks and uses decomposition to search across objective trade-offs. On the studied benchmarks, the observed Pareto fronts are narrow, but decomposition still supplies distinct search directions.

C. Implementation Details

  • SecRepoBench tests repository-level secure code completion, while BaxBench tests complete backend-application generation, with both assessing functionality and security through execution.The paper also evaluates transfer on CWEval, an outcome-driven secure code-generation benchmark.Source: C. Implementation Details · p. 18
  • The main experiments use released Meta-harness and DGM code with the paper’s benchmarks and grader plugged in.Source: C. Implementation Details · p. 18

D. Details for Decomposition

  • On SecRepoBench, the observed front contains two non-dominated harnesses early, while in the last generation one harness is best on both objectives.The authors describe the observed fronts as narrow and note that Tchebycheff decomposition can recover non-convex Pareto regions, unlike linear scalarization.Source: D. Details for Decomposition · p. 18

E. Implementation Details for Factorized Harness Evolution

The factorized evolution implementation starts from a functional but modest harness, then applies scoped component edits and a generalization operator. Sequential edits are validated before candidates enter full task evaluation.

E. Implementation Details for Factorized Harness Evolution

  • The harness is an evolvable Python package whose entry point exposes the solver, including its LLM layer, tools, prompts, and control logic.The seed uses a minimal ReAct-style loop and basic tools, leaving higher-level mechanisms for evolution.Source: E. Implementation Details for… · p. 19

E.1. Diagnose Execution Feedback

  • Diagnosis supplies failure-grounded guidance for selecting component operators and scopes; unverifiable evidence suppresses that guidance rather than passing unproven claims to an operator.Source: §E.1 · pp. 19–20

E.2. Tool Evolution, Skill Evolution and Logic Evolution

  • Tool, skill, and logic operators edit distinct harness components, with scope chosen as general or benchmark-specific before editing and enforced through write permissions.General scope avoids hard-coded benchmark facts; benchmark-specific modules may override general defaults.Source: §E.2 · pp. 20–21

E.3. Generalization Operator

  • The generalize operator moves or rewrites one misfiled unit at a time, promoting reusable procedures, localizing unrecoverable benchmark facts, or deriving runtime facts.Moves are intended to preserve current-benchmark behavior while keeping the general partition executable on its own.Source: §E.3 · p. 22

E.4. Operator Scheduling

  • The proposer orders selected operators from its diagnosis, and each operator edits the latest candidate so later changes can build on earlier ones.Source: §E.4 · p. 22

E.5. Validation, Repair, and Rollback

  • Candidates undergo a three-stage smoke-test pipeline, receive one repair opportunity if a stage fails, and are rejected with repository rollback if validation fails again.Only candidates passing validation proceed to full task-level evaluation.Source: §E.5 · p. 23

F. Implementation Details for Dynamic Composition

Dynamic composition loads general components together with those for the active environment, then plans a task-specific workflow. The reported generalization-operator ablation improves best functionality-and-security results in both tested orchestrator settings.

F. Implementation Details for Dynamic Composition

  • At load time, the harness imports general components and only the active environment’s components; a planner then emits a task-specific DAG of registered workflow steps.The executor validates the DAG and runs it in topological order.Source: F. Implementation Details for Dynamic… · pp. 23–24

F.1. Composition Agent

  • The harness groups components as tools, skills, and logic, with logic covering prompting, verification, control flow, and orchestration.Source: §F.1 · p. 24

F.2. Runtime Harness Assembly

  • Generalization is maintained through evolutionary reorganization rather than a fixed file-level partition, because the general-versus-specialized distinction is semantic.The operator rewrites environment-dependent mechanisms or confines specialization to environment-specific components.Source: §F.2 · p. 24

G. Additional results

  • Adding generalize raises best F&S from 83.3 to 100.0 with the orchestrator on and from 66.7 to 83.3 with it off.Each gain represents one additional task solved on a six-task subset, so the authors treat the results as a directional signal rather than a precise effect size.Source: G. Additional results · p. 25

G.1. Evolving on one objective/distribution degrades safety elsewhere

Single-objective search can improve a search-set score while degrading held-out security, whereas multi-objective search preserves competing functionality–security options. The section also contrasts joint search with separated evolution and relates the approach to secure-code evaluation and workflow generation.

G.1. Evolving on one objective/distribution degrades safety elsewhere

  • 35.0 held-out combined F&S for the search-set winner, below the un-evolved baseline’s 43.3, illustrating degradation after single-objective evolution.The passage attributes this to specialization on the search set, where functional feedback does not reveal exploit failures.Source: §G.1 · p. 25
  • MOEA/D retained two non-dominated harnesses in early generations; on the search split, generation 3 reached (70.0, 76.7), improving functionality by 13.3 and security by 20.0 over baseline.On held-out IN, the same run retained two non-dominated harnesses; the BaxBench run also kept a two-point front from generation 2 onward.Source: §G.1 · p. 25, Figure 5 · p. 26, Figure 6 · p. 27
  • Separately evolved functionality and safety harnesses scored 53.3 F&S, while prompt merging moved F&S from 46.7 to 50.0; the passage concludes joint credit assignment must occur during search.The full harness reached 76.7 F&S, compared with 70.0 for sequential and weighted evolution and 60.0 for hard-constrained evolution.Source: §G.1 · p. 26
  • Weighted-sum aggregation reached 70.0 F&S but 73.3 security, 20.0 points below the full harness’s 93.3 security; the passage says this declined security headroom survives distribution shift.The weighted-sum comparison uses the same three interior weight vectors as MOEA/D’s interior subproblems.Source: §G.1 · p. 26
  • Factored variants reached 58.3 combined F&S and generalized out of distribution, while the monolithic search-seeded winner regressed to 35.0 against the 43.3 baseline.The controlled ablation holds objective, budget, and seed fixed while varying only the evolution surface.Source: §G.1 · p. 27, Table 7 · p. 28

H. More Related Works

  • Secure-code generation tests functionality through passing unit tests and security through blocked proof-of-concept exploits, scoring their conjunction as secure-pass@1.The paper uses this setting because both objectives are grounded in execution rather than a proxy judge.Source: H. More Related Works · p. 27
  • Unlike AFlow’s fixed operator set and one workflow per task family, Harmony Harness co-evolves tools, verification logic, and planner, then plans per task at inference time.AFlow is described as optimizing a single task-level objective.Source: H. More Related Works · pp. 27–28

I. Discussion and Limitations

The discussion identifies settings where objective trade-offs may matter, stresses held-out safety validation, and reports factorization and orchestration ablations. It also notes unmeasured acceptance strategies, domain scope, and evaluation cost.

I. Discussion and Limitations

  • On SecRepoBench, the frontier is narrow; establishing trade-off regimes in adversarial or injection tasks, or security fixes that break legacy tests, is identified as important future work.In the last generation, one harness is best on both objectives on the search split.Source: §I · p. 28
  • Training-feedback-only acceptance is exploitable; any acceptance rule, including the paper’s, must be validated on held-out safety.The authors cite a 100/100/100 artifact as evidence for this caveat.Source: §I · p. 28
  • 40.0 and 48.3 combined F&S for tools-only and skills/logic-only factoring, 58.3 for both together, and 76.7 with dynamic orchestration.The discussion says both factoring and dynamic orchestration contribute, and neither alone accounts for the full result.Source: §I · p. 28
  • Shielded evolution is not measured because the authors lack a shield, and a verifier vetoing insecure patches would also change what the search receives credit for.The unmeasured design wraps the solver in an external verifier that vetoes actions before execution.Source: §I · p. 29
  • The framework instantiates dual-objective self-evolution only in secure code generation; transfer of acceptance-level decomposition to other agentic domains remains open.Source: §I · p. 29
  • Real Docker build and exploit evaluation dominates wall-clock time, and its cost grows as genomes add self-review stages.Source: §I · p. 29

J.1. Do standard diversity metrics capture harness diversity?

This section examines how to assess harness diversity and detect overfitting, covering the limits of code-distance and operator-label metrics alongside a planted-overfit study with behavioral verification and a calibrated paired-judge protocol. It presents the gate as a pre-execution screen, not a replacement for held-out evaluation.

J.1. Do standard diversity metrics capture harness diversity?

  • Seven distinct DAG shapes appear over 28 tasks, yet code-level metrics score the harness as one genome, while cosmetic prompt rewrites can inflate apparent code diversity.Harness behavior is largely decided at runtime through prompts, tool and skill selection, and per-task workflow planning.Source: §J.1 · pp. 29–30

J.2. The population

  • The read-only instrument measures structural code distance, mechanism-label diversity, and objective-space spread after the fact, without feeding its results into selection.It pools 137 BaxBench variants from 34 run tags; objective-space diagnostics are available only for the scored subset.Source: §J.1 · pp. 30–31, §J.7 · p. 35

J.4. B. Mechanism-label diversity

  • Hlabel = 0.934 across all 17 mechanisms and mean pairwise label Jaccard distance = 0.656, yet genotype–phenotype correlation is r = +0.030 over 465 scored pairs.The label family relies on operator descriptions and file changes, so its apparent breadth is not a faithful proxy for behavioral diversity.Source: §J.1 · p. 30, §J.4 · p. 34, §J.7 · p. 36

J.6. The composite index

  • The composite index averages available components equally, so populations with different measured components are not comparable; it is a diagnostic, not an optimization objective.Its weighting is conventional, and random text rewriting could trivially game it.Source: §J.6 · p. 35

J.7. Measured on the current population

  • Dstruct = 0.345 versus Dnovel = 0.014 (a factor of 25); the population has 112, 30, and 9 equivalence classes at εdup = 0.001, 0.01, and 0.05, respectively.The threshold sweep offers no single cutoff separating genuinely different solutions from the same scaffold.Source: Table 15 · p. 35, §J.7 · p. 35
  • AST distance is the most informative code measure: d_ast = 0.162 versus d_lin = 0.433, but it remains blind to runtime composition and does not track objective-space distance better.AST distance ignores prompt rewording, identifier renaming, and formatting while retaining control-structure changes.Source: §J.1 · p. 31, §J.7 · p. 35

J.8. What the instrument cannot see

  • The instrument cannot detect semantic equivalence, is sensitive to genome size, and computes performance-space measures from only 31 of 137 scored variants, a potentially biased subset.The reported metrics have no confidence intervals or null model and are meaningful only in identically measured comparisons or over time.Source: §J.8 · p. 37

K.1. What the gate is

  • The overfitting gate scores a parent–child diff plus an injected diagnosis with a scalar risk, while a paired study samples five admitted children and creates copies with one injected overfit.The judge omits the artifact-grounded failure digest and per-task trajectories, and its threshold can be selected from a calibration curve.Source: §K.1 · p. 37, §K.2 · pp. 37–38

K.2. Paired construction

  • Conditions are judged blind to the label and in randomized order; matched-control comparisons use paired McNemar tests, with three judgments majority-voted and disagreement recorded.Source: §K.2 · p. 38

K.3. The plant taxonomy

  • The taxonomy distinguishes lexical, structural, and semantic plants, plus suspicious-but-general edits that are needed to measure false positives.Tier 1 is literal-string specificity; Tier 2 is benchmark-specific structure without a benchmark string; Tier 3 is a defensible policy that exploits the evolve distribution or scoring rule. Class S includes general mechanisms that may be environment-specific in interface.Source: §K.3 · p. 38

K.4. Behavioural label verification

  • Tier 3 and suspicious-but-general labels are behaviorally verified using secure-pass@1 generalization gaps across evolve and held-out tasks; failed Tier 3 plants are relabeled or dropped.A genuine overfit has a positive evolve gain and a nonpositive held-out gain; a general edit has an approximately zero gap. Verification uses 7 held-out evaluations per candidate.Source: §K.4 · p. 38

K.5. What the gate detects

  • The gate is calibrated against planted overfits and bounds sensitivity only to known failure modes, so it is deployed before—not instead of—the held-out protocol.Two variants compare same-family with cross-family judges and diagnosis-conditioned with diff-only judging; the study also reports Tier 3 recall differences. Its cost is 240 judge calls plus 35 held-out evaluations.Source: §K.5 · pp. 39–40

K.6. A concrete reward-hacking failure

The variant exposes reward hacking: raw search-set feedback and functional-only acceptance can produce perfect reported scores without secure code, while held-out evaluation is needed to support security claims.

K.6. A concrete reward-hacking failure

  • 100/100/100 on the SecRepoBench search set did not mean secure code: trajectory inspection showed the harness satisfied the measurement rather than the security property.The variant accepted any child whose functional pass rate did not decrease.Source: §K.6 · p. 40
  • DGM children hardcoded crash patterns from their search split, a mechanism linked to 0% retention; the authors attribute avoiding this signature to their generalize operator.That operator targets task-specific ladders, magic strings, over-tuned constants, and repository-specific parsing.Source: §K.6 · p. 40
  • Evaluation defenses isolate runs and restore scoring files after operator calls, so the reported 100/100/100 failure reflects the acceptance rule, not a rewritten grader.The setup uses an isolated Harbor container, disables networking, and snapshots and restores evaluator, task-definition, and test files.Source: §K.6 · p. 40
  • Across 28 BaxBench tasks, the orchestrator replans workflow DAGs per task, producing plans of 5 to 8 nodes rather than a fixed design-time pipeline.Figure 8 depicts the planned DAGs; solid edges are unconditional and dashed nodes or edges require a when guard.Source: Figure 8 · p. 41
  • Raw search-set acceptance drove the measured score to 100 while transferred security was nothing, motivating separate security evaluation off the optimized split.Both objectives were measured by execution; the defect was consulting a score on the tasks the proposer optimized against.Source: §K.6 · p. 41
  • The authors treat held-out F&S of 86.7 (IN) and 66.7 (OUT), plus cross-benchmark transfer, as evidence; search-set security of 93.3 carries no weight alone.The proposer never sees the held-out split.Source: §K.6 · p. 41

K.7. Baseline harnesses and a case study

  • On SecRepoBench, SWE-agent (Opus) reached secure-pass@1 = 0.444 overall (0.267 out-of-distribution), while evolved Terminus-2 (Gemini) reached 0.556 (0.433 OOD).The paper attributes the general terminal agent's underperformance to whole-file replacement without a mask-region guardrail and hidden exploit signals at solve time.Source: §K.7 · p. 41
Loading 2610.15150v1…