Source-linked AI summary
ConflictGuide: AutoResearch Improves When Competing Behaviors Are Made Visible
Binqian Xu, Qiran Zou, Xiangbo Shu, Dianbo Liu
TL;DR
Scalar task feedback supports broad early exploration but does not reveal how edits affect competing behaviors, a limitation for AutoResearch when desirable model behaviors are in tension. ConflictGuide adds qualified, model-specific probes in a two-stage search, reducing task and conflict-related errors relative to scalar-only AutoResearch across five model families.
Problem
Scalar task metrics do not reveal how code edits affect competing behaviors, leaving AutoResearch without feedback about behavior-level trade-offs.
Source: §1 · pp. 1–2Method
ConflictGuide-Skill uses a literature-grounded taxonomy and model-specific evidence to specify qualified probes, which guide proposals and marginal-gain retention after scalar-guided exploration.
Source: §1 · p. 2, §5 · p. 10Results
Up to 28% lower task error and 14% lower conflict-related error than scalar-only AutoResearch across five model families, with gains extending to other code agents.
Source: §1 · p. 3Takeaways & Limitations
Competing-behavior feedback redirects search toward conflict-relevant changes and sustains progress beyond scalar-only plateaus.
Source: §5 · p. 10Applicability beyond the tested settings remains to be established, and model-specific probe specification, implementation, and qualification require setup effort not included in a uniform end-to-end cost comparison.
Source: K LIMITATIONS AND FUTURE WORK · p. 64
Abstract
from arXiv · showhide
When designing machine learning models, desirable properties are often in tension: improving one behavior can impair another, so task progress can depend on alleviating the conflict. LLM-based AutoResearch systems, which iteratively edit model code and retain edits based on scalar task-performance feedback, have largely ignored this trade-off. We find that scalar feedback supports broad exploration early in search, but it does not reveal how edits affect competing behaviors. In matched-budget experiments, introducing competing-behavior feedback as task gains diminish increases the share of proposals that improve both behaviors and sustains progress beyond scalar-only plateaus. Obtaining this feedback for a given model requires identifying its competing behaviors and designing probes to measure them. To make competing-behavior feedback actionable, we introduce ConflictGuide. Its reusable ConflictGuide-Skill combines a literature-grounded taxonomy with model-specific evidence to identify competing behaviors and specify probes for a code agent to implement as metrics. Evolution proceeds in two stages: Stage I explores with task feedback; Stage II uses probe feedback to steer proposals toward conflict alleviation and retains marginal-gain edits only when probes indicate sufficient alleviation. Across five diverse model families, ConflictGuide reduces task and conflict-related errors by up to 28% and 14%, respectively, relative to scalar-only AutoResearch, with gains extending to other code agents.
1 INTRODUCTION
ConflictGuide addresses the gap between scalar task metrics and model-design trade-offs by making competing-behavior effects visible. It introduces a two-stage search, first exploring with scalar feedback and then refining with qualified conflict probes.
- Desirable behaviors can conflict, and the sources of tension vary across models, although design or training advances can improve the attainable balance.Source: §1 · p. 1
- Scalar task metrics do not show how code edits affect competing behaviors, so competing-behavior feedback can complement task-specific diagnostics.Source: §1 · pp. 1–2
- With delayed competing-behavior feedback, joint-improving proposals rose from 7.2% to 13.1%, joint-degrading proposals fell from 29.0% to 20.7%, and task progress continued beyond scalar-only plateaus.These matched-budget exploratory studies motivated a two-stage design.Source: §1 · p. 2, Figure 1 · p. 2
- ConflictGuide-Skill uses a literature-grounded taxonomy and model-specific evidence to identify conflicts and specify qualified probes, then Stage II uses probe results to guide proposals and retain marginal-gain edits only when alleviation is sufficient.Source: §1 · p. 2, Figure 2 · p. 3
- Across five model families, ConflictGuide achieves up to 28% lower task error and 14% lower conflict-related error than scalar-only AutoResearch, with gains extending to other code agents.Source: §1 · p. 3
2 RELATED WORK
Related automated model-evolution systems use feedback from code structure, verification, domain requirements, or performance bottlenecks. ConflictGuide instead targets persistent mechanism-level conflicts between competing behaviors.
- Automated evolution systems derive guidance from code structure, verification, domain requirements, training-state diagnostics, or diagnosed performance bottlenecks.Source: §2 · p. 3
- ConflictGuide diagnoses persistent mechanism-level conflicts between competing behaviors and uses quantitative probes to track whether edits alleviate or aggravate them.Source: §2 · p. 3
- Introduced after task gains diminish, conflict-specific evidence guides subsequent proposals and informs edit retention.Source: §2 · p. 3
3 METHOD
ConflictGuide combines a model-specific conflict skill with a two-stage evolution pipeline. It uses scalar feedback for initial exploration, then qualified probes to guide conflict-aware proposals and provide an auxiliary retention path.
3 METHOD
- ConflictGuide pairs scalar-feedback exploration with probe-derived conflict feedback for later proposals and auxiliary retention.Source: §3 · p. 3
3.1 PROBLEM SETUP
- A task-gain scalar indicates whether an edit improves overall performance, but does not reveal its effects on competing behaviors; a conflict is two desirable behaviors coupled through a model mechanism.Probe-improvement components are positive when their corresponding behavior improves.Source: §3.1 · p. 4
3.2 CONFLICTGUIDE-SKILL
- ConflictGuide-Skill screens evidence-supported taxonomy paths, specifies a minimal non-redundant probe set, and qualifies read-only metrics before passing them to the evolution pipeline.Probes compute behavior metrics from observables such as model outputs, internal states, or parameters.Source: §3.2 · p. 4
3.3 TWO-STAGE CONFLICTGUIDE EVOLUTION
- Stage I explores with scalar feedback; after task gains diminish, Stage II exposes probe changes to guide proposals and retains marginal-gain edits only when qualified probes indicate sufficient alleviation.The described tables cover five model–conflict settings and primary probes, including SpecB–FNO task and conflict metrics.Source: §3.3 · pp. 4–5
4 EXPERIMENTS
Experiments compare matched continuations from shared branch points and report consistent task-performance gains across five model families. Ablations and proposal analyses examine the roles of conflict-specific guidance, retention, and probe feedback.
Main Results
- ConflictGuide improves mean task performance over the unmodified reference and AutoResearch across five model families and three search rounds, with simultaneous task and conflict-metric gains in most settings.AutoResearch often plateaus after the shared branch point while ConflictGuide continues improving under matched continuation budgets.Source: Figure 3 · p. 6, §4 · pp. 7–8
Ablation Studies
- Guidance contributes larger gains on Navier–Stokes, while auxiliary retention contributes more on CIFAR-100, indicating complementary roles.Guidance redirects proposals; retention filters noisy task gains and preserves conflict-alleviating edits through a separate path.Source: Table 7 · p. 7, §4 · p. 8
Extension to Other Agents and Frameworks
- Probe-derived feedback improves task and conflict-related metrics across four coding agents and in AIDE under two LLM configurations.These results indicate the benefit is not tied to one code agent or search framework.Source: Figure 4 · p. 8, §4 · p. 9
Hyperparameter Analysis
- With a fixed 200-iteration budget, the 100/100 Stage I and ConflictGuide split achieves the lowest test NRMSE.The authors fixed this default before inspecting test results for models with sufficient evolution capacity.Source: Figure 5 · p. 8, §4 · p. 9
Proposals Analysis
- Probe feedback nearly doubles SNGP’s relief rate from 10.8% to 21.5% and raises GCNII’s from 20.0% to 28.8%.It also shifts proposed edits toward targeted mechanisms, including random-feature-map and kernel-geometry edits for SNGP and neighbor suppression for GCNII.Source: §4 · p. 9, Figure 6 · p. 9
Value of Conflict Feedback
- ConflictGuide outperforms multi-metric and prompt-only alternatives across GCNII metrics and leads on SNGP’s NLL and OOD AUPR.The comparison is for search round 2.Source: §4 · p. 9, Table 8 · p. 9
5 CONCLUSION
ConflictGuide uses a literature-grounded taxonomy and model-specific probes to guide search after scalar-feedback exploration. The conclusion also describes reproducibility resources, while the appendix overview maps additional analyses and documentation.
5 CONCLUSION
- After scalar-guided exploration, ConflictGuide uses probe signals to steer proposals and filters marginal-task-gain edits when conflict alleviation is insufficient.The skill identifies model-specific conflicts and designs measurable probes; the agent implements them before qualification and evolution.Source: §5 · p. 10
- The search loop can be reproduced by extending the public AutoResearch codebase with the paper’s feedback and retention rules; supplementary materials include the skill and experimental code.Appendices C–I document datasets, five base models, settings, and probe-metric definitions.Source: REPRODUCIBILITY STATEMENT · p. 10
APPENDIX OVERVIEW
- The appendix overview lists sections on exploratory studies, related work, skill details, the taxonomy, identifiability, model-specific probes, agent extensions, additional analyses, and limitations.The listed topics span Appendices A–K, with some letters omitted from the overview.Source: APPENDIX OVERVIEW · p. 14
A EXPLORATORY STUDIES AND DESIGN MOTIVATION
The exploratory studies favor introducing competing-behavior feedback after scalar-guided exploration: outset probes narrowed proposal types without improving both reported metrics, while delayed feedback improved both. The paper situates this feedback among automated model-evolution approaches and defines ConflictGuide-Skill to map model evidence to behavior probes.
A EXPLORATORY STUDIES AND DESIGN MOTIVATION
- The studies compare outset feedback and delayed feedback from a shared scalar-only branch under matched proposal or refinement budgets.The proposal audit uses five non-exclusive labels spanning frequency-specific spectral, spatial/local, architectural, fusion, and optimization changes.Source: A EXPLORATORY STUDIES AND DESIGN… · p. 14, Table 9 · p. 14
- From the shared Scalar-only Stage I branch, ConflictGuide reduced Test NRMSE by 9.9% and ND-NMSE by 14.0% versus the matched Scalar-only continuation.The delayed-feedback evaluation reports means and standard deviations over three paired training seeds.Source: Table 11 · p. 15, A EXPLORATORY STUDIES AND DESIGN… · p. 15
- At feedback outset, frequency-specific spectral edits rose from 18.1% to 31.8%, while normalization, activation, or channel-structure edits fell from 59.6% to 31.8%.Scalar-only evolution improved both Test NRMSE and ND-NMSE over the reference, whereas Probe-from-outset improved neither.Source: A EXPLORATORY STUDIES AND DESIGN… · p. 15
B.1 AUTOMATED SCIENTIFIC AGENTS AND MODEL EVOLUTION
- ConflictGuide builds on automated ML experimentation loops in which coding agents iteratively modify and evaluate existing model implementations.Source: §B.1 · p. 16
B.2 FEEDBACK SIGNALS IN AUTOMATED MODEL EVOLUTION
- ConflictGuide adds quantitative Probes for how candidate edits affect competing behaviors, effects that scalar task metrics can obscure.The paper presents this as a complementary, underexplored form of feedback in model evolution.Source: §B.2 · p. 17
B.3 DIAGNOSIS-GUIDED AUTOMATED MODEL EVOLUTION
- ConflictGuide maintains a consistent mechanism-level diagnostic target and recomputes Probes after each edit to track conflict changes across the evolution trajectory.Source: §B.3 · p. 17
C CONFLICTGUIDE-SKILL; C.1 INTERFACE AND WORKFLOW
- ConflictGuide-Skill maps model-specific evidence to a taxonomy mechanism, a competing behavior pair, and Probes; its workflow returns a machine-checkable card or an abstention or rejection record.It records the model context and unresolved assumptions, and conflict selection precedes Probe construction without using evolution outcomes.Source: C CONFLICTGUIDE-SKILL · p. 17, §C.1 · p. 18
C.2 TAXONOMY SCREENING AND CONFLICT INSTANTIATION
- Conflict screening uses pre-evolution evidence to select fixed taxonomy entries and requires independently desirable behaviors with a defensible shared coupling and plausible interference.The skill abstains when evidence is insufficient and rejects unsupported instantiations or Probes that fail its requirements.Source: §C.2 · pp. 18–19
C.3 PROBE DESIGN, IMPLEMENTATION, AND QUALIFICATION
Probe design, implementation, and qualification keep behavior measurements fixed and inspectable while separating read-only diagnostics from model intervention.
C.3 PROBE DESIGN, IMPLEMENTATION, AND QUALIFICATION
- For an instantiated conflict, the skill designs the smallest fixed Probe set that reveals behavior-specific changes not already visible in the scalar task metric.Candidate observables are prioritized from direct ground-truth measurements and existing outputs through internal states, geometry, gradients, data structure, counterfactual inference, and fixed heuristics.Source: §C.3 · p. 19
- Before evolution, the skill fixes evaluation data, preprocessing, partitions, references, perturbations, randomness, aggregation, normalization, and desirable metric directions.Probe definitions cannot depend on future candidate errors, unavailable final-test information, or thresholds chosen after observing outcomes.Source: §C.3 · pp. 19–20
- In IMPLEMENT mode, a code agent adds only read-only diagnostic code to compute fixed Probes, without modifying training or candidate-retention rules.The Probe specification makes intended behavior, additional information, computational cost, and possible failure modes inspectable before exposure to the evolution agent.Source: §C.3 · p. 20, Table 14 · p. 20
- Calibration checks measurement variability under no intended model change, with additional model-specific checks including candidate replay and perturbation controls.Continuation calibration occurs at the Stage-I boundary before probe-guided search.Source: §C.3 · p. 20
C.4 OUTPUTS AND REPRODUCIBILITY
This section documents Conflict Probe Card outputs through a SpecB–FNO worked example and specifies the fixed Root–Axis–Mechanism taxonomy, its selection rules, and mechanism inventory. It also defines taxonomy boundaries and limits selection to taxonomy assignments and evidence across the covered competing-behavior areas.
C.4 OUTPUTS AND REPRODUCIBILITY
- ConflictGuide-Skill produces a human-readable summary and machine-readable Probe Card, separating probe definitions and qualification from the Stage-I/Stage-II evolution policy.The schemas specify model context, behavior instantiation, and Probe Card fields; validation scripts check structural consistency, and examples illustrate the reasoning and output format rather than model-name lookup rules.Source: §C.4 · p. 20, Table 15 · p. 21
C.5 WORKED EXAMPLE: SPECB–FNO
- In SpecB–FNO, the worked example maps dominant- and non-dominant-component accuracy to R2.M2 and measures non-dominant relative NMSE (ZND), with dominant-mode and late-rollout guard metrics.All model-specific gates passed in the initial qualification. Before Stage II, the continuation protocol was calibrated with four paired no-edit comparisons and a four-seed replay; the recorded auxiliary route required Gt ≥2.5 × 10−4 and at least a 1% relative improvement in ZND, with anchor guards, while the direct task route required Gt ≥5 × 10−4.Source: §C.5 · p. 21
D FULL ROOT–AXIS–MECHANISM TAXONOMY
- The fixed taxonomy contains nine Roots, six Axes, 110 Root-specific mechanisms, and three standalone optimization artifacts, with R7 divided into R7a–R7c.The appendix preserves active mechanism identifiers, names, and core diagnostic definitions; repeated examples and implementation instructions remain in the Skill specification.Source: D FULL ROOT–AXIS–MECHANISM TAXONOMY · pp. 21–22, Figure 9 · p. 22
D.1 TAXONOMY CONVENTIONS AND SELECTION RULES
- ConflictGuide-Skill selects a Root, admissible axis, and mechanism using target-model evidence, with a 100×-compute test helping distinguish compute-relievable from persistent conflicts.Partial cases retain and link both Roots; optimization signatures annotate mechanisms, while conflicts that disappear with independent parameters are represented as OPT.M1–OPT.M3 rather than substantive Roots.Source: §D.1 · p. 22, Table 16 · p. 23
D.2 ROOT AND AXIS OVERVIEW
- The axis whitelists exclude population from R2 and R6, routing population shifts to R4, protected-group trade-offs to R7, and population-dependent annotation failures to R8.R6 concerns solution-space coverage rather than demographic coverage.Source: §D.2 · p. 23
D.3 COMPLETE MECHANISM INVENTORY
- The complete inventory lists all 110 active mechanisms and their diagnostic definitions, including R1.M12’s allocation or iteration pathology, which is distinct from a budget shortfall.R1.M12 covers unstable or degenerate allocation mechanisms or iteration operators, rather than insufficient capacity alone.Source: §D.3 · pp. 23–24, Table 17 · p. 24
- R4 distinguishes robustness and shift conflicts, including clean-versus-worst-case accuracy, OOD calibration, worst-group-versus-average performance, and client/domain interference.Source: §D.3 · p. 27
- R5 catalogs continual-learning conflicts including forgetting versus plasticity, shared-parameter overwrite, representation drift, insufficient replay, new-old distribution conflict, and personalization versus global generality.It also includes module proliferation, constraint forgetting, calibration drift, recency bias, negative backward transfer, and capability tax from constraint training.Source: §D.3 · pp. 27–28
- R6 covers exploration deficits and excesses, diversity-fidelity and novelty-constraint conflicts, search breadth versus depth, rare-mode coverage versus common quality, and misidentified exploration targets.Source: §D.3 · p. 28
- R7a captures incompatibilities such as fairness-criteria and preference-aggregation impossibility, while R7b covers privacy-accuracy, privacy-personalization, and fairness-average-performance trade-offs.Source: §D.3 · pp. 28–29
- R7b and R7c include safety-helpfulness, harmlessness-truthfulness, policy-compliance and constraint-stacking conflicts, plus over-refusal, under-refusal, and uneven boundary generalization.Source: §D.3 · p. 29
- R8 catalogs annotation disagreement, label ambiguity, noise-versus-long-tail indistinguishability, sparse supervision, label drift, leakage, and aggregation loss; R9 covers proxy-goal divergence, metric insensitivity, benchmark artifacts, and saturation.Source: §D.3 · p. 29
D.4 OPTIMIZATION ARTIFACTS AND BOUNDARY DECISIONS
- The taxonomy separates capacity limits, prior-signal mismatch, and information loss; distinguishes shift, upstream label, and measurement failures; and assigns sequential overwrite to R5 while routing simultaneous interference to its substantive root.It also separates exploration from population coverage, impossible R7a incompatibilities from R7b frontiers and R7c decision-boundary calibration, and objective aggregation from evaluation-metric aggregation.Source: §D.4 · p. 30
- The selection stage outputs taxonomy assignments and evidence only; later ConflictGuide-Skill stages handle probes, controlled faults, experiments, repairs, training strategies, and code modifications.The active inventory contains 110 R*.M* mechanisms and three OPT.M* artifacts, with no historical ID migration table in the fixed library.Source: §D.4 · p. 30
E IDENTIFIABILITY OF CONFLICT-ALLEVIATING EDITS
The appendix treats conflict alleviation as an inference problem: scalar task gain may not reveal whether both behaviors improved, while probes can provide behavior-specific evidence. It states conditions under which probes improve prediction and operational retention decisions, while qualifying what those results establish.
E IDENTIFIABILITY OF CONFLICT-ALLEVIATING EDITS
- The target event is a judged region of behavior changes that alleviate a conflict, distinct from the probe-based operational criterion computed using model-specific references and guards.One example requires both behaviors to be non-decreasing and at least one to improve.Source: E IDENTIFIABILITY OF… · p. 31
- Equal scalar gains can conceal different behavioral outcomes: one illustrative edit harms a behavior, while another improves both, and distinguishing probes can resolve the ambiguity.The example illustrates scalar aggregation and is not an assumption about the experimental task metrics.Source: E IDENTIFIABILITY OF… · p. 31
- When probes contain information about true conflict alleviation beyond scalar gain and shared context, they strictly reduce optimal squared-error risk and likewise reduce optimal logarithmic-loss risk.The claim concerns feedback information, not whether the coding agent uses it optimally.Source: E IDENTIFIABILITY OF… · pp. 30–31
- When the operational rule requires probe-based qualification and that qualification is uncertain, scalar-only feedback cannot exactly reproduce the rule, whereas probe feedback can.This is a statement about the specified retention rule; agreement with population-level conflict alleviation depends on probe validity.Source: E IDENTIFIABILITY OF… · p. 32
- Stage-II history records probe changes alongside scalar history, providing a conditional information rationale for later proposals but not guaranteeing informative probes, effective agent use, or better task performance.Information gain alone does not guarantee improved task performance.Source: E IDENTIFIABILITY OF… · p. 32
F MODEL-SPECIFIC CONFLICTS AND PROBE METRICS
This appendix documents model-specific competing behaviors, shared mechanisms, and probes designed to expose effects scalar task metrics may hide, including probes for GCNII and TCM–Lite and the distinction between ESN long-horizon transfer error and its probes. Probe construction is frozen before conflict-aware evolution, with test data excluded.
F.1 COMMON NOTATION AND REPORTING PROTOCOL
- Probe components state their desirable directions separately from the scalar task metric, and qualification checks determinism, noise sensitivity, and behavioral discrimination.Probe construction and calibration choices are frozen before conflict-aware evolution; the official test set is excluded from construction and online evaluation.Source: §F.1 · pp. 32–33, F MODEL-SPECIFIC CONFLICTS AND PROBE… · p. 32
F.2 SPECB–FNO
- SpecB–FNO’s dominant-mode accuracy can improve while non-dominant spectral structure remains underfit; ZND measures non-dominant residual energy, with ZD and Zlate guarding other behavior.The dominant/non-dominant partition is fixed from calibration targets; ZND aligns with the formal non-dominant spectral metric, while full-field NRMSE remains the task metric.Source: §F.2 · pp. 33–34
F.3 SNGP
- SNGP’s discriminative compression can improve class separation while distorting pre-GP geometry; Zmar measures vulnerable-tail margins and Zdist measures pair-dependent geometry distortion.Zdist is a mechanism-aligned proxy, not a direct estimate of predictive uncertainty; the probes do not directly optimize OOD performance, and accuracy serves only as a safety guard.Source: §F.3 · pp. 35–36
F.4 ESN
- ESN reservoir dynamics trade past-state sensitivity against state-dependent nonlinear processing, measured by Jacobian-product memory retention and local nonlinear variation probes.Memory retention measures transmission of past-state perturbations, not input recovery; nonlinear variation is a mechanism proxy rather than a direct measure of task-useful nonlinearity.Source: §F.4 · pp. 37–38
- For ESN, Mackey–Glass long-horizon transfer tests whether selected reservoir dynamics improve downstream forecasting, but its prediction error is not identical to either Probe.Appendix G specifies the transfer horizons and implementation settings.Source: §F.4 · p. 39
F.5 GCNII
- GCNII’s Probe measures the extra validation loss from frozen high-disagreement message substitutions, while non-degraded clean NLL guards against retaining uniformly insensitive, low-quality predictors.The conflict is between using informative neighborhood messages and resisting incompatible-message contamination; smaller disagreement sensitivity indicates greater resistance.Source: §F.5 · pp. 39–40
- On Wisconsin, ConflictGuide reduces mean contaminated NLL in all three rounds, but the contamination-induced increase falls versus AutoResearch only in round 2.Formal evaluation uses the same contaminated-minus-clean sensitivity functional as the Probe, but its held-out intervention information cannot affect evolution.Source: §F.5 · p. 40
F.6 TCM–LITE
- TCM–Lite separately probes estimated latent rate and multiscale luminance-gradient distortion, exposing the trade-off between entropy-efficient coding and fine-detail preservation.The scalar rate–distortion objective can mask offsetting changes in rate and detail; the detail Probe does not measure chromatic detail or complete perceptual quality.Source: §F.6 · pp. 40–42
G EXPERIMENTAL DETAILS
The experiments use matched two-stage searches, fixed data and evaluation roles, and held-out evaluation only after source freezing. Reduced-cost model-specific evolution substrates are followed by full-scale retraining for formal evaluation.
G EXPERIMENTAL DETAILS
- Across experiments, held-out metrics are computed only after the terminal source is frozen, and candidates are evaluated with fixed data splits, seeds, perturbations, and resource limits.Candidate sessions are fresh; candidates follow model-specific training procedures rather than inheriting incumbent weights or optimizer state.Source: G EXPERIMENTAL DETAILS · p. 42, §G.1 · p. 43
G.1 COMMON TWO-STAGE EVOLUTION PROTOCOL
- Each round branches from an independently generated Stage-I scalar-only incumbent; its matched Stage-II arms share the source and experimental budget, while ConflictGuide alone receives probe feedback for retention.The scalar-only arm follows the standard AutoResearch retention rule; hidden probes, when computed there, are not exposed or used for retention.Source: §G.1 · p. 43
G.2 MODEL-SPECIFIC EVOLUTION AND EVALUATION PROTOCOLS
- Search uses reduced-cost evolution substrates, while formal evaluation restores full models and benchmark-scale data after source freezing; only source code and architecture transfer, and held-out data never guide selection.Search-time data roles and derived probe assets are fixed before candidate selection; formal sources are retrained from scratch.Source: §G.2 · pp. 43–44, Table 19 · p. 45
G.2.1 SPECB–FNO
- For SpecB–FNO, search uses 200 fitting trajectories and separate 48-example mask and keep-validation panels; formal evaluation retrains the frozen source at full scale and reports official-test task and spectral metrics.The exact search allocations are computational choices, not benchmark splits prescribed by the cited literature.Source: §G.2.1 · pp. 44–45, Table 19 · p. 45
G.2.2 SNGP
- For SNGP, CIFAR-100’s training set is class-stratified into 400/20/30/50-image candidate-training, temperature-calibration, keep-validation, and audit panels; the audit panel is unavailable to proposal generation and retention.Temperature is fitted only on the calibration panel; the allocation is a search-throughput design, not an SNGP benchmark split.Source: §G.2.2 · pp. 45–46
G.2.3 GCNII
The appendix specifies model-specific evolution and evaluation protocols, then defines shared validation, retention, budget, and agent-history controls for matched AutoResearch comparisons. These choices constrain what proposals can change and how candidate edits are retained and evaluated.
G.2.3 GCNII
- GCNII search uses only Chameleon split 0 as a computational proxy, while formal evaluation retrains the frozen source across ten splits of seven datasets.Test data are unavailable online; the proxy choice does not make Chameleon the formal test distribution.Source: §G.2.3 · pp. 46–47
G.2.4 ESN
- ESN evolution uses one sequence to make comparisons inexpensive, then tests frozen sources across 200 sequence–reservoir pairs and a separate Mackey–Glass task.The single-sequence proxy is not evidence of sequence-level generalization; independent formal sequences address that limitation.Source: §G.2.4 · p. 48
G.2.5 TCM–LITE
- TCM–LITE evolution uses disjoint image-source roles and a reduced patch-training proxy, followed by full-image compression testing after source freezing.The hidden audit panel is unavailable to the agent and retention controller, and Kodak outcomes do not guide source selection.Source: §G.2.5 · pp. 48–49
G.3 PROPOSAL VALIDATION AND RETENTION
- ConflictGuide directly retains clear task improvements, while marginal or tied results require the model-specific conflict condition; negative task gains are never retained.The retention thresholds are fixed before Stage II, and parent and candidate comparisons use the same evaluator manifest and resources.Source: §G.3 · pp. 50–51
G.4 EVOLUTION BUDGETS AND PROTOCOL EXCEPTIONS
- Main searches use matched Stage-II proposal opportunities, while ESN uses a shorter 20 + 50 budget because its editable space is narrower.The same Stage-I source and 50 Stage-II opportunities anchor the ESN comparison; unexecuted proposals are not counted.Source: §G.4 · p. 51, Table 22 · p. 51
G.5 AGENT CONFIGURATION, PROMPTS, AND HISTORY
- Paired arms share their pre-branch source and history-construction policy, while histories diverge afterward as a consequence of differing feedback and retention policies.The proposal agent receives feedback specified by its condition and cannot inspect held-out data or protected evaluation pipelines.Source: §G.5 · p. 52
G.6 RELATIONSHIP TO THE UPSTREAM AUTORESEARCH IMPLEMENTATION
- “AutoResearch” denotes a controlled scalar-only arm that shares the ConflictGuide harness and budget but differs in feedback and retention policy.The harness is a controlled, multi-model implementation of the AutoResearch abstraction, not a literal execution of the upstream repository.Source: §G.6 · pp. 52, 54, Table 23 · p. 53
G.7 COMPUTE AND RUNTIME CONTROLS
The appendix separates fixed training and proposal budgets from administrative watchdogs and accounts for evaluator-side Probe computation. It also defines the feedback information exposed in the baseline comparisons.
G.7 COMPUTE AND RUNTIME CONTROLS
- Watchdogs cap process failure time, but fixed model-specific training and proposal budgets—not elapsed runtime—determine search evaluation and retention.Amendments changed only allowed process completion time, not data, training budgets, metrics, or retention rules.Source: §G.7 · p. 54
- Probe feedback adds evaluator-side computation for SNGP and GCNII, while SpecB–FNO, ESN, and TCM–Lite use matched workloads with zero realized evaluator-time difference.For the latter models, Probes are computed but hidden from the scalar-only agent; runtime is not part of retention.Source: §G.7 · p. 54, Table 25 · p. 55
H FEEDBACK BASELINES
- The feedback-baseline comparison tests conventional additional measurements and qualitative trade-off prompting against model-specific, quantitatively measured conflict feedback.Conditions share scalar task feedback but differ in added information and whether it supports conflict-specific retention.Source: H FEEDBACK BASELINES · p. 55, §H.1 · p. 55
H.2 MULTI-METRIC FEEDBACK
The feedback-baseline comparison distinguishes conventional metrics and qualitative trade-off reminders from ConflictGuide’s model-specific Probes. Matched conditions share task feedback and evaluation procedures while varying the additional information and retention checks.
H.2.1 SNGP
- SNGP multi-metric feedback adds expected calibration error and development OOD average precision, but these metrics do not separately measure vulnerable-tail separation and input-relative representation geometry.The metrics therefore differ from ConflictGuide’s Zmar and Zdist probes.Source: §H.2.1 · p. 56
H.2.2 GCNII
- GCNII multi-metric feedback measures generic robustness to random message replacement, unlike ConflictGuide’s targeted high-disagreement intervention and contaminated-minus-clean Probe.Random replacement sources are not selected using labels, anchor-prediction disagreement, or Jensen–Shannon divergence.Source: §H.2.2 · p. 57
H.3 TRADE-OFF-ONLY PROMPTING
- Trade-off-only prompting adds a warning to consider trade-offs but names neither the competing behaviors nor how to measure or assess them.The reminder is the only condition-specific information beyond ordinary task-only feedback.Source: §H.3 · p. 57
H.4 MATCHED COMPARISON PROTOCOL
- Within each model, conditions share the branch source, agent, data, training recipe, editable boundary, history policy, resource limits, and continuation budget.All conditions use task performance as the primary retention criterion, with ConflictGuide adding qualified Probe checks.Source: §H.4 · p. 57
H.5 FORMAL EVALUATION AND INTERPRETATION
- SNGP multi-metric evolution observes CIFAR-10 training examples, so its CIFAR-10 test OOD result is held-out evaluation within an observed domain, not unseen-domain generalization.GCNII’s random-perturbation metrics also differ from the formal incompatible-message stress test, which agents do not see during search.Source: §H.5 · pp. 57–58
- ConflictGuide makes competing behaviors observable through model-specific Probes and uses those measurements in proposal feedback and retention, unlike the two feedback baselines.Multi-metric feedback supplies conventional measures; trade-off-only prompting supplies a qualitative reminder without quantitative evidence.Source: §H.2 · pp. 55–56, §H.3 · p. 57, §H.5 · p. 58
I EXTENSION TO OTHER AGENTS AND FRAMEWORKS
The appendix transfers ConflictGuide feedback to a different proposal agent and to AIDE’s solution-tree search using matched scalar-only comparisons. The transfer settings preserve fixed evaluation procedures while changing feedback and, for AIDE, parent selection.
I EXTENSION TO OTHER AGENTS AND FRAMEWORKS
- Across transfer settings, scalar-only feedback is compared with ConflictGuide under matched continuation budgets, but the experiments do not rank the underlying LLMs.The transfer tests use Kimi K2.7 Code and two AIDE code-generation and execution-review configurations.Source: I EXTENSION TO OTHER AGENTS AND… · p. 58, §I.3 · p. 59
I.1 COMMON SPECB–FNO SETUP
- The shared SpecB–FNO setup probes non-dominant spectral error, with dominant-mode and late-rollout errors serving as guards.Formal evaluation retrains the selected architecture from scratch after discarding proxy-trained weights.Source: §I.1 · pp. 58–59
I.2 PROPOSAL-AGENT TRANSFER: KIMI K2.7 CODE
- Kimi receives a common Stage-I anchor, then runs matched scalar-only and ConflictGuide continuations; only the latter sees qualified probes and uses probe-informed retention.The unchanged controller validates, trains, evaluates, and retains candidates; Kimi does not receive evaluator files or held-out data.Source: §I.2 · p. 59
I.3 SEARCH-FRAMEWORK TRANSFER: AIDE
- AIDE retains its solution-tree operations but starts from the registered SpecB–FNO source as a fixed root shared by both feedback conditions.Candidates pass through a fixed executor, and a separate execution-review model cannot access probe results.Source: §I.3 · p. 59
I.3.1 MATCHED CONTINUATIONS
- Each AIDE configuration branches from a shared 100-expansion checkpoint into scalar-only and ConflictGuide refinements with 100 further expansions each.The checkpoint includes the solution tree, node metadata, candidate sources, and random-number-generator state.Source: §I.3.1 · p. 59
I.3.2 PROBE FEEDBACK AND PARENT SELECTION
- AIDE’s ConflictGuide parent selection can prefer probe-qualified nodes only within the scalar-equivalent frontier; probes cannot promote nodes outside it.In the o4-mini/GPT-4.1-mini run, no refinement node met the full qualification rule, so parent selection did not activate.Source: §I.3.2 · p. 60
I.4 MATCHED FORMAL EVALUATION
- Formal evaluation discards proxy weights and retrains architectures from scratch, using held-out test examples only after training and selection are complete.A shared GroupNorm portability rule is applied to both o4-mini/GPT-4.1-mini continuations before formal training.Source: §I.4 · p. 60
J ADDITIONAL RESULTS AND ANALYSES
- The additional analyses examine behavior trajectories, ambiguity under similar scalar gains, edit-mechanism distributions, and representative edits linked to established design motifs.The passage describes the analyses’ scope, not their findings.Source: J ADDITIONAL RESULTS AND ANALYSES · p. 60
J.1 COMPETING-BEHAVIOR TRAJECTORIES AND SCALAR-GAIN AMBIGUITY
In TCM–Lite’s task-only search, aggregate task improvement coincided with one behavior improving and another worsening, while similar scalar gains were compatible with different behavior outcomes. The section also audits how proposal types and representative edit mechanisms vary across models.
J.1 COMPETING-BEHAVIOR TRAJECTORIES AND SCALAR-GAIN AMBIGUITY
- 3.78 detail thresholds of improvement coincided with rate ending 0.30 rate thresholds below its initial value in TCM–Lite’s task-only Stage I.The trajectories show how an aggregate rate–distortion objective can improve while its components move in opposite directions.Source: §J.1 · p. 60, Figure 10 · p. 61
- 0.892 mean normalized outcome entropy (maximum 1), with no category above 35.8% in any quartile, shows behavior outcomes varied across task-gain quartiles.Three- and five-bin checks yielded mean entropies of 0.891 and 0.888, respectively.Source: Figure 11 · p. 61, §J.1 · p. 61
J.2 CODE-EDIT DISTRIBUTIONS AND MECHANISM EXAMPLES
- SNGP kernel-scale edits rose from 12.7% to 24.6%, while TCM–Lite pointwise-mixing edits fell from 27.0% to 7.4% and residual-refinement edits rose from 29.0% to 34.1%.The audit notes that proposal tags are non-exclusive.Source: §J.2 · p. 61
- These edit distributions are descriptive, not pure feedback-effect estimates, because retained-edit frequencies combine proposal generation and retention.The SNGP AutoResearch audit includes all 15 retained edits but only a nonrandom subset of rejected proposals.Source: §J.2 · p. 62
J.3 REPRESENTATIVE EDITS AND CONNECTIONS TO PRIOR WORK
- SpecB–FNO edits use a learned gain or gated bypass to correct selected retained modes without replacing the full spectral operator.The connection to FNO is representational; the per-mode correction and placement are proposal-specific.Source: §J.3 · p. 63
- Other examples include stabilizing SNGP’s precision matrix before mean-field correction and using receiver-dependent gates in GCNII to attenuate incompatible messages.The SNGP edit is reported as a model-specific numerical mechanism, not a new GP formulation.Source: §J.3 · p. 63
- A TCM–Lite gated residual correction combines depthwise–pointwise factorization with motifs related to depthwise separable convolution and residual learning.Those references do not specify the edit’s codec placement or gate.Source: §J.3 · p. 63
K LIMITATIONS AND FUTURE WORK
ConflictGuide applies when model-specific evidence supports a conflict and probes can measure its competing behaviors. Its tested scope, setup costs, and prespecified timing leave broader applicability and adaptive feedback as open questions.
K LIMITATIONS AND FUTURE WORK
- ConflictGuide applies only when model-specific evidence supports a conflict and its competing behaviors can be measured with probes.ConflictGuide-Skill abstains when evidence is insufficient, and probes are qualified and fixed before evolution.Source: K LIMITATIONS AND FUTURE WORK · p. 63
- Applicability beyond the five tested model families and alternative code-agent settings remains to be established.Model-specific probe specification, implementation, and qualification require setup effort excluded from a uniform end-to-end cost comparison.Source: K LIMITATIONS AND FUTURE WORK · p. 64
- Adaptive timing and strength of probe feedback, interacting conflicts, broader evaluations, and systematic setup-effort measurement remain future work.Source: K LIMITATIONS AND FUTURE WORK · p. 64
Related papers
This paper cites
- Squeeze-and-Excitation Networks2017
- On Calibration of Modern Neural Networks2017
- Fourier Neural Operator for Parametric Partial Differential Equations2020
- Robustness May Be at Odds with Accuracy2018
- Multi-Task Learning as Multi-Objective Optimization2018
- AutoSaddler: Automatic Harness Optimization with Durable Updates from Agent Execution Traces2026