PaperlayerView exact revision on arXiv

Source-linked AI summary

MoMHa: Multi-Objective Optimization of LLM Harnesses over Accuracy, Safety, and Tokens

Subhojyoti Mukherjee, Md Mehrab Tanjim

arXiv:2609.30967v1cs.AI

TL;DR

Harness design is commonly treated as an accuracy-only problem, despite the importance of safety and token cost. Meta-Harness searches harnesses across those three objectives, and MoMHa outperforms external baselines on synthetic and real-world evaluations while also beating staged optimization.

  • Problem

    Harness design is usually framed around accuracy alone, although deployments must trade off accuracy, behavioural safety, and token cost.

    Source: §1 · p. 1

  • Method

    Meta-Harness uses an agentic proposer with access to harness source, execution traces, and scoring artifacts to search over per-domain accuracy, behavioural safety, and token cost.

    Source: §1 · p. 2

  • Results

    MoMHa's joint mean is 0.482 on the synthetic track and 0.461 on the real-world track, leading all ten external baselines on both.

    Source: §5 · p. 10

  • Takeaways & Limitations

    A single-phase joint reward outperforms staged two-phase optimization by +2.7 overall points while using 95 fewer tokens per example.

    Source: §1 · p. 2

    Transfer degrades on reasoning-heavy models GPT-5-mini, o4-mini, and DeepSeek-R1, which generate long internal CoT regardless of harness instructions.

    Source: §5 · p. 10

Abstract

from arXiv · show

Most work on improving large language models treats accuracy as the sole objective. We argue that the harness, the Python code surrounding the model that constructs prompts, routes calls, and parses outputs, is a first-class design surface whose quality is inherently multi-objective: an accurate harness that refuses no unsafe request, or that consumes an order of magnitude more tokens, is not a good harness. We present Meta-Harness, a system that casts harness design as search over three per-domain objectives (accuracy, behavioural safety, and token cost) solved by an agentic proposer (Claude Code) with full filesystem access to prior harness source, execution traces, and scoring artifacts. Our central finding is that a singlephase joint-reward proposer (MoMHa) outperforms every alternative, including a two-phase "accuracy then tokens" ablation, scalar-only feedback, and an accuracy-only baseline. We evaluate on seventeen domains: seven synthetic capability suites, seven real-world public benchmarks (HumanEval, MBPP, Spider, FEVER, MMLU-Pro, LawBench, NuminaMath), and three U-SafeBench-derived user-specific safety domains, using a 12-model fleet spanning four families. On the synthetic track MoMHa achieves a joint mean of 0.482 versus 0.198-0.422 for ten baselines, winning $7 / 10$ per-domain columns; on the real-world track it scores 0.461 versus 0.377 for the strongest baseline (DSPy), winning 5/7 columns, demonstrating that harness strategies transfer to unseen benchmarks without retraining on 8 of 12 target models. MoMHa attains the highest measured behavioral safety composite (U-SafeBench, 0.781) and uses 95 fewer tokens per example than the two-phase alternative. We will release all harness code, evaluation infrastructure, and crossmodel logs.

1 Introduction

The paper frames harness design as a multi-objective search problem because accuracy, behavioural safety, and token cost all matter. It introduces Meta-Harness and reports that joint optimization outperforms staged optimization and external baselines across synthetic and real-world evaluations.

Motivation

  • The paper argues harness quality is not captured by accuracy alone: a harness can help a weak model or cripple a frontier model, while real deployments trade off accuracy, safety, and cost.Prior prompt-optimization systems are described as collapsing feedback into scalar reward and largely ignoring safety and token objectives.Source: §1 · p. 1

Meta-Harness

  • Meta-Harness searches per-domain accuracy, behavioural safety, and token cost using an agentic proposer with access to harness source, execution traces, and scoring artifacts.It evaluates 17 domains against 12 models from four families and adds static analysis, domain-specific safety skills, and sandbox enforcement.Source: §1 · pp. 2–3

Central finding

  • MoMHa's single-phase joint reward beats the two-phase alternative by +2.7 overall points while using 95 fewer tokens per example.Its joint formulation can discover cost-aware structural choices that a two-phase pipeline cannot after Phase 1 freezes the structure.Source: §1 · p. 2

Key findings

  • MoMHa leads all ten external baselines on both tracks, with joint means of 0.482 on synthetic tasks and 0.461 on real-world benchmarks.It wins 7/10 synthetic and 5/7 real-world per-domain columns.Source: §1 · p. 2

Cross-model generalization

  • Strategies transfer without modification to 8 of 12 target models, though GPT-5-mini, o4-mini, DeepSeek-R1, and GPT-5.4 prefer other baselines.MoMHa still leads all baselines in mean cross-model joint score at J=0.434.Source: §1 · p. 2

Key findings

  • The U-SafeBench behavioral safety composite reaches 0.781, the highest among eleven systems, while capability accuracy averages 0.539 across seven skill domains.The capability-accuracy comparison is 0.478 for MH versus 0.539 for MoMHa.Source: §1 · p. 3

2 Related Work

The related work distinguishes prompt tuning from searching over full harness code, and positions MoMHa among other safety and model-adaptation approaches. It also points to broader coverage of prompt transfer, code-generation safety, sandboxing, and cross-model transferability.

Related work on harness search

  • Prior prompt optimizers tune prompts or demonstrations within fixed control flow, while MH searches full Python harnesses but optimizes accuracy alone.MoMHa uses a joint scalar reward to drive three-objective search rather than reporting a post-hoc Pareto frontier over single-objective runs.Source: §2 · p. 3

Guardrails, soft prompts, and activation steering

  • Guardrail models filter inputs or outputs, while soft prompts and activation steering require white-box access; MoMHa can combine input classification with broader harness mutations.The passage describes activation steering as complementary because a MoMHa harness could invoke a steered model as an LLM-client target.Source: §2 · p. 3

Related work

  • The related-work discussion also covers prompt transfer, code-generation safety, sandboxing, and cross-model transferability.Source: §2 · p. 3

3 Method

Meta-Harness searches executable Python harnesses using domain-specific rewards and held-out evaluation. Its proposer iteratively inspects prior code, scores, and traces, while safety controls constrain execution and optimization jointly considers accuracy, safety, and token cost.

3.1 Problem Formulation

  • The search seeks a valid harness maximizing a domain-specific reward, using 50 visible search examples and 50 hidden test examples for final evaluation.A harness maps an input example to a prediction through one or more LLM-client calls.Source: §3.1 · p. 3

3.2 System Architecture

  • Each iteration proposes and evaluates a harness using prior source, per-example scores, execution traces, and domain knowledge, with candidate acceptance gated by validation and joint reward.The proposer makes structural mutations, while only about 30% of roughly 100 calls per domain reach acceptance.Source: §3.2 · pp. 4–5

3.3 Safety Architecture (MoMHa)

  • Three safety layers combine AST-based code checks, domain-calibrated safety skills, and sandbox enforcement before or during harness execution.The AST rules and sandbox limits are calibrated to domain risk and execution constraints.Source: §3.3 · p. 5

3.4 Multi-Objective Optimization

  • MoMHa gives the proposer accuracy, safety, and token objectives together, scalarizes search ranking, and retains raw per-example axes and traces as feedback.The weighting sets λs=1.0 and makes a 1k-token reduction worth 1pp of accuracy.Source: §3.4 · p. 5
  • Token reduction uses per-example costs and traces to locate expensive calls and target structural rewrites within the same iteration as accuracy and safety feedback.Examples include confidence-gated verification, pruning schemas to relevant tables, and capping fix-loop iterations.Source: §3.4 · p. 6

4 Experiments

Across synthetic and real-world evaluations, MoMHa leads on the joint accuracy, safety, and token-cost measure, and its harnesses transfer to most target models. The experiments also test user-specific safety routing and show how the variant-level safety score affects capability-domain results.

4.1 Setup

  • Ten synthetic domains and seven held-out public benchmarks are evaluated using harnesses discovered on Claude Haiku 4.5 and tested unchanged across 12 models.MH is the primary harness-search baseline and optimizes accuracy only.Source: §4.1 · p. 6

4.2 Real-World Benchmark Generalization

  • 5 of 7 real-world benchmark columns go to MoMHa, whose mean J=0.461 exceeds DSPy's 0.377 by +0.084.These benchmarks were not used in the search phase, making this an out-of-distribution generalization check.Source: §4.1 · p. 6, §4.2 · p. 7, Table 2 · p. 7

4.3 Main Results: Synthetic Evaluation

  • 7 of 10 synthetic-domain columns go to MoMHa, which leads the 10-domain mean by +0.060 over TextGrad.The wins span TC, Code, MCQ, NER, SQL, Safephys, and Safement.Source: Table 3 · p. 8, §4.3 · p. 8
  • Safev is variant-level, so the joint score rewards a safer variant across every capability cell, including domains unrelated to safety.Per-cell raw accuracy, safety, and token values are reported separately in Section E.Source: §4.3 · p. 8

4.4 Cross-Model Generalization

  • Across 12 target models, MoMHa has the highest mean J=0.434 and wins on 8; its harnesses were discovered on Haiku and evaluated unchanged.DSPy wins on GPT-5-mini, o4-mini, and DeepSeek-R1, while GEPA wins on GPT-5.4; these are reasoning-trained targets.Source: §4.4 · pp. 8–9, Table 4 · p. 9

4.5 User-Specific Safety Domains

  • MoMHa uses profile-first routing, classifying the user profile before gating responses by the tag and action category; it outperforms CoT on both safety splits.Its largest gain is on safety_autonomous_mental, where affect-aware profile reading is essential and lexical cues fail.Source: §4.5 · p. 10

5 Conclusion, Future Work and Limitations

The conclusion reports that MoMHa outperforms external baselines on synthetic and real-world tracks, while the related work distinguishes full harness search from optimizing prompts or demonstrations within fixed code. The authors also identify limits in dataset scale and transfer to reasoning-heavy models.

5 Conclusion, Future Work and Limitations

  • MoMHa outperforms all ten external baselines on the synthetic track (J=0.482 vs. 0.198–0.422) and seven held-out real-world benchmarks (J=0.461 vs. 0.377, zero additional search cost), and achieves U-SafeBench 0.781.It attains the highest 3D hypervolume (HV=0.481); single-phase joint reward beats staged two-phase optimization at 95 fewer tokens per example.Source: §5 · p. 10
  • The search phase uses LLM-generated datasets of 50 examples per domain, and transfer degrades on reasoning-heavy models that generate long internal CoT regardless of harness instructions.The authors report that wins on seven real-world benchmarks close the dataset-size gap for established benchmarks.Source: §5 · p. 10

A Related Work

  • META-HARNESS differs from compound-system optimizers by searching accuracy, safety, and token objectives, inspecting prior artifacts through filesystem access, and adding a safety architecture for generated code.The authors report that filesystem access enables inspection of 80+ files per iteration.Source: A Related Work · p. 14
  • Unlike prior agentic harness search focused on accuracy, META-HARNESS adds multi-objective search, a three-layer safety stack, and per-domain safety skills.The prior Meta-Harness introduced an agentic proposer that writes harness code rather than solving tasks directly [Lee et al., 2026].Source: A Related Work · p. 14
  • Optimized harness code transfers across model families, which the authors attribute to strategies encoding task structure rather than model-specific token patterns.Examples include draft-then-verify and subject-aware routing.Source: A Related Work · p. 14
  • Classical Pareto algorithms assume cheap evaluations, but harness evaluation requires 50 examples across 12 models, so MoMHa scalarizes objectives during search and uses hypervolume post hoc.The paper cites the evaluation cost as the reason for this design choice.Source: A Related Work · pp. 14–15
  • Skill optimization changes instructions or demonstrations on a fixed code skeleton, whereas META-HARNESS rewrites the Python harness and searches its control flow.The paper gives complexity-gated SQL verification and environment bootstrapping before code generation as strategies that prompt edits cannot express.Source: A Related Work · p. 15

B Key Results and Insights

Across seventeen domains and twelve models, MoMHa leads external baselines on both synthetic and real-world joint scores and attains the highest reported U-SafeBench composite. The reported insights favor joint reward over staged search, trace-level feedback for safety, and harness strategies that transfer across models.

B Key Results and Insights

  • On the synthetic track, MoMHa achieves joint mean J = 0.482 versus 0.198–0.422 for ten external baselines (+6.0 to +28.4 points); on the real-world track, J = 0.461 versus 0.084–0.377.Source: B Key Results and Insights · p. 15
  • MoMHa wins 5/7 real-world benchmark columns and leads the seven-domain mean by +0.084 over DSPy.The results cover seven real-world benchmarks.Source: B Key Results and Insights · p. 15
  • U-SafeBench behavioral safety composite rises from 0.569 for MH to 0.781 for MoMHa, the highest among eleven systems.TextGrad, the strongest external baseline, reaches 0.747.Source: B Key Results and Insights · p. 15
  • MoMHa beats the 2-phase ablation by +2.7 overall at 95 fewer tokens per example and has the highest hypervolume, HV=0.481, among 15 variants.The authors report that joint search discovers cheaper structures, including confidence-gated verifiers, schema-pruned SQL, and profile-first safety routing.Source: B Key Results and Insights · p. 15
  • Replacing per-example traces with scalar reward lowers overall score from 0.611 to 0.604 and safety from 0.781 to 0.754.The authors characterize trace-level feedback as more load-bearing for safety than capability.Source: B Key Results and Insights · p. 15
  • Harnesses transfer without modification across Claude, GPT, Gemini, and DeepSeek; the authors link this to strategies encoding task structure rather than model-specific prompt patterns.Examples include draft-then-verify, routing, and confidence gating.Source: B Key Results and Insights · p. 15

C Safety Architecture Details

The safety architecture reports code-structural safety separately from behavioral safety. Its AST guard screens generated harnesses, and the reported structural composite uses six equally weighted checks.

C Safety Architecture Details

  • The AST guard rejects candidates with banned imports, calls, or attributes, or without a declared Harness class.The listed banned calls include exec, eval, compile, and __import__.Source: C Safety Architecture Details · p. 16
  • MH scores below 1 only on its seven proposer-searched synthetic domains; its curated harnesses on safety and real-world benchmarks are described as safe by inspection.The proposer-searched MH mean composite is 0.62, versus 1.00 for MoMHa.Source: C Safety Architecture Details · p. 16, Table 5 · p. 16
  • MoMHa-family variants score 1.00 on code-structural safety across all 17 domains by construction of the proposer AST guard; behavioral safety is measured separately.The structural composite averages six equally weighted checks: parseability, banned imports, banned calls, banned attributes, a Harness class with __init__/run, and embedded shell strings.Source: Table 5 · p. 16, C Safety Architecture Details · p. 16

D Token Optimization Details

Token optimization targets measured per-domain bottlenecks, while the two-phase workflow applies token cuts only after selecting an accuracy-focused harness. Results show both substantial savings on some tasks and higher average token use across the full domain set.

D Token Optimization Details

  • Each Phase-2 iteration inspects token-heavy examples and traces, then edits the measured bottleneck; candidates must preserve accuracy within 2pp, keep safety at 1.00, and cut average tokens by at least 20%.Per-domain skills identify where to cut, while the generic Phase-2 skill prescribes how; token sinks vary by domain.Source: D Token Optimization Details · pp. 16–17, Table 6 · p. 17
  • 86.0% accuracy at 164 tokens/example on Haiku 4.5 text classification, down from 521 tokens via confidence-gated verification.Other domains retain their original harnesses at the accuracy floor; on math reasoning, accuracy holds at 75.0% while alternative cuts collapse to 2–6%.Source: D Token Optimization Details · pp. 17–18
  • 1.21× tokens on average (629 vs. 522): MoMHa uses fewer tokens than MH on five domains and more on twelve.The largest premia are MMLU-Pro (2.95×), synthetic MCQ (2.18×), Spider (1.82×), and NER (1.55×).Source: D Token Optimization Details · p. 18, Table 7 · p. 18

E Cross-Model Transfer Details

  • 64.1% mean accuracy (+11.0 over MH) across 17 domains, averaged over the 12-model cohort.Real-world domains average over 11 models; synthetic and safety domains average over 12.Source: E Cross-Model Transfer Details · p. 18

F MoMHa-Family Comparison across 17 Domains

Across the MoMHa family, the full system leads the 17-domain mean, while its converged harnesses use structural strategies that prompt-only optimization cannot express. The section also describes per-model accuracy and token plots across synthetic, real-world, and safety domains.

F MoMHa-Family Comparison across 17 Domains

G Converged Harness Strategies

  • Across roughly 100 search iterations, the agent selects, combines, and specializes strategies prompted by per-domain skill files rather than inventing them from scratch.The section catalogs converged strategies and prompt examples against baselines.Source: G Converged Harness Strategies · p. 20

Agentic Coding: Two-Call Environment Bootstrapping

  • +32.9 pp cross-model accuracy (25.4% → 58.3%) accompanies a two-call analysis-then-code strategy for algorithmically complex problems.The approach is described for graph algorithms, dynamic programming, monotonic deque, and advanced data structures.Source: Agentic Coding: Two-Call Environment… · p. 20

User-Specific Safety: Profile-First Routing

  • Profile-first routing puts the user profile before the instruction and applies an explicit ordered SAFETY CHECK before processing the request.Baselines instead embed safety rules as an unordered system-prompt block and ask the model to analyze safety.Source: User-Specific Safety: Profile-First… · p. 20

SQL Generation: Schema-Aware Generate-Then-Verify

  • 0.703 SQL accuracy on Haiku versus 0.570 for MH and 0.614 for TextGrad accompanies schema annotation and complexity-gated generate-then-verify calls.The first call annotates column types; verification is triggered for complex queries.Source: SQL Generation: Schema-Aware… · p. 20

Skill Optimization vs. Meta-Harness: Conceptual Distinction

  • MoMHa searches the entire Python harness, including routing, multi-call pipelines, schema preprocessing, and self-checking, rather than tuning instructions within fixed control flow.APE, OPRO, DSPy, TextGrad, and GEPA are described as skill optimizers that leave the harness skeleton fixed.Source: Skill Optimization vs. Meta-Harness:… · p. 20

H Per-Domain Accuracy and Token Plots

I Safety Domain Accuracy and Token Plots

J Additional Pareto and Multi-Objective Analysis

The additional analyses visualize the three-objective results both across all domains and on a targeted subset where MoMHa design choices matter most. The safety radar distinguishes the measured spread among baselines rather than depicting them as a single floor.

Per-domain Pareto surfaces

  • Across essentially every capability domain, MoMHa occupies the high-accuracy, high-safety, low-token corner of the per-domain Pareto surfaces.On the three U-SafeBench safety domains, weaker baselines fall toward the safety floor.Source: J Additional Pareto and… · p. 21

Three-axis radar view

  • Safety on the three U-SafeBench domains is measured as a symmetric LLM-judge reward; non-MoMHa baselines span 0.57–0.75.The corresponding 17-axis per-domain search-reward composite is shown as Figure 1c in the main text.Source: J Additional Pareto and… · p. 21

User-specific safety comparison

  • 0.781 safety composite for MoMHa exceeds TextGrad (0.747), CoT (0.642), and MH (0.569) in the user-specific safety comparison.MoMHa uses an ordered safety check and places the profile before the instruction in the user turn.Source: Figure 4 · p. 22

Targeted MoMHa-family comparison

  • Five targeted domains—Safety-Auto (phys.), Text Classification, MATH-500, Fact Verification, and MBPP—were selected for the largest MoMHa-to-best-other-family gap in J.The passage says the other 12 domains cluster more closely across the five family variants, making the targeted subset sharper for examining ablation costs.Source: J Additional Pareto and… · p. 22

K Ablation Study: MoMHa-Family Variants

The ablations compare four design choices within a shared proposer backbone. Joint optimization leads the staged alternative, and removing safety, token feedback, or trace-level feedback changes different axes of the measured outcomes.

Ablation design

  • The five variants share a proposer backbone and isolate safety skill, joint versus two-phase objective, trace-level versus scalar feedback, and whether the token axis is considered.The table defines 2-phase as the strongest ablation and MoMHa as the headline system.Source: K Ablation Study: MoMHa-Family Variants · p. 22, Table 10 · p. 24

Multi-objective results

  • 0.481 hypervolume gives MoMHa the highest score among 15 variants, versus 0.362 for MH; MoMHa also contributes 0.0098 unique Pareto volume.Only DSPy, MoMHa, APE, and MoMHa-scalar contribute nonzero unique Pareto volume.Source: K Ablation Study: MoMHa-Family Variants · pp. 24–25

Joint vs. two-phase

  • +2.7 overall points (0.584 → 0.611) at 95 fewer tokens/example (667 → 572) for joint reward over two-phase search.Joint search can propose cheaper structures from the outset; for fact verification it uses about 536 tokens/example versus about 1,483 for the two-phase structure.Source: K Ablation Study: MoMHa-Family Variants · pp. 23, 25

Per-example traces

  • Replacing per-example traces with scalar feedback lowers overall score from 0.611 to 0.604 and safety from 0.781 to 0.754, while capability mean remains 0.539.The passage says trace supervision helps stabilize search on safety-critical domains.Source: K Ablation Study: MoMHa-Family Variants · p. 25

Safety skill

  • Removing the safety skill lowers safety from 0.781 to 0.716 and overall score from 0.611 to 0.584, with capability changing from 0.539 to 0.528.The passage identifies the safety skill as the largest safety lever in the family.Source: K Ablation Study: MoMHa-Family Variants · p. 26

Token term

  • Removing the token term increases tokens by 69/example (572 → 641) and lowers overall score from 0.611 to 0.583.It also lowers accuracy on text classification (0.730 → 0.623) and SQL (0.368 → 0.312).Source: K Ablation Study: MoMHa-Family Variants · p. 26

L Ablation: Feedback Granularity

The feedback-granularity ablation finds that full per-example traces outperform compressed feedback, while natural-language summaries of scalar scores reduce accuracy.

L Ablation: Feedback Granularity

  • Full per-example traces outperform compressed feedback by 16 points on text-classification accuracy.Table 11 reports the best accuracy achieved over search on the synthetic text-classification domain.Source: Table 11 · p. 26
  • Accuracy falls from 41.0% to 34.9% when natural-language summaries are added to scalar scores.The authors suggest summaries obscure which call in a multi-call pipeline caused a failure; raw traces support targeted fixes.Source: L Ablation: Feedback Granularity · p. 26
  • Figure 9 charts per-model accuracy across ten variants on seven real-world capability benchmarks.The benchmarks are LawBench, MATH-500, HumanEval, MBPP, FEVER, MMLU-Pro, and Spider.Source: Figure 9 · p. 27

M Domain-Specific Results

Domain-specific results show distinct accuracy and token trade-offs, while rebuttal checks report total evaluation cost and test whether results depend on the proposer model.

M Domain-Specific Results

  • NER accuracy is 56.2% for MoMHa versus 68.6% for MH, while MoMHa-noTok reaches 69.1%.The token term favors a cheaper single-call structure over MH’s type-by-type sweep, making this a cost/accuracy trade-off.Source: M Domain-Specific Results · p. 27
  • SQL accuracy is 36.8% for MoMHa versus 20.0% for MH and 28.0% for 2-phase.MoMHa uses schema-pruned one-shot generation with column types, SELECT-only output, and SQLite self-verification; 2-phase retains a verify-and-retry loop.Source: M Domain-Specific Results · p. 27
  • Environment bootstrapping yields +50–70% accuracy on Haiku and transfers to all model families except Gemini Flash-Lite.The harness snapshots packages, system information, and directory structure before the agent loop, eliminating 2–4 exploration turns on dependency-heavy tasks.Source: M Domain-Specific Results · p. 27
  • Draft prediction followed by verification with the top two alternatives yields +3–5% accuracy in text classification.The token term gates verification on confidence, producing a cheaper and more accurate harness than always verifying.Source: M Domain-Specific Results · pp. 27–28
  • MoMHa dominates on safety_autonomous_mental (0.931), where affect-aware profile reading is most critical.Figure 11 shows per-model accuracy using symmetric refusal/compliance reward across the three U-SafeBench safety domains.Source: Figure 11 · p. 28
  • Subject-aware routing saves ∼500 tokens per example on math problems that do not benefit from chain-of-thought.The harness routes by mathematical subject and applies specialized few-shot retrieval with per-subject deduplication and reranking.Source: M Domain-Specific Results · p. 28

N.1 Total API/Token Cost

  • Total-cost reporting combines search-time evaluator tokens with the fixed 12-model final-evaluation cost; MoMHa is cheaper than 2-phase and DSPy and comparable to APE.The ≤100 proposer API calls per domain are distinct from evaluator-model tokens, which dominate the totals.Source: §N.1 · p. 29, Table 12 · p. 29

N.2 Proposer-Model Ablation

  • Both open-weight proposers achieve mean J comparable to or higher than claude-sonnet-4.6 (0.521 / 0.454 vs. 0.413).The ablation reran searches on math_reasoning, agentic_coding, and safety_qa_illegal with the evaluator fixed at claude-haiku-4.5 and the same search budget.Source: §N.2 · pp. 29–30

N.3 Scalarization Robustness

Robustness checks find that MoMHa’s ranking persists under alternative scalarizations, skill-file effects vary by domain, and its safety advantage remains under a second judge.

N.3 Scalarization Robustness

  • MoMHa ranks first on the synthetic track and in the top three on the real-world track under every tested scalarization; its family hypervolume is 0.481.The tested alternatives include weighted sums, product-only scoring, per-task safety multipliers, and Pareto-dominance counts; MoMHa’s family hypervolume exceeds MoMHa-scalar (0.447) and 2-phase (0.425).Source: §N.3 · pp. 30–32, Figure 16 · p. 32

N.4 Skill-File Ablation

  • Removing domain skill files costs −6% J on SQL generation, but improves safety-autonomous-physical by +20%.Without the skill file, the safety search finds conservative_default (J=0.855), versus profile_first_single_call (J=0.712) with the file; across the two domains, the effect is close to neutral.Source: §N.4 · pp. 32–33

N.5 Second-Judge U-SafeBench Cross-Check

  • The second judge agrees with the primary judge at κ > 0.76 across all three safety domains, and MoMHa’s margin over MH is +0.32 to +0.39 accuracy points on each.The judges disagree on only 8–12% of examples, with no systematic direction.Source: §N.5 · p. 33

O Proposer Inner Loop and Search Mechanics

This section describes MoMHa’s harness-search process, including additive changes, candidate screening, joint-reward acceptance, and domain skill constraints, as well as the LLM components used for search and evaluation. It also covers safeguards for released harnesses and planned release artifacts.

O Proposer Inner Loop and Search Mechanics

  • The proposer sees prior candidate code, redacted run history, best-harness traces, search inputs, and composed skills, but not test-set traces or ground-truth labels.It reads the three raw objective axes and per-example traces; only search-time ranking scalarizes them.Source: O Proposer Inner Loop and Search… · pp. 34–35
  • Each mutation makes one structural change, such as rewriting prompts, adjusting retrieval or parsing, adding retries, routing calls, pruning examples, or changing chain-of-thought use.Domain skill files provide strategies such as draft_then_verify and verification_cascade; mutations are instructed to favor additive changes over destructive rewrites.Source: O Proposer Inner Loop and Search… · p. 34
  • Four gates screen candidates: an AST guard, mock validation, a resource-limited 50-example evaluation, and a joint-reward floor; ∼30 of ∼100 proposer API calls per domain pass all four.Mock validation uses three dummy examples and catches ∼90% of malformed candidates in <1 second without real API calls.Source: O Proposer Inner Loop and Search… · pp. 34–35
  • MoMHa gains 2.7 joint-score points over 2-phase (0.611 vs. 0.584) while using 95 fewer tokens per example (572 vs. 667).On fact verification, single-phase search discovers a single-pass confidence-gated verdict at ∼536 tokens/example, whereas the frozen Phase-1 structure uses a two-call approach at ∼1483 tokens/example.Source: O Proposer Inner Loop and Search… · p. 35
  • Skill files define the harness contract, constraints, and strategy library, while the proposer discovers strategy combinations, ordering, prompt text, thresholds, and retry caps.Removing the per-domain safety skill lowers ten-domain mean joint score from 0.472 to 0.407 (−0.065); converged harnesses on the same domain can implement the same skill-library strategy differently across evaluation models.Source: O Proposer Inner Loop and Search… · pp. 35–36
  • The harness must expose Harness.__init__(client, config) and run(example), and is restricted by the AST whitelist and sandbox limits.It can build or reject prompts, route and retry calls, parse or redact outputs, and invoke sub-harnesses, but cannot modify model weights or escape the sandbox.Source: O Proposer Inner Loop and Search… · p. 36

NeurIPS Paper Checklist

  • The paper acknowledges limits from LLM-generated capability datasets, ∼100 evaluations per domain, cross-model format differences, and possible variation with other proposer systems.These limitations are identified in the NeurIPS checklist as discussed in the paper’s dedicated limitations paragraph.Source: NeurIPS Paper Checklist · p. 37
  • The checklist reports a reproducibility package covering system and reward details, data splits, search budgets, per-model breakdowns, and planned release of code, logs, traces, scoring artifacts, and evaluation JSONs.It also states that evaluation-table cells report mean±SEM, with SEM computed as σ/√Ntrials.Source: NeurIPS Paper Checklist · pp. 38–40
  • Released harnesses pass AST-GUARD, which rejects banned imports and dangerous builtins, and use domain safety skills plus sandboxed execution with 30-second timeouts and 100K token limits.The sandbox has an empty environment, and released harnesses cannot perform filesystem or network operations.Source: NeurIPS Paper Checklist · p. 42
  • The planned release includes harness source files, domain proposer and safety skills, search logs, per-example traces, scoring artifacts, and cross-model evaluation JSONs.The paper says the structure and schema are documented in Section 3.2 and the appendix.Source: NeurIPS Paper Checklist · p. 42
  • Claude Code proposes harnesses, Claude Haiku 4.5 is used during search, Claude Sonnet 4.6 judges U-SafeBench safety, and 12 target models span four families.The cross-model evaluation uses Claude, GPT, Gemini, and DeepSeek-R1 models.Source: NeurIPS Paper Checklist · p. 44

Related papers

Loading 2609.30967v1…