PaperlayerView exact revision on arXiv

Source-linked AI summary

WHALE: A Simple Recipe for Joint Harness-Weight Optimization

Haechan Kim, Yoonho Lee, Gisang Lee, Chelsea Finn, Kangwook Lee

arXiv:2609.00196v1cs.LGcs.AI

TL;DR

Agent performance depends on jointly adapting model weights and executable harnesses, but existing approaches leave the broader harness fixed. WHALE alternates weight updates with harness search and outperforms single-component baselines by 7.67–24.38 percentage points across three domains.

  • Problem

    Agent systems require joint optimization of model weights and executable harnesses because either component can become the performance bottleneck.

  • Method

    WHALE alternates model training under the current harness with executable harness search under the updated model.

  • Results

    Across SearchQA, Math, and chess puzzles, WHALE exceeds weight-only and harness-only baselines by 7.67–24.38 percentage points and FST by 4.15–13.00 points.

  • Takeaways & Limitations

    Performance bottlenecks vary by domain, and small alternating updates outperform one large weight-then-harness pass.

  • Takeaways & Limitations

    The study remains to be scaled to larger models and additional domains.

Abstract

from arXiv · show

Agent performance depends jointly on the model parameters and the executable harness code that manages context and control flow. Optimizing either component in isolation can leave the system bottlenecked by its frozen counterpart: weight updates can change which harness is effective, while harness updates can change which model capabilities are exposed. Existing joint-adaptation methods optimize weights and textual prompts but leave the broader harness fixed. We propose Weight-Harness Alternating LEarning (WHALE), a simple recipe that alternates two phases: updating the model under the current harness, then searching for a better harness under the updated model. We instantiate these two phases with online rejection-sampling fine-tuning and Meta-Harness, respectively. When to switch is a key design choice: to separate real improvements from noise without over-optimizing against a changing counterpart, WHALE uses either fixed phase durations or an adaptive patience rule over training signals. Using Qwen3.5-2B/4B agents across three domains (search question answering, mathematical reasoning, and chess puzzles), WHALE outperforms weight-only, harness-only, and Fast-Slow Training by 4.15-24.38 percentage points in best mean@8 accuracy. Either component can be the bottleneck: harness search matches peak weight-only accuracy with far fewer rollouts in SearchQA, but improves math accuracy only after a weight update. Small interleaved updates also outperform stagewise weight-then-harness optimization in accuracy and rollout cost. The code is available at https://github.com/krafton-ai/WHALE.

1 INTRODUCTION

WHALE addresses shifting bottlenecks between model weights and executable harnesses by alternating model training under the current harness with harness search under the updated model. It evaluates this approach across three agent domains against weight-only, harness-only, and Fast–Slow Training baselines.

  • Motivation: Model weights and executable harnesses jointly determine agent performance, because retrieval, tool use, error handling, and control flow can limit what model capabilities achieve.Stronger weights cannot use evidence a brittle harness fails to retrieve, while better retrieval cannot help a model that cannot synthesize returned evidence.
  • Motivation: Prior joint-adaptation methods target textual prompts, whereas executable agents expose harness code governing tools, observations, error handling, and control flow.Recent LLM-based optimizers make direct search over harness code feasible.
  • Method: Alternation addresses mismatched update timescales: harness search uses long proposal-and-accept cycles, whereas online fine-tuning interleaves rollout collection with gradient updates.Concurrent updates can confound credit assignment or require synchronization barriers.
  • Method: WHALE alternates model training under the current harness with harness search under the updated model.The framework instantiates these phases with online rejection-sampling fine-tuning and Meta-Harness.
  • Experiments: WHALE improves over single-component baselines by 7.67–24.38 percentage points across SearchQA, Math, and chess puzzles.The evaluation compares WHALE with weight-only, harness-only, and Fast–Slow Training, whose implementation uses the same update methods and schedule but searches only system and user prompts.

2 RELATED WORK

Prior work trains multi-turn reasoning and tool use, optimizes prompts and agent workflows around fixed models, and studies alternating updates for coupled components. However, existing joint prompt-weight methods restrict the context-side changes, motivating broader weight-harness adaptation.

  • Training for multi-turn reasoning and tool use: Prior methods train agents to interleave reasoning with actions in search and code-execution environments.Reward-ranked and reinforced self-training methods motivate SFT-style, reward-filtered updates as simpler and more stable alternatives to policy-gradient methods.
  • Optimizing systems around fixed models: System-optimization work expands beyond model weights from natural-language instructions and modular prompt programs to agent architectures and workflows.Recent methods also search executable harnesses and their components through source-code editing, configuration search, and trajectory-based approaches.
  • Joint and alternating optimization: Alternating optimization updates one component while holding others fixed, matching weight-harness adaptation because each component shapes the trajectories used to improve the other.Prior methods jointly adapting prompts and model weights differ in coordination but restrict context-side changes.

3 PRELIMINARIES

WHALE optimizes a coupled model–harness system by alternating weight updates with fixed harnesses and harness search with fixed model parameters. The paper instantiates these phases with online RSFT and Meta-Harness, respectively.

  • System formulation: A model θ and executable harness h jointly induce a trajectory distribution πθ,h over model messages, tool interactions, and environment transitions.The harness includes system instructions, tool schemas, context management, parsing and execution logic, and termination policy.
  • Alternating adaptation: The objective seeks a model–harness pair (θ⋆, h⋆), separating adaptation into weight updates with h fixed and harness search with θ fixed.The phases remain coupled because each update changes the policy on which the other operates.
  • Weight updates: Online RSFT updates model weights using supervised learning only on verifier-accepted rollouts generated by a frozen θold under fixed h.For each prompt batch, it samples G trajectories and retains accepted (x, τ) pairs.
  • Weight updates: RSFT applies minibatch stochastic-gradient updates and synchronizes updated weights to rollout workers before collecting the next trajectory group.The updates ascend the token-normalized supervised log-likelihood over accepted rollouts.
  • Harness search: Meta-Harness performs iterative proposal–evaluation–selection search over executable harnesses with θ fixed, maintaining an archive seeded by hinit.Each evaluated harness contributes an aggregate score, verifier outcomes, and artifacts; the highest-scoring archive candidate becomes accepted.

4 WHALE: WEIGHT-HARNESS ALTERNATING LEARNING

WHALE alternates model-weight updates under the current harness with harness search under the updated model, allowing the two components to co-adapt through their induced behavior. Its RSFT–Meta-Harness instantiation supports fixed phase budgets or adaptive per-phase stopping based on training progress.

  • Alternating procedure: WHALE alternates training model weights under the current harness with searching for a better harness under the updated model.The procedure repeats these two phases from an initial model–harness pair and requires only black-box interfaces for both update procedures.
  • Alternating procedure: WHALE instantiates ModelUpdate with RSFT and HarnessSearch with Meta-Harness, with phase budgets defined by training epochs, search iterations, and proposed candidates.For RSFT, E counts training epochs; for Meta-Harness, I counts search iterations and M counts candidate harnesses per iteration.
  • Co-adaptation: The phases co-adapt because weight updates alter behavior under the current harness, while harness search seeks a harness better matched to the updated model.The coupling operates through the induced trajectory distribution over prompts, tools, observations, and termination policy.
  • Adaptive scheduling: Adaptive WHALE replaces fixed phase budgets with per-phase stopping rules based on sliding-window training reward or archived harness-score improvements after a minimum phase length.The weight phase stops after reward stagnation for a fixed number of steps, while harness search stops after archive-score stagnation for a fixed number of iterations.

5 EXPERIMENTS

The experiments evaluate whether alternating weight updates with full-harness search improves tool-using agents across SearchQA, mathematical reasoning, and chess puzzles. With matched budgets and a fixed schedule, WHALE achieves the highest accuracy across all three domains and improves over the stronger single-component baseline by 7.67–10.05 percentage points.

  • Experimental goals: The experiments test joint adaptation against optimizing weights or harnesses alone, and assess gains beyond prompt-only adaptation across three tool-using domains.Shared initializations and matched update budgets isolate the effect of joint adaptation.
  • Domains and datasets: The evaluation covers SearchQA, mathematical reasoning, and chess puzzles using domain-specific datasets, environments, and tool interactions.SearchQA uses retrieval questions, mathematical reasoning permits Python programs, and chess puzzles use UCI moves with environment-mediated board updates.
  • Harness search: Harness search can modify prompts, tool formatting, tool feedback, stopping criteria, and turn allocation while keeping the model, verifier, datasets, and environments fixed.The verifier supplies a sparse trajectory-level reward.
  • Results: With fixed schedule (E, I) = (0.6, 6), WHALE achieves the highest accuracy in all three domains.Weight-only and harness-only are effectively tied in SearchQA, while weight-only is substantially stronger in mathematical reasoning and chess puzzles.
  • Results: 7.67–10.05 percentage points: WHALE outperforms the stronger single-component baseline despite the baseline ordering changing across domains.WHALE also outperforms weight-only, harness-only, and prompt-restricted FST across all three domains.

6 ANALYSIS

The analysis finds domain-dependent bottlenecks: harness search is rollout-efficient in SearchQA, while mathematical reasoning requires weight updates to unlock effective harness search. Short alternating phases outperform stagewise optimization, and adaptive switching achieves the best SearchQA result while approaching the strongest fixed schedule in Math.

  • Bottleneck regimes: Harness search matches weight-update peak accuracy using 5.79% as many rollouts in SearchQA, but reaches only 0.42% versus 15.42% for weight updates in Math.These comparisons count target-agent rollouts and exclude proposer compute during harness search.
  • Bottleneck regimes: 65.41%: WHALE reaches the highest SearchQA retrieval accuracy, after harness-only raises retrieval from 26.88% to 60.61%.Harness search controls query post-processing and document return, while weight-only peaks at 79.93% for answer extraction versus 57.28% for harness-only.
  • Alternation versus stagewise optimization: Alternating phases dominate stagewise optimization in both final accuracy and rollout cost by limiting over-optimization against a counterpart that later changes.Stagewise reaches 43.02% in SearchQA and 15.63% in Math, while WHALE with (0.6, 6) passes those accuracies after 29% and 49% of stagewise rollouts, respectively.
  • Schedule ablation: 50.09% and 28.33%: the (0.2, 6) schedule is strongest in both SearchQA and Math, exceeding (0.6, 6) by +1.75 and +3.54 percentage points.Longer per-cycle budgets reduce accuracy, while the (0.2, 2) Math run destabilizes after accepting a chance-inflated candidate.
  • Adaptive switching: 52.82%: adaptive WHALE attains the best SearchQA result, +4.48 points over (0.6, 6), +2.73 points over the best hand-tuned schedule, and with 23% fewer rollouts.In Math, adaptive WHALE reaches 26.46%, +1.67 points over (0.6, 6), with 4% fewer rollouts but 1.87 points below the best hand-tuned schedule.
  • Adaptive switching: Adaptive schedules use median phase lengths of 0.24 and 0.29 weight-update epochs with I = 7 harness-search iterations per cycle in both domains.Individual phases extend when training signals keep improving, reaching up to 1.16 epochs and I = 13.

7 CONCLUSION · A ALGORITHMS · A.1 ADAPTIVE WHALE

WHALE jointly alternates weight updates and executable harness search, improving performance across three domains while motivating joint treatment of models and harnesses. Adaptive WHALE replaces fixed phase budgets with patience-based stopping rules for both optimization phases.

  • 7 CONCLUSION: WHALE alternates weight updates with executable harness search and outperforms weight-only and harness-only baselines by +7.67 to +24.38 percentage points across SearchQA, Math, and Chess Puzzles.It also exceeds FST, whose harness search is restricted to prompts, by +4.15 to +13.00 points.
  • 7 CONCLUSION: The analysis identifies harness-dominant and model-dominant regimes across domains.These results support treating the model and its harness as one jointly trained system rather than separately engineered artifacts.
  • 7 CONCLUSION: Future work includes combining WHALE’s black-box operators with other weight-update and harness-search algorithms and scaling experiments to larger models and additional domains.The proposed interfaces permit alternative algorithm instantiations.
  • A ALGORITHMS · A.1 ADAPTIVE WHALE: Adaptive WHALE replaces fixed phase budgets E and I with per-phase stopping rules.The weight phase uses an averaging window W, minimum phase length Tmin, and patience Pw, all measured in rollout steps.
  • A ALGORITHMS · A.1 ADAPTIVE WHALE: During weight updates, the stopping signal is mean verifier reward over the most recent W steps, while the algorithm tracks the phase-best signal and its step.The phase stops only after exceeding Tmin and meeting the patience condition.
  • A ALGORITHMS · A.1 ADAPTIVE WHALE: After each weight phase, adaptive WHALE initializes harness search from the current harness and evaluates proposed candidates under the updated model.Candidate harnesses and their evaluation artifacts accumulate across search iterations.
  • A ALGORITHMS · A.1 ADAPTIVE WHALE: Harness search uses a minimum iteration count Jmin and patience Ph before stopping, then selects the harness with the best evaluated objective.The overall procedure returns the parameter–harness pair with the best accuracy on Dtest.

B EXPERIMENTAL DETAILS … B.3.1 SEARCHQA LLM-AS-A-JUDGE PROMPT

The experiments compare weight-only, harness-only, WHALE, and FST under matched domain-specific budgets, using fixed binary verifiers and harness spaces tailored to SearchQA, mathematical reasoning, and chess puzzles. SearchQA additionally uses a strict GPT-5.4-mini judging rubric that requires a concise, committed answer and emits exactly CORRECT or INCORRECT.

  • B.1 PRINCIPAL SETTINGS: Four systems share domain-specific blocks, while WHALE and FST use both components under per-cycle budgets; weight-only and harness-only receive matching cumulative budgets.The weight-only epoch budgets match WHALE and FST’s cumulative training epochs, and harness-only iteration budgets match their cumulative harness-search iterations.
  • B.1 PRINCIPAL SETTINGS: Each harness-search iteration scores every candidate with one rollout per harness-search training example, costing |Dharness| × M rollouts.The weight update instead runs one SFT epoch over trajectories accepted in the generating step, with no later trajectory revisitation.
  • B.2 HARNESS-SEARCH SPACES AND BASE HARNESS: Harness search can modify retrieval queries, retrieval parameters, and document ranking in SearchQA, while the base harness permits one query and at most 16 assistant turns.The initial harness forwards queries unchanged, returns one passage truncated to 200 tokens, and requires the final answer within <answer> tags.
  • B.2 HARNESS-SEARCH SPACES AND BASE HARNESS: Mathematical-reasoning harness search changes Python extraction, normalization, and execution-output handling, with one program per call and at most 16 assistant turns.Its initial harness requires a final answer in \boxed{...} and has no system prompt.
  • B.2 HARNESS-SEARCH SPACES AND BASE HARNESS: Chess-puzzle harness search can alter board presentation, UCI parsing, retry behavior, feedback, move retention, and validation, but cannot generate or search for moves.Retry budgets are capped at 10 each, and the base harness allows at most 18 assistant turns.
  • B.3 BINARY VERIFIERS: A shared binary verifier R(x, τ) ∈{0, 1} remains fixed with reference answers while model parameters and harnesses change, governing training acceptance, candidate scoring, and evaluation.The verifier is domain-specific and deterministic in its acceptance criterion.
  • B.3 BINARY VERIFIERS: SearchQA parses the <answer>-tagged response, then GPT-5.4-mini at temperature zero and a 256-token cap judges it against references; parsing failure immediately yields R(x, τ) = 0.A CORRECT judgment yields R(x, τ) = 1.
  • B.3.1 SEARCHQA LLM-AS-A-JUDGE PROMPT: The SearchQA judge accepts equivalent surface forms, aliases, contained less-specific values, unit variants, same-feature descriptions, and self-correcting final answers, but rejects contradictions, mismatched entity types, noncommittal alternatives, and evidence passages.It receives the question, reference answers, and prediction, and must respond with exactly one word: CORRECT or INCORRECT.

C ADDITIONAL RESULTS … MODEL RESPONSE

WHALE wins across every reported benchmark and illustrates how harness changes can turn an incorrect SearchQA trajectory into a correct answer. The examples contrast base-harness retrieval failure with WHALE’s perfect result on a specific question.

  • C.1 MAIN COMPARISON: PER-BENCHMARK RESULTS: WHALE is best on all seven SearchQA datasets, both AIME datasets, and the Lichess puzzle test set.Its domain-level gains are therefore not carried by only a subset of benchmarks.
  • C.1 MAIN COMPARISON: PER-BENCHMARK RESULTS: Table 2 reports test mean@8 accuracy (%) across domain-specific dataset subsets, with WHALE using (E, I) = (0.6, 6).
  • C.2 SCHEDULE COMPARISON: BEST POINTS: Table 3 compares schedules using each Figure 4 run’s best test mean@8 accuracy (%) and rollouts consumed up to that point.
  • D TRAJECTORY EXAMPLES: The trajectory examples pair base-model failures under h0 with WHALE successes using (E, I) = (0.6, 6), showing one test item per domain.The displayed trajectories place message roles side by side and trim long spans where marked.
  • D.1 SEARCHQA: 0/8 versus 8/8: on the Octavie Coudreau question, the base pair fails while WHALE answers with the reference answer.
  • D.1 SEARCHQA: The base harness permits two assistant turns and one returned passage, but the first retrieval omits the birthplace and the rollout ends without an <answer> tag.
  • USER PROMPT: The user prompt asks where Henri Coudreau was born, permits search if needed, and requires the final answer only inside <answer>...</answer>.
  • MODEL RESPONSE: The model response identifies the need to find information about Octavie Coudreau, specifically her husband’s birthplace, then recognizes that the first result lacks it.It proposes a more specific search for Henri Coudreau’s birthplace.

TOOL CALL … TOOL CALL

The examples show WHALE improving tool-mediated question answering and mathematical reasoning by enforcing targeted retrieval, answer verification, and executable computation. In the mathematical example, the base harness fails through prolonged prose reasoning, while WHALE obtains the correct result using a code interpreter and structured finalization.

  • TOOL CALL: The search harness permits up to three searches and four assistant turns, requires retrieval, and mandates a single specific answer in <answer>...</answer>.It also requires follow-up searches to target a new, more specific missing fact rather than repeat a query.
  • TOOL CALL: The retrieval sequence first identifies Octavie Coudreau, then searches for Henri Coudreau’s birthplace and returns Sonnac.The final response is formatted as <answer>Sonnac</answer>.
  • D.2 MATHEMATICAL REASONING: The AIME 2025 counting example has reference answer 279: the base pair scores 0/8, whereas WHALE scores 8/8.The base model never calls the code interpreter, reasons for 20,035 characters, reaches the token limit, and produces no extractable boxed answer.
  • USER PROMPT: The mathematical task counts permutations of digits 1–8 divisible by 22 and asks for N − 2025 in boxed format.The prompt states there are 8! = 40320 candidate eight-digit integers.
  • USER PROMPT: WHALE preserves the opening prompt but expands the budget to seven assistant turns and reminds the model to return a settled boxed result after execution.The observed switch from prose to one code call is attributed to weight updates.
  • TOOL CALL: The harness executes Python over all 8! permutations, converts each permutation to an integer, and tests divisibility by 22.The computation finds N = 2304, so 2304 − 2025 = 279.

D.3 CHESS PUZZLES · SYSTEM PROMPT

In the chess-puzzle example, WHALE solves the mate-in-two position perfectly, while the base harness causes prolonged, ambiguous, and budget-exhausting deliberation. The system prompt instead constrains the agent to legal moves from the displayed position and requires exactly one UCI move in a `<move>` tag.

  • D.3 CHESS PUZZLES: WHALE scores 8/8 on the mate-in-two puzzle, compared with 0/8 for the base pair.The example solution is 1...Qb1+ 2.Bd1 Qxd1#.
  • D.3 CHESS PUZZLES: Under the base harness h0, the model reaches the 8,129-token cap without committing to a move or emitting a <move> tag.The harness provides a FEN string, ASCII board, flat legal-move list, and a one-move request.
  • D.3 CHESS PUZZLES: 2,025 of 2,048 base rollouts stop at the same cap, averaging 8,112 assistant tokens versus 764 under WHALE.The retry offered by the harness cannot execute because the budget is exhausted.
  • SYSTEM PROMPT: The system prompt instructs the agent to choose one legal move at a time using only the shown board position and legal moves.It explicitly prohibits hidden engine analysis and external knowledge.
  • SYSTEM PROMPT: The prompt supplies the puzzle’s side to move, FEN, ASCII board, move number, and legal moves with UCI and SAN notation.The listed position has Black to move on puzzle move number 1.
  • SYSTEM PROMPT: The required output is exactly one move formatted as <move>uci</move>, without explanation.The prompt gives <move>e2e4</move> as the formatting example.
  • SYSTEM PROMPT: The base model’s internal reasoning shows uncertainty while parsing the FEN and identifying the black pieces.It repeatedly revises interpretations of the board before considering candidate moves.

ENVIRONMENT RESPONSE

The environment rejected an ambiguous multi-move response, while the searched WHALE harness enforced exact move formatting and enabled a forcing chess continuation from e4b1 to b1d1 mate.

  • Verifier feedback: The verifier rejected the response because it could not parse exactly one UCI move from an ambiguous multi-move output.It instructed the agent to return exactly one move, such as <move>e2e4</move>, without explanation.
  • Harness behavior: The searched WHALE harness parses and groups legal moves, annotates tactical properties, and restates accepted moves with the opponent’s reply.This design avoids carrying the full position through reasoning and fixes how the move is read from the response.
  • First move: e4b1 was selected as Black’s only available check from the supplied legal-move list.The response committed to <move>e4b1</move>.
  • Continuation: After the verifier marked e4b1 correct and reported the reply e2d1, the next position offered b1d1 as a mate-and-capture move.The subsequent reasoning and response identified <move>b1d1</move> as immediate checkmate.
Loading 2609.00196v1…