PaperlayerView exact revision on arXiv

Source-linked AI summary

LLM Agents for Automated Post-Training via Structured Recipe Optimization

Ruohong Liu, Bohan Tang, Lekang Jiang, Jack Umenberger, Yiwen Guo

arXiv:2610.18572v1cs.AIcs.LG

TL;DR

Predefined recipe spaces may miss target-specific choices, while open-ended agents can leave discoveries implicit and difficult to reuse across trials. SRO constructs an explicit, task-adapted recipe space and searches it using structured trial evidence and multi-objective Bayesian optimization. It achieves the highest average score across evaluated configurations, including a reported 41.1 versus 15.4 comparison in joint post-training.

  • Problem

    Predefined recipe spaces may omit task-specific choices, while open-ended agents leave discoveries implicit, limiting systematic reuse and recombination across trials.

    Source: §1 · p. 2

  • Method

    SRO constructs a task-adapted recipe space of explicit choices and uses structured trial evidence to guide LLM candidate construction and multi-objective Bayesian selection.

    Source: §1 · pp. 2–3

  • Results

    SRO achieves the highest average score across evaluated configurations; on Qwen3-1.7B with GPT-5.4, it reaches 41.1 versus 15.4 for the strongest baseline in joint six-benchmark post-training.

    Source: §1 · p. 3, §5 · p. 10

  • Takeaways & Limitations

    SRO combines task adaptability with systematic optimization under limited experimental budgets.

    Source: §5 · p. 10

    Within each run, SRO can expand choices but keeps searchable component types and their organization fixed.

    Source: A LIMITATIONS · p. 13

Abstract

from arXiv · show

Automated post-training transforms pre-trained large language models (LLMs) into models capable of handling diverse domains by automatically constructing and optimizing a recipe of coupled decisions over data sources, data processing, cross-domain composition, and training configuration under a limited experimental budget. Existing automation either searches predefined recipe spaces that enable systematic optimization but remain fixed rather than adapting their available choices to the target tasks, or uses open-ended agents that discover task-specific choices but leave them implicit in trial-level edits, limiting their systematic reuse and recombination across trials. To fill this gap, we introduce optimization over a task-adapted recipe space, a novel paradigm that can adapt the recipe space itself to the target tasks while representing its choices explicitly for systematic reuse, recombination, and joint optimization. We instantiate this paradigm with Structured Recipe Optimization (SRO), comprising an LLM-based Recipe-Space Designer that constructs the task-adapted recipe space and a Multi-Domain Recipe Optimizer that uses structured trial evidence to guide LLM candidate construction and multi-objective Bayesian trial selection. Extensive experiments show that SRO achieves the highest average score in all evaluated target-model and agent-LLM combinations. In joint post-training across three capability domains and six benchmarks, SRO reaches 41.1 versus 15.4 for the strongest baseline on Qwen3-1.7B with GPT-5.4, and 53.8 versus 40.8 on Qwen3-4B with DeepSeek-V4-Flash. Our code is available at https://github.com/RuohLiuq/SRO.

1 INTRODUCTION

Existing automated post-training methods trade task adaptability against systematic optimization; SRO addresses this by constructing explicit, task-adapted recipe choices and optimizing their combinations. Its introduction reports the highest average score across evaluated configurations.

1 INTRODUCTION

  • Predefined spaces support systematic optimization but may omit task-specific choices, while open-ended agents discover choices that remain implicit and hard to reuse across trials.The paper describes this as a trade-off between task adaptability and optimization effectiveness under limited training budgets.Source: §1 · p. 2
  • Optimization over a task-adapted recipe space represents target-relevant alternatives as explicit, reusable, composable choices for joint search.The space covers data sources, data processing, cross-domain composition, and training configuration, with executable and comparable combinations.Source: §1 · p. 2
  • SRO pairs a Recipe-Space Designer with a Multi-Domain Recipe Optimizer that uses trial evidence to guide candidate construction and multi-objective Bayesian selection.The Designer populates recipe components with task-based choices; the Optimizer organizes trial records by domain for reuse and recombination.Source: §1 · pp. 2–3
  • 41.1 versus 15.4 for the strongest baseline on Qwen3-1.7B with GPT-5.4 in joint post-training across six benchmarks; SRO leads evaluated configurations.The introduction also reports 53.8 versus 40.8 on Qwen3-4B with DeepSeek-V4-Flash.Source: §1 · p. 3

2 RELATED WORKS

Related work distinguishes component-level automation from pipeline-level automation, where methods either search predefined choices or delegate broader design to LLM agents.

2 RELATED WORKS

  • Post-training couples data preparation, supervised fine-tuning, and reinforcement learning, with recipe design requiring substantial expertise and costly training and evaluation.The passage describes data preparation as constructing supervision for subsequent training stages.Source: §2 · p. 3
  • Predefined recipe spaces enable systematic search but can omit task-specific choices; open-ended agents discover such choices but struggle to organize and optimize them across trials.The paper frames these approaches as trading task adaptability against optimization effectiveness.Source: §2 · p. 3
  • Pipeline-level automation broadly includes searching choices specified in advance and delegating broader post-training design to LLM agents.AutoPipe is an example of predefined-space search, while PostTrainBench gives agents autonomy to discover data and design strategies.Source: §2 · p. 3

3 METHODOLOGY

SRO defines automated post-training as constrained recipe search, then constructs and iteratively searches a task-adapted space using structured trial evidence and multi-objective selection.

3.1 TASK DEFINITION

  • The task supplies a pretrained model, target domains and benchmarks, compute environment, and time limit; executable recipes are evaluated across target benchmarks.The task is decomposed into recipe-space design and feedback-driven search, with the selected checkpoint evaluated at the end of the time limit.Source: §3.1 · p. 4

3.2 STRUCTURED RECIPE OPTIMIZATION (SRO)

  • SRO first constructs an initial task-adapted recipe space, then iteratively proposes and selects recipes, executes evaluations, and updates trial history and the space.New data-source choices can expand the space during this iterative workflow.Source: §3.2 · p. 5

3.2.1 RECIPE-SPACE DESIGNER

  • The Recipe-Space Designer defines components, supplies an executable workflow and design skills, and instantiates and validates choices for the initial space.Components cover data sources and processing, cross-domain composition, and training configuration; the workflow validates recipes, enforces compute constraints, and excludes evaluation data from training. Valid generated datasets become data-source choices, and valid combinations populate the initial space.Source: §3.2.1 · p. 5

3.2.2 MULTI-DOMAIN RECIPE OPTIMIZER

  • The optimizer retrieves domain-structured trial records for LLM candidate construction, then uses domain-wise surrogate predictions and Pareto-aware expected hypervolume improvement to select a recipe.Records pair executed recipes with domain scores and include strong and weaker trials, choice statistics, and under-explored choices. After execution and evaluation, the history is updated and validated generated datasets can expand the recipe space.Source: §3.2.2 · pp. 5–6

4 EXPERIMENTS

The experiments compare SRO with automated post-training baselines, test its overall performance, and examine the recipe-space design and search mechanisms. Across settings, they assess benchmark outcomes and how structured choices and trial evidence contribute to search.

4.1 EXPERIMENTAL SETUP

  • Runs use Qwen3-1.7B-Base or Qwen3-4B-Base with GPT-5.4 or DeepSeek-V4-Flash, comparing SRO with PostTrainBench and AutoTrainess under matched task, interface, agent, and compute conditions.Four DeepSeek-V4-Flash ablations remove explicit recipe-space search, LLM proposals, EHVI selection, or selective structured memory; single-domain and multi-domain budgets are 10 and 20 hours, respectively.Source: §4.1 · pp. 6–7

4.2 RQ1: OVERALL EFFECTIVENESS

  • 41.1% vs. 15.4% is SRO’s multi-domain average on Qwen3-1.7B with GPT-5.4; its single-domain GSM8K score is 87.4% vs. 45.7% on Qwen3-4B with GPT-5.4.SRO achieves the highest average score in every evaluated base-model and agent-LLM configuration in both settings.Source: §4.2 · p. 7, Table 2 · p. 8, Table 1 · p. 8
  • SRO matches or exceeds the strongest baseline in 11 of 12 single-domain combinations; its exception is GSM8K on Qwen3-1.7B with DeepSeek-V4-Flash, where AutoTrainess scores 79.1 vs. 68.2.On Qwen3-4B with DeepSeek-V4-Flash, SRO’s domain averages are 45.5 vs. 32.1 in math, 87.0 vs. 66.9 in coding and tool use, and 28.9 vs. 23.4 in science and health; its agent-switch sensitivity is 5.3 points versus 25.6 for PostTrainBench and 14.9 for AutoTrainess.Source: §4.2 · p. 7

4.3 RQ2: RECIPE-SPACE DESIGNER EFFECTIVENESS

  • 45.3 to 34.5 on Qwen3-1.7B and 53.8 to 40.3 on Qwen3-4B are the average-score drops when task-adapted structured recipe search is removed.The variant retains structured trial memory but lets the agent design and execute recipes directly during experimentation.Source: §4.3 · p. 7

4.4 RQ3: MULTI-DOMAIN RECIPE OPTIMIZER EFFECTIVENESS

  • 1.3 and 3.6 points are the score drops from uniform sampling instead of LLM proposals; replacing EHVI with confidence selection reduces scores by 4.2 and 1.9 points.The ablations compare search mechanisms under the same per-model compute budget with DeepSeek-V4-Flash.Source: §4.4 · pp. 8–9, Table 3 · p. 9

4.3 RQ2: RECIPE-SPACE DESIGNER EFFECTIVENESS

  • 52.4, 55.5, and 62.2 are the initial SFT, Rewrite, and Correction HumanEval scores; GRPO reaches 64.0, and configuration tuning reaches 64.6.Validated generated datasets become reusable data-source choices for subsequent recipes.Source: §4.3 · p. 8, Figure 3 · p. 9

4.4 RQ3: MULTI-DOMAIN RECIPE OPTIMIZER EFFECTIVENESS

  • 1.9 and 5.6 points are the score drops when the proposer receives the full raw history instead of selective retrieval, domain-structured summaries, and choice-coverage statistics.The history used for surrogate fitting remains unchanged in this comparison.Source: §4.4 · p. 9

5 CONCLUSION

The paper introduces optimization over task-adapted recipe spaces and instantiates it with SRO, which uses accumulated trial evidence for recipe optimization. SRO consistently outperforms existing automated post-training methods in the reported experiments.

5 CONCLUSION

  • The proposed paradigm adapts the recipe space to target tasks while making its choices explicit for reuse, recombination, and joint optimization.SRO constructs this space and uses accumulated multi-domain trial evidence to guide recipe optimization.Source: §5 · p. 9
  • Figure 4 tracks each method’s best multi-domain average over completed trials under the same compute budget.The caption describes optimization trajectories for SRO and its variants but does not state a winner.Source: Figure 4 · p. 10
  • SRO consistently outperforms existing automated post-training methods, combining task adaptability with systematic optimization under limited experimental budgets.The conclusion also describes its optimization as sample-efficient.Source: §5 · p. 10

A LIMITATIONS

SRO can expand recipe choices during a run, but its searchable components and their organization remain fixed. The paper also does not study carrying experience forward across sequential tasks or base models.

A LIMITATIONS

  • Recipe choices can expand during search, but the searchable components and their hierarchy remain fixed within each run.Schema-level evolution that introduces new components or hierarchies is left to future work.Source: A LIMITATIONS · p. 13
  • SRO is evaluated on a fixed set of target tasks and one base model per run, not on sequential tasks or models with experience carried forward.Reusing choices, trial evidence, or optimization priors across successive problems is proposed as future work.Source: A LIMITATIONS · p. 13

B.1 BENCHMARKS, EVALUATION PROTOCOL, AND SCORING

The evaluation covers separate single-domain tasks and joint multi-domain benchmarks, using common evaluation components across methods. Search-time scores guide optimization and checkpoint selection, while final evaluation uses complete benchmark sets.

B.1 BENCHMARKS, EVALUATION PROTOCOL, AND SCORING

  • Single-domain runs target GSM8K, GPQA Main, or HumanEval; multi-domain runs cover six benchmarks across mathematics, coding and tool use, and science and health.Table 4 summarizes the evaluation sets, output formats, and scoring rules.Source: §B.1 · p. 13, Table 4 · p. 13
  • All compared methods use the same evaluation scripts, prompts, and chat templates, and agents cannot modify these components.HealthBench-Easy uses GPT-5.4-mini as judge with the same configuration across methods.Source: §B.1 · p. 13
  • SRO search-time evaluation uses 50% of each benchmark’s examples, except HealthBench-Easy, which uses 50 samples.For PostTrainBench and AutoTrainess, agents choose search-time evaluation sample sizes through the provided interface.Source: §B.1 · p. 13
  • Final evaluation uses complete target benchmark sets outside the time budget; SRO selects its highest observed search score, while the baselines choose their final models based on target-benchmark evaluation.Multi-domain scores are reported as the mean across domains, each computed by averaging its benchmark scores.Source: §B.1 · p. 13

B.2 COMPUTE BUDGET AND INFRASTRUCTURE

Runs use fixed compute budgets and hardware, with an LLM agent handling recipe-space discovery and optimization. SRO separates design from optimization and uses SFT in all reported multi-domain runs, with a conditional GRPO phase described for single-domain runs.

B.2 COMPUTE BUDGET AND INFRASTRUCTURE

  • Multi-domain runs receive 20 hours and single-domain runs 10 hours, with one GPU per run and equal budgets, task descriptions, interfaces, and hardware across methods.Final independent evaluation is outside the budget; each GPU has 141 GB of memory.Source: §B.2 · pp. 13–14
  • All methods use the same OpenCode scaffold with DeepSeek-V4-Flash or GPT-5.4; in SRO, the same agent LLM handles discovery, recipe proposals, and auxiliary data-generation calls.The auxiliary calls include Rewrite and Correction workers.Source: §B.2 · p. 14
  • An SRO run first designs an executable recipe space, then performs proposal, acquisition, training, and evaluation rounds until the wall-clock budget is reached.The description specifies at most iterative rounds but gives no count.Source: §B.2 · p. 14

B.3 TRAINING CONFIGURATION

  • SRO starts from the provided base model without replacing or merging external weights; both Qwen models use an effective SFT batch size of 32.The 1.7B model uses per-device batch size 8 with accumulation 4, while the 4B model uses batch size 2 with accumulation 16; loss applies to assistant targets only.Source: §B.3 · p. 14
  • Single-domain search enters GRPO after three consecutive SFT trials without Pareto-hypervolume growth or domain-wise historical-best improvement; no reported multi-domain run entered GRPO.GRPO starts from the best SFT checkpoint and continues LoRA training; all Table 2 results therefore come from SFT trials.Source: §B.3 · p. 14
  • For GRPO, the proposer searches learning rate, KL coefficient, group size, training budget, and prompt selector, while the remaining settings are fixed.Short and long budgets mean one and two passes over the prompt pool; the unstable selector keeps prompts whose sampled answers disagree.Source: §B.3 · p. 14

B.4 BAYESIAN OPTIMIZER IMPLEMENTATION

This section details the Bayesian optimizer’s objectives, surrogate models, and trial-selection procedure, along with prompts for recipe-space discovery and recipe proposal. It also reports a predefined-recipe-space experiment that restricts search to training configuration and supplies task-specific data in advance.

B.4 BAYESIAN OPTIMIZER IMPLEMENTATION

  • Bayesian optimization normalizes benchmark scores against base and instruction-tuned references, averages scores within each domain, and fits an independent random forest to each domain objective.Recipes are represented by features for domain budget shares, assembly mode, registered data sources, generation methods, and active-phase training choices; forests use 24 trees of maximum depth 4 and estimate uncertainty from tree disagreement.Source: §B.4 · pp. 14–15
  • For each candidate, 64 sampled outcome vectors estimate EHVI over the observed Pareto front, and the highest-scoring candidate runs next; single-domain runs use scalar improvement.Before five feasible trials, selection uses the proposing LLM’s self-declared priors; cross-domain predictions are treated as independent, so their error correlations are not modeled.Source: §B.4 · p. 15

C.1 DISCOVERY-STAGE TASK INSTRUCTION

  • The discovery instruction asks the LLM to construct and validate a task-adapted recipe space and executable initial recipe, while treating that initial recipe as a handoff example, not the eventual best configuration.The design covers data sources, processing methods, source combinations, and cross-domain composition; training-configuration choices come from the workflow, and evaluation questions or answers must not be used for training.Source: §C.1 · pp. 15–16

C.2 DISCOVERY SKILLS

  • Discovery skills direct benchmark analysis and comparison of dataset examples by relevance, task form, supervision, and source diversity, with evaluation data and derivatives excluded from training roles.Adaptation should use deterministic filtering and faithful field conversion, and verifier-backed roles require genuine source supervision.Source: §C.2 · pp. 16–17
  • Discovery skills define Rewrite and Correction as optional, validated augmentation methods and require a finite recipe space with complementary source choices, positive mixture ratios, and cross-domain modes.The workflow checks source loading, conversion, method validation, and initial-recipe execution; the handoff is revised until accepted.Source: §C.2 · pp. 17–18

C.3 RECIPE-PROPOSER PROMPT

  • The proposer uses the current recipe space and retrieved trial evidence to construct a diverse candidate set spanning exploitation, recombination, and exploration, while distinguishing untested choices from weak observed results.It uses domain-wise results and Pareto evidence for cross-domain proposals; when source-mode evidence is incomplete or confounded, it includes controlled ablations that change one domain at a time.Source: §C.3 · pp. 18–19
  • The recipe-proposer prompt requests up to N candidate recipes as JSON, specifying data sources, optional data generation, SFT hyperparameters, and expected effects, confidence, and risk.It includes examples of two recipe formats: one using interleaved base sources and one using another recipe format.Source: §C.3 · p. 19

C.4 BASELINE TASK PROMPT

  • The baselines retain their original optimization procedures, with only the task objective and evaluation interface adapted for joint post-training across six benchmarks.The overall score is the mean of the six benchmark scores, equivalent to the mean of the three domain scores.Source: §C.4 · p. 19

D ADDITIONAL EXPERIMENTAL RESULTS

  • The additional experimental results section presents recipe spaces and generated data, optimization trajectories, and SRO’s remaining performance limitations.Source: D ADDITIONAL EXPERIMENTAL RESULTS · p. 19

D.1 OPTIMIZATION OVER A PREDEFINED RECIPE SPACE

  • 55.7 GSM8K accuracy (%) for Qwen3-1.7B after GRPO, while Qwen3-4B retains its best SFT model at 74.8.Both models’ best SFT results used a learning rate of 2 × 10−4, three epochs, and LoRA rank 64; search was restricted to training configuration with task-specific data supplied in advance.Source: Table 7 · p. 20, §D.1 · p. 20

D.2 AGENT-CONSTRUCTED RECIPE SPACES

Agent-built recipe spaces vary their data sources and processing choices across model–agent runs, while SRO’s search trajectories outperform the optimizer ablations and expose cross-domain trade-offs.

D.2 AGENT-CONSTRUCTED RECIPE SPACES

  • Across model–agent configurations, agents independently construct recipe spaces with complementary sources and benchmark-specific Rewrite and Correction choices.The Qwen3-1.7B GPT-5.4 run selected some sources different from the DeepSeek run, while Qwen3-4B with DeepSeek also constructed a different space.Source: §D.2 · p. 20
  • Using the independent ARC diagnostic source—not the GPQA evaluation set—the SFT checkpoint missed 97 of 500 examples, and Correction produced verified responses for 96 of those 97 (98.97%).Source: §D.2 · p. 22

D.3 RECIPE OPTIMIZATION TRAJECTORIES

  • SRO finishes with a higher search-time score than all three optimizer ablations on both target models under the same recipe space and compute budget.Figure 5 plots best-so-far six-benchmark means against completed trials; trial counts differ with recipe execution costs, and the comparison excludes the variant without the shared execution workflow.Source: §D.3 · p. 22, Figure 5 · p. 23
  • In several runs, the best recipe was found before search ended, and later trials did not improve the best search-time aggregate score.Table 10 summarizes completed trial counts, selected trials, and search-time and final evaluation scores across four multi-domain runs.Source: §D.3 · p. 22, Table 10 · p. 23
  • Higher aggregate scores can coincide with lower individual benchmark scores, including GPQA or AIME declines in the described runs.The recipes differ in multiple data and training choices, so these observations do not isolate any single component’s effect.Source: §D.3 · p. 23

D.4 LIMITATIONS AND FAILURE CASES

The reported limitations show that strong aggregate results do not ensure strong performance on every capability, and executable recipes can still regress on individual benchmarks.

D.4 LIMITATIONS AND FAILURE CASES

  • Across four multi-domain runs, selected models score 0.0–6.7 on AIME, below Qwen3-1.7B and Qwen3-4B instruction-tuned references at 26.7 and 53.3.With only 30 evaluation problems, each additional correct answer changes the score by about 3.3 points.Source: §D.4 · p. 23
  • Recipe validation checks executability but does not guarantee capability preservation; some Qwen3-1.7B recipes severely regress on BFCL.The observed scores do not determine whether the regressions stem from data composition, training settings, or output-format failures.Source: §D.4 · p. 23

E CONTAMINATION SAFEGUARDS AND RUN HYGIENE

The runs apply common contamination rules, with SRO adding structural checks and reproducibility safeguards and all systems constrained to the assigned base model.

E CONTAMINATION SAFEGUARDS AND RUN HYGIENE

  • All reported runs apply identical safeguards because automated post-training systems have been shown to engage in reward hacking when given the opportunity.The safeguards apply to SRO and both baselines.Source: E CONTAMINATION SAFEGUARDS AND RUN… · p. 24
  • Task-level rules reserve benchmark questions, answers, solutions, tool-call targets, rubrics, and their transformations for evaluation rather than training or generation.The restriction covers training sources, Rewrite inputs, diagnostic sources, generated-data prompts, and examples.Source: E CONTAMINATION SAFEGUARDS AND RUN… · p. 24
  • SRO structurally rejects invalid sources, isolates diagnostic data, verifies generated supervision, and records source and method digests for auditability.Its guards include benchmark-specific adapters and exclusion of specified HealthBench rubric and canary fields.Source: E CONTAMINATION SAFEGUARDS AND RUN… · p. 24
  • Only the assigned base model may be fine-tuned, and final evaluation uses the same benchmark entrypoints used during the run.SRO enforces base-checkpoint training by construction; baselines receive the same rule in their task prompt.Source: E CONTAMINATION SAFEGUARDS AND RUN… · p. 24
Loading 2610.18572v1…