AutoTrainess: Teaching Language Models to Improve Language Models Autonomously

1 Introduction

• The paper investigates whether LM agents can benefit from human researchers’ accumulated training expertise, empirical intuitions, and artifacts such as sophisticated data-generation pipelines.The motivation is that improving language models still relies on extensive human effort despite the growing capabilities of frontier LM agents.Source: §1 · p. 2
• CLI agents can mispack maximum-length sequences or use incorrect chat templates, producing suboptimal datasets and frequent dataloader exceptions; the authors argue for an ACI that packages human-curated best practices [5].These examples illustrate why direct interaction with a raw Linux CLI can leave even powerful agents struggling to manage training tasks.Source: §1 · p. 2
• AutoTrainess’s training-specialized ACI provides meaningful training heuristics and standard data-preparation, training, and evaluation pipelines, enabling diagnosis and efficient strategy refinement.The authors report stable, iterative improvement with reduced human supervision.Source: §1 · p. 2
• 26.94 average on PostTrainBench with GPT-5.4 (Codex) exceeds CLI-only’s 23.21, while DeepSeek-V4-Flash (OpenCode) scores 19.58 versus CLI-only’s 12.13.On the PostTrainBench Qwen3-4B subset, ablations attribute 3–8 point improvements to the ACIs.Source: §1 · p. 2
• The authors present AutoTrainess as a practical pathway toward scalable LLM self-improvement through reliable autonomous end-to-end post-training.Their contributions include identifying raw CLI limitations and embedding human expertise into agent-environment interaction.Source: §1 · p. 2

AutoTrainess

• AutoTrainHub guides each iteration through planning, data preparation, training, benchmark evaluation, and structured logging, specifying expected artifacts and operational constraints.The agent uses prior experimental evidence to set the next iteration’s goal and records the completed iteration for subsequent decisions.Source: §2 · p. 3

Data processing

• Data processing requires the agent to select data directions, construct task-aligned datasets, and validate them against the benchmark interface before training.Validation can approve samples or return work to construction or selection, distinguishing execution errors from an incorrect data-optimization direction.Source: §2 · p. 3

Training

• Training fixes LlamaFactory [7] as the backend and requires stable, comparable workflows rather than framework switching or custom training loops.Supervised fine-tuning requires full-parameter training, a small validation run before scaling, and an evaluation-ready export; reinforcement learning is used only when recent evaluation evidence supports it, with the reward or feedback signal specified.Source: §2 · p. 3

Evaluation

• Evaluation uses the benchmark’s canonical pipeline and records raw outputs, a summary, 15 randomly selected examples, and classified failure modes.Failures are classified as primarily data, training, or inference/template problems to inform subsequent planning.Source: §2 · p. 3

Logging & Planning

• Logging preserves each iteration’s context, data, method, configuration, evaluation, results, analysis, artifacts, and next action as long-horizon experimental memory.Planning uses prior evaluation results, training outcomes, and workspace state to identify problems, choose an objective and intervention, and define a success criterion.Source: §2 · pp. 3–4

3.1 Evaluation Setup

• PostTrainBench evaluates agents on seven benchmarks spanning mathematical reasoning, function calling, general knowledge, health, and code generation.The benchmarks are run across four base models; CLI-only agents interact with the training environment through internal tools, a system prompt, and shell-launched code.Source: §3.1 · p. 4

3.2 Main Results

• 26.94 overall and 32.60 on the Qwen3-4B subset make AutoTrainess with GPT-5.4 (Codex) the best all-around system; its overall score is 15% above CLI-only.AutoTrainess shows 15%–25% relative increases over CLI-only, with a decrease of less than 5% on the Gemma-4B subset; OpenCode results also show consistent gains.Source: §3.2 · p. 5

3.3 Ablations

• Removing the data interface raises train action failures from 7.2% to 12.7%, while evaluation failures remain near the full-interface rate.Without it, data-loading errors are more likely to surface during debug or validation training; with it, agents more often inspect, clean, format, and construct datasets before training.Source: §3.3 · p. 5
• Removing the train interface raises evaluation action failures from 7.6% to 12.0%, despite a similar train action failure rate of 6.7%.The interface stabilizes checkpoint locations, merged-model outputs, and artifact handoff conventions for downstream evaluation.Source: §3.3 · pp. 5–6
• Removing the eval interface increases evaluation action failures from 7.6% to 22.8%, while train failures remain similar at 6.6% versus 7.2%.The eval interface preserves benchmark invocation conventions, limited-evaluation checks, output paths, and model-serving prerequisites.Source: Figure 3 · p. 5, §3.3 · p. 6
• Removing logging and planning raises evaluation action failures from 7.6% to 19.6%, as agents lose support for maintaining state across iterations.The ablation leads to brittle train-to-evaluation handoffs; its 3.2% train failure rate reflects fewer commands, not improved training reliability.Source: §3.3 · p. 6

3.4 Exploration vs. Exploitation

• 111 train-to-eval handoffs and 7 retained improvements (6.3% yield) give the full interface the highest reported exploration and retained gains.Removing the train interface yields 58 handoffs and 7 improvements (12.1%); removing data yields 95 handoffs and 4 improvements (4.2%). The eval ablation has 70 handoffs and 4 improvements, while logging/planning removal falls to 30 handoffs and 2 improvements; CLI-only records 86 handoffs and 5 improvements.Source: §3.4 · pp. 6–7, Figure 5 · p. 6

3.5 Analysis of Agent Behavior

• Agents move from early benchmark adaptation to mid-run data construction and training, then late-stage diagnosis of remaining failures and full-benchmark evaluation.Baseline-based planning appears 18 times in the first two hours and not in later blocks; data synthesis rises from 19 to 40 occurrences and remains at 36 in the next block. DPO-style training emerges in later blocks, as does self-distillation; examples include benchmark-aligned data for GPQA, format normalization for HumanEval, and targeted data plus LoRA for ArenaHard.Source: §3.5 · pp. 7–8, Figure 7 · p. 8

4 Related Work

• Recent research agents span automated scientific discovery [19], long-horizon literature exploration and trajectory synthesis [20] [21], and domain-focused hypothesis generation and refinement.These systems combine literature understanding, experimentation, and iterative refinement with sustained coordination.Source: §4 · p. 9
• Research-agent benchmarks cover ML experimentation [2], artifact reproduction [23], end-to-end LLM post-training workflows, and broader scientific and engineering tasks [24] [25] [26] [27].The passage identifies PostTrainBench as extending evaluation to end-to-end LLM post-training workflows.Source: §4 · p. 9

5 Conclusion

• AutoTrainess consistently outperforms CLI-only baselines on PostTrainBench, and ablations indicate that its data, training, evaluation, and planning interfaces each contribute to the gains.The ablations support contributions from all four interfaces; no individual effect sizes are specified in the passage.Source: §5 · p. 9
• AutoTrainess externalizes prior human experience as reusable skills for planning and logging, data preparation, training, and evaluation.These interfaces turn post-training into a more structured and reliable long-horizon workflow.Source: §5 · p. 9
• The work presents AutoTrainess as a training-specialized agent-computer interface for autonomous LLM post-training.Source: §5 · p. 9

Eval skill

• With the eval skill, AutoTrainess uses evaluation feedback to create targeted benchmark-shaped data and makes a substantial late-run gain over the no-eval-skill ablation.It targets weaknesses including procedural guidance, preventive and travel summaries, multilingual safety, and structured patient-facing responses; the ablation instead retries parameters and recipes without constructing targeted follow-up supervision.Source: A Case Study on the Effects of… · p. 13

Data skill

• Figure 9 presents an ablation study of the data skill on ArenaHard using Qwen3-4B, but its caption does not specify mixture details or comparative scores.Source: Figure 9 · p. 13

B Detailed Explanation of the Agent Behavior Taxonomy

• The section briefly explains the meaning of the behavior taxonomy used in Section 3.5.Source: B Detailed Explanation of the Agent… · p. 13
• Figure 10 presents an evaluation-skill ablation on HealthBench and Qwen3-4B, but its caption provides no results.Source: Figure 10 · p. 14

B.1 Evaluation Strategy

• Sampled-subset evaluation supports quick comparison, smoke testing, or low-cost iteration.Source: §B.1 · p. 14
• Full evaluation runs the complete benchmark or official evaluation set.Source: §B.1 · p. 14
• Fine-grained evaluation adds statistical analyses from multiple perspectives beyond the final result score.Source: §B.1 · p. 14

B.2 Input Format Strategy

• Prompt alignment matches the benchmark prompt contract or user-message style, while template adjustment changes the model-side template or wrapper.Template changes include chat templates, role/message schemas, or think/non-think switches.Source: §B.2 · p. 14
• Prompt style changes alter wording, system instructions, or answer-behavior instructions without changing the core task family.Source: §B.2 · p. 14

B.3 Output Format Strategy

• Output data may contain direct answers only or include both rationales and final answers.Source: §B.3 · p. 14
• Targeted-format output adopts a benchmark-facing contract or exact structured answer schema, such as “ANSWER: <number>” or “ANSWER: <LETTER>.”Source: §B.3 · p. 14

D1 Benchmark-related Data Source Selection

• D1 selects a data source highly similar to the benchmark task distribution, often seeking benchmark-like public data on Hugging Face.Source: §B.4 · p. 15

D2 Training Dataset Expansion

• D2 expands the dataset with more data from a similar task family, prioritizing volume over difficulty or narrow targeting.Source: §B.4 · p. 15

D3 Data Difficulty Improvement

• D3 deliberately selects harder, expert-level, or challenge-style data to increase task difficulty rather than dataset size.Source: §B.4 · p. 15

D4 Data Synthesis

• D4 adds genuinely new examples created through self-generation or programmatic construction rather than taken directly from a public dataset.Source: §B.4 · p. 15

D5 Data Augmentation

• D5 rewrites or transforms existing examples into derived supervision variants grounded in the same source rows, rather than adding a new source.Source: §B.4 · p. 15

D6 Data Cleanup/Dedup/Rebalance

• D6 cleans, deduplicates, trims, filters, or rebalances the dataset without adding new data.Source: §B.4 · p. 15

B.5 Training Strategy

• Full SFT applies a standard supervised update to all model parameters.This is the full-parameter training strategy.Source: §B.5 · p. 15
• Alternative updates include LoRA-style PEFT, preference objectives such as DPO or ORPO, and self-distillation from accepted or model-generated data.Source: §B.5 · p. 15
• An iteration may continue from a previously trained checkpoint or start from the original base model.Source: §B.5 · p. 15
• Annealing reduces update strength during continued training, for example by using fewer steps or a lower learning rate.Source: §B.5 · p. 15

B.6 Planning Strategy

• Baseline-based planning evaluates starting behavior first, then uses it to choose the initial optimization direction.Source: §B.6 · p. 15
• The agent can diagnose recurring broad errors, including wrong templates, truncation, or repeated format violations.Source: §B.6 · p. 15
• For a proposed change, the agent states or implies a concrete hypothesis and uses the next iteration to test it against evidence.Source: §B.6 · p. 15
• It can also inspect specific failed samples to identify underlying capability gaps or unresolved weaknesses.Source: §B.6 · p. 16

C.1 AGENTS.md

• The workflow restricts benchmark APIs to evaluation and bars benchmark examples or data with unruled-out overlap from training-data construction.It also requires fine-tuning from the task’s exact target base model or checkpoints derived from it.Source: §C.1 · p. 16
• The workflow defines the task, evaluates the base model on the real benchmark, runs local iterations to establish a reliable training/evaluation loop, then expands beyond local optimization using Stage 2 evidence.Fix a stage and retry it; advance only when it has been explored enough to justify moving on.Source: §C.1 · pp. 16–17

C.2 Plan

• Each full iteration follows iteration_plan → data → train → eval → log, while the plan sets an evidence-based goal, proposed changes, focus, success criterion, and downstream guidance.The plan draws on prior experiments, evaluation evidence, and current training and data context.Source: §C.1 · pp. 16–17, §C.2 · p. 18

C.3 Data Process

• Data preparation targets concrete problems, prioritizes the benchmark’s real evaluation interface, favors the smallest effective dataset, and runs selection, construction, and validation in sequence.Validation issues return the workflow to construction or selection, depending on whether the samples or the data direction is at fault.Source: §C.3 · p. 19, Figure 14 · p. 20

C.3.2 Construction

• Construction turns selected data needs and source directions into a training-ready dataset with a concise account of its origins, transformations, target behaviors, format, limitations, and decisions.Source: §C.3.2 · p. 20

C.3.3 Validation

• Validation checks data structure, benchmark alignment, and content quality, then approves it for training or returns it to construction or selection with problems documented.Checks include evaluation-style input and output formats, answer boundaries, duplication, unrealistic synthesis, and leakage or contamination risk.Source: §C.3.3 · pp. 21–22

C.4 Training

• The appendix presents separate instruction materials for the Training and SFT skills, without specifying their procedures in the supplied passages.Source: Figure 17 · p. 23, Figure 18 · p. 24

C.4.2 RL

• The RL skill calls for the minimum valid LlamaFactory workflow only when current evidence supports RL, starting with a small validation run and stating the reward or feedback signal used.Its inputs include recent evaluation evidence for RL, the model to continue from, and the selected setup’s minimum reward, feedback, or RL data requirements.Source: §C.4.2 · p. 24
• RL proceeds from confirming it is justified by recent evaluation evidence through a small validation run, then the intended run and export of `final_model/`.The procedure also requires checking LlamaFactory usability and preparing the minimum reward setup or RL data before validation.Source: §C.4.2 · p. 24

C.4.3 Shared Instruction

• Before training, use LlamaFactory rather than another framework, install it if needed, and verify that the script, model, environment, dependencies, and training inputs are usable.If training fails, debug the actual prerequisite, configuration, data, or resource issue and leave the result ready for evaluation.Source: §C.4.3 · p. 25
• Inspect the LlamaFactory config to align the base model, finetuning stage, evaluation-matched template, prepared dataset, training strength, and cutoff length with the intended run.When an existing workspace config is available, start from it and change only what is needed.Source: §C.4.3 · p. 25
• For missing or broken LlamaFactory, run the install script; for dependency-version-check failures, retry with `DISABLE_VERSION_CHECK=1`.The provided training runner already sets this variable.Source: §C.4.3 · p. 25

C.5 Evaluation

• Evaluate `final_model/` through the benchmark’s real entrypoint, using at least `max(32, ceil(5% of the benchmark))` samples for checkpoint comparisons or iteration choices, or the full benchmark if it has fewer than 32.Smaller runs are only smoke tests for command or runtime validity, not evidence that one checkpoint or approach is better.Source: §C.5 · p. 26
• Record reproducible evaluation outputs, the exact command or config, a metrics summary, 15 random samples, and the main observed failure modes with likely data, training, or inference/template causes.Use the sample summarizer for compatible `inspect_ai` logs; otherwise add the minimum benchmark-specific logging needed to produce the sample summary.Source: §C.5 · p. 26

C.6 Log

• After each completed iteration, append one new experiment-log entry, creating the file if absent and organizing entries by stage when available.Record only the iteration just finished.Source: §C.6 · p. 27
• Log entries use an iteration heading and capture context, status, motivation, and references consulted.The specified status values are completed, failed, or blocked.Source: §C.6 · p. 27

• PostTrainBench results are reported for AIME2025 and ArenaHard Writing.Source: Table 3 · p. 28, Table 4 · p. 29
• Additional PostTrainBench results cover BFCL, GPQA Main, and GSM8K.Source: Table 5 · p. 29, Table 6 · p. 29, Table 7 · p. 30
• The section also reports PostTrainBench results for HealthBench Easy and HumanEval.Source: Table 8 · p. 30, Table 9 · p. 30