Language Models Learn to Improve Language Models

Motivation

• Experimental records can guide proposal decisions through in-context learning and outcome prediction, motivating parameter learning to improve generation beyond in-context use alone.The paper frames this as a way to learn from experimental feedback.Source: §1 · p. 2

Challenges

• Real-execution feedback is costly: evaluating 25,600 proposals at the cited per-run cost would consume approximately 1,152,000 GPU-hours and cost over $16 million.The example assumes 100 RL updates, 32 training states per update, and 8 proposals per state; one 486M-parameter model run on 8B tokens takes approximately 45 B200 GPU-hours.Source: §1 · p. 2
• Surrogate predictions on past experiments may become unreliable as the policy changes its proposals, allowing RL to favor overestimated proposals without corresponding real gains.The feedback must remain grounded in real outcomes on proposals generated by the current LLM.Source: §1 · p. 2

Approach

• A frozen research world model predicts proposal gains from real experimental records in context and supplies rewards for reinforcement learning.On-policy calibration periodically executes highly ranked current-policy proposals and adds their measured outcomes to the RWM’s context.Source: §1 · p. 2

Contributions

• LM2LM enables Qwen3.8-27B to outperform frontier models and past-record baselines at lower data-generation cost, while calibration mitigates reward overestimation and preserves ranking quality.The analysis contrasts calibrated feedback with a fixed-record RWM, which eventually performs worse than random.Source: §1 · p. 2

AutoResearch with LLM Agents

• AutoResearch iterates through idea generation, experimental implementation, execution, and analysis, with early work demonstrating research ideas and end-to-end experiments.Benchmarks also evaluate machine-learning engineering and AI R&D capabilities.Source: §2 · p. 2
• Some methods improve candidate solutions with execution feedback while keeping the underlying agent fixed, whereas others train the agent on execution feedback.The latter includes work on machine-learning engineering and language-model development.Source: §2 · p. 2
• Prior methods that train agents on execution feedback rely on costly real execution for training signals; this paper instead uses a world model to predict outcomes with fewer real executions.The contrast is between real-execution training signals and predicted experimental outcomes.Source: §2 · p. 2

3.1 Preliminaries

• Each task specifies a model family, size, pretraining budget, and reference configuration; a proposer generates proposals from the task and experiment history, and a builder implements them.The training objective is to improve the proposer’s decision-making so its proposals have high experimental gain.Source: §3.1 · p. 3

3.2 Scalable Feedback: Research World Model as the Verifier

• The frozen RWM predicts gains from experimental records, avoiding a model-training run for each proposal; rewards normalize and clip gains and account for invalid, repeated, and less-diverse proposals.The reward design uses task-specific scaling, ranks repeated proposals below valid ones, and rewards less-common mechanism directions.Source: §3.2 · p. 4
• The RWM also expands the training-state pool by assigning predicted outcomes to sampled task–history pairs, reducing reliance on costly real outcomes.The record pool starts with real experimental records and is periodically expanded with state–proposal pairs sampled during RL.Source: §3.2 · p. 4

3.3 Reliable Feedback: On-Policy Calibration of the RWM

• Because policy optimization can favor highly rated proposals poorly covered by initial records, LM2LM periodically executes high-rated current-policy proposals and adds their measured outcomes to the RWM context.Calibration updates the in-context records without retraining and requires additional real executions independent of the number of proposals sampled during RL.Source: §3.3 · p. 5

4.1 Experimental Setup

• The experiments span four autoregressive and four diffusion model families, with 40 training tasks, 10 held-out tasks, and real-execution evaluation on held-out research states.Evaluation generates four proposals per state and measures gains using BPB for AR models and NLL for diffusion models.Source: §4.1 · p. 5

4.2 Main Comparison with Baselines

• 17.06 Avg@4 and 23.40 Best@4 on AR, plus 7.10 Avg@4 and 9.93 Best@4 on diffusion, make LM2LM the top baseline on both task types.The comparison uses real-execution evaluations on held-out research states.Source: Table 1 · p. 6, §4.2 · p. 6
• At 200 real runs, LM2LM achieves higher gains than 300-record ICL at approximately 33% lower data-generation cost and time.At step 96, its Avg@4 is 17.06 on AR and 7.10 on diffusion; total cost including RL training also remains below 300-record ICL.Source: Figure 3 · p. 6, §4.2 · p. 7

4.3 Generalization to Unseen Model Families

• Without Qwen3 records in training or RWM grounding, LM2LM reaches 53.72 Avg@4 and 86.69 Best@4, versus 21.63 and 25.27 for GPT-6-sol-ICL.Base-ICL performs similarly to Base, while GPT-6-sol-ICL only slightly improves over GPT-6-sol using out-of-family records.Source: §4.3 · p. 7

4.4 Ablation Studies

• With 200 real runs, calibration beats adding random records, 17.06 vs. 14.76 Avg@4; random calibration lowers AR Avg@4/Best@4 from 17.06/23.40 to 10.84/18.60.LM2LM also exceeds static-record variants on AR Avg@4 and Best@4, outperforming them in Avg@4 on 74.2% and 70.0% of held-out states.Source: §4.4 · p. 7, Figure 4 · p. 7

4.5 In-depth Dissection of LM2LM

• LM2LM’s advantage cosine of 0.50–0.71 and pairwise accuracy of 0.61–0.69 exceed chance; the static RWM’s metrics instead fall below chance as training progresses.At step 128, static RWM advantage cosine is −0.42 and pairwise accuracy is 0.37, while LM2LM maintains pairwise accuracy of 0.61.Source: §4.5 · p. 8, Figure 5 · p. 8
• Calibration keeps predicted rewards closer to real ones: LM2LM’s reward gap falls from 0.85 at step 96 to 0.17 at step 128, while static RWM reaches about 2.0 at step 96.The authors attribute this to calibration correcting overestimated proposals that RL might otherwise reinforce.Source: §4.5 · p. 8

• LM2LM trains LLMs with RL to generate LM-development proposals, using a frozen LLM grounded in real experimental records and on-policy calibration to maintain feedback reliability.Source: §5 · p. 8
• Across autoregressive and diffusion environments, LM2LM enables Qwen3.8-27B to outperform frontier models and record-based baselines at lower data-generation cost.The conclusion characterizes RWM-guided RL as a promising paradigm for learning research judgment from experimental feedback.Source: §5 · p. 8

• The study evaluates proposal-level improvements in LM development, not end-to-end research campaigns.Source: §6 · p. 9

7 Ethics Statement

• The authors recommend human oversight, independent evaluation, and safeguards because predicted rewards and measured gains do not guarantee safety or broader usefulness.The framework is evaluated in controlled training environments; the caution applies especially to use with more capable models or in higher-stakes settings.Source: §7 · p. 9

A.1 World Models for LLM Agents.

• Unlike WMRL’s per-step rescaling, LM2LM grounds its frozen world model in real records and adds outcomes from selected top-rated proposals.This contrast describes how the two approaches use real experimental outcomes to improve or adjust world-model feedback.Source: §A.1 · p. 13

A.2 Reinforcement Learning for LLMs

• LM2LM uses RWM-predicted gains as RL rewards and periodically calibrates the RWM with selected real executions, avoiding real execution at every training step.LM development rewards require costly model-training experiments, unlike inexpensive checks often used in RL with verifiable rewards.Source: §A.2 · p. 13

A.3 Self-Evolving LLMs and Agents

• Unlike self-evolving approaches relying on cheap feedback, LM2LM learns from RWM-predicted gains and calibrates with real executions for costly LM-development experiments.The stated target is the ability to improve LM development, which the passage identifies as ultimately required for recursive self-improvement.Source: §A.3 · p. 13

B.1 Reward Design

• The task-specific reward scale is the maximum of reference-run variability and scaled median absolute deviation of prior RWM-predicted gains.The scale is recomputed at each training-segment start and fixed within that segment; proposal validity and repetition also affect rewards.Source: §B.1 · p. 13

B.3 Algorithm of LM2LM

• Algorithm 1 initializes real records, trains with RWM rewards, periodically selects highly rated current-policy proposals, and adds their measured outcomes to RWM records.Selected proposals are implemented and executed to measure real gains before their outcomes are incorporated.Source: §B.1 · p. 14

B.2 Synthetic Training-State Expansion

• After each rollout, LM2LM adds proposals and their RWM-predicted gains to a pool used to resample histories for the next rollout.The combined pool contains real and predicted records.Source: §B.2 · p. 14

C.1 Model Families

• The evaluation spans autoregressive and diffusion model families; MDM is a training-only family without a held-out task.Tasks vary by model family, size, and training budget, with reference configurations and held-out tasks detailed in the appendix.Source: §C.1 · pp. 14–15

Mechanism Library

• The mechanism library covers architecture, optimization, and training objectives; each reviewed card specifies a hypothesis, expected effects, evidence, and valid family-specific configurations.The AR library contains 40 accepted cards, and the library defines the action space shared by all proposers.Source: §C.2 · p. 15

Action Space

• Proposals can combine mechanisms and modify valid recipe settings, while model size, budget, data, tokenizer, evaluation, seeds, and hardware remain fixed.Numerical and categorical settings may be proposed within the allowed ranges and implemented by the builder.Source: §C.2 · pp. 15–16

Experimental Records

• 300 executed proposal–outcome records cover 40 training tasks; 226 use one mechanism direction and 74 combine two.One hundred records are selected as the initial RWM set M0.Source: §C.2 · p. 16, Table 5 · p. 16

Training and Baseline Data

• Initial RL histories contain 0–8 task-matched real records and are expanded with RWM-predicted outcomes; ICL baselines can use up to 300 executed records.SFT uses 300 state–proposal pairs, while RFT-Real keeps the 114 pairs with positive measured gains.Source: §C.2 · p. 16

Held-out States

• The held-out evaluation uses 60 autoregressive and 18 diffusion research states from 10 held-out tasks, with real-outcome histories fixed across methods.No held-out experimental records are used for proposer training, RWM grounding, or calibration.Source: §C.2 · p. 16

D.1 Metrics

• Each proposer is tested on 60 autoregressive and 18 diffusion held-out states, generating four proposals per state for real training-run evaluation.Invalid or unchanged proposals receive zero gain, while negative gains are retained.Source: §D.1 · pp. 16–17
• Avg@4 and Best@4 summarize mean and maximum real gain across four proposals per state; win rate compares methods’ Avg@4 by state.Model-performance measures are BPB for autoregressive models and NLL for diffusion models; lower values indicate better performance.Source: §D.1 · p. 17

D.2 Cost Accounting

• Data-generation cost counts real-execution GPU expenses and RWM prediction token costs, while time sums execution durations assuming sequential runs and prediction time.Cost excludes proposer and builder inference, proposer fine-tuning, and shared research-state construction.Source: §D.2 · p. 18

E.1 Models and Prompts.

• Qwen3.8-27B is the proposer, GPT-6-sol is the RWM, and GPT-6-astra runs the builder as a Codex agent.The RWM is queried once per proposal at high reasoning effort without tools.Source: §E.1 · p. 18

E.2 RL Training

• RL uses eight policy updates per rollout, a global batch of 256, temperature 1.0 sampling, and an initial learning rate of 10^-6.The learning rate is multiplied by 0.7 after each training segment.Source: §E.2 · p. 18

E.3 On-Policy Calibration

• Calibration executes selected current-policy proposals once each; the three rounds add 31, 38, and 31 records to RWM records, not training-state histories.Selected proposals are spread across training tasks and states.Source: §E.3 · p. 19

E.4 SFT and RFT-Real

• SFT trains on 300 executed-proposal pairs, while RFT-Real retains the 114 pairs with positive measured gains.Separate LoRA adapters are trained for autoregressive and diffusion tasks.Source: §E.4 · p. 19

F.1 Per-Family Results

• LM2LM has the highest Avg@4 and Best@4 on OLMo-3, Qwen3, E2D2, and BD3LM, but family-level exceptions occur on Marin and E2D2-SW.It has the highest Best@4 on Marin, but trails GPT-6-sol-ICL in Avg@4 on Marin and E2D2-SW and Base-ICL in Best@4 on E2D2-SW.Source: §F.1 · p. 19
• 4.42 Avg@4 and 10.68 Best@4 on autoregressive tasks excluding Qwen3 remain above the baselines’ maxima of 3.09 and 7.58.Source: §F.1 · p. 19
• On NanoChat, no method improves by more than about 2σe, and GPT-6-sol-ICL alone achieves positive Avg@4.Best@4 is at most 1.23 × 10^-3 BPB.Source: §F.1 · p. 19

F.2 Statistical Robustness

• 7.29σe Avg@4 on autoregressive tasks exceeds the baselines’ maximum 4.91σe; 50.8% of LM2LM proposals exceed 2σe, versus 22.5% for Base and 40.0% for GPT-6-sol-ICL.LM2LM’s Avg@4 corresponds to a 1.28% reduction in reference loss.Source: §F.2 · p. 19

G.1 Data-Generation Cost

• 18.91 at $23.9k exceeds the AR baseline maximum of 16.54 at $60.0k; on diffusion tasks, Best@4 reaches 9.93 at M3 versus a baseline maximum of 8.50.LM2LM exceeds every baseline on AR tasks before its first calibration and on diffusion tasks at M3.Source: §G.1 · p. 19

G.2 Total Cost

• $56.1k total at M3 keeps LM2LM below ICL with K = 300 ($60.0k), while Avg@4 is higher on AR (17.06 vs. 6.15) and diffusion tasks (7.10 vs. 4.94).LM2LM’s M3 total includes $40.1k in data generation and $16.1k in RL training.Source: §G.2 · p. 19

H Case Studies: Impact of Calibration Strategies

• Top-rated calibration corrects the overestimated M-UPD-72 reward after its first update, while random calibration delays correction and spends more training on the harmful change.The change’s held-out real reward is about −1.46; top-rated calibration executes seven proposals applying it in the first update, while random calibration executes none.Source: H Case Studies: Impact of Calibration… · pp. 19–20, Figure 7 · p. 20
• Random calibration’s OLMo token-mixing record raises RWM rewards and proposal share, but the change has a mean held-out gain of −16.4 × 10−3 BPB; top-rated calibration keeps its share below 5%.The random record came from a training-task improvement of 1.43%.Source: H Case Studies: Impact of Calibration… · pp. 20–21, Figure 8 · p. 21

I.1 Diagnosing RWM Feedback Reliability

• At step 128, static-RWM sign agreement falls to 0.43 and wrong-reinforcement reaches 55%, while LM2LM sign agreement is 0.62 and wrong-reinforcement stays between 30% and 36%.From step 96 onward, static-RWM-reinforced proposals have a negative mean real advantage (−0.31), whereas LM2LM’s remain positive (0.18 to 0.59).Source: §I.1 · pp. 21–22, Figure 9 · p. 23

I.2 Effect of Updating Records on Identical Proposals

• With identical LM2LM proposals, calibrated records raise pairwise accuracy from 0.44 to 0.61 at step 128, compared with scoring those proposals using initial records M0.At steps 64 and 96, calibrated-record accuracy is 0.64 and 0.61, versus 0.49 and 0.46 with M0.Source: §I.2 · p. 22, Figure 10 · p. 23

I.3 Case Studies

• The orthogonalized-momentum optimizer is favored on OLMo-3 and Qwen3 but rejected on NanoChat, with held-out execution supporting those family-dependent choices.The change improves 93% of OLMo-3 and 100% of Qwen3 proposals that apply it, while losing 44 × 10−3 BPB on average on NanoChat.Source: §I.3 · p. 22
• −1.89 versus −10.84 is LM2LM’s NanoChat Avg@4 at step 96 versus the static variant, and LM2LM outperforms it on all 17 held-out states.After calibration, the RWM reward for M-UPD-72 shifts to −1.66, close to its held-out real reward.Source: §I.3 · p. 22

J Information about AI Assistants

• GPT-6.1 assisted with grammar, clarity, and figure or table presentation; authors report no contribution to research ideation, experiments, implementation, data analysis, or technical content beyond surface edits.The authors reviewed and edited all outputs and take responsibility for the final text and visuals.Source: J Information about AI Assistants · p. 22