PaperlayerView exact revision on arXiv

Source-linked AI summary

Self-Adapting Language Models

Adam Zweiger, Jyothish Pari, Han Guo, Ekin Akyürek, Yoon Kim, Pulkit Agrawal

arXiv:2506.10943v2cs.LG

TL;DR

LLMs remain difficult to adapt to new tasks and knowledge because they generally lack mechanisms for transforming data and learning procedures to fit each context. SEAL lets models generate self-edits that control finetuning data, updates, and tools, training this behavior with downstream rewards. It improves knowledge incorporation and few-shot performance in the reported evaluations, while remaining vulnerable to catastrophic forgetting and requiring explicit downstream tasks for reward computation.

  • Problem

    LLMs are powerful but adapting them to specific tasks, new information, and novel reasoning remains challenging because task-specific data is limited and models lack bespoke data-learning strategies.

  • Method

    SEAL uses reinforcement learning to train LLMs to generate self-edits that specify training data, finetuning directives, optional hyperparameters, and tool use, followed by supervised weight updates.

  • Results

    SEAL improves no-passage-in-context SQuAD question answering from 33.5% to 47.0% and improves few-shot performance on simplified ARC-AGI over standard ICL and non-RL self-editing.

  • Takeaways & Limitations

    SEAL demonstrates that LLMs can autonomously incorporate knowledge and adapt to novel tasks through synthetic self-edit data and lightweight weight updates.

  • Takeaways & Limitations

    Performance on earlier tasks gradually declines as edits accumulate, and current instantiations require explicit downstream tasks with held-out queries or reference QA for reward computation.

Abstract

from arXiv · show

Large language models (LLMs) are powerful but static; they lack mechanisms to adapt their weights in response to new tasks, knowledge, or examples. We introduce Self-Adapting LLMs (SEAL), a framework that enables LLMs to self-adapt by generating their own finetuning data and update directives. Given a new input, the model produces a self-edit-a generation that may restructure the information in different ways, specify optimization hyperparameters, or invoke tools for data augmentation and gradient-based updates. Through supervised finetuning (SFT), these self-edits result in persistent weight updates, enabling lasting adaptation. To train the model to produce effective self-edits, we use a reinforcement learning loop with the downstream performance of the updated model as the reward signal. Unlike prior approaches that rely on separate adaptation modules or auxiliary networks, SEAL directly uses the model's own generation to control its adaptation process. Experiments on knowledge incorporation and few-shot generalization show that SEAL is a promising step toward language models capable of self-directed adaptation. Our website and code is available at https://jyopari.github.io/posts/seal.

1 Introduction

SEAL addresses the difficulty of adapting pretrained LLMs by letting them generate training data and finetuning directives tailored to new inputs. Reinforcement learning trains these self-edits using downstream performance, with gains shown in knowledge incorporation and few-shot learning.

  • Motivation: Current LLMs typically consume task data as-is and lack bespoke strategies for transforming and learning from it.The paper frames self-adaptation as transforming or generating training data and learning procedures.
  • Approach: Each RL iteration generates candidate self-edits, applies the resulting weight updates, evaluates downstream performance, and uses the reward to improve self-edit generation.This outer loop is summarized in Figure 1.
  • Approach: SEAL trains LLMs to generate self-edits: natural-language directives specifying training data and optionally optimization hyperparameters for weight updates.The framework uses reinforcement learning to train self-edit generation in response to new inputs.
  • Results: 47.0% versus 33.5%: finetuning on SEAL-generated synthetic data improves no-passage-in-context SQuAD question answering after RL training.SEAL-generated data also outperforms synthetic data generated by GPT-4.1.
  • Results: On a simplified ARC-AGI subset, SEAL autonomously selects data augmentations and optimization hyperparameters, improving few-shot performance over standard ICL and non-RL self-editing.The tools include learning-rate, epoch, and selective-loss configurations.

2 Related Work

SEAL builds on synthetic data generation, knowledge updating, test-time training, reinforcement learning, meta-learning, and self-improvement. Its distinctive focus is using RL to optimize self-edit data for downstream weight updates rather than directly optimizing final answers.

  • Synthetic Data Generation: SEAL extends synthetic-data methods by using reinforcement learning to maximize the downstream utility of data applied through gradient-based self-updates.This contrasts with static or heuristic synthetic-data generation.
  • Knowledge Updating: For knowledge updating, SEAL follows approaches that generate additional finetuning data from contextual information rather than directly locating parameters for individual facts.Related methods include implication-based finetuning from facts.
  • Test-Time Training: SEAL incorporates a round of test-time training in its inner loop, using efficient weight updates to evaluate which generated data yields the greatest performance gain.The paper reports training with single-example test-time-training episodes.
  • Reinforcement Learning for LLMs: Unlike RL methods that optimize final answers or trace revisions, SEAL applies reinforcement learning to generating self-edit data that is then used for weight updates.The reward targets the utility of the generated adaptation material.
  • Meta-Learning and Self-Modifying Systems: SEAL reflects meta-learning by learning an adaptation strategy—how to generate effective self-edits—through an outer optimization loop.The stated goal is to learn how to learn efficiently from task contexts.
  • Self-Improvement: Unlike self-improvement methods that use models to provide rewards or confidence signals, SEAL learns to generate data for subsequent parameter updates.The related-work discussion places SEAL among broader self-training approaches.

3 Methods

SEAL trains LLMs to generate self-edits that specify synthetic adaptation data, tool use, and optimization settings, then evaluates the resulting weight updates with reinforcement learning. The framework is instantiated for knowledge incorporation and few-shot learning, using supervised finetuning—often with LoRA—inside an adaptation loop.

  • General Framework: SEAL generates a self-edit from task context and applies it through supervised finetuning to produce updated parameters.The self-edit form varies by domain, and the update is represented as θ′ ← SFT(θ, SE).
  • General Framework: The adapted model’s downstream performance supplies the reward used to train self-edit generation with reinforcement learning.The reward depends on the updated model’s performance on the task evaluation.
  • General Framework: SEAL uses ReSTEM after GRPO and PPO training proved unstable, reinforcing sampled self-edits with positive reward through supervised finetuning.ReSTEM is described as filtered behavior cloning, or rejection sampling plus SFT.
  • Knowledge Incorporation: In knowledge incorporation, the model generates passage implications, finetunes on them with LoRA, and is evaluated on passage questions without the original text.The evaluation accuracy serves as the reinforcement-learning reward.
  • Few-Shot Learning: In few-shot ARC learning, SEAL selects data augmentations and optimization hyperparameters through tools before LoRA adaptation and held-out evaluation.The self-edit specifies which tools to invoke and how to configure them; the held-out result determines the reward.

4 Results

SEAL is evaluated on few-shot learning and knowledge incorporation, where reinforcement-learned self-edits improve adaptation over non-RL and no-adaptation baselines. Knowledge-incorporation results show gains across single-passage and continued-pretraining settings, with performance surpassing GPT-4.1 synthetic data after two RL iterations in the no-context SQuAD setup.

  • 4.1 Few-Shot Learning: The few-shot experiments use Llama-3.2-1B-Instruct on 11 training and 8 evaluation ARC tasks selected for solvability under optimal TTT configurations.The model has no ARC-specific pretraining, and the curated setting supports learning a fixed self-edit strategy from limited examples.
  • 4.1 Few-Shot Learning: 72.5% adaptation success exceeds 20% for base-model self-edits without RL and 0% with no adaptation, but remains below Oracle TTT.Evaluation applies five independently generated self-edits per held-out task and reports the percentage producing correct outputs.
  • 4.2 Knowledge Incorporation: 47.0% no-context SQuAD accuracy follows reinforcement learning, compared with 46.3% for GPT-4.1 synthetic data and 33.5% for direct passage finetuning.Base-model synthetic data reaches 39.7%, while direct passage finetuning yields only a negligible gain over the frozen base model.
  • 4.2 Knowledge Incorporation: 58.2% accuracy is achieved by SEAL in continued pretraining on n = 200 documents, although GPT-4.1 synthetic data slightly outperforms SEAL in that setting.Aggregating five self-edit generations per passage produces a similar method ranking while exceeding SEAL’s single-passage performance.
  • 4.2 Knowledge Incorporation: Two outer RL iterations suffice for SEAL to overtake GPT-4.1 synthetic data on no-context SQuAD, after which accuracy shows diminishing returns.Each iteration uses 50 contexts and five sampled self-edits per context; the learned policy tends toward atomic, easily learnable facts.
  • 4.2 Knowledge Incorporation: Figure 5 shows that RL can produce more detailed self-edits associated with better performance, while prompting for longer self-edits also yields gains.The progression is clearer in some examples than others, and RL further improves performance by a similar margin to longer-generation prompting.

5 Limitations

SEAL’s continual self-editing remains limited by catastrophic forgetting, computational overhead, and evaluation settings that require explicit downstream tasks. Performance on earlier tasks declines as edits accumulate, while each reward evaluation requires costly finetuning and evaluation.

  • Catastrophic forgetting: The current training setup does not explicitly optimize retention during sequential self-edits.The evaluation establishes a baseline for handling sequential edits without dedicated forgetting mechanisms.
  • Catastrophic forgetting: Performance on earlier tasks gradually declines as the number of self-edits increases, indicating susceptibility to catastrophic forgetting.The model can perform multiple updates without complete collapse, but retention is not explicitly optimized.
  • Computational overhead: Each self-edit reward evaluation requires finetuning and evaluating the entire model, taking approximately 30–45 seconds.This creates substantially more computational overhead than reward loops based on a forward pass or simple pattern matching.
  • Context-dependent evaluation: SEAL’s current instantiations require every context to include an explicit downstream task, preventing reinforcement-learning training on unlabeled corpora.The paper suggests generating evaluation questions or synthetic test cases as a possible way to broaden applicability.

6 Discussion and Conclusion

The discussion positions SEAL as a route toward models that generate training signals, update their weights, and adapt across tasks and interactions. It also outlines broader applications while documenting the controlled scope of the few-shot evaluation.

  • Future directions: SEAL could generate synthetic pretraining corpora as publicly available human-generated text approaches a projected data wall.The paper frames self-generated high-utility training signals as a possible response once web-scale corpora are exhausted.
  • Future directions: SEAL could let models ingest papers and generate explanations and implications for self-refinement on rare or underrepresented topics.The proposed loop is described as operating without additional external supervision.
  • Reasoning and adaptation: SEAL may complement chain-of-thought reinforcement learning by learning when and how to update weights during or after reasoning.The paper suggests that mid-reasoning updates could guide trajectories, while post-reasoning updates could distill insights into parameters.
  • Agentic systems: Structured self-modification could support agentic systems that acquire and retain knowledge across extended interactions.After an interaction, an agent could synthesize a self-edit that triggers a weight update and reduces reliance on repeated supervision.
  • Evaluation scope: The few-shot ARC evaluation used Llama-3.2-1B-Instruct with 11 training tasks and 8 held-out evaluation problems selected for test-time-training solvability.These sets were chosen to isolate self-edit learning rather than general ARC ability.
  • Evaluation procedure: SEAL generated 15 self-edit configurations per training problem, selecting data-augmentation and optimization settings for test-time training.Configurations varied augmentation types, learning rate, epochs, and whether loss used all or only output tokens; costly configurations were discarded.

A.3 Evaluation Details

The evaluation details describe a held-out ARC protocol in which multiple self-edit configurations are generated and scored after adaptation, alongside the knowledge-incorporation setup and training procedure.

  • Few-shot evaluation: For each of 8 held-out evaluation tasks, SEAL generated 5 self-edit configurations, yielding 40 configurations scored by post-adaptation correctness.Success was measured as the percentage of configurations producing correct outputs.
  • Knowledge incorporation: Knowledge incorporation used Qwen-2.5-7B with SQuAD v1.1, training on the training set and evaluating on 200 evaluation-set articles without passage context.The training and evaluation sets had overlapping topics but no passage overlap.
  • Knowledge incorporation: SQuAD supervised finetuning used batch size 10, 2 epochs, learning rate 3e-4, and LoRA with rank 64 and alpha 128.LoRA was applied to all MLP and attention projection layers.

B.3 Synthetic Data Generation and Finetuning Details

The synthetic-data pipeline prompts models to generate implications, converts those outputs into training documents, and then applies supervised finetuning under specified training configurations.

  • Synthetic data generation: Synthetic data is generated by prompting models to produce implications derived directly or indirectly from each passage.The resulting implications serve as the basis for subsequent finetuning.
  • Data processing: Single-passage generations are split by newline into training documents, whereas multi-passage generations are retained as one training document.GPT-4.1 outputs receive an additional rule that may omit the first line when the second line begins with “1.”.
  • Training configuration: The synthetic-data experiments report separate single-passage and multi-passage knowledge-incorporation hyperparameter tables.These are labeled Table 3 and Table 4, respectively.
  • Data processing: In the multi-passage n = 200 setting, five self-edit completions are sampled per passage and aggregated into one training dataset.The aggregate of all self-edits across passages is used for training.
  • Evaluation: The finetuned model answers each corresponding question using a direct and concise question-answering prompt.The prompt places the question before an “Answer:” continuation.

B.4 Evaluation Details

The evaluation uses a 200-passage SQuAD subset with 974 questions and automated grading by gpt-4.1-2025-04-14. Training and evaluation run on 2×H100 or 2×H200 infrastructure with specified compute optimizations.

  • The SQuAD evaluation uses 200 passages and 974 questions, roughly five questions per passage.
  • Automated grading uses gpt-4.1-2025-04-14 through the OpenAI API with greedy decoding.
  • The grader judges answers solely against the gold answer and returns only “yes” or “no.”
  • Experiments run on 2×H100 or 2×H200, using DeepSpeed ZeRO-3 for SFT and vLLM for efficient inference.
  • A single training round processes 50 passages across 5 completions and 3 runs per completion, totaling 750 inner-loop iterations.

B.6 Standard Error of the Mean in Catastrophic Forgetting Experiment

Table B.6 reports the standard errors of the mean for entries in Figure 6. These values provide uncertainty estimates for the reported experiment results.

  • Table B.6 reports standard errors of the mean for each entry in Figure 6.
  • The SEM values correspond specifically to the entries displayed in Figure 6.
  • The table supplies uncertainty information alongside the Figure 6 results.

B.7 Scaling Model Size

The scaling experiments compare SEAL across model sizes and against alternative synthetic-data and adapter approaches. Results indicate stronger relative gains at larger size, while broader scaling conclusions remain uncertain.

  • Model Size Scaling: SEAL’s improvement ratio over base-model self-edits is 1.75× for the 3B model and 2.04× for the 7B model.
  • Model Size Scaling: The larger relative improvement for the 7B model provides some evidence that reinforcement learning benefits may increase with model capacity.
  • Model Size Scaling: The authors caution that stronger scaling conclusions are difficult without experiments at still larger model sizes.
  • Comparison to Generative Adapter: Generative Adapter comparisons use the Mistral-7B-based model and evaluate both single-passage and continued-pretraining settings.
  • Comparison to Generative Adapter: Synthetic-data parameterization allows reuse for continued pretraining, application to arbitrary base models, and update types beyond LoRA finetuning.
  • Comparison to Entigraph: In SQuAD synthetic continued pretraining, Entigraph with ten generations is competitive with SEAL, especially at larger scale.

B.10 Proxy Reward

The paper compares a manually designed proxy reward with the full post-finetuning reward and examines prompt sensitivity. Proxy metrics reduce cost, while predefined prompts and RL training materially affect outcomes.

  • Proxy Reward: The proxy reward scores length, diversity, quality, and correctness from 1 to 5 using a GPT-4.1 grader.
  • Proxy Reward: The full SEAL loop avoids manually specifying a reward rubric by learning which self-edits improve downstream performance.
  • Proxy Reward: Proxy metrics offer dramatically lower cost, but further rubric or metric tuning may strengthen their reward signal.
  • Prompting: The prompting study evaluates seven formats, including implications, rewrite, self-QA, chain-of-thought, and no-prompt variants.
  • Prompting: ReSTEM improves performance by roughly 6 to 11 percentage points across the tested prompt variants.
  • Prompting: Chain-of-thought placement produced no substantial difference in this setting, while no-prompt reached only 18.9% after two training rounds.
Loading 2506.10943v2…