PaperlayerView exact revision on arXiv

Source-linked AI summary

SEIF: Self-Evolving Reinforcement Learning for Instruction Following

Qingyu Ren, Qianyu He, Jiajie Zhu, Xingzhou Chen, Jingwen Chang, Zeye Sun, Han Xia, Fei Yu, Jiaqing Liang, Yanghua Xiao

arXiv:2605.07465v1cs.CL

TL;DR

Continuously improving LLM instruction following is difficult because external supervision is costly and static self-play instructions do not evolve with model capabilities. SEIF co-evolves challenging instructions and followers through a filtered, judged reinforcement-learning loop, improving performance across model scales and architectures.

  • Problem

    Existing instruction-following methods either require costly external supervision or use self-play with static-difficulty instructions that cannot evolve with model capabilities.

  • Method

    SEIF alternately trains an Instructor and Follower while frozen Filter and Judger roles ensure instruction quality and provide constraint-satisfaction rewards.

  • Results

    SEIF improves instruction-following performance across five model families from 1.5B to 14B parameters and multiple architectures, including +4.7 on Qwen2.5-7B-Instruct IFEval.

  • Takeaways & Limitations

    Effective self-evolution uses sufficient early-stage training followed by moderate late-stage training, while dynamic instruction difficulty evolution is important for performance.

  • Takeaways & Limitations

    Training data may be substantially simpler than real-world instructions involving thousands of tokens, multiple constraint types, external documents, and implicit requirements.

Abstract

from arXiv · show

Instruction following is a fundamental capability of large language models (LLMs), yet continuously improving this capability remains challenging. Existing methods typically rely either on costly external supervision from humans or strong teacher models, or on self-play training with static-difficulty instructions that cannot evolve as the model's capabilities improve. To address these limitations, we propose SEIF (Self-Evolving Reinforcement Learning for Instruction Following), a self-evolving framework for enhancing the instruction-following ability of LLMs. SEIF forms a closed self-evolution loop that improves the model's instruction-following ability, where instruction difficulty evolution and model capability evolution reinforce each other. SEIF consists of four roles: an Instructor that generates increasingly challenging instructions, a Filter that removes conflicting or invalid instructions to ensure data quality, a Follower that learns to follow evolved instructions, and a Judger that provides reward signals for reinforcement learning. The Instructor and Follower are alternately trained and co-evolve throughout the process. Experiments across multiple model scales and architectures show that SEIF consistently improves instruction-following performance, suggesting strong generality. Further analyses reveal the sources of improvement and identify an effective training strategy for self-evolution on open-ended tasks: sufficient early-stage training to build a solid foundation, followed by moderate late-stage training to mitigate overfitting and achieve better final performance. The code and data are publicly available at https://github.com/Rainier-rq1/SEIF.

1 Introduction

SEIF addresses the difficulty of continuously improving instruction following by creating a self-evolving loop for open-ended tasks. Its Instructor, Filter, Follower, and Judger roles support evolving instructions, quality control, learning, and reward acquisition, while experiments show general improvements and favor intensive early training followed by moderate late training.

  • Motivation: Instruction following is important for LLM evaluation and real-world use, but continuously improving it efficiently remains challenging.Strong instruction following requires understanding user requirements and constraints and generating responses that meet expectations.
  • Limitations: Existing approaches either depend on costly external supervision or use self-play with instructions whose difficulty remains static.Current self-evolving methods mainly target verifiable tasks such as mathematics and code, leaving open-ended instruction following less explored.
  • SEIF framework: SEIF targets three challenges: evolving instruction difficulty, preserving instruction quality, and obtaining reliable rewards for open-ended tasks without ground-truth answers.These challenges arise as instruction complexity grows and require preventing constraint conflicts while acquiring usable reward signals.
  • SEIF framework: SEIF creates a closed self-evolution loop with four roles: Instructor, Filter, Follower, and Judger.The Instructor generates increasingly challenging instructions, the Filter removes conflicting or meaningless instructions, the Follower learns to follow them, and the Judger provides rewards.
  • Results: SEIF improves instruction-following ability across multiple parameter scales and model architectures, demonstrating strong generalization.The study also analyzes improvement sources and training strategies under the self-evolution paradigm.
  • Results: Sufficient early-stage training followed by moderate late-stage training is particularly effective for self-evolution.The reported strategy uses intensive training early and moderate training later to mitigate overfitting and achieve better final performance.

2 Method

SEIF is a self-evolving framework that alternates training an Instructor to generate increasingly challenging instructions and a Follower to satisfy them. Frozen Filter and Judger roles maintain instruction quality and provide adaptive reward signals, while GRPO updates both trainable components.

  • Framework overview: SEIF alternates between an Instructor that evolves seed instructions into more complex ones and a Follower that learns to satisfy them.The Instructor and Follower are the two trainable roles in the framework.
  • Auxiliary roles: The Filter removes conflicting or invalid evolved instructions, while the Judger scores how well responses satisfy their constraints.Both auxiliary roles are frozen and support data quality and reward construction for open-ended tasks.
  • Alternating training: At each iteration, the Instructor is trained against the current frozen Follower, then the updated Instructor generates instructions for Follower training.Refreshing the auxiliary roles from the latest Follower makes filtering and reward criteria adaptive.
  • Adaptive evolution: The Follower trains on an adaptive instruction distribution near its capability boundary, encouraging responses that satisfy more constraints and driving progressive improvement.The training prompts come from the latest Instructor rather than a fixed dataset.
  • Optimization: GRPO updates both Instructor and Follower by comparing grouped candidate outputs and standardizing rewards without training an additional value function.For the Instructor, rewards evaluate evolved instructions; for the Follower, rewards evaluate responses to those instructions.

3 Experiment

SEIF is evaluated across five models from 1.5B to 14B parameters and six instruction-following benchmarks. Across model scales, SEIF improves instruction following, with dynamic difficulty evolution outperforming static self-play while generally preserving general capabilities.

  • Experimental Setup: SEIF is evaluated on five models spanning 1.5B to 14B parameters against frontier, specialized instruction-following, and other training baselines.The evaluation uses six benchmarks covering constraint satisfaction, multi-turn dialogue, writing, agentic tool use, and multilingual instruction following.
  • Self-Evolving Enhances Instruction-Following: On Qwen2.5-7B-Instruct, SEIF gains +4.7 on IFEval, +4.0 on CFBench, and +6.6 on WritingBench, while Distill-Qwen-14B reaches 80.0 on IFEval (+5.1).The results span multiple model families and three iterative training stages, and SEIF outperforms other instruction-following optimization baselines.
  • Small Models Can Also Benefit from Self-Evolution: Qwen2.5-1.5B-Instruct gains +4.7 on AgentIF and +3.9 on IFEval, showing that small models also benefit consistently from self-evolution.Absolute scores remain lower than those of larger models, but gains are steady across nearly all benchmarks.
  • Instruction Difficulty Evolution is Critical: SEIF reaches 78.6 on IFEval, 51.0 on CFBench, and 59.0 on FollowBench, outperforming static instruction self-play; Meta-Rewarding reaches 76.6 on IFEval, only +2.7 over BASE.The comparison is conducted on SEIF-7B, and static methods show only marginal improvements over the base model.
  • General Capability Preservation: Distill-Qwen-14B reaches the best average of 60.3 on general-capability benchmarks, while Llama-3.1-8B improves from 24.0 to 24.4.These results indicate that SEIF generally maintains or improves general capabilities rather than compromising them.

4 Analysis

SEIF’s gains arise from quality control, adaptive evaluation, and iterative self-evolution: filtering and refreshed Filter/Judger components preserve useful reward signals, while training saturates after Turn3. Analyses further show that data distributions evolve across turns and that front-loaded training with moderate later refinement produces the strongest self-evolution.

  • Component ablations: Removing the Filter drops IFEval by −3.2 and CFBench by −6.0, showing that filtering invalid or conflicting instructions preserves training-data quality.Without filtering, the Instructor can generate noisy training signals and infeasible or inconsistent instructions.
  • Component ablations: Keeping the Judger and Filter fixed instead of refreshing them from the latest Follower decreases benchmark performance, indicating that adaptive criteria keep rewards aligned with evolving capabilities.The shared-parameter design refreshes evaluation and filtering components as the Follower improves.
  • Iteration turns: Performance improves during the first few training turns, saturates after Turn3, and can degrade with further training; therefore, SEIF adopts three iteration turns.Later training yields only marginal gains for some models and harms others.
  • Evolution mechanism: Training representations shift in distribution and centroid position across iterative turns, indicating that SEIF generates evolving data rather than reusing a fixed instruction set.The analysis uses PCA visualizations of training-instruction representations across three iteration turns.
  • Self-Evolving Training Strategy: The E1 = 3, E2 = 1, E3 = 1 schedule achieves the best performance, whereas late-intensive E1 = 1, E2 = 1, E3 = 3 can hurt through distributional bias or weakened complex instruction following.Sufficient first-turn training builds a foundation, while later turns provide moderate refinement; uniform schedules are generally inferior to asymmetric ones.
  • Judger and Filter Reliability: On 400 VerInstruct examples, the Filter reaches Accuracy around 0.79–0.80 and F1 around 0.78–0.80, while the Judger reaches Accuracy around 0.73–0.74 and F1 around 0.70–0.72.These human-agreement results indicate reliable Filter detection of conflicting instructions and provide evidence for Judger reliability.

5 Related Work

Prior instruction-following reinforcement learning relies on costly external supervision or self-play with often-static training data. Self-evolving training reduces external dependence, but open-ended instruction following remains difficult because it lacks unique ground-truth answers.

  • Reinforcement learning for instruction following: Instruction-following reinforcement learning commonly derives rewards from human annotations or stronger teacher models, creating costly and difficult-to-scale external supervision.These approaches are effective but depend on supervision that is expensive to obtain and scale.
  • Self-play: Self-play methods use model-generated feedback to improve instruction following, but their training data is often static.Self-play reduces reliance on external feedback while retaining a limitation in data evolution.
  • Self-evolving training: Self-evolving training iteratively combines data generation, feedback, and learning to reduce external supervision, with strongest progress demonstrated in verifiable domains.Mathematical reasoning and code generation can use automatically obtained rewards, unlike open-ended instruction following, which has no unique ground-truth answer.

6 Conclusion · Appendix

SEIF is a self-evolving framework that improves LLM instruction following through a closed loop of four specialized roles. Alternating GRPO optimization adapts instruction difficulty to the Follower’s capability boundary.

  • 6 Conclusion: SEIF targets improvement of LLM instruction-following ability.
  • 6 Conclusion: SEIF constructs a closed self-evolution loop.
  • 6 Conclusion: The Instructor evolves increasingly challenging instructions.
  • 6 Conclusion: The Filter ensures instruction quality.
  • 6 Conclusion: The Follower learns to follow evolved instructions.
  • 6 Conclusion: The Judger provides constraint satisfaction rewards through alternating GRPO optimization.

A Training Data Construction … A.3 Hard Constraint Types

SEIF constructs training data through an Instructor–Filter–Follower–Judger pipeline, combining evolving instructions with soft and hard constraint taxonomies. The taxonomy spans 25 soft constraint types and 24 hard constraint subtypes organized across five high-level categories.

  • A.1 Pipeline: The Instructor generates more complex instructions from seed instructions, the Filter removes invalid or conflicting cases with reward 0, and the Judger evaluates Follower responses by constraint satisfaction.If an instruction passes filtering, the latest Follower generates a response; the Judger uses 1 minus the constraint satisfaction rate as the training signal.
  • A.2 Soft Constraint Types: The soft-constraint taxonomy contains 25 types, and the Instructor selects five types to add to each instruction.Types cover content, elements, semantics, counts, style, audience, formatting, language, syntax, morphology, phonology, roles, tasks, examples, exclusions, contradictions, and rules.
  • A.2 Soft Constraint Types: Soft constraints include presentation and linguistic requirements such as tone, form, audience, authorial style, JSON or HTML, custom formats, language, syntax, capitalization, and sound patterns.The taxonomy also includes role-based, task-specific, complex-context, example-based, inverse, contradictory, and rule constraints.
  • A.3 Hard Constraint Types: Hard constraints comprise 24 subtypes in five high-level categories, with the Instructor selecting three different categories and exactly one subtype from each.Each selected subtype is used to generate one complete constraint.
  • A.3 Hard Constraint Types: Lexical hard constraints regulate required or forbidden words, keyword and letter frequencies, and the frequency of ALL-CAPS words.Structural layout constraints regulate sentence, paragraph, and bullet-point counts, sectioning, paragraph-initial words, and word-count conditions.
  • A.3 Hard Constraint Types: Formatting and language constraints require specified markup, wrappers, valid JSON, response language, capitalization, or punctuation restrictions.Examples include highlighted spans, title or quotation wrappers, English uppercase or lowercase, and no commas.
  • A.3 Hard Constraint Types: Special-pattern constraints impose global response structures such as repeating the request before answering, exact endings, two distinct responses, postscripts, and placeholder counts.These are grouped as Category E: Special pattern constraints.

B Training Details · C Pseudocode · D Filter and Judger Prompt

The appendices specify GRPO implementation details, present SEIF’s pseudocode, and define Filter and Judger prompts for checking constraint consistency and satisfaction. The algorithm alternates Instructor and Follower optimization using filtered instructions and Judger-derived rewards.

  • B Training Details: Instructor and Follower are trained with GRPO using the EasyR1 framework, with experimental hyperparameters summarized in Table 8.The appendix also provides a prompt template for adding constraints.
  • B Training Details: The training-details appendix documents the constraint-adding prompt template and GRPO hyperparameter table used in the experiments.
  • C Pseudocode: Algorithm 1 presents SEIF’s self-evolving procedure, while the pseudocode requires a seed instruction set, initial Instructor and Follower models, and an iteration count.
  • D Filter and Judger Prompt: The Filter checks whether instruction constraints are internally consistent and assigns rejected instructions an Instructor reward of 0.Accepted instructions receive a Follower response, whose satisfaction score determines the Instructor reward as 1 − si.
  • D Filter and Judger Prompt: The Judger evaluates each constraint in a reply and outputs [[1]] when all constraints are satisfied, otherwise [[0]].The prompt requires analysis first and the final score in [[score]] format, with examples covering insufficient characters and forbidden commas.
  • D Filter and Judger Prompt: At each iteration, frozen Filter and Judger models are instantiated from the current Follower before Instructor optimization begins.
  • D Filter and Judger Prompt: Instructor optimization samples grouped evolved instructions, skips updates when reward standard deviation is zero, and otherwise applies group-relative advantages with GRPO.
  • D Filter and Judger Prompt: Follower optimization generates filtered evolved instructions, samples grouped responses, computes Follower rewards, skips zero-variance updates, and returns the updated Instructor and Follower.

E Benchmark Details · F Baseline Details · F.1 Specialized Instruction-Following Models

SEIF is evaluated on six instruction-following benchmarks spanning distinct capability dimensions and compared with specialized instruction-following and self-training baselines. The benchmarks test verifiable, multi-constraint, writing, agentic, multi-level, multilingual, and multi-turn instruction following.

  • E Benchmark Details: SEIF is evaluated on six widely adopted benchmarks, each targeting a distinct dimension of instruction-following capability.
  • E Benchmark Details: IFEval uses automatically checked constraints, while CFBench evaluates complex multi-constraint instructions across real-life scenarios and NLP tasks.IFEval contains approximately 500 prompts and 25 constraint templates; CFBench covers more than 200 scenarios, 50 NLP tasks, and over 25 subcategories.
  • E Benchmark Details: FollowBench progressively adds constraints across five types, whereas WritingBench assesses criteria-aware writing quality across six domains and 100 subdomains.FollowBench reports HSR, SSR, and CSL; WritingBench contains 1,000 queries averaging 1,500+ tokens and uses five instance-specific criteria.
  • E Benchmark Details: AgentIF evaluates instruction following in agentic scenarios using 707 human-annotated instructions from 50 applications and code-based, LLM-based, or hybrid evaluation.Each instruction averages 1,723 words and 11.9 constraints spanning tools, conditions, formatting, and safety.
  • E Benchmark Details: Multi-IF tests 4,501 three-turn multilingual conversations across eight languages and finds higher failure rates with additional turns, especially in non-Latin-script languages.
  • F Baseline Details: SEIF is compared against baselines covering specialized instruction-following models and self-training methods.
  • F.1 Specialized Instruction-Following Models: Specialized RL baselines use decomposed curricula, verifiable rewards, and mixed rule- or LLM-based verification; RAIF-7B reports an 11.74% IFEval improvement for 1.5B models.Self-Supervised-7B derives pseudo-labels from instructions, VERIF-8B trains on approximately 22,000 instances, and RAIF-7B adds expert-reasoner behavior cloning.
  • F.1 Specialized Instruction-Following Models: Preference-optimization baselines refine data through self-play search, constraint back-translation, or GPT-4-driven progressive refinement.SPAR-8B-DPO reaches 81.3% on IFEval and surpasses GPT-4-Turbo; Crab-7B-DPO reduces noise and generation cost; Conifer-7B-DPO outperforms open-source models 10x larger.

F.2 Baseline Method · F.2.1 SFT and ProxyReward · F.2.2 Self-Play Baselines

The baselines are reproduced from a constrained seed-instruction dataset using the backbone model, while comparison methods span supervised fine-tuning, proxy rewards, and self-play or self-alignment loops. These methods use self-generated data, iterative correction, filtering, judgment, or reward signals, with several reporting substantial instruction-following gains.

  • F.2 Baseline Method: Baseline reproduction begins by adding soft or hard constraints to 5,120 seed instructions using the backbone model and Table 7’s prompt.The remaining reproduction settings follow the original papers.
  • F.2.1 SFT and ProxyReward: SFT uses GPT-5.1-generated responses to complex initial-dataset instructions for full-parameter training on 8 H200 GPUs.ProxyReward automatically constructs proxy question-answer pairs and targeted rewards to assess comprehensiveness and accuracy in long-context generation.
  • F.2.2 Self-Play Baselines: Self-Correct iteratively refines responses by having models identify and correct instruction-following errors using their own judgment without external supervision.
  • F.2.2 Self-Play Baselines: Humpback back-translates unlabeled web content into instruction-response pairs and iteratively self-curates high-quality examples without distillation from larger models.
  • F.2.2 Self-Play Baselines: SELF iteratively fine-tunes models on their own high-quality outputs, progressively improving instruction-following through self-generated data.
  • F.2.2 Self-Play Baselines: Three iterations of Self-Rewarding let fine-tuned Llama 2 70B use itself as a reward model with iterative DPO, outperforming Claude 2, Gemini Pro, and GPT-4 0613 on AlpacaEval 2.0.The method jointly improves instruction following and reward generation.
  • F.2.2 Self-Play Baselines: 8.88% absolute improvement on IFEval and 78.2% relative improvement on AlpacaEval are achieved by I-SHEEP on Qwen-1.5 72B.I-SHEEP cycles through self-synthesis, self-assessment, filtering, and supervised fine-tuning.
  • F.2.2 Self-Play Baselines: 22.9% to 39.4% win rate on AlpacaEval 2.0 is achieved by Meta-Rewarding for Llama-3-8B-Instruct without human supervision.It extends self-rewarding with a meta-layer that evaluates and refines the model’s own judgments.

G Judger and Filter Reliability · H Instruction Evolving Example

SEIF evaluates its self-instantiated Filter and Judger against human judgments, tests refreshing them from the latest Follower, and directly assesses human preference for final outputs. It also illustrates instruction evolution across ten example tasks spanning generation, summarization, question answering, rating, and policy analysis.

  • G Judger and Filter Reliability: SEIF evaluates Filter and Judger reliability through agreement with human judgments, initialization ablations, and pairwise human evaluation of final outputs.The agreement evaluation uses 400 VerInstruct examples, while final-output evaluation compares BASE, Ours-SEIF, w/o Instructor Evolving, and Meta-Rewarding.
  • G Judger and Filter Reliability: Using the latest Filter and latest Judger achieves the best performance on all three benchmarks.The ablation varies initialization from the base model versus refreshing from the latest Follower.
  • G Judger and Filter Reliability: Pairwise human evaluation provides direct evidence that SEIF improves final instruction-following quality rather than merely optimizing for Judger bias.The evaluation collects anonymized responses from BASE, Ours-SEIF, w/o Instructor Evolving, and Meta-Rewarding on Qwen2.5-7B-Instruct.
  • H.1 Case 1: Restaurant Description – Midsummer House; H.2 Case 2: Electric Vehicle Conversation Generation: Instruction evolution includes converting restaurant key-value pairs into fluent English and constructing a conversation about electric vehicles.These examples correspond to the Midsummer House restaurant description and electric-vehicle conversation tasks.
  • H.3 Case 3: Restaurant Review – xname; H.4 Case 4: Navy News Article Summary: The examples include writing a restaurant review from structured information and generating a highlight summary of a Navy article about adopting NICE.The restaurant review specifies recommendation, cuisine, decor, location, price, and service attributes.
  • H.5 Case 5: WWF Five-Sentence Summary; H.6 Case 6: Gangs of New York Question Generation: Other evolved tasks require five-sentence summarization of a WWF passage and question generation about “The blood stays on the blade” from a movie discussion.The WWF task concerns conservation, ecosystems, wildlife, and global environmental protection.
  • H.7 Case 7: Employee Monthly Performance Rating; H.8 Case 8: Casino Royale CIA Plot Question Title: The instruction set also covers rating an employee’s monthly performance from reported work activities and summarizing a Casino Royale CIA plot question.The performance-rating task uses a 1-to-5 scale and lists customer-service, technical-support, and project-related activities.
  • H.9 Case 9: Computer Adjective Generation; H.10 Case 10: Impact of Increasing Business Taxes: Further examples ask for at least five adjectives describing a computer and an impact description for increasing business taxes.These examples extend instruction evolution to attribute generation and policy-impact analysis.

I Limitations … L Impact

SEIF acknowledges that real-world instructions may exceed its training-data complexity, while presenting self-evolving challenge construction as a path toward scalable, autonomous post-training with less intensive external supervision. The paper also reports its annotation, AI-assisted writing, and human-subjects practices.

  • I Limitations: Real-world instructions may be thousands of tokens long, combine multiple constraint types, reference external documents, and contain implicit requirements.AgentIF instructions average 1,723 words and nearly 12 constraints each.
  • I Limitations: SEIF’s training data may not fully represent the substantially greater complexity of real-world user instructions.The paper identifies this mismatch as a limitation of the work.
  • J Crowdsourcing and research with human subjects: Three computer science students annotated the data, received pay above the local minimum wage, and consented to its stated research use.Conflicting annotations were resolved through majority voting.
  • J Crowdsourcing and research with human subjects: When annotations conflicted, the final label was determined by majority vote.The annotation process involved three computer science students.
  • K Declaration of LLM usage: Cursor was used solely to polish language in an early manuscript version, while the authors independently developed the paper’s core ideas.The paper acknowledges this AI-assisted writing use.
  • L Impact: SEIF explores models that continuously construct training challenges and learn from their own evolving experience.This frames self-evolution as a paradigm for improving large language models.
  • L Impact: The paradigm may support more scalable and autonomous post-training while reducing long-term dependence on intensive human supervision or increasingly stronger teacher models.The paper further suggests adaptive AI systems could improve on open-ended tasks through iterative self-evolution.
Loading 2605.07465v1…