Self-play with Execution Feedback: Improving Instruction-following Capabilities of Large Language Models

1 INTRODUCTION

• Instruction-following errors can lead to unintended outcomes, potentially with severe consequences in critical scenarios.The introduction presents this as a motivation for improving instruction following.Source: §1 · p. 2
• Existing data approaches face scalability and reliability constraints: complex manual annotation is costly, while behavior imitation is limited by teacher capability and response errors.The passage notes that complex instruction design is difficult to scale and accurate execution by humans can require costly validation.Source: §1 · p. 2
• AUTOIF generates code-verifiable instructions, checks verification code with unit tests and back-translation, then uses execution-feedback rejection sampling to produce SFT or preference-training data.Passing responses can train SFT; passing and failing response pairs can form DPO chosen-rejected pairs.Source: §1 · p. 2
• 90.4% IFEval Loose Instruction (Acc.) with LLaMA3-70B marked the first result above 90%, while FollowBench SSR (avg) rose over 5%.Qwen2-72B reached 88.0% IFEval Loose Instruction (Acc.); the reported gains span SFT, Offline DPO, and Online DPO in self-alignment and strong-to-weak distillation.Source: §1 · p. 2

2 RELATED WORKS

• Instruction-following evaluation spans verifiable, compositional, format-related, refuting, and fine-grained instructions, with a reported gap between open-source and proprietary LLMs.The cited passage lists the evaluation contexts but does not detail their benchmarks.Source: §2 · p. 2
• Open-source instruction-following research has used distillation from proprietary LLMs, but related approaches rely on proprietary models for response distillation or judgment.The passage says this reliance limits their potential and subjects them to OpenAI’s terms of use.Source: §2 · pp. 2–3
• Execution feedback has been used for tool-use and coding alignment; AUTOIF extends the paradigm by generating verification functions and unit tests for natural-language instructions.Examples cited include unit-test feedback for code synthesis, debugging suggestions, and learning from execution results.Source: §2 · p. 3

3.1 PRELIMINARIES

• Verifiable instructions have code-executable checks that return whether a response follows the instruction, enabling supervision through self-generated checks and execution feedback.The paper reports benefits for handling more complex instructions that simple code snippets cannot verify.Source: §3.1 · p. 4

3 AUTOIF

• AUTOIF synthesizes instruction-following data through self-evolution, rejection sampling, and execution feedback, with automated augmentation and verification at both instruction and query levels.The framework is evaluated with three training strategies and two experimental settings.Source: §3 · p. 3, §3.1 · p. 4

3.2 INSTRUCTION AUGMENTATION AND VERIFICATION

• AUTOIF starts with handwritten single-constraint seed instructions, expands them with LLM rewrites, and combines the rewritten and seed instructions after deduplication.The seed instructions are denoted Dseed; the combined set is Dins.Source: §3.2 · p. 4
• For each instruction, an LLM generates verification functions and test cases; cross-validation filters samples using compilation and accuracy criteria, followed by back-translation to check instruction-function consistency.Contradiction between an instruction and its back-translated form is filtered to preserve intent consistency.Source: §3.2 · pp. 4–5

3.3 QUERY AUGMENTATION AND VERIFICATION

• AUTOIF pairs verified instructions with ShareGPT queries, augments the query inputs and responses, then checks whether responses satisfy the instruction constraints.The query-level verification requires responses to exceed 0.5 accuracy across verification functions.Source: §3.3 · p. 5
• A separate query-quality filter removes instruction-query-response samples with matching scores below 8, addressing semantic conflicts that a simple NLI model may miss.The LLM assigns matching scores on a 1-to-10 scale.Source: §3.3 · p. 5

3.4 TRAINING STRATEGIES

• SFT trains on the filtered input-response pairs, where each input concatenates an instruction and a user query.The paper also mines positive and negative pairs from instruction- and query-level quality filtering for preference training.Source: §3.4 · p. 6
• Offline DPO applies preference optimization to an SFT model using mined preferred and dispreferred samples, while iterative online DPO samples responses and ranks them by average verification pass rates.The online process uses verification functions as filtering standards across training iterations.Source: §3.4 · p. 6

4.1 EXPERIMENTAL SETUP

• Experiments use Qwen2 and LLaMA3 models in strong-to-weak distillation and self-alignment, evaluating instruction following and general capabilities.Evaluation covers IFEval, FollowBench, C-Eval, MMLU, GSM8k, and HumanEval.Source: §4.1 · pp. 6–7

4.2 MAIN RESULTS

• AUTOIF improves instruction-following performance across models, self-alignment and strong-to-weak distillation configurations, and training methods on two benchmarks.Table 1 presents the main results on two instruction-following and four general benchmarks.Source: Table 1 · p. 7, §4.2 · p. 7

4.4 ANALYSES

• With 1/64 DPO data, Qwen2-7B reaches nearly 55% in loose prompt accuracy, an increase of 11.4% pts; instruction-following improves with more data.Figure 4 presents the SFT and DPO data scaling analysis.Source: Figure 4 · p. 8, §4.4 · p. 9

4.2 MAIN RESULTS

• Online DPO outperforms Offline DPO; larger models typically gain more on FollowBench, while general abilities are preserved across training setups.The authors report slight gains in general abilities in on-policy settings and attribute preservation largely to incorporating ShareGPT data during synthesis.Source: §4.2 · p. 8

4.3 QUALITY ABLATION STUDY

• GPT-4 supervision gains over 15% in the loose prompt in IFEval versus Qwen72B supervision for strong-to-weak distillation.The supervision model performs text augmentation, verification-code generation, and quality assessment.Source: §4.3 · p. 8, Table 2 · p. 8
• Higher pass-rate thresholds reduce SFT data; instruction-level performance improves, while query-level performance peaks at 0.8 and declines beyond 1.The authors describe this as a trade-off between data quality and quantity.Source: §4.3 · pp. 8–9
• Removing any quality-filtering process reduces performance; cross-verification removal causes the largest single-component drop, while removing overall filtering causes a larger drop still.The authors characterize instruction- and query-level filtering as mutually reinforcing.Source: Table 3 · p. 8, §4.3 · p. 9

4.4 ANALYSES

• Both reported contamination rates are lower than ShareGPT’s; supervision-model coding ability is consistent with data-quality pass rate and final instruction-following capability.The authors report no contamination between AUTOIF-generated training samples and the IFEval and FollowBench test sets.Source: §4.4 · p. 9

• AUTOIF uses self-instruct and rejection sampling to strengthen seed-instruction supervision, with self-generated execution feedback filtering data quality.Source: §5 · p. 9
• The study evaluates AUTOIF through three training strategies and two alignment settings.Source: §5 · p. 9
• Across IFEval and FollowBench, performance significantly improves in all settings, with the first LLM exceeding 90% loose-instruction accuracy.Source: §5 · p. 10

• Cross-instruction construction is outside the paper’s focus, despite results on two instruction-following benchmarks demonstrating generalizability to complex tasks.Source: LIMITATIONS · p. 10
• AUTOIF scales to over 10,000 instructions, while automating and scaling cross-instruction tasks remains future work.Source: LIMITATIONS · p. 10
• The proposed extension combines multiple simple instructions into cross-instructions before AUTOIF augmentation and quality filtering.The authors describe this as a potential strategy to amplify the method’s effectiveness.Source: LIMITATIONS · p. 10

ETHIC CONSIDERATION

• Malicious prompts can lead the model to produce harmful or inappropriate outputs during rejection sampling.The authors call for ethical guidance and safeguards to prevent misuse and reduce harmful outputs.Source: ETHIC CONSIDERATION · p. 10

A SEED INSTRUCTIONS

• The seed instruction set is presented in the Appendix, and Figure 5 shows examples of seed instructions.Source: ETHIC CONSIDERATION · p. 10, Figure 5 · p. 15

ETHIC CONSIDERATION

• The authors report that concatenated ShareGPT queries underwent multiple quality-filtering steps to minimize potential safety and ethical risks.Source: ETHIC CONSIDERATION · p. 10

B IMPLEMENTATION DETAILS

• SFT uses full fine-tuning for three epochs, with learning rates of 7e-6 and batch sizes of 128 or 512 depending on model scale.Training uses bf16, an 8192-token context, linear scheduling with 20 warm-up steps, weight decay 0.1, DeepSpeed ZeRO Stage 3, and Flash-Attention 2.Source: B IMPLEMENTATION DETAILS · p. 15
• DPO runs for two epochs with learning rate 5e-7, batch size 64, sigmoid loss with beta 0.3, and a 4096-token context.It uses a cosine scheduler, 0.1 warm-up ratio, bf16, DeepSpeed ZeRO Stage 3, and Flash-Attention 2.Source: B IMPLEMENTATION DETAILS · p. 15
• Experiments use A100 and H800 GPUs; HumanEval uses pass@1 with greedy decoding, GSM8K uses zero-shot accuracy, and results average five random seeds.Qwen2-7B and LLaMA3-8B use 8 A100 GPUs; Qwen2-72B-Instruct and LLaMA3-70B-Instruct use 64 H800 GPUs.Source: B IMPLEMENTATION DETAILS · p. 15

C DETAILS OF AUTOIF

• Instruction-level self-instruct uses RFT with K=100, while NLI filtering removes samples predicted as Contradiction, approximately 15%.The NLI filter is mDeberta, and automated quality cross-verification applies four criteria described in the main text.Source: C DETAILS OF AUTOIF · p. 15
• For each instruction, query-level sampling selects 16 ShareGPT samples and performs response rejection sampling with K=8; query consistency scores below 8 are filtered.Instruction-following verification uses the two standards described in the main text.Source: C DETAILS OF AUTOIF · p. 15

D CASE STUDY OF DATA COMBINATION

• All model-based and rule-based data-combination metrics are lower than ShareGPT’s, which the authors take as confirmation that AutoIF has no test-set data combination.The study also reports the top five training-test n-gram overlaps for IFEval and Followbench.Source: D CASE STUDY OF DATA COMBINATION · p. 16
• Figure 6 presents the top five training-test n-gram overlaps for IFEval and Followbench.The accompanying analysis uses n-gram overlap and reports that the data-combination metrics are lower than ShareGPT’s.Source: Figure 6 · p. 16, D CASE STUDY OF DATA COMBINATION · p. 16

E PROMPT TEMPLATES

• The instruction-generation prompt requests format-focused, Python-verifiable instructions and excludes style-oriented examples such as metaphor or translation.It asks for one instruction per line, using seed instructions as examples.Source: E PROMPT TEMPLATES · p. 16
• The verification-code prompt asks for an evaluate function and three test cases, including expected true and false outputs.The prompt specifies JSON fields for the function and cases.Source: E PROMPT TEMPLATES · pp. 16–17
• For back translation, the prompt converts evaluation-function code into corresponding instruction text, including a request to turn supplied functions into a list of instructions.Examples include function code and cases.Source: E PROMPT TEMPLATES · p. 17
• The response-generation prompt conditions answers on both an instruction and a query, while separate query-quality judging scores how helpful and relevant a response is.The judging prompt requests a 0–10 score after analysis.Source: E PROMPT TEMPLATES · p. 17

F CASE STUDY OF AUTOIF

• The illustrated data format includes a query, a response, and a verification function.Source: F CASE STUDY OF AUTOIF · p. 17, Table 6 · p. 18
• The described response is paired with verification functions whose accuracy exceeds 0.8.Source: F CASE STUDY OF AUTOIF · p. 17
• Table 6 provides examples of AutoIF’s data formats.The accompanying description identifies query, response, and verification function fields.Source: F CASE STUDY OF AUTOIF · p. 17, Table 6 · p. 18

G BASELINES & DATASETS

• Model background covers LLaMA3’s expanded training data, context length, and vocabulary, and Qwen2’s multilingual, coding, mathematics, and long-context capabilities.The Qwen2-72B-Instruct context length is reported as up to 128K tokens.Source: G BASELINES & DATASETS · pp. 18–19
• Other baseline descriptions include Mistral-7B’s GQA and sliding-window attention, Mixtral-8×7B’s sparse mixture of experts, and GPT-3.5/GPT-4’s RLHF training.The text says Mixtral-8×7B (Jiang et al., 2024a) uses GQA like Mistral 7B.Source: G BASELINES & DATASETS · p. 19
• ShareGPT contains 86K human-chat histories; 20,000 samples were selected to train the LLaMA3-8B and Qwen2-7B ShareGPT baselines.The setup uses a cleaned raw ShareGPT dataset and Vicuna preprocessing.Source: G BASELINES & DATASETS · p. 19
• HumanEval reports greedy-decoding Pass@1, while HumanEval+ increases average test cases per problem from about 9.6 to 774.8.Source: G BASELINES & DATASETS · p. 19
• The evaluation overview describes MMLU’s 57 subjects and C-Eval’s 52 disciplines, alongside GSM8K’s grade-school math problems.C-Eval also includes the challenging C-Eval Hard subset; GSM8K has 7,473 training and 1,319 test samples.Source: G BASELINES & DATASETS · p. 19