PaperlayerView exact revision on arXiv

Source-linked AI summary

Genetic Instruct: Scaling up Synthetic Generation of Coding Instructions for Large Language Models

Somshubra Majumdar, Vahid Noroozi, Mehrzad Samadi, Sean Narenthiran, Aleksander Ficek, Wasi Uddin Ahmad, Jocelyn Huang, Jagadeesh Balam, Boris Ginsburg

arXiv:2407.21077v3cs.CLcs.LGcs.NE

TL;DR

Expert-curated coding instructions are costly to produce, motivating synthetic data for LLM alignment. Genetic-Instruct evolves seed instructions into coding pairs and evaluates them with LLMs; models trained on its dataset outperform other instruction-generation methods and some public SFT datasets across coding benchmarks.

  • Problem

    Paired instruction-solution examples support LLM alignment, but diverse and complex coding instructions can be expensive to create with expert input.

    Source: §1 · p. 1

  • Method

    Genetic-Instruct uses LLM-driven crossover and mutation to generate coding instruction-solution pairs from seed instructions, with LLM evaluation of pair quality.

    Source: §1 · pp. 1–2

  • Results

    Models trained on the synthetic dataset outperform other instruction-generation methods and some public SFT datasets across coding benchmarks.

    Source: §1 · p. 2, §5 · p. 7

  • Takeaways & Limitations

    The authors released a 7.5M synthetic instruction-solution dataset to support open-source LLM development.

    Source: §5 · p. 7

    Execution-based code correctness checking is challenging because of language constraints, missing dependencies, or unavailable larger codebases.

    Source: §3.3 · p. 4

Abstract

from arXiv · show

Large Language Models (LLMs) require high quality instruction data for effective alignment, particularly in code generation tasks where expert curated datasets are expensive to produce. We present Genetic-Instruct, a scalable algorithm for synthesizing large-scale, high quality coding instructions using evolutionary principles. Starting from a small set of seed instructions, Genetic-Instruct generates diverse and challenging instruction-code pairs by leveraging an Instructor-LLM for generation, a Coder-LLM for code synthesis, and a Judge-LLM for automatic quality evaluation. Our proposed approach is highly parallelizable and effective even with a small seed data and weaker generator models. We generated more than 7.5 million coding instructions with the proposed approach. Then we evaluated it by fine-tuning LLMs with the synthetic samples and demonstrated a significant improvement in their code generation capability compared to the other synthetic generation approaches and publicly available datasets. Our results highlight the efficiency, scalability, and generalizability of the Genetic-Instruct framework.

1 Introduction

Genetic-Instruct uses evolutionary operations and LLM-based evaluation to generate coding instruction-solution pairs from a small seed set. Models trained on its synthetic data outperform other instruction-generation methods and some public SFT datasets across coding benchmarks.

1 Introduction

  • Expert-curated coding instructions are costly to produce, motivating synthetic instruction generation as an alternative.Paired instruction-solution examples support LLM alignment for code generation.Source: §1 · p. 1
  • Genetic-Instruct starts from a small seed set and uses crossover and mutation to generate new instruction-code pairs.Crossover expands topic coverage, while mutation locally varies an instruction; generated instructions can seed later operations.Source: §1 · pp. 1–2
  • A Coder-LLM produces solutions, and a Judge-LLM evaluates pair correctness and quality before accepted samples enter the population.The process continues until the target population size is reached.Source: §1 · p. 2
  • The pipeline runs multiple population colonies in parallel and can repeat generations using prior outputs as seeds.The final population is decontaminated.Source: Figure 1 · p. 1, §1 · p. 2
  • Models trained on Genetic-Instruct data outperform other instruction-generation methods and some public SFT datasets across coding benchmarks.The work reports generating more than 7.5 million samples from 512 seed questions.Source: §1 · p. 2

2 Previous Works

Prior synthetic-instruction approaches either target general tasks, evolve instructions for greater complexity, or derive prompts from code. Code-seeded methods rely on large, high-quality processed code samples, which can be challenging for less common languages.

2 Previous Works

  • Self-Instruct generates instruction-output pairs from a small seed set for general tasks, but its samples are often too simple to require additional solution steps.It can enhance topic coverage but is not coding-focused.Source: §2 · p. 2
  • Evol-Instruct uses mutation to create more complex and diverse tasks, and WizardCoder adapts this approach to coding.The methods use meta-instructions to increase reasoning depth, impose constraints, or promote conceptual evolution.Source: §2 · p. 2
  • Code-seeded methods generate instructions from code snippets, but depend on large, high-quality processed code samples that may be scarce for less common languages.INVERSE-CODER matches instructions to code, while OSS-Instruct and WaveCoder generate code-inspired instructions.Source: §2 · p. 2

3 Genetic-Instruct

Genetic-Instruct evolves a seed population through LLM-driven mutation, crossover, and selection, then generates, validates, and decontaminates instruction-code pairs. Its design also uses parallel colonies and periodic generation aggregation to scale the process.

3 Genetic-Instruct

  • The core loop evolves high-quality seed instructions through mutation, crossover, and selection, using LLMs with in-context learning to form new generations.Generated pairs are assessed by a Judge-LLM, accepted samples can become seeds, and the process continues until the target population size is reached.Source: §3 · pp. 2–3

3.1 Mutation Operation

  • For mutation, the method randomly selects one of five predefined Evol-Instruct tasks for a chosen sample and applies it with an LLM to generate a new instruction.The five tasks were introduced by (Luo et al., 2024), with minor prompt modifications for the Instructor-LLM.Source: §3.1 · p. 3

3.2 Crossover Operation

  • Crossover uses multiple few-shot seed instructions in one Instructor-LLM call to produce multiple diverse new instructions.The operation draws on Self-Instruct and Unnatural Instructions.Source: §3.2 · pp. 3–4

3.3 Code Generation

  • The Coder-LLM generates code for new instructions, and parser or compiler checks filter out code that cannot be parsed or compiled.Execution-based correctness checking is challenging because of language constraints, missing dependencies, or unavailable larger codebases.Source: §3.3 · p. 4

3.4 Fitness Function

  • The fitness function combines post-processing checks with a Judge-LLM binary score for whether a solution meets minimum requirements.The Judge-LLM receives the instruction and code solution, with few-shot in-context learning and Chain-of-Thought prompting.Source: §3.4 · p. 4

3.5 Scaling Up the Process

  • Multiple colonies evolve independently from the same seed population, run in parallel, and merge after reaching the desired size.The process synchronizes colonies periodically and selects distinct seed examples within a batch.Source: §3.5 · p. 4

3.6 LLM Decontamination

  • Decontamination finds each synthesized question’s nearest benchmark question by embedding similarity, then uses an LLM to check for paraphrase matches.Question-pair order is reversed for a second check, and a synthesized question is removed if either pair is judged similar.Source: §3.6 · p. 4

4 Experiments

The experiments evaluate Genetic-Instruct-generated coding data by fine-tuning LLMs and testing them on four code-generation benchmarks. They report higher average accuracy than the best-performing public dataset, gains as dataset size grows, and ablation results favoring both genetic operations together.

4 Experiments

  • The evaluation fine-tuned base LLMs with supervised fine-tuning and tested them on HumanEval, MBPP, HumanEval+, and MBPP+.The + benchmarks extend the original test sets with additional test cases.Source: §4 · p. 4

4.1 Experimental Settings

  • Experiments generally began with 512 Tiger-Leetcode seed questions and constrained generated solutions to Python.Generation used 20 parallel colonies; default mutation and crossover batch sizes were 100 and 10, respectively.Source: §4 · p. 5, §4.1 · p. 5

4.2 Performance Evaluation

  • More than 3 million samples outperformed Llama3.1-8B-Instruct, while accuracy gains began to plateau beyond approximately 6 million.Accuracy averages model performance across four benchmarks; the synthetic dataset reached around 7.8 million samples.Source: §4.2 · p. 5, Figure 2 · p. 5
  • 69.7% average accuracy for the five-generation dataset exceeded OpenCoder Stage 1’s 62.9%; the 4M-sample dataset achieved 68.0%.The comparison uses average accuracy across the benchmarks.Source: Table 1 · p. 6, §4.2 · p. 7

4.3 Ablation Study

  • Combining crossover and mutation yielded the highest average accuracy across benchmarks, although Mutation-Only slightly outperformed the full method on HumanEval.Each setup used 4 million samples to fine-tune Llama3.1-8B Base.Source: Table 2 · p. 6, §4.3 · p. 7

4.4 Influence of the Generator Model

  • Qwen models outperformed the Mixtral family across most benchmarks, and Qwen-7B performed closely to Qwen-32B.The generator-model comparison used 1.5 million samples per experiment and evaluated Llama3.1-8B-Base and Qwen2.5-7B-Base.Source: §4.4 · p. 7, Table 3 · p. 7

5 Conclusion

Genetic-Instruct is an evolutionary algorithm designed for parallel generation of synthetic coding instructions. The authors report improved code-generation performance and release a 7.5M instruction-solution dataset for open-source development.

5 Conclusion

  • Genetic-Instruct applies evolutionary principles to generate synthetic coding instructions for LLMs.Source: §5 · p. 7
  • The authors report that combining crossover and mutation achieves the best performance in their ablations.They describe Genetic-Instruct as designed for parallel generation and report consistent effectiveness against baseline methods and public datasets.Source: §5 · p. 7
  • The publicly released dataset contains 7.5M synthetic instruction-solution pairs for open-source LLM development.Source: §5 · p. 7

A Mutation Prompts

The mutation prompts ask the Instructor-LLM to make a programming question harder while producing only a new instruction. Their rules offer several ways to alter the original task, including adding constraints or changing a requirement.

A Mutation Prompts

  • Mutation prompts request one harder programming instruction and prohibit hints, solutions, or outputs.The prompt asks for a slight difficulty increase.Source: A Mutation Prompts · p. 10
  • One prompt adds constraints and requirements, targeting approximately 10 additional words.Source: A Mutation Prompts · p. 10
  • Other mutation rules replace a common task requirement with a more specific one or add reasoning steps to a short task.Source: A Mutation Prompts · p. 10
  • The erroneous-code operation adds wrong Python reference code as misdirection, then asks for the correct solution without supplying it.Source: A Mutation Prompts · p. 10

B Crossover Prompt

The crossover prompt requests 20 diverse coding-task instructions for evaluating a GPT model. It specifies variation in task form and language, with realistic inputs and appropriate outputs where applicable.

B Crossover Prompt

  • The crossover prompt requests 20 coding-task instructions intended for GPT-model evaluation.Source: B Crossover Prompt · p. 11
  • It seeks diversity in verbs, language, and task types, including generation, classification, editing, and optimization.Source: B Crossover Prompt · p. 11
  • For tasks needing input, the prompt calls for a realistic specific example under 100 words and an appropriate output; otherwise it permits a no-input marker.Source: B Crossover Prompt · p. 11

C Prompts for Coder-LLM

Figure 5 is a prompt template for code generation with the Coder-LLM.

C Prompts for Coder-LLM

  • Figure 5 presents a prompt template for code generation with the Coder-LLM.Source: Figure 5 · p. 12
  • The template is identified as a prompt for the Coder-LLM.Source: Figure 5 · p. 12
  • The supplied description identifies the template's purpose as code generation.Source: Figure 5 · p. 12

D Fitness Prompt for Judge-LLM

Figure 6 is a prompt template for quality judgment by the Judge-LLM.

D Fitness Prompt for Judge-LLM

  • Figure 6 presents a prompt template for code quality judgment with the Judge-LLM.Source: Figure 6 · p. 12
  • The template is identified as a prompt for the Judge-LLM.Source: Figure 6 · p. 12
  • The supplied description identifies the template's purpose as judging code quality.Source: Figure 6 · p. 12

E Decontamination Prompt

The section presents a contamination-detection prompt that asks whether two coding problems are the same.

E Decontamination Prompt

F Evaluation Prompts

The evaluation prompts ask for Python code generation on MBPP and MBPP+, and code completion on HumanEval and HumanEval+. Both include an instruction to continue the code in Python.

F Evaluation Prompts

Loading 2407.21077v3…