Source-linked AI summary
Learning to Solve and Verify: A Self-Play Framework for Code and Test Generation
Zi Lin, Sheng Shen, Ilia Kulikov, Jingbo Shang, Jason Weston, Yixin Nie
TL;DR
Limited high-quality coding data and unreliable self-generated data motivate better verification for synthetic training. SOL-VER uses iterative self-play in which one model jointly generates solutions and tests, improving both capabilities. On MBPP and LiveCodeBench with Llama 3.1 8B, it reports average relative improvements of 19.63% for code generation and 17.49% for test generation.
Problem
Scarce high-quality coding data and accumulated errors in self-generated training outputs create a need for effective synthetic-data verification.
Source: §1 · p. 1Method
SOL-VER iteratively trains one model as both a code solver and a test verifier using execution feedback, SFT, and DPO preference pairs.
Source: §3.1 · p. 4Results
19.63% and 17.49% average relative improvements were achieved for code and test generation, respectively, on MBPP and LiveCodeBench.
Source: §1 · p. 2, §4.2 · p. 6Takeaways & Limitations
The framework improves both code and test generation without human annotations or larger teacher models.
Source: §1 · p. 2The framework’s applicability beyond function-level generation and its generalizability across larger and more diverse models remain for future evaluation.
Source: §6 · p. 9
Abstract
from arXiv · showhide
Recent advances in large language models (LLMs) have improved their performance on coding benchmarks. However, improvement is plateauing due to the exhaustion of readily available high-quality data. Prior work has shown the potential of synthetic self-instruct data, but naively training on a model's own outputs can cause error accumulation, especially in coding tasks, where generalization may collapse due to overly simple or erroneous training data, highlighting the need for rigorous quality checks on synthetic data. In this work, we explore an effective approach whereby the model itself verifies the correctness of its own data. We thus propose Sol-Ver, a self-play solver-verifier framework that jointly improves a single model's code and test generation capacity. By iteratively refining code (LLM-as-a-solver) and tests (LLM-as-a-verifier) together, we boost both capabilities without relying on human annotations or larger teacher models. Experiments with the Llama 3.1 8B model demonstrate substantial performance enhancements, achieving average relative improvements of 19.63% in code generation and 17.49% in test generation on MBPP and LiveCodeBench.
1. Introduction
The paper addresses scarce high-quality coding data and the solver–verifier imbalance by using iterative self-play to improve code and test generation together. SOL-VER trains both roles from execution-based feedback, achieving relative gains on both tasks without human annotations or larger models.
1. Introduction
- Data scarcity has become a key bottleneck as high-quality online coding resources and supervised sources saturate.Source: §1 · p. 1
- Self-generated training data can accumulate errors, creating a need for effective verification of synthetic code.Teacher-based synthetic data also depends on stronger models with greater parameter size and computation costs.Source: §1 · p. 1
- LLM test-generation ability lags code-generation ability because fine-tuning data predominantly emphasizes code generation rather than diverse, high-quality unit tests.Source: §1 · p. 1
- SOL-VER has one LLM generate solutions and unit tests for the same problems, then uses execution outcomes to construct training feedback.Passed examples support SFT, while passed and failed examples form DPO preference pairs for both generation roles; the process repeats iteratively.Source: §1 · pp. 1–2
- 19.63% and 17.49% average relative improvements were achieved for code and test generation, respectively, on MBPP and LiveCodeBench.The results use Llama 3.1 8B and do not rely on human-annotated data or larger models.Source: §1 · p. 2
- The framework is presented as jointly improving solver and verifier capabilities through synthetic preference data and self-alignment without external annotations or teacher models.Source: §1 · p. 2
2. Related Work
Related work uses synthetic data, execution feedback, self-correction, and preference optimization to improve coding models. SOL-VER differs by co-evolving the code solver and test verifier rather than relying on static or externally generated verification data.
2. Related Work
- SELF-INSTRUCT generates instruction–output pairs from a small seed dataset using a pretrained LLM, establishing a broader synthetic-data precedent.Source: §2 · p. 2
- Earlier synthetic-code approaches distill data from stronger teacher models, including ChatGPT or GPT-4, into weaker student models.Examples include CODEALPACA and CODELLAMA, while other methods increase instruction complexity or draw problems from open-source code.Source: §2 · p. 2
- Training on a model’s own generated data can degrade performance, motivating execution-based validation and methods such as CodeT, Self-Debug, Llama 3.1 self-correction, RLTF, and CodeDPO.Source: §2 · p. 2
- SOL-VER uses both passing and failing model-generated responses as chosen–rejected pairs for DPO and is complementary to self-correction and RLTF.Improved unit tests can provide more accurate execution feedback for those settings.Source: §2 · p. 2
- SOL-VER jointly improves code solving and test verification, addressing the limitation that weak or static self-generated tests can produce suboptimal preference learning.Its self-play mechanism iteratively enhances the verifier’s ability to generate more discerning and comprehensive tests.Source: §2 · pp. 2–3
3. A Self-play Solver-verifier Framework
Sol-Ver frames one LLM as both a solver that generates code and a verifier that generates tests, training the two roles through correctness-based feedback. Its pipeline constructs diverse problems, tests, and preference pairs, then applies supervised and preference tuning to improve both roles.
3.1. Problem Formulation
- The solver generates candidate code for a problem, while the verifier generates challenging tests intended to distinguish correct from incorrect solutions.The solver seeks code that passes the verifier's tests; the verifier seeks tests that fail incorrect code.Source: §3.1 · p. 3
- A score in [0, 1] measures the fraction of tests passed, and examples are labeled chosen when the candidate solution passes every test.The score is computed by executing the candidate code on generated tests.Source: §3.1 · p. 3
- The framework first uses SFT on correct solution-test pairs to train the same model to generate solutions and meaningful verification tests.Chosen pairs provide the desired behavior for both solver and verifier roles.Source: §3.1 · pp. 3–4
- It then applies DPO-style preference tuning by pairing chosen outputs with rejected code or test alternatives for the solver and verifier.DPO strengthens both roles using pairwise comparisons; other preference-tuning methods could also be used.Source: §3.1 · p. 4
3.2. Synthetic Data Generation
- Synthetic problems are generated from diverse code snippets and benchmark-inspired templates, then completed with starter code and function signatures, yielding 103,280 deduplicated descriptions.The prompts accommodate multiple descriptions of equivalent programming tasks.Source: §3.2 · p. 4
- Unit tests are generated through staged input and output generation, with majority voting, chain-of-thought reasoning, coverage optimization, and output diversification improving test quality.Rejected examples are constructed by cross-validating generated code and tests, although disagreement does not always reveal which side is wrong.Source: §3.2 · pp. 4–5
4. Experiments
Across MBPP and LiveCodeBench, SOL-VER improves code and unit-test generation over iterations, while base-model evaluation exposes a solver–verifier gap and iteration agreement supports ensembling.
4.1. Experimental Setup
- The experiments use Llama 3.1 8B on Python benchmarks, evaluating code with Pass% and tests with accuracy and false positive rate.The benchmarks include MBPP and LiveCodeBench; MBPP test-output prediction uses its gold unit tests.Source: §4.1 · p. 5
4.2. Evaluating SOL-VER
- 19.63% and 17.49% relative improvements were achieved for code and test generation, respectively, across MBPP and LiveCodeBench.SOL-VER consistently improves code pass rate, test accuracy, and false positive rate across iterations.Source: §4.2 · p. 6, Table 1 · p. 6
- The SFT+DPO configuration suggests that incorporating negative examples during preference tuning helps the model learn from errors and refine generation strategies.The comparison is part of SOL-VER’s iterative evaluation against SFT-only training.Source: §4.2 · p. 6
4.3. Evaluating the Base Model
- Unit-test generation significantly underperforms code generation on the same MBPP examples, revealing a solver–verifier performance gap.This gap motivates improving verification while leveraging strong code-generation ability.Source: §4.3 · pp. 6–7
- Majority voting and chain-of-thought prompting improve unit-test quality, but their combination remains insufficient for optimal verification and can reduce reranked code-generation performance.The combined strategies produce the best reported base-model test-generation result, yet generated tests still harm downstream reranking.Source: §4.3 · p. 7
4.4. Evaluating Agreement between Iterations
- Models from the first and second iterations generate largely consistent unit tests, and an ensemble passing both test sets modestly improves over the second iteration alone.The ensemble is consequently used to generate synthetic data for the third iteration.Source: §4.4 · p. 7, Table 3 · p. 7
5. Ablation Study
SOL-VER’s ablations examine prompt coverage, iterative test refinement, and preference-pair scoring, showing broader synthetic problem coverage and improved test quality across iterations. The scoring analysis favors retaining fewer, higher-quality preference pairs over indiscriminately enlarging the dataset.
5.1. Prompt and Test Analysis
- Synthetic problem descriptions broadly cover topics, difficulty levels, and programming paradigms represented in MBPP, APPS, and LiveCodeBench.The comparison uses Gecko sentence embeddings and PCA visualization.Source: §5.1 · p. 7, Figure 3 · p. 8
- Higher pass rates across iterations primarily reflect fewer assertion errors and more accurate predicted test outputs.Table 4 illustrates verifier corrections and improvements across iterations for two programming problems.Source: §5.1 · p. 7, Table 4 · p. 8
5.2. Discussion on the Scoring Function
- Only 45% of Iteration 1 examples produce agreed solution-test pairs, limiting the number of available preference-tuning pairs.Agreement requires at least one generated code solution to pass all generated tests.Source: §5.2 · p. 8
- Soft pass-rate selection explores thresholds above 0, 0.5, and 0.75 while varying the rejected score from random to median alternatives.The chosen pair must have a higher score than the rejected pair and meet the reliability threshold.Source: §5.2 · p. 8
- Randomly selecting a high-quality-test threshold degrades performance despite increasing data size, whereas rejected-score selection is less sensitive.The study retains SOL-VER’s original settings and concludes that synthetic-data quality matters more than quantity.Source: §5.2 · p. 8, Table 5 · p. 9
6. Conclusion
The conclusion presents SOL-VER as a self-play framework addressing data scarcity and the solver-verifier performance gap through iterative co-evolution. It reports gains without human annotations or larger teacher models while identifying several directions for broader applicability and efficiency.
6. Conclusion
- SOL-VER jointly improves code synthesis and test generation by iteratively co-evolving the solver and verifier.The framework generates synthetic code-test pairs without human annotations or larger teacher models.Source: §6 · p. 9
- The authors identify controlled input generation, more complex coding scenarios, lower computational overhead, and larger diverse models as future directions.These directions target synthetic-data quality, applicability, efficiency, and generalizability.Source: §6 · p. 9
A. Prompt Template
The prompt templates generate programming problems, function signatures, inputs, tests, and outputs through structured examples and instructions. The appendix also includes reusable problem and code-generation templates.
A. Prompt Template
- Separate templates request problem descriptions, function signatures, test inputs, and test outputs as distinct generation stages.The appendix labels each prompt component explicitly.Source: A. Prompt Template · pp. 11–13
- The appendix also provides a code-snippet template and a programming-problem template based on summing last digits of page numbers divisible by m.A short code fragment accompanies the natural-language problem template.Source: A. Prompt Template · p. 11
- Problem-generation instructions transform code snippets and templates into self-contained Python programming problems without solutions or explicit examples.The instructions require a single-function problem and permit deviation from the inspiration.Source: A. Prompt Template · p. 11
- Input-generation prompts cover general and empty-list cases for finding the longest string, while additional cases address ASCII-sequence checking.The prompt asks the model to handle different sequence types for the ASCII task.Source: A. Prompt Template · p. 12
- Test-generation prompts include explicit analyses and assertions for multiple coding questions, including tuple overlap, primality, and string filtering.The examples pair each input with an expected assertion output.Source: A. Prompt Template · p. 13
B. Performance Change Across Iterations
The supplied passage identifies a figure presenting SOL-VER framework performance, but it does not provide the plotted metrics, values, or iteration comparisons.
B. Performance Change Across Iterations
- Figure 5 presents the performance of the Solver-Verifier Framework, without reported values or specific comparisons in the supplied passage.The figure reference alone does not establish an outcome.Source: Figure 5 · p. 14