Source-linked AI summary
SpecCoder: Specification-Aware Code Generation with Curriculum Dual-Task Reinforcement Learning
Yixuan Li, Mingxuan Huang, Jiajing Wang, Weidong Yang, Xinyi Liu, Ben Fei, Lipeng Ma
TL;DR
Challenging code-generation tasks require models to understand rich requirements, but existing methods provide limited supervision for deriving structured specifications and grounding them in code behavior. SpecCoder combines specification-guided SFT with curriculum dual-task GRPO for generation and discrimination. It reports consistent improvements in standalone code generation and agent-based workflows across APPS, CodeContests, and xCodeEval.
Problem
Existing methods provide limited supervision for deriving structured specification analyses from raw requirements and ensuring that implementations reflect them.
Source: §1 · p. 3Method
SpecCoder uses specification-guided SFT to derive structured analyses and generate conditioned code, followed by curriculum dual-task GRPO for generation and discrimination.
Source: §1 · pp. 3–4Results
SpecCoder consistently improves standalone code generation and agent-based workflows across APPS, CodeContests, and xCodeEval, with agent gains of up to 11.3 percentage points in Pass@1 and 13.2 percentage points in AvgPassRatio over the base model.
Source: §1 · p. 4Takeaways & Limitations
Structured specification analyses provide useful intermediate guidance for code synthesis.
Source: §1 · p. 4The model's specification-understanding ability is limited by external teacher models used in data construction, which also adds API costs.
Source: §8.2 · p. 32
Abstract
from arXiv · showhide
Large language models (LLMs) have made substantial progress in code generation but still struggle with challenging programming tasks that require understanding rich natural language requirements. These requirements often specify problem goals, input/output formats, constraints, examples, and edge cases. Overlooking even one may produce executable but functionally incorrect code. Existing training-free methods mainly rely on prompting or agent-based workflows, while training-based methods typically optimize final code outputs. However, existing approaches provide limited supervision for learning the intermediate mapping from raw requirements to structured specifications and for grounding them in concrete implementation behavior. Consequently, models may omit critical constraints, and even when an explicit specification is produced, the implementation may fail to reflect it consistently. Motivated by this gap, we propose SpecCoder, a specification-aware two-stage training framework for code generation. SpecCoder first employs specification-guided SFT to train LLMs to derive structured specification analyses and generate code conditioned on them. It then introduces curriculum dual-task GRPO, which jointly optimizes specification-guided generation and discrimination to encourage stronger correspondence between specifications and code behavior. Experiments on APPS, CodeContests, and xCodeEval demonstrate the effectiveness of specification-aware training, with SpecCoder consistently improving both standalone code generation and agent-based workflows. Additional evaluations on BigCodeBench-Hard and ClassEval, alongside human evaluation and perturbation studies, further validate the role of structured specifications in guiding code generation and discrimination.
1. Introduction
SpecCoder addresses limited supervision for deriving structured specifications from rich requirements and aligning them with implementation behavior. Its two-stage training combines specification-guided generation with discrimination, and evaluations report gains in standalone and agent-based code generation.
1. Introduction
- Raw requirements can include goals, input/output formats, constraints, examples, edge cases, and implicit assumptions that must be understood for functionally correct code.A model may produce a reasonable analysis yet still generate code that ignores, simplifies, or contradicts its constraints.Source: §1 · pp. 2–3
- Existing training-based methods provide limited supervision for deriving structured specification analyses from raw requirements and using them to guide code generation.Prompt- and agent-based approaches primarily alter inference-time workflows, while execution or preference feedback often supplies sparse, outcome-level signals.Source: §1 · pp. 2–3
- SpecCoder uses specification-guided SFT to derive structured analyses and generate conditioned code, then curriculum dual-task GRPO to optimize generation and implementation discrimination.The curriculum progresses from easier samples to harder cases, then mixes difficulty levels to consolidate behavior and mitigate forgetting.Source: §1 · pp. 3–4
- SpecCoder improves standalone generation and agent-based workflows across APPS, CodeContests, and xCodeEval, with agent gains reaching 11.3 percentage points in Pass@1 and 13.2 percentage points in AvgPassRatio over the base model.Standalone results include Pass@1/AvgPassRatio scores of 0.186/0.346 on APPS, 0.091/0.207 on CodeContests, and 0.147/0.324 on xCodeEval.Source: §1 · p. 4
- Structured specification analyses serve as a shared intermediate representation connecting requirement understanding to code generation and implementation discrimination.The paper evaluates SpecCoder in standalone and agent-based settings across in-distribution and out-of-distribution benchmarks, with additional analyses of intermediate representations and specification dependency.Source: §1 · pp. 4–5
2. Related Work
Prior inference-time methods use prompts or agent workflows to handle complex requirements, while training-based approaches mainly optimize implementation outcomes. SpecCoder targets the remaining supervision gap with structured specification analyses and paired generation and discrimination objectives.
2.1. Training-Free Methods for LLM-Based Code Generation
- Prompt-based approaches add structured reasoning or iterative repair, but manually designed prompts may not robustly preserve critical requirement details.Examples include Chain-of-Thought prompting, Self-Debug, Self-Edit, Self-Planning, SCoT, Program of Thoughts [27], ArchCode, and 𝜇FiX.Source: §2.1 · p. 5
- Agent workflows such as MetaGPT, FlowGen, and Reflexion add analysis, feedback, or revision, but repeated interactions and execution increase inference cost.Specine [18] and REA-Coder [19] address requirement misunderstanding and specification alignment at inference time.Source: §2.1 · p. 5
2.2. Training-Based Methods for LLM-Based Code Generation
- Training-strategy methods organize tasks progressively or decompose capabilities, including StepCoder, RECRL [32], and FixAudit [33].These approaches target learning efficiency, exploration, or iterative code refinement.Source: §2.2 · p. 6
- Existing training signals emphasize final implementation outcomes, leaving limited supervision for understanding raw requirements and ensuring code reflects intermediate analyses.SpecCoder combines specification-guided SFT with curriculum dual-task GRPO to train structured analyses, conditioned generation, and discrimination.Source: §2.2 · p. 6
3. Problem Formulation
SpecCoder formulates code generation around raw requirements, structured specification analyses, and candidate implementations, using available tests as a proxy for functional correctness. Its generation and discrimination tasks make the intermediate analysis explicit and connect candidate behavior to the shared specification.
3. Problem Formulation
- For each requirement p, SpecCoder defines f(c, T_p) as the fraction of its test cases passed by candidate c and uses available tests as an executable proxy for observable functional correctness.The formulation defines P, S, and C as the spaces of raw requirements, structured specification analyses, and candidate implementations.Source: §3 · p. 6
- The test-pass signal is approximate, not a complete measure of requirement satisfaction.Available tests may not cover every semantic constraint.Source: §3 · pp. 6–7
3.1. Specification-Guided Code Generation Task
- Direct generation from raw requirements can overlook goals, constraints, edge cases, or implicit assumptions, so the generation task produces a structured analysis before code.The analysis makes requirement understanding explicit before implementation.Source: §3.1 · p. 7
- The generated specification analysis is an intermediate representation, not a unique ground-truth analysis.Source: §3.1 · p. 7
- Code is generated conditioned on both the raw requirement and the preceding specification analysis in the autoregressive context.Source: §3.1 · p. 7
3.2. Specification-Guided Code Discrimination Task
- The discrimination task compares two candidate implementations against the same raw requirement and structured specification analysis.Candidate pairs are constructed so one has a higher available-test pass rate than the other.Source: §3.2 · p. 7
- Candidates are randomly ordered to reduce position bias, and the model outputs a binary choice between them with respect to the shared specification.Source: §3.2 · p. 7
4. Methodology
SpecCoder uses structured specification analyses as an intermediate interface between raw requirements and code, with data construction followed by specification-guided SFT and curriculum dual-task GRPO. Training covers both generating code from specifications and discriminating between candidate implementations.
4.1. Data Construction Pipeline
- SpecCoder builds generation and discrimination datasets from APPS and CodeContests, reserving a split of the APPS test set for evaluation.The remaining APPS test subset is used only for data construction; CodeContests uses its complete training set.Source: §4.1 · p. 8
4.1.1. Specification-Guided Code Generation Dataset Dgen
- Dgen contains verified requirement–specification–implementation triplets, filtered through execution-based correctness checks and backward checks for specification consistency.When multiple triplets remain for a problem, one is selected at random; problems with none are discarded.Source: §4.1.1 · pp. 8–10
4.1.2. Specification-Guided Code Discrimination Dataset Ddisc
- Ddisc pairs candidate implementations under a shared requirement and specification, labeling the candidate with the higher test pass rate and grouping pairs by pass-rate gap.The gap defines Easy, Medium, and Hard difficulty levels for curriculum learning.Source: §4.1.2 · pp. 10–11
4.2. Two-Stage Training Paradigm
- The two-stage training paradigm first warm-starts the model with specification-guided SFT, then jointly improves generation and discrimination with curriculum dual-task GRPO.The two stages correspond to Figure 1’s Stage 2 and Stage 3.Source: §4.2 · p. 11
4.2.1. Specification-Guided SFT
- SFT trains the model to produce a structured specification analysis and then code conditioned on the requirement and analysis, applying loss to target tokens rather than input tokens.The target includes the specification analysis, implementation, and their delimiters; the raw requirement and its delimiters are masked from the loss.Source: §4.2.1 · pp. 11–12
4.2.2. Curriculum Dual-Task GRPO
- GRPO uses separate task-specific reward structures and normalizes advantages within each task to prevent reward-scale differences from dominating policy updates.Generation rewards combine format compliance, compilation, and test-based correctness; discrimination rewards combine format compliance and decision correctness.Source: §4.2.2 · pp. 12–13
- The GRPO curriculum moves from Easy samples, to Medium and Hard samples, then to a full mixture to consolidate behavior and mitigate forgetting.A fixed sample-level update budget keeps total sample exposure constant while the curriculum changes training order and phase composition.Source: §4.2.2 · pp. 13–14
5. Experimental Design
The experimental design evaluates specification-guided generation and discrimination on three benchmarks, using complementary correctness metrics and comparisons with prompting, training-based, and agent-based methods.
5.1. Datasets
- Experiments use APPS, CodeContests, and xCodeEval to construct specification-guided generation and discrimination datasets.Table 2 summarizes the final training and testing data distributions for the two tasks.Source: §5.1 · p. 14, Table 2 · p. 14
5.2. Evaluation Metrics
- Pass@1 measures whether a single generated solution passes all test cases, while AvgPassRatio measures the average fraction of test cases passed.The study reports 95% confidence intervals estimated by bootstrap resampling with 10,000 iterations.Source: §5.2 · p. 15
5.3. Baselines
- Baselines include prompting and training-based code-generation methods, plus four training-free agent frameworks used to test backbone compatibility.The agent frameworks are AgentCoder, MetaGPT, PairCoder, and Specine.Source: §5.3 · p. 15
5.4. Implementation Details
- Trainable methods share the Qwen2.5-7B-Coder-Instruct backbone, training split, evaluation protocol, hardware, and comparable optimization budgets.SpecCoder uses LoRA SFT followed by curriculum dual-task GRPO; Table 3 compares methods using Pass@1 and AvgPassRatio with 95% confidence intervals.Source: §5.4 · p. 16, Table 3 · p. 16
6. Results and Analysis
SpecCoder performs competitively or better in standalone code generation and improves agent-based workflows across the evaluated ID and OOD benchmarks. Ablations indicate that its training components, optimization strategy, and structured specifications contribute to performance.
6.1. Effectiveness of the SpecCoder Framework
- On APPS, SpecCoder scores Pass@1 0.186 and AvgPassRatio 0.346 versus CodePRM’s 0.186 and 0.340; on CodeContests, it ties FixAudit at Pass@1 0.091 but leads in AvgPassRatio, 0.207 to 0.162.On xCodeEval, it reaches Pass@1 0.147 versus FixAudit’s 0.143 and μFix’s 0.133, and has the highest AvgPassRatio at 0.324, 4.5 percentage points above the best baseline.Source: §6.1 · p. 17
6.2. Generalization as an Agent Backbone
- Across agent frameworks, SpecCoder improves ID and OOD results; CodeContests gains reach 10.9-point Pass@1 and 12.3-point AvgPassRatio, while Specine’s xCodeEval gains are 11.3 and 12.5 points.SpecCoder-based agents also maintain higher Pass@1 at each refinement round across the evaluated frameworks.Source: §6.2 · pp. 17–19, Figure 2 · p. 18, Table 4 · p. 19
6.3.1. Training Component
- Removing SFT lowers CodeContests Pass@1 from 0.091 to 0.043 (52.7%); removing GRPO decreases Pass@1 by 34.1%, and removing discrimination also degrades performance.The authors attribute SFT’s role to structural initialization and discrimination’s role to connecting specification satisfaction with code behavior.Source: Figure 3 · p. 20, §6.3.1 · pp. 20–21
6.3.2. Optimization Strategy
- Removing curriculum lowers CodeContests Pass@1 from 0.091 to 0.079; sequential generation-then-discrimination and discrimination-then-generation both underperform joint dual-task optimization.The authors describe curriculum as a stable, progressive path and joint optimization as allowing generation and discrimination to reinforce each other.Source: Figure 3 · p. 20, §6.3.2 · pp. 21–22
6.3.3. Intermediate Representation
- Under matched training conditions, SpecCoder (w/o Disc) is best or tied-best among generation-only variants; the full model adds 14.7 tokens and improves Pass@1 by 1.4, 3.0, and 2.0 points across the three datasets.On xCodeEval, SpecCoder (w/o Disc) scores 0.127, outperforming Code-Only, Free-Form, and Planning-CoT by 4.7, 2.4, and 3.4 percentage points, respectively.Source: §6.3.3 · pp. 22–24, Table 5 · p. 24
- Across five Specine iterations, structured-specification models maintain higher Pass@1 as gains for non-structured representations slow; at iteration 5, full SpecCoder leads the best non-structured representation by 5.2, 6.0, and 5.3 points on APPS, CodeContests, and xCodeEval.The authors describe structured specifications as a consistent reference for iterative generation, checking, and repair.Source: §6.3.3 · p. 24, Figure 5 · p. 25
7. In-depth Analysis
SpecCoder performs strongly across problem difficulties and realistic programming benchmarks, while alignment evaluations, specification perturbations, and case studies show how structured specifications relate to generation and discrimination outcomes.
7.1. Performance across Problem Difficulties
- Across difficulty levels, SpecCoder posts strong AvgPassRatio results and increases agent-based repair gains, including a 0.090 net gain on APPS Hard and 0.143 on xCodeEval Easy.Its Easy xCodeEval standalone scores are Pass@1 0.247 and AvgPassRatio 0.461; on Medium tasks, its AvgPassRatio is 0.171, while CodePRM has higher Pass@1 at 0.057 versus 0.043.Source: §7.1 · pp. 25–26
7.2. Specification Alignment and Dependency Analysis
- The analysis combines human assessment of requirement coverage and code consistency with controlled tests of generation and discrimination under specification perturbations.The broader evaluation examines whether model behavior depends on specification semantics, not merely their structured format.Source: §7.2 · p. 26
7.2.1. Specification Alignment Evaluation
- In blinded human evaluation, SpecCoder scores 4.55 for Requirement Coverage and 4.48 for Spec-Code Consistency, with weighted Cohen’s κ values of 0.81 and 0.77.The evaluation uses a 5-point Likert scale.Source: Table 6 · p. 27, §7.2.1 · p. 27
7.2.2. Dependency in Code Generation
- For code generation, Standard specifications perform best; Shuffled causes the largest overall degradation, with xCodeEval Pass@1 falling from 0.147 to 0.014.Corrupted specifications also generally underperform Removed and Empty; on APPS, Pass@1 falls from 0.170 to 0.132.Source: §7.2.2 · p. 28
7.2.3. Dependency in Code Discrimination
- For code discrimination, accuracy is 91% with Standard specifications, versus 68% for Empty, 60% for Removed, 43% for Shuffled, and 49% for Corrupted.All content perturbations score below Empty, indicating reliance on semantic correspondence rather than specification format alone.Source: §7.2.3 · pp. 28–29, Figure 7 · p. 29
7.3. Evaluation on Realistic Benchmarks
- SpecCoder leads the evaluated realistic settings: Pass@1 is 0.34 on BigCodeBench-Hard, 0.692 on ClassEval function-level, and 0.230 on ClassEval class-level.On BigCodeBench-Hard, the base model scores 0.29 and CodePRM 0.32; on ClassEval class-level, the base model scores 0.182.Source: §7.3 · pp. 29–30, Figure 8 · p. 30
7.4. Case Studies
- The base model remains at a 40% pass rate after five refinements, while SpecCoder reaches 100% in three using separated preferred elements and sliding-window consumption.The SpecCoder refinement uses dynamic padding and truncates the residual pool to avoid index-offset errors.Source: §7.4 · pp. 30, 32, Figure 9 · p. 31
8. Discussion
The data pipeline retains 13,532 high-quality triplets at an estimated API cost of $596.96. The paper identifies dependence on external teacher models and incomplete test suites as limitations.
8.1. Cost Analysis
- Starting from 20,000 seed problems, bidirectional filtering retains 13,532 triplets for Dgen at approximately $596.96 in API costs.The amortized API cost is approximately $0.044 per retained sample.Source: §8.1 · p. 32
8.2. Limitations and Future Work
- Data construction relies heavily on external teacher models, whose capabilities limit the model’s ability to understand specifications.The authors identify additional API costs and propose exploring data synthesis without closed-source models, including self-play and weak-to-strong generalization.Source: §8.2 · pp. 32–33
- Test-case pass rates are only approximate reward signals, since incomplete black-box suites may miss edge cases and internal-logic or external-constraint violations.The authors propose static code analysis or process reward models for multidimensional feedback.Source: §8.2 · p. 34
9. Conclusion
SpecCoder uses structured specification analyses between raw requirements and code, combining specification-guided SFT with curriculum dual-task GRPO. Across the reported evaluations, it improves standalone and agent-based code generation, and the analyses find reliance on specification semantics.
9. Conclusion
- SpecCoder combines specification-guided SFT and curriculum dual-task GRPO to train code generation and discrimination through structured specification analyses.The paper reports improvements on APPS, CodeContests, and xCodeEval, with further evaluations on BigCodeBench-Hard and ClassEval.Source: §9 · p. 34
Appendix A. Data Construction Prompts and Verification
The appendix documents the prompts and evaluation procedures used to construct and verify specification-guided training data.
Appendix A.1. Multi-Path Generation and Verification Prompts
- The Dgen construction prompts use GPT-4o for multi-path structured-specification generation and DeepSeek-V3-0324 for triplet consistency evaluation.The generation prompt uses seven predefined dimensions.Source: Figure A.11 · p. 35, Appendix A.1 · p. 35
Appendix A.2. Bidirectional Verification Evaluation Rubrics and Prompt
- Backward verification checks that each generated specification preserves raw requirements and that candidate implementation logic can be traced to it.The triplet is ⟨p, s, c⟩: raw requirement, structured specification analysis, and candidate implementation.Source: Appendix A.2 · pp. 35–36
- Semantic Completeness scores whether the specification preserves objectives, I/O formats, constraints, boundaries, and special cases.Source: Appendix A.2 · p. 36
- Algorithmic Traceability assesses whether implementation logic, I/O handling, state updates, and boundary-case processing can be traced to the specification.Source: Appendix A.2 · p. 36
- Expression Clarity evaluates whether specifications are clear and specific enough to condition code generation without ambiguity, contradictions, or irrelevant content.Source: Appendix A.2 · p. 36
- Triplets are retained only if they score at least 4 out of 5 on all three backward-verification dimensions, filtering some test-passing but semantically deficient examples.The rubric distinguishes semantic consistency from forward verification of executable correctness.Source: Appendix A.2 · p. 36
- In a random audit of 400 accepted triplets, annotators achieve κ = 0.79, and 88% meet the human acceptance criterion.The audit uses the same three dimensions and five-point criteria as the automatic evaluator.Source: Appendix A.2 · pp. 36–37
Appendix B. Expanded Descriptions of Benchmark Datasets
Appendix B describes the benchmark data and evaluation splits used for APPS, CodeContests, and xCodeEval; Appendix C explains the criteria used in human evaluation.
Appendix B. Expanded Descriptions of Benchmark Datasets
- The three benchmarks are processed with bidirectional filtering to construct specification-based generation and discrimination tasks.Appendix B also says the following subsections describe each dataset and its difficulty classifications.Source: Appendix B · p. 37
Appendix B.1. APPS
- APPS has 5,000 training and 5,000 test samples; 3,000 test samples support data generation, while 500 are sampled for evaluation across difficulty levels.The evaluation sample preserves the original distribution of easy, medium, and hard problems.Source: Appendix B.1 · p. 37
Appendix B.2. CodeContests
- CodeContests contains 13,610 training problems and 165 test problems, divided into easy, medium, and hard levels by platform rating.The full test set is used for evaluation; training problems serve as base data for generation-stage data construction.Source: Appendix B.2 · p. 37
Appendix B.3. xCodeEval
- xCodeEval contains about 7,500 problems and is held out from training; evaluation uses a 300-problem sample.Its difficulty levels are based on D: easy at D≤1400, medium at 1400<D≤2000, and hard at D>2000.Source: Appendix B.3 · p. 38
Appendix C. Human Evaluation Rubrics
- Human evaluators use a 5-point Likert scale to assess requirement coverage and consistency between intermediate reasoning and generated code.The seven specification dimensions are semantic references, not separately scored items or required output fields.Source: Appendix C · p. 38
Appendix D. Implementation and Hyperparameter Configurations
The appendix reports SpecCoder’s training resources and compute, then details the three-phase curriculum’s sample exposure schedule.
Appendix D. Implementation and Hyperparameter Configurations
- Training used two NVIDIA A100-SXM4 GPUs; SFT processed 13,532 samples in 1.50 hours, while dual-task GRPO used 10,220 samples.GRPO comprised 6,811 generation and 3,409 discrimination samples; peak memory was 73,058 MB per GPU for SFT and 76,445 MB for GRPO.Source: Appendix D · p. 38
- The easy warmup, hard focus, and full consolidation phases took 13.17, 25.67, and 18.38 hours, respectively.Each phase initialized from the preceding checkpoint and reset optimizer state.Source: Appendix D · pp. 38–39
8 Curriculum Dual-task GRPO Stage
- Each sample is seen three times: easy samples twice during warmup and once during consolidation, while medium and hard samples receive that allocation in hard focus and consolidation.This matches the total sample exposure of a standard three-epoch scheme.Source: Appendix D · p. 39, Table C.8 · p. 40
Appendix E. Reproducibility and Artifact Availability
The appendix documents the seed and publicly released artifacts for reproducing the experimental data splits.
Appendix E. Reproducibility and Artifact Availability
- The authors used random seed 42 and publicly released the exact training and evaluation samples with task identifiers, along with the code.The code and additional information are linked at the SpecCoder GitHub repository.Source: Appendix E · p. 41