Source-linked AI summary
TAROT: Test-driven and Capability-adaptive Curriculum Reinforcement Fine-tuning for Code Generation with Large Language Models
Chansung Park, Juyong Jiang, Fan Wang, Sayak Paul, Jiasi Shen, Jing Tang, Jianguo Li
TL;DR
Code-generation RFT can suffer from imbalanced reward signals because existing approaches overlook heterogeneous test-case difficulty. TAROT uses four-tier test suites and capability-conditioned curriculum policies, and finds that curriculum effectiveness depends on model capability while improving code correctness and robustness.
Problem
Existing code-generation RFT approaches overlook heterogeneous intra-problem test difficulty, creating imbalanced reward signals and biased gradient updates during training.
Method
TAROT builds four-tier test suites and decouples curriculum progression from raw rewards to select capability-conditioned policies from a curriculum portfolio.
Results
TAROT consistently improves code-generation performance, functional correctness, and robustness, with less-capable models favoring basic-focused progression and more-capable models favoring complex-focused curricula.
Takeaways & Limitations
Optimal RFT curricula depend on effective model capability rather than size alone, supporting adaptive and task-specific curriculum policies.
Takeaways & Limitations
The framework depends on synthetically generated four-tier tests, is currently restricted to Python, and selects policies from a predefined portfolio using static baseline assessments.
Abstract
from arXiv · showhide
Large Language Models (LLMs) are changing the coding paradigm, known as vibe coding, yet synthesizing algorithmically sophisticated and robust code still remains a critical challenge. Incentivizing the deep reasoning capabilities of LLMs is essential to overcoming this hurdle. Reinforcement Fine-Tuning (RFT) has emerged as a promising strategy to address this need. However, most existing approaches overlook the heterogeneous difficulty and granularity inherent in test cases, leading to an imbalanced distribution of reward signals and consequently biased gradient updates during training. To address this, we propose Test-driven and cApability-adaptive cuRriculum reinfOrcement fine-Tuning (TAROT). TAROT systematically constructs, for each problem, a four-tier test suite (basic, intermediate, complex, edge), providing a controlled difficulty landscape for curriculum design and evaluation. Crucially, TAROT decouples curriculum progression from raw reward scores, enabling capability-conditioned evaluation and principled selection from a portfolio of curriculum policies rather than incidental test-case difficulty composition. This design fosters stable optimization and more efficient competency acquisition. Extensive experimental results reveal that the optimal curriculum for RFT in code generation is closely tied to a model's inherent capability, with less capable models achieving greater gains with an easy-to-hard progression, whereas more competent models excel under a hard-first curriculum. TAROT provides a reproducible method that adaptively tailors curriculum design to a model's capability, thereby consistently improving the functional correctness and robustness of the generated code. All code and data are released to foster reproducibility and advance community research at https://github.com/deep-diver/TAROT.
1 Introduction
TAROT addresses reward imbalance in code-generation RFT by modeling intra-problem test difficulty and adapting curriculum policies to model capability. Experiments report improved correctness and robustness, with different curriculum preferences for less- and more-capable models.
- Motivation: Existing code-generation curricula often sequence problems by coarse difficulty, overlooking heterogeneous test-case difficulty within each problem.This oversight can produce imbalanced rewards and biased gradient updates, hindering robust and sophisticated reasoning.
- Method: TAROT decouples curriculum progression from raw rewards and selects capability-conditioned policies from a curriculum portfolio.The framework is designed to support stable optimization and efficient competency acquisition.
- Method: TAROT constructs four-tier test suites—basic, intermediate, complex, and edge—to create an intra-problem difficulty spectrum for curriculum progression.Differential emphasis on test tiers supplies a structured signal for reinforcement learning.
- Results: TAROT consistently improves functional correctness, robustness, model performance, and training efficiency across coding benchmarks compared with strong baselines.The reported evaluations cover state-of-the-art LLMs and well-known coding benchmarks.
- Results: Less-capable models benefit from basic-to-complex curricula, whereas more-capable models learn more effectively from complex-focused curricula.The optimal curriculum is therefore capability-dependent rather than fixed across models.
2 Related Work
Prior code-generation work uses curriculum learning to organize training by task or subtask difficulty, while reinforcement-learning methods optimize desired behaviors. TAROT differs by integrating tiered test suites directly into a capability-adaptive, curriculum-aware reward mechanism.
- Curriculum Learning: Curriculum learning presents training data in a structured difficulty order, typically progressing from simple to complex examples.Prior work applies this idea through progressively generated instructions, curated examples, and code-task difficulty metrics.
- Curriculum Learning: Code-generation curricula have used automatic complexity metrics or decomposed complex problems into simpler code-completion subtasks.These approaches organize training at the problem or subtask level rather than through TAROT’s tiered test suites.
- Reinforcement Learning: Reinforcement learning methods align LLM behavior through approaches including RLHF, DPO, PPO, GRPO, and GSPO.Code-focused RL work also includes stepwise training and adaptive curriculum selection based on evolving capability.
- TAROT’s Position: TAROT makes rewards curriculum-aware by weighting success according to the difficulty of the solved test tier.This produces a more nuanced learning gradient that encourages mastery of harder problems within stable policy optimization.
3 Methodology
TAROT augments each coding problem with four verified test tiers and trains with a capability-conditioned curriculum that separates training allocation from reward valuation. This design focuses optimization on difficulty tiers suited to the model’s baseline capability.
- Dataset: The four-tier suite provides an intra-problem difficulty gradient for curriculum progression across levels of functional correctness.The framework trains LLMs on test cases of varying difficulty rather than relying only on coarse problem-level difficulty.
- Dataset: TAROT augments each problem with basic, intermediate, complex, and edge test tiers while preserving the original statement and reference solution.Every test case is validated against the reference solution, and curriculum order is imposed during training rather than encoded in the dataset.
- Training Mechanism: TAROT separates curriculum allocation from reward weights, allowing training effort and success valuation to be specified independently for each difficulty tier.The allocation α_l controls the share of training updates, while w_l controls a tier’s contribution to the overall return.
- Training Mechanism: During training, generated solutions are evaluated on tiered tests, and their pass rates are accumulated into a tier-weighted return.Tier-level success is defined as the average pass rate over the corresponding test set, with Pass(π, t) indicating whether the generated solution satisfies test case t.
- Training Mechanism: Curriculum allocation and reward weights are chosen before training according to the model’s effective capability and baseline coding proficiency.This produces a fixed but customized schedule that concentrates the training signal on a model-specific zone of optimal difficulty.
4 Experiments
Experiments validate TAROT’s four-tier dataset and show consistent gains across coding benchmarks, while optimal curricula vary with model capability, specialization, and task domain.
- Experimental Setup: 15k Python coding interview problems were augmented with validated basic, intermediate, complex, and edge test suites, using forward, reversed, static, and weighted curriculum policies.Policies vary allocation order and reward weighting, with staged transitions at 0.2, 0.4, and 0.6 of total training epochs.
- Dataset Validation: The four tiers exhibit increasing structural complexity, while complex and edge cases separately target algorithmic complexity and boundary conditions.Input length, token diversity, and character transitions shift rightward across tiers; complexity-focused tests peak in complex, whereas boundary checks concentrate in edge.
- Main Results: TAROT consistently improves pass@1 over base checkpoints across HumanEval, HumanEval+, MBPP, and MBPP+ for Qwen2.5-Instruct and Qwen2.5-Coder-Instruct models.Figure 3 reports benchmark scores, gains in percentage points, and the best curriculum strategy for each model and benchmark.
- Capability Dependence: Optimal curriculum depends on model scale and specialization: larger or code-specialized models favor complex-focused strategies, while smaller general-purpose models favor basic-focused training.Qwen2.5-Coder-3B matches the complex-focused preference of Instruct-7B despite having fewer parameters.
- Generalization: +2.12 to +4.26 percentage points were achieved by Qwen3-4B-Instruct-2507 with C/E Weighted training across all reported benchmarks.These gains occurred relative to an already strong base-model baseline and support capability-dependent curriculum preferences.
- Analysis: For more-capable models, basic tests may be too trivial while complex tests provide informative signal; less-capable models can experience sparse rewards on complex tiers.The authors associate these differences with a zone of optimal difficulty and recommend capability-calibrated curricula.
- Out-of-Distribution Evaluation: On out-of-distribution benchmarks, TAROT outperforms baselines, but the best curriculum is task-dependent rather than universal.Qwen2.5-7B prefers Basic Only on LiveCodeBench v5, whereas C/E Weighted performs best on CruxEval and CodeForces.
5 Conclusion
TAROT adapts curriculum design to model capability through a four-tier, intra-problem test suite. Experiments find that less-capable models benefit from basic-focused progressions, while more-capable models perform best with complex-focused curricula.
- 5 Conclusion: TAROT constructs a four-tier, intra-problem test suite to tailor curriculum design to a model’s effective capability.The tiers comprise basic, intermediate, complex, and edge cases.
- 5 Conclusion: TAROT consistently improves code-generation performance over strong baselines.
- 5 Conclusion: Less-capable models benefit most from basic-focused progression, whereas more-capable models excel with complex-focused challenges.The optimal curriculum depends on effective capability rather than model size alone.
- 5 Conclusion: The findings support future research into automated and task-specific curriculum policies for code generation.
Limitations
TAROT’s scope and learning signal are constrained by synthetically generated test suites, Python-only experiments, and a predefined policy portfolio selected from static baseline assessments.
- Limitations: TAROT depends on four-tier test suites synthetically generated by frontier LLMs.Potential generator biases or latent coverage gaps may propagate to the policy model and constrain learning-signal diversity.
- Limitations: The study is restricted to Python coding tasks, leaving multilingual and low-resource generalization unverified.
- Limitations: The capability-adaptive mechanism selects from a predefined curriculum-policy portfolio using static baseline assessments.Continuous curriculum spaces and dynamic schedule optimization during training remain future work.
B Implementation Details
The experiments augment approximately 15k Python problems with four-tier test suites and evaluate diverse models using specified training, generation, and benchmark protocols.
- Dataset: The TAROT dataset augments approximately 15k Python problems with four-tier test suites generated by OpenAI’s o3 and o4 models.The tiers span basic, intermediate, complex, and edge cases, and generated tests undergo validation.
- Model Selection: The model suite includes Qwen2.5 models at 1.5B, 3B, and 7B scales to test whether optimal curricula depend on capability.The study also examines code-specialized counterparts and architectural generalizability.
- Training Details: All selected models are fine-tuned for a single epoch using TAROT.
- Training Details: Policy optimization uses GRPO with AdamW, a constant learning rate of 1 × 10^-6, and eight candidate completions per prompt.Global batch size is 8, reduced to 4 for specified larger models.
- Evaluation Metrics: Evaluation measures pass@1 on HumanEval, MBPP, HumanEval+, and MBPP+, plus overall accuracy on LiveCodeBench v5 and CodeForces.The evaluation pipeline is managed by EvalChemy and follows benchmark-specific generation configurations.
- Evaluation Protocol: Generated code is served with vLLM and executed in a sandboxed Python 3.11 environment with a strict 10-second timeout per test case.
D Hyperparameter Sensitivity Analysis
Hyperparameter sensitivity is benchmark-dependent: exploration-favoring settings help HumanEval variants, while MBPP favors more conservative or longer-generation configurations.
- Sensitivity Analysis: Ablations examine GRPO’s KL-regularization coefficient β and training-time sampling temperature.
- Impact of GRPO’s β: For HumanEval and HumanEval+, β = 0.01 yields the best results by allowing greater policy exploration.
- Impact of Training Temperature: A training temperature of 1.0 is optimal for HumanEval and HumanEval+, whereas MBPP peaks at 0.7.The preferred exploration level is task-specific.
- Generation Length: On HumanEval and HumanEval+, performance generally declines beyond a 4,096-token inference limit, while MBPP and MBPP+ often benefit from 8,192 or 16,384 tokens.
- Generation Length: The ideal maximum completion length depends strongly on benchmark characteristics.Longer limits may encourage verbose, error-prone code on concise function-completion tasks but support fuller logic development on MBPP.
F Additional Results on Gemma2-2B-IT
Gemma2-2B-IT shows curriculum fragility: most curricula reduce performance, while a fundamentals-first schedule provides the most reliable gains. This supports emphasizing simpler tiers for weaker models.
- Most curricula depress Gemma2-2B-IT performance, consistent with sparse rewards causing collapse in less-capable models.
- Basic Only yields the most reliable gains among the tested strategies.
- For weaker models, simpler tiers are a prerequisite for successful fine-tuning, while complex-focused or mixed curricula can be harmful.
G Full Benchmark Tables (Qwen2.5 & Qwen3-4B)
The appendix reports complete curriculum-strategy results across Qwen2.5 and Qwen3-4B models and several coding benchmarks. The stronger Qwen3-4B model favors C/E Weighted, while exploration and regularization settings remain benchmark-dependent.
- HumanEval and HumanEval+ peak at β=0.01, whereas MBPP and CodeForces peak at β=0.05.
- Training temperature 1.0 is optimal for HumanEval and HumanEval+, while MBPP prefers 0.7.
- Complete tables report pass@1 and accuracy across Qwen2.5, Qwen3-4B, and multiple coding benchmarks for every curriculum strategy.
- C/E Weighted tends to perform best for the more-capable Qwen3-4B model and improves over the base across four code-function benchmarks.
H Training Dynamics Analysis
Training rewards increase stably and separate by model capacity, while completion length provides a stronger negative association with benchmark performance than reward. The appendix also documents TAROT’s tiered test-suite construction and evaluation tables.
- Training dynamics: Training rewards increase stably and separate according to model capacity during reinforcement fine-tuning.Initial rewards are low because of strict formatting requirements and sandbox execution timeouts, which models quickly adapt to.
- Inference sensitivity: For HumanEval and HumanEval+, performance tends to decline when the inference token limit exceeds 4,096.
- Training dynamics: Mean completion length has a stronger negative correlation with benchmark scores than reward does.More capable models tend to produce shorter solutions as training progresses.
- Tiered test cases: TAROT test tiers progress from basic core-logic checks to edge cases probing boundaries, constraints, and performance-intensive inputs.