Source-linked AI summary
Composition-RL: Compose Your Verifiable Prompts for Reinforcement Learning of Large Language Models
Xin Xu, Clive Bai, Kai Yang, Tianhao Chen, Yangkun Chen, Weijie Liu, Hao Chen, Yang Wang, Saiyong Yang, Can Yang
TL;DR
RLVR training is constrained by costly prompt expansion and by the growing prevalence of easy, pass-rate-1 prompts that reduce effective data. Composition-RL composes multiple existing problems into new verifiable questions for RL training, consistently outperforming training on original prompts across model sizes, with further gains from curriculum and cross-domain composition.
Problem
RLVR datasets contain uninformative prompts, while collecting additional high-quality verifiable prompts is costly and easy prompts become more prevalent during training.
Method
Composition-RL automatically composes multiple existing prompts into more challenging verifiable prompts and trains RL models on them.
Results
Composition-RL consistently outperforms RL on original prompts across 4B–30B models, with gains increasing with model size and reaching +10.5% for Qwen3-30B-A3B.
Takeaways & Limitations
Compositional prompts improve the utility of existing RLVR data and support cross-domain RL, including prompts spanning physics and mathematics.
Takeaways & Limitations
The method currently requires the first question’s answer to contain at least one numeric value, and some generated prompts may remain invalid after filtering.
Abstract
from arXiv · showhide
Large-scale verifiable prompts underpin the success of Reinforcement Learning with Verifiable Rewards (RLVR), but they contain many uninformative examples and are costly to expand further. Recent studies focus on better exploiting limited training data by prioritizing hard prompts whose rollout pass rate is 0. However, easy prompts with a pass rate of 1 also become increasingly prevalent as training progresses, thereby reducing the effective data size. To mitigate this, we propose Composition-RL, a simple yet useful approach for better utilizing limited verifiable prompts targeting pass-rate-1 prompts. More specifically, Composition-RL automatically composes multiple problems into a new verifiable question and uses these compositional prompts for RL training. Extensive experiments across model sizes from 4B to 30B show that Composition-RL consistently improves reasoning capability over RL trained on the original dataset. Performance can be further boosted with a curriculum variant of Composition-RL that gradually increases compositional depth over training. Additionally, Composition-RL enables more effective cross-domain RL by composing prompts drawn from different domains. Codes, datasets, and models are available at https://github.com/XinXU-USTC/Composition-RL.
1. Introduction
RLVR benefits reasoning, but limited verifiable prompts become less informative as training progresses because both solve-none and increasingly solve-all prompts yield no gradient signal. Composition-RL addresses this by composing existing prompts into harder training questions.
- RLVR has improved text-only reasoning and multimodal question answering in large language models.
- Prompts with rollout accuracy 0 or 1 yield zero gradient signals, reducing the informative training data available to RLVR.
- As training progresses, an increasing proportion of prompts attain rollout accuracy 1, creating an additional source of ineffective training data beyond hard solve-none prompts.
- Composition-RL automatically composes existing prompts into more challenging verifiable prompts for reinforcement-learning training.
- Experiments across 4B–30B models show Composition-RL outperforms RL on original prompts, with larger gains under a compositional-depth curriculum and potential for cross-domain training.
2. Preliminary
RLVR evaluates sampled responses with a binary verifier and updates the policy using advantage-weighted policy gradients. Prompts whose sampled responses are uniformly correct or incorrect have zero advantage and produce no effective update, motivating dynamic filtering.
- The policy likelihood factorizes over response tokens, conditioning each token on the query and preceding response tokens.
- A verifier v(q, r) ∈ {0, 1} indicates whether a response matches the ground-truth answer for query q.
- GRPO estimates advantage by sampling a group of responses from the old policy and comparing verifier rewards with a query-dependent baseline.
- Prompts with empirical success rates of 0 or 1 receive zero advantage, so their policy-gradient updates vanish.
- Dynamic sampling oversamples a candidate prompt set and filters out uninformative prompts when constructing each training batch.
3. Methodology & Meta-Experiments
The paper introduces Sequential Prompt Composition, which recursively combines verifiable problems into harder prompts, and uses the resulting data for RLVR. Meta-experiments show that solve-all prompts become prevalent during training, while composition lowers solve-all rates and can improve reasoning performance.
- Sequential Prompt Composition: Sequential Prompt Composition recursively composes K distinct prompts, with K denoting compositional depth.
- Composing Two Prompts: The two-prompt operator modifies the first problem using its answer, replaces a numeric value in the second with a variable, and links them through a computed relation.
- Composing Two Prompts: The composed prompt preserves the second problem’s ground-truth answer and is asymmetric because solving it requires solving the first problem before the second.
- Meta-Experiments: Solve-all prompts rise from near zero to over 50% within 50 RL steps and stabilize around 75% for Qwen3-4B-Base on MATH training data.
- Meta-Experiments: Switching to compositional prompts reduces avg@8 by 19.7% for OpenMath-Reasoning-1.5B and 15.4% for JustRL-1.5B on a MATH500 subset.
- Meta-Experiments: JustRL-1.5B improves performance by 2.3% on the MATH500 subset and 6.6% on the compositional test set, while RL on normal prompts also improves compositional performance.
- Composition-RL: Composition-RL applies the GRPO objective to prompts generated by SPC, using compositional datasets rather than the original prompts.
- Composition-RL: To control compositional dataset size, the method uses small independently resampled candidate pools, setting |D_k| = 20 for k < K and D_K = D.
4. Experiments
Experiments across 4B–30B models evaluate Composition-RL against original-prompt RL, curriculum training, controlled data sizes, and cross-domain composition. Composition-RL consistently improves in-domain and out-of-domain reasoning, with further gains from increasing compositional depth and composing physics with math.
- Main Results: Composition-RL improves overall mathematics performance by +3.6%, +4.8%, +6.1%, and +14.3% for Qwen3-4B/8B/14B/30B-A3B, respectively.It also improves multi-task overall performance by +2.7%, +1.3%, +0.7%, and +2.9% across those four models.
- Main Results: Overall average gains are +3.3%, +3.7%, +4.3%, and +10.5% for Qwen3-4B/8B/14B/30B-A3B, respectively.The reported gains increase with model size, especially on in-domain mathematics.
- Controlled Data Size: With the same training-set size, Composition-RL improves over original-prompt RL by 2.2%–11.5% and outperforms MetaMath by 7.5% and SAND-MATH by 1.3%.The 12K compositional-prompt setting has smaller gains than MATH-Composition-199K but remains the best controlled-size result.
- Curriculum RL: Curriculum Composition-RL improves AIME24 by +9.7% and MMLU-Pro by +5.9% after adding Depth 2, then adds +2.0% overall with a Depth 3 stage.Depth 1→Depth 2 also exceeds direct training on MATH-Composition-199K by +3.0% overall.
- Curriculum RL: A 4B curriculum Composition-RL model achieves 37.9% on AIME24, exceeding Beyond-80/20-8B at 34.6%, Alpha-RL-8B at 28.3%, and RL-ZVP-8B at 24.6%.Composition-RL uses MATH12K and Qwen3-4B-Base, while the cited baselines use DAPO-MATH-17K and Qwen3-8B-Base.
- Cross-Domain RL: Physics–math compositional RL outperforms Math-then-Physics by +1.3% on MMLU-Pro and +7.1% on AIME24, while exceeding MATH12K-only RL by +4.3% and +9.1%, respectively.The compositional method performs best on in-domain math and physics and out-of-domain law, engineering, and chemistry.
5. Analysis
The analysis finds that Composition-RL benefits from diverse candidate prompts, remains effective on fully solvable data, and may work through compositional generalization and implicit process supervision.
- Candidate-set ablations: Composition-RL gains +6.0% accuracy over variant B on AIME24, supporting broader answer coverage from a full D2.The authors hypothesize that verifying answers from the full prompt set provides more diverse training signals.
- Candidate-set ablations: Directly concatenating question pairs underperforms RL on the original dataset, while Composition-RL exceeds it by 5.5% in overall accuracy.The authors attribute the degradation to distribution shift from combining questions without conditioning q2 on gt1.
- Fully solvable prompts: Composition-RL improves by +2.7% on a fully solvable 7.2K-prompt subset, even with eight rollouts per prompt.This comparison uses a DAPO-style baseline with adaptive extra sampling for easier problems.
- Why Composition-RL works: Composition-RL substantially improves performance on harder Depth-2 compositional tests relative to training on Depth-1 data.The result is presented as evidence that composition can encourage recombination of learned skills.
- Why Composition-RL works: Steady improvement in recovering v1 supports implicit process supervision despite rewards verifying only final answers.Solving q1:2 requires obtaining v1 before using it to solve the second subproblem.
6. Related Work
Related work addresses RLVR data utility, prompt augmentation, compositional generalization, and prompt enlargement. Composition-RL differs by making easy prompts harder through automated sequential composition and extending composition across reasoning domains.
- RLVR training efficiency: Prior RLVR methods filter uninformative prompts, shape advantages, or allocate more samples to harder prompts to better exploit limited data.Hint-based augmentation instead transforms hard prompts into easier ones.
- Compositional generalization: Figure 3 compares avg@8 accuracy across model sizes and tracks q1:2 solution rates alongside v1 recovery accuracy over training.Darker colors and embedded numbers indicate Composition-RL improvement over the MATH12K RL baseline.
- RLVR training efficiency: Composition-RL contrasts with hint-based augmentation by making easy prompts harder through compositional prompt generation.This distinction targets the increasing prevalence of solve-all prompts during training.
- Enlarging RLVR training prompts: A substantial body of work collects diverse high-quality RLVR data and synthesizes data for evaluation or supervised fine-tuning.These efforts motivate enlarging or improving the prompt supply available for RLVR.
- Compositional generalization: Prior compositionality studies examine controlled language, math, algorithmic, or synthetic settings, while this work applies composition to RL training prompts for broader reasoning.The paper also analyzes cross-domain composition and implicit process supervision.
7. Conclusion & Discussion
The paper concludes that Composition-RL increases the utility of existing RL prompts, generalizes across topics, and may provide implicit process supervision through intermediate-step requirements.
- Conclusion: Composing prompts from different topics shows potential for cross-domain RL training.The conclusion identifies broader domain coverage as a direction for future work.
- Conclusion: Compositional prompts may provide implicit process supervision by encouraging correct intermediate steps.This conclusion follows the analysis of intermediate-value recovery during composed-prompt solving.
- Future directions: The authors identify extending composition beyond MATH12K, expanding domains, and adapting Composition-RL to on-policy distillation as future directions.They also plan to release code, compositional datasets, and trained models.
Impact Statement
The impact statement presents Composition-RL as an RLVR research contribution and announces two compositional datasets intended as resources for future RL research on LLMs.
- Impact: The paper presents Composition-RL as an approach intended to advance research on RLVR.The authors state that no specific societal consequences require highlighting.
- Impact: The authors plan to release MATH-Composition-199K and Physics-MATH-Composition-141K for future RL research on LLMs.These datasets are described as compositional resources.
Limitation
The method’s current scope is constrained by its reliance on numeric answers and rule-based verification, while some invalid compositions may remain after filtering.
- Data quality: Some generated prompts may remain invalid despite the verification procedure filtering many low-quality compositions.The authors report an erroneous-prompt rate below 2% after filtering, based on prior work.
- Evaluation scope: The main experiments use roughly 12K MATH training prompts across five difficulty levels and approximately 8.2K physics prompts retained after verifier-based filtering.The physics subset is filtered because rule-based verification is not reliably effective for all physics outputs.
- Evaluation scope: Evaluation spans mathematical benchmarks AIME24, AIME25, BeyondAIME, and IMO-Bench, plus GPQA-Diamond and MMLU-Pro for multi-task reasoning.The benchmarks use different sampling-based metrics and evaluation settings.
B.4. Details of Ablation for Candidate Sets Dk
Candidate-set ablations produce approximately matched final compositional dataset sizes, isolating candidate-set configuration rather than dataset size as the main comparison.
- Candidate-set settings: The default configuration uses the full prompt pool for D2 and a 20-prompt random subset for D1.The default sizes are |D2| = 12,000 and |D1| = 20.
- Candidate-set settings: Variant A samples both candidate sets randomly with |D1| = |D2| = 500, while Variant B reverses the default sizes with |D1| = 12,000 and |D2| = 20.These variants are designed to yield roughly the same theoretical compositional dataset size.
- Dataset sizes: Composition-RL produces a final dataset of 199K prompts after filtering.This is the reported final size for the default configuration.
- Dataset sizes: Variant A reaches 200K final prompts after filtering, following intermediate sizes of 240K after step 1 and 202K after step 2.The filtering pipeline reduces the initially constructed set before the final dataset is formed.
- Dataset sizes: Variant B reaches 200K final prompts after filtering, following 231K after step 1 and 201K after step 2.The final size is therefore close to the default and Variant A.
- Dataset sizes: Final dataset sizes are approximately matched across the three configurations.This supports comparing candidate-set choices under similar data volume.
C. Analysis Details
Figure 4 presents the prompt used to verify the correctness of finding v1 in an LLM response.
- Figure 4 is a verification prompt for checking whether finding v1 in an LLM response is correct.
D. More Analysis
Additional analyses examine implicit process supervision, compositional failure modes, response length, pass@k, effective training data, and automated composition verification. Together, they report stronger intermediate reasoning and broader solvability without significantly longer responses, while filtering removes many low-quality compositions.
- Implicit process supervision: Reflection-conditioned accuracy rises from 11.0 to 19.8, while self-correction success rate rises from 9.1 to 26.1.These results provide additional evidence that compositional prompts improve reflective and self-corrective behavior.
- Error analysis: Linkage errors dominate the original model’s 40 inspected failures, accounting for 22 cases versus 7 q1 errors and 11 q2 errors.After Composition-RL, all three error types decrease, with linkage errors falling from 22 to 7.
- Efficiency and coverage: Composition-RL does not significantly increase response length across model scales.The analysis reports average response length across in-domain, out-of-domain, and overall settings.
- Efficiency and coverage: Composition-RL consistently improves pass@k across model scales, indicating a broader set of solvable problems.
- Efficiency and coverage: At later training stages, Composition-RL has a lower solve-all ratio and comparable solve-none ratio, yielding more effective training data overall.
- Composition verification: Automated verification checks q1 modification, q2 modification, and q1–q2 connection before retaining composed prompts.The checks compare extracted and recomputed variables and filter inconsistent variable names or definitions.
- Composition verification: The verification procedure removes many low-quality compositions, with an erroneous-prompt rate below 2% reported after filtering.