Source-linked AI summary
From Imitation to Discrimination: Toward A Generalized Curriculum Advantage Mechanism Enhancing Cross-Domain Reasoning Tasks
Changpeng Yang, Jinyang Wu, Yuchen Liu, Shuai Zhang, Yang Li, Qiliang Liang, Hongzhen Wang, Shuai Nie, Jiaming Xu, Runyu Shi, Ying Huang, Guoquan Zhang
TL;DR
Mixing positive and negative advantage signals from the start can make guidance ambiguous. CAPO stages training from positive-only imitation to discrimination with negative signals, using advantage as an adaptive curriculum signal. The paper reports improvements across mathematical reasoning and multimodal GUI tasks and across several optimization methods.
Problem
Existing RL methods mix positive and negative advantage samples, and the introduction identifies ambiguous guidance from this simultaneous training as a research challenge.
Source: §1 · p. 1Method
CAPO uses positive-only advantage samples in an imitation phase before introducing negative samples in a discrimination phase, with advantage estimates guiding adaptive curriculum scheduling.
Source: §1 · pp. 1–2, §2.1 · p. 2Results
CAPO improves mathematical reasoning across advantage-based RL methods, with gains of +2.4 to +4.0 on 1.5B models and +1.7 to +3.9 on 7B models, and reports +3.81 on multimodal GUI reasoning tasks.
Source: §1 · p. 2Takeaways & Limitations
The reported cross-method and cross-domain results support advantage-based curricula as a mechanism for unifying stability and exploration.
Source: §4.1 · p. 4
Abstract
from arXiv · showhide
Reinforcement learning has emerged as a paradigm for post-training large language models, boosting their reasoning capabilities. Such approaches compute an advantage value for each sample, reflecting better or worse performance than expected, thereby yielding both positive and negative signals for training. However, the indiscriminate mixing of the two signals in existing methods, especially from the early stages, may lead to ambiguous guidance and limited gains. To address this issue, we propose **CAPO** (**C**urriculum **A**dvantage **P**olicy **O**ptimization), an adaptive curriculum mechanism based on advantage signals. The proposed mechanism bootstraps imitation learning with positive-only advantage samples to establish robust foundations, and subsequently introduces negative signals to cultivate discriminative capabilities, thereby improving generalization across complex scenarios. Compatible with diverse optimization methods including GRPO, PPO, RLOO, and Reinforce++, our method consistently achieves stable and significant improvements in mathematical reasoning tasks, and further generalizes effectively to multimodal Graphical User Interface (GUI) reasoning scenarios, establishing itself as a versatile and robust optimization framework.
1 Introduction
The introduction frames mixed positive and negative advantage signals as a source of ambiguous early guidance, then proposes CAPO’s staged curriculum to use them in sequence. It presents this approach as an adaptive, broadly applicable method with reported gains in mathematical and GUI reasoning.
1 Introduction
- Mixing positive and negative advantage samples from the start can create ambiguous guidance, motivating the question of whether advantage can structure their integration.The introduction defines advantage as indicating whether a trajectory performs above or below expectation.Source: §1 · p. 1
- CAPO first uses positive samples for imitation, then combines positive and negative samples for discrimination to support stable foundations and generalization.Figure 1 contrasts this staged approach with mixing both signals from the outset.Source: §1 · p. 1, Figure 1 · p. 1
- The curriculum is motivated by staged learning and framed through a variance–bias tradeoff: imitation reduces gradient variance, while discrimination restores unbiasedness for generalization.The introduction relates the progression to children first acquiring behaviors through positive imitation and later incorporating corrective feedback.Source: §1 · pp. 1–2
- CAPO uses advantage estimates as an intrinsic, competence-aware signal for dynamic scheduling, unlike curricula based on static heuristics or external proxies.The introduction contrasts these proxies with signals aligned to the model’s evolving competence.Source: §1 · p. 2
- Improvements range from +2.4 to +4.0 on 1.5B models and +1.7 to +3.9 on 7B models across advantage-based RL algorithms, with +3.81 on multimodal GUI reasoning tasks.The introduction also reports consistent gains across GRPO, PPO, RLOO, and Reinforce++.Source: §1 · p. 2
2 Method
CAPO structures advantage-based policy optimization as a positive-only imitation phase followed by a phase using the full advantage spectrum. Its theoretical justification frames the schedule as trading early variance reduction for later unbiasedness and generalization.
2.1 Curriculum Advantage Policy Optimization
- CAPO uses algorithm-computed advantages to schedule positive-only imitation before introducing negative advantages for discriminative refinement.The switch is intended to stabilize early learning and improve later generalization.Source: §2 · p. 2, §2.1 · pp. 2–3
2 Method
- Policy optimization can replace action values with advantages to reduce gradient-estimate variance, while algorithms differ in how they estimate advantages.The paper treats advantage as both a gradient signal and a way to structure the training curriculum.Source: §2 · p. 2
2.2 Theoretical Justification
- Under Robbins–Monro conditions, the two-phase update is claimed to converge almost surely to a local optimum as Phase 1 bounds variance and Phase 2 restores unbiasedness.The stated result is that the mean squared error vanishes asymptotically and the limit point is stationary.Source: §2.2 · p. 3
3 Experimental Setups
The experiments cover mathematical reasoning and GUI-based multimodal tasks, with CAPO-augmented baselines compared against their underlying methods. Mathematical evaluation spans in-domain and out-of-domain benchmarks, while GUI evaluation includes planning and perception tasks.
3 Experimental Setups
- The setup evaluates CAPO on mathematical reasoning and GUI-based multimodal tasks using datasets, benchmarks, and baselines.The experiments are presented as covering both the paper’s primary mathematical focus and a supplementary multimodal evaluation.Source: §3 · p. 4, §3.2 · p. 4
3.1 Mathematical Reasoning Tasks
- Mathematical training uses 5.5K MATH Level 3–5 problems, with evaluation across seven math benchmarks plus ARC-C and GPQA-Diamond for out-of-domain testing.GRPO, PPO, RLOO, and Reinforce++ are each evaluated with and without CAPO.Source: §3.1 · p. 4
3.2 GUI-based Multimodal Tasks
- GUI training uses 3K vision-language-action examples, with planning benchmarks across web and Android tasks and ScreenSpot Pro for perception.The baseline comparison is GRPO versus its CAPO-augmented variant; the evaluation also includes qualitative case studies.Source: §3.2 · p. 4
4 Experimental Results
CAPO improves mathematical reasoning across optimization methods and model scales, and extends gains to GUI reasoning. Detailed analyses examine training dynamics, switch timing, curriculum strategy, and out-of-distribution performance.
4.1 Main Results
- CAPO delivers +1.7 to +4.0 points across four optimization methods on seven reasoning benchmarks, with gains on both 1.5B and 7B models.On the 7B model, AMC improves from 52.5 to 65.0 (+12.5) and AIME24 from 16.7 to 20.0 (+3.3).Source: §4.1 · p. 4, Table 1 · p. 5
- CAPO on 1.5B models closes much of the performance gap to the 7B baseline, while the 7B model attains the highest performance.The reported results also describe substantial improvements on high-difficulty benchmarks.Source: §4.1 · p. 4
- CAPO yields an overall gain of +3.81 points over GRPO on GUI planning benchmarks, extending its results beyond mathematical reasoning.The evaluation covers low- and high-level planning tasks across four GUI datasets.Source: §4.1 · pp. 4–5, Table 2 · p. 5
4.2 Detailed Analysis
- After imitation stabilizes, CAPO's entropy and rewards continue rising while GRPO's entropy plateaus, a pattern the authors link to better generalization.The authors say deferring negative samples stabilizes the imitation stage and avoids sharp entropy collapses associated with prematurely mixing them.Source: §4.2 · p. 5, Figure 3 · p. 6
- On AIME24 and AMC23, performance peaks when the switch from imitation to discrimination occurs around 20%–30% of training.The figure presents these as two representative results from evaluations across eight benchmarks.Source: §4.2 · p. 6, Figure 4 · p. 7
- GRPO(+Ours) achieves the strongest improvements across most tasks, while ADARFT and static-curriculum GRPO show limited and inconsistent gains over vanilla GRPO.The comparison contrasts dynamic advantage adjustment with a fixed sample order based on pass@16 difficulty estimates.Source: §4.2 · p. 6, Table 3 · p. 6
- CAPO shows robustness under distribution shift when evaluated on ARC-C and GPQA-Diamond after training exclusively on mathematical data.The supplied prose and figure caption report different average-accuracy gains over GRPO, so this point does not combine their values.Source: §4.2 · pp. 6–7, Figure 5 · p. 7
5 Related Work
The related work contrasts CAPO with feedback-based optimization and curriculum approaches that rely on static or external heuristics. CAPO instead schedules advantage signals in stages and adapts to the model's evolving competence.
Reinforcement Learning for Large Reasoning Model
- Prior approaches use positive and negative feedback in different ways, while CAPO first uses positive advantages and later adds negative samples within advantage-based RL.The cited discussion contrasts this with Zhu et al. (2025) and Xu et al. (2025b), whose approaches do not provide the same staged integration with advantage-based RL.Source: §5 · p. 7
Curriculum Learning
- Unlike data-centric curricula that order samples using external difficulty criteria, CAPO uses advantage estimates as an intrinsic, competence-aware signal.The discussion identifies static difficulty measures and external proxies as characteristic of prior curriculum methods.Source: §5 · p. 7
6 Conclusion
The conclusion presents CAPO as an adaptive curriculum that moves from positive-only imitation to discrimination with negative signals. Experiments report gains over strong baselines across mathematical reasoning benchmarks and multimodal reasoning tasks.
6 Conclusion
- CAPO uses advantage as an intrinsic learning signal to schedule a two-phase curriculum, moving from positive-only imitation to discrimination with negative signals.The conclusion reports consistent improvements over strong baselines across benchmarks and cross-domain generalization from mathematical to multimodal reasoning.Source: §6 · p. 7
A Theoretical Derivations
The appendix derives a variance–bias motivation and convergence guarantee for CAPO. Its experimental settings cover mathematical reasoning and multimodal GUI tasks.
A Theoretical Derivations
- The derivation assumes zero-mean advantage-estimation noise with variance σ2, independent and bounded across state–action pairs, and relates estimator MSE to bias and variance.When the advantage estimate is unbiased, the gradient estimator's bias term vanishes; large advantage variance can destabilize training.Source: A Theoretical Derivations · p. 10
A.1 Phase 1: Variance Reduction
- Phase 1 filters updates to positive advantages, reducing variance by excluding negative samples, though this introduces bias that the analysis treats as tolerable early in training.For a symmetric normal advantage estimate, the derivation says this filtering halves the variance contribution.Source: §A.1 · p. 10
A.2 Phase 2: Convergence Guarantee
- As policy improvement lowers advantage variance, CAPO switches to an unbiased estimator; under bounded gradients and advantages and Robbins–Monro conditions, SGD converges almost surely to a stationary point of expected reward.The convergence guarantee is conditional on the stated boundedness and step-size assumptions.Source: §A.2 · p. 10
B Experimental Settings and Hyperparameters
- The experiments are divided into mathematical reasoning and multimodal GUI tasks, with detailed settings and hyperparameters provided for reproducibility.Source: B Experimental Settings and… · p. 10
B.1 Mathematical Reasoning Experiments
Experiments used separate Qwen2.5-Math and Qwen2.5-VL-3B-Instruct setups; GUI training also specified grounding rewards and a default switch point.
B.1 Mathematical Reasoning Experiments
- Math experiments used Qwen2.5-Math models at 1.5B and 7B, with 1024-token input and completion limits and AdamW at 1 × 10−5.GRPO used 16 candidates per prompt and two training epochs; PPO, RLOO, and Reinforce++ used batch size 128.Source: §B.1 · p. 11
B.2 GUI reasoning tasks
- GUI experiments used Qwen2.5-VL-3B-Instruct with 1024-token input and completion limits, AdamW at 5 × 10−6, cosine decay, and batch size 128.Training ran for three epochs with eight gradient-accumulation steps, KL regularization 0.02, dropout 0.1, and bfloat16 mixed precision.Source: §B.2 · p. 11
- GUI grounding used token-level group-relative advantage and two GRPO iterations per batch, with a hard CAPO switch at 30% of training steps.The coordinate reward was 1 when the predicted point fell within the bounding box and 0 otherwise; a format reward applied to outputs matching the required format.Source: §B.2 · p. 11
C.1 Full Results of Switch Stage Sensitivity Study
Switch-point sensitivity varies by benchmark: early switches improve four named math benchmarks, while College Math and OlmpaidBench show more stable trends.
C.1 Full Results of Switch Stage Sensitivity Study
- AIME24, AMC23, GSM8K, and MATH improve clearly when negative samples are introduced early, at approximately 10%–40%.Figure 6 examines switch-point effects across eight benchmarks.Source: Figure 6 · p. 11
- College Math and OlmpaidBench show more stable trends with moderate fluctuations as the switch point changes.Source: Figure 6 · p. 11
C.2 Results on GUI-based Perception results
On ScreenSpot-Pro, CAPO improves GUI grounding over GRPO; a mathematical case study also contrasts a valid constraint-satisfying solution with an invalid one.
C.2 Results on GUI-based Perception results
- On ScreenSpot-Pro, CAPO beats GRPO by +3.54 text-accuracy and +3.59 icon-accuracy points in Creative; CAD and Scientific gains reach +7.11 and +6.94 points.The table reports grounding accuracy across Dev, Creative, CAD, and Scientific domains.Source: Table 4 · p. 12, §C.2 · p. 12
D.1 Mathematical Reasoning
- In the number-theory case, CAPO satisfies all constraints and obtains 236, whereas GRPO's 248 violates the median condition.CAPO constructs a four-element list with two 9s, 5, and 7, producing a median of 8 absent from the list.Source: §D.1 · p. 12
D.2 Cross-Domain Case Studies on Graphical User Interface(GUI) Interaction
The case studies contrast GRPO's flawed or untargeted reasoning with CAPO's constraint-aware mathematical solution and use of visible interface cues.
D.2 Cross-Domain Case Studies on Graphical User Interface(GUI) Interaction
- In the number-theory comparison, GRPO's median-violating construction yields incorrect 248, while CAPO's condition-by-condition solution gives 236.Source: Figure 7 · p. 13
- For the shopping task, GRPO explores generically, whereas CAPO uses the visible School Supplies category filter for a more accurate and efficient solution.The category button is already visible on screen, avoiding unnecessary scrolling or exploration.Source: §D.2 · p. 12, Figure 8 · p. 14