Source-linked AI summary
Guidance Contrastive Token Credit Assignment for Discrete Policy Optimization
Shufan Li, Konstantinos Kallidromitis, Akash Gokul, Yuta Kyuragi, Aditya Grover
TL;DR
Uniform sample-level advantages in GRPO-style optimization can miss token-level contribution differences. GCPO contrasts positive and negative predictions from the same policy to assign per-token weights, and experiments report improvements over GRPO and DAPO across image-generation and multimodal-reasoning benchmarks.
Problem
GRPO broadcasts sample-level advantages uniformly across tokens, potentially neglecting differences in token contributions.
Method
GCPO assigns per-token importance weights by contrasting positive and negative predictions from the same policy, extending the approach to multimodal chain-of-thought reasoning.
Results
GCPO outperforms GRPO and DAPO on text-to-image and multimodal reasoning benchmarks; on GenEval, it achieves an overall score of 0.89, improving 0.09 over Janus-Pro-7B.
Takeaways & Limitations
GCPO provides a general method for assigning per-token credits when only sample-level rewards are available.
Takeaways & Limitations
GCPO requires reinforcement-learning tasks to be strictly prompt dependent so positive and negative prompts can provide contrastive guidance.
Abstract
from arXiv · showhide
Group-advantage-based reinforcement learning methods, such as GRPO and DAPO, have demonstrated strong performance across diverse domains, including mathematical reasoning and text-to-image generation. However, their reliance on sample-level rewards introduces a key limitation as uniform credit assignment across all tokens fails to capture fine-grained, token-level contributions. To address this issue, we propose Guidance Contrastive Policy Optimization (GCPO), a novel algorithm that enables per-token credit assignment by contrasting model predictions under positive and negative prompts. Rather than uniformly broadcasting sample-level advantages, GCPO assigns token-level advantages proportional to the difference between these contrastive predictions, allowing more precise and informative learning signals. Empirically, we find that GCPO emphasizes semantically relevant regions such as visual areas aligned with textual prompts in text-to-image generation, and critical keywords within reasoning traces for chain-of-thought tasks. Through extensive experiments, GCPO consistently outperforms GRPO and DAPO baselines on both text-to-image generation and chain-of-thought reasoning benchmarks, demonstrating its effectiveness as a general and scalable optimization strategy for discrete policy learning.
1 Introduction
GRPO improves efficiency by using group-normalized sample rewards, but uniform token credit can overlook differences in token contributions. GCPO addresses this with contrastive positive–negative predictions, extending token weighting from image regions to multimodal reasoning.
- GRPO broadcasts each sample-level advantage uniformly across its tokens, potentially neglecting important intra-token differences.
- Reasoning traces and generated images contain tokens or regions that differ in their contribution to the prompt-related outcome.
- GCPO assigns per-token importance weights by contrasting positive and negative predictions from the same policy.
- GCPO extends contrastive token weighting from visual synthesis to multimodal chain-of-thought reasoning.
- GCPO’s multimodal extension uses an added “please generate a wrong answer” instruction as a negative prompt and addresses normalization challenges for importance weights.
- GCPO outperforms GRPO and DAPO on text-to-image and multimodal reasoning benchmarks, including GenEval, MathVerse, MathVision, LogicVista, and MMMU-Pro.
2 Background and Related Works
GRPO estimates advantages from group-normalized rewards, while related methods modify stability or clipping and token-credit approaches seek non-uniform token signals. GCPO applies classifier-free guidance contrasts to derive semantic token weighting for policy optimization.
- GRPO samples grouped responses, computes normalized per-sample rewards as advantages, and may add a KL penalty for training stability.
- GSPO uses sequence-level ratios, DAPO removes the KL penalty with asymmetric clipping and online filtering, and Dr. GRPO removes standard-deviation scaling.
- Existing token-credit methods estimate non-uniform advantages using signals such as confidence, gradients, entropy, or statistical tests.
- Classifier-free guidance constructs a negative prompt and modifies predictions using the difference between positive and negative conditioning signals.
- GCPO extends classifier-free guidance into reinforcement-learning post-training for large language models and text-to-image generation.
- Histogram equalization is presented as a normalization strategy that produces smoother per-token weights than outlier-sensitive min-max and softmax alternatives.
3 Method
GCPO derives token-level learning signals by contrasting positive and negative prompt predictions, then normalizes these signals before scaling GRPO’s sample advantages. The method extends this approach to language and multimodal generation while emphasizing tokens whose predictions are most affected by guidance.
- 3 Method: GCPO’s guidance signal is high in image regions containing prompt-described entities, supporting its use for emphasizing semantically relevant visual tokens.The method is also positioned as broader than visually grounded approaches such as VPPO.
- 3.1 Contrastive Guidance: GCPO computes contrastive guidance from the KL divergence between positive- and negative-prompt token distributions.The divergence measures how differently the two distributions predict each token and correlates with its expected sensitivity to classifier-free guidance.
- 3.2 Normalization: The method normalizes unbounded KL divergences with a within-image cumulative distribution function.This maps token weights to [0,1], assigning 0 to the lowest value, 1 to the highest, and 0.5 to the median, producing smoother optimization across image sequences.
- 3.3 Guidance Contrastive Policy Optimization: GCPO converts normalized guidance into per-token advantages by scaling each sample-level GRPO advantage instead of broadcasting it uniformly.The resulting objective assigns greater weight to key tokens whose logits are more affected by classifier-free guidance.
- 3.4 Extending to language generation: For multimodal reasoning, GCPO computes divergences under a hypothetical classifier-free-guidance condition without using guidance during rollout.Because instruction-tuned VLMs require prompts, the method explores negative prompts including an instruction to generate a wrong answer, which works best in practice.
- 3.4 Extending to language generation: 0.89 overall GenEval score is reported for Janus-Pro-R1 + GCPO, which outperforms same-size baselines and matches significantly larger models.The result is reported as the highest overall score in the benchmark table.
4 Experiments
Experiments evaluate GCPO across text-to-image generation, multimodal reasoning, and ablations of scoring, normalization, and negative-prompt choices. GCPO improves over relevant baselines, while KL divergence, histogram-equalization-style normalization, and the proposed negative suffix perform best in their respective studies.
- Text-to-Image Generation: GCPO achieves a 0.89 overall GenEval score, improving Janus-Pro-7B by +0.09 and outperforming Janus-Pro-R1 and GRPO.The largest subcategory gains are counting, 0.56 → 0.84, and color attribution, 0.66 → 0.83.
- Text-to-Image Generation: Qualitative GenEval comparisons show GCPO producing higher-quality images that better follow instructions such as placing an apple above a television.The comparison includes Clock, Boats, and Scissors outputs from Janus-Pro-7B, GRPO, and the base model.
- Multimodal Reasoning: GCPO outperforms GRPO, VPPO, and DAPO across mathematical and generic visual reasoning tasks using different base models.The gains over DAPO appear across all reported tasks, while improvements over VPPO are more pronounced on LogicVista and MMMU-Pro.
- Ablation Studies: KL divergence performs best among tested scoring metrics, whereas information gain emphasizes tokens associated with the model’s believed correct answer.Absolute differences behave similarly to KL divergence, which also emphasizes tokens likely associated with incorrect answers.
- Ablation Studies: The proposed histogram-equalization-style normalization works best for both text-to-image generation and multimodal understanding tasks.The normalization study validates the technique across GenEval and MM12k.
- Ablation Studies: Adding “generate the wrong answer” as a suffix to the original prompt performs best among tested negative-prompt designs for MM12k.The study compares empty strings, removed image tokens, generic prompts, and alternatives reported from VPPO.
5 Conclusion
GCPO uses classifier-free-guidance signals to assign per-token advantages instead of broadcasting sample-level advantages uniformly. The method extends to text generation through a negative prompt and performs effectively when only sample-level rewards are available.
- Conclusion: GCPO assigns per-token advantages from classifier-free-guidance signals, emphasizing learning on important regions rather than uniformly broadcasting sample-level advantages.The conclusion contrasts GCPO with GRPO and DAPO.
- Conclusion: GCPO extends to text generation by designing a negative prompt that reveals the model’s implicit belief about token correctness.Classifier-free guidance is not used during these rollouts.
- Conclusion: Extensive experiments demonstrate GCPO as a generalizable and effective method for assigning per-token credit when only sample-level reward is available.The conclusion positions this as a direction for discrete policy optimization.
A Additional Implementation Details
Implementation details specify separate training configurations for text-to-image and multimodal understanding experiments. Evaluation follows the VPPO protocol by sampling eight responses per question and reporting average accuracy.
- Text-to-Image: Text-to-image experiments train GCPO on Janus-Pro-7B for 1,600 steps with AdamW and cosine-decay learning rate.Hyperparameters are listed in Table 5.
- Multimodal Understanding: Multimodal-understanding experiments use Qwen-2.5-VL-7B-Instruct and Qwen-3-VL-8B-Instruct for two epochs, totaling 202 steps on ViRL39K.Hyperparameters are listed in Table 6.
- Evaluation: Evaluation samples 8 responses per question and reports their average accuracy.This follows the VPPO evaluation setup.
B Training Dynamics
GCPO’s validation reward remains ahead of DAPO at most training stages, with the gap widening as training progresses. Experiments use fixed compute settings across models.
- GCPO consistently outperforms DAPO at most training stages, and the performance gap grows as training progresses.
- Text-to-image experiments run for 30 hours, while multimodal understanding and reasoning experiments run for 40 hours.
- All models are trained with 8 B200 GPUs.
C Visual Examination of Contrastive Guidance
Contrastive-guidance visualizations indicate that GCPO focuses learning signals on critical regions and tokens. Figure 5 compares validation accuracy over training steps for GCPO, DAPO, and GRPO.
- GCPO effectively focuses the learning signal on critical regions and tokens in text-to-image and multimodal understanding tasks.
- The contrastive-guidance visualizations cover both text-to-image tasks and multimodal understanding tasks.
- Figure 5 visualizes MM12K validation accuracy for GCPO, DAPO, and GRPO across training steps.
D Qualitative Results for multimodal reasoning.
Qualitative examples from GCPO-finetuned Qwen2.5-VL-7B-Instruct highlight the model’s multimodal reasoning capabilities across several figures.
- GCPO-finetuned Qwen2.5-VL-7B-Instruct outputs are presented as qualitative examples of multimodal reasoning.
- The qualitative results include outputs shown in Figures 9, 10, and 11.
- The examples specifically use the Qwen2.5-VL-7B-Instruct model after GCPO fine-tuning.
E Limitation
GCPO applies only when rewards depend on prompts and contrastive guidance can be derived. For multimodal understanding, it also assumes sufficient initial ability to distinguish correct from incorrect answers.
- GCPO cannot apply to reinforcement-learning settings whose reward is simply response length.
- GCPO requires strictly prompt-dependent reinforcement-learning tasks so it can derive guidance from positive and negative prompts.
- Smaller 1B-scale models may fail to generate wrong answers when instructed, making their guidance unreliable.
F Broader Impact
GCPO is presented with deployment cautions: misuse could enable malicious actions, while fine-tuning may retain base-model biases and hallucinations. The authors also report intended-use compliance for the artifacts used.
- Fine-tuning with the proposed algorithm may be used to train models for malicious actions when used improperly.
- Fine-tuned models may inherit biases and hallucinations from their base models, even without malicious intent.
- The authors do not recommend non-research use of fine-tuned models.
- The authors report following the intended use of the datasets and models used.