PaperlayerView exact revision on arXiv

Source-linked AI summary

GCPO: When Contrast Fails, Go Gold

Hao Wu, Wei Liu

arXiv:2510.07790v1cs.AI

TL;DR

Prior approaches assume that pairing a problem with its standard answer establishes a useful link, while small models may fabricate plausible reasoning trajectories. GCPO uses gold-standard answers to steer updates when rollouts fail. Experiments report superior performance, faster convergence, and smaller models emulating larger models’ reasoning style.

  • Problem

    Prior approaches assume that pairing a problem with its standard answer establishes a useful link, while small models may fabricate plausible reasoning trajectories.

    Source: §1 · p. 1

  • Method

    GCPO replaces a failed rollout with a gold-standard answer, giving GRPO updates a positive example and a clearer direction.

    Source: §1 · p. 2

  • Results

    GCPO achieves superior performance, converges faster, and enables smaller models to emulate larger models’ reasoning style.

    Source: §1 · p. 2

  • Takeaways & Limitations

    GCPO enables smaller models to learn larger models’ reasoning patterns while improving training sample efficiency.

    Source: §1 · p. 2, §5 · p. 5

    Evaluation is limited to mathematical tasks, and preparing GAs requires resources such as LLM calls or manual annotation.

    Source: §6 · p. 5

Abstract

from arXiv · show

Reinforcement learning has been widely applied to enhance the reasoning capabilities of large language models. Extending the inference limits of smaller models has become a prominent research focus. However, algorithms such as Group Relative Policy Optimization (GRPO) suffer from a clear drawback: the upper bound of a model's rollout responses is entirely determined by the model itself, preventing the acquisition of knowledge from samples that are either all incorrect or all correct. In this paper, we introduce Group Contrastive Policy Optimization (GCPO), a method that incorporates external standard reference answers. When the model cannot solve a problem, the reference answer supplies the correct response, steering the model toward an unequivocally accurate update direction. This approach offers two main advantages: (1) it improves training efficiency by fully utilizing every sample; (2) it enables the model to emulate the problem solving strategy of the reference answer during training, thereby enhancing generalization in reasoning. GCPO achieves outstanding results across multiple benchmark datasets, yielding substantial improvements over the baseline model. Our code is available at: https://github.com/AchoWu/GCPO.

1. Introduction

The introduction asks how to improve sample use when GRPO rollouts provide no useful positive examples. GCPO supplies a gold-standard answer to guide updates, with the stated aim of improving efficiency and reasoning generalization.

1. Introduction

  • When rollouts are all incorrect or mostly correct, intra-group advantage can collapse to zero, making the policy gradient vanish and hampering learning efficiency.The introduction notes that filtering such samples still leaves GRPO unable to use all training data.Source: §1 · p. 1
  • A standard answer alone may prompt small models to fabricate a plausible reasoning trajectory, so the proposed guidance includes a complete, correct chain of thought.The authors argue this can steer updates toward the intended reasoning path.Source: §1 · pp. 1–2
  • GCPO replaces one response with a gold-standard answer when all rollouts fail, supplying a positive example that indicates the optimization direction.The reference may be ground truth or a response generated by a larger language model; any response in an all-incorrect group can serve as the surrogate.Source: §1 · p. 2
  • The authors say reference-guided updates give each step a clear direction, speed convergence, and improve training sample efficiency.They describe this as incorporating chain-of-thought data and a reference answer into GRPO updates.Source: §1 · p. 2
  • For reasoning tasks, the authors say KL divergence limits model performance rather than enhancing stability, and propose sequence-level importance sampling to align reward signals with policy optimization.Their rationale is that rule-based rewards evaluate whole response sequences, whereas token-level optimization can mismatch those sequence-level signals.Source: §1 · p. 2
  • Experiments report that GCPO achieves superior performance, converges faster, and lets a smaller model emulate a larger model’s reasoning style.The introduction gives no benchmark values for these outcomes.Source: §1 · p. 2

2. Related Work

Related work covers chain-of-thought approaches to reasoning and reinforcement-learning methods for LLMs. It also frames the challenge of directing updates toward a pre-specified training direction.

2. Related Work

  • Chain-of-thought reasoning breaks complex tasks into logical steps; related approaches include tool-using Program-of-Thought, self-consistency, reflection, and CoT-based fine-tuning or distillation.The section notes that direct supervised fine-tuning may not provide enough feedback to push beyond a model’s reasoning boundary.Source: §2 · pp. 2–3
  • LLM reasoning work includes PPO, DPO, and GRPO, which estimates advantages through in-group comparisons, alongside later efforts on sample efficiency, reward engineering, and data curation.The section identifies RL with verifiable rewards as part of this line of work.Source: §2 · p. 3
  • The paper characterizes existing methods as spending steps on ineffective updates and proposes directing each update toward a pre-specified training direction.Source: §2 · p. 3

3. Method

GCPO addresses groups with no correct rollouts by substituting a gold-standard answer and using the resulting rewards to guide updates. The method also uses sequence-level importance sampling and omits a KL penalty.

3.1. Preliminaries

  • GPRO samples a group of responses for one question, derives advantages from parsed-response rewards, and uses a relative objective that favors positive responses within each group.The group reward average supplies a baseline without a reward model.Source: §3.1 · p. 3

3.2. GCPO

  • The golden answer is treated as a correct reference, which the authors expect can help the model learn problem-solving strategies.They propose using human annotations or responses generated by a larger language model as references.Source: §3.2 · pp. 3–4
  • When all rollout rewards are zero, GCPO substitutes a failed response with a golden answer and normalizes the resulting rewards to provide an update direction.When any reward is nonzero, it instead normalizes the original rewards.Source: §3.2 · p. 4
  • GCPO replaces token-level importance sampling with sequence-level sampling, which experiments report improves performance and training stability.The paper notes that rule-based rewards assess the whole response sequence rather than individual tokens.Source: §3.2 · p. 4

3.3. Other techniques

  • The method omits the KL-divergence penalty because the authors argue that it would impede updates when extensive chain-of-thought reasoning shifts outputs from the base model.The final objective is shown in Eq. (5).Source: §3.3 · p. 4

4. Experiments

GCPO outperformed DAPO on nearly all evaluated datasets, with gains on AIME 2024 and MQA. Ablations favored sequence-level over token-level importance sampling and removing the KL penalty.

4.1. Experiment Setup

  • Evaluation covered 10 math and STEM benchmarks, using pass@1 by default and mean@32 for AIME 2024.The selected benchmarks included AIME2024, GSM8K, MATH, ASDiv, MAWPS, TabMWP, MathQA, STEM MMLU, and SAT math.Source: §4.1 · p. 4
  • Training used R1-1.5B and 9,975 filtered DAPO-Math samples, with generated golden answers replacing a response when all 16 rollouts were incorrect.Golden answers were generated by DeepSeek-R1; samples exceeding the token limit or failing format requirements were excluded.Source: §4.1 · p. 4

4.2. Main Results

  • 25% improvement over DAPO at mean@32 on AIME 2024 and roughly 54% gain over R1-1.5B on MQA; GCPO outperformed both comparators across almost all datasets.The reported MQA gain is relative to the baseline model R1-1.5B.Source: §4.2 · pp. 4–5

4.3. Ablation Study

  • 36.95% average performance for full GCPO versus DAPO’s 30.37%; token-level importance sampling reduced GCPO’s average to 32.33%, and adding KL degraded performance.The authors attribute the KL result to constrained updates after the model distribution diverges from the baseline.Source: Table 1 · p. 5, §4.3 · p. 5

5. Conclusion

GCPO uses a standard answer when rollouts are all incorrect to provide an update direction, alongside sequence-level importance sampling and removal of KL divergence. The paper reports improved training efficiency and smaller models learning larger models’ reasoning patterns.

5. Conclusion

  • When all rollouts are incorrect, GCPO supplies a standard answer to steer the update and provide external knowledge.The answer may be ground truth or generated by a larger language model.Source: §5 · p. 5
  • The conclusion identifies sequence-level importance sampling and removing KL divergence as contributors to more stable training and efficient performance.Source: §5 · p. 5
  • GCPO is reported to boost training efficiency and help smaller models learn and incorporate larger models’ reasoning patterns.The authors also report outstanding performance in experiments.Source: §5 · p. 5

6. Limitations

The authors identify golden-answer collection costs and a mathematics-only experimental evaluation as limitations.

6. Limitations

  • Using GCPO requires collecting golden answers beforehand, which consumes resources such as LLM calls or manual annotation.Source: §6 · p. 5
  • The experimental evaluation is limited to mathematical tasks, although the authors argue GCPO may have broader applicability.Source: §6 · p. 5
  • The authors suggest integrating GCPO with tool-using chain-of-thought frameworks as a possible application beyond the evaluated setting.Source: §6 · p. 5
Loading 2510.07790v1…