SEED-GRPO: Semantic Entropy Enhanced GRPO for Uncertainty-Aware Policy Optimization

Minghan Chen; Guikun Chen; Wenguan Wang; Yi Yang

Source-linked AI summary

SEED-GRPO: Semantic Entropy Enhanced GRPO for Uncertainty-Aware Policy Optimization

Minghan Chen, Guikun Chen, Wenguan Wang, Yi Yang

arXiv:2505.12346v1May 18, 2025cs.AI

TL;DR

GRPO treats prompts equally despite differences in model uncertainty, motivating uncertainty-aware policy optimization. SEED-GRPO measures semantic entropy from response-meaning diversity to scale updates, and experiments report state-of-the-art average accuracy across five mathematical reasoning benchmarks.

Problem
GRPO and its variants assign equal importance to training prompts despite varying model confidence and uncertainty across input problems.
Method
SEED-GRPO uses semantic entropy to quantify prompt-specific uncertainty and dynamically modulate policy-update magnitudes, making updates more conservative for highly uncertain questions.
Results
SEED-GRPO achieves state-of-the-art average accuracy across five mathematical reasoning benchmarks.
Takeaways & Limitations
Uncertainty-aware policy optimization preserves stronger learning signals for confident problems while using more conservative updates for challenging ones.
Takeaways & Limitations
The current implementation clusters only final answers, which may inadequately capture valid reasoning-path diversity in open-ended tasks or domains requiring evaluation of intermediate reasoning.

Abstract

from arXiv · show

Large language models (LLMs) exhibit varying levels of confidence across input prompts (questions): some lead to consistent, semantically similar answers, while others yield diverse or contradictory outputs. This variation reflects LLM's uncertainty about the input prompt, a signal of how confidently the model understands a given problem. However, vanilla Group Relative Policy Optimization (GRPO) treats all prompts equally during policy updates, ignoring this important information about the model's knowledge boundaries. To address this limitation, we propose SEED-GRPO (Semantic Entropy EnhanceD GRPO), which explicitly measures LLMs' uncertainty of the input prompts semantic entropy. Semantic entropy measures the diversity of meaning in multiple generated answers given a prompt and uses this to modulate the magnitude of policy updates. This uncertainty-aware training mechanism enables dynamic adjustment of policy update magnitudes based on question uncertainty. It allows more conservative updates on high-uncertainty questions while maintaining the original learning signal on confident ones. Experimental results on five mathematical reasoning benchmarks (AIME24 56.7, AMC 68.7, MATH 83.4, Minerva 34.2, and OlympiadBench 48.0) demonstrate that SEED-GRPO achieves new state-of-the-art performance in average accuracy, validating the effectiveness of uncertainty-aware policy optimization.

1 Introduction

Existing GRPO methods treat training prompts uniformly despite meaningful differences in model confidence. SEED-GRPO uses semantic entropy to adapt policy-update magnitudes, and experiments report strong results across five mathematical reasoning benchmarks.

GRPO assigns equal importance to all prompts, overlooking prompt-specific uncertainty and the model’s varying understanding of different problems.
Semantically diverse responses typically indicate uncertainty and may make large policy updates unstable, whereas consistent responses support safer updates.
SEED-GRPO dynamically scales updates using semantic entropy, reducing update magnitudes for challenging questions while preserving stronger signals for confident ones.The approach functions like a prompt-specific dynamic learning rate.
Semantic entropy measures diversity of meanings among multiple responses, with low entropy indicating consensus and high entropy indicating uncertainty.In the illustrated examples, five of six responses share one meaning for Question 1, while Question 2 yields six distinct meaning classes.
SEED-GRPO establishes new state-of-the-art average accuracy across AIME24, AMC, MATH, Minerva, and OlympiadBench.The introduction reports strong performance across all five mathematical reasoning benchmarks.

2 Related Work

Prior work improves reasoning through prompting and reinforcement-learning methods, including GRPO and variants. Related methods differ in how they use sampling, resampling, filtering, bias correction, and semantic entropy during optimization.

Reasoning LLMs: Chain-of-Thought, Tree-of-Thoughts, and self-consistency prompting improve reasoning by generating or aggregating multiple intermediate reasoning paths.
Group Relative Policy Optimization and Variants: GRPO is a PPO variant that removes the value model and uses group-based relative rewards to train reasoning models across mathematics, coding, and question answering.
Group Relative Policy Optimization and Variants: SRPO reuses stored problems, DAPO filters completely correct and incorrect samples, and Dr.GRPO addresses length bias in GRPO training.
Group Relative Policy Optimization and Variants: EMPO also incorporates semantic entropy, but SEED-GRPO uses it to measure uncertainty and modify advantage calculations rather than as a direct optimization objective.

3 SEED-GRPO: Uncertainty-Aware Policy Optimization

SEED-GRPO uses semantic entropy from multiple sampled responses to estimate prompt-specific uncertainty and modulate GRPO advantages. It makes updates more conservative for high-uncertainty questions while preserving the original learning signal for confident ones.

Prompt uncertainty: SEED-GRPO samples multiple responses and uses their semantic diversity to quantify uncertainty for each prompt.Responses are grouped into meaning clusters before semantic entropy is estimated from the observed clusters.
GRPO objective: For each prompt, GRPO computes binary correctness rewards, uses the group-average reward as a baseline, and forms advantages Ai = ri − r̄.The same scalar advantage is broadcast across all tokens in each sampled response.
GRPO objective: The resulting objective averages the clipped surrogate loss over the response group, after which model parameters are updated by maximizing it.The importance-sampling ratio compares the current and old policies.
Prompt uncertainty: Semantic entropy is low when responses share one meaning and high when responses occupy distinct meaning clusters.With K=1, entropy is 0; with K=G, it reaches its maximum for the fixed response group.
Uncertainty-aware updates: High semantic entropy indicates greater uncertainty, so SEED-GRPO scales down advantages to produce more conservative policy updates.The scaling sensitivity is controlled by α, and the weighting function can be linear, exponential, or focal.
Why semantic entropy: Semantic entropy is preferred to vanilla information entropy because surface-level wording differences can inflate uncertainty despite identical underlying meanings.Semantic clustering instead treats responses with equivalent meanings as one class.

2) What benefits does uncertainty-aware advantage bring to policy optimization?

SEED-GRPO incorporates uncertainty into advantage computation so policy updates are scaled per question rather than applied uniformly. This creates an adaptive learning-rate effect, with larger steps for confidently solvable problems and more cautious steps for challenging ones.

Incorporating uncertainty into the advantage computation adaptively modulates the learning process.The uncertainty-aware advantage computation scales each input’s policy-update contribution according to model uncertainty.
The resulting question-specific adaptive learning rate produces larger steps on confidently solvable problems and more cautious steps on challenging ones.This mechanism creates an implicit curriculum-learning effect and helps prevent overfitting to noise in difficult problems.
The policy update combines the global learning rate, log-probability gradient, importance-sampling ratio, and advantage term.SEED-GRPO adds a non-negative uncertainty-dependent factor that modulates update magnitude per question.

4 Experiments

Experiments evaluate SEED-GRPO on five mathematical reasoning benchmarks using Qwen2.5-Math and DeepSeek-R1-Distill-Qwen base models, with Pass@1 as the primary metric. Across configurations and ablations, uncertainty-aware optimization produces strong average performance and benefits from additional rollouts.

Experimental Setup: The evaluation covers AIME24, AMC, MATH500, Minerva, and OlympiadBench, using Qwen2.5-Math 1.5B/7B and DeepSeek-R1-Distill-Qwen-7B base models.Dr.GRPO is the default baseline, and comparisons include multiple state-of-the-art reasoning methods.
Experimental Setup: Pass@1 is the primary comparison metric, and the default uncertainty function is linear with α = 0.02.Pass@1 evaluates a single generated response, making it a more challenging setting than Pass@k with multiple responses.
Quantitative Comparison Results: 43.5% average score is achieved by SEED-GRPO 1.5B under the Qwen-Math-base setting across all benchmarks.This result is reported as a significant average improvement over the corresponding Qwen-Math-base1.5B model.
Quantitative Comparison Results: 56.6% average score is achieved by SEED-GRPO 7B, improving 5.2% over the Dr.GRPO 7B baseline.With the same initial Qwen2.5-Math-base 7B architecture and identical hyperparameters, SEED-GRPO improves AIME from 43.3% to 46.7%.
Quantitative Comparison Results: 58.2% average score is reached when rollouts increase from 8 to 16, including an AIME24 increase from 46.7% to 56.7%.The configuration uses 8.5k training examples and batch size 128, and surpasses several 32B models while using lower computational cost than training large 32B models.
Quantitative Comparison Results: 64.0% average Pass@1 is achieved by SEED-GRPO with the DeepSeek-R1-Distill-Qwen-7B base model, outperforming 7B and 32B models on key benchmarks.The reported comparisons include AIME24, MATH, and OlympiadBench.

5 Limitation and Future Work

SEED-GRPO’s current scope is mathematical reasoning with verifiable final answers, leaving broader domains and richer reasoning signals for future work. The authors propose extending uncertainty modeling beyond final-answer clustering and applying entropy signals at inference time.

Limitation: Final-answer semantic clustering is effective for mathematical problems with unique, well-defined answers but may miss diverse valid reasoning paths in open-ended tasks.The limitation arises because intermediate reasoning steps are not included in the current semantic entropy calculation.
Limitation: The method may be insufficient in domains where evaluation depends on nuanced reasoning processes rather than clear final-answer correctness.
Future Work: SEED-GRPO currently focuses on mathematical reasoning, with multimodal tasks, code generation, and open-ended question answering proposed as future extensions.
Future Work: Incorporating intermediate reasoning steps could enable finer-grained uncertainty modeling along the reasoning trajectory and potentially improve training dynamics.
Future Work: Entropy signals could support test-time strategies such as increasing rollout counts or triggering fallback mechanisms when the model is uncertain.

6 Conclusion

SEED-GRPO integrates semantic entropy into GRPO so policy updates reflect prompt-specific confidence. Across five mathematical reasoning benchmarks, it outperforms strong baselines and achieves new state-of-the-art performance using a 7B model.

Conclusion: SEED-GRPO integrates semantic entropy into GRPO’s training objective to adaptively scale policy updates by prompt-specific confidence.
Conclusion: High-uncertainty prompts receive more conservative updates, while confidently solvable problems retain effective learning signals.
Conclusion: AIME24 56.7, AMC 68.7, MATH 83.4, Minerva 34.2, and OlympiadBench 48.0 are reported across five mathematical reasoning benchmarks.
Conclusion: SEED-GRPO consistently outperforms strong baselines, including Dr.GRPO and 32B-scale competitors, achieving new state-of-the-art performance using only a 7B model.