CorrGRPO: Correlation-Normalized GRPO for Multi-Reward Learning

1 Introduction

• Multi-reward objectives can reinforce one another or trade off, as in code correctness signals and the balance between agent utility and security.The paper studies how these reward relationships affect advantage estimation in GRPO.Source: §1 · p. 2
• With marginal variances and centered total reward fixed, stronger positive reward correlations reduce advantage magnitude, while weaker or more negative correlations increase it.The variance of the summed reward depends on all pairwise reward covariances, so GRPO adjusts advantage scaling with reward relationships.Source: §1 · p. 2
• Covariance-based normalization entangles reward dependence with scale, so large-scale components can dominate even when smaller-scale rewards encode useful distinctions.This imbalance is especially relevant when heterogeneous rewards differ substantially in numerical range or within-group variability.Source: §1 · p. 2
• Across code generation, tool calling, and agent security, CorrGRPO improves code accuracy and efficiency, tool-call accuracy, and the balance between agent utility and security.Experiments use models from 0.5B to 8B parameters and report an expanded empirical Pareto frontier between reward objectives.Source: §1 · p. 3
• CorrGRPO replaces covariance terms in the denominator with Pearson correlations while preserving the centered total reward, preventing scale alone from dominating normalization.This lets correlations involving smaller-scale rewards influence advantage normalization without being downweighted by their scales.Source: §1 · pp. 2–3

2 A Covariance Normalization View of Multi-Reward GRPO

• The denominator aggregates all entries of the reward covariance matrix, including individual variances and pairwise covariances.Positive covariances increase summed-reward variability and reduce the shared scaling coefficient; negative covariances offset variability and increase it.Source: §2 · p. 3
• At fixed marginal variances and centered reward, stronger positive correlations reduce advantage magnitude, while weaker or more negative correlations increase it.The scaling coefficient is shared across trajectories in the rollout group.Source: §2 · p. 3
• Because covariance terms weight correlations by component standard deviations, large-scale rewards can dominate normalization and limit the influence of dependencies among smaller-scale rewards.The resulting scaling coefficient adapts as reward statistics are recomputed from newly sampled trajectories across prompts and training.Source: §2 · p. 4
• In the Figure 2 example, reward r3’s variance of 14.1696 accounts for about 91.1% of the covariance sum, despite its weak correlations with r1 and r2.The strongly correlated r1–r2 pair has correlation about 0.9756, while each correlation involving r3 is about 0.1098.Source: §2 · p. 4, Figure 2 · p. 4

3 CorrGRPO: Correlation-Normalized GRPO

• For zero-variance reward components, CorrGRPO sets the corresponding correlation-matrix rows and columns, including diagonal entries, to zero.The resulting matrix is positive semidefinite, so its correlation sum is nonnegative and ε > 0 keeps the denominator strictly positive.Source: §3 · p. 4
• Replacing covariance terms with sample Pearson correlations removes reward-scale weighting, giving each nonzero-variance reward a diagonal contribution of one.Off-diagonal terms reflect the strength and direction of reward dependence, allowing strong correlations among small-scale rewards to influence normalization.Source: §3 · p. 4
• Table 1 covers LeetCodeDataset, HumanEval, MBPP, and LCB v6; its caption defines Efficiency, Executable, and average Pass@1 measures.Efficiency is the mean percentage of eligible programs faster than reference code; Executable denotes running-time success rate, and Avg. averages Pass@1 across four datasets.Source: Table 1 · p. 5

4.1 Coding Reasoning

• CorrGRPO extends the coding correctness–efficiency Pareto frontier, with every reported nondominated configuration and the highest Pass@1 and average benchmark Pass@1 at each model scale.Against GRPO, average Pass@1 rises by 2.09, 0.80, 2.27, and 4.21 percentage points at 0.5B, 1.5B, 3B, and 7B, respectively.Source: Figure 3 · p. 6, §4.1 · p. 6
• Coding evaluation uses Qwen2.5-Coder-Instruct models from 0.5B to 7B on LeetCodeDataset, with additional generalization tests on HumanEval, MBPP, and LiveCodeBench v6.The coding task assesses functional correctness and execution efficiency against reference implementations.Source: §4.1 · pp. 5–6

4.2 Tool Calling

• CorrGRPO achieves the highest RLLA-4K and API-Bank average all-exact scores for all three backbones, improving complete-call correctness and generalization.On RLLA-4K, Qwen2.5-7B all-exact rises from 63.38% to 67.61%; API-Bank average scores also improve for each backbone.Source: Table 2 · p. 7, §4.2 · p. 7
• Tool-calling experiments assess correct function and argument generation, training on RLLA-4K and testing generalization on API-Bank across three model backbones.RLLA-4K reports field-level scores and all-exact complete-call correctness; API-Bank uses execution-based final-call correctness.Source: §4.2 · pp. 6–7

4.3 Agent Utility vs. Security

• Agent experiments test task completion under prompt injection alongside attack success, training in AgentDojo and evaluating transfer on Agent Security Bench and InjecAgent.Utility and security are competing objectives: avoiding tool interactions can prevent attacks but also prevent task completion.Source: §4.3 · pp. 7–8
• CorrGRPO improves agent utility/security balance: ASB Joint Accuracy rises for all three models, InjecAgent attack success rates fall, and AgentDojo Joint Accuracy improves for two models.The largest reported AgentDojo gain is for Qwen2.5-3B, from 60.13% to 89.62%.Source: Table 3 · p. 8, §4.3 · pp. 8–9

5 Related Work

• Scalarization methods combine multiple rewards, while Safe RLHF uses dynamically updated Lagrange multipliers and dynamic reward weighting adapts scalarization weights.The cited approaches include MORLAIF, Safe RLHF, and hypervolume- or gradient-based dynamic weighting.Source: §5 · p. 9
• Group-relative methods include variance-based reward reweighting in MO-GRPO and independent within-group reward normalization followed by batch-level normalization in GDPO.These approaches modify how multiple rewards contribute to group-relative advantage estimation.Source: §5 · p. 9
• RDPO uses magnitude-aware quantile normalization and Mahalanobis whitening; CorrGRPO instead preserves prescribed reward weights and replaces covariance normalization with aggregate Pearson correlations.The passage characterizes RDPO as reducing redundant variation among correlated reward dimensions.Source: §5 · p. 9

• CorrGRPO replaces summed reward covariances in the normalization with Pearson correlations while preserving the centered total reward and its prescribed relative weights.Source: §6 · p. 9
• GRPO’s normalization couples reward dependence with component scales, whereas CorrGRPO lets advantage normalization respond to inter-reward correlations without domination by larger-scale components.Source: §6 · p. 9
• Experiments on code generation, tool calling, and agent security with 0.5B–8B models show improved performance and an outward expansion of the empirical Pareto frontier.Source: §6 · p. 10

• Generative AI tools were used to improve the manuscript’s clarity, language quality, and grammatical correctness.Source: AI Use Statement · p. 10
• The authors state that, beyond the studied language models, AI tools were not used to develop the method, formulate claims, design or implement experiments, process data, or interpret results.Source: AI Use Statement · p. 10
• The authors reviewed and revised all AI-assisted text and take responsibility for the final manuscript and its claims and results.Source: AI Use Statement · p. 10

• The work evaluates code generation, tool calling, and agent security through existing benchmarks.Source: Ethics Statement · p. 10
• Security experiments test resistance to benchmark prompt-injection attacks, but do not establish safety in real-world deployments or against unseen attacks.Source: Ethics Statement · p. 10
• Correlation-based normalization does not correct biased or misspecified rewards, and behavior remains dependent on selected objectives and their weights.Source: Ethics Statement · p. 10
• The statement calls for application-specific safety evaluation, appropriate tool-access restrictions, and human oversight before deployment.Source: Ethics Statement · p. 10

Reproducibility Statement

• Reproduction materials include model backbones, rewards, settings, metrics, hyperparameters, datasets, evaluation subsets, benchmark settings, prompts, and theoretical derivations.Source: Reproducibility Statement · p. 10

A Further Discussion

• At fixed policy parameters, CorrGRPO preserves each group’s nonzero reward-driven gradient direction while adapting its magnitude; clipping thresholds and the KL coefficient remain unchanged.Source: A Further Discussion · p. 13
• The correlation-norm view treats standardized, centered rewards as correlated input channels: GRPO weights channels by their standard deviations, while CorrGRPO assigns equal gain.Source: A Further Discussion · p. 13
• CorrGRPO can be integrated with DAPO and CISPO by replacing the group-relative advantage, and with GDPO by replacing its final batch standard-deviation denominator with a correlation norm.Source: A Further Discussion · p. 13
• CorrGRPO retains higher policy entropy during training, particularly on AgentDojo and LeetCodeDataset; the pattern varies across settings and is proposed as an explanation for stronger performance.The discussion links retained entropy to policy diversity and opportunities to explore alternative reasoning paths and higher-reward responses.Source: A Further Discussion · pp. 13–14

B Core Implementation of CorrGRPO

• The implementation computes weighted, group-centered rewards and uses sample covariance to form correlations; zero-variance components receive zero rows and columns, and the correlation sum is clamped against negative round-off.The listing uses epsilon 10^-6 for CorrGRPO and places it inside the square root, unlike the paper formula and GRPO implementation.Source: B Core Implementation of CorrGRPO · p. 14

Correlation analysis

• Runtime success and passing are binary rewards, and passing all tests requires successful execution.The correlation analysis considers Pearson correlation when both within-group variances are nonzero.Source: C Correlation Analysis for Two… · p. 14
• Once runtime success stabilizes below one, increasing the mean pass reward increases the correlation between the two rewards.With equal reward weights, CorrGRPO applies its correlation-based normalization to their combined reward.Source: C Correlation Analysis for Two… · p. 15
• As reward correlation strengthens, CorrGRPO enlarges the denominator and attenuates the combined signal for a fixed centered total reward, moderating reinforcement from overlapping signals.In the limiting case where the rewards are identical, the advantage equals the centered runtime-success reward, so duplicating it does not double the advantage.Source: C Correlation Analysis for Two… · p. 15

Experimental analysis

• After about 100 steps, runtime success stays around 0.92–0.96 while pass rate rises from approximately 0.36 to 0.54 and reward correlation increases overall.The plotted experiment uses equally weighted runtime-success and all-test passing rewards.Source: Figure 5 · p. 15, C Correlation Analysis for Two… · p. 15

D Additional Training Dynamics

• LeetCodeDataset and RLLA-4K training and validation curves plot reward components and total reward over training steps for each model backbone.The curves use exponential moving average smoothing with decay 0.6.Source: D Additional Training Dynamics · p. 15, Figure 6 · p. 16, Figure 7 · p. 17
• AgentDojo reward curves show task-specific validation components and total reward, alongside mean training components and total reward, over overlapping recorded steps.Figure 12 also tracks mean response length and policy entropy across all three datasets; curves use smoothing with decay 0.6.Source: D Additional Training Dynamics · p. 15, Figure 10 · p. 20, Figure 11 · p. 21

E Reference Scores from the Original Benchmark Papers

• Tables 4–7 report benchmark-paper scores alongside the base model, GRPO, and CorrGRPO; all scores are percentages and evaluation protocols differ across papers.The appendix identifies original sources and specifies evaluation settings for each benchmark.Source: E Reference Scores from the Original… · p. 23

E.1 Code Generation

• The code-generation comparison reports Pass@1 across LeetCodeDataset, HumanEval, MBPP, and LiveCodeBench v6, with the evaluation splits indicated in Table 4.Reference SFT scores use Qwen2.5-Coder-7B, whereas the RL runs start from Qwen2.5-Coder-7B-Instruct and train on a 2,641-problem LeetCodeDataset split.Source: Table 4 · p. 23, §E.1 · p. 23

E.2 AgentDojo

• The AgentDojo setup trains Qwen2.5-3B-Instruct on 1,584 task-grouped cases and evaluates on 21 clean and 390 attacked cases.The attacks inject important_instructions or tool_knowledge through tool responses; Table 5 includes original 95% confidence intervals.Source: Table 5 · p. 23, §E.2 · p. 23

E.3 Agent Security Bench

• Agent Security Bench evaluates the same Qwen2.5-3B-Instruct checkpoints on 400 paired clean and attacked cases without further training, reporting Clean Utility and OPI ASR.Attacks append context_ignoring payloads to intermediate tool responses.Source: Table 6 · p. 24, §E.3 · p. 24

E.4 InjecAgent

• The InjecAgent comparison uses the base setting and standard prompt without further training, with ASR-valid excluding invalid outputs.Table 7 reports ASR-valid and separates S1 data extraction from S2 data transmission, conditional on reaching that stage.Source: Table 7 · p. 24, §E.4 · p. 24

F.1 Compatibility Experiments

• Table 8 compares CISPO, GDPO, and DAPO with their CorrGRPO-integrated variants across LeetCodeDataset Efficiency, Executable, and Pass@1, plus Pass@1 on HumanEval, MBPP, and LiveCodeBench v6.The reported average summarizes Pass@1 across the four benchmarks; DAPO group filtering uses the accuracy reward. No table cell values are supplied here.Source: §F.1 · p. 25, Table 8 · p. 25

F.2 Compatibility Analysis

• Integrations are derived on fixed sampled data with advantages and normalization statistics held fixed during policy differentiation; separately configured KL or auxiliary terms retain their definitions.The objectives are reward surrogates, and numerical stabilizers follow CorrGRPO’s convention.Source: §F.2 · p. 25

F.2.1 DAPO

• For DAPO (Yu et al., 2025), substituting CorrGRPO’s advantage scales the same retained-group contribution by a positive factor, preserving asymmetric clipping and token averaging.DAPO sampling filters and reward shaping remain applicable under the same configured rules; later sampled groups may differ as training proceeds.Source: §F.2.1 · p. 25

F.2.2 CISPO

• For CISPO (MiniMax, 2025), replacing the group-relative advantage preserves importance-weight clipping and stop-gradient computation while rescaling each group’s reward-driven contribution.The construction adds neither a PPO-style minimum nor a token-dropping rule; the supplied checkpoint comparison does not assume equal training steps.Source: §F.2.2 · p. 26

F.2.3 GDPO

• For GDPO (Liu et al., 2026), CorrGRPO changes only the final batch denominator to correlation-based normalization, preserving group normalization, aggregation weights, and the batch-centered numerator.Correlations use the per-reward advantages across the batch with the same sample scope and covariance convention; this establishes compatibility, not a universal benefit, and can reduce to a common scale correction when component batch variances are equal.Source: §F.2.3 · pp. 26–27

J.1 AST Structural Similarity / J.2 Tool-Call Matching / J.4 Tool-Call Format

• AST structural reward uses a 0-to-1 traversal-sequence similarity, while tool-call content uses greedy matching and format reward separately checks the required response structure.The structural ratio is 1 for identical traversal sequences; format credit depends on delimiters and block structure independently of JSON parsing and field correctness.Source: §J.1 · p. 32, §J.2 · p. 33, §J.4 · p. 34

J.3 Coding Efficiency

• Coding efficiency is measured as the mean of eight round-level percentages of eligible pairs where the generated program is strictly faster than its reference.Each round includes only pairs where both programs pass all tests and have positive finite runtimes, with execution order alternated across rounds.Source: §J.3 · p. 34

K Effects of Reward Scale and Correlation on Advantages

• At a = 5, GRPO and CorrGRPO advantages are approximately 0.1802 and 0.4385; increasing reward scale attenuates GRPO, while CorrGRPO remains constant.The third reward contributes 81.2% of the squared GRPO denominator at a = 5, despite its weak correlations with the other rewards.Source: Figure 13 · p. 34, K Effects of Reward Scale and… · p. 35
• At a = 5, GRPO’s response to an equal correlation perturbation is five times stronger for the large-scale reward pair, while CorrGRPO responds equally to either pair.In GRPO, positive perturbations decrease the advantage and negative perturbations increase it; CorrGRPO removes the scale factors from this normalization response.Source: Figure 13 · p. 34, K Effects of Reward Scale and… · p. 35

L.1 Variance of the Total Reward as a Sum of Covariances / L.2 Exact Equivalence of Finite-Sample Variance and Covariance Estimates

• Total-reward variance equals the sum of component covariances, and the sample identity holds exactly when statistics use the same samples, means, and divisor.The identity requires no independence assumption; computing GRPO’s denominator by either route gives the same result in exact arithmetic.Source: §L.1 · p. 36, §L.2 · p. 37

L.3 Group Rescaling and Compatibility with Clipping

• Within a fixed group, CorrGRPO advantages positively rescale GRPO advantages, preserving signs, ordering, and clipping branches, but not generally the full-objective gradient scaling.The group-level surrogate and its gradient scale by a positive group-specific coefficient; adding an unchanged KL term prevents that scaling identity from generally extending to the full objective.Source: §L.3 · pp. 37–38

L.4 Positive Affine Invariance of the Denominator

• Positive componentwise affine transformations leave CorrGRPO’s correlation-based denominator unchanged, but the full advantage is not invariant.For a common positive scale, the full advantage scales by that factor; exact denominator invariance assumes exact Pearson coefficients and an unchanged set of nonconstant components.Source: §L.4 · p. 38

L.5 Positivity of the Normalization Denominators

• Both normalization quantities under the square roots are nonnegative, and adding ε > 0 makes each denominator strictly positive, including degenerate cases.The correlation matrix remains positive semidefinite under the implementation’s zero-variance convention; perfect negative correlation can still make the unregularized correlation sum zero.Source: §L.5 · pp. 38–39
• Clamping guards against negative floating-point round-off, and max(Q, 0) + ε remains strictly positive.Source: §L.5 · p. 39

M.1 Training Benchmarks

• LeetCodeDataset uses 2,641 reinforcement-learning problems and 228 evaluation problems, measuring correctness, executability, and execution efficiency.The dataset contains Python programming problems with reference solutions, executable tests, and temporal metadata.Source: §M.1 · p. 39
• RLLA-4K uses 3,920 reinforcement-learning examples and 80 evaluation examples, with tool-call metrics computed on the 71 evaluation examples requiring tool calls.Metrics cover component-level matching and complete-call correctness.Source: §M.1 · p. 39
• AgentDojo includes 1,584 training cases and 411 evaluation cases: 21 clean tasks and 390 attacked cases using the important_instructions and tool_knowledge settings.Task-specific checkers assess legitimate task completion and attacker-objective success.Source: §M.1 · p. 40

• HumanEval (Chen et al., 2021) tests Python function completion from natural-language specifications with executable tests and is used only for evaluation.Source: §M.2 · p. 40
• MBPP (Austin et al., 2021) evaluates short natural-language Python tasks using reference code and test cases, without further training.Source: §M.2 · p. 40
• LiveCodeBench v6 (Jain et al., 2024) uses 175 contest problems to assess transfer to competition-style code generation without further training.Source: §M.2 · p. 40
• API-Bank (Li et al., 2023) evaluates API selection and invocation across v1–v3 using an execution-based checker, without further training.Source: §M.2 · p. 40
• Agent Security Bench (Zhang et al., 2025) tests utility and security transfer on 400 paired clean and attacked cases in an observation prompt-injection setting.Source: §M.2 · p. 40
• InjecAgent (Zhan et al., 2024) evaluates responses to indirect prompt injections in attacker-controlled tool outputs using its base setting and standard prompt.Source: §M.2 · p. 40