PaperlayerView exact revision on arXiv

Source-linked AI summary

Fed-GRPO: Reward-Signal-Driven Federated Group Relative Policy Optimization

Pengxin Guo, Shuang Zeng, Zonggen Li, Weiying Zheng, Mengting Liu, Liangqiong Qu

arXiv:2610.11502v1cs.AI

TL;DR

Federated reasoning training is needed where privacy, ownership, or regulation prevents pooling data, but GRPO creates a need to coordinate clients by their reward signals. Fed-GRPO uses those signals to guide aggregation, local objectives, and communication; across mathematical reasoning tasks it outperforms federated baselines and approaches centralized performance.

  • Problem

    GRPO methods generally assume centralized training data, although high-value reasoning data may be distributed across organizations unable to pool it for privacy, ownership, or regulatory reasons.

    Source: §1 · p. 1

  • Method

    Fed-GRPO uses GRPO reward statistics to weight clients by reward standard deviation, calibrate per-prompt objectives with local–global reward gaps, and allocate sparse communication bandwidth.

    Source: §1 · p. 3

  • Results

    Across several mathematical reasoning tasks, Fed-GRPO significantly outperforms federated baselines and approaches centralized performance while achieving up to 32× lossless communication compression.

    Source: §8 · p. 10

  • Takeaways & Limitations

    GRPO's naturally computed reward statistics can guide federated aggregation, local training, and communication without additional training cost.

    Source: §1 · pp. 2–3

    The convergence analysis assumes the average round-to-round drift of the global objective along the iterate path is bounded in expectation.

    Source: §5 · p. 7

Abstract

from arXiv · show

Large Language Models (LLMs) have shown strong reasoning capabilities when fine-tuned with reinforcement learning (RL), particularly through Group Relative Policy Optimization (GRPO). However, existing GRPO methods assume centralized access to training data, which may not hold in practice due to privacy or regulatory constraints. To this end, we propose Fed-GRPO, a federated GRPO training framework that addresses these privacy constraints by enabling collaborative reasoning training without sharing raw data, which leverages the reward statistics naturally produced during GRPO training as zero-cost signals to guide aggregation, local training, and communication. Fed-GRPO contains three reward-signal-driven mechanisms: (i) \emph{signal-weighted aggregation} that weights clients by their reward standard deviation, prioritizing clients with stronger learning signals; (ii) \emph{global reward calibration} that re-weights per-prompt objectives based on the local-global reward gap, steering each client toward its relative weaknesses; and (iii) \emph{adaptive sparse communication} that allocates bandwidth based on the informativeness of each client's update. Extensive experiments on mathematical reasoning tasks demonstrate that Fed-GRPO achieves the best performance among all federated methods, clearly outperforms FedAvg and approaches centralized training performance, while losslessly reducing communication by $32\times$ and supporting up to $621\times$ compression under tight bandwidth budgets with only graceful accuracy degradation. Our code is available at https://github.com/HKU-HealthAI/Fed-GRPO.

1 INTRODUCTION

Fed-GRPO addresses the mismatch between centralized GRPO training and privacy-constrained data by using GRPO reward statistics to coordinate federated training. It combines reward-driven aggregation, local objective calibration, and communication allocation, and is reported to approach centralized performance across mathematical reasoning benchmarks.

1 INTRODUCTION

  • Centralized access to training data is increasingly unrealistic in high-value reasoning domains where institutions cannot pool data because of privacy, ownership, or regulatory constraints.Examples include hospitals, financial firms, educational platforms, and enterprise coding environments.Source: §1 · p. 1
  • Federated GRPO requires identifying useful clients by reward signal: a nearly saturated large client may contribute less than a smaller client with high reward dispersion.This differs from conventional supervised federated learning, which commonly uses sample counts or drift correction.Source: §1 · pp. 1–2
  • Naïve FedAvg-GRPO misaligns aggregation, local objectives, and communication with reward information, potentially under-training globally lagging prompts and wasting bandwidth on uninformative clients.Existing compression heuristics may over-compress reward-informative clients while allocating bandwidth to less informative ones.Source: §1 · p. 2
  • Fed-GRPO uses reward standard deviation for signal-weighted aggregation and adaptive bandwidth, and local–global reward-mean gaps to re-weight per-prompt objectives.The reward statistics are computed during GRPO training at essentially zero additional training cost.Source: §1 · pp. 2–3
  • Across five math reasoning benchmarks, Fed-GRPO outperforms all federated baselines, approaches centralized GRPO, losslessly reduces communication by 32×, and supports up to 621× compression with graceful accuracy–bandwidth trade-offs.The 621× compression applies under tight bandwidth budgets.Source: §1 · p. 3

2 PRELIMINARY

The preliminaries describe GRPO's group-relative reward-based policy optimization and the standard federated-learning loop. In federated learning, clients train locally on private data and the server aggregates their updates.

2.1 GROUP RELATIVE POLICY OPTIMIZATION

  • GRPO computes advantages from group-relative reward comparisons without a separate critic, then optimizes the policy using sampled rollouts from the old policy.The objective uses a clipping parameter and a KL penalty; rewards are normalized across the rollout group.Source: §2.1 · p. 3

2.2 FEDERATED LEARNING

  • In standard FedAvg, the server broadcasts a global model, clients train locally on private datasets and return updates, and the server aggregates them by weighted averaging.Client weights are typically set according to dataset size.Source: §2.2 · pp. 3–4

3 ANALYSIS ON FEDERATED GRPO TRAINING

A five-client, difficulty-partitioned MATH diagnostic of FedAvg-GRPO tracked reward statistics and model-update sparsity. It found client-level differences in reward signals and update density that motivate reward-driven aggregation, objective calibration, and communication.

3 ANALYSIS ON FEDERATED GRPO TRAINING

  • Reward standard deviation varied by up to 4× across clients within a round, indicating differing learning-signal strength that motivates weighting clients by σk.The diagnostic trained Qwen2.5-3B-Instruct on MATH with five difficulty-based clients, 12 rounds, and five local steps per round.Source: §3 · p. 4, Figure 2 · p. 4
  • Persistent gaps between client and global reward means expose misallocated effort: clients can over-train locally mastered prompts and under-train prompts where they lag globally.This observation motivates re-weighting per-prompt objectives using the local–global reward-mean gap.Source: §3 · p. 4
  • Over 96.2% of parameters remained unchanged after local GRPO training, and clients with stronger σk produced denser updates, motivating adaptive bandwidth allocation.The high sparsity remained stable across training rounds; low-σk clients can be compressed more aggressively.Source: §3 · p. 4
  • The reward statistics µk and σk are computed by GRPO at no extra cost and carry information about clients’ aggregation, training, and communication needs.Fed-GRPO uses these statistics in a unified framework addressing the three identified challenges.Source: §3 · pp. 4–5

4 METHODOLOGY

Fed-GRPO uses reward statistics produced during GRPO to coordinate federated aggregation, local objectives, and communication without relying on dataset size alone.

4.1 SIGNAL-WEIGHTED AGGREGATION

  • Clients are weighted by reward standard deviation σk, a zero-cost measure of learning-signal strength that adapts as client rewards evolve.σk is already computed during GRPO advantage normalization.Source: §4.1 · p. 5

4.2 GLOBAL REWARD CALIBRATION

  • Global reward calibration reweights each prompt using the local-global reward gap, directing clients toward relative weaknesses and away from mastered prompts.Weights are bounded by the calibration strength; in the first round, unavailable global statistics leave weights at 1.Source: §4.2 · pp. 5–6

4.3 ADAPTIVE SPARSE COMMUNICATION

  • Adaptive sparse communication assigns bandwidth by σk and uses module update efficiency to retain more informative updates under tighter budgets.Lossless sparse encoding yields approximately 32× compression while retaining all updated parameters; further module selection is lossy when Bk < 1.0.Source: §4.3 · p. 6

4.4 FED-GRPO ALGORITHM

  • Each round broadcasts the global model and reward statistics, runs locally calibrated GRPO and compressed update transmission, then aggregates updates and refreshes global statistics.The procedure repeats until convergence or the maximum number of rounds.Source: §4.4 · p. 7

5 CONVERGENCE ANALYSIS

Under Assumptions 1–4, Fed-GRPO converges to a neighborhood of a stationary point of the time-varying calibrated objective. The bound’s neighborhood size reflects calibration drift and adaptive sparse communication, and both floors vanish when the objective is stationary across rounds and encoding is lossless.

5 CONVERGENCE ANALYSIS

  • The analysis assumes L-smooth local objectives, uniformly bounded stochastic-gradient second moments, contractive compression, and bounded average calibration drift.Source: §5 · p. 7
  • For the terminal round, any uniformly bounded extension of the round statistics can be used without affecting the bound.Source: §5 · p. 7
  • O(1/T) convergence reaches a neighborhood of a stationary point of the time-varying calibrated objective, with drift and compression floors of O(RM) and O(K(1−B̄)G²).The theorem’s bound holds under Assumptions 1–4.Source: §5 · p. 7
  • When R = 0 and B̄ = 1, both floors vanish and Fed-GRPO recovers FedAvg’s standard O(1/T) rate in the non-convex setting.Source: §5 · p. 7

6 EXPERIMENTS

Across five math benchmarks, Fed-GRPO performs best among federated methods and approaches centralized GRPO. Its communication budget can be reduced substantially with gradual accuracy degradation, and results extend to another base model and coding tasks.

6.1 EXPERIMENTAL SETUP

  • The math setup uses five difficulty-partitioned clients and evaluates on five benchmarks, with Qwen2.5-3B-Instruct and Qwen3-4B-Instruct as base models.Federated training runs for 12 communication rounds with five local steps; coding uses a separate non-IID partition based on reference-solution length.Source: §6.1 · p. 8

6.2 MAIN RESULTS

  • Fed-GRPO performs best among federated methods and approaches the centralized upper bound across five math reasoning benchmarks.FedAvg-SW and FedAvg-UW perform worse than Local-Only; at B=100%, communication falls from 6,794.2 MB to 645.5 MB per round with zero accuracy loss.Source: §6.2 · p. 8, Table 1 · p. 8
  • At B=5%, Fed-GRPO reaches 621× parameter compression with a moderate accuracy drop of about 3–4 points versus lossless transmission.The paper presents B=5% for severely bandwidth-constrained deployments and describes the accuracy degradation as gradual.Source: Table 1 · p. 8, §6.2 · p. 9

6.3 MORE RESULTS ON OTHER BASE MODELS AND TASKS

  • Fed-GRPO consistently outperforms other federated methods on math reasoning with Qwen3-4B-Instruct and on coding tasks with Qwen2.5-3B-Instruct.The coding evaluation uses MBPP and HumanEval; the additional results support performance across different base models and tasks.Source: §6.3 · p. 9, Table 2 · p. 9, Table 3 · p. 9

6.4 ABLATION STUDIES

  • Adding signal-weighted aggregation raises average accuracy from 37.10% to 38.01%, and adding global reward calibration raises it further to 39.07%.Adaptive sparse communication at B=1.0 preserves accuracy (39.06% versus 39.07%) while reducing per-client uplink from 6,794.2 MB to 645.5 MB.Source: §6.4 · p. 9, Table 4 · p. 9

7 RELATED WORK

Related federated GRPO work addresses several different settings and design choices. Fed-GRPO focuses on standard reasoning with shared accuracy-style rewards and non-IID prompt distributions, using GRPO reward statistics to guide multiple parts of training.

7 RELATED WORK

  • Fed-GRPO targets standard federated GRPO reasoning with shared accuracy-style rewards and non-IID prompt distributions, unlike several federated GRPO approaches with other emphases.The paper identifies reward statistics µk and σk from GRPO advantage normalization as a zero-cost signal for aggregation, training, and communication.Source: §7 · p. 10
  • FedGRPO uses clients’ domain knowledge to guide privacy-preserving server updates, while FedMOA uses hypergradient-based weighting for multi-objective RL.FedMOA also uses task-aware server aggregation.Source: §7 · p. 10
  • FGRPO dynamically prioritizes learning trajectories by comparing client performance gains with their historical baselines.The paper distinguishes this approach from its own focus on reward statistics for standard federated GRPO reasoning.Source: §7 · p. 10

8 CONCLUSION

Fed-GRPO is presented as a federated framework for GRPO training that uses reward statistics to guide aggregation, local objectives, and communication. Its algorithm broadcasts a global model and reward statistics, gathers client updates and statistics, then aggregates the updates into the next global model.

8 CONCLUSION

  • Fed-GRPO combines signal-weighted aggregation, global reward calibration, and adaptive sparse communication using GRPO reward statistics.The conclusion describes the framework as supporting collaborative reasoning training across distributed data sources without sharing raw data.Source: §8 · p. 10

A FED-GRPO ALGORITHM

  • Each round broadcasts the global model and global reward statistics; clients train locally, compute reward means and standard deviations, and form model updates.Clients calibrate per-prompt objectives using local-global reward gaps when global statistics are available.Source: A FED-GRPO ALGORITHM · p. 14
  • Clients sparsely encode and, when needed, further compress updates before sending them with reward statistics; the server decompresses and aggregates updates using signal-weighted client weights.The server then updates the global reward statistics for the next round.Source: A FED-GRPO ALGORITHM · p. 14

B PROOF OF THEOREM 1

The proof tracks a round through local client updates and the resulting compressed global update. It controls compression-related error, derives a descent inequality, and telescopes the bound across rounds.

B PROOF OF THEOREM 1

  • The proof distinguishes the immediate global update from uncompressed client updates from the actual update after compressed communication.It introduces intermediate quantities for local gradients, compressed updates, and signal-weighted aggregation weights.Source: B PROOF OF THEOREM 1 · pp. 14–15
  • The compression argument bounds dropped update energy by ordering modules by efficiency and dropping the lowest-efficiency modules first.The proof uses the retained fraction of changed parameters to characterize the compression step.Source: B PROOF OF THEOREM 1 · p. 17
  • The proof combines the descent inequality with a boundary convention for the final round, then sums the inequalities across rounds to obtain a telescoped bound.The proof also balances η-dependent terms using an AM–GM choice.Source: B PROOF OF THEOREM 1 · p. 18

C EXPERIMENTAL DETAILS

Experiments evaluate federated GRPO on mathematical reasoning and coding tasks using several centralized, local, and federated baselines. Training and evaluation use matched procedures across methods, with accuracy reported across checkpoints.

C EXPERIMENTAL DETAILS

  • The study evaluates math reasoning on MATH, MATH-500, GSM8K, AIME 2024, AIME 2025, and AMC 2023, plus coding on MBPP and HumanEval.MATH training data is split across five clients by difficulty; MBPP data is partitioned by reference-solution length using a Dirichlet distribution (α = 0.5).Source: C EXPERIMENTAL DETAILS · p. 19
  • Baselines span base-model and centralized GRPO references, local-only training, FedAvg variants, sparse-communication methods, and FGRPO.FedAvg-SW uses sample-weighted aggregation, while FedAvg-UW uses uniform weights.Source: C EXPERIMENTAL DETAILS · p. 19
  • Federated methods run for 12 communication rounds with five local steps per round, using Qwen2.5-3B-Instruct and Qwen3-4B-Instruct as base models.For Fed-GRPO, global reward calibration strength is λ = 0.5, and the minimum communication budget is Bmin = 0.01.Source: C EXPERIMENTAL DETAILS · p. 19
  • Evaluation reports the best accuracy across checkpoints and uses avg@32, the mean accuracy over 32 generated samples per prompt.Federated checkpoints are saved each communication round; centralized and local-only checkpoints are saved every 10 training steps.Source: C EXPERIMENTAL DETAILS · p. 19

D.1 RESULTS ON DIFFERENT DATA HETEROGENEITY

Under subject-based data heterogeneity, Fed-GRPO outperforms all baselines across five math benchmarks. Its signal-weighted aggregation also shifts weights across rounds toward clients with higher reward standard deviations.

D.1 RESULTS ON DIFFERENT DATA HETEROGENEITY

  • Fed-GRPO consistently outperforms all baselines across five benchmarks under subject-based Dirichlet partitioning with 20 clients.The evaluation uses Qwen2.5-3B-Instruct and α = 0.5.Source: Table 5 · p. 20, §D.1 · p. 20

D.2 DYNAMICS OF SIGNAL-WEIGHTED AGGREGATION

  • Across 12 MATH communication rounds, client reward standard deviations evolve differently, and aggregation weights shift toward clients with higher σk.Some clients' σk rises while others' falls as they master their local data.Source: §D.2 · p. 20, Figure 3 · p. 20

Related papers

Loading 2610.11502v1…