PaperlayerView exact revision on arXiv

Source-linked AI summary

F-GRPO: Factorized Group-Relative Policy Optimization for Unified Candidate Generation and Ranking

Rohan Surana, Gagan Mundada, Junda Wu, Xintong Li, Yizhu Jiao, Bowen Jin, Sizhe Zhou, Tong Yu, Ritwik Sinha, Jiawei Han, Jingbo Shang, Julian McAuley

arXiv:2605.12995v1cs.LG

TL;DR

LLM-based list generation and ranking lack phase-specific feedback, while staged pipelines optimize separate modules rather than the coupled decision end-to-end. F-GRPO factorizes these phases within one rollout and improves ranking metrics across recommendation and multi-hop QA benchmarks, especially when coverage is limiting.

  • Problem

    LLM-based approaches either rank fixed candidate pools without separating coverage from ordering or stage separate modules with independently optimized proxy objectives.

  • Method

    F-GRPO jointly trains a single LLM to generate a candidate slate and rank it using separate group-relative advantages for coverage and ordering.

  • Results

    F-GRPO consistently improves Recall@k and NDCG@k over zero-shot, supervised, decoupled, and GRPO baselines across sequential recommendation and multi-hop question answering.

  • Takeaways & Limitations

    The strongest gains appear when proposal coverage is the main bottleneck, supporting phase-specific optimization for unified generation and ranking.

Abstract

from arXiv · show

Traditional retrieval pipelines optimize utility through stages of candidate retrieval and reranking, where ranking operates over a predefined candidate set. Large Language Models (LLMs) broaden this into a generative process: given a candidate pool, an LLM can generate a subset and order it within a single autoregressive pass. However, this flexibility introduces a new optimization challenge: the model must search a combinatorial output space while receiving utility feedback only after the full ranked list is generated. Because this feedback is defined over the completed sequence, it cannot distinguish whether a poor result arises from failing to generate a relevant subset or from failing to rank that subset correctly. This credit assignment gap makes end-to-end optimization unstable and sample-inefficient. Existing systems often address this by separating candidate generation from ranking. However, such decoupling remains misaligned with downstream utility because ranking is limited by the candidate set it receives. To bridge this gap, we propose a unified framework that performs both within a single autoregressive rollout and optimizes them end-to-end via factorized group-relative policy optimization (F-GRPO). Our framework factorizes the policy into candidate generation and ranking while sharing a single LLM backbone, and jointly trains them with an order-invariant coverage reward and a position-aware utility reward. To address the resulting phase-specific credit assignment problem, we use separate group-relative advantages for generation and ranking within a two-phase sequence-level objective. Across sequential recommendation and multi-hop question answering benchmarks, F-GRPO improves top-ranked performance over GRPO and decoupled baselines, outperforms supervised alternatives, and remains competitive with strong zero-shot rerankers, with no architectural changes at inference time.

1 Introduction

F-GRPO addresses the coupled problem of selecting relevant candidates and ordering them by factorizing generation and ranking within one autoregressive rollout. It assigns phase-specific group-relative credit while jointly optimizing coverage and listwise ordering, improving ranking performance across recommendation and multi-hop question-answering tasks.

  • Motivation: LLM rerankers over fixed pools cannot separate candidate coverage from ordering quality, while modular retriever–reranker systems introduce additional models and independently optimized proxy objectives.
  • Unified formulation: F-GRPO first constructs a candidate slate and then ranks it within the same autoregressive trajectory, allowing ranking to condition directly on generated candidates.This unified process is termed in-context exploration because slate construction searches the candidate space before ranking.
  • Credit assignment: Separate group-relative advantages for slate construction and ranking resolve the ambiguity caused by a single sequence-level reward conflating coverage and ordering.
  • Method: F-GRPO factorizes the policy over a proposal slate and permutation, jointly optimizing coverage and listwise ordering within a single rollout.
  • Evaluation: F-GRPO achieves consistent improvements in Recall@k and NDCG@k over zero-shot, supervised, decoupled, and GRPO baselines across sequential recommendation and multi-hop question answering.The evaluation uses MovieLens, LastFM, HotpotQA, and MuSiQue with Qwen3-4B and Qwen3.5-2B.

2 Problem Formulation

The problem is a coupled decision: the model constructs a candidate slate and then orders it, optimizing both relevant-item coverage and early placement of high-utility candidates. F-GRPO models these phases with a shared autoregressive backbone but uses distinct order-invariant and position-aware feedback to resolve their different credit-assignment needs.

  • Problem formulation: For each context x, the model constructs a slate τ of n candidates from E and selects a permutation σ defining their final ranked order.The context contains all available task information, reducing each task to one coupled slate-construction-and-ranking decision.
  • Factorized policy: A factorized policy separates slate generation from ranking while sharing one autoregressive model across both phases.The model first generates tokens defining τ, then produces an ordering over those candidates within the same decoding trajectory.
  • Feedback signals: The slate generator receives the order-invariant reward Rslate(x, τ) = ∑e∈uniq(τ) U(x, e), whereas the ranker receives the position-aware reward Rrank(x, τ, σ) = Urank(x, τσ).Rslate measures coverage of distinct relevant items, while Rrank evaluates the reordered list τσ.
  • Feedback signals: Applying one scalar reward to both phases would conflate coverage and ordering objectives, obscuring which phase caused success or failure.The learning objective therefore trades off candidate coverage and ranking quality with λ ≥ 0, and Section 3 optimizes it using factorized group-relative credit assignment.

3 F-GRPO: Factorized Group-Relative Policy Optimization

F-GRPO factorizes the unified autoregressive rollout into slate generation and ranking, assigning phase-specific group-relative advantages to close the credit-assignment gap. Its combined objective trains both phases through a shared backbone while preserving ranking’s dependence on the generated slate.

  • Factorized credit assignment: Each rollout produces a slate τ(i) and ranking permutation σ(i), yielding separate coverage and ranking rewards for phase-specific optimization.F-GRPO computes separate group-relative advantages rather than combining the two scalar rewards into one.
  • Factorized credit assignment: Unlike standard GRPO, F-GRPO applies different rollout-level advantages to slate and rank token subsequences within the same autoregressive rollout.The slate generator receives signal from coverage quality, while the ranker receives signal from ranking quality.
  • Two-phase objective: The rank loss conditions on the prompt augmented with the generated slate, preserving the autoregressive dependency between candidate generation and ordering.Conditioning on the slate makes the ranker learn to order the specific candidates it receives.
  • Two-phase objective: The total loss is L(θ) = Lslate + λ Lrank + βKL DKL(πθ∥πref), with both phases optimized on shared parameters through a single backward pass.Rollouts missing the required delimiter tags receive a constant format penalty pfmt < 0 instead of the computed reward.
  • Comparison with standard GRPO: A single GRPO advantage contaminates both phases: ranking quality can influence slate gradients, while good coverage can be penalized by poor ranking.F-GRPO’s first-order separability eliminates this cross-phase gradient contamination.

4 Experiments

Experiments evaluate F-GRPO on sequential recommendation and multi-hop QA across multiple LLM scales and baselines. Results show gains from RL and factorized credit assignment, plus stronger robustness than decoupled pipelines.

  • Tasks and datasets: Experiments cover recommendation on LastFM and MovieLens and multi-hop QA on MuSiQue and HotpotQA, selecting and ranking from 20 candidates.Recommendation ranks items, while QA selects 2–4 gold evidence passages.
  • Sequential recommendation: F-GRPO improves over SFT by +53.7% relative on LastFM Recall@3 and +82.9% relative on MovieLens Recall@3.These gains support sequence-level reward optimization over token-level imitation.
  • Sequential recommendation: F-GRPO improves over GRPO most at broader cutoffs, including a +10.6% relative gain in LastFM Recall@5.The broader slate coverage compounds with the ranker’s ordering.
  • Decoupled baselines: Decoupled pipelines underperform because rankers trained on gold slates face generated-slate distribution shifts, whereas F-GRPO co-adapts both phases during training.This pattern appears in both recommendation and QA, where even strong decoupled variants remain below F-GRPO when coverage matters.
  • Multi-hop QA: On MuSiQue, F-GRPO gains +13.2% relative in Qwen3-4B Recall@3; on HotpotQA, Qwen3.5-2B gains +5.9% relative at Recall@3.The 4B HotpotQA models are essentially tied, indicating larger benefits when coverage is harder.
  • Reranker baselines: F-GRPO 4B models outperform all dedicated rerankers on HotpotQA and exceed MonoT5 at MuSiQue R@1 and @3 while remaining competitive at @5.Dedicated rerankers are trained on MS MARCO and applied zero-shot, whereas F-GRPO is trained in-domain.

5 Analysis

Analysis shows that recall-based slate rewards, phased optimization, and factorized roles shape F-GRPO’s behavior. Ablations identify λ=1.0 and slate size n=10 as robust defaults for balancing ranking quality and coverage.

  • Slate reward formulation: Recall rewards improve monotonically, yielding +48% relative Recall@5 on LastFM and +85% on MovieLens, whereas F1 rewards saturate early.F1 saturates by k=3 on LastFM and k=4 on MovieLens because its precision penalty discourages candidates beyond the gold set.
  • Phase separation: The slate generator reaches 90% of peak recall by step 150, while the ranker reaches 90% of peak NDCG by step 200.The ranker depends on the slate generator producing an informative candidate set before it can rank effectively.
  • Precision–recall redistribution: At convergence, the ranker raises precision over the slate from 0.090 to 0.163 for Qwen3-4B and from 0.082 to 0.140 for Qwen3.5-2B.The slate generator instead maintains high recall: 0.90 for Qwen3-4B and 0.82 for Qwen3.5-2B.
  • Ablation studies: Sensitivity analysis supports λ=1.0 and slate size n=10 as robust defaults across Recall@k and NDCG@k.λ=0.5 degrades top-position quality, n=5 limits coverage, and n=15 dilutes the pool.

6 Related Work

Prior work applies LLMs to retrieval and ranking through generative retrieval, pointwise/pairwise/listwise reranking, and reinforcement-learning-based ranking. F-GRPO builds on critic-free group-relative policy optimization, including GRPO and its refinements.

  • LLM Retrieval and Ranking: LLM retrieval and ranking methods include direct document-identifier generation and pointwise, pairwise, or listwise reranking over candidate pools.The cited paradigms span generative retrieval and candidate-pool reranking.
  • LLM Retrieval and Ranking: Neural PG-RANK represents reinforcement-learning approaches that optimize ranking with LLMs.The passage identifies Neural PG-RANK as training a Placke…
  • Policy Optimization: GRPO replaces PPO’s critic with group-relative normalization and is widely adopted for large-scale reasoning-oriented reinforcement-learning training.GRPO follows PPO’s use in LLM alignment through RLHF while removing the critic.
  • Policy Optimization: Recent GRPO refinements include Dr. GRPO’s unbiased normalization, DAPO’s decoupled clipping and token-level loss, and GVPO’s provably optimal group weights.These methods modify normalization, clipping, token-level optimization, or group weighting.

7 Conclusion · A Theoretical Analysis

The paper concludes that F-GRPO jointly trains candidate generation and ranking with phase-specific group-relative advantages, addressing limitations of single-reward GRPO and decoupled pipelines. Its theoretical analysis formalizes first-order separability of the two credit signals, while experiments span sequential recommendation and multi-hop QA.

  • 7 Conclusion: F-GRPO trains a single LLM to jointly generate and rank candidates through separate group-relative advantages for each phase.This factorized optimization framework treats generation and ranking as distinct phases within one model.
  • 7 Conclusion: Separate phase-specific advantages eliminate the gradient interference associated with single-reward GRPO.The conclusion identifies gradient interference as a limitation of using one reward signal for both phases.
  • 7 Conclusion: F-GRPO eliminates the distribution mismatch arising in decoupled retrieval-and-ranking pipelines.The framework retains joint optimization while separating credit signals across phases.
  • A Theoretical Analysis: The theoretical analysis formalizes the first-order separability of the generation and ranking credit signals.This analysis provides the stated theoretical basis for using separate advantages in the two phases.
  • 7 Conclusion: Experiments evaluate F-GRPO across sequential recommendation and multi-hop question answering.The conclusion reports empirical evaluation on both benchmark categories.
  • A Theoretical Analysis: AI writing tools assisted with drafting and verifying the paper’s theoretical proofs.This is reported as a usage disclosure in the conclusion materials.

A.1 Gradient Derivation for Two-Phase Sequence-Level Loss · B Dataset Details

At θ = θold, the two-phase sequence-level loss decomposes into separate GRPO-style updates for slate generation and ranking, each driven only by its phase-specific reward. This establishes phase-specific credit assignment despite shared model parameters.

  • A.1 Gradient Derivation for Two-Phase Sequence-Level Loss: The slate and ranking updates receive no cross-phase reward information at the expansion point.Each gradient term is weighted solely by its own phase-specific reward signal.
  • A.1 Gradient Derivation for Two-Phase Sequence-Level Loss: At θ = θold, the combined gradient decomposes into slate, weighted ranking, and KL-gradient terms.The decomposition is ∇θL = ∇θLslate + λ ∇θLrank + βKL ∇θDKL.
  • A.1 Gradient Derivation for Two-Phase Sequence-Level Loss: The slate gradient applies a phase-specific group-relative advantage uniformly across slate content tokens with per-rollout length normalization.This is the standard REINFORCE gradient specialized to the slate phase.
  • A.1 Gradient Derivation for Two-Phase Sequence-Level Loss: The ranking gradient analogously applies its ranking-phase advantage to ranking tokens conditioned on the generated slate.The rank loss uses per-token ratios for the rank segment.
  • A.1 Gradient Derivation for Two-Phase Sequence-Level Loss: Although both phases update shared parameters θ, the slate optimizes coverage quality while ranking optimizes ordering quality without cross-phase interference.The advantage weighting determines each phase’s update direction from its own reward alone.
  • A.1 Gradient Derivation for Two-Phase Sequence-Level Loss: The combined loss is equivalent to two GRPO optimizations on shared parameters, with λ controlling the relative importance of ranking quality.Each optimization operates on a different token subsequence of the same rollout and uses a different reward function.

B.1 Dataset Statistics … E.2 GRPO Hyperparameters

The appendix specifies the datasets, structured two-phase implementation, baselines, and training procedures used to evaluate F-GRPO. It standardizes candidate-pool construction and output handling while detailing SFT initialization, comparison systems, and GRPO evaluation settings.

  • B.2 Dataset Formulation: HotpotQA selects and ranks exactly 2 gold bridge passages among 20 candidates, whereas MuSiQue uses 2–4 gold passages among 20 and constructs disjoint validation and test subsets from validation data.MuSiQue subsamples 10,000 training examples and uses abstract passage IDs after shuffling.
  • C Implementation Details; C.1 Structured Output Format: All tasks generate <SLATE> and <RANK> segments in one autoregressive pass, with task-specific item representations and normalization applied during reward parsing.Recommendation ranks movie or artist names by predicted preference; QA ranks passage IDs by query relevance.
  • C.2 Delimiter Masking: Delimiter tokens remain in the forward pass but are excluded from the two-phase loss through position-based masking of opening and closing delimiter positions.The implementation caches delimiter counts and masks the corresponding positions within each segment.
  • C.3 Malformed Output Handling; C.4 Rank Subset Validation: Malformed rollouts receive case-specific rewards: normal rewards when both tags appear, a rank format penalty when only <SLATE> appears, and penalties for both phases when <SLATE> is missing.The format penalty is pfmt = −1.0, and rank outputs containing items absent from the slate are also penalized.
  • D Baseline Details; D.1 Recommendation Baselines; D.2 QA Reranking Baselines; D.3 LLM Baselines: Baselines share candidate pools, data splits, and metrics; recommendation comparisons include staged, single-step, supervised, decoupled, and GRPO paradigms, while QA rerankers reorder all passages without selection.QA reranking baselines are zero-shot and include RankZephyr, MonoT5, DuoT5, and LiT5.
  • E Training Details; E.1 SFT Warm-Start; E.1.1 Factorized SFT (Ours); E.1.2 Baseline SFT; E.1.3 Decoupled SFT: RL methods use SFT warm-starts because structured slate-and-rank generation otherwise causes malformed outputs and sparse rewards; factorized SFT trains both tags, whereas baseline and decoupled SFT separate their targets.Decoupled SFT trains selector and ranker specialists independently, chains them through two LLM calls at inference, and trains the ranker on gold slates.
  • E.1.4 SFT Hyperparameters; E.2 GRPO Hyperparameters: SFT checkpoints initialize GRPO with learning rate 5 × 10−6, or 2 × 10−6 for Qwen3.5-2B on QA; GRPO sets βKL = 0 and evaluates greedily with T=0 and Geval=1.Shared SFT and GRPO configurations are summarized in Tables 5 and 6.

E.3 Ablation Results · F Additional Results · F.1 Optimization dynamics.

The ablations identify λ=1.0 and slate size n=10 as the strongest settings, while additional metrics and optimization curves show consistent advantages for F-GRPO. F-GRPO also converges faster and reaches higher performance than GRPO, especially at higher cutoffs.

  • E.3 Ablation Results: λ=1.0 provides the best or near-best performance across metrics, whereas λ=0.5 degrades top-position quality by weakening ranking-loss gradients.The results confirm that the ranker requires sufficient gradient signal.
  • E.3 Ablation Results: All metrics peak at slate size n=10; n=5 limits candidate coverage, while n=15 dilutes the pool with low-relevance items.The larger slate slightly degrades Precision@1 and NDCG@1.
  • F Additional Results: The additional recommendation results report Precision@k for LastFM and MovieLens alongside the main-text Recall@k and NDCG@k metrics.Table 7 contains these Precision@k results for Qwen3-4B and Qwen3.5-2B, with values reported as percentages.
  • F Additional Results: Across all cutoffs, F-GRPO achieves the best or second-best performance among LLM-based methods on Precision@k and Hit@k for MuSiQue and HotpotQA.These trends are consistent with the main-text results.
  • F.1 Optimization dynamics.: On MovieLens, F-GRPO reaches Recall@5≈0.70 while GRPO plateaus at≈0.60, with the clearest gains appearing at higher cutoffs.The factorized slate combines broader coverage with ranking quality, producing the most pronounced advantage on Recall@5 and NDCG@5.
  • F.1 Optimization dynamics.: F-GRPO converges faster than GRPO, achieving strong performance within the first 1,000 steps on LastFM and converging to higher values on most metrics across both datasets.The comparison uses Qwen3-4B evaluation curves.

F.2 Error attribution

F-GRPO attributes test errors to failures in either candidate generation or ranking. Because both phases contribute errors on both datasets, the findings support using phase-specific advantages.

  • Error attribution: F-GRPO separates errors into slate misses, where the gold item is absent, and rank drops, where it is present but not surfaced.This decomposition distinguishes candidate-generation failures from ranking failures.
  • Error attribution: Errors occur across both generation and ranking phases on both datasets rather than being dominated by one phase.The balanced distribution validates the factorized design.
  • Error attribution: The cross-phase error distribution supports phase-specific advantages over a single undifferentiated GRPO advantage.If errors were concentrated in one phase, phase-specific advantages would offer little benefit over GRPO.

G Representative Examples … J.2 Reinforcement Learning for LLMs

The paper supplements its main results with representative HotpotQA outputs, exact prompt templates for unified and two-step inference, and related-work context spanning LLM retrieval, reranking, and reinforcement learning. These materials show F-GRPO’s phase-specific design in examples and position it among coupled retrieval-ranking and critic-free RL methods.

  • G Representative Examples: F-GRPO correctly ranks the gold volleyball answer first, while the baseline instead places Icheon first in the Shahnoza Nazirova example.The example’s gold truth is “Volleyball at the 2014 Asian Games -- Women”; F-GRPO’s slate and ranking include it first, whereas the baseline ranks Icheon first.
  • G Representative Examples: In a second HotpotQA example, F-GRPO ranks Mariinsky Ballet and Arthur Saint-L´eon together, while the baseline inserts Larissa Lezhnina and Mariinsky Theatre.F-GRPO’s ranking matches the two named gold-truth entities; its slate additionally includes Mariinsky Theatre.
  • H Prompt Templates: The factorized prompt makes the model emit a slate and ranking in one pass using <SLATE>...</SLATE> and <RANK>...</RANK> tags, unlike the baseline’s ranking-only prompt.Both prompts specify “up to 10” slate items and “top 5” ranked items, matching training-time output caps.
  • I Two-Step Zero-Shot Prompt Templates: The two-step zero-shot baseline uses two greedy-decoded LLM calls: the first generates a slate, and the second ranks that slate as context.It uses T=0 decoding and a single rollout per prompt, with system prompts shared with the factorized templates.
  • I.2 Question Answering: For question answering, Step 1 selects relevant passage IDs, and Step 2 ranks the selected IDs by relevance without inventing identifiers.Both stages require outputs containing only passage IDs from the provided candidate list.
  • J Extended Related Work: Related work spans generative retrieval, fixed-pool LLM reranking, and methods that jointly optimize candidate generation with final ordering.The paper situates F-GRPO among approaches including generative document identification, listwise reranking, and coupled candidate-generator/ranker optimization.
  • J.1 LLM-based Retrieval and Reranking: F-GRPO differs from prior coupled methods by using one autoregressive rollout with separate slate-quality and within-slate-ordering signals, extending critic-free RL’s credit-assignment focus.The RL discussion contrasts GRPO and later credit-assignment refinements with offline objectives such as DPO and MaxRL.
Loading 2605.12995v1…