FAPO: Flawed-Aware Policy Optimization for Efficient and Reliable Reasoning

1 INTRODUCTION

• RLVR gives the same positive outcome reward to flawed-positive and fully correct rollouts, potentially reinforcing unreliable reasoning such as answer-guessing and jump-in-reasoning.The paper identifies analyzing flawed positives through RL and developing mitigation strategies as open needs.Source: §1 · p. 1
• FAPO improves outcome correctness across AIME24, AIME25, and GPQA-Diamond while reducing flawed-positive ratios and improving training stability, efficiency, and process reliability without increasing the token budget.Figure 1 compares flawed-positive ratios and performance; the reported advantages appear at nearly all intermediate evaluation checkpoints.Source: Figure 1 · p. 2, §1 · p. 2
• Flawed positives accelerate capability gains early but may later hinder learning, motivating FAPO’s parameter-free penalty and gradual shift toward reliable problem-solving.The proposed training trajectory initially exploits flawed positives as shortcuts, then shifts the objective as model capabilities advance.Source: §1 · p. 2
• FAPO introduces a generative reward model with process-level rewards to detect flawed positives and locate intermediate reasoning errors.The paper reports that FAPO-GenRM-4B improves on FlawedPositiveBench and ProcessBench.Source: §1 · p. 2

2.1 PROBLEM DEFINITION AND MOTIVATION

• Flawed positives are correct-answer trajectories with at least one logically invalid reasoning step, and rule-based rewards can reinforce them alongside correct solutions.Prior work cited in the section reports flawed-positive rates as high as 50%; their mechanisms and RL impact remain underexplored.Source: §2.1 · p. 3

2.2 FLAWED POSITIVE ANALYSIS IN REINFORCEMENT LEARNING

• Across three evaluated models on DAPO-Math, flawed positives account for 20%–40% of correct rollouts.Qwen3-32B judges whether reasoning trajectories contain unreliable patterns, alongside a manual case study.Source: §2.2 · p. 3
• In a simulated learning-stage analysis, flawed positives are most prevalent early and diminish as training progresses, consistent with a stepping-stone role before fully correct solutions emerge.Learning stages are approximated by rollout accuracy, with samples grouped by that accuracy.Source: §2.2 · p. 3
• During RL on DAPO-Math, Qwen2.5-Math-7B’s flawed-positive ratio stays near 30% even as rollout accuracy improves, indicating difficulty shifting to genuine problem-solving.The section attributes a concern to flawed positives receiving the same rewards as correct solutions, thereby reinforcing unreliable reasoning and hindering progress.Source: §2.2 · pp. 3–4
• Penalizing flawed positives in preliminary RL trials yields significant AIME24 performance gains over baseline RLVR, though improvements emerge more gradually early in training.The trials use Qwen3-32B to detect flawed positives and assign them negative signals like negative rollouts.Source: §2.2 · p. 4

3 FAPO: FLAWED-AWARE POLICY OPTIMIZATION

• FAPO combines a compact generative reward model with adaptive penalization, aiming to use flawed positives during warm-up and suppress them as reliable solutions become more common.The optimization direction follows whether positive or negative samples dominate; the default penalization strength is λ = 1.Source: §3 · p. 4, §3.2 · pp. 5–6

3.1 FLAWED POSITIVE DETECTION

• Existing detectors often over-criticize: they can achieve high recall but low precision by flagging minor or unnecessary errors, while stronger models remain too slow for online RL.FlawedPositiveBench is constructed from positive samples, including flawed ones, collected from ProcessBench.Source: §3.1 · pp. 4–5
• Flawed-positive detection is evaluated using precision, recall, and F1, which respectively measure prediction correctness, coverage of true flawed positives, and their balance.Figure 3 compares generative models with discriminative PRMs; Table 2 reports subset-level results and additional models.Source: §3.1 · p. 4, Figure 3 · p. 5
• GenRM training adds a distance-sensitive step-wise penalty that rewards locating an error nearer its true step, encouraging precise detection rather than guessing.The reward design naturally shifts emphasis from prediction correctness toward process optimization as correctness saturates, without an additional controlling hyperparameter.Source: §3.1 · p. 5

3.2 FLAWED POSITIVE PENALIZATION

• Detected flawed-positive rollouts receive a −λ reward penalty, with λ controlling its strength relative to the standard RLVR baseline.The majority-guided strategy sets λ = 1 by default and shifts optimization toward refinement when positive samples dominate.Source: §3.2 · pp. 5–6

4.1 TRAINING DETAILS

• Training uses GRPO with common optimization strategies, an 85K flawed-positive critic dataset, and an asynchronously deployed GenRM for reasoning-task RL.The dataset identifies step-level error locations in correct-answer responses, while fully correct responses use an infinite first-error index; GenRM requests run through multiple workers and a load-balancing router.Source: §4.1 · p. 6

4.2 EVALUATION SETUP

• Evaluation covers FlawedPositiveBench and ProcessBench for detection, plus AIME24, AIME25, and GPQA-Diamond for reasoning performance throughout training.Figure 4 compares FAPO-GenRM with baseline outcome reward models and FAPO-Reasoning with the baseline setting.Source: §4.2 · p. 6, Figure 4 · p. 7

4.3 MAIN RESULTS

• Across mathematical and general-domain benchmarks, FAPO outperforms baselines, lowers flawed-positive ratios, smooths training, and improves performance without longer responses.FAPO-GenRM also improves on FlawedPositiveBench and ProcessBench, outperforming Qwen3-32B; Figure 4 compares its training dynamics and those of FAPO-Reasoning with baselines.Source: §4.3 · p. 7, Figure 4 · p. 7

4.4 ABLATION STUDY

• Ablations report that stronger GenRM detection aligns with better RL performance, while FAPO shifts from early self-correction toward shorter, fully correct rollouts.The study also states that FlawedPositiveBench is aligned with final performance and that even small detection improvements can yield substantial performance gains.Source: §4.4 · p. 8, Figure 5 · p. 8, Figure 6 · p. 8

4.5 DISCUSSION: GENRM APPLICATION IN FUTURE RL SYSTEMS

• Fine-grained process rewards can invite reward hacking: a step-ratio reward leads policies to skip uncertain reasoning, stall later, and exhibit jump-in-reasoning.The authors also report asynchronous infrastructure and training strategies, with training time increased by less than 20% relative to the baseline.Source: §4.5 · pp. 8–9, Figure 7 · p. 8

LLM Reinforcement Learning

• RL on verifiable tasks has been associated with planning and tool use, which prior work reports can foster generalization across broader domains.The cited work frames this direction as building on RL's potential to advance LLM reasoning capabilities.Source: §5 · p. 9

Reward Models in Reinforcement Learning

• Generative reward models serve as flexible verifiers and rubric-based reward providers, while discriminative models produce dense token-, step-, or turn-level rewards.The passage notes that dense reward complexity can expose policies to reward-hacking shortcuts.Source: §5 · p. 9
• FAPO trains generative reward models to detect flawed positives and provide interpretable rewards, with experiments and theory reported as evidence of robustness and scalability.The passage positions this approach against limitations attributed to reward hacking in dense reward models.Source: §5 · p. 9

• Flawed-positive rollouts can accelerate early capability gains but later constrain reasoning quality by reinforcing unreliable patterns.Source: §6 · p. 9
• FAPO applies a parameter-free reward adjustment that uses flawed positives as warm-up shortcuts and progressively steers optimization toward reliable reasoning.Source: §6 · p. 9
• A process-level generative reward model detects flawed positives and localizes reasoning errors as part of FAPO's approach to efficient and reliable LLM RL.The conclusion cites empirical experiments and theoretical analysis as demonstrating FAPO's effectiveness in future large-scale RL systems.Source: §6 · p. 9

• The authors state that the work adheres to ethical research practices.Source: ETHICS STATEMENT · p. 10
• All data and language models used are publicly available, including through API access, and are properly cited.Source: ETHICS STATEMENT · p. 10
• The work involved no human subjects, and its statement on LLM use appears in Appendix E.Source: ETHICS STATEMENT · p. 10

REPRODUCIBILITY STATEMENT

• Algorithm design details are provided in Section 3.The statement identifies where to find the algorithm design.Source: REPRODUCIBILITY STATEMENT · p. 10
• Infrastructure setup is described in Section 4.The statement identifies where to find the infrastructure setup.Source: REPRODUCIBILITY STATEMENT · p. 10
• Dataset, training hyperparameters, and evaluation setups are presented in Section 4 and Appendix B.The statement identifies where to find these experimental details.Source: REPRODUCIBILITY STATEMENT · p. 10

Advantage estimation

• FAPO modifies flawed-positive rewards using λ, changing their group-relative advantages while GRPO treats flawed and fully correct positives alike.The analysis compares advantage estimates under these reward assignments and describes how the optimization direction evolves during RL.Source: A THEORETICAL ANALYSIS OF FAPO ALGORITHM · p. 14

Optimization-direction shift

• When the proportion of fully correct rollouts rises relative to negatives, FAPO shifts from reaching correct answers toward reinforcing reliable reasoning.The analysis represents the rollout mix using α for fully correct positives, β for negatives, and 1 − α − β for flawed positives; the shift is tied to a rising ρ = α/β.Source: A THEORETICAL ANALYSIS OF FAPO ALGORITHM · pp. 14–15

Summary: The Whole Optimization Process of FAPO

• As ρ rises beyond 4λ^-1, FAPO scales advantages more conservatively than GRPO for flawed-positive and fully correct rollouts, which the analysis links to training stability.The passage states that this conservative scaling prevents overly aggressive updates while continuing to prioritize correct and reliable rollouts.Source: A THEORETICAL ANALYSIS OF FAPO ALGORITHM · p. 15

Determining λ

• A majority-guided rule keeps flawed positives positively advantaged while negatives dominate, then shifts toward reliable reasoning when fully correct rollouts become the majority.The rule sets the transition around α > β, allowing flawed positives to support early correct-answer acquisition.Source: A THEORETICAL ANALYSIS OF FAPO ALGORITHM · pp. 15–16

FAPO-Reasoning

• GenRM training uses a higher rollout temperature for exploration, while reasoning-model training follows DAPO settings except for reducing rollouts from 16 to 8.The reduction is stated to accelerate overall training speed; Table 1 summarizes training configurations and hyperparameters.Source: B IMPLEMENTATION DETAILS · p. 16, Table 1 · p. 16

FAPO-GenRM

• GenRM data synthesis keeps an instance only when three sampled generations agree on the process-error positions.The robust objective uses soft noise-robust supervision so the student can still receive an appropriate reward when the annotated error location is slightly off.Source: B IMPLEMENTATION DETAILS · p. 17

Reward Loop Design

• The reward loop sends each rollout to the reward model as soon as it is generated, rather than waiting for the full batch.The partially asynchronous design supports custom reward functions, including DiSRM and GenRM.Source: B IMPLEMENTATION DETAILS · p. 17

C ADDITIONAL RESULTS AND DISCUSSION

• FAPO-GenRM-4B, built on Qwen3-4B-Instruct, improves on FlawedPositiveBench and ProcessBench, surpassing Qwen3-32B and Qwen2.5-Math-PRM-72B.The comparison covers publicly available discriminative and generative reward models.Source: C ADDITIONAL RESULTS AND DISCUSSION · p. 17, Table 2 · p. 18
• FAPO consistently improves across a broad range of tasks and outperforms the baseline on code reasoning benchmarks.The additional evaluation includes MATH, AMC, and LiveCodeBench; Table 3 summarizes these benchmark results.Source: C ADDITIONAL RESULTS AND DISCUSSION · p. 17, Table 3 · p. 18
• GenRM inference time stays nearly constant across model trajectories, while rollout time grows with long traces, reducing GenRM’s relative inference cost for long-CoT models.The passage gives GenRM output lengths of 3.2k → 3.8k and cites 12k-token Qwen3-4B trajectories as an example of long rollouts.Source: C ADDITIONAL RESULTS AND DISCUSSION · p. 18, Table 4 · p. 19
• All tested λ configurations outperform the baseline, but tuning is needed; ρ = 1/2 performs best when about one third of rollouts are fully correct.The passage characterizes the 7B setting’s more aggressive strategy as producing larger gains; Table 5 covers λ tuning.Source: C ADDITIONAL RESULTS AND DISCUSSION · p. 18, Table 5 · p. 19
• Human verification and LLM-as-a-judge evaluation both support FAPO’s improvement over the baseline in mitigating flawed positives.The human review samples 20 positive cases with correct final answers and notes that multiple-choice questions have a higher flawed-positive proportion than math word problems.Source: C ADDITIONAL RESULTS AND DISCUSSION · pp. 18–19, Table 6 · p. 19

D LIMITATIONS AND FUTURE WORK

• The work’s stated limitations are that broader algorithmic settings and model architectures remain to be validated, and fully asynchronous RL applicability is uncertain.The authors identify multi-choice, multi-turn, agent-based RL, MoE models, and larger scales as directions for future validation.Source: D LIMITATIONS AND FUTURE WORK · p. 20

E LLM USAGE

• The authors report that AI assistants were used only for minor language refinement, with suggestions reviewed and revised by the authors.They state that the assistance did not influence research design, experiments, or conclusions.Source: E LLM USAGE · p. 20

F PROMPTS

• The supplementary prompts ask a model to critique solutions step by step, identify the earliest error, or judge correctness against a ground-truth answer.The prompt set also includes a process-reward prompt for paragraph-indexed solutions.Source: F PROMPTS · pp. 20–21

G FLAWED POSITIVE SAMPLES WITH HUMAN ANALYSIS

• In the AIME24 example, the answer is correct despite an incorrect set count and a guessed value near the target.The human analysis identifies the count error as making the derivation fundamentally flawed.Source: G FLAWED POSITIVE SAMPLES WITH HUMAN… · p. 23
• In the cubic-roots example, the response makes a simplification error and ultimately guesses k = 5 after failed attempts to correct itself.The human analysis says the correct equation is k3 −6k2 + 12k −35 = 0, not k3 −6k2 + 24k −35 = 0.Source: G FLAWED POSITIVE SAMPLES WITH HUMAN… · p. 24
• In the common-root example, the response assumes α = 1 without justification and reaches the correct sum coincidentally.The human analysis says deriving α = 1 requires a complex reasoning process.Source: G FLAWED POSITIVE SAMPLES WITH HUMAN… · p. 25