Source-linked AI summary
Function-Structured Reinforcement Learning with Executable Verifiers for Mathematical Reasoning
Zihan Liu, Xurong Xie
TL;DR
Reliable mathematical reasoning requires decomposition and computation, but execution and final-answer feedback do not by themselves verify intermediate correctness. FSG-RL combines function graphs, executable code, and multi-verifier GRPO with staged SFT training. Verifier-guided GRPO improves final-answer accuracy and complete solution success over SFT, with additional observed gains from teacher supervision.
Problem
Tool execution does not establish that a function implements the intended computation or that intermediate results are correctly combined, while final-outcome feedback leaves a question about local credit for executable reasoning.
Source: §1 · p. 1Method
FSG-RL trains policies first with SFT to construct function graphs and generate code, then with GRPO using execution, tests, dependency checks, and final-outcome verification.
Source: §1 · p. 1Results
Takeaways & Limitations
The reported gains extend beyond valid interfaces to correct answers and implementations that pass verification and support those answers.
Source: §4 · p. 4, §3.2 · p. 4, §3.3 · p. 4The program output may match the reported answer even when both are incorrect, so the answer-match signal alone cannot establish correctness.
Source: §2.2 · p. 2
Abstract
from arXiv · showhide
Algorithmic mathematical reasoning requires reliable decomposition, computation, and aggregation. Final-answer rewards provide limited guidance on intermediate errors, while successful execution does not guarantee mathematical correctness. This work proposes Function-Structured Graph Reinforcement Learning (FSG-RL), connecting subproblem graphs and Python implementations with multi-verifier feedback. The policy first learns to generate code from function graphs through supervised fine-tuning (SFT). Group Relative Policy Optimization (GRPO) then optimizes the policy using answer-gated rewards and span-level credit assignment. The framework also supports teacher supervision and structured memory. A benchmark curated from Grade School Math 8K (GSM8K), MathQA, MATH, and Omni-MATH pairs public function graphs with private verification specifications. Under a unified evaluation protocol, GRPO improves final-answer accuracy from 43.25% to 67.50% and full solution success from 32.25% to 52.25% over SFT. Continued reinforcement learning (RL) with teacher supervision yields additional gains. The gains extend beyond producing correctly formatted code, supporting verifier-guided reinforcement learning for mathematical reasoning. Code is available at https://github.com/ZihanLiummyycc/FSG-RL.
1. INTRODUCTION
Mathematical reasoning requires decomposition, computation, and aggregation, while tool execution and final-answer feedback alone leave intermediate correctness difficult to verify. FSG-RL connects subproblem graphs, executable Python functions, and multi-verifier feedback, with staged training and modular extensions.
1. INTRODUCTION
- Tool-integrated methods can execute computations, but execution alone does not establish that a function implements the intended computation or that outputs are correctly combined.This motivates checks of intermediate behavior and subproblem consistency.Source: §1 · p. 1
- Existing intermediate-feedback approaches leave open how local credit should connect to function behavior, dependency consistency, and final-answer correctness.The question arises for executable mathematical reasoning despite prior advances in finer-grained learning signals.Source: §1 · p. 1
- FSG-RL connects typed subproblem dependency graphs, executable Python interfaces, and multi-verifier rewards through staged SFT-to-GRPO training.Answer gating reduces intermediate-step reward weight when the final answer is incorrect, while span-level credit assigns verification signals to corresponding code spans.Source: §1 · p. 1
- The curated executable mathematical-reasoning benchmark uses public function graphs and private verification specifications to support evaluation of interface compliance across problems drawn from four datasets spanning varied mathematical topics and difficulty levels.The problems come from GSM8K, MathQA, selected Level 1–3 MATH problems, and Omni-MATH.Source: §1 · p. 1
- Verifier-guided GRPO improves solution correctness beyond the interface compliance established by SFT, and teacher supervision provides additional observed gains.The framework also reports that its trained policy outperforms the API reference for all outcome metrics.Source: §1 · p. 2
2. METHOD
The method represents problems as dependency graphs with executable Python nodes and checks generated solutions using private verification. It combines gated multi-verifier rewards with span-level credit assignment in GRPO.
2.1. Function Graphs and Private Verification
- Function graphs encode necessary subproblems and their dependencies, while private tests check executable behavior and compare a specified function call with the reported answer.An optional memory module retrieves Mathlib4 theorem statements with natural-language descriptions for retrieval-augmented generation.Source: §2.1 · p. 2
2.2. Multi-Verifier Rewards
- The reward combines interface validity, local functional evidence, and final-answer correctness, while answer gating reduces process credit for incorrect answers.Local checks include unit and property tests and dependency evidence; program-answer agreement is insufficient because both may be wrong.Source: §2.2 · pp. 2–3
2.3. Span-Level Group Relative Optimization
- Span-level GRPO combines node verification scores, adjacent-edge evidence, and shared outcome credit, then normalizes total and node rewards separately across a problem’s rollouts.Node content tokens receive the corresponding node advantage; outcome credit is shared, so span rewards need not sum to the total reward.Source: §2.3 · p. 3
3. EXPERIMENTS AND ANALYSIS
The experiments evaluate executable mathematical reasoning on a benchmark built from four datasets, using common public graphs and private checks. They compare staged SFT and GRPO, teacher-supervised continuation, and an external API reference, while distinguishing execution from complete correctness.
3.1. Benchmark and Evaluation Protocol
- The benchmark pairs problems from GSM8K, MathQA, selected MATH, and Omni-MATH with public function graphs and private verification specifications.It contains 866 GRPO training problems and 400 evaluation problems; models receive common public graphs and identical private checks.Source: §3.1 · p. 3
- Full success requires valid tags, complete code blocks, matching signatures, successful execution, all hidden tests passing, and agreement between the reported and gold answers.Dependency score contributes to training rewards but is excluded from this evaluation criterion.Source: §3.1 · p. 3
3.2. Staged SFT to RL Training
- SFT teaches the required interfaces, then GRPO uses implementation-correctness differences to guide policy updates.The comparison includes base, SFT, and GRPO policies; the exact McNemar test favors GRPO over SFT for final-answer correctness (p = 1.72 × 10^-18).Source: §3.2 · p. 4
- GRPO yields larger gains in final-answer accuracy and full success than in execution success after SFT largely resolves structured-format difficulties.Format compliance changes little during GRPO, so the gains extend beyond producing valid code.Source: §3.2 · p. 4, Table 1 · p. 4
3.3. Teacher Supervision and Qwen API Comparison
- FC-GRPO continuation on 60 hard problems shows additional gains under teacher-free evaluation, while both GRPO-trained policies exceed the Qwen3.8-27B API reference in final-answer accuracy and full success.The API comparison uses one response per problem at temperature zero and a 2,048-token limit.Source: §3.3 · p. 4, Table 2 · p. 4
3.4. Illustrative Case Study
- Two constructed programs execute successfully and match their reported answers, but only the parity-aware solution is correct; an even-branch test detects the missing condition.Execution and answer consistency alone do not establish correctness.Source: §3.4 · p. 4, Figure 2 · p. 4
4. CONCLUSION
The conclusion presents FSG-RL as a framework connecting mathematical decomposition, executable implementations, and multi-verifier feedback. Its reported results show gains from verifier-guided GRPO and further observed gains from teacher supervision.
4. CONCLUSION
- FSG-RL connects mathematical decomposition, executable implementations, and multi-verifier feedback through staged SFT-to-RL training, teacher supervision, and structured memory.The release includes annotated training records and public evaluation inputs, while private verifier specifications remain withheld.Source: §4 · p. 4
- Verifier-guided GRPO improves both final-answer accuracy and complete solution success, and teacher supervision provides further observed gains.Under the reported inference settings, Qwen3.5-9B exceeds the Qwen3.8-27B API reference across reported verification and outcome metrics.Source: §4 · p. 4