PaperlayerView exact revision on arXiv

Source-linked AI summary

From Outcomes to Strategies: Learning Strategy Utility for Mathematical Reasoning

Ruikang Zhang, Xiao An, Xuli Shen, Jiaxing Sun, Xiaoyi Yu, Jin Zeng, Jiang Wu, Tong Lin

arXiv:2609.32482v1cs.AI

TL;DR

Terminal correctness indicates whether mathematical reasoning succeeds, but provides limited insight into high-level strategies separately from their execution. SURE learns relative strategy utility from pairwise preferences and uses a frozen Strategy Reward Model alongside correctness and format rewards during policy optimization. Across three backbones, it improves average pass@1 over matched outcome-and-format GRPO baselines by 1.87%, 2.64%, and 2.93%.

  • Problem

    Terminal correctness evaluates the final answer but does not identify which strategic decisions made success probable, although strategy selection and execution are distinct.

    Source: Introduction · p. 1

  • Method

    SURE learns relative strategy utility from pairwise preferences and uses a frozen Strategy Reward Model score with correctness and format rewards during sequence-level GRPO.

    Source: Introduction · p. 2

  • Results

    1.87%, 2.64%, and 2.93% average pass@1 gains over matched outcome and format GRPO baselines across three policy backbones.

    Source: Introduction · p. 2

  • Takeaways & Limitations

    SURE provides a way to reuse strategic supervision across training runs by separating offline strategy evaluation from policy optimization.

    Source: Conclusion · p. 8

    The SRM ranking analysis supports relative group-level utility, not calibration of individual strategy scores, because each strategy receives only one binary answer label.

    Source: SRM Utility Ranking Details · p. 12

Abstract

from arXiv · show

Reinforcement learning with verifiable rewards has substantially improved mathematical reasoning. However, terminal correctness alone provides limited insight into the quality of high-level strategies, such as theorem selection and subgoal decomposition, when considered separately from their subsequent execution. This paper studies strategy utility, which is defined as the likelihood that a strategy supports a correct downstream solution under a given executor. We introduce SURE, a framework for learning and leveraging relative strategy utility. In this framework, high-level strategies are separated from their detailed reasoning. Based on the pairwise preferences constructed from strategy-conditioned rollouts and teacher-generated contrasts, a Strategy Reward Model is learned to estimate relative strategy utility. During reinforcement learning, the frozen reward model reads only the extracted strategy, whose score is combined with the correctness and format rewards in a sequence-level GRPO objective. Compared with outcome-and-format GRPO baselines, experiments show that SURE improves average pass@1 by 1.87%, 2.64%, and 2.93% across three policy backbones. Our method also achieves competitive or better accuracy than stronger reward baselines while requiring substantially lower GRPO-stage compute.

Introduction

Terminal correctness reveals whether a solution succeeds but not which strategic choices made success likely, motivating supervision that separates strategy from execution. SURE learns relative strategy utility from pairwise preferences and adds it to policy optimization alongside correctness and format rewards.

  • Terminal correctness conflates strategy selection with detailed execution, so the final answer gives limited guidance about which solution route the policy should learn.A sound strategy can fail through execution error, while a fragile one can succeed through an unusual derivation or lucky correction.Source: Introduction · p. 1, Figure 1 · p. 1
  • Strategy utility measures the likelihood that a strategy supports a correct solution under a specified executor and decoding configuration, rather than as an intrinsic property.The learned SRM is a relative ranking signal under that reference setting, not a universal estimator for all downstream executors.Source: Introduction · pp. 1–2
  • Unlike prior planning methods that generally use plans as prompting formats or inference procedures, SURE asks whether relative strategy utility can be learned as a training signal.Source: Introduction · p. 2
  • SURE teaches a policy to emit a high-level strategy, then trains an SRM on pairwise preferences from confidence-filtered strategy rollouts and teacher-generated structural contrasts.The SRM estimates relative utility from the problem and strategy alone, without observing execution or the final answer.Source: Introduction · p. 2
  • During GRPO, SURE freezes the SRM and combines its group-normalized utility score with correctness and format rewards, suppressing utility for incorrect responses.This design preserves terminal correctness as the primary objective and reduces incentives to exploit the learned reward.Source: Introduction · p. 2
  • 1.87%, 2.64%, and 2.93% average pass@1 gains over matched outcome and format GRPO baselines across three policy backbones.Source: Introduction · p. 2

Related Work

SURE makes high-level strategy an explicit learning target, distinct from detailed execution, and estimates its relative utility for policy optimization. Its frozen Strategy Reward Model (SRM) scores the problem and strategy, while strategy-conditioned rollouts and teacher contrasts supply preference data.

Explicit Plans and Decompositions

  • Planning methods expose high-level decisions through plans or decompositions, while SURE treats strategy as an explicit learning target and estimates its relative utility for policy optimization.Prior work also uses plans to guide inference-time search.Source: Explicit Plans and Decompositions · p. 2

Reward Models for Mathematical Reasoning

  • SURE introduces strategy utility as an intermediate target between rewards for complete solutions and process rewards for individual reasoning steps.The SRM uses execution evidence to learn preferences but scores only the problem and extracted strategy.Source: Reward Models for Mathematical Reasoning · p. 2

Reinforcement Learning with Planning-Aware Rewards

  • SURE learns relative strategy utility offline and reuses a frozen SRM during optimization, rather than repeatedly evaluating plan quality through online executions.The SRM scores the problem and strategy, not the individual rollout’s execution tokens; experiments use GRPO, but the utility definition and SRM training do not inherently depend on GRPO.Source: Reinforcement Learning with… · p. 3

Strategy and Execution Decomposition

  • A response is decomposed into a high-level strategy specifying tool choice, intermediate targets, and subgoal order, and an execution containing the concrete derivation and final answer.Strategy utility is the probability of a correct solution under a fixed executor and decoding configuration.Source: Strategy and Execution Decomposition · p. 3

Strategy-Guided SFT

  • SFT uses 17,467 manually verified question–strategy–execution examples, with teacher-generated strategies and reference executions under structured tags.The policy learns the full sequence through standard next-token prediction.Source: Strategy-Guided SFT · p. 4

SRM Training Data Construction

Rollout-Induced Preference Data

  • For rollout preferences, each problem yields 8 candidate strategies and 24 executions per strategy; a confidence-aware Beta-Binomial filter retains likely utility orderings before judge verification.The process leaves 48,413 rollout pairs in Drollout.Source: Rollout-Induced Preference Data · p. 4

Teacher-Constructed Contrastive Data

  • Teacher contrasts add 17,197 pairs, and the merged 65,610-pair set balances its two sources to reduce sensitivity to rollout noise and teacher-specific phrasing.The SRM uses a margin-augmented Bradley–Terry ranking loss with a scale-anchoring regularizer; rollout pairs receive confidence-derived weights, teacher pairs uniform weights, and the trained model is frozen before policy optimization.Source: Teacher-Constructed Contrastive Data · p. 4, Merging and Filtering · p. 4, SRM Training · p. 4

Experiments

Across three policy backbones, SURE improves pass@1 over matched outcome-and-format GRPO, while experiments also examine strategy utility ranking, reward combinations, ablations, and GRPO-stage compute.

Experiments

  • Experiments compare outcome-only GRPO, process-reward integration, and online plan-reward estimation across competition-math benchmarks and three policy backbones.Evaluation reports pass@1 on AIME 2024, AIME 2025, AMC 2023, and MATH500; methods vary SFT/RL initialization, reward composition, and token-matched SFT controls.Source: Experiments · p. 5, Experimental Setup · p. 5

Experimental Setup

  • The setup uses three policy backbones, manually verified strategy-guided SFT data, and a 7,200-problem DeepMath subset for GRPO.The SFT corpus contains 17,467 triples, and GRPO uses group size G=8 with KL regularization toward the SFT policy.Source: Experimental Setup · p. 5

Main Results (RQ2)

  • +1.87%, +2.64%, and +2.93%: SURE improves over GRPO from the same Strategy-SFT initialization across three backbones.Explicit strategy structure also improves outcome-only GRPO over CoT-SFT; token-matched SFT comparisons support gains before RL. On Qwen3-8B, outcome-and-format GRPO averages 57.92% versus 59.36% for the SFT checkpoint, while SURE reaches 60.56%.Source: Main Results (RQ2) · pp. 5–6, Table 1 · p. 6

SFT Teacher Strength Ablation

  • Replacing the strong SFT strategy teacher with Qwen3-4B lowers average performance by only 0.58% on Qwen3-4B.The change affects only SFT strategy construction; the SRM preference data and judge remain unchanged.Source: SFT Teacher Strength Ablation · p. 6, Table 2 · p. 6

Compatibility with Process Rewards

  • Combining PRM and SRM yields the highest average on both Qwen3 backbones, while their separate reward profiles differ.The combined averages are 61.93 on Qwen3-4B and 61.31 on Qwen3-8B; the comparison includes each reward model separately and together.Source: Table 3 · p. 6, Compatibility with Process Rewards · pp. 6–7

Do SRM Scores Track Relative Strategy Utility?

  • SRM quantiles show increasing accuracy: the top 20% exceeds the bottom 20% by 3.74% and the second-highest quantile by 2.12%.With one answer label per strategy, this evaluates relative group-level utility, not individual-score calibration; frozen-executor ranking accuracy is 68.1%, 64.8%, and 64.6% across three backbones.Source: Do SRM Scores Track Relative Strategy… · p. 7, Table 4 · p. 7

SRM Data and Reward Design Ablations

  • Removing rollout-induced or teacher-constructed preference pairs lowers average pass@1 from 60.24% to 59.69% or 59.90%, respectively.Removing either correctness gating or group-wise normalization also hurts: no gating permits a late epoch-3 regression, while no normalization diverges in epoch 3 and lowers average by 0.98%.Source: Table 5 · p. 7, SRM Data and Reward Design Ablations · p. 7, Figure 3 · p. 7

GRPO-Stage Cost Given a Pre-Trained SRM (RQ3)

  • SURE uses 45% less GRPO-stage compute than GRPO+PRM while achieving comparable accuracy.The comparison is 174.05 versus 318.30 GPU-h; SURE also outperforms PTA-GRPO on AMC23 and MATH500 while running 61 GPU-hours faster at the GRPO stage.Source: Table 6 · p. 8, GRPO-Stage Cost Given a Pre-Trained… · p. 8

Conclusion

SURE trains a Strategy Reward Model on confidence-aware pairwise preferences and uses it during policy optimization. The paper reports improved accuracy across three backbones, competitive results with lower GRPO-stage compute, and reusable strategy selection.

Conclusion

  • SURE improves over outcome-only GRPO across three backbones, integrates with step-level PRMs, and achieves competitive accuracy with substantially lower GRPO-stage compute than online plan-reward baselines.The SRM also transfers as a standalone strategy selector.Source: Conclusion · p. 8
  • SURE trains a Strategy Reward Model on confidence-aware pairwise preferences and uses it during policy optimization.The method uses correctness gating and group-wise normalization.Source: Conclusion · p. 8
  • The findings indicate that explicitly evaluating high-level decisions can improve reasoning policies beyond terminal correctness alone.Separating offline strategy evaluation from policy optimization allows strategic supervision to be reused across training runs and model backbones.Source: Conclusion · p. 8

Comparison with Online Plan-Reward

SURE differs from online plan-reward methods in when it estimates utility, what it evaluates, and how GRPO-stage cost is counted. Its strategy evaluator is trained offline and reused during policy optimization.

Comparison with Online Plan-Reward

SFT Strategy Generation Prompt

The SFT prompt distills a high-level strategy from a problem and its reference chain-of-thought solution, then verifies the resulting training triples.

SFT Strategy Generation Prompt

SRM Preference Data Construction Details

The preference data combines execution-grounded strategy comparisons with teacher-generated contrasts, followed by structural checks and judge verification.

Data Pipeline and Statistics

  • From 32,000 DeepMath and OpenThoughts problems, SURE samples 8 strategies and 24 executions per strategy, retaining 48,413 rollout pairs and 17,197 teacher pairs for 65,610 total after filtering and deduplication.The rollout pairs use a Beta-Binomial confidence filter with P[q(s+) > q(s−)] ≥ 0.9; the associated table reports preference-data statistics.Source: Data Pipeline and Statistics · p. 10, Table 7 · p. 10

Strategy Extraction and Deterministic Tag Checking

  • A deterministic parser requires one strategy block, followed by one think block and a boxed final answer, and filters format-invalid outputs before SRM training.Tag validation and answer extraction use deterministic rules, not the LLM judge.Source: Strategy Extraction and Deterministic… · pp. 10–11

LLM-Assisted Preference Reasonableness Verification

  • The judge receives the proposed preference label and checks whether the preferred strategy is mathematically sound and more likely to lead to a correct solution.For rollout-induced pairs, only pairs judged REASONABLE are retained; invalid, UNREASONABLE, and ambiguous responses are discarded.Source: LLM-Assisted Preference Reasonableness… · p. 11

Teacher-Constructed Contrastive Prompt

  • Teacher-generated contrasts add recognizable structural errors, such as a wrong theorem or infeasible subgoal order, that rollout pairs may underrepresent.Rollout pairs are limited to strategies the SFT policy can generate.Source: Teacher-Constructed Contrastive Prompt · p. 11

Implementation Details

Implementation uses a three-stage SURE pipeline: strategy-guided SFT, SRM training, and GRPO policy optimization. Evaluation samples ten answers per problem and reports mean pass@1.

Stage 1 Strategy-Guided SFT

  • Stage 1 fine-tunes Qwen3-4B to emit structured <strategy>/<think> outputs; this establishes the format rather than improving raw reasoning ability.The policy is fully fine-tuned with FSDP2 and standard next-token prediction loss.Source: Implementation Details · p. 11

Stage 2 SRM Training

  • Stage 2 trains a LoRA-adapted SRM with a margin-augmented Bradley–Terry loss and freezes it before Stage 3.SRM training takes approximately 4.38 hours; preference data construction takes 47 wall-clock hours on one eight-GPU RTX 4090 node.Source: Implementation Details · p. 11

Stage 3 SURE Policy Optimization

  • Stage 3 initializes GRPO from the Stage 1 checkpoint and combines correctness, format, and frozen-SRM rewards with group-wise normalization and correctness gating.Policy optimization uses full parameters under DeepSpeed ZeRO-2, with KL regularization following Shao et al. (2024).Source: Implementation Details · p. 11

Inference and evaluation

  • Evaluation draws 10 independent samples per problem and reports mean pass@1, using a deterministic parser and math-aware verifier for answers.Rollouts use temperature 0.6, top-p 0.95, and a maximum of 32,768 new tokens.Source: Implementation Details · p. 11

SFT Teacher Strength Ablation Details

The teacher-strength ablation changes only the teacher that generates Stage 1 strategy targets for Qwen3-4B. Using the weaker teacher lowers the average by 0.58%, suggesting the final gains are not attributable solely to the largest SFT teacher.

SFT Teacher Strength Ablation Details

  • The weaker SFT teacher lowers the average by only 0.58%, supporting that SURE’s final gains are not attributable solely to distillation from the largest teacher.The comparison changes only the Stage 1 strategy-target teacher; the data size, optimization schedule, SRM data and parameters, and LLM judge remain unchanged.Source: SFT Teacher Strength Ablation Details · p. 11

SURE Policy Optimization Loop

SURE’s policy loop scores extracted strategies with a frozen SRM and combines that signal with correctness and format rewards in GRPO. Ranking studies find monotonic quantile ordering across training checkpoints, while noting that individual SRM scores are not calibrated.

SURE Policy Optimization Loop

  • For each problem, the loop samples G rollouts, parses strategy and execution, verifies correctness, and combines group-normalized, correctness-gated SRM scores with format and correctness rewards for GRPO.The resulting composite scalar drives the standard GRPO update.Source: SURE Policy Optimization Loop · p. 11

SRM Utility Ranking Details

  • Ranking evaluation uses 600 held-out problems, 20 strategies per problem, and one answer per strategy; its five quantiles assess relative group-level utility, not individual-score calibration.Each quantile contains 2,400 strategies.Source: SRM Utility Ranking Details · p. 12

SRM Ranking Across SURE Checkpoints

  • At each checkpoint, strategies are regenerated by that policy and evaluated on the same 600 problems, so the ranking reflects both executor evolution and changing strategy distributions.Each strategy receives one downstream execution.Source: SRM Ranking Across SURE Checkpoints · p. 13
  • The checkpoint ranking is strictly monotonic, with top-to-bottom accuracy gaps of 3.24%, 3.57%, and 3.74% at epochs 1, 2, and 3.The analysis does not isolate executor drift from strategy-distribution drift.Source: SRM Ranking Across SURE Checkpoints · p. 13, Table 10 · p. 13

Cross-Executor Strategy Utility Validation

SRM scores rank strategy utility above chance across evaluated executors, and choosing the highest-scored rather than lowest-scored strategy raises pass@1 for all three. The paper’s example separates a high-level strategy from its detailed execution.

Cross-Executor Strategy Utility Validation

  • Choosing the highest- rather than lowest-scored strategy raises pass@1 from 55.5% to 59.0% for Qwen3-4B, 56.4% to 59.4% for Qwen3-8B, and 17.7% to 20.8% for Llama-3.2-3B.Pairwise ranking accuracy is above chance for every executor; the results show transfer to a larger Qwen executor and the Llama family.Source: Cross-Executor Strategy Utility… · p. 13

Qualitative Example

  • The strategy block specifies theorem choice, transformations, and subgoal order without numerical computation; the execution block carries out the derivation.Figure 4 presents a complete question, strategy, and execution example.Source: Qualitative Example · p. 13, Figure 4 · p. 14

Dataset Licenses and Ethical Considerations

The paper documents dataset and model licensing alongside safeguards against evaluation-data leakage. Its appendix collects prompts for strategy generation, preference verification, and teacher-constructed contrasts.

Dataset Licenses and Ethical Considerations

  • SFT draws from MATH-train, OpenThoughts, and DeepMath after deduplication against evaluation sets; MATH500 is excluded from SFT, SRM training, and RL.The MATH subset uses only the training split.Source: Dataset Licenses and Ethical… · p. 14
  • Released checkpoints use Apache 2.0; the teacher model is accessed through a public API under the provider’s terms of service.The listed Qwen models use Apache 2.0, while Llama-3.2-3B-Instruct uses the Meta Llama 3 Community License.Source: Dataset Licenses and Ethical… · p. 14

SFT Strategy Generation Prompt

  • The prompt set covers strategy extraction, preference verification, and teacher-generated positive-negative strategy pairs that expose structural errors underrepresented in rollout data.The verification judge checks whether a labeled preference is justified.Source: Figure 5 · p. 15, Figure 6 · p. 16, Figure 7 · p. 17
Loading 2609.32482v1…