Source-linked AI summary
STRIDE: Strategic Trajectory Reasoning via Discriminative Estimation for Verifiable Reinforcement Learning
Qinjian Zhao, Zhihao Dou, Dinggen Zhang, Xiangyu Li, Chaoda Song, Zhongwei Wan, Xinpeng Li, Yanyan Zhang, Kaijie Chen, Qingtao Pan, Chengcheng Feng, Zhiqiang Gao, Xiaoyu Xia
TL;DR
Existing RLVR methods rely heavily on final-answer correctness, yielding sparse supervision and leaving intermediate signals unable to reliably distinguish beneficial from harmful reasoning. STRIDE contrasts successful and failed trajectories to estimate discriminative n-gram patterns, combines them with reasoning-saliency entropy, and assigns differentiated advantages. Across mathematical, multimodal, and agent tasks, STRIDE consistently outperforms strong baselines, while remaining bounded by its focus on automatically verifiable outcomes and sampled response groups.
Problem
Existing RLVR methods use final-answer correctness for trajectory rewards, while intermediate signals may not reliably distinguish beneficial from harmful reasoning patterns.
Method
STRIDE contrasts successful and failed trajectories to estimate outcome-discriminative n-gram patterns, combines them with reasoning-saliency entropy, and assigns differentiated advantages during RL optimization.
Results
STRIDE consistently outperforms strong baselines across mathematical reasoning, multimodal reasoning, and agent tasks.
Takeaways & Limitations
STRIDE provides fine-grained, verifiable credit assignment by distinguishing beneficial and harmful strategic patterns rather than treating all trajectory tokens uniformly.
Takeaways & Limitations
The study focuses mainly on automatically verifiable outcomes, estimates patterns from sampled response groups, and uses n-gram patterns as its credit-assignment unit.
Abstract
from arXiv · showhide
Reinforcement Learning with Verifiable Rewards (RLVR) has become an effective post-training paradigm for improving the reasoning abilities of large language models. However, existing RLVR methods typically rely on final-answer correctness to assign trajectory-level rewards, providing sparse supervision and treating all tokens uniformly regardless of their actual contribution to reasoning. Although recent studies introduce intermediate signals such as process rewards, high-entropy tokens, and semantic uncertainty, these signals are often not inherently verifiable and may fail to distinguish beneficial strategic patterns from harmful ones. To address this limitation, we propose STRIDE (Strategic Trajectory Reasoning with Discriminative Estimation), a fine-grained RLVR framework that derives strategic reasoning supervision from verifiable outcomes. STRIDE contrasts successful and failed trajectories within each response group to estimate the outcome-discriminative preference of each $n$-gram strategic pattern, and further combines this signal with reasoning saliency entropy to identify decision-relevant strategic patterns. These patterns are assigned differentiated advantage values during RL optimization, enabling more precise credit assignment while preserving the verifiability of RLVR. Extensive experiments demonstrate that STRIDE consistently improves reasoning performance across diverse models, tasks, and extended settings, including VLMs and agent-based systems.
1 Introduction
Existing RLVR methods provide sparse, outcome-level supervision and intermediate signals that are not reliably verifiable. STRIDE addresses this by estimating outcome-discriminative strategic patterns, combining them with saliency entropy, and assigning differentiated advantages.
- RLVR commonly rewards an entire trajectory using only final-answer correctness, leaving intermediate reasoning weakly supervised and tokens optimized uniformly.
- Intermediate signals such as process rewards, high-entropy tokens, and semantic uncertainty identify salient regions but may not determine whether those regions help or harm correctness.
- Correct and incorrect responses can exhibit nearly indistinguishable entropy peaks, frequencies, and density distributions, limiting entropy’s ability to identify beneficial reasoning regions.
- STRIDE contrasts successful and failed trajectories within each response group to estimate whether n-gram strategic patterns are associated with beneficial or harmful outcomes.
- STRIDE combines outcome-discriminative patterns with reasoning-saliency entropy, then assigns differentiated advantages to improve credit assignment while preserving RLVR verifiability.
2 Preliminaries and related work
RLVR methods such as GRPO improve reasoning efficiently but still rely on sparse final-answer rewards, while related approaches seek denser or more informative intermediate supervision. High-entropy token distributions can overlap across correct and wrong trajectories, motivating more discriminative signals.
- Outcome-level RLVR can encourage locally optimized responses without a coherent global plan because intermediate reasoning receives limited supervision.
- Existing approaches pursue more informative rollout rewards or intermediate-process supervision to provide denser guidance.
- GRPO removes the value model and estimates advantages from sampled response groups, making large-scale reasoning optimization more efficient than PPO.
- Standard GRPO usually derives response-level rewards from final-answer correctness, producing sparse supervision and potentially rewarding redundant or spurious reasoning.
- High-entropy token ranges can be shared by correct and wrong trajectories with similar frequencies, so entropy alone does not reliably distinguish useful reasoning.
3 Approach of STRIDE
STRIDE identifies reasoning-critical strategic patterns by contrasting successful and failed trajectories, then combines outcome discriminability with saliency entropy to reweight RLVR advantages. This differentiates reinforcement and penalties for patterns associated with correct and failed reasoning.
- Framework: STRIDE models heterogeneous token roles and distinguishes strategic patterns associated with successful versus failed outcomes instead of uniformly optimizing all tokens.Its DSR-RL framework provides the basis for identifying reasoning-critical regions within generated trajectories.
- Discriminative reasoning estimation: Within each response group, sampled responses are partitioned into successful and failed subsets using final-answer correctness.Successful responses satisfy the ground-truth answer, while failed responses do not.
- Overview and evaluation: Figure 3 summarizes STRIDE as contrasting successful and failed trajectories to assign pattern-level advantages that reinforce beneficial patterns and penalize harmful ones.The method is evaluated through performance comparisons and ablations, including the benchmarks summarized in Tables 1 and 2.
- Discriminative reasoning estimation: Sfreq(g) compares an n-gram pattern’s occurrence frequency in successful and failed trajectories to estimate outcome discriminability.Positive values indicate stronger association with successful reasoning, whereas smaller or negative values indicate greater association with failed reasoning; smoothing and clipping control numerical instability and rare-pattern effects.
- Strategic saliency: Because frequency association alone may reflect peripheral execution patterns, STRIDE uses saliency entropy to identify patterns more likely to contain decision-critical reasoning.Hsal(g) averages token-generation entropy over the pattern, with higher values indicating greater potential reasoning significance.
- Advantage reweighting: SA(g) activates frequency-based pattern scoring only when saliency entropy exceeds a threshold, producing stronger rewards for beneficial patterns and stronger penalties for harmful ones.The resulting significance score is added to the base outcome-level advantage during RL optimization, reweighting conventional uniform RLVR rewards.
4 Experiment
Experiments evaluate STRIDE across model families, mathematical, multimodal, and agent benchmarks, finding consistent gains over strong baselines. Ablations and sensitivity analyses support the contributions of its discriminative estimation and entropy filtering.
- Experiment setting: STRIDE is evaluated across Llama-3.1-8B-Instruct, Qwen2.5-7B-Instruct, Qwen3-4B-Base, and Qwen2.5-VL-7B, with agent tests on WebShop, ALFWorld, and BabyAI.The comparisons use shared SFT/RL data and identical sampled-response counts across RLVR methods where stated.
- Mathematical reasoning: 2.53 points: STRIDE raises Qwen2.5-7B-Instruct’s average mathematical score from 42.18 with HET-DAPO to 44.71.It also improves averages by 1.90 points on Llama-3.1-8B-Instruct and from 47.27 to 49.24 on Qwen3-4B-Base.
- Ablation and sensitivity: Removing frequency-based estimation lowers the average score from 44.71 to 42.46, while removing the entropy filter lowers it to 42.31.These ablations support the reported roles of discriminative frequency estimation and reasoning-saliency information.
- Multimodal reasoning: STRIDE consistently outperforms baselines on general-domain and scientific multimodal benchmarks, including MMMU-Pro, MMMU, EMMA, and Bio.Against Qwen2.5-VL-7B, scores rise from 30.5 to 44.2 on MMMU-Pro, 20.2 to 31.5 on EMMA, and 54.0 to 72.3 on Bio.
- Agent tasks: 84.3 average: STRIDE outperforms AgentSTaR’s 82.6 and HET-DAPO by 1.5 points across WebShop, ALFWorld, and BabyAI.The gains are reported consistently across all three interactive decision-making tasks.
- Ablation and sensitivity: STRIDE outperforms GRPO across six mathematical benchmarks under different rollout budgets, while gains gradually saturate from 8 to 10 rollouts.The method retains an advantage at every tested rollout setting, including challenging AIME and Olympiad benchmarks.
5 Conclusion
The paper presents STRIDE as a fine-grained RLVR framework that identifies outcome-discriminative strategic patterns from successful and failed trajectories. Across mathematical, multimodal, and agent tasks, it consistently outperforms strong baselines.
- Conclusion: STRIDE combines verifiable trajectory comparisons with frequency-based estimation and reasoning-saliency entropy to assign differentiated advantages to strategic patterns.This design targets more precise credit assignment than standard outcome-level RLVR.
- Conclusion: STRIDE consistently outperforms strong baselines across mathematical reasoning, multimodal reasoning, and agent tasks.The conclusion characterizes the method as effective, robust, and broadly generalizable within these evaluated settings.
Limitations
The study’s scope is concentrated on automatically verifiable outcomes, and it identifies broader open-ended generation, estimation efficiency, and semantic pattern units as future directions.
- Scope: The current study mainly focuses on tasks with automatically verifiable outcomes, leaving open-ended generation as a future extension.Such scenarios make automatic verification less straightforward.
- Efficiency: STRIDE estimates strategic patterns from sampled response groups, so improving pattern-estimation efficiency could broaden applicability.The experiments show effectiveness under moderate rollout budgets, but efficiency remains a stated direction.
- Representation: The method uses n-gram patterns as a simple credit-assignment unit, while more flexible semantic units remain for future work.The authors connect this direction to potential improvements in interpretability and generalization.
Ethics statement
STRIDE is evaluated in reasoning-focused settings, but its models may retain risks from pretrained systems and training data. The paper therefore emphasizes safeguards before deployment in sensitive domains.
- STRIDE models may inherit biases, factual inaccuracies, or unsafe behaviors from pretrained backbones and training data.
- Stronger reasoning capabilities may be misused in inappropriate or harmful applications without proper safeguards.
- The experiments prioritize mathematical, scientific, multimodal, and structured decision-making benchmarks rather than sensitive personal or harmful content.
- The paper recommends safety evaluation, bias analysis, and domain-specific human review before real-world deployment involving sensitive decisions.
A.1 Experimental parameter setup
The experiments use GRPO as the base RLVR algorithm and generate grouped rollouts for each question. STRIDE estimates strategic patterns by contrasting successful and failed trajectories within each group.
- All RL experiments use GRPO as the base RLVR algorithm.
- For each input question, the policy generates a group of 6 responses during rollout.
- STRIDE contrasts successful and failed trajectories within each group to estimate outcome-discriminative n-gram strategic patterns.
A.2.1 Results of test-time scaling
STRIDE improves over GRPO across test-time sampling budgets, model families, and mathematical benchmarks. Its training dynamics remain stable while reinforcing strategic reasoning patterns through differentiated signals.
- A.2.1 Results of test-time scaling: Across six mathematical reasoning benchmarks, STRIDE consistently outperforms GRPO under different Pass@K settings as sampled rollouts increase.The gains are especially clear on AIME 2024, AIME 2025, and Olympiad across all evaluated K values.
- A.2.2 Training Dynamics of STRIDE: STRIDE achieves consistently higher accuracy rewards than GRPO during training while producing longer reasoning responses.
- A.2.2 Training Dynamics of STRIDE: STRIDE maintains entropy loss within a range comparable to GRPO, indicating that its performance improvement occurs without unstable exploration behavior.
- A.2.3 Statistical Significance Analysis: Across three base models and six mathematical reasoning benchmarks, STRIDE shows positive improvements over GRPO, with many comparisons reaching p < 0.001.
- A.4 Examples of Strengthened and Weakened Patterns: High-frequency reasoning tokens shared by correct and wrong trajectories are not necessarily outcome-discriminative.