PaperlayerView exact revision on arXiv

Source-linked AI summary

PRISM: Pushing the Frontier of Deep Think via Process Reward Model-Guided Inference

Rituraj Sharma, Weiyuan Chen, Noah Provenzano, Tu Vu

arXiv:2603.02479v1cs.AI

TL;DR

DEEPTHINK inference lacks reliable correctness signals, making deeper refinement vulnerable to error amplification and weak compute returns. PRISM decomposes the process and uses PRM-guided step-level verification for refinement and aggregation, achieving strong results across mathematics and science benchmarks while showing directional correction and robustness in low-correctness settings.

  • Problem

    Existing DEEPTHINK frameworks lack reliable correctness signals during inference, so refinement can amplify errors, suppress correct minority traces, and yield weak returns to additional compute.

  • Method

    PRISM is a Process Reward Model-guided inference algorithm that uses step-level signals to resample, stochastically refine, and aggregate candidate reasoning trajectories.

  • Results

    Across mathematical and scientific benchmarks, PRISM is competitive with or outperforms state-of-the-art DEEPTHINK methods, with gpt-oss-20b reaching 90.0% on AIME25, 75.4% on HMMT25, and 71.4% on GPQA Diamond.

  • Takeaways & Limitations

    PRISM produces consistent directional correction, remains robust when few initial candidates are correct, and often lies on the compute-accuracy Pareto frontier.

  • Takeaways & Limitations

    The study evaluates one deterministic prompted PRM derived from the same base LLM, so it does not fully characterize performance under richer reward sources.

Abstract

from arXiv · show

DEEPTHINK methods improve reasoning by generating, refining, and aggregating populations of candidate solutions, which enables strong performance on complex mathematical and scientific tasks. However, existing frameworks often lack reliable correctness signals during inference, which creates a population-enhancement bottleneck where deeper deliberation amplifies errors, suppresses correct minority solutions, and yields weak returns to additional compute. In this paper, we introduce a functional decomposition of DEEPTHINK systems and propose PRISM, a Process Reward Model (PRM)-guided inference algorithm that uses step-level verification to guide both population refinement and solution aggregation. During refinement, PRISM treats candidate solutions as particles in a PRM-defined energy landscape and reshapes the population through score-guided resampling and stochastic refinement, which concentrates probability mass on higher-quality reasoning while preserving diversity. Across mathematics and science benchmarks, PRISM is competitive with or outperforms existing DEEPTHINK methods, reaching 90.0%, 75.4%, and 71.4% with gpt-oss-20b on AIME25, HMMT25, and GPQA Diamond, respectively, while matching or exceeding gpt-oss-120b. Additionally, our analysis shows that PRISM produces consistent net-directional correction during refinement, remains reliable when the initial population contains few correct candidates, and often lies on the compute-accuracy Pareto frontier.

1 Introduction

DEEPTHINK systems lack a transparent account of how their components affect reasoning quality and compute efficiency, while refinement can amplify errors without reliable correctness signals. PRISM addresses this bottleneck with PRM-guided refinement and aggregation, achieving strong benchmark performance and robust scaling behavior.

  • DEEPTHINK frameworks are difficult to analyze mechanistically because their monolithic presentation obscures design choices, failure modes, component interactions, and compute-performance tradeoffs.
  • The proposed taxonomy separates DEEPTHINK into population creation, population enhancement, and solution aggregation for systematic comparison.Population creation generates candidates, enhancement refines them, and aggregation selects the final answer.
  • Population enhancement is identified as the primary bottleneck because parallel sampling with majority voting can compete with sophisticated iterative systems.This suggests that initial diversity and aggregation contribute substantially to observed performance.
  • Existing refinement can preserve, propagate, or amplify errors and suppress infrequent correct reasoning through majority dilution.Increasing refinement depth therefore does not reliably improve population quality.
  • 90.0%, 75.4%, and 71.4% are PRISM’s gpt-oss-20b results on AIME25, HMMT25, and GPQA Diamond, respectively.PRISM is reported as competitive with or better than state-of-the-art DEEPTHINK methods and matching or exceeding gpt-oss-120b.
  • PRISM uses step-level PRM correctness signals to guide population refinement and final solution aggregation.Its refinement treats candidates as particles in a PRM-defined energy landscape and uses score-guided resampling and stochastic refinement.

2 Dissecting DEEPTHINK frameworks

DEEPTHINK frameworks can be dissected into candidate creation, iterative enhancement, and final aggregation, revealing that refinement often fails to convert extra computation into reliable correctness gains. The paper motivates PRM-guided alternatives by identifying weak quality signals, error propagation, and compute inefficiency.

  • 2.1 A functional taxonomy of DEEPTHINK frameworks: The functional decomposition contains population creation, population enhancement, and solution aggregation, enabling mechanistic analysis of how compute translates into correctness.These stages generate candidates, iteratively refine them, and synthesize the final answer.
  • 2.1 A functional taxonomy of DEEPTHINK frameworks: Population creation increases diversity and the chance that at least one correct trajectory exists, but does not itself concentrate probability on correct solutions.Later stages must identify and amplify high-quality candidates.
  • 2.1 A functional taxonomy of DEEPTHINK frameworks: Population enhancement iteratively refines candidates to increase the density of correct reasoning trajectories while preserving useful diversity.Existing approaches include critic-guided refinement and multi-agent interaction.
  • 2.1 A functional taxonomy of DEEPTHINK frameworks: Solution aggregation converts the final population into one prediction, commonly through majority voting or model-based aggregation.Aggregation assumes that enhancement has produced a sufficiently high-quality candidate set.
  • 2.2 Empirical motivation: The population-refinement bottleneck: Parallel sampling with majority voting is often competitive at low compute, while refinement-heavy frameworks can provide only marginal gains despite higher token usage.This challenges the assumption that additional refinement iterations monotonically improve population quality.
  • 2.2 Empirical motivation: The population-refinement bottleneck: Without correctness-sensitive signals, refinement updates can preserve, propagate, or amplify errors and limit returns from additional inference-time compute.The paper therefore motivates mechanisms that guide both refinement and aggregation with explicit correctness signals.

3 PRISM: PRM-guided Refinement, Iteration, and Selection Mechanisms

PRISM injects step-level PRM correctness signals into refinement and aggregation. It combines score-guided population management with stochastic, probabilistic refinement to favor higher-quality reasoning while preserving exploration.

  • PRISM uses PRM-based step-level correctness signals to guide both population refinement and final solution selection.
  • Refinement: Each refinement iteration scores candidates, resamples when probability mass collapses, and applies stochastic refinement moves for correction and exploration.
  • Refinement: Lower temperatures concentrate probability mass on high-scoring candidates, while higher temperatures preserve diversity in the implicit PRM-defined energy landscape.
  • Refinement: PRISM accepts proposals with probability A(τ → τ′) = min(1, r_w), favoring higher PRM scores while allowing some decreases to escape local modes.
  • Safeguards: Conflict arbitration and clone capping stabilize resampling by handling similarly scored answers and limiting duplicated traces.
  • Aggregation: PRM-score voting selects the answer with the highest aggregated reasoning score rather than the most frequent answer alone.

4 Experimental setup

The experiments compare PRISM with diverse DEEPTHINK strategies under controlled inference settings across mathematical and scientific benchmarks. Evaluation includes final accuracy, population behavior, directional correction, and accuracy–compute tradeoffs.

  • Controlled comparison: All methods share backbone models, population width, refinement depth, and initialization, while token usage varies and is reported explicitly.
  • Compared methods: The comparison includes no-refinement voting, critic-guided rewriting, multi-agent debate, majority-driven refinement, and recursive self-aggregation.
  • Benchmarks: The evaluation uses AIME25, HMMT25, and GPQA Diamond, with GPQA Diamond capped at its first 120 examples while the mathematics benchmarks are evaluated in full.
  • Models: Nine generator models are evaluated across gpt-oss and Qwen families, with gpt-oss-20b as the primary PRISM backbone and gpt-oss-120b mainly as a zero-shot baseline.
  • Metrics: Population accuracy measures the fraction of correct candidates at each refinement depth and whether refinement increases correct-candidate density.
  • Metrics: NetFlip compares incorrect-to-correct and correct-to-incorrect transitions, with positive values indicating net error correction and negative values indicating degradation.
  • Metrics: Compute is measured as total token usage across generation, verification, and comparison, enabling accuracy–compute tradeoff analysis and estimated cost reporting.

5 Results and discussion

PRISM improves accuracy and compute efficiency across benchmarks while producing more stable, directionally corrective refinement and greater robustness to weak initial populations. Its PRM-guided dynamics also stabilize population diversity and aggregation.

  • Accuracy and compute efficiency: PRISM consistently improves final accuracy over lightweight and refinement-heavy baselines while remaining on or near the compute–accuracy Pareto frontier.It frequently converts additional inference-time compute into gains more efficiently than refinement-heavy methods.
  • Accuracy and compute efficiency: 90.0% on AIME25, 75.4% on HMMT, and 71.4% on GPQA Diamond are achieved by PRISM with gpt-oss-20b.These results are competitive with or exceed strong DEEPTHINK baselines, while matching or outperforming gpt-oss-120b.
  • Refinement dynamics: Refinement-heavy baselines often oscillate or degrade with depth, whereas PRISM shows stable upward population dynamics.PRM-guided acceptance filtering is described as suppressing harmful updates and preserving correct trajectories.
  • Refinement dynamics: PRISM produces substantially larger positive NetFlip values, indicating that refinement corrects incorrect trajectories more often than it degrades correct ones.Positive NetFlip distinguishes directional correction from random-walk reshuffling.
  • Robustness and aggregation: PRISM maintains markedly higher accuracy when initial populations contain few correct candidates, preserving recovery capacity in low-correctness regimes.Step-level correctness signals make promising minority trajectories less likely to be overwritten by more frequent incorrect reasoning.
  • Robustness and aggregation: PRISM maintains stable performance under LLM aggregation, while many baselines lose accuracy when switching from majority vote to LLM aggregation.PRM-score voting further mitigates majority dilution when correct answers are rare.
  • Population stabilization: By t = 4, ESS/N rises to 0.81-0.88 and resampling falls to 3.3%–8.5% after early concentration triggers 50.3%–79.5% resampling.ESS/N measures how evenly distributed candidate weights are across the population.
  • Generalization: PRISM generalizes across models and benefits most when the verifier is stronger than the generator.Under Qwen3, it remains competitive on the accuracy–compute Pareto frontier, suggesting more effective compute allocation rather than token usage alone.

6 Related work

The paper situates PRISM within DEEPTHINK frameworks that refine and combine interdependent reasoning paths, while distinguishing its use of process reward models as operational guidance during inference.

  • Test-time scaling and DEEPTHINK frameworks: DEEPTHINK extends parallel reasoning by iteratively refining and aggregating interdependent candidate paths rather than treating samples as independent.Examples include SciMaster and Recursive Self-Aggregation.
  • Process reward modeling: Process reward models provide step-level supervision over intermediate reasoning steps instead of evaluating only final answers.Prior work primarily uses PRM outputs for ranking, filtering, or reinforcement learning.
  • Process reward modeling: PRISM differs by interpreting PRM scores as an implicit energy landscape over reasoning trajectories.This interpretation supports using process rewards directly in refinement and selection rather than only for evaluation or training.

7 Conclusion

The paper concludes that PRISM addresses population enhancement bottlenecks by embedding step-level correctness signals into refinement and aggregation. Across benchmarks, it improves accuracy, directional correction, robustness, and compute efficiency.

  • Conclusion: PRISM combines a functional taxonomy of DEEPTHINK systems with PRM-guided refinement and aggregation based on step-level correctness signals.PRM scores define an implicit energy landscape over reasoning trajectories.
  • Conclusion: Across mathematical and scientific benchmarks, PRISM consistently improves accuracy, produces stable directional correction, remains robust in low-correctness regimes, and often lies on the compute–accuracy Pareto frontier.The conclusion identifies reliable step-level verification as a key ingredient for scalable inference-time reasoning.

8 Limitations

The study’s scope is constrained by its PRM design and by the assumption that reasoning can be segmented into meaningful steps.

  • PRM instantiation: PRISM experiments use a deterministic prompted PRM derived from the same base LLM, representing only one possible step-level reward realization.The authors note that externally grounded feedback, such as executable tests or formal verification, could provide substantially more reliable scoring.
  • PRM instantiation: Results demonstrate integrating step-level signals into inference but do not fully characterize performance under richer reward sources.
  • Step segmentation: PRISM assumes reasoning can be decomposed into meaningful steps for informative step-level scoring.Misaligned segmentation can weaken refinement guidance, motivating improved segmentation or more structured reasoning representations.

A Step verifier and score construction

The verifier audits each tagged reasoning step and the proposed answer, assigning structured step-level judgments plus a final-answer check.

  • Verifier inputs: The verifier receives the problem, an ordered tagged step sequence, and the proposed final answer.Each input step is wrapped in a 1-indexed <step i="k"> tag before verification.
  • Output format: Verifier output contains one line per step with a score in {+1, 0, −1}, followed by a FINAL ANSWER CHECK token.Free-form notes are ignored; only score tokens are parsed.
  • Step judgments: The verifier judges each step independently, marking dependent later steps incorrect when an earlier mistake propagates.Trivial steps receive 0, while missing, ambiguous, or unverifiable content receives -1.
  • Answer decision: The final answer is accepted only when all steps and the FINAL ANSWER CHECK receive +1.Otherwise, the verifier returns an answer score of -1.

A.2 Parsing and failure handling

Verifier calls are validated as structured text, and malformed or incomplete outputs trigger a verification failure with a fixed fallback score.

  • Parsing validation: A successful verifier call must emit one parsable score token for every non-final step and a parsable FINAL ANSWER CHECK token.All score tokens must belong to {+1, 0, −1}.
  • Failure handling: Missing lines, malformed tags, out-of-range tokens, or parsing-breaking extra text are treated as VERIFICATION_FAILED.The affected trace receives the fixed fallback score, which is clamped above a small positive epsilon for ratio and weight computations.

A.3 Scoring, particle weights, and acceptance

PRISM converts verifier feedback into scalar scores, uses those scores to guide refinement proposals and population weighting, and arbitrates incompatible high-scoring answers.

  • Score construction: The scalar verifier score combines step-level cleanliness with the final verdict, allowing within-verdict ranking without overriding final correctness.Correct steps receive full credit, neutral steps partial credit, and incorrect steps no credit in the step ratio.
  • Score construction: Incorrect final answers are capped below 0.3, while correct answers are floored at 0.5 and scaled upward with step quality.Neutral or missing verdicts receive intermediate scaling to reflect uncertainty.
  • Mixture proposal distribution: The refinement proposal mixes local PRM-guided edits with exploration from the original problem, with η controlling the exploration fraction.Local refinement fixes incorrect steps and preserves correct ones; exploration attempts a qualitatively different approach to reduce mode collapse.
  • Decoding and formatting: Unparsable refinement outputs are treated as no-ops, leaving the current trace unchanged.An explicit final-answer line is required for deterministic answer extraction.
  • Arbitration: Arbitration is triggered when multiple extracted answers are near-perfect or top answers are near-tied, using a comparator that returns A, B, or NEITHER.The comparator checks both reasoning and final answers and uses NEITHER when both are wrong or indeterminate.
  • Arbitration: Score clamping lowers the resampling weights of unresolved conflicting traces, limiting their influence on later refinement.Weights depend on scores, so clamping a losing or unresolved trace reduces its subsequent resampling probability.

D Additional figures

The additional figures show PRISM's stable refinement dynamics, resilience to weak initial populations, broad gains across model settings, and favorable compute–accuracy tradeoffs. They also examine verifier scaling and model-type effects.

  • Population refinement: PRISM exhibits stable upward population dynamics as refinement depth increases, unlike non-PRM methods that often oscillate or degrade.
  • Directional correction: Positive NetFlip values indicate more incorrect→correct than correct→incorrect transitions across enhancement depth on AIME25 and HMMT25.
  • Initial population quality: PRISM maintains substantially higher final accuracy when initial candidate pools contain few correct solutions, demonstrating resilience to weak initial populations.
  • Model scaling: PRISM consistently improves over zero-shot Qwen3 performance across model sizes on AIME25, HMMT25, and GPQA Diamond, with the largest gains for smaller models.
  • Compute–accuracy tradeoff: PRISM remains competitive on compute–accuracy Pareto frontiers across AIME25, HMMT25, and GPQA Diamond, indicating gains are not driven solely by increased token usage.
  • Verifier and model-type effects: Performance improves as verifier strength increases, with the largest gains when the verifier is larger than the generator; PRISM also helps weaker model variants more.
Loading 2603.02479v1…