AgentV-RL: Scaling Reward Modeling with Agentic Verifier

1 Introduction

• Existing outcome- and process-level reward models provide scalar values, while generative reward models typically assess candidate solutions with single-turn reasoning.The paper identifies error propagation on plausible incorrect answers and insufficient external grounding for computation- or knowledge-heavy tasks as limitations of existing verifiers.Source: §1 · pp. 1–2
• Agentic Verifier coordinates forward and backward agents to check premise-to-conclusion sufficiency and conclusion-to-premise necessity using multi-turn reasoning and tools.The agents can decompose solutions into verifiable sub-steps and invoke tools such as code interpreters for numerical calculation.Source: §1 · p. 2
• AgentV-RL distills multi-agent capabilities into one model through synthetic verification data and rejection-sampling fine-tuning followed by reinforcement learning.Its synthetic data engine generates and quality-controls trajectories spanning logical and computational challenges.Source: §1 · p. 2
• The 4B variant outperforms the state-of-the-art ORM and attains up to 25.2% improvements.Across parallel TTS, Agentic Verifier also outperforms proprietary reasoning models and off-the-shelf BT-RMs on mathematics; for sequential TTS, it serves as a critique model that helps correct actor flaws.Source: §1 · p. 2

2 Related Work

• Outcome-level reward models assign scalar rewards to complete responses, while process-level models provide dense supervision on intermediate steps.Generative reward models produce natural-language feedback, and rubric-based variants construct task-specific evaluation criteria.Source: §2 · p. 2
• Existing approaches may not tightly integrate tool execution into reasoning or provide the point-wise feedback required for test-time scaling.The paper contrasts these gaps with verification as an agentic, multi-turn process for test-time exploration and assessment.Source: §2 · p. 3
• Unlike prior test-time scaling work, this approach makes verification bidirectional, multi-turn, and tool-augmented for both parallel and sequential scaling.Prior work includes critique-based actor correction and studies of additional inference-time computation for reward models and process verifiers.Source: §2 · p. 3

3.2 Task Definition

• The task studies actor-verifier interaction on mathematics, with the actor solving problems and the verifier supervising generated solution chains.The formal setup represents each example as a question and ground truth, and examines parallel and sequential test-time scaling.Source: §3 · p. 3, §3.2 · p. 4

3 Method

• BoN samples k candidate solutions and selects the highest-confidence one using the verifier’s True-token likelihood; sequential scaling iterates critique and actor refinement.The verifier assesses candidate solutions and generates a rationale, while the actor revises its solution based on feedback.Source: §3 · p. 3

3.1 Agentic Verifier

• Forward and backward agents check solution steps from premises to conclusions and conclusions against problem constraints, using multi-turn reasoning and tools such as Python.The forward agent plans atomic steps, validates their logical sufficiency, and issues a binary verdict; the backward agent checks necessity and omitted constraints.Source: §3.1 · p. 4, §3.2 · p. 4

3.3 AgentV-RL

• AgentV-RL distills the multi-agent verifier into a single model through a synthetic data engine and a two-stage training schema.The recipe is designed to provide scalable training for the bidirectional verification capability.Source: §3.3 · p. 4

3.3.1 Synthetic Trajectories Sampling

• Synthetic trajectories are generated by having an LLM act as either verifier agent, then retaining only trajectories whose verdict matches the solution’s ground-truth correctness label.The initial candidates come from filtered public datasets; questions with eight sampled solutions that are all correct or all incorrect are filtered out. The resulting dataset includes positive and negative verification examples.Source: §3.3.1 · pp. 4–5

3.3.2 Rejection Fine-tuning

• Supervised fine-tuning on the synthetic dataset trains the verifier to reproduce stepwise reasoning and effective tool interaction in multi-turn decisions.The objective uses annotated verification trajectories containing thought, action, and observation steps.Source: §3.3.2 · p. 5

3.3.3 Reinforcement Learning

• After supervised fine-tuning, GRPO optimizes multi-turn, long-horizon, tool-integrated reasoning through sampled candidate trajectories and mixed forward/backward agent roles.The mixed sampling strategy is intended to balance optimization across the two perspectives.Source: §3.3.3 · p. 5

4.2 Main Result

• Agentic-Verifier-Qwen3-4B reaches up to 79.0% accuracy on MATH500, exceeding Skywork-V2-Llama-8B by 25.2 percentage points.Similar leads are reported on GSM8K, Gaokao2023, and AIME24.Source: Table 1 · p. 6, §4.2 · pp. 6–7

4.1 Setup

• Evaluation covers GSM8K, AIME24, MATH500, and Gaokao2023 under parallel BoN and iterative verifier revision, using shared candidate pools and initial solutions.BoN accuracy is measured at different rollout counts; sequential evaluation reports accuracy by iteration alongside correction and degradation rates.Source: §4.1 · p. 6

4.2 Main Result

• On first-turn revision, Agentic-Verifier-Qwen3-4B reaches 94.6% GSM8K and 84.2% MATH500 accuracy, with up to 41.6% ∆↑ and 0.6% ∆↓.Performance gains appear in the first one or two rounds, then remain stable or decline slightly in later iterations.Source: Table 2 · p. 7, §4.2 · pp. 7–8

4.3.1 Ablation Study

• Forward-only and backward-only variants are competitive, but the full design performs best; adding tools improves on the no-tool variant, which already beats the base model.Tool usage is described as moderate and stable in practice.Source: Figure 4 · p. 7, §4.3.1 · p. 8
• SFT+RL is strongest on BoN; Train-free beats the base model by up to 2.6 points on Gaokao2023, SFT adds gains, and RL further unlocks reasoning potential.The controlled study compares Train-free, SFT, and SFT+RL configurations.Source: Figure 5 · p. 8, §4.3.1 · p. 8

4.3.2 Scaling Effect

• Sampling more verification trajectories improves BoN accuracy, while scaling Qwen3 from 0.6B to 1.7B yields 5.2 points on Gaokao2023, with further gains at 4B.The model-size comparison is conducted under the train-free setting.Source: Figure 6 · p. 8, §4.3.2 · p. 8, Table 3 · p. 9

4.3.3 Generalization Performance

• Agentic-Verifier-Qwen3-4B achieves superior performance on LiveCodeBench and HotpotQA.These evaluations test competitive code and multihop question answering.Source: Table 4 · p. 9, §4.3.3 · p. 9

4.3.4 Latency Analysis

• Agentic Verifier achieves the highest accuracy at the expense of increased tokens and latency; forward and backward variants offer resource trade-offs.The overhead stems from the multi-turn deliberative process.Source: §4.3.4 · p. 9

• Agentic Verifier achieves substantial improvements in parallel and sequential TTS, outperforming state-of-the-art ORM and PRM.The conclusion describes the results as supporting a promising direction for agentic reward modeling.Source: §5 · p. 9
• Forward and backward agents support comprehensive, interpretable solution checking through multi-turn, tool-augmented verification.Source: §5 · p. 9
• Computational cost and latency are averaged over verification trajectories, with latency measured on one A100 GPU using vLLM at batch size 128.Source: Table 5 · p. 9
• The paper aims to inform reward-model development and inspire research on autonomous verification and reasoning.Source: §5 · p. 9

• Synthetic, tool-augmented data may not represent the variety of real-world reasoning problems, impeding generalization.Source: §6 · p. 9
• The multi-turn process increases computational cost, challenging real-time or resource-limited deployment.Source: §6 · p. 9
• Performance depends on external-tool coverage and reliability, which can bottleneck some tasks.Source: §6 · p. 9

• The methods and findings are intended solely for research purposes.Source: §7 · p. 9
• The study does not pertain to malicious applications or unintended uses.Source: §7 · p. 9
• The statement excludes fairness, privacy, security, crowdsourcing, and human-subject research issues.Source: §7 · p. 9

B.1 Text Reasoning Prompt

• Best-of-N evaluates each candidate independently in one turn and returns a Boolean judgment; Refine-style prompting revisits an initial judgment with feedback.Source: §B.1 · p. 13

B.2 Agentic Verifier Prompt

• The proposed framework has Plan, Validation, and Verdict stages, with forward or backward planning variants and shared Validation and Verdict prompts.The system automatically inserts intermediate prompts that advance execution between stages.Source: §B.2 · pp. 13–14

B.2.3 Stage B: Validation Prompt

• Validation checks planned steps across multiple turns and can call Python for a computation before continuing.Turns end with a Python block, a step-complete marker, or an end-of-analysis marker; Python execution returns results for the next turn.Source: §B.2.3 · p. 15

B.2.4 Stage C: Verdict Prompt

• The Verdict stage reviews prior verification steps and returns true only if all are correct, or false if any error or new inconsistency is found.Source: §B.2.4 · p. 15

C.1 Evaluation Protocol Details

• All verifier variants use the same 128 Qwen2.5-7B-Instruct rollouts per problem, generated with identical sampling settings, for comparability.The initial sequential-refinement solution is also fixed across variants.Source: §C.1 · p. 16

C.2 Baselines.

• The baselines cover text-only reasoning judges, outcome reward models that score whole solutions, and process reward models that provide step-level supervision.The listed examples include instruction-tuned judges, ORMs, and PRMs.Source: §C.2 · p. 16

C.3 Aggregation for Forward & Backward Agents

• For BoN, the verifier averages forward- and backward-trace confidence scores to rank candidates; for verifier revision, both traces must judge a solution correct or refinement is triggered.When either trace identifies an error, the verifier generates targeted modification suggestions for the actor.Source: §C.3 · p. 16

Experiment Setup

• Training data are synthesized from diverse mathematical benchmarks and grouped by difficulty using the number of correct candidates in Best-of-8 verification as a coarse signal.Qwen2.5-7B-Instruct generates question–answer pairs, and Qwen3-4B performs verification.Source: §C.4 · p. 16
• Supervised fine-tuning uses one epoch with a 5 × 10−6 learning rate and batch size 128, masking user inputs and tool outputs from loss computation.The maximum sequence length is 21,600 tokens.Source: §C.4 · p. 16
• GRPO training samples eight responses per query, caps tool invocations at three per rollout, and excludes user inputs and tool outputs from loss computation.Each rollout is capped at 21,600 total tokens; the actor uses a 1 × 10−6 learning rate.Source: §C.4 · p. 17
• Inference-time scaling samples k verification trajectories for the same query and candidate answer, then aggregates their scores.Tables 6 and 7 report BoN accuracy under different sampling budgets on Math500 and Gaokao2023, respectively.Source: §C.4 · p. 17, Table 6 · p. 17, Table 7 · p. 17

C.5 Additional Ablation on Tool Usage

• Adding Python tools improves performance on both MATH500 and Gaokao2023, while the no-tool verifier already substantially improves over the Base model.The comparison is reported in Tables 6 and 7.Source: §C.5 · p. 17
• Tool use is capped at three Python calls per sample; correct verdicts use tools slightly more often, while execution-outcome breakdowns are broadly similar across verdict groups.A successful call finishes normally and returns an executable result without runtime errors, timeout, or budget blocking.Source: §C.5 · p. 17, Table 8 · p. 17

C.6 Experimental Details for Generalization Beyond Math

• Generalization evaluation uses Python execution for LiveCodeBench and web retrieval for HotpotQA, judging sampled rollouts without access to ground-truth answers.The setting isolates verifier capability rather than the full test-time scaling pipeline.Source: §C.6 · pp. 17–18

C.7 Data Contamination Check

• The authors report no direct overlap between evaluation benchmarks and training data based on documented dataset provenance.The cited sources exclude AIME 2024, GSM8K, and Gaokao2023; MATH500 is explicitly removed from Light-R1.Source: §C.7 · p. 18

D.1 Generative Verifier

• In the running-geometry case, the verifier rejects the proposed 90° conclusion because the assumed distances do not match the runners’ motion and the resulting configuration is degenerate.Under d = t, the runners coincide at O, so ∠APB is undefined.Source: Generative Verifier · pp. 18–19

D.2 Forward Verifier

• The forward verifier checks a claimed root mapping by substitution and finds it does not follow from the polynomial’s root condition, so the proposed pairing is unjustified.It also identifies a Vieta claim that conflicts with the polynomial coefficients.Source: §D.2 · pp. 20–22

D.3 Backward Verifier

• Its backward plan checks expansion correctness, whether coefficient equations follow from that expansion, and whether the reported parameters satisfy those equations.The example also uses a symbolic tool check to confirm the product expansion.Source: §D.3 · pp. 22–23
• The verifier compares the candidate’s coefficient expressions with the true product; mismatches break the coefficient-matching system and the dependency leading to the final value.The reported b = 11/2 fails the verified equation 3b + 20 = 26.Source: §D.3 · p. 23
• The backward verifier rejects the claimed a + b = 5/2 because the solution’s expansion is incorrect and its reported b = 11/2 contradicts the verified equation 3b + 20 = 26.The correct expansion has coefficients 3b + 20 for t^3, 4a + 5b − 6 for t^2, and ab − 10 for t.Source: §D.3 · p. 23