Rewarding the Scientific Process: Process-Level Reward Modeling for Agentic Data Analysis

Zhisong Qiu; Shuofei Qiao; Kewei Xu; Yuqi Zhu; Lun Du; Ningyu Zhang; Huajun Chen

Source-linked AI summary

Rewarding the Scientific Process: Process-Level Reward Modeling for Agentic Data Analysis

Zhisong Qiu, Shuofei Qiao, Kewei Xu, Yuqi Zhu, Lun Du, Ningyu Zhang, Huajun Chen

arXiv:2604.24198v2Apr 27, 2026cs.CLcs.AIcs.CEcs.LGcs.MA

TL;DR

General-domain process reward models struggle to supervise dynamic data-analysis agents, especially when errors are silent or exploratory steps are necessary. DataPRM addresses this with environment-aware verification and ternary rewards, improving performance in both test-time scaling and reinforcement learning, including 78.73% on DABench.

Problem
General-domain process reward models cannot reliably detect silent errors or distinguish recoverable exploratory grounding errors in data-analysis agents.
Method
DataPRM actively interacts with the environment to verify steps and uses ternary rewards to distinguish incorrect, correct, and exploratory actions.
Results
DataPRM improves performance across test-time scaling and reinforcement learning, achieving 78.73% on DABench.
Takeaways & Limitations
The results indicate that environment interaction and diverse process supervision are important for reliable data-analysis agents.
Takeaways & Limitations
The study focuses on reasoning and visualization tasks, while DataPRM is trained solely through supervised fine-tuning requiring high-quality trajectory data.

Abstract

from arXiv · show

Process Reward Models (PRMs) have achieved remarkable success in augmenting the reasoning capabilities of Large Language Models (LLMs) within static domains such as mathematics. However, their potential in dynamic data analysis tasks remains underexplored. In this work, we first present a empirical study revealing that general-domain PRMs struggle to supervise data analysis agents. Specifically, they fail to detect silent errors, logical flaws that yield incorrect results without triggering interpreter exceptions, and erroneously penalize exploratory actions, mistaking necessary trial-and-error exploration for grounding failures. To bridge this gap, we introduce DataPRM, a novel environment-aware generative process reward model that (1) can serve as an active verifier, autonomously interacting with the environment to probe intermediate execution states and uncover silent errors, and (2) employs a reflection-aware ternary reward strategy that distinguishes between correctable grounding errors and irrecoverable mistakes. We design a scalable pipeline to construct over 8K high-quality training instances for DataPRM via diversity-driven trajectory generation and knowledge-augmented step-level annotation. Experimental results demonstrate that DataPRM improves downstream policy LLMs by 7.21% on ScienceAgentBench and 11.28% on DABStep using Best-of-N inference. Notably, with only 4B parameters, DataPRM outperforms strong baselines, and exhibits robust generalizability across diverse Test-Time Scaling strategies. Furthermore, integrating DataPRM into Reinforcement Learning yields substantial gains over outcome-reward baselines, achieving 78.73% on DABench and 64.84% on TableBench, validating the effectiveness of process reward supervision. Code is available at https://github.com/zjunlp/DataMind.

1 Introduction

DataPRM addresses failures of general process reward models in automated data analysis by combining environment interaction with ternary rewards. Across test-time scaling and reinforcement learning, it improves downstream performance while using a scalable process-supervision data pipeline.

Motivation: General PRMs struggle to verify silent logical errors and grounding errors in data-analysis tasks.They miss incorrect results that do not trigger interpreter exceptions and can mistake necessary trial-and-error exploration for irrecoverable failure.
Method: DataPRM dynamically interacts with real-world data contexts to validate procedural steps and avoid being deceived by code-execution success.Its ternary reward strategy distinguishes incorrect, correct, and neutral exploratory steps.
Results: 7.21% improvement on ScienceAgentBench and 11.28% on DABStep result from incorporating a 4B-parameter DataPRM in test-time scaling.The model also outperforms powerful self-rewarding baselines while achieving 58× parameter efficiency.
Results: 78.73% on DABench and 64.84% on TableBench are achieved by models trained with DataPRM process supervision in reinforcement learning.These results surpass methods relying solely on outcome supervision.
Data pipeline: Over 7K annotated instances are produced through diversity-driven trajectory generation and knowledge-augmented step-level annotation.The pipeline supplies fine-grained process-supervision data for evaluating DataPRM in test-time scaling and reinforcement learning.

2 Preliminary

The paper formulates agentic data analysis as a partially observable interaction process and defines trajectories as sequences of tasks, actions, observations, and reasoning. A process reward model scores individual steps, aggregates them into trajectory rewards, and supports both search-based inference and reinforcement learning.

Problem Formulation: Data analysis is modeled as a partially observable Markov decision process with environment states, code-generation actions, and interpreter-derived observations.The environment typically includes a code interpreter and files, while the agent observes the current state only through the interpreter.
Problem Formulation: An interaction history records the task, prior actions, and observations; under ReAct, explicit reasoning also guides action generation.At each time step, reasoning, action, and observation are treated as one unified data-analytic agent step.
Process Reward Modeling: A standard PRM assigns a step-level reward from the agent’s history and current step, then aggregates these rewards into a trajectory-level score.The aggregation function is typically Sum or Mean.
Process Reward Modeling: Step- or trajectory-level rewards support search methods such as Best-of-N and Beam Search and provide fine-grained signals for reinforcement learning.These rewards are used to enhance policy-model reasoning performance and supervise learning.

3 General PRMs on Data Analysis Tasks

The study finds that general-domain PRMs lack the discriminative capability to reliably supervise data-analysis agents, despite improving Best-of-N search over single-path generation. Their main failures are penalizing recoverable exploratory errors and missing silent logical errors, motivating an environment-aware verifier that supports exploration and actively checks execution results.

Performance Bottleneck of General PRMs: 40.00%: ThinkPRM improves from 32.67% to 40.00% at N=16, yet PRM-guided search still fails to surpass Majority Voting.The comparison covers Qwen2.5-Math-PRM-72B, GenPRM, and ThinkPRM against a Majority Voting baseline on DABStep.
Misjudgment of Exploratory Failures (Grounding Errors): General PRMs harshly penalize recoverable grounding errors, mistaking necessary exploratory steps for failures and impeding environment adaptation.Grounding errors include syntax or schema mistakes caused by insufficient prior knowledge of the data file, such as guessing an incorrect column name.
Inability to Detect Silent Errors: Static PRMs cannot reliably detect silent errors because code executes without exceptions while logical flaws produce incorrect results.The analysis attributes this limitation to PRMs primarily reading code text rather than verifying execution-result semantics.
Inability to Detect Silent Errors: Environment interaction improves verification of data-analysis step correctness.The study’s takeaway is that PRMs with environment interaction can better verify whether intermediate steps are correct.
Motivation for DataPRM: DataPRM is motivated as an environment-aware verifier that forgives recoverable grounding errors and actively interacts with data to catch silent errors.These requirements are intended to encourage exploration while uncovering execution errors that static reasoning misses.

4 Methodology · 4.1 Environment-Aware Verifier Architecture

DataPRM is an environment-aware generative process reward model built on ReAct, enabling interactive, context-sensitive verification of data-analysis trajectories. It combines multi-step environment probing, tool-augmented coverage, persistent feedback, and reflection-aware ternary rewards.

4.1 Environment-Aware Verifier Architecture: DataPRM adopts the ReAct paradigm and interacts with the environment to provide context-sensitive verification comparable to solution generation.The verifier is modeled using the same ReAct paradigm as the data-analysis agent.
4.1.1 Generative ReAct Paradigm for Verification.: At each policy step, DataPRM receives the trajectory and immediate outcome, then performs an internal multi-step reasoning and verification loop.Each internal step generates a verification tuple containing reasoning, an action, and an observation.
4.1.1 Generative ReAct Paradigm for Verification.: The verifier terminates with a scalar quality score and explanatory rationale rather than a code action.The score evaluates the policy step, while the rationale is derived from the verification trajectory.
4.1.1 Generative ReAct Paradigm for Verification.: DataPRM appends prior verification scores and rationales to subsequent contexts, preserving consistency and continuity across evaluations.Historical verification results are explicitly carried into the next time step.
4.1.2 Tool-Augmented Capability Integration.: DataPRM bridges internal code generation with external tool usage to support multimodal understanding and long-context comprehension during verification.Its verification coverage spans data files, manual documents, and images.
4.1.2 Tool-Augmented Capability Integration.: DataPRM expands binary step rewards to the ternary set R = {0, 0.5, 1} to distinguish grounding errors from other agentic behaviors.The strategy addresses the inability of existing PRMs to distinguish grounding errors when assigning scores.
4.1.3 Reflection-Aware Reward Strategy.: Strictly correct steps receive 1.0, irrecoverable errors receive 0.0, and correctable errors receive 0.5 when environmental feedback enables potential correction.Correctable errors include syntax errors or incorrect file paths, whereas irrecoverable errors contain fundamental logic flaws or hallucinations.

4.2 Step-Level Data Construction

The authors address the scarcity of step-annotated scientific data analysis datasets with a diversity-driven trajectory generation and knowledge-augmented annotation pipeline. The process combines automated collection and query generation with model-assisted step annotation, error attribution, category merging, and human verification.

Motivation: The pipeline targets the lack of public datasets containing both source files and fine-grained step annotations.This limitation makes off-the-shelf process-supervised training difficult.
Trajectory and Query Generation: The authors adapt AutoSDT to crawl GitHub for scientific data-analysis files and have human experts revise and extend a subset.DeepSeek-V3.2 synthesizes reasoning-focused queries, while validated AutoSDT queries are reused for visualization tasks.
Trajectory and Query Generation: The collection strategy retains cases where final answers are not all identical to focus the data on informative boundary cases.
Step-Level Annotation: Qwen3-235B-A22B-Instruct segments trajectories into steps and performs initial annotation and error attribution, while AutoManual merges similar error categories.Human experts verify the merged-category rationales and inject them as structured few-shot examples into the annotation prompt.

4.3 End-to-End RL Training with PRM

The section presents an end-to-end RL framework using GRPO with stabilization strategies and a reward combining outcome correctness with PRM-based process validity. It also aligns terminal PRM estimates with ground-truth outcomes to prevent conflicting optimization signals.

Optimization: GRPO trains the agent end to end, using clip-higher and token-level loss to stabilize optimization.The objective is defined through the ratio between current and previous policies.
Reward Design: The total reward combines outcome reward with PRM scores, balancing outcome correctness against process validity.The weighting coefficient β controls this trade-off, while PRM scores are associated with agent steps.
Optimization: Group-normalized advantages are computed across outputs using group size G to support GRPO updates.The advantage is calculated for each output and agent step.
Consistency Control: A consistency check resolves discrepancies between ground-truth outcomes and the PRM’s final-step estimates.This prevents the model from learning from conflicting signals at trajectory termination.

5 Experiments

DataPRM consistently outperforms strong verification baselines across Best-of-N, Beam Search, and DVTS, while ablations show that environment interaction and multi-turn reasoning are important. Its process supervision also improves reinforcement-learning outcomes, despite higher verification cost than GenPRM.

Test-Time Scaling: DataPRM outperforms GenPRM-32B, Qwen2.5-Math-PRM-72B, DeepSeek-V3.2 LLM-as-a-judge, and Qwen3-235B-A22B-Instruct self-rewarding despite using only 4B parameters.These results are reported in the Best-of-N setting, where stronger reward supervision helps discern high-quality data-analysis trajectories.
Test-Time Scaling: DataPRM consistently outperforms all baselines across Beam Search and Diverse Verifier Tree Search at every computation budget.Other baselines show instability under Beam Search, including Qwen2.5-Math-PRM-72B.
Ablation Study: Environment interaction improves over the Chain-of-Thought baseline, while combining multi-turn interaction with the environment significantly boosts performance.Multi-turn interaction alone provides only marginal gains, indicating that executable feedback is central to grounding reasoning.
Inference Cost: DataPRM uses 2.57 turns and 0.87 tool calls on average, trading higher token usage and latency than GenPRM for executable verification of intermediate states.Compared with Self-Rewarding, it reduces total token consumption from 25.3K to 21.5K tokens, a 15.1% reduction.

6 Related Work

Related work spans process reward modeling and autonomous data analysis agents. PRMs provide granular supervision mainly in non-interactive domains, while data analysis agents have increasingly targeted end-to-end real-world tasks through prompting and predefined workflows.

Process Reward Models: PRMs provide granular rewards and support Test Time Scaling and Reinforcement Learning applications.Existing PRM research has primarily emphasized settings without environmental interaction.
Process Reward Models: Current PRMs mainly address non-interactive domains including mathematics, code generation, and tabular reasoning.These domains generally do not require environmental interaction during supervision.
Data Analysis Agents: Data analysis agents aim to autonomously complete end-to-end data analysis tasks in real-world scenarios.Early approaches relied primarily on prompt engineering and predefined workflows to leverage closed-source models’ reasoning and coding capabilities, including visualization and insight or report generation.

7 Conclusion

DataPRM is an environment-aware process reward model that uses active verification and ternary rewards to provide precise step-level supervision for interactive data analysis. Its scalable construction pipeline and empirical results support gains in Test-Time Scaling and Reinforcement Learning.

Conclusion: DataPRM addresses general PRMs’ limitations in detecting silent and grounding errors during interactive data analysis.It is designed as an environment-aware process reward model.
Conclusion: Active environment verification and a ternary reward strategy enable DataPRM to deliver precise step-level supervision.The model was constructed using diversity-driven trajectory generation and knowledge-enhanced expert annotation.
Conclusion: DataPRM significantly enhances performance under both Test-Time Scaling and Reinforcement Learning.These results validate the model’s effectiveness across both training and inference paradigms.

8 Limitations and Ethical Considerations

The work is limited to reasoning- and visualization-focused data analysis and trains DataPRM solely through SFT, which depends heavily on high-quality trajectory data. It follows established ethical practices by using synthesized or publicly available datasets and citing sources accurately.

Limitations: The study focuses primarily on data analysis involving reasoning and visualization, leaving complex engineering tasks such as model training and prediction for future exploration.This scope excludes more complex engineering-oriented tasks.
Limitations: DataPRM is trained solely via Supervised Fine-Tuning, a paradigm that relies heavily on high-quality trajectory data.The authors identify this reliance as a limitation and data dependency.
Ethical Considerations: The work uses only synthesized or publicly available datasets and accurately cites all sources for transparency and proper attribution.These practices are presented as adherence to established ethical research standards.

A Theoretical Perspective for Environment-Aware Verifier

The paper models data analysis as a POMDP with an unknown environment state, explaining why static PRMs can misreward out-of-distribution scientific trajectories. DataPRM addresses this uncertainty through environment interaction and a theoretically grounded reward combining task progress with information gain.

POMDP Formulation: Data analysis is formalized as a POMDP in which the true environment state ε is latent, while static PRMs rely on an internal prior P_prior(ε|h_t).Real-world scientific data may be heterogeneous and out-of-distribution relative to this learned prior.
Bayesian Evidence Gathering: DataPRM interacts with the environment to obtain groundtruth observations and update its uncertain prior into an accurate posterior via Bayes’ theorem.This interaction gathers evidence that grounds latent variables and reduces reward-estimator variance.
Reward Decomposition: Theoretical analysis decomposes each step’s reward into progress toward the final goal and information gain about the hidden environment, balanced with λ=0.5.The formulation captures the need to balance exploitation for task progress with exploration for uncertainty reduction.
Reward Decomposition: R(a_t) = λ·G(a_t) + (1 −λ)·I(a_t), with information gain defined as D_KL(P_post∥P_prior) and approximated by I[I(a_t) > ε].The indicator approximation uses a small threshold ε because continuous KL-divergence rewards are difficult to annotate reliably.
Ternary Reward: Strictly Correct receives R=1 when an action advances the task and confirms the validity of the current logic.This connects the progress and information-gain components to the paper’s three-value reward mechanism.