PaperlayerView exact revision on arXiv

Source-linked AI summary

AdaPRA: Process Reward Agents with Adaptive Rubrics for Data Science

Atharva Naik, Vivek Chari, Pol Llado, Daniel Fried, Carolyn Rosé, Benjamin Van Durme, Sam Thomson, Justin Svegliato

arXiv:2610.33245v1cs.AIcs.LG

TL;DR

Data-science agents can complete end-to-end ML workflows, but their leaderboard scores lag amid process errors and repeated work across attempts. ADAPRA combines persistent workspaces with rubric-guided supervision that checks trajectories and workspaces and adapts criteria to each task. With five attempts, it improves mean Kaggle percentile on both DSPredict benchmarks.

  • Problem

    Data-science agents’ leaderboard performance lags, with process errors and repeated exploratory work across attempts remaining challenges.

    Source: §1 · p. 1

  • Method

    ADAPRA combines a persistent per-task workspace with a PRA that checks trajectories and workspaces against discovered, task-adapted rubrics.

    Source: §1 · pp. 1–2

  • Results

    24.5 points on DSPredict-Easy and 14.8 points on DSPredict-Hard: ADAPRA improves mean Kaggle percentile for an 80B Qwen3-Coder-Next agent, matching or exceeding GPT-5.1 (high) across metrics.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Rubrics learned from easier tasks generalize to harder tasks, and adapting them to each task further improves performance.

    Source: §1 · p. 2

    The study does not examine how supervision gains change as the number of attempts grows.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Data science agents can automate end-to-end machine learning workflows, yet they remain far from competitive on human leaderboards. Agents make process errors such as insufficient exploration, unsound validation, coding bugs, and poor use of a fixed budget (re-running the same exploratory data analysis and baseline models in every attempt). We propose AdaPRA to address these issues: a persistent coding agent harness that accumulates knowledge across attempts on the same task, with rubric-guided process supervision that identifies and corrects errors. Our process reward agent audits the agent's trajectory and workspace by executing checks, and adapts its rubrics to each task. We find that rubrics seeded from easier tasks generalize to harder tasks, with rubric adaptation further improving performance. With five attempts per task, AdaPRA improves an 80B Qwen model's leaderboard percentile by 24.5 points on DSPredict-Easy and 14.8 points on DSPredict-Hard while reducing process errors. We also show that supervision with rubrics mined from one base model's runs improves another (DeepSeek-V3.2), and that AdaPRA beats DataPRM, a competitive PRM for data science. We also share our code for reproducibility.

1 Introduction

ADAPRA targets process errors that can remain invisible in trajectories and wasted repeated attempts by pairing persistent workspaces with rubric-guided supervision. Its checks inspect both trajectories and workspaces, while rubrics are discovered and adapted for each task.

1 Introduction

  • Process errors include insufficient exploration, unsound validation, and coding bugs, and some produce valid submissions without leaving evidence in the trajectory.A wrong file path can silently turn every test clip into silent audio and every prediction into “unknown.”Source: §1 · p. 1, Figure 1 · p. 1
  • A persistent workspace carries exploratory insights and trained models across attempts, while the PRA checks trajectories and workspaces against rubrics using tools and code execution.Seed rubrics are discovered from past agent runs and adapted to each task.Source: §1 · pp. 1–2
  • 24.5 points on DSPredict-Easy and 14.8 points on DSPredict-Hard: ADAPRA improves mean Kaggle percentile for an 80B Qwen3-Coder-Next agent, matching or exceeding GPT-5.1 (high) across metrics.The Hard tasks span vision, text, speech, and molecular prediction.Source: §1 · p. 2
  • ADAPRA combines a persistent harness, rubric-guided process supervision, seed-rubric discovery, and task-specific rubric adaptation.Its rubric discovery contrasts whole attempts using both trajectories and workspace artifacts, with rubrics transferring from easier tasks to harder tasks and another base model.Source: §1 · p. 2

2 Related Work

ADAPRA differs from data-science agent scaffolds and prior rubric or process-reward approaches in where search resides, what evidence is checked, and how rubrics are adapted and discovered.

Agents for data science

Rubric adaptation

  • RubricEM and Shao et al. adapt rubrics for training, whereas ADAPRA adapts them during task supervision without gold feedback such as Kaggle scores (Li et al., 2026; Shao et al., 2025).Its PRA checks rubrics with tools and code, and stores task-specific pitfalls for later reviews.Source: §2 · p. 2

Process reward agents

Rubric discovery

  • Unlike rubric discovery from responses or trajectories alone, ADAPRA also mines workspace artifacts and reports transfer from easier to harder tasks (Li et al., 2026; Zhang et al., 2026b).Source: §2 · p. 3

3 Method

ADAPRA combines a persistent per-task workspace with a PRA that reviews agent progress against seed and task-adapted rubrics. Rubrics are discovered from prior attempts using both trajectory and workspace evidence.

3 Method

  • ADAPRA combines a persistent harness with a PRA that checks the agent against offline-discovered rubrics, grounds judgments in code, and adapts rubrics to each task.Source: §3 · p. 3

3.1 Problem Setup

  • For each attempt, the PRA evaluates the partial trajectory and current workspace against each rubric, returning satisfaction probability, importance, rationale, and a suggested next action.Feedback includes only rubrics with importance at least 0.5 and satisfaction probability below 0.6.Source: §3.1 · p. 3

3.2 Agent Harness with Memory

  • The workspace preserves files and results across attempts while the model context resets, allowing later attempts to reuse prior work and summaries.Only the agent’s and PRA’s outputs are carried across attempts; the carried scores are the agent’s own validation scores.Source: §3.2 · pp. 3–4

3.3 Rubric Discovery

  • Offline rubric discovery contrasts stronger and weaker attempts using trajectories and workspace artifacts, then clusters selected behaviors into a mutually exclusive, collectively exhaustive rubric set.The 19 seed rubrics are applied unchanged to DSPredict-Hard after discovery on DSPredict-Easy.Source: §3.3 · p. 4

3.4 Process Reward Agent

  • The PRA checks rubric claims against the agent’s files and can execute its own code; periodic reviews return prioritized failing rubrics while course correction is still possible.A failed finish review allows one final refinement round.Source: §3.4 · p. 5

3.5 Adaptive Rubrics

  • A per-task rubric store lets the PRA add, update, or retire criteria as task-specific pitfalls emerge, and each addition requires a recorded check in the same review.Stores are isolated across tasks; on House Prices, an added rubric checks whether out-of-fold predictions are underdispersed relative to the target.Source: §3.5 · p. 5

4 Experiments

Experiments compare ADAPRA and ablations on DSPredict-Easy and DSPredict-Hard using a five-attempt setup, with additional single-attempt studies. Evaluation reports task-level outcomes and considers multiple submission-selection policies.

Benchmarks

  • DSPredict-Easy covers 38 mostly tabular Kaggle competitions, while DSPredict-Hard covers 54 competitions across tabular, vision, text, audio, and molecular data.Easy is the development benchmark for seed rubrics; Hard tests transfer to harder tasks.Source: §4 · p. 5

Models and budget

  • Every main condition uses five attempts per task with Qwen3-Coder-Next as the base agent and Claude Sonnet 4.6 as the PRA.Hard tasks receive a larger budget and GPU access.Source: §4 · p. 5

Baselines and Ablations

  • The comparisons include workspace-reset and persistent-harness baselines, DataPRM, ADAPRA, fixed seed rubrics, and a condition without PRA code execution.DataPRM judges individual steps; ADAPRA uses rubrics that can cover errors spanning many steps.Source: §4 · pp. 5–6

Additional ablations

  • Additional DSPredict-Easy ablations use one attempt to vary PRA review cadence or substitute DeepSeek-V3.2 as the base agent.Source: §4 · p. 6

Metrics

  • Metrics include mean percentile, valid submissions, above-median outcomes, and medals; invalid or missing submissions count as zero percentile.Main tables report best and mean attempts, and mean-percentile differences use one-sided paired permutation tests over tasks.Source: §4 · p. 6

5 Results

Across DSPredict-Easy and DSPredict-Hard, AdaPRA improves Qwen3-Coder-Next’s leaderboard results over the DSGym harness. Ablations distinguish the contributions of a persistent workspace, code-executing process supervision, and task-adaptive rubrics; seed rubrics also transfer from Easy to Hard, with adaptation adding domain-specific criteria.

5.1 DSPredict-Easy

  • On DSPredict-Easy, mean percentile rises from 44.5 to 69.0 and the above-median rate from 50.0% to 86.8%.These are the reported Qwen3-Coder-Next results with five sequential attempts per task.Source: §5.1 · p. 6, Table 1 · p. 7

5.2 DSPredict-Hard

  • On DSPredict-Hard, mean percentile rises from 21.3 to 36.1, valid rate from 68.5% to 90.7%, and above-median rate from 11.1% to 33.3%.ADAPRA’s best attempt exceeds a single GPT-5.1 (high) attempt on valid, above-median, and medal rates; per attempt, it trails GPT-5.1 slightly in medal rate (4.1% vs. 4.8%).Source: §5.2 · p. 6, Table 2 · p. 7

5.3 Ablations

  • The persistent harness contributes 16.3 percentage points on Easy, while supervision contributes 10.6 points on Hard.The respective contributions are 8.2 points for supervision on Easy and 4.2 points for the harness on Hard; the harness alone raises Easy mean percentile from 44.5 to 60.8 and Hard from 21.3 to 25.5.Source: §5.3 · p. 6
  • Freezing seed rubrics lowers mean percentile by 1.7 points on Easy and 4.3 on Hard, where the above-median rate falls from 33.3% to 22.2%.The Hard gap is significant (p = 0.024); adaptation adds 177 rubrics and updates one across four adaptive runs, mostly in the first two attempts.Source: §5.3 · p. 6
  • Reading-only scoring lowers mean percentile to 63.0 on Easy and 31.8 on Hard; without execution, supervision still beats the harness alone on both.The ablation comparison finds ADAPRA has the highest mean percentile on both benchmarks, indicating that code execution and rubric adaptation are complementary.Source: §5.3 · p. 7
  • With one attempt per task, DeepSeek-V3.2’s mean percentile rises from 36.9 to 44.7 over the harness alone.For cadence, invoking the PRA every 10 turns performs best in the tested single-attempt setting (56.4 vs. 48.4 for the harness alone); Hard uses c=20 to bound cost because attempts take 1.6 times as many agent turns and the PRA accounts for 75% of run cost.Source: §5.3 · pp. 7–8

5.4 Easy-to-Hard Generalization

  • Frozen seed rubrics add 6.3 percentile points over the harness on Hard, close to their 6.5-point gain on Easy.The seed set targets failures shared across benchmarks, including simplicity bias; invalid cross-validation rises from 32% of Easy attempts to 57% of Hard attempts, and unverified submissions from 33% to 52%.Source: §5.4 · p. 8
  • Adaptation adds 1.2 rubrics per Hard task versus 0.8 per Easy task, adding at least one on 79% versus 51% of tasks.Hard-task additions include domain-specific rules such as non-negative time-to-failure predictions and excluding quantum-mechanical target components from features.Source: §5.4 · p. 8

6 Conclusion

ADAPRA combines a persistent workspace, rubric-guided process supervision, and offline rubric discovery to improve data-science agent performance. Its results span two benchmarks, rubric transfer, and a second base model.

6 Conclusion

  • Mean Kaggle percentile rises from 44.5 to 69.0 on DSPredict-Easy and from 21.3 to 36.1 on DSPredict-Hard, matching or exceeding GPT-5.1 (high).Each of the harness, process reward agent, and adaptive rubrics contributes to the gains; removing any one lowers performance on both benchmarks.Source: §6 · p. 8
  • Rubrics discovered on mostly tabular DSPredict-Easy tasks transfer to vision, text, audio, and molecular DSPredict-Hard tasks, while adaptation adds domain-specific criteria.The conclusion attributes transfer to targeted process failures coming from the agent rather than the domain.Source: §6 · p. 8
  • Rubrics mined from Qwen3-Coder-Next runs also improve DeepSeek-V3.2, and ADAPRA outperforms DataPRM (Qiu et al., 2026).Source: §6 · p. 8
  • ADAPRA’s persistent workspace carries task knowledge across attempts, while its process reward agent audits trajectories and workspaces and adapts rubrics; seed rubrics are discovered offline.Source: §6 · p. 8

7 Future Work

The authors identify four directions: training the base agent on PRA feedback, improving rubric credit assignment, replacing the closed PRA, and evolving the harness.

7 Future Work

  • Training the base agent on dense PRA feedback could help it internalize process criteria and depend less on test-time supervision.Source: §7 · p. 8
  • Credit assignment for individual rubrics could make adaptation more selective by informing rubric revision or retirement.Current adaptation does not measure each rubric’s effect on outcomes and retired none in the reported runs.Source: §7 · p. 8
  • Training smaller open-weight models to replace the closed-source PRA could extend Gandhi et al. (2026) from non-agentic critics to agentic PRAs.Source: §7 · p. 8
  • Harness evolution such as Meta-Harness (Lee et al., 2026) could further improve the persistent harness.Source: §7 · p. 8

Limitations

The evaluation and supervision design have several stated scope limits, including cadence, benchmark coverage, attempt count, rubric adaptation, and model choice. The supervisor design comparisons also use read-only workspace and trajectory access.

Limitations

  • DSPredict-Hard uses PRA supervision every 20 turns to bound cost, though cadence 20 is the weakest tested Easy setting and Hard results may understate gains from more frequent supervision.The PRA accounts for 75% of a Hard run’s cost; 10 turns works best on Easy.Source: Limitations · p. 9
  • Evaluation covers only Kaggle leaderboard competitions in DSPredict-Easy and DSPredict-Hard, not open-ended analysis or tasks without predictive objectives.Source: Limitations · p. 9
  • Multi-attempt results use five attempts per task, while additional ablations use one; the study does not test how supervision gains change as attempts increase.The authors note that selecting among more unsupervised attempts may recover part of the difference.Source: Limitations · p. 9
  • Rubric adaptation mostly adds criteria—177 additions, one update, and no retirements—because the PRA lacks local validation credit assignment for individual rubrics.The authors suggest better credit assignment could make adaptation more selective.Source: Limitations · p. 9
  • Main results use one base agent and one PRA; DeepSeek-V3.2 is tested only with one attempt per task on DSPredict-Easy.Source: Limitations · p. 9
  • The PRA is Claude Sonnet 4.6, a closed commercial-API model, so exact reproduction depends on its continued availability and unchanged behavior.Source: Limitations · p. 9

B Supervision Design Choices

  • The supervisor ablations vary operationalization and rubric adaptation; file-based trajectory access performed better in short DSPredict-Easy experiments, and the PRA cannot modify the agent’s workspace or trajectory.The PRA can read both the workspace and trajectory.Source: B Supervision Design Choices · p. 10

C Harness Implementation Details

The implementation isolates each task’s workspace and applies benchmark-specific time limits, model settings, and compute resources. The design comparisons distinguish execution and rubric-store choices, while the PRA has separate inference and rubric-discovery configurations.

C Harness Implementation Details

  • Each task runs in its own container and workspace, and the PRA mounts that workspace read-only; tasks execute in parallel without sharing files.Source: C Harness Implementation Details · p. 10
  • Time limits are 2 hours for unsupervised Easy attempts, 3 hours for supervised Easy attempts, and 4 hours for all Hard conditions.Most unsupervised Easy attempts finish well before their limit, with a median of 23 minutes; supervised runs also cap agent-set timeouts.Source: C Harness Implementation Details · pp. 10–11
  • The design coverage compares work domain, operationalization, and rubric-store choices, including execute versus read and fixed versus adaptive rubrics.Table 3 marks explicit variation, ablation-only variation, or fixed/not applicable dimensions; Table 4 gives ADAPRA’s supervisor configuration and the alternatives used in two ablations.Source: C Harness Implementation Details · p. 11, Table 3 · p. 11, Table 4 · p. 11
  • Qwen3-Coder-Next uses temperature 0 and up to 16,000 output tokens; DeepSeek-V3.2 uses high reasoning effort and up to 8,000 tokens, with provider-default temperature.The PRA uses native function calling, extended thinking, high reasoning effort, a 63,999-token thinking budget, and up to 64,000 output tokens; offline rubric discovery uses Claude Sonnet 4.6 without extended thinking.Source: C Harness Implementation Details · p. 11

D Computational Environment

  • Runs use Slurm, AMD EPYC 7763 CPUs, and NVIDIA A100 80GB GPUs; Hard containers use GPUs, while Easy containers run on CPUs alone.Each harness run solves four tasks in parallel; base agents and the PRA use Amazon Bedrock, while DataPRM is locally served on one A100 GPU.Source: D Computational Environment · p. 11

E Prompts

Appendix E documents the prompts and operating variants used to guide the base agent and the Process Reward Agent across attempts. It also describes how the PRA conducts evidence-grounded audits and how observable, outcome-relevant behaviors are derived into reusable process rubrics.

E Prompts

  • The reproduced prompts use “PRM” for PRA, frame scoring as best-of-N, and note that DSPredict-Hard supervised runs reword two timeout instructions.Angle-bracket text is filled at runtime, and bracketed identifiers may be elided; the prompts are otherwise verbatim apart from normalized punctuation.Source: E Prompts · pp. 11–12

E.1 Base Agent

  • From attempt two onward, the base agent receives earlier attempt summaries and final PRA feedback before the task message, with the objective framed as maximizing across attempts.The first attempt instead begins with the competition description and dataset path.Source: §E.1 · p. 12
  • The prompt requires a modular, reusable workspace across attempts and asks the agent to cache fitted models, predictions, and CV scores rather than retrain scored models.It recommends one model family per script, reusing saved predictions for ensembles, and making scripts idempotent.Source: §E.1 · p. 12
  • The prompt directs the agent to edit canonical files in place rather than create numbered copies, keeping the workspace organized as it iterates.It also advises clean, efficient code and separating larger functions or files into smaller pieces.Source: §E.1 · p. 12
  • The agent is instructed to maintain a README ledger of approaches and CV scores, consult it at the start of later attempts, and carry forward terse observations and tips.A machine-readable cv_scores.json mirrors the ledger.Source: §E.1 · p. 13
  • The workflow calls for broad exploration across model families, features, tuning, and ensembles while keeping experiments on the same fixed CV scheme and reusing cached predictions.The prompt describes exploration, modularization, a baseline, iteration, and submission as its workflow stages.Source: §E.1 · p. 13
  • The prompt requires correct submission formatting, a fixed and robust CV scheme across models and attempts, and a final check that the submitted predictions come from the best-scoring model.The agent must copy that model’s saved test predictions, verify the submission on disk, and report its validation score before finishing.Source: §E.1 · p. 13
  • The base agent is instructed to preserve reusable scripts, processed data, notes, model predictions, and CV scores so later attempts can reuse prior work instead of repeating it.The prompt describes the workspace as shared code across attempts and calls for a scoreboard that makes model comparisons available without rerunning them.Source: §E.1 · p. 14
  • The prescribed workflow moves from checking saved EDA notes through reusable setup, baseline and iterative modeling, cross-validation, submission, and reporting a final validation score.The prompt also advises estimating runtime, starting with fast models, simplifying after timeouts, and prioritizing a valid submission when budgets run low.Source: §E.1 · p. 14
  • In supervised DSPredict-Hard runs, the prompt replaces bash-tool timeout guidance with shell timeouts to avoid blocking commands consuming the attempt.It says the bash-tool timeout is capped and a large setting disables the no-output safety limit; shell `timeout` bounds the command instead.Source: §E.1 · pp. 14–15
  • For best-of-N runs, the base agent should carry forward lessons but favor a genuinely different modeling approach rather than reproduce a previous attempt.Only the single best attempt is scored, so repeating an approach wastes a limited attempt while a distinct approach can still improve the best result.Source: §E.1 · p. 15

E.2 Process Reward Agent

  • The PRA reviews an ongoing attempt at periodic checkpoints and must treat work not yet reached as not-yet-applicable, rather than as a defect.A transcript ending at the latest turn is only a snapshot and does not show that the base agent has finished or abandoned the attempt.Source: §E.2 · p. 15
  • The PRA’s two equally important duties are to verify submission soundness and identify concrete modeling opportunities using diagnostics run on existing artifacts.Soundness checks include whether the submission matches OOF predictions and is well formed; modeling diagnostics can test residual signal, calibration, fold stability, and leakage.Source: §E.2 · p. 15
  • The PRA must independently reproduce claimed CV scores, avoid retraining models, and keep status tied to soundness rather than model quality.Modeling feedback must cite measured findings; a modeling suggestion alone cannot make an otherwise sound submission fail.Source: §E.2 · pp. 15–16
  • The prompt prioritizes checking whether the submitted file matches the out-of-fold predictions, because reported cross-validation is only a claim and may not describe the submission.It instructs the auditor to check submission-versus-OOF consistency first.Source: §E.2 · p. 16
  • The auditor may run code and inspect data only in read-only base repositories and data directories, while keeping its scripts and scratch work in its own writable workspace.With no test labels, it verifies methodology rather than the answer.Source: §E.2 · p. 16
  • Rubrics are specifications, not prewritten checks, so the auditor writes small bespoke scripts using the task’s actual columns, metric, and transformations.Each check should read a few files and print measured numbers with a one-line PASS, WARN, or FAIL result.Source: §E.2 · p. 16
  • Because checks persist across attempts while message history resets, the auditor reads the script index, reuses fitting checks, and updates its README whenever a script changes.The README records each script’s filename, rubric, purpose, and exact run command.Source: §E.2 · p. 16
  • The procedure runs soundness checks first and records each result immediately, then looks once for OOF predictions and proceeds to relevant modeling diagnostics if turns remain.If OOF predictions are not found promptly, the auditor records uncertainty and moves on rather than continuing to search.Source: §E.2 · p. 16
  • The auditor may add a rubric for a task-specific failure or recurring weakness, but must submit a verdict covering applicable rubrics and grounding scores in recorded results or inspected facts.The harness derives the status and finish gate from those scores; the auditor does not set them directly.Source: §E.2 · pp. 16–17
  • For each rubric, the auditor assigns relevance and success scores, grounding dynamic success probabilities in executed checks and static judgments in inspected artifacts or code.A rubric for an unreached stage gets low importance rather than a low success score; 0.5 denotes maximum uncertainty.Source: §E.2 · p. 16
  • In best-of-N runs, the auditor rewards genuinely different approaches and flags duplicate attempts because only the best submission is scored.It reviews prior trajectories when judging cross-attempt redundancy or process behavior.Source: §E.2 · p. 17
  • New rubric specs must describe reusable invariants rather than task-specific columns or thresholds, and adding one requires first running and recording its check.The prompt prioritizes soundness checks against out-of-fold predictions, submission coverage, and reproducibility of reported cross-validation results.Source: §E.2 · pp. 17–18
  • The PRA records executed checks with evidence and scores before citing them in verdicts, then submits per-rubric scores that drive thresholded feedback in the agent’s conversation.Feedback is ordered by importance and carried into the next attempt with instructions to address each issue.Source: §E.2 · p. 17
  • The −PRA ablation removes terminal access and replaces executable checks with file reading, while the −adaptive-rubrics ablation retains PRA instructions but disables rubric editing tools.The rubric store, scoring interface, and feedback format remain unchanged in the −PRA variant.Source: §E.2 · p. 18

E.3 Rubric Discovery

  • Of 906 mined behaviors, stage two pools 519 good behaviors from top attempts and issues from bottom attempts, then clusters them into a MECE rubric set.The pool contains 238 good behaviors and 281 issues; stage two drops 112 behaviors contrary to their attempt outcomes, middle-attempt behaviors, and behaviors from unscored valid attempts.Source: §E.3 · pp. 18–19
  • Rubric discovery uses attempt transcripts and persistent workspaces to mine concrete, generalizable behaviors associated with outcomes among sibling attempts on the same competition.The agent’s end-of-attempt summary is an orientation aid, not ground truth.Source: §E.3 · p. 19
  • The discovery prompt evaluates validation discipline, whether the final submission uses the best-CV model, and whether the submission covers test rows with valid predictions.Its validation examples include leakage, overfitting, and noisy gains; submission examples include partial, default-filled, corrupted, or invalid predictions.Source: §E.3 · p. 19
  • Other discovery lenses include efficient exploration, modeling strength, and decisive recovery through catching mistakes, reverting regressions, and finishing with a verified submission.The prompt flags repeated crashes, repeated EDA, timed-out searches, and plateaued churn as wasted motion.Source: §E.3 · p. 19
  • Behaviors should be atomic, generalizable, evidence-grounded, and plausibly outcome-moving, with artifact grounding checkable from workspace files and trajectory grounding requiring process evidence.Examples include submission integrity as an artifact and repeated crashes or failure to pivot as trajectory behaviors.Source: §E.3 · p. 19
  • The first discovery stage labels attempts as top- or bottom-band among valid sibling attempts, extracting behaviors for a PRM to reward or issues to penalize.Top-band attempts may also have an issue noted, and bottom-band attempts may still exhibit a good behavior.Source: §E.3 · p. 19
  • The second stage derives a small, mutually exclusive and collectively exhaustive rubric set bottom-up, treating good behaviors and issues as positive and negative evidence for the same criteria.The pool contains behaviors mined from real trajectories and associated with top-outcome, bottom-outcome, or invalid attempts.Source: §E.3 · p. 20
  • Rubrics should be specific, observable, domain-grounded, and prioritized by frequency and outcome relevance, with both artifact- and trajectory-grounded rubrics represented.The guidance folds a good behavior and its mirror-image issue into one rubric and excludes rare or incidental cases.Source: §E.3 · p. 20
  • The output schema specifies good-practice rubric proposals with four scoring labels and concrete fix suggestions, targeting a compact set of 12–20 rubrics.Each proposal includes a grounding, category, descriptions, scoring instruction, supporting behavior count, and example evidence.Source: §E.3 · p. 20
  • The 17 rubrics covered 378 of 387 dropped behaviors (97.7%), versus 59 of 60 kept behaviors (98.3%).The nine unassigned behaviors were mostly isolated, low-severity cases; the largest group was close to Timeout-Aware Experiment Design.Source: §E.3 · p. 20

F DataPRM Baseline

DataPRM is used as an online supervisor under ADAPRA’s otherwise identical setup, while the related-work section distinguishes rubric-based reward approaches by what they supervise and how rubrics are supplied.

F DataPRM Baseline

  • DataPRM is a generative process reward model for agentic data analysis that reasons about each trajectory step, executes Python checks, and assigns a ternary score.The evaluation uses its released ScienceAgentBench checkpoint, the closer of its two released checkpoints to model-building tasks.Source: F DataPRM Baseline · p. 21
  • DataPRM replaces the PRA while the base agent, harness, five-attempt protocol, review cadence, finish refinement, and feedback channel remain fixed.This isolates the supervisor comparison; DataPRM verifies each turn since its previous invocation and returns FAIL, WARN, or PASS based on step scores.Source: F DataPRM Baseline · p. 21
  • Online use requires scoring only newly taken steps and linking DataPRM to the agent’s live workspace, because rebuilding workspaces by re-executing hours of training code is infeasible.The online adaptation also clips and drops history to fit the checkpoint’s 24K-token context and retains a query_image tool description despite not serving a vision model.Source: F DataPRM Baseline · p. 21
  • DataPRM verifies a median of 10 steps per invocation in 126 seconds, returning FAIL on 43% of invocations, WARN on 32%, and PASS on 24%.It scores 10% of verified steps 0; many such scores flag code or analysis errors, while others penalize approaches tried in earlier attempts.Source: F DataPRM Baseline · p. 21

G Rubric and Principle Reward Models

  • Rubric-based reward work includes structured criteria for reward modeling, reinforcement learning, checklists, and principle following, while the PRA can revise rubrics and judge long-horizon trajectories and artifacts.Unlike approaches focused on a single final response, the PRA receives rubrics as input rather than learning them in training.Source: G Rubric and Principle Reward Models · p. 22

H Selection-Policy Analysis

The selection-policy analysis contrasts attempt choices that are available during a sequential run with the best attempt, which can only be identified after the fact.

H Selection-Policy Analysis

  • The last attempt is the natural deployment choice for a sequential shared-workspace run, whereas reporting the best attempt assumes after-the-fact identification.The analysis also reports first, last, mean-over-scored-attempts, and best-attempt selection policies.Source: H Selection-Policy Analysis · p. 22
  • On DSPredict-Easy, the last attempt beats the first by 4.5 points without supervision and 6.5 with ADAPRA, whose last-attempt score of 62.9 is 6.1 points below its best.On DSPredict-Hard, the last attempt improves on the first in every condition; ADAPRA has the highest mean and best-attempt percentiles, 32.1 and 36.1, while its last-attempt 30.2 is 0.2 below the highest last-attempt score, −PRA’s 30.4.Source: H Selection-Policy Analysis · p. 22

I Paired Significance of Process Supervision

Paired task-level tests compare supervision conditions with the harness alone and compare ADAPRA with its ablations; the section also documents the seed rubrics used to initialize rubric stores.

I Paired Significance of Process Supervision

  • The primary significance test is a one-sided paired permutation test on task-level mean differences, using 200,000 random sign flips without assuming symmetric differences.The analysis also reports sign and Wilcoxon tests and a nested bootstrap interval resampling tasks and attempts.Source: I Paired Significance of Process… · p. 22
  • On DSPredict-Easy, ADAPRA’s gains over the harness alone are +8.2 on best attempt and +9.7 per attempt, both p < 0.001; its best-attempt gain over −PRA is +6.0 (p = 0.012).Its best-attempt advantage over −adaptive rubrics is +1.7 and not significant (p = 0.147).Source: I Paired Significance of Process… · p. 22
  • On DSPredict-Hard, ADAPRA’s gains over the harness alone are +10.6 on best attempt (p < 0.001) and +7.8 per attempt (p = 0.001); its best-attempt gain over −adaptive rubrics is +4.3 (p = 0.024).Its best-attempt gain over −PRA is also +4.3 but not significant (p = 0.077), and neither direct per-attempt difference is significant.Source: I Paired Significance of Process… · pp. 22–23

J Seed Rubrics

  • The 19 seed rubrics include 17 mined from harness-only DSPredict-Easy runs and two hand-curated rubrics targeting failures that arise only across attempts.The two hand-curated rubrics account for 11–17% of rubric failures across supervised runs and are the only failing rubrics in 1–3% of reviews that report failure.Source: J Seed Rubrics · p. 23

K Seed Rubric Failures Across Benchmarks

The seed-rubric analysis compares rubric failure patterns across benchmarks and reports how often flagged issues are resolved within DSPredict-Hard attempts.

K Seed Rubric Failures Across Benchmarks

  • Across 19 seed rubrics, failure-rate correlations between DSPredict-Easy and DSPredict-Hard are ρ = 0.66 (p = 0.002), and mean-importance correlations are ρ = 0.91 (p < 10^-7).The compared frozen-rubric runs score the same rubrics at every review.Source: K Seed Rubric Failures Across Benchmarks · p. 23
  • Seven examples pair failures flagged on non-tabular DSPredict-Hard tasks with the same rubric failing on a tabular DSPredict-Easy task, using rubric text discovered on DSPredict-Easy.A flag counts as resolved when a later review reports success probability at least 0.7 and the rationale confirms the agent fixed the issue.Source: K Seed Rubric Failures Across Benchmarks · pp. 23–24
  • In DSPredict-Hard’s frozen-rubric run, 685 of 1,277 flags (54%) are resolved within the attempt across 53 of 54 tasks.Resolution is highest for pipeline checks, including submission integrity (83%) and regression detection (82%), and lowest for simplicity-bias checks, including model diversity (16%).Source: K Seed Rubric Failures Across Benchmarks · p. 24

L Rubric Store Operations

Rubric-store adaptation mainly adds rubrics, with most additions arriving early. Code execution changes the kinds of checks added and increases rubric coverage on DSPredict-Hard, while the −PRA ablation edits stores at a similar rate.

L Rubric Store Operations

  • Across four runs, the PRA adds 177 rubrics, updates one, and retires none; 81%–94% of additions occur in the first two attempts.Later attempts are therefore reviewed against a largely stable store.Source: L Rubric Store Operations · p. 24
  • The −PRA ablation adds rubrics at a similar rate: 0.76 vs. 0.84 per task on DSPredict-Easy and 1.10 vs. 1.21 on DSPredict-Hard.Removing code execution does not reduce how often the store is edited.Source: L Rubric Store Operations · p. 24
  • With code execution, the PRA adds at least one rubric on 79% of DSPredict-Hard tasks versus 55% for −PRA, but on DSPredict-Easy the shares are 51% and 63%.Figure 6 reports how many rubrics each task’s store gains.Source: L Rubric Store Operations · p. 24
  • Prediction-sanity checks make up 35% of DSPredict-Easy and 19% of DSPredict-Hard additions with code execution, versus 3% and 7% without it.Without code execution, train–test and submission consistency checks account for 41% and 44% of additions, respectively.Source: L Rubric Store Operations · p. 24

M Example PRA Feedback

These examples illustrate how the PRA audits agent trajectories and workspaces, uses executed checks and task-specific rubrics to identify process and validation problems, and guides revisions across attempts. They also cover rubric persistence and limitations, feedback cadence and alternative agents, and the choice to report only failed rubrics.

M Example PRA Feedback

  • Nested cross-validation showed isotonic calibration worsened RMSE from 0.073994 to 0.074386, and after reverting it the agent’s private percentile rose from 49.7 to 62.2 across attempts 1–2.The PRA added a task-specific rubric against in-sample calibration; the agent subsequently finished with a Ridge stack at CV 0.073815.Source: M Example PRA Feedback · p. 25
  • A PRA rubric capped image-level salt classification at 0.5228 AP versus approximately 0.88 for U-Net segmentation, and later attempts reached private percentiles of 34.3–36.6.The agent replaced its image classifier with pixel-level segmentation; the rubric passed in all 21 subsequent reviews. Attempt 1 had reached a private percentile of 10.1, and the rubric also caught an out-of-fold prediction mismatch.Source: M Example PRA Feedback · pp. 26–27
  • In obesity-risk prediction, the PRA traced a reported XGBoost accuracy of 0.000000 to a label-prefix bug; the final soft vote reached CV 0.910059 and private percentile 73.3.The agent corrected the accuracy calculation to 0.909240 and fixed its score registry; the private percentile had been 49.4 in attempt 2.Source: M Example PRA Feedback · p. 27
  • At Digit Recognizer’s finish gate, the PRA found OOF accuracy of 0.9899 instead of reported CV 0.9922; after reloading best-epoch checkpoints, the agent finished at OOF accuracy 0.9912.The attempt reached public percentile 87.0, up from 73.3 in attempt 4, but the paper cautions that the gain also reflects an architecture change.Source: M Example PRA Feedback · p. 28
  • A PRA review also caught a ResNet script error: loading each fold’s best weights made reported CV match the OOF score at 0.9912.The agent had been advised to report OOF CV 0.9899 before fixing the weight-loading issue; the corrected run reported categorization accuracy with 5-fold CV of 0.9912.Source: M Example PRA Feedback · p. 28
  • On Titanic, cross-attempt review prompted a diverse vote that raised public accuracy from 0.77033 to 0.77511 and public percentile from 32.8 to 52.4, but validation remained unresolved.The agent did not compute an out-of-fold score for the ensemble and reused the CatBoost score as a proxy despite later PRA reviews flagging the issue.Source: M Example PRA Feedback · pp. 28–29
  • A Ridge stack reached CV 72,590 and private percentile 80.5 on used-car prices after PRA reviews prompted model diversification and validated stacking.The agent first improved on the LightGBM baseline with HistGradientBoosting, then reverted from an unvalidated ensemble before stacking out-of-fold predictions.Source: M Example PRA Feedback · pp. 29–30
  • A 0.45/0.55 CatBoost–CNN blend reached CV 0.240659 and private percentile 52.3 on the iceberg task, outperforming the feature-based attempts’ percentile of 21.8.The PRA used cached out-of-fold predictions to identify the blend; attempt 5 had a lower CV of 0.232 but private percentile 3.7, so attempt 4 remained the best-of-5 result.Source: M Example PRA Feedback · pp. 30–31
  • The concrete-strength example shows adaptation can create an unattainable rubric: its prediction-dispersion threshold conflicted with the metric and was never retired.With CV RMSE about 12.1 and target standard deviation 16.4, the PRA measured a ratio about 0.68; meeting the rubric’s 0.80 threshold would require RMSE near 9.8, and attempted weighting worsened CV from 12.121 to 12.184.Source: M Example PRA Feedback · p. 31
  • Prediction-dispersion checks indicated compression and extreme-value underfitting: high-strength samples (>65) had mean |residual| 25.92, 3.2x the mid-range MAE of 8.08; the weighted model achieved CV 12.18.Source: M Example PRA Feedback · pp. 31–32
  • Across examples, executed checks produced verifiable numbers and prompted fixes, while verified task-specific rubrics persisted as useful checks across attempts.Broader exploration feedback worked best when followed by a check; without validation, agents could comply without justifying their choices, and repeatedly failing rubrics with no CV loss should be retired.Source: M Example PRA Feedback · p. 32

N PRA Cadence

  • Every PRA cadence tested improves over the harness alone on DSPredict-Easy; c=10 performs best, while c=5 interrupts more and c=20 delays feedback.The paper uses c=10 on DSPredict-Easy and c=20 on DSPredict-Hard; at c=20, PRA still improves on the harness alone on Easy.Source: N PRA Cadence · p. 32, Table 19 · p. 40

O Alternative Base Agent

  • 7.8 points: ADAPRA raises DeepSeek-V3.2’s mean percentile on DSPredict-Easy and doubles its above-median rate in single-attempt runs.Across tasks, it wins 25 times, loses 11, and ties twice; the one-sided paired permutation test gives p = 0.005.Source: O Alternative Base Agent · p. 32, Table 20 · p. 40

R Feedback Content in an Earlier Version

Related papers

Loading 2610.33245v1…