ARISE: Adapting to Evolving Capability Gaps in Agentic Reinforcement Learning

1 INTRODUCTION

• Task-level rewards can mask shifting bottlenecks, while identical group rewards yield zero advantage despite differing intermediate behavior.Consistently satisfied criteria can also become uninformative, leaving out-of-scope weaknesses unmeasured.Source: §1 · p. 1
• ARISE turns rollout evidence into behavioral requirements, each represented by a rubric for partial progress and a paired skill for action guidance.Low rubric pass rates trigger skill activation or refinement; consistently high rates indicate mastery and allow criteria to be retired.Source: §1 · pp. 1–2
• A capability-based adaptive sampler selects tasks using rubric-based estimates of performance within each task type.This accounts for behaviors that may be reliable in one task type but remain difficult in another.Source: §1 · p. 2
• ARISE couples capability identification, targeted exploration, fine-grained behavioral feedback, and adaptive task sampling.Its requirements and training priorities are updated as rollout evidence reveals emerging bottlenecks.Source: §1 · p. 2
• ARISE reports substantial, efficient gains across two agent benchmarks and outperforms same-scale baselines on both.On SkillsBench, it also improves over Qwen3.5-27B and performs competitively with substantially larger models.Source: §1 · p. 2, Figure 1 · p. 2

2 RELATED WORK

• OnlineRubrics (Rezaei et al., 2026) and DR Tulu (Shao et al., 2026) adapt criteria, while RuscaRL (Zhou et al., 2026) uses fixed criteria and a predefined guidance-decay schedule.ARISE evaluates intermediate behaviors and revises active criteria as evidence reveals new or mastered requirements.Source: §2 · pp. 2–3
• Voyager (Wang et al., 2024) and ExpeL (Zhao et al., 2024) reuse experience with frozen policy parameters, while SkillRL evolves a hierarchical skill library.ARISE uses paired behavioral evaluations to decide when guidance is needed and refine skills when weaknesses persist.Source: §2 · p. 3
• GoalGAN (Florensa et al., 2018), DAPO (Yu et al., 2025), and VADE select or filter tasks using outcome-level success or reward signals.ARISE instead uses rubric evaluations to identify which behaviors remain weak within each task type.Source: §2 · p. 3

3.1 PROBLEM FORMULATION

• The agent samples actions from a policy over interaction histories to produce task trajectories, and learning seeks to improve expected task performance.A task-specific verifier supplies the task-completion reward.Source: §3.1 · p. 3

3.2 OVERALL FRAMEWORK

• ARISE combines rubric–skill co-evolution with capability-based adaptive sampling to determine behavioral feedback, guidance, and training tasks.Figure 2 shows sampled tasks generating rollouts with selectively activated skills; rubric evaluations inform policy optimization and, with rollout evidence, subsequent rubric–skill evolution and sampling.Source: §3.2 · p. 3, Figure 2 · p. 4

3.2.1 RUBRIC–SKILL CO-EVOLUTION

• Rubric–skill pairs encode observable behavioral requirements, with trajectory-based pass rates guiding skill activation, refinement, and rubric retirement.Recent rollout evidence supports new pairs; consistently satisfied rubrics are retired, and persistent failures despite active guidance prompt skill refinement.Source: §3.2.1 · pp. 4–5

3.2.2 CAPABILITY-BASED ADAPTIVE SAMPLING

• Capability-based sampling estimates rubric-based performance within task types, uses coverage-guided discovery, and prioritizes task-type–capability pairs with intermediate pass probabilities.A task is sampled uniformly from the selected pair’s available members; evidence for unchanged rubrics is preserved, obsolete contributions discarded, and new criteria receive fresh discovery needs.Source: §3.2.2 · pp. 5–6

3.3 TRAINING STRATEGY

• Policy optimization combines separately normalized task rewards and rubric verdicts into a trajectory advantage, with task-only fallback when usable rubric feedback is absent.The combined advantage is used for GRPO-style optimization, applying trajectory advantages to model-generated tokens while excluding tool outputs and environment observations.Source: §3.3 · p. 6

4.1 EXPERIMENT SETTINGS

• Training uses 1,728 tasks; evaluation covers SkillsBench v1.1 and Terminal-Bench v2.1, with evolved skills withheld and own pass rates averaged over three runs.The reported baselines include comparable-scale models and substantially larger models.Source: §4.1 · pp. 6–7

4.2 MAIN RESULTS

• ARISE achieves the highest overall pass rates among comparable-scale models on both benchmarks, surpasses larger Qwen3.5 variants on SkillsBench, and remains competitive with them on Terminal-Bench.The SkillsBench advantage is consistent across three evaluation runs.Source: Table 1 · p. 7, §4.2 · pp. 7–8

4.3 MODEL ANALYSIS

• By step 50, ARISE reaches outcome-only RL’s step-150 pass rate under the same per-step rollout budget, then continues improving.This comparison uses pass rates over training on SkillsBench v1.1.Source: Figure 3 · p. 8, §4.3 · p. 9
• ARISE outperforms five matched-budget ablations; frozen full-lifecycle rubrics, removing skill guidance, and VADE sampling each fall short.The study also compares adaptive sampling with non-adaptive sampling and outcome-only RL.Source: §4.3 · p. 8, Figure 3 · p. 8
• 36 criteria were added and 21 retired, leaving 15 active; ARISE also had lower all-failure rollout rates than the no-skill variant across training stages.The lower rates were most pronounced early in training.Source: Figure 4 · p. 8, §4.3 · p. 9
• Sampling priorities shift from execution and verification early to debugging and efficiency later; blank early cells still receive discovery sampling.The probability figure excludes discovery sampling, and blank cells reflect an initially empty pool and limited evaluation coverage.Source: §4.3 · p. 9, Figure 5 · p. 9

• ARISE adapts the learning process to evolving capability gaps by turning rollout evidence into evolving rubrics and paired skills.The framework connects behavioral weakness identification with targeted exploration and task selection.Source: §5 · p. 9
• Across SkillsBench and Terminal-Bench, ARISE improves over evaluated comparable-scale baselines, competes with substantially larger models, and transfers across agent harnesses.Source: §5 · p. 9
• The analysis reports improved training efficiency under a matched per-step rollout budget, supporting continual adaptation of behavioral feedback and learning opportunities.Source: §5 · p. 9

• Training exclusively uses Qwen3.5-27B, leaving effectiveness across other model scales and architectures unverified.Source: A LIMITATIONS · p. 16
• Evaluation is limited to SkillsBench and Terminal-Bench, leaving generalization to diverse agent environments and real-world workflows for future work.Source: A LIMITATIONS · p. 16
• Rubric generation and evaluation rely heavily on reflection and judge models, whose errors or biases may compromise behavioral supervision and training decisions.Source: A LIMITATIONS · p. 16

B.1 TRAINING PROCEDURE

• ARISE initializes an empty rubric–skill pool, uses rollout diagnostics and judgments to update the policy, then periodically refines, hides, retires, and generates pairs as rubric pass rates change.New skills begin hidden, and unchanged rubric evidence is preserved while obsolete contributions are discarded.Source: §B.1 · p. 16

B.2 MODEL CONFIGURATIONS

• Training initializes Qwen3.5-27B with AReaL 2.0, uses fixed DeepSeek-V4-Flash for judging and reflection, and runs on 32 NVIDIA H800 GPUs for approximately two days.AReaL 2.0 is described as an asynchronous reinforcement learning framework with decoupled agent execution, inference, and policy optimization services.Source: §B.2 · p. 17

B.3 ROLLOUT EVIDENCE

• Rollout evidence includes interaction traces and verifier outcomes, while pool updates select up to 256 records from the preceding ten steps and provide compressed diagnostics rather than full trajectories or per-rollout reward labels.Issue and contrast evidence support new rubrics, while failed-task evidence and outcome context guide skill refinement.Source: §B.3 · p. 17

B.4 POLICY OPTIMIZATION

• Policy optimization samples eight trajectories per task with group-relative reward normalization, applies conditional length regularization, and rejects out-of-range importance-ratio tokens rather than clipping them.Length regularization is active only for mixed positive and non-positive reward groups meeting the length threshold; the objective uses dual clipping for negative advantages and no reference-policy KL penalty.Source: §B.4 · pp. 17–18

B.5 ADAPTIVE SAMPLING

• Adaptive sampling mixes discovery with capability-targeted selection, limits each target pair to eight adaptive picks per batch, and falls back to inverse-attempt weighting when adaptive selection is unavailable.With batch size B = 32 and quota fraction b = 0.25, discovery picks are exempt from the per-pair quota.Source: §B.5 · p. 19

B.6 PROMPT TEMPLATES

• Prompts bootstrap diagnostics from individual rollouts, generate paired rubrics and skills from contrasting behavioral evidence without per-trajectory reward labels, and evaluate applicability and verdicts against interaction traces.The evaluation prompt compares uncovered behaviors with the complete active pool; verifier outcomes are weak context and cannot alone ground a verdict.Source: §B.6 · pp. 19, 20, 22
• Skill-refinement prompts use failed-rollout evidence to make guidance more operational while preserving the paired rubric’s behavioral requirement.They ask for checks, decision points, and verification discipline, while restricting guidance to task-visible evidence and excluding hidden evaluator internals or sample-specific answers.Source: §B.6 · pp. 23–24
• Offline task-type annotation matches tasks to registry tags using explicit task-flow requirements and observable outcomes, with a focused second pass restricted to nearby candidate tags.The prompt rejects matches based only on likely solver actions, domain resemblance, or a downstream process described in an artifact the solver is asked to create.Source: §B.6 · p. 24

C.1 TRAINING CORPUS

• The training corpus contains 1,728 executable tasks across seven domains and excludes SkillsBench, Terminal-Bench, and paraphrased variants.Each task includes an instruction, task-specific resources, a containerized environment, and a verifier; the policy generates trajectories online during reinforcement learning.Source: §C.1 · p. 25

C.2 TASK-TYPE ANNOTATION

• The predefined taxonomy contains 82 task types defined by operations or workflows, not application domain or agent behavior quality.Source: §C.2 · p. 25
• GLM-5.2 assigns task types offline from instructions, available skills, and artifact paths, excluding hidden tests, reference solutions, and rollout trajectories.Matches are grounded in required operations and observable results, distinguishing workflow execution from merely describing a workflow.Source: §C.2 · p. 25
• The training-corpus distribution by broad domain is summarized in Table 4.Source: Table 4 · p. 26
• Annotations cover 67 task types across the corpus, allow multiple tags per task, and remain fixed as rubric evaluations update behavioral evidence.Source: §C.2 · p. 26
• Table 5 presents representative tasks and annotations, illustrating shared task types across domains, multilabel assignments, and workflow execution versus synthesis.Source: Table 5 · p. 26, §C.2 · p. 26

D.1 BENCHMARKS

• SkillsBench v1.1 contains 87 expertise-intensive tasks across eight domains, evaluated with official verifiers in containerized environments.Source: §D.1 · p. 26
• SkillsBench execution uses Kilo Code with each task’s expert-curated skills directory available, distinct from ARISE-evolved skills.ARISE-evolved skills are training-time guidance and are not provided on either evaluation benchmark.Source: §D.1 · pp. 26–27
• Terminal-Bench v2.1 contains 89 containerized tasks across software engineering, system administration, data processing, and scientific computing, with success verified from final task state.Tasks include programmatic tests and a time limit; evaluation uses Terminus-2 under official settings.Source: §D.1 · p. 27

D.2 EVALUATION SETTINGS

• SkillsBench evaluations use model-native context limits, specified generation settings, up to 32,768 output tokens, and a 10,000-second task limit; interactive tools are disabled.Terminal-Bench uses comparable-scale models under official settings, with official leaderboard scores for remaining models.Source: §D.2 · p. 27

D.3 BASELINE CONFIGURATIONS

• ARISE and training-method baselines share the Qwen3.5-27B initialization, training data, and policy-training budget.The baselines adapt VADE sampling, OnlineRubrics criteria elicitation, and a fixed RuscaRL rubric pool; other listed schedules and weights follow their original papers.Source: §D.3 · p. 27

D.4 METRICS

• Task pass rate is the percentage of verifier-confirmed successful attempts averaged across tasks and repeated runs.Source: §D.4 · p. 27
• Both benchmarks use three attempts per task, pooling all tasks for the overall rate rather than equally averaging SkillsBench’s eight domain rates.This yields 261 SkillsBench attempts and 267 Terminal-Bench attempts; attempts deemed unsuccessful under the official protocol contribute zero.Source: §D.4 · p. 27

E EVALUATION STABILITY

• Across three runs on the same 87 tasks, ARISE’s lowest pass rate exceeds every comparable-scale baseline’s highest pass rate.Table 6 reports individual run rates, their mean, and sample standard deviation.Source: E EVALUATION STABILITY · pp. 27–28, Table 6 · p. 28

F RUBRIC–SKILL EVOLUTION CASES

• The four cases illustrate activating then hiding guidance, retiring a pair after hiding guidance, and retaining or retiring rubrics while skills stay hidden.They are drawn from the first 150 training steps; reported pass rates summarize applicable rubric evaluations within each update window.Source: F RUBRIC–SKILL EVOLUTION CASES · p. 28, Table 7 · p. 28, Table 8 · p. 29

G BEHAVIORAL RETENTION AFTER RETIREMENT

• All three rubrics maintain pass rates above 90% at both later checkpoints, with individual rates fluctuating.The fixed task sets comprise tasks sampled during the 10-step window ending at retirement; later evaluations use fixed rubrics and no ARISE-generated skill guidance.Source: G BEHAVIORAL RETENTION AFTER RETIREMENT · p. 30, Figure 6 · p. 30
• The three Figure 6 panels show rubrics retiring at steps 60, 50, and 80, with training-window rates compared against fixed-task evaluations.The later evaluations are at steps 100 and 150.Source: Figure 6 · p. 30, G BEHAVIORAL RETENTION AFTER RETIREMENT · p. 30

H TRAINING TIME ANALYSIS

• ARISE reaches comparable performance in one-third as many training steps, implying an estimated 65% reduction in training time despite approximately 4.2% higher average per-step time.The comparison uses matched hardware and rollout budgets.Source: H TRAINING TIME ANALYSIS · p. 30, Table 11 · p. 30