Source-linked AI summary
ARISE: Agent Reasoning with Intrinsic Skill Evolution in Hierarchical Reinforcement Learning
Yu Li, Rui Miao, Zhengling Qi, Tian Lan
TL;DR
Existing verifiable-reward methods discard reusable reasoning strategies, limiting how skill knowledge accumulates across mathematical problems. ARISE integrates skill management and response generation in a shared hierarchical policy with an evolving library and hierarchical rewards. Across two base models and seven benchmarks, it consistently outperforms GRPO-family and memory-augmented baselines, while policy improvement and library enrichment improve in tandem.
Problem
Existing methods treat problem instances separately and decouple skill selection and library updates from the reinforcement-learning objective, discarding successful reasoning strategies after rollouts.
Method
ARISE uses a shared Manager-Worker policy, policy-driven skill selection, structured skill generation, an endogenous evolving-skill MDP, and hierarchical rewards to jointly optimize reasoning and the skill library.
Results
ARISE consistently outperforms GRPO-family algorithms and memory-augmented baselines across two base models and seven competition and Omni-MATH benchmarks.
Takeaways & Limitations
The results indicate that library quality and reasoning performance co-evolve, with later accuracy gains coming from improved selection of existing skills after library size saturates.
Abstract
from arXiv · showhide
The dominant paradigm for improving mathematical reasoning in language models relies on Reinforcement Learning with verifiable rewards. Yet existing methods treat each problem instance in isolation without leveraging the reusable strategies that emerge and accumulate during training. To this end, we introduce ARISE (Agent Reasoning via Intrinsic Skill Evolution), a hierarchical reinforcement learning framework, in which a shared policy operates both to manage skills at high-level and to generate responses at low-level (denoted as a Skills Manager and a Worker, respectively). The Manager maintains a tiered skill library through a dedicated skill generation rollout that performs structured summarization of successful solution traces (after execution), while employing a policy-driven selection mechanism to retrieve relevant skills to condition future rollouts (before execution). A hierarchical reward design guides the co-evolution of reasoning ability and library quality. Experiments on two base models and seven benchmarks spanning both competition mathematics and Omni-MATH show that ARISE consistently outperforms GRPO-family algorithms and memory-augmented baselines, with particularly notable gains on out-of-distribution tasks. Ablation studies confirm that each component contributes to the observed improvements and that library quality and reasoning performance improve in tandem throughout training. Code is available at \href{https://github.com/Skylanding/ARISE}{https://github.com/Skylanding/ARISE}.
1 Introduction
ARISE treats reusable reasoning skills as an intrinsic, evolving component of hierarchical reinforcement learning rather than discarding successful strategies after each rollout. Its shared Manager-Worker policy, structured skill library, policy-driven selection, and hierarchical rewards jointly improve mathematical reasoning across models and benchmarks.
- Existing verifiable-reward methods discard successful reasoning strategies after each problem instance instead of retaining them for future use.
- ARISE uses a shared policy as both Skills Manager and Worker, selecting skills before execution and generating solution traces conditioned on them.
- The Evolving-Skill MDP models the skill library as endogenous state, enabling joint optimization of policy and library under one reinforcement-learning objective.
- Policy-driven selection uses conditional log-probability scoring so policy gradients directly shape skill preferences without an external retriever.
- A hierarchical reward and two-tier library architecture support co-evolution between policy improvement and library enrichment.
- ARISE consistently outperforms vanilla GRPO variants and skill-augmented baselines across competition and Olympiad-level benchmarks on two base models.
2 Backgrounds
Prior work improves mathematical reasoning with verifiable rewards or external memory, but these approaches generally separate policy learning from memory and skill management. ARISE addresses this separation by integrating evolving skills into hierarchical policy optimization.
- GRPO enables critic-free policy optimization by estimating advantages from normalized rewards within groups of rollouts for each query.
- External-memory methods retrieve trajectories or distilled knowledge into context, but their memories are populated independently of policy learning and remain fixed once constructed.
3 ARISE: Hierarchical Agent RL with Evolving Skills
ARISE integrates skill selection, solution generation, and skill evolution within a shared-policy Manager-Worker hierarchy. Its evolving library, hierarchical rewards, and two-phase training jointly couple policy optimization with reusable-skill management.
- Evolving-Skill MDP: ARISE models the skill library as part of the agent’s state through an Evolving-Skill MDP, enabling joint optimization of policy and library.The augmented state includes the query and current library configuration, while library actions change future states.
- Library Management: The two-tier library uses a compact cache for active selection and a reservoir for surplus skills, maintained by ADD, UPDATE, EVICT, LOAD, and DELETE.New skills enter through Upload, selected skills reach workers through Download, and utilities are updated with an exponential moving average.
- Manager-Worker Hierarchy: A shared policy πθ selects skills before execution, generates solution traces, and summarizes successful traces into new skill documents afterward.The Manager handles selection and generation, while the Worker produces the solution trace.
- Skill Selection: The Manager scores candidate skills with conditional log-probabilities and uses the resulting selection distribution to choose skills for worker rollouts.A confidence gate admits the selected skill only when its maximum selection probability reaches δ; otherwise the worker solves unaided.
- Skill Generation: Successful positive-advantage traces are converted into structured skill documents by a dedicated skill-generation rollout beyond the ordinary solution rollouts.This grounds skill induction in concrete successful solutions rather than open-ended strategy generation.
- Hierarchical Reward and Training: The hierarchical reward assigns r2 > r1 > r0 to correct skill-augmented, correct unaugmented, and incorrect solutions, respectively.Training begins with warm-up GRPO using binary task rewards, then activates selection and the full hierarchical pipeline in Phase II.
4 Results
ARISE is evaluated across two instruction-tuned models, in-distribution competition benchmarks, and out-of-distribution Omni-MATH. It consistently outperforms GRPO-family and memory- or skill-augmented baselines, with especially large gains on Omni-MATH and measurable dependence on its hierarchical components.
- Main Results: ARISE achieves the highest scores across every benchmark for both Qwen3-4B-Instruct-2507 and Phi-4-mini-instruct.The evaluation uses average Pass@1 over 32 runs, with all methods trained on DeepScaleR under the same group size and computational budget.
- Out-of-Distribution Generalization: 2.9 and 1.9 points are ARISE’s Omni-MATH average-accuracy gains over GRPO on the two base models, respectively.Gains occur across all four Omni-MATH domains, with the largest improvements in Algebra.
- Baseline Comparisons: ARISE outperforms memory-augmented methods through end-to-end policy-driven skill selection, whereas EvolveR and SimpleMem do not consistently surpass DAPO or GSPO.The comparison attributes ARISE’s larger gains to advantage signals shaping which skills are retrieved and retained.
- Ablation Study: Binary reward reduces skill utilization from 73% to 31% and causes the largest accuracy drop in the ablation study.Random injection preserves utilization but harms accuracy through mismatched context, while removing skill generation freezes the library at 24 seed skills.
- Training Dynamics: Library growth and accuracy improvement widen together after Phase II, while later accuracy gains continue after library size saturates.This pattern indicates that later improvements come from more effective selection of existing skills rather than continued accumulation of new ones.
5 Conclusion
ARISE improves mathematical reasoning while its skill library and policy co-evolve during training. The conclusion identifies library curation and broader agent-task evaluation as important next steps.
- ARISE unifies skill selection, generation, and policy optimization under one shared policy, allowing the library to co-evolve with the agent.
- Experiments on two base models across seven benchmarks show consistent improvements over GRPO-family baselines and memory-augmented methods.
- Ablations identify hierarchical reward as the primary driver of skill adoption, while late-stage accuracy gains come from better selection of existing skills after library growth saturates.
- ARISE is evaluated exclusively on mathematical reasoning, leaving multi-tool agent tasks and code generation as future evaluation directions.
A.1 Training Configuration and Hyperparameters
Training uses a two-phase GRPO-based procedure: a binary-reward warm-up builds the initial library, followed by hierarchical skill selection, rollout, updating, and library maintenance.
- Training configuration: Training uses the DeepScaleR dataset for 2,000 optimization steps with GRPO, group size G=8, effective batch size 64, and AdamW learning rate 1×10−6.
- Training phases: Phase I spans the first Nw=500 steps with binary reward, while Phase II begins at step 501 with hierarchical reward and full skill selection.
- Training loop: Each step samples queries, selects a cache skill in Phase II when confidence exceeds δ, prepends it to the prompt, and generates G trajectories.
- Optimization: Dynamic sampling removes zero-variance groups before advantage estimation, after which group-relative advantages and GRPO updates train the policy.
- Skill evolution: Positive-advantage traces produce a new skill through the OG+1 summary rollout, followed by utility updates and cache-reservoir maintenance operations.
- Training phases: Skill generation and library maintenance run in both phases, ensuring the library is populated before Phase II selection begins.
A.3 Evaluation Protocol
Evaluation averages greedy Pass@1 accuracy over 32 independent runs, using the complete ARISE pipeline and benchmark-specific answer verification.
- Evaluation procedure: All results report average Pass@1 accuracy over 32 independent runs, with one greedy-decoded solution generated per problem.
- Evaluation procedure: At evaluation time, the manager selects a cache skill with ε=0, while the worker conditions on the augmented prompt or solves unaided when confidence is below δ.
- Answer verification: In-distribution AMC and AIME answers use exact numerical matching, whereas Omni-MATH uses symbolic equivalence checking across four domains.
- Formal evaluation context: The ES-MDP state combines the independently sampled query qt with the endogenous skill library Mt shaped by preceding library actions.
- Formal evaluation context: Library transitions are deterministic given management actions and a validated skill, while stochasticity enters through policy-generated skills.
- Hierarchical reward: When the group mean reward exceeds one, correct but unaugmented solutions receive negative advantage, strengthening the signal for skill utilization.
C Skill Library Architecture
ARISE stores reusable reasoning strategies as uniformly formatted documents in a two-tier cache-reservoir library. Skills are generated from successful traces, validated, scored, selected, and maintained through utility-based operations.
- Skill documents: Each skill is a JSON document with a uniform five-field schema, designed so log-probability scoring reflects semantic relevance rather than format variation.
- Seed skills: The library begins with five seed skills encoding generic mathematical heuristics, providing non-trivial selection candidates when Phase II starts.
- Two-tier architecture: The active cache holds 10 skills, while a 100-entry reservoir archives surplus skills for possible future promotion.
- Library operations: Library maintenance applies UPDATE, ADD, EVICT, LOAD, and DELETE operations to adjust utilities, insert skills, and manage cache-reservoir movement.
- Library operations: The fixed operation order is UPDATE → ADD → EVICT → LOAD → DELETE, so new insertions can trigger eviction before reservoir loading.
- Skill generation: After rollouts, OG+1 summarizes positive-advantage traces into reusable documents, which pass extraction, parsing, truncation, and insertion validation.
- Skill generation: The primary summarization path succeeds in over 85% of attempts after warm-up, with a fallback constructing a minimal skill from the first successful trace when validation fails.
D.3 Information Gain Metric
ARISE uses information gain alongside EMA utility to assess whether skills contribute causally to successful reasoning rather than merely correlating with easy problems.
- Information gain measures how much conditioning on a skill increases the policy’s confidence on known correct reasoning solutions.Exact computation requires two forward passes per skill–trace pair, so ARISE maintains an online proxy and computes exact IG every 100 steps.
- The proxy compares skill-conditioned and global running reward and success-rate statistics to estimate a skill’s contribution.The corresponding statistics are maintained for each skill and globally.
- High utility with low or negative information gain identifies skills associated with easy problems rather than skills causally contributing to solutions.Utility tracks cumulative reward, whereas information gain isolates the difference made by conditioning on the skill.
E.1 Hyperparameter Sensitivity
ARISE is sensitive to skill-selection, exploration, cache-size, and warm-up choices, with default settings balancing utilization, precision, repertoire, and training timing.
- Confidence gate δ: At δ=0.15, nearly 90% of rollouts receive skills but accuracy falls by 1.2 points; at δ=0.55, utilization drops to 42% and accuracy trails the full model by 1.4 points.The default δ=0.35 injects skills in 73% of rollouts.
- Exploration rate ε: Performance remains robust for ε ∈[0.05, 0.15], while ε=0 misses under-explored skills and ε>0.2 increasingly injects irrelevant context.The sensitivity reflects a trade-off between exploiting known skills and exploring recently added entries.
- Cache size Cc: A cache size of 10 balances skill diversity and scoring cost; size 5 restricts the repertoire, while size 50 dilutes the active pool and raises scoring overhead 5× without commensurate gains.The larger cache increases the number of low-utility entries considered during selection.
- Warm-up length Nw: Nw=200 produces the largest accuracy deficit (−1.6 points), whereas Nw=1000 delays co-evolution; Nw=500 builds a sufficient policy and approximately 50–80 skills.The shorter warm-up begins Phase II with both a weak policy and an underpopulated library.
- Skill library evolution: The library progresses from broad seed-like heuristics to specialized Number Theory and Combinatorics techniques, then to targeted methods such as Vieta’s formulas and inscribed angle chasing.This specialization is presented as a qualitative explanation for disproportionately large out-of-distribution gains.
- Skill library evolution: Library size plateaus around step 1,500 while accuracy continues rising, indicating that late gains come from better selection over existing skills.LOAD/EVICT curates the active cache by promoting specialized skills and retiring redundant ones.
- Phase transition: Skill utilization rises from 0% to approximately 40–50% during the first 10–20 Phase II steps, then stabilizes as the confidence gate filters low-relevance injections.The initial surge is driven primarily by ε-greedy exploration before the policy learns to discriminate among candidates.
- Phase transition: The hierarchical reward creates a three-level advantage landscape whose mixed R=0, R=1, and R=2 groups provide stronger learning signals than binary rewards.Higher reward variance amplifies advantage magnitudes per gradient step.
E.4 Token Overhead Analysis
ARISE adds modest prompt overhead while producing a favorable accuracy-to-overhead trade-off and leaving solution-response length comparable to GRPO.
- ARISE adds an average of 66 prompt tokens per problem and gains 2.9 accuracy points on Omni-MATH over GRPO.The injected skill document is constrained by a 220-character hard cap.
- ARISE achieves a better accuracy-to-overhead trade-off than EvolveR, which incurs 7.9% overhead for a 0.7-point Omni-MATH gain.EvolveR uses longer experience narratives than ARISE’s injected skill documents.
- ARISE’s 2,876-token response length is comparable to vanilla GRPO’s 2,847 tokens, indicating that skill injection does not inflate solution length.The injected skill guides reasoning strategy without encouraging more verbose output.
E.5 Computational Cost Breakdown
ARISE incurs a 16.6% per-step training overhead, dominated by skill generation and scoring, while library maintenance remains negligible.
- The skill-generation rollout costs 4.8 seconds, or 9.4% of total step time, and is the dominant overhead component.It generates one sequence of up to 192 tokens.
- Skill scoring adds 2.4 seconds, or 4.7%, through ten forward passes for log-probability computation.The scoring cost arises from evaluating the cached skills.
- The total per-step overhead is 16.6%, while library maintenance contributes less than 0.2%.The reported overhead is considered modest relative to the observed accuracy gains, with batching and caching proposed as further reductions.