Source-linked AI summary
ARISE-RL: Agentic Rubric-Grounded Iterative Self-Evolution with Reinforcement Learning
Fanrui Zhang, Ruixue Ding, Qiang Zhang, Xi Chen, Boli Chen, Shihang Wang, Qiuchen Wang, Hongmin Zhan, Jinxin Bian, Li xingchao, Peijin Zheng, Hao cheng, Pengjun Xie, Kaipeng Zhang, Jiawei Liu, Zheng-Jun Zha
TL;DR
Open-ended agentic RL lacks verifiable gold answers, scalable rubrics, and stable long-horizon rewards. ARISE-RL couples a rubric-grounded Generator and Solver with reward-gated same-policy distillation, and reports average state-of-the-art performance across four benchmarks while remaining evaluated only on 8/9B backbones.
Problem
Open-ended agentic tasks lack a single gold standard, scalable expert rubrics, and stable rewards for fine-grained reinforcement-learning optimization.
Method
ARISE-RL uses rubric-mediated Generator–Solver co-evolution with tool-grounded criteria, capability-frontier task generation, and reward-gated distillation from a memory-augmented variant of the same policy.
Results
ARISE-RL achieves average state-of-the-art performance across four benchmarks with a 9B open-source backbone, with gains reported on interactive multi-tool and rubric-based research benchmarks.
Takeaways & Limitations
The results support unified closed-loop self-evolution as a promising approach for scalable open-ended agent training.
Takeaways & Limitations
Experiments use only 8/9B open-source backbones, so scalability to substantially larger models remains unverified.
Abstract
from arXiv · showhide
Training open-ended agents via reinforcement learning (RL) is hindered by the lack of verifiable gold answers and scalable rubrics. Moreover, even near the model's capability boundary, long-horizon open-ended agentic tasks often yield brittle and unstable rewards, resulting in weak or noisy rollout contrast that obscures fine-grained optimization signals for group-based policy learning. To address these challenges, we propose ARISE-RL, a novel full-cycle self-evolution framework that couples a task/rubric Generator and a reasoning Solver through rubric-mediated co-evolution. The Generator grounds tool-related rubric criteria in real tool observations and is rewarded for producing valid, intermediate-difficulty tasks aligned with the Solver's evolving capability boundary. The Solver, in turn, learns from fine-grained rubric satisfaction signals through multi-step reasoning and tool use. We further introduce Reward-Gated Self-Evolution Distillation (RG-SED), which selectively distills a memory-augmented variant of the same policy back into itself only when the memory yields empirical reward improvement, thereby reducing distribution mismatch and avoiding blind imitation of noisy guidance. Finally, to support rigorous evaluation, we present ECR-Bench, an expert-calibrated rubric benchmark suite covering single-tool deep research and multi-tool travel planning. Extensive experiments demonstrate that ARISE-RL consistently achieves robust and stable overall state-of-the-art performance across all evaluated benchmarks.
1 Introduction
Open-ended agentic RL lacks scalable data, expert rubrics, and stable fine-grained rewards. ARISE-RL addresses these constraints through rubric-mediated Generator–Solver co-evolution, reward-gated self-distillation, and expert-calibrated evaluation.
- Motivation: Open-ended tasks require multidimensional criteria such as factual grounding, constraint satisfaction, reasoning coherence, and utility rather than a single gold answer.
- Motivation: High-quality open-ended training data and expert rubrics are expensive to scale, while existing open-ended suites mainly support static post-hoc evaluation.
- Motivation: Long-horizon tool interactions produce brittle, unstable rewards and weak rollout contrast, limiting fine-grained policy optimization near the capability boundary.
- ARISE-RL: ARISE-RL co-evolves a Generator and Solver through tool-grounded rubrics, with difficulty-shaped rewards keeping generated tasks near the Solver’s capability frontier.
- ARISE-RL: RG-SED distills a transient, memory-augmented teacher from the same policy only when its rollouts empirically improve reward.
- Evaluation: ECR-Bench evaluates open-ended agents on expert-calibrated single-tool research and multi-tool travel planning, including final quality and process-level tool-use correctness.
2 Related Work
Prior self-evolution and distillation methods address parts of agent improvement but remain limited by task scope, teacher mismatch, or ungated guidance. ARISE-RL combines open-ended tool grounding with reward-gated same-policy distillation.
- Self-Evolving Agents: Prior self-evolution frameworks mainly target closed-form, single-tool, or reasoning-centric settings rather than open-ended tasks requiring tool grounding and rubric evaluation.
- Off-Policy Distillation and Self-Distillation: Fixed-teacher distillation can compound importance-sampling mismatch across tool-use turns, impose a static ceiling, and provide no mechanism for rejecting noisy guidance.
- Off-Policy Distillation and Self-Distillation: RG-SED constructs a teacher from the same policy with query-specific coach memory and distills only after coach-augmented rollouts improve empirical reward.
3 Method
ARISE-RL trains open-ended agents through a rubric-mediated loop in which a Generator creates tool-grounded tasks and a Solver learns from rubric-based rewards. RG-SED selectively distills reward-improving, memory-augmented behavior into both roles to improve self-evolution stability.
- Rubric-mediated co-evolution: ARISE-RL couples task generation, rubric construction, and Solver policy learning in a self-evolving loop near the Solver’s capability boundary.The Generator creates queries and rubrics, while the Solver uses multi-step reasoning and tools under rubric-mediated rewards.
- Rubric-mediated co-evolution: The generated rubric provides fine-grained evaluation criteria, process-aware Solver supervision, and tool-grounded feedback for Generator training.
- Generator reward: Generator rewards combine tool-grounding and format-validity gates with a difficulty reward based on repeated Solver success rates.The tool-grounding gate requires observing real tool output before specifying tool-dependent criteria; the format gate ensures downstream executability.
- Generator reward: The difficulty reward peaks when approximately half of Solver rollouts succeed and falls to zero when all succeed or fail, targeting informative intermediate-difficulty tasks.
- Solver reward: Solver rewards combine partial rubric satisfaction with a full-completion bonus, providing fine-grained supervision while incentivizing satisfaction of all criteria.The same empirical Solver reward is reused as the Generator’s task-difficulty signal.
- Reward-Gated Self-Evolution Distillation: RG-SED addresses sparse or delayed feedback and weak updates by applying role-conditioned self-distillation to both Generator and Solver.A coach produces role-specific memory, and the same policy uses that memory to form a transient enhanced teacher.
- Reward-Gated Self-Evolution Distillation: The memory-augmented teacher shares parameters with the student, constraining distributional mismatch without relying on an external teacher.
- Reward-Gated Self-Evolution Distillation: RG-SED activates only after empirical reward improvement and then applies token-level reverse KL on standard on-policy trajectories.This gating avoids blind imitation of noisy guidance and is reported to improve self-evolution stability and efficiency.
4 ECR-Bench
ECR-Bench is an expert-calibrated suite for evaluating open-ended agents in single-tool deep research and multi-tool travel planning. Its manually reviewed rubrics and quality-control procedures target reliable, fine-grained assessment of final responses and tool-use processes.
- Benchmark scope: ECR-Bench covers ECR-DeepResearch for single-tool research and ECR-Travel for multi-tool travel planning.
- Evaluation metrics: Table 1 reports rubric score rate for ResearchRubrics and ECR-DeepResearch, task pass rate for VitaBench and ECR-Travel, and an average across four benchmarks.VitaBench and ECR-Travel are aggregated by averaging their corresponding sub-task scores.
- Quality control: All test queries and rubrics undergo manual review for clarity, difficulty, specificity, coverage, consistency, and reliability, with failed items revised or removed.A rule-augmented LLM checker additionally filters formatting errors, logical inconsistencies, invalid tool calls, dialogue errors, and inconsistent final answers.
- ECR-DeepResearch: ECR-DeepResearch contains 100 test queries with rubrics assessing factual grounding, evidence coverage, reasoning quality, completeness, and report structure.Its main metric is the rubric score ratio.
5 Experiments
ARISE-RL is evaluated across four open-ended agentic benchmarks, including the proposed ECR-Bench suite and existing rubric-based benchmarks. Results show strong overall performance, monotonic gains across co-evolution cycles, and evidence that reward gating and increasingly difficult generated queries support training.
- Experimental Settings: ARISE-RL is evaluated on four benchmarks spanning single-tool research and multi-tool planning, including ECR-DeepResearch, ECR-Travel, ResearchRubrics, and VitaBench.ECR-Travel covers route planning, transportation comparison, nearby POI search, and one-day and multi-day itinerary planning.
- Further Analysis: Each of three co-evolution cycles produces monotone Solver improvement across all four benchmarks, with the strongest gains on the two multi-tool benchmarks.The Generator correspondingly produces progressively harder yet valid queries as the Solver improves.
- Main Results: ARISE-RL consistently achieves the best overall performance, with the largest gains on interactive multi-tool benchmarks and column-wise best results on DeepResearch-style benchmarks.The advantage persists on both Qwen3.5-9B and Qwen3-8B backbones and exceeds contemporary self-evolution baselines under the same backbone.
- Further Analysis: Removing RG-SED causes the largest ablation drop on multi-tool benchmarks, while removing reward gating substantially degrades performance and can amplify noisy supervision.Tool-grounded rubrics and difficulty-shaped rewards also provide consistent, complementary gains.
- Further Analysis: On both multi-tool benchmarks, the frozen Qwen3.5-9B base policy’s single-shot pass rate decreases monotonically across cycles, indicating increasingly challenging generated queries.This behavior is consistent with the Generator inducing a curriculum aligned with the Solver’s improving capability.
- Further Analysis: 72.1% of ECR-Travel training groups show a reliable positive reward gap, while 8.3% are negative and 19.6% ambiguous, motivating selective gate activation.The reward gate activates distillation only when coach-augmented rollouts empirically improve reward, avoiding neutral or harmful supervision.
6 Conclusion
The paper presents ARISE-RL, RG-SED, and ECR-Bench as a unified approach to open-ended agent training and evaluation. Across four benchmarks, ARISE-RL achieves average state-of-the-art performance with a 9B open-source backbone.
- Conclusion: ARISE-RL couples a Generator and Solver through rubric-mediated co-evolution, while RG-SED distills memory-augmented behavior only when it empirically improves reward.ECR-Bench provides expert-calibrated evaluation for multi-tool agents.
- Conclusion: Across four benchmarks, ARISE-RL achieves average state-of-the-art performance with a 9B open-source backbone.The conclusion attributes this result to rubric-mediated co-evolution and reward-gated self-distillation for open-ended agent training.
Limitations
The experiments are limited to 8/9B open-source backbone models because of computational resource constraints. Whether ARISE-RL and RG-SED scale to substantially larger models remains unverified.
- Limitations: Experiments use only 8/9B open-source backbone models, leaving scalability to substantially larger base models for future validation.The authors explicitly identify large-scale validation across stronger backbones as future work.
Ethical Considerations
The study reports research-purpose data collection without personally identifiable information and describes the tools and benchmark composition used for evaluation. ECR-Travel balances five subtasks and includes map, search, transport, and weather tools, while topic distributions characterize the benchmark domains.
- Ethical Considerations: The study states that all data are intended for research purposes, no personally identifiable information was collected, and the protocol received internal organizational review.The supplied passage truncates the final approval wording.
- Benchmark Tools: ECR-DeepResearch uses a single Google Custom Search-based web_search tool, with pages exceeding 7,500 characters summarized before being returned to the agent.This reduces context saturation from lengthy retrieved pages.
- Benchmark Tools: ECR-Travel exposes seven primitives covering POI lookup, nearby search, routing, web search, flights, trains, and weather.The tools support structured location retrieval, multimodal routing, textual recommendations, transport search, and weather-aware planning.
- Benchmark Composition: The ECR-Travel and ECR-DeepResearch word clouds summarize topic frequencies using curated token mappings, with ECR-DeepResearch showing no dominant research theme.The ECR-Travel lexicon covers cities, activities, transport, time of day, and weather.
- Benchmark Tools: Flight and train searches are simulated by GPT-5-mini under deterministic prompts encoding a closed database for a fixed planning horizon.This implementation applies specifically to search_flights and search_train_tickets.
B.2 Benchmark Statistics
ECR-Bench spans balanced travel-planning subtasks and distinct rubric dimensions, with expert-authored evaluation and broad baseline coverage. ECR-Travel emphasizes coverage and spatial-temporal constraints, while ECR-DeepResearch emphasizes reasoning, sourcing, and structure.
- ECR-Travel: ECR-Travel contains 500 queries across five perfectly balanced subtasks, with direction, around_search, poi_search, weather, and transport tools most frequently invoked.Each subtask contributes 100 queries; direction is used 327 times, followed by around_search at 184 and poi_search at 163.
- Rubric composition: ECR-Travel rubrics emphasize coverage plus location and time constraints, whereas ECR-DeepResearch concentrates more on reasoning, sourcing, and structural formatting.The Other category mainly captures free-form logical-consistency criteria.
- Annotation: Six domain experts authored the rubrics using a standardized bilingual handbook and independently verifiable pass criteria.The annotators comprised three travel-planning experts and three deep-research or academic-writing experts.
- Baselines: All baselines use the same benchmark interface, tool environment, and scoring pipeline, while on-policy distillation baselines are reserved for RG-SED comparisons.The comparison includes commercial models, open-source backbones, self-evolution frameworks, and on-policy distillation methods.
D.1 Gate Dynamics over Training.
RG-SED’s gate remains reward-responsive throughout ECR-Travel training: it follows its theoretical response, retracts under degraded coach guidance, and generally limits student–teacher divergence relative to fixed-teacher distillation.
- Gate dynamics: The gate-active rate rises from ≈5% to ≈22% during the first ∼40 rollouts, then adapts to reward changes rather than following a fixed schedule.It visibly retracts around rollouts 80–90 when coach-memory guidance degrades.
- Gate stability: Empirical gate values tightly track the theoretical sigmoid across ECR-Travel training without observable drift or additional calibration.This supports stable selective amplification of distillation according to ∆r.
- Distribution divergence: RG-SED generally operates at lower token-level DKL than GKD, typically ≈0.2 to 0.35 versus ≈0.4 to 0.55.The runs still fluctuate and cross at several rollout segments; the comparison uses GKD with a fixed Qwen3.5-397B teacher.
- Evaluation: The evaluation includes human validation of gpt-5.2 rubric judgments on 50 sampled responses from each benchmark.Three domain-familiar annotators independently score each response, using majority vote as the human label.
- Prompt protocol: Prompt templates coordinate tool-grounded generation, tool-augmented solving, rubric judging, and coach-memory conditioning across roles.The Generator must ground research rubrics in web-search evidence, while the Solver produces structured reports and receives coach feedback when RG-SED is active.
F.1 VitaBench — OTA: Attraction Tickets + Hotel for Parents
The VitaBench OTA case illustrates how ARISE-RL combines tool-grounded task generation, multi-step solving, rubric judging, and coach-memory distillation. The supplied passages emphasize prompt protocols and a successful memory-augmented trajectory.
- Case study: A successful OTA trajectory used memory-augmented solving to satisfy six rubrics for an attraction ticket and parents’ hotel booking.The case is presented as a late-stage ARISE-RL rollout whose memory-augmented sample succeeded while sibling rollouts failed.
- Generator: The Generator grounds tasks in real tool observations, verifies expected tools, and incorporates returned IDs, prices, dates, POIs, routes, and weather into instructions.This protocol is stated for ECR-Travel and VitaBench, with coach memory summarizing recent generation outcomes.
- Case study: A complementary delivery case shows memory-guided recovery of the prior restaurant, delivery timing, menu constraints, and user confirmation before ordering.The successful trajectory contrasts with sibling failures involving arbitrary store selection and unverified delivery feasibility.
- Task setting: VitaBench uses an interactive setting in which a UserSimulator reveals needs across turns and the agent may ask clarifying questions before tool calls.The Generator is specialized for OTA and daily-life service tasks.
- Solver: The Solver follows tool-use guidelines, receives group-level actionable coach advice, and can be evaluated against explicit rubric outcomes.Solver memory summarizes attempts, rewards, expected versus actual tools, missing tools, and rubric pass/fail details.
Trajectory A — VitaBench: Attraction Tickets + Hotel for Parents
In the OTA trajectory, coach memory converts a coupled attraction-and-hotel request into a verified sequence of date, weather, ticket, history, room, and payment decisions. The resulting plan passes all six rubrics.
- Planning: The agent first resolves li-qiu as 2025-08-07, checks Jincheng weather, and identifies Aug. 5 as the sunny attraction date.The weather response reports Aug. 5 as sunny at 22–29 °C, while Aug. 2–3 are rainy, Aug. 4 cloudy, and Aug. 6–7 cloudy.
- Hotel resolution: It retrieves the previous Wuzhou booking and verifies that Atour Wuzhou offers a deluxe twin-bed room at ¥214 per night.The prior booking used a king-bed room, so the room variant had to be checked explicitly.
- Attraction booking: The attraction option costs ¥180 for one adult ticket including the “Rendition of Kangxi” performance.The agent surfaces the ticket details before asking the user to confirm both orders.
- Confirmation: The confirmed plan books Aug. 7→Aug. 9 for two nights in a deluxe twin room and places the attraction order on Aug. 5.The user confirms the dates, ticket variant, room type, and purchase before execution.
- Outcome: Both orders are paid successfully, totaling ¥608, and the final trajectory passes all six rubrics.The rubric checks cover the performance ticket, sunny date, attraction price, li-qiu-anchored hotel dates, recovered hotel, and twin-bed room.
Trajectory B — ECR-Travel: Beijing One-Day Itinerary
The travel agent converts weather and itinerary requirements into a time-windowed Beijing plan using place lookup and route estimation. The resulting schedule covers the requested stops, travel modes, dwell windows, and weather rationale, with all five rubrics passing.
- Task: The task required ordering Sanlihe Park, Fayuan Temple Museum, and Qianmen Street around cloudy-to-sunny weather, with walk or public-transport durations.The itinerary also needed shopping, museum, dinner, and old-brand activities.
- Method: The coach memory prescribed weather lookup, POI searches for coordinates and opening hours, and route calls for every leg.It specifically targeted failures involving vague ordering, missing durations, and absent weather-based scheduling rationale.
- Schedule: The recommended schedule placed a 09:00–10:30 Sanlihe Park stroll during the cloudy phase, followed by a 53-minute transfer.The transfer consisted of 12 minutes walking, 30 minutes by bus, and 11 minutes walking.
- Schedule: Fayuan Temple Museum occupied 11:30–15:00, with an approximately 3-hour dwell window during the hotter sunny phase.The indoor museum placement follows the stated weather rationale.
- Schedule: Qianmen Street was scheduled for 16:00–20:00, including dinner and old brands, with sunny late-afternoon light supporting the evening walk.The plan also aimed to avoid backtracking and keep all three POIs within their opening windows.
- Evaluation: All five rubrics passed: weather-comfort ordering, POI coverage, per-leg minutes, dwell windows, and geographic consistency.The reported rubric outcome marks each criterion as satisfied.