ARISE-RL: Agentic Rubric-Grounded Iterative Self-Evolution with Reinforcement Learning

1 Introduction

• Open-ended tasks lack gold answers and scalable training rubrics, while long-horizon tool use produces brittle rewards and weak rollout contrast that hinders fine-grained learning.Existing open-ended suites largely serve as static post-hoc testbeds; fixed-teacher distillation can accumulate distribution gaps, and unreliable guidance risks transferring poor tool choices or reasoning.Source: §1 · pp. 1–2
• ARISE-RL couples a tool-grounded rubric-generating Generator with a reasoning-and-tool-use Solver, rewarding intermediate empirical solvability to keep tasks near the Solver’s evolving capability boundary.Rubrics connect task construction with policy optimization, and the Generator must use relevant tools before writing criteria that depend on their outputs.Source: §1 · p. 2
• RG-SED distills reward-improving Generator patterns and Solver behaviors from memory-augmented same-policy rollouts only when they outperform standard rollouts.It uses on-policy token-level reverse KL and avoids distribution mismatch and blind imitation of unreliable guidance.Source: §1 · pp. 2–3
• Across ECR-DeepResearch, ECR-Travel, ResearchRubrics (Sharma et al., 2025), and VitaBench (He et al., 2025), ARISE-RL achieves best average performance and robust gains on 8B/9B-scale models.Deep research is measured by rubric score rate, while multi-tool task planning is measured by task pass rate.Source: §1 · p. 3
• ECR-Bench covers single-tool deep research and multi-tool travel planning, evaluating final-response quality and process-level tool-use correctness.It includes 100 expert-calibrated research queries and five travel task types: route planning, transportation comparison, nearby POI search, and one-day or multi-day itinerary planning.Source: §1 · pp. 3–4

Self-Evolving Agents

• Prior self-evolution frameworks target closed-form, single-tool, or reasoning-centric settings; ARISE-RL closes task generation, rubric construction, and policy learning for open-ended agentic tasks.EvolveR (Wu et al., 2025) combines offline self-distillation with online interaction, ASL (Sun et al., 2025) unifies prompt generation, policy learning, and reward modeling, and RAGEN (Wang et al., 2025) studies multi-turn RL.Source: §2 · p. 4

Off-Policy Distillation and Self-Distillation

• RG-SED gates same-policy, coach-prompted self-distillation on empirical reward gains, using on-policy token-level reverse KL to limit mismatch and retain training–deployment consistency.Static-teacher importance-sampling corrections can destabilize multi-turn optimization (OPCD; Ye et al., 2026), while standard self-distillation lacks a principled criterion for when to distill (Zhang et al., 2019; Allen-Zhu & Li, 2020).Source: §2 · p. 4

3.1 Rubric-Mediated Generator–Solver Co-Evolution

• Rubrics couple task construction and policy learning by providing explicit criteria, process-aware Solver supervision, and tool-grounded feedback for Generator training.Source: §3.1 · p. 4
• The Generator is rewarded for tool-grounded, format-valid tasks whose difficulty peaks when approximately half of the Solver’s sampled rollouts succeed.The tool-grounding gate requires real tool observations before tool-dependent rubric criteria are specified; the difficulty reward falls to zero when all rollouts succeed or fail.Source: §3.1 · pp. 4–5
• The Solver receives fine-grained rubric-satisfaction supervision with a full-completion bonus, and its reward also supplies the Generator’s empirical difficulty signal.Source: §3.1 · p. 5

3.2 Role-Conditioned Reward-Gated Self-Evolution Distillation

• RG-SED distills memory-augmented behavior only when it improves empirical group reward, applying token-level reverse KL to standard on-policy trajectories to avoid blind imitation of noisy guidance.The method is motivated by sparse or weak learning signals for both roles and is designed to mitigate distribution mismatch while improving self-evolution stability and efficiency.Source: §3.2 · pp. 5–6
• RG-SED uses role-specific coach memories to guide Generator task construction and Solver responses, conditioning the same policy rather than introducing an external teacher.Because the student and memory-augmented teacher share parameters and differ only in prompt conditioning, their distributional gap is constrained.Source: §3.2 · p. 6

Benchmark overview

• ECR-Bench pairs ECR-DeepResearch for single-tool research with ECR-Travel for multi-tool travel planning.The domains assess complementary capabilities: evidence-grounded long-form research and constraint-aware multi-tool planning.Source: §4 · pp. 6–7

Stage I: Benchmark design and expert rubric annotation

• Stage I manually reviews test queries and rubrics for clarity, difficulty, specificity, coverage, and consistency, revising or removing those that fail.The review is intended to ensure evaluation reliability.Source: §4 · p. 7

ECR-DeepResearch

• ECR-DeepResearch contains 100 test queries and uses rubric score ratio to assess factual grounding, evidence coverage, reasoning, completeness, and report structure.It evaluates web-search-based research requiring information retrieval, evidence synthesis, and structured research-style responses.Source: §4 · p. 7

ECR-Travel

• ECR-Travel covers five categories with 100 queries each, evaluating itinerary quality and tool-use correctness by task success rate.Categories are Direction, Compare, Search, 1-Day, and M-Day; tools include POI and nearby search, navigation, web, flight, train-ticket, and weather lookup.Source: §4 · p. 7

Stage II: Quality control

• Stage II uses a rule-augmented LLM checker to validate tool calls, dialogue, formatting, logical consistency, and final-answer consistency, rewriting or removing failed samples.Test queries and rubrics are also manually reviewed for reliability.Source: §4 · p. 7

5.1 Experimental Settings

• Evaluation covers ECR-DeepResearch, ECR-Travel, ResearchRubrics, and VitaBench, using rubric score rate for research and four-run-average task pass rate for multi-tool tasks.The benchmarks span single-tool deep research and multi-tool travel planning and daily-life tasks; baselines include closed-source and open-source LLMs and self-evolution frameworks.Source: §5.1 · pp. 7–8

5.2 Main Results

• On Qwen3.5-9B, ARISE-RL achieves the best overall performance, surpassing closed- and open-source non-self-evolving baselines and outperforming Dr. Zero and Absolute Zero under matched backbones.Its strongest gains are on the two interactive multi-tool benchmarks, while it reaches the column-wise best on both DeepResearch-style benchmarks; the advantage also holds on Qwen3-8B.Source: §5.2 · p. 8
• In Figure 3, RG-SED attained the best score on every one of the eleven benchmark axes; despite complementary strengths on the radar, OPCD and GKD remained consistently inferior to RG-SED overall.Its group-level memory-augmented self-distillation gates the distillation signal by empirical reward improvement rather than applying it blindly.Source: §5.2 · pp. 8–9, Figure 3 · p. 9

5.3 Further Analysis

• On ECR-Travel, 8.3% of groups have ∆r < −τa, 19.6% have |∆r| ≤ τa, and 72.1% have a reliable positive reward gap.The gate activates distillation only when ∆r > τa, avoiding neutral or harmful supervision from coach memory.Source: Figure 5 · p. 9, §5.3 · p. 10
• Removing RG-SED causes the largest performance drop on the multi-tool benchmarks, while ungated distillation degrades those tasks and tool-grounded rubrics and difficulty-shaped rewards each add gains.The ablations indicate that reward gating, tool-grounded rubrics, and difficulty-shaped rewards are complementary design choices.Source: Table 2 · p. 8, §5.3 · p. 9
• Across three co-evolution cycles, Solver performance improves on all four benchmarks, with the largest gains on multi-tool tasks, as generated queries become harder.The frozen Qwen3.5-9B base policy’s single-shot pass rate decreases monotonically on both multi-tool benchmarks across cycles, indicating increasingly challenging generated queries.Source: Table 3 · p. 8, §5.3 · pp. 9–10, Figure 4 · p. 9

6 Conclusion

• Across four benchmarks, ARISE-RL achieves average state-of-the-art performance with a 9B open-source backbone.The authors cite this result as evidence for rubric-mediated co-evolution and reward-gated self-distillation in open-ended agent training.Source: §6 · p. 10
• ARISE-RL couples a Generator and Solver in rubric-mediated co-evolution with fine-grained rewards, and distills memory-augmented behavior only when it empirically improves reward.Source: §6 · p. 10
• ECR-Bench is an expert-calibrated rubric benchmark introduced for multi-tool agents.Source: §6 · p. 10

• Experiments used only 8/9B open-source backbones because of computational resource constraints, leaving validation on larger, stronger models to future work.Source: Limitations · p. 10

Ethical Considerations

• All study data are research-only, no PII was collected, and an internal Ethics Review Board approved the data protocol.Existing artifacts are used under their stated licenses, access conditions, and terms; created artifacts and derivatives of research-only data remain restricted to research contexts.Source: Ethical Considerations · p. 10

B ECR-Bench Construction Details

• The appendix supplements Section 4 with tool inventories and benchmark statistics; ECR-Travel balances five subtasks at 100 queries each, while direction dominates expected tool use.Weather and transport-search tools together account for nearly a third of expected calls.Source: B ECR-Bench Construction Details · p. 13, Figure 6 · p. 14

B.1 Annotated Tool Inventory

• ECR-DeepResearch gives agents one Google Custom Search web_search tool and summarizes parsed pages exceeding 7,500 characters with Qwen3-Max.The summarization is intended to avoid context saturation from lengthy retrieved pages.Source: §B.1 · p. 13
• ECR-Travel’s poi_search resolves addresses or named places into ranked POIs, while around_search retrieves nearby POIs by center, radius, and optional type or keyword.Both use Amap place search and return POI candidates with addresses, coordinates, and business metadata.Source: §B.1 · p. 13
• The Amap-based direction tool returns stepwise walking, driving, or transit routes, including leg distances and estimated durations.Queries use origin and destination coordinates, with optional waypoints and travel mode.Source: §B.1 · p. 13
• ECR-Travel’s Google Custom Search web_search supplies prose recommendations for open-ended questions beyond structured map APIs and summarizes pages over 7,500 characters.Examples include finding quiet places or family-suitable evening activities.Source: §B.1 · p. 13
• Date-specific flight and train searches return ranked intercity options, while GPT-5-mini simulates both under a deterministic prompt and fixed-horizon closed database.Flight results include fares, airports, flight numbers, and times; train results include fares, stations, train IDs, times, and transfer status.Source: §B.1 · pp. 13–14
• The ECR-Travel weather tool returns city-level historical or forecast conditions, including daily conditions, temperatures, wind, and humidity, grounding schedule planning in real weather.It is a departure from prior Amap-only travel benchmarks.Source: §B.1 · p. 13

B.2 Benchmark Statistics

• Across 500 ECR-Travel queries, its five sub-tasks each contain 100 queries; direction leads tool use at 327, followed by around_search (184), poi_search (163), and weather (135).The two transport-search tools each see approximately 125 uses.Source: §B.2 · p. 14
• ECR-Travel’s annotated expected_tools cover required tool-use behavior across its queries.The passage frames Figure 6 as summarizing sub-task composition and tool-use frequency.Source: §B.2 · p. 14
• ECR-Travel topics center on concrete travel verbs, transport modes, and Chinese cities, whereas ECR-DeepResearch covers a flatter spread including technology, health, economy, policy, and environment.The word clouds use jieba-segmented query tokens, with frequent Chinese terms mapped to English.Source: §B.2 · p. 14
• ECR-Travel rubrics emphasize coverage and location/time constraints, while ECR-DeepResearch rubrics span coverage, reasoning, and presentation criteria such as sourcing and structure.Figure 9’s “Other” category mostly captures free-form logical-consistency criteria.Source: Figure 9 · p. 15, §B.2 · p. 15

B.3 Instructions Given to Rubric Annotators

• Six domain experts—three in travel planning and three in deep research or academic writing—received a bilingual handbook specifying independently verifiable rubrics, pass criteria, scoring, tool-call formats, and participation protections.Annotation was conducted by in-house research collaborators without external compensation.Source: §B.3 · p. 15

C Baseline Details

• Baselines comprise closed-source LLMs, open-source backbones, same-backbone self-evolution frameworks, and on-policy distillation methods, evaluated through a shared interface, tool environment, and scoring pipeline.On-policy distillation methods are used only for the RG-SED mechanism comparison in Figure 3.Source: C Baseline Details · p. 15
• On ECR-Travel, RG-SED’s gate-active sample rate rises to about 22% during warm-up and is modulated by reward improvement, while empirical gate values track the theoretical sigmoid.The gate retracts around rollouts 80–90 when coach memory degrades, and the reported selectivity is adaptive rather than scheduled.Source: Figure 10 · p. 16, Figure 11 · p. 17
• The principal ARISE-RL run and same-backbone self-evolution and RG-SED comparisons use Qwen3.5-9B.The passage identifies Qwen3.5-9B as the main 9B-scale base policy for these comparisons.Source: C Baseline Details · p. 16

D.1 Gate Dynamics over Training.

• The gate-active sample fraction rises from about 5% to 22% over the first 40 rollouts, then responds to reward changes rather than flattening.The smoothed mean ∆r stays above activation threshold τa; around rollouts 80–90, degraded coach rewards coincide with a dip in gate activity.Source: §D.1 · p. 17

D.2 Empirical Gate Response.

• Across ECR-Travel training, empirical gate values closely follow the theoretical sigmoid with no observable late-training drift or additional calibration.The gate selectively amplifies distillation according to ∆r.Source: §D.2 · pp. 17–18

D.3 Distribution Divergence.

• RG-SED’s student–teacher divergence typically stays around 0.2–0.35, versus GKD’s 0.4–0.55, although their curves cross at several rollout segments.The comparison is on ECR-Travel; RG-SED uses a same-policy memory teacher, while GKD uses a fixed external Qwen3.5-397B teacher.Source: §D.3 · p. 18, Figure 12 · p. 18

D.4 LLM–Human Judge Consistency.

• Human–LLM judge agreement was 82.3% on ECR-DeepResearch and 77.9% on ECR-Travel.For each benchmark, 50 final-model responses were scored by gpt-5.2 and three annotators using the same rubric, with majority vote as the human label.Source: §D.4 · p. 18, Figure 13 · p. 19

D.5 Generator Query Generation Pipeline

• The Generator refreshes queries every 120 training steps, retaining 500 deduplicated queries and using its ten most recent queries as in-context negatives to reduce near-duplicates.RL pauses during each refresh, and each prompt receives one rollout; the passage describes this history-aware prompting and post-hoc deduplication as preserving training-distribution coverage and diversity without additional training overhead.Source: §D.5 · p. 19

E Prompt Templates; E.1 Generator Prompts; E.2 Solver Prompts; E.3 Judge Prompts

• ARISE-RL defines role-specific Generator, Solver, and Judge prompts, with coach-memory prompts for the Generator and Solver.The prompts specify tool-grounded rubric generation, task solving with coach feedback when active, and rubric-based judging; the translated templates are English, while production uses task-native language.Source: E Prompt Templates · p. 19, §E.1 · p. 19, §E.2 · p. 19

F Case Studies: Coach-Memory-Guided Successful Trajectories

• Late-stage successful samples on VitaBench, ECR-Travel, and ECR-DeepResearch had non-trivial coach memories, while same-group siblings without memory failed the instruction.Each selected sample had reward 1.0; the full annotated trajectories are identified as Trajectories A, B, and C.Source: F Case Studies: Coach-Memory-Guided… · p. 20

F.1 VitaBench — OTA: Attraction Tickets + Hotel for Parents

• With coach memory, the agent resolved the solar term and date, recovered the prior hotel, verified a twin-bed room, then placed and paid both orders, satisfying all six rubrics.It surfaced a consolidated plan before creating the orders.Source: §F.1 · pp. 20–21

F.2 VitaBench — Delivery: Sichuan Re-order for Office Visit

• With memory, the agent recovered the previous Sichuan restaurant, checked the delivery address and 30-minute ETA against the 13:00 deadline, confirmed the dish and spice level, then placed and paid the order.Without memory, sibling rollouts chose an arbitrary store or ordered without checking whether delivery would be on time.Source: §F.2 · p. 21

F.3 ECR-Travel — Beijing One-Day Itinerary

• With memory, the agent used weather, POI, and navigation tools to schedule outdoor walking in cooler cloudy weather and the indoor museum in the sunny phase, satisfying all five rubrics.The schedule also specified transfer durations and dwell windows; memory-free rollouts omitted weather rationale and per-leg durations.Source: §F.3 · p. 23

F.4 ECR-DeepResearch — ¥500 K Restaurant in Baotou

• With memory, five focused searches supported a Baotou restaurant report with a verdict, statistics, segmented opportunities, a ¥500 K budget, and a three-scenario profitability model; all five rubrics were satisfied.Memory-free outputs tended to cite yearbooks generically and describe opportunities broadly, such as healthy meals or delivery.Source: §F.4 · pp. 24–25

F.5 VitaBench Generator — Weather-Triggered Booking

• Generator-side RG-SED uses memory of authoring failures to steer harder, more diverse, rubric-grounded tasks; here it produced a weather-triggered Yinchuan-or-Ordos booking task under ¥1500 with eight verifiable rubrics.The memory suggested combining weather, budget, and on-time constraints; the Generator queried weather, attraction, and flight catalogues before authoring the task.Source: §F.4 · p. 26, §F.5 · p. 26

F.6 ECR-Travel Generator — Transport-Mode Comparison

• Guided by memory, the Generator created a Guangzhou–Shenzhen 2-day-1-night itinerary requiring two train recommendations before 21:00, POI opening-hour checks, and driving-distance verification, paired with seven falsifiable rubrics.The memory addressed prior open-ended tasks lacking verifiable constraints.Source: §F.6 · pp. 26–27

Trajectory A — VitaBench: Attraction Tickets + Hotel for Parents

• The prescribed workflow separates the attraction and hotel needs, verifies dates and prior booking details, obtains user confirmation, and only then creates and pays orders.The coach specifies search→info→user-confirm→create→pay, with the previous hotel recovered through order-history tools.Source: Trajectory A — VitaBench: Attraction… · p. 28
• Li-qiu was resolved as Aug. 7, Jincheng’s sunny day was Aug. 5, the prior Wuzhou booking was Atour, and the twin-bed room was listed at ¥214 per night.The performance ticket was listed at ¥180.Source: Trajectory A — VitaBench: Attraction… · p. 29
• Both orders were paid: the Huangcheng Xiangfu ticket with performance cost ¥180, and the Atour Wuzhou twin-bed stay cost ¥428.The attraction was booked for Aug. 5, 2025, and the hotel for Aug. 7–9.Source: Trajectory A — VitaBench: Attraction… · pp. 29–30
• All six rubric checks passed, covering the performance ticket, sunny-day date, ¥180 attraction price, li-qiu-anchored hotel dates, recovered hotel, and twin-bed room.Source: Trajectory A — VitaBench: Attraction… · p. 30

Trajectory B — ECR-Travel: Beijing One-Day Itinerary

• 53-minute transfers connect the 09:00–10:30 Sanlihe Park stroll, 11:30–15:00 Fayuan Temple Museum visit, and 16:00–20:00 Qianmen evening stop.The park and museum dwell times are approximately 1.5 and 3 hours; Qianmen is reached before the dinner rush.Source: Trajectory B — ECR-Travel: Beijing… · p. 31
• All five rubrics passed for the weather-based route: cloudy-phase park, sunny-phase indoor museum, then Qianmen in the evening, avoiding sun exposure and backtracking.The passage also reports that all three POIs were open during the proposed windows.Source: Trajectory B — ECR-Travel: Beijing… · pp. 31–32

Trajectory C — ECR-DeepResearch: ¥500 K Restaurant in Baotou

• The Baotou restaurant analysis should first assemble recent economic and district-level evidence, then give a conditional verdict, segmented opportunities, a ¥500 K budget, and a sensitivity-tested profitability model.The coach memory calls for indicator tables and location advice, with payback, scenario contrasts, reserves, and cash flow among the commonly missed considerations.Source: Trajectory C — ECR-DeepResearch: ¥500… · p. 32

Trajectory D — VitaBench Delivery: Sichuan Re-order for Office Visit

• The delivery workflow converts “before lunch” into a hard 13:00 deadline, estimates ETA before committing, and rechecks the scheduled time before payment.It also calls for confirming the drop-off details and order preferences, and switching stores or modifying the order if the ETA misses the deadline.Source: Trajectory D — VitaBench Delivery:… · p. 34
• 4/4 rubrics were satisfied: [7] recovered Xiao Sichuan (Shifan St. Branch), 787 m away, with an approximately 30-minute ETA and a 12:13 drop-off before 13:00.The order contained Boiling Fish and a garlic-flavoured stir-fried vegetable; the stated minor gap was that rice was unavailable in the menu.Source: Trajectory D — VitaBench Delivery:… · pp. 35–36

Trajectory E — VitaBench Generator: Weather-Triggered Booking

• The weather-triggered booking guidance recommends varied, tool-answerable tasks with quantified rubric hits and difficulty adjusted through constraint conflicts or a simpler fixed-destination query.Suggested directions include choosing a weather-safe city for a refundable flight-and-ticket pair under a price cap, or combining transport, a hotel, and an attraction under budget and arrival constraints.Source: Trajectory E — VitaBench Generator:… · p. 36

Trajectory F — ECR-Travel Generator: Guangzhou-Shenzhen 2D1N

• The Guangzhou–Shenzhen generator guidance avoids unavailable real-time facts and tunes difficulty with trip scope, transfer constraints, plan comparisons, and multiple tool calls.It proposes comparing train candidates and checking two POIs’ opening hours, as well as same-city or short regional itineraries with accessibility, budget, or rain-fallback constraints.Source: Trajectory F — ECR-Travel Generator:… · p. 37