PaperlayerView exact revision on arXiv

Source-linked AI summary

RISED: RubrIcs for agentic multi-environment Selection and sElf-Distillation

Jingtan Wang, Sirajul Salekin, Young mok Jung, Javier Movellan, Bryan Kian Hsiang Low, Manjot Bilkhu

arXiv:2610.00979v1cs.AI

TL;DR

Multi-environment training strategies often lack prompt-group selection based on cross-environment rollout relationships, and identical rewards provide no within-group contrast. RISED uses shared rubric profiles for online selection and self-distillation; across model backbones it achieves the highest mean pass rate across environments.

  • Problem

    Existing allocation and online-selection strategies do not explicitly account for relationships among current rollouts across environments, while identical rewards provide no within-group reward contrast.

    Source: §1 · pp. 1–2

  • Method

    RISED uses a shared rubric vocabulary to select prompt groups by batch-profile alignment and limited overlap, conditions an on-policy teacher with positive rubrics, and uses negative rubrics to steer later rollouts.

    Source: §1 · pp. 2–3

  • Results

    RISED achieves the highest mean pass rate across environments across model backbones and ranks first or tied for second in all six backbone–environment comparisons.

    Source: §1 · p. 2, §7 · p. 10

  • Takeaways & Limitations

    Rubric-level analyses provide an interpretable view of how behaviours evolve during training.

    Source: §1 · p. 2, §7 · p. 10

    For selection, an incomplete rubric set can make rollouts appear more similar and weaken the signal, so consistent tagging must capture enough variation to distinguish them.

    Source: §2.2 · p. 3

Abstract

from arXiv · show

Training a single LLM agent jointly across diverse interactive environments has attracted increasing attention as a route to generalist agents. Existing curriculum and data-selection strategies often allocate training at the environment level or prioritize local reward-based signals, without explicitly considering relationships between current rollouts across environments for prompt-group selection. Meanwhile, as environments are learned at different rates, all-failure and all-success rollout groups can coexist within a batch, leaving those data without group-relative reward signals. Both challenges highlight limitations of relying solely on scalar rewards in multi-environment RL: they provide limited information about cross-environment relationships and no within-group reward contrast when rewards are identical. This motivates richer textual feedback, such as rubrics describing rollout behaviours, to guide learning. Beyond rubrics' usage as reward, we repurpose rubrics to guide both online data selection and policy supervision. An LLM judge tags each rollout using a predefined rubric vocabulary shared across environments. The resulting profiles guide the selection of data that aligns with the overall behavioural composition of the mixed-environment batch while limiting overlap with already-selected data. Available positive rubrics (describing desired behaviours) provide privileged context for an on-policy self-distillation teacher, supplying additional token-level supervision, while negative rubrics (describing undesired behaviours) guide subsequent rollout generation away from recurring failure modes. Together, these components form RISED. Across model backbones, RISED achieves the highest mean pass rate across environments and ranks first or second in every individual environment. Rubric-based analysis of RISED can further characterize the behavioural changes accompanying these gains.

1 Introduction

Existing multi-environment training strategies do not explicitly select prompt groups by relationships among current rollouts, while identical rewards leave some groups without within-group contrast. RISED uses shared rubric profiles for online selection and self-distillation, and achieves the highest mean pass rate across environments.

1 Introduction

  • Existing online selectors prioritize local rollout signals, while multi-environment batches can contain many groups with identical rewards and no group-relative signal.All-failure and all-success groups constitute more than half of candidate prompt groups in the experiments.Source: §1 · pp. 1–2
  • RISED achieves the highest mean pass rate across environments, with selection and self-distillation providing complementary benefits.Rubric analysis also finds more frequent positive behaviours than in baseline rollouts.Source: §1 · pp. 2–3
  • Rubric-profile similarity correlates more strongly with cross-environment reward improvements than gradient similarity.This preliminary analysis motivates using rubric profiles for prompt-group selection.Source: §1 · pp. 2–3
  • RISED uses a shared rubric vocabulary to compare rollout behaviours, select groups aligned with the batch profile while limiting overlap, and reuse rubrics for self-distillation.Positive rubrics condition the teacher; negative rubrics guide later rollout generation away from recurring failures.Source: §1 · pp. 2–3

2 Rubrics For Data Selection

RISED uses a shared, fine-grained rubric representation to analyze cross-environment transfer and select prompt groups online, then reuses annotations for supervision when rewards provide no within-group contrast. The analysis reports stronger transfer associations for combined rubric profiles, while selection and reward variance capture distinct properties.

2.1 Problem Setup

  • At each step, the method oversamples a candidate pool by 2×, then selects half for the policy update; online selection adapts to the current policy while limiting scoring cost.The candidate pool contains 2k groups and the selected set contains k groups.Source: §2.1 · p. 3

2.2 Rubric Construction

  • Rubrics serve as behavioural descriptions for comparing rollouts rather than calibrated scalar rewards, so selection needs consistent tags but not exhaustive coverage of desirable behaviours.An incomplete set can make rollouts appear more similar and weaken the selection signal.Source: §2.2 · p. 3
  • The rubric vocabulary covers positive and negative behaviours, and an LLM judge assigns each rollout hierarchical skill-and-rubric tags with role and occurrence count.The vocabulary is built from warm-up and stronger teacher rollouts; extraction uses one judge call per rollout and can overlap annotation with generation.Source: §2.2 · pp. 3–4

2.3 Does Rubric Similarity Track Cross-Environment Transfer?

  • The transfer analysis compares training–validation rubric-profile similarity and gradient similarity with reward improvements across checkpoints and environments.It uses WebShop-only training and evaluates held-out environments; rubric profiles are count vectors compared with cosine similarity.Source: §2.3 · pp. 4–5, Table 1 · p. 5
  • Combined positive-and-negative rubric profiles identify the most-improved environment at 19 evaluated checkpoints, versus 8 for gradient similarity, while skill-level aggregation generally weakens the signal.Independent re-tagging changes reported Spearman coefficients by at most 0.060.Source: §2.3 · p. 5

2.4 Rubrics for Online Data Selection

  • Greedy round-robin selection chooses groups aligned with the candidate batch’s rubric profile while penalizing overlap with already-selected groups, preserving fixed environment quotas.Each choice changes redundancy scores across environments and can favour complementary behaviours.Source: §2.4 · p. 5
  • Selected groups have mean within-group reward variance 0.0557 versus 0.0507 in the candidate pool, while a variance-targeted selector reaches approximately 2×.Rubric overlap captures exercised behaviours, whereas reward variance captures within-group reward contrast.Source: §2.4 · pp. 5–6
  • 62.9% of candidate groups have zero within-group reward variance, and a similar fraction remains among rubric-selected groups, motivating complementary self-distillation supervision.The same rubric annotations are reused to provide token-level supervision on selected rollouts.Source: §2.4 · p. 6

3 Rubrics For Distillation

RISED uses positive rubrics to condition an on-policy self-distillation teacher and negative rubrics to steer subsequent training rollouts. The teacher supplies token-level supervision independently of rollout reward contrast.

On-policy self-distillation

  • On-policy self-distillation uses the same policy as teacher and student, conditioning the teacher on privileged context unavailable to the student.The student matches the teacher’s output distribution on its own rollouts, providing dense per-token targets.Source: §3 · p. 6
  • The self-distillation loss supplies dense token-level supervision independently of GRPO advantage coefficients and is added to the GRPO loss on selected groups.The teacher’s log-probabilities are treated as constants when differentiating the distillation loss.Source: §3 · p. 6

Positive rubrics distill; negative rubrics steer

  • Positive rubrics provide privileged teacher context, while negative rubrics steer the next step’s rollout prompts when violations outnumber positive behaviours.The teacher context combines each rollout’s top three positive rubrics with its group’s top three; negative rubrics are injected as an “anti-patterns” block.Source: §3 · p. 6

RISED

  • RISED combines rubric-guided selection with rubric-conditioned distillation in one multi-environment approach.Negative rubrics steer rollout generation, while positive rubrics provide the self-distillation target.Source: §3 · pp. 6–7

4 Experiments and Results

RISED achieves the highest mean overall pass rate on both evaluated backbones and ranks first or tied for second in every individual environment comparison. Experiments compare it with five baselines under a joint-training setup.

4.1 Experimental Setting

  • Training jointly covers ALFWorld, WebShop, and DBBench, with 64 prompts and eight rollouts per prompt and 64 of 128 candidate groups selected per step.Validation pass rates are measured every 10 steps on fixed held-out sets, with overall performance defined as the unweighted mean of the three environment-level rates.Source: §4.1 · p. 7

4.2 Baselines

  • Five baselines include two unselected GRPO batch sizes, two reward-variance selectors, and an all-rubric self-distillation method.The variance selectors choose 64 of 128 candidate groups; All-Rubric SD uses 64 groups without selection or negative-rubric guidance.Source: §4.2 · p. 7

4.3 Main Results

  • RISED achieves the highest mean overall pass rate on both backbones, improving over GRPO-64 by 5.8 and 6.9 percentage points, and ranks first or tied for second in all six backbone–environment comparisons.Against the strongest baseline on each backbone, gains are 1.2 and 0.6 points; RISED also outperforms GRPO-128 while updating on 512 of its 1024 generated rollouts per update.Source: §4.3 · pp. 7–8, Table 2 · p. 8

5 Ablations and Further Analysis

Ablations, behavioural analyses, and a four-environment stress test examine RISED’s components and the behaviours accompanying its performance. The results indicate that selection and distillation contribute complementarily, with effects varying by environment.

Component ablations

  • Removing either rubric-guided selection or rubric-conditioned distillation reduces overall performance, though their relative contributions vary by environment.Distillation contributes more on WebShop, while selection provides larger gains on ALFWorld and DBBench.Source: §5 · p. 8, Figure 3 · p. 9

Qualitative analysis

  • RISED accumulates rollouts with environment-relevant positive behaviours earlier and in greater numbers than GRPO-64, alongside an emerging validation-performance gap.The analyses track relevant item selection in ALFWorld, search-query formulation and execution in DBBench, and item navigation in WebShop.Source: §5 · pp. 8–9, Figure 4 · p. 9

Stress test with a challenging fourth environment

  • With AppWorld added, RISED achieves higher overall performance than GRPO-64 and begins making progress where GRPO-64’s AppWorld validation pass rate remains near zero.RISED produces more frequent AppWorld successes late in training, although performance is described as low and variable.Source: §5 · p. 9, Figure 5 · p. 9

6 Related Work

Prior online-selection approaches use example-level signals such as loss, gradient importance, reward variability, or estimated difficulty. RISED instead compares current rollout rubric profiles across environments while limiting redundancy.

Online data selection

  • Online data selection has prioritized examples using loss, gradient-based importance estimates, or estimated reducible loss.These approaches estimate which training examples merit priority.Source: §6 · p. 10

RL online selection

  • RL selection methods use reward variability or policy-dependent difficulty, with some approaches also incorporating replay, information gain, or historical outcomes.For binary rewards, success rates closer to 0.5 correspond to higher reward variance.Source: §6 · p. 10

RISED and comparison protocol

  • RISED selects a fixed number of already-generated groups using current rollout rubric profiles for mixed-environment coverage while limiting redundancy.Its comparison protocol excludes validation-guided scoring and per-group gradient computation.Source: §6 · p. 10

7 Conclusion

This section summarizes RISED’s rubric-guided data selection and rubric-conditioned self-distillation, along with the construction and use of its rubric vocabulary and rollout-tagging prompts. It also describes rollout-tagging coverage, the rarity of discovered rubric names, and the training and validation data scope across four environments, distinguishing main experiments from ablations.

7 Conclusion

  • RISED combines cross-environment rubric-guided selection and self-distillation, consistently improving over competitive baselines across two model backbones and multiple agent environments.Rubric-level analyses offer an interpretable view of behavior change during training; rubric tagging and teacher inference add overhead, motivating future efficiency improvements and scaling to larger policies and more environments.Source: §7 · p. 10

A.1 Further Details on Vocabulary Construction

  • Vocabulary construction augments weak-policy failure coverage with teacher successes, producing 82 rubrics across 11 skills and four environments; a shared taxonomy avoids cross-checkpoint fragmentation from unaligned free-form names.The initial vocabulary was expanded using warm-up and teacher rollouts; descriptions support teacher conditioning and rollout guidance, not tagging or sample selection.Source: §A.1 · p. 15

A.2 Examples of Rubrics and Skills

  • The vocabulary groups specific behaviors under broader skills and covers desired and undesired behaviors, including environment-general patterns and rubrics tailored to particular environments.For example, search_and_navigation spans WebShop, ALFWorld, and DBBench, while explores_database_schema is specific to DBBench.Source: §A.2 · pp. 15–16

A.3 Main Prompts Used for Rubric Generation

  • Rubric prompts trace successful rollouts backward to abstract prerequisite steps, count and rank recurring errors in failures, and apply both strategies to partial successes.For partial successes, prompts report both demonstrated and violated rubrics; tagging uses the fixed taxonomy while permitting discovered positive behaviors when needed.Source: §A.3 · pp. 16, 19

A.4 Further Detail on Rubric Tagging.

  • Rubric tagging is implemented with an asynchronous primary path and a retry path.The supplied passage does not specify further details about either path.Source: §A.4 · p. 20
  • Approximately 98% of rollouts are processed, and discovered names account for less than 1% of emitted rubric names.The primary tagging path uses the predefined vocabulary; a retry path uses a shorter candidate list refreshed every ten steps. Returned names are normalized and mapped to predefined rubric identifiers, whose descriptions are used for teacher conditioning and rollout guidance.Source: §A.4 · p. 20
  • The data table covers four environments, while main experiments and ablations use the first three.AppWorld training includes only easy and medium tasks, whereas validation includes all difficulty levels; sizes count task instances, not generated rollouts.Source: Table 5 · p. 21

B.1 Training Datasets and Environments

The experiments use four interactive environments for training and validation data, with the main experiments and ablations focused on three. AppWorld training is limited to easy and medium tasks, while its validation set includes all difficulty levels.

B.1 Training Datasets and Environments

  • The main experiments and ablations use ALFWorld, DBBench, and WebShop; preliminary correlation analysis and additional experiments also include AppWorld.The four environments cover household manipulation, database operations, online shopping, and application-based digital tasks, respectively.Source: §B.1 · p. 21
  • AppWorld training uses 72 easy and medium tasks across 24 scenarios, while validation uses 57 tasks spanning all difficulty levels.Validation uses each environment’s designated set without subsampling; sizes count task instances rather than generated rollouts.Source: §B.1 · p. 21

B.2 Training Hyperparameters

  • Training hyperparameters are reported in Table 6, with shared values listed once and backbone-specific values listed separately.Source: §B.2 · p. 21

B.3 Training Loss

Training follows AgentRL’s GRPO formulation and setup, using group-relative advantages and a clipped objective with a reference-policy term. The appendix also outlines RISED’s training procedure and identifies the default backbone for the additional empirical analysis.

B.3 Training Loss

  • GRPO uses group-relative advantages and a loss-minimization objective, with token-level loss computation and normalization following AgentRL.The compact objective is group-indexed; the implementation details and shared hyperparameters are referenced in AgentRL and Table 6.Source: §B.3 · pp. 21–22
  • The GRPO objective uses the rollout policy and a frozen reference policy, with backbone-specific training configurations reported in Table 6.The passage defines θold as the rollout policy and πref as the frozen reference policy.Source: §B.3 · p. 22, Table 6 · p. 22

B.4 Algorithm Pseudo-code

  • Algorithm 1 presents RISED’s conceptual training step, implemented with AgentRL’s rollout and optimization conventions.Source: §B.4 · p. 22

C Additional empirical analysis

C.1 Additional Cross-Environment Transfer Analysis

The additional analysis tests rubric-profile transfer signals under DBBench-only training, using unchanged evaluation environments, profile construction, and correlation protocol. It reports stronger held-out correlations for rubric similarity than skill or gradient similarity and compares rubric-role conditioning variants.

C.1 Additional Cross-Environment Transfer Analysis

  • All-tag rubric similarity achieves higher held-out correlation than all-tag skill similarity and gradient similarity under DBBench-only training.Correlations pool checkpoint–environment pairs at steps 50–500 relative to the step-25 baseline; held-out results exclude DBBench.Source: §C.1 · p. 23, Table 7 · p. 23
  • Positive-only profiles have higher correlations in three of four comparisons, while combined profiles attain the highest all-environment Pearson correlation.Combined profiles remain among the strongest positively correlated signals in both DBBench- and WebShop-training settings.Source: §C.1 · p. 23
  • The gradient comparison averages reward-weighted training and validation policy gradients within each pool, then computes cosine similarity between the averages.Source: §C.1 · p. 23
  • Positive-only conditioning outperforms all-rubric conditioning overall, especially on ALFWorld, while full RISED learns faster initially, most clearly on DBBench.Source: Figure 6 · p. 24

C.2 Tagger reliability

The tagger produces identical tag sets for 68.8% of rollouts, while aggregate rubric profiles and downstream transfer-correlation analyses remain consistent across runs using different inference endpoints.

C.2 Tagger reliability

  • The reproducibility check retags the same rollouts in two runs with identical model and tagging settings but different inference endpoints.Table 8 reports signal agreement across checkpoint–environment pairs using Pearson correlation and mean absolute difference, and transfer correlations as pooled Spearman coefficients.Source: §C.2 · p. 24, Table 8 · p. 24
  • Mean cross-run profile cosine similarity ranges from 0.9622 to 0.9992 across training and validation profiles.The two runs produce identical tag sets for 68.8% of rollouts; mean Jaccard overlap is 0.856 and median overlap is 1.000.Source: §C.2 · p. 24
  • All reported Spearman correlations change by at most 0.060, preserving their signs and feature-variant ranking.The held-out rubric-all correlation changes from 0.595 to 0.588.Source: §C.2 · p. 24

C.3 Role Filter in the Teacher Context

The role-filter analysis supports using positive rubrics for teacher conditioning and negative rubrics for rollout guidance; additional analyses compare rubric-judge sizes and track changes in DBBench failure behaviours.

C.3 Role Filter in the Teacher Context

  • Conditioning the teacher on all rubrics performs worse overall than positive-only conditioning, particularly on ALFWorld.The authors suggest that avoid-violation instructions may create tension with distillation on the sampled rollout, whereas positive rubrics highlight demonstrated behaviours.Source: §C.3 · pp. 24–25

C.4 Additional Qualitative analysis

  • Repeated failed actions and false completion claims reach half their initial rates substantially earlier than hallucinated observations or actions.Figure 7 tracks the fraction of rollouts exhibiting each undesired behaviour and marks when each rate falls to half its initial level.Source: Figure 7 · p. 25

C.3 Role Filter in the Teacher Context

  • Full RISED improves faster initially than positive-only conditioning, most clearly on DBBench, where negative rubrics identify actionable failures despite few positive tags.The analysis motivates using positive rubrics for distillation and negative rubrics for subsequent exploration; the compared variants differ in teacher conditioning and rollout-prompt guidance.Source: §C.3 · pp. 24–25

C.5 Effect of judge model size

  • The 32B judge achieves the highest overall validation performance, while 14B remains competitive and 8B performs consistently worse overall.The comparison keeps the rest of the RISED pipeline fixed.Source: §C.5 · pp. 25–26, Figure 8 · p. 25

C.6 Training-Time Analysis

The appendix discusses RISED’s training-time overhead and scope, the rationale and costs of a fixed rubric vocabulary, and limits on what its similarity analysis and identical-reward supervision establish.

C.6 Training-Time Analysis

  • RISED’s total step time is 1.63× that of GRPO-64 and comparable to GRPO-128, with added latency primarily from rubric tagging and the teacher forward pass.Selection takes less than 0.1 s per step; elapsed-time comparisons use a dedicated judge server and therefore do not compare total compute. Rollout wait is exposed blocking time, not total generation cost, and lower wait should not be read as faster generation.Source: §C.6 · p. 26

D Additional Literature Review

  • RISED selects prompt groups from current mixed-environment rollouts under a fixed environment allocation, unlike approaches that allocate domains, environments, or rubric-based rewards.Its shared behavioural rubric vocabulary also supplies teacher context; the review describes prompt-group selection and distillation as complementary components.Source: D Additional Literature Review · p. 27

E.1 Why use a fixed rubric vocabulary?; E.2 Will the warm-up process in building vocabulary introduce additional time cost?

  • A fixed rubric vocabulary provides a stable coordinate system for comparing profiles across environments and training steps, and its one-time warm-up cost is reused across runs and backbones.Predefined labels reduce recurring tagging to label selection; rubric consolidation is performed offline, while online tagging uses a locally served judge.Source: §E.1 · p. 28, §E.2 · p. 28

E.3 Why evaluate rubric similarity at the environment level?

  • Environment-level similarity analysis is a computationally tractable diagnostic, not a direct estimate of individual prompt groups’ marginal benefits.Estimating prompt-group effects would require more expensive controlled updates and evaluations; practical utility is assessed in the main experiment and component ablations.Source: §E.3 · p. 28

E.4 Does reward-conditioned tagging affect the interpretation of rubric similarity?

  • Because the judge uses rollout rewards and outcome-dependent instructions, rubric profiles reflect both behaviour and outcome composition, so stronger correlations do not establish reward-independent association.The comparison tests how well two reward-informed representations track observed environment-level reward improvements.Source: §E.4 · p. 28

E.5 Are negative rubrics injected during validation?

  • Validation uses original task prompts without negative-rubric injection, and reported training-guidance benefits persist under that unguided evaluation protocol.Source: §E.5 · p. 28

E.6 Does every group with identical rewards receive positive-rubric supervision?

  • Positive-rubric distillation does not provide privileged behavioural context to groups without positive rubrics; when teacher and student parameters and inputs coincide, the distillation term has zero loss and gradients.Negative rubrics can still guide subsequent rollout generation.Source: §E.6 · pp. 28–29

E.7 Why not evaluate larger policy backbones?

  • The evaluation is limited to two policy backbones because repeated agent-RL runs, baselines, ablations, seeds, and validation incur substantial compute costs.For Qwen2.5-3B-Instruct, a run uses 1000 steps on 32 H100 GPUs; measured averages are about 1.8 days for GRPO-64 and 2.9 days for RISED.Source: §E.7 · p. 29

Related papers

Loading 2610.00979v1…