PaperlayerView exact revision on arXiv

Source-linked AI summary

Structuring MoE Expert Selection for Agentic Reinforcement Learning

Bolian Li, Ting-Yao Hu, Cheng-Yu Hsieh, Sanjoy Chowdhury, Oncel Tuzel, Raviteja Vemulapalli

arXiv:2610.07332v1cs.LGcs.CL

TL;DR

Co-design of agentic behavior and MoE routing is underexplored, even though agentic trajectories exhibit operation-related expert-selection structure that standard RL does not explicitly optimize. The paper introduces hierarchical routing control, combining operation-aware turn-level routing with selective token-level consistency and entropy gating. It reports over 10-point task performance improvement, alongside improved training stability and inference efficiency.

  • Problem

    Co-design of agentic frameworks and MoE routing remains underexplored, and standard agentic post-training does not account for how policy updates affect expert selection.

    Source: §1 · p. 1

  • Method

    The framework combines operation-aware turn-level routing control with selective token-level consistency and gates auxiliary routing gradients using policy entropy.

    Source: §1 · p. 2, §3.3 · p. 6

  • Results

    Over 10-point task performance improvement is reported, with improved performance across RL algorithms and benchmarks, long-term training stability, and inference efficiency.

    Source: §1 · p. 2, §6 · p. 10

  • Takeaways & Limitations

    Agentic trajectory structure provides a useful signal for organizing MoE expert selection during post-training.

    Source: §1 · p. 2

Abstract

from arXiv · show

Long-horizon LLM agents are frequently implemented using sparse mixture-of-experts (MoE) models, yet the co-design of agentic behavior and MoE structures remains underexplored. In this work, we comprehensively study the connections between agentic post-training and MoE expert selection. In off-the-shelf MoE models, we observe expert selection exhibits a specialized structure that naturally aligns with agentic trajectories. Specifically, expert routing overlaps more between turns where the agent performs semantically similar operations (e.g., READ, UPDATE) than between turns with differing operations. However, standard RL algorithms ignore this specialization, allowing the MoE routing to go uncontrolled during training, which empirically limit task performance and inference efficiency. To address this, we introduce a hierarchical routing control framework for agentic tasks. We explicitly encourage turn-level expert selections to align with agentic operations while regularizing token-level expert selections to maintain local consistency. To resolve stability issues that arise during post-training with the proposed methods, we further introduce an entropy-gated control mechanism. Overall, our routing control framework achieves over 10-point improvements in success rate on all evaluated benchmarks. These results demonstrate that agentic trajectory structure provides an effective signal for optimizing MoE capacity during RL post-training.

1 Introduction

The co-design of agentic frameworks and MoE routing remains underexplored, and standard post-training algorithms optimize task performance without accounting for their effects on expert selection. The paper proposes hierarchical routing control and reports improved task performance, training stability, and inference efficiency.

1 Introduction

  • Co-design of agentic frameworks and MoE routing remains underexplored, while current agentic post-training optimizes downstream policy performance without accounting for effects on expert selection.Source: §1 · p. 1
  • MoE routers naturally exhibit operation-grouped specialization and temporal consistency, but standard RL overlooks these routing structures.The paper identifies these structures as factors related to task performance and inference efficiency.Source: §1 · pp. 1–2
  • The framework combines operation-aware turn-level specialization with selective, outlier-tolerant token-level consistency, and uses policy entropy to gate routing gradients for stability.It preserves the standard top-k router and is compatible with different RL algorithms.Source: §1 · p. 2
  • Routing control improves agentic RL across AppWorld and AutomationBench, with over 10-point task performance improvement, greater long-term stability, and better inference efficiency.The framework is evaluated with multiple underlying RL algorithms.Source: §1 · p. 2

2 Preliminary

Agentic reinforcement learning proceeds through repeated turns of reasoning, tool use, and environment feedback, with reward-based policy optimization. MoE policies sparsely route each token to selected experts, and default routing statistics show overlap patterns associated with operations and fields.

2.1 Agentic Reinforcement Learning

  • Each agentic turn includes thinking, tool use, and environment feedback; trajectories repeat this interaction across multiple rounds.The RL signal is an outcome reward, and practical algorithms such as PPO or GRPO use advantage scores in its place.Source: §2.1 · pp. 2–3

2.2 MoE Models: Efficiency and Expert Specialization

  • An MoE layer routes each token to a top-k subset of its experts, combining their outputs with routing probabilities for sparse activation.Sparse activation supports efficient parameter usage during inference.Source: §2.2 · p. 3
  • Default Qwen3-30B-A3B routing on AppWorld shows greater expert overlap for tokens sharing an operation label and for tokens within the same field and turn.Source: Figure 2 · p. 4
  • Standard RL optimizes task completion without explicitly optimizing routing strategy, while agentic tasks use different semantic operations across turns.Source: §2.2 · p. 4

3 Method

The method combines turn-level operation-aware and token-level local routing controls, with entropy gating added to address stability during agentic RL.

3 Method

  • Off-the-shelf MoEs show operation-grouped expert specialization and greater within-field, within-turn token overlap, but RL perturbs these routing statistics.The reported RL-induced noise hinders task performance and inference efficiency.Source: §3 · p. 4

3.1 Turn-Level Operation-Aware Control

  • For thinking and tool-use fields, turn-level MI links expert use to operation labels, reinforcing shared preferences for repeated operations and distinct preferences across operations.Unlike load balancing, it does not guarantee uniform expert utilization; no MI signal is applied when fewer than 8 turns are available.Source: §3.1 · pp. 4–6

3.2 Token-Level Local Consistency Control

  • Token-level control reinforces prior experts only for small within-field adjacent-token gaps, preserving large-gap transitions; it also improves rollout throughput and reduces rollout time.Source: §3.2 · p. 6

3.3 Entropy-Gated Control for Stable MoE RL

  • With Hhigh = 0.2, training remains stable through 200 steps and retains task performance on both splits; without gating, both collapse around step 100.Source: §3.3 · p. 7, Figure 4 · p. 7

4 Experiments

Experiments on AppWorld and AutomationBench test routing controls across RL algorithms, with gains in task performance, routing structure, and rollout efficiency. The analyses also examine how the controls affect expert selection.

4.1 Setup

  • Experiments train on 90 AppWorld tasks and 480 public AutomationBench training tasks spanning six domains, and augment PPO, GRPO, LOOP, and GiGPO with routing controls.AppWorld uses instruction-following Qwen3-30B-A3B-2507, while AutomationBench uses Qwen3.5-35B-A3B; training and rollout generation for both are colocated on 8 NVIDIA B200 GPUs.Source: §4.1 · p. 7

4.2 End-to-End Task Performance

  • Up to 12.9 points in AppWorld and 11.34 points for LOOP on AutomationBench show task-performance gains from routing control.Table 1 reports improvements across both benchmarks and diverse RL algorithms; PPO's AppWorld improvement is inconsistent, which the authors attribute to unstable MI estimates from fewer turns per prompt.Source: §4.2 · pp. 7–8, Table 1 · p. 8

4.3 Changes in Routing Behavior

  • Token-level control raises within-field Jaccard similarity by 7.7%, while late-stage mutual information between turn-level expert distributions and operation labels rises 23.5%.Figure 5 summarizes routing similarity, operation-label mutual information, and operation separability; results support increased operation separability under turn-level control.Source: §4.3 · p. 8, Figure 5 · p. 9

4.4 Inference Efficiency

  • In steps 100–200, combined control raises throughput from 224 to 316 tokens/GPU/s (+41.1%) and cuts wall time from 224 to 174 seconds (22.3%).Combined control has the highest late-stage throughput, while token-level control alone has the shortest wall time at 133 seconds; both controls improve inference throughput, with token-level control contributing most early.Source: §4.4 · p. 8, Figure 6 · p. 9

5 Related Works

Prior interactive-agent RL work refines learning signals, exploration, and environment interaction, while MoE routing research addresses routing stability, reuse, and specialization. This work uses agentic turn structure to regularize expert selection during RL.

5.1 RL for Interactive LLM Agents

  • Interactive-agent RL methods refine credit assignment, exploration, or environment interaction; this work instead studies MoE expert selection and complements the policy-gradient estimator.Source: §5.1 · pp. 9–10

5.2 MoE Routing Control

  • Unlike routing replay, reward-based selection, phase-dependent experts, or modality bins, this framework regularizes native expert distributions with tool-field operation labels and selective local consistency.Source: §5.2 · p. 10

6 Conclusion

The paper concludes that agentic trajectory structure is a useful signal for organizing MoE usage during post-training. Its hierarchical routing controls improve task performance and inference efficiency while inducing intended routing structure and stabilizing long-term training.

6 Conclusion

  • The framework combines operation-aware turn-level control, selective token-level routing control, and policy-entropy gating of auxiliary routing gradients.Source: §6 · p. 10
  • Experiments on AppWorld and AutomationBench report improved performance across multiple RL algorithms, intended routing structure, long-term training stability, and better inference efficiency.Source: §6 · p. 10
  • The authors conclude that agentic trajectory structure provides a useful signal for organizing MoE usage during post-training.Source: §6 · p. 10

AI Use Statement

The authors used generative AI tools to improve manuscript clarity and readability, assist with tables and figures, and support code development. They take full responsibility for the final content.

AI Use Statement

  • Generative AI tools assisted with manuscript clarity, tables and figures, and code development; the authors take full responsibility for the final content.Source: AI Use Statement · p. 10
  • The authors state that responsibility for the final text, tables, figures, and scientific claims remains with them.Source: AI Use Statement · p. 10

A Metric Definitions

This section defines metrics for evaluating MoE routing during RL, including expert specialization, routing consistency, and load balance. The reported default-routing statistics show operation- and field-related patterns in expert overlap.

A Metric Definitions

  • Default MoE routing shows higher expert overlap for tokens sharing an operation label and for tokens within the same field and turn; Figure 7 tracks routing consistency and specialization during training.The framework improves both routing consistency and expert specialization, as measured by expert switches and expert-operation classification.Source: Figure 8 · p. 15, A Metric Definitions · p. 15, Figure 7 · p. 15

A.1.1 Expert-Operation Classification

  • A nearest-centroid classifier predicts turn labels from expert indices using K-fold cross-validation, and its metric measures operation discriminability within the evaluation collection.The passage gives no formula for the metric, so this summary describes its stated purpose rather than its calculation.Source: §A.1.1 · p. 15

A.1.2 Similarity Gap

  • The similarity gap compares cosine similarity of expert-usage frequency vectors for turns with the same versus different operation labels to measure routing specialization.Within-group averages use distinct fields sharing a label, while across-group averages compare fields from different groups; cached group sums permit exact computation in O(E(Mn + M^2)) rather than O(EM^2n^2) for equally sized groups.Source: §A.1.2 · p. 16

A.2.1 Jaccard Similarity

  • Token-level Jaccard similarity compares selected-expert sets, with within-field values averaged over token pairs and a random-routing reference of approximately 0.032 for Qwen3-30B-A3B.The reference uses E = 128 experts and k = 8 selected experts per token.Source: §A.2.1 · p. 17

A.2.2 Expert Switches

  • Expert switches count changes across within-field token pairs, providing a measure of routing consistency within each turn.The cited prior work also uses expert switches per token to quantify consistency (Yang et al., 2026).Source: §A.2.2 · p. 17

A.3 Measuring Expert Load Balance

  • Load imbalance is measured with the unsquared coefficient of variation of expert assignments, averaged across trajectories and MoE layers; lower values indicate more balanced usage.Experts with zero assignments are included, and zero CV corresponds to equal assignment counts across experts.Source: §A.3 · p. 17

B Variant Routing Controls

The appendix compares early routing-control variants and technical choices. Operation-label-free load balancing improves expert-load balance but not task performance until turn-level control is applied; several direct consistency controls either underperform or destabilize training.

B.1 Routing Regularization: Field Consensus

  • Field-consensus regularization pushes each token toward its field’s majority-voted experts, but can be too restrictive when tokens need different experts.Consensus is computed separately for each field; the majority-voted set means the most frequently selected experts, not necessarily a strict majority.Source: §B.1 · p. 18, B Variant Routing Controls · p. 18

B.2 Switch Regularization: Adjacent-Distribution Agreement

  • The switch regularizer uses full router distributions to penalize disagreement between adjacent tokens, with gradients applied to both endpoints.The pairwise inner product represents the probability that independent expert draws agree.Source: §B.2 · p. 18
  • The evaluated switch regularizer raised AppWorld test-challenge TGC to 46.28%, but remained below the final framework and could destabilize RL by altering router entropy.Even identical adjacent distributions incur a concentration-favoring penalty.Source: §B.2 · p. 19

B.3 Outlier Regularization

  • Outlier regularization penalizes selected experts outside a configurable field-consensus set, leaving unselected experts without a direct penalty.The design applies probability-weighted downward signals to the outlier logits.Source: §B.3 · p. 19
  • The outlier-suppression run crashed after perturbing routing too strongly, motivating reinforcement of existing local agreement instead.The explored consensus-set sizes were K = 8 and K = 16.Source: §B.3 · p. 19

B.4 Routing Reward and Routing Clipping

  • Routing-reward and routing-clipping variants did not effectively change routing statistics because their discrete consistency scores supplied no differentiated signal to the current router.Ordinary policy-loss gradients could still reach router parameters, but did not explicitly reinforce consensus experts.Source: §B.4 · p. 19

B.5 Load-Balancing Loss

  • Load-balancing loss reduces expert-load CV, but test-challenge task performance improves only after turn-level control is added.The authors conclude that operation labels are still required to improve hard task solving.Source: §B.5 · p. 20, Table 3 · p. 20

C.1 MoE Model Architecture

  • The technical setup uses fused MoE routing operations, separate trajectory processes with sandboxes, and SGLang servers that generate tokens for HTTP requests.The inference settings balance speed and GPU memory capacity.Source: §C.1 · p. 21, Figure 9 · p. 21, Table 4 · p. 21

C.4 Training Settings

The technical sections describe the training setup, injected routing gradients, and task-dependent operation-label rules. Additional analyses find that hard fixed routing lowers trajectory likelihood, while direct router-entropy control can destabilize training.

C.4 Training Settings

  • AppWorld training used Qwen3-30B-A3B-Instruct-25073 for 200 steps, with 160 nominal rollout trajectories per step and evaluations every 10 steps.Training trajectories were capped at 30 interaction turns; evaluation trajectories could run for up to 50 turns.Source: §C.4 · p. 22

C.5 Injected Gradient Flow in RL Regularization

  • Auxiliary routing gradients are injected at router logits alongside the standard RL update, with per-token scaling and an entropy gate; the forward computation remains unchanged.The backward rule is applied at every MoE layer in generated thinking and tool-use fields, updating router weights and pre-router representations.Source: §C.5 · p. 22

C.6 Rubrics for Grouping Agentic Operations

  • Operation labels are assigned by simple, deterministic, task-dependent rules, and a turn’s label is the sorted set of its detected operation categories.The labels are assigned to both thinking and tool-use fields.Source: §C.6 · p. 22

C.6.1 AppWorld Rubrics

  • AppWorld labels are derived from generated API calls; read-oriented prefixes take precedence over HTTP-method rules because some read-only APIs use POST.A turn may receive multiple labels, such as Create and Delete.Source: §C.6.1 · pp. 22–23

C.6.2 AutomationBench Rubrics

  • AutomationBench labels come from structured tool calls, whose api_fetch entries include the HTTP method without an external API-to-method lookup.Local formatting tools and malformed or unsupported calls receive no operation label.Source: §C.6.2 · pp. 23–24

D Additional Discussion

  • The discussion distinguishes total action entropy, route-fixed expected action entropy, and router entropy when analyzing MoE stability.The entropy decomposition gives an upper bound on total action entropy as route-fixed action entropy plus router entropy.Source: D Additional Discussion · p. 24, §D.2 · p. 24

D.1 Is It Possible to Manually Fix the Within-Field Routing?

  • Enforcing fixed expert selections substantially lowers mean token log-probability on example AppWorld trajectories.The comparison is between default routing and replacing token-specific routes with each field’s majority-selected experts.Source: §D.1 · p. 24, Table 5 · p. 24

D.2 Is It Possible to Directly Control the Routers’ Entropy?

  • Direct router-entropy regularization caused training collapse, with small router-score changes near the top-k boundary able to replace active experts and disrupt later hidden representations.The authors link this disruption to an explosion in total action entropy.Source: §D.2 · p. 25, Figure 10 · p. 25

D.3 Why Does Routing Consistency Improve Efficiency?

Routing consistency improves batched inference efficiency through increased reuse across active experts, though the controlled router-collapsing analysis is a synthetic diagnostic. Additional experiments favor operation labels, show fewer expert shifts after routing control, and report a stability–exploration trade-off with R3.

D.3 Why Does Routing Consistency Improve Efficiency?

  • Because fused inference kernels make manual expert selection difficult, the analysis mixes router-weight rows into groups, fixing selection within each group to its first expert.Varying the number of groups controls the number of expert shifts.Source: §D.3 · p. 25
  • Routing consistency improves inference efficiency only for batch sizes larger than 1, regardless of whether expert parallelism is enabled.Figure 11’s controlled comparison varies routing consistency across batch sizes and expert-parallelism settings.Source: §D.3 · p. 26, Figure 11 · p. 26
  • With B tokens and top-k routing, concentrating the Bk assignments on fewer active experts can increase tokens per expert and improve weight reuse and grouped computation.At batch size 1, each selected expert receives one token, so this batching opportunity is absent.Source: §D.3 · p. 26
  • The router-collapsing intervention is a synthetic diagnostic, not a quality-preserving inference configuration, so its evidence concerns batched efficiency.The authors interpret Figure 11 in light of the real evaluation setting, which uses batched inference.Source: §D.3 · p. 26

E.1 Ablation Study on Turn Labels

  • Operation labels provide the best turn-labeling scheme for improving agentic RL performance among operation, random, and application-name labels.Table 6 compares the labeling schemes at training step 199.Source: §E.1 · p. 26, Table 6 · p. 27

E.2 Histogram of the Number of Shifted Experts

  • With appropriate routing controls, RL training reduces expert shifts, making more tokens’ expert selections consistent with those of preceding tokens.Figure 12 shows high expert-shift counts becoming less frequent after training with routing_reg and switch_reg.Source: §E.2 · p. 26, Figure 12 · p. 27

E.3 Effect of R3 on Agentic RL

  • R3 stabilizes entropy dynamics on agentic tasks, but this stability does not improve performance when it comes at the cost of RL exploration.Removing R3 yields slightly better performance but introduces instability, including a risk of entropy explosion.Source: Figure 13 · p. 27, §E.3 · p. 27

E.4 Ablation Study on Entropy Loss

The entropy-loss ablation finds that changing total action entropy does not alter router behavior or inference efficiency.

E.4 Ablation Study on Entropy Loss

  • Changes in total action entropy from entropy loss have no observable effect on router behavior or inference efficiency.Figure 14 compares training curves with entropy loss.Source: §E.4 · pp. 27–28, Figure 14 · p. 28

E.5 Ablation Study on the Token-Level Control Threshold

These additional experiments identify a token-level control threshold and coefficient ranges associated with performance and training stability, distinguish the controls’ contributions, and assess rollout efficiency. The AppWorld example also records a discrepancy between the agent’s success counter and the environment’s final evaluation.

E.5 Ablation Study on the Token-Level Control Threshold

  • A token-level control threshold of hδ = 0.5 performs best; smaller thresholds reduce the framework to standard MoE-agnostic RL, while larger ones destabilize training.The threshold comparison uses AppWorld TGC with GRPO.Source: §E.5 · p. 28, Table 7 · p. 28

E.6 Hyperparameter Magnitude Search

  • Turn-level and token-level controls each have optimal coefficient ranges, and larger magnitudes require the entropy gate to prevent unstable router updates.The search evaluates the controls separately; larger magnitudes can otherwise lead to training collapse.Source: §E.6 · p. 28, Table 8 · p. 28, Table 9 · p. 29

E.7 Ablation Study on Routing Control Components

  • Turn-level control contributes more to task performance, while token-level control contributes more to inference efficiency.Table 10 reports the individual and combined component effects on AppWorld.Source: §E.7 · p. 29, Table 10 · p. 29

E.8 Ablation Study on Rollout Efficiency

  • Routing control improves rollout efficiency, with token-level control contributing especially strongly; both controls reduce single-trajectory rollout time and improve inference throughput.Figure 15 reports AppWorld rollout comparisons under GRPO.Source: §E.8 · p. 29, Figure 15 · p. 29

F.1 AppWorld Trajectory

  • In the 20-turn AppWorld example, the environment records successful completion with all six tests passed despite the agent’s success counter printing zero.A repeated like request reports that the transaction had already been liked.Source: §F.1 · p. 30

F.2 AutomationBench Trajectory

The AutomationBench trajectory documents an agent completing a four-vendor email and Pipefy workflow, including updating cards, marking related emails as read, and sending procurement confirmations. It also records vendor matching and operation labels, with all scored assertions passing and no tool errors.

F.2 AutomationBench Trajectory

  • Under the stated labeling rule, api_search is Search, GET api_fetch calls are Read, and POST api_fetch calls are Create; local base64 formatting and the final response have empty label sets.Each turn’s label set applies to both its response and tool-use fields.Source: §F.2 · p. 36
  • Score 1.0, with all 10/10 scored assertions passed and zero tool errors; three unscored negative assertions also passed, including leaving Northwind card_902 unmoved.The log records task_completed_correctly = 1.0.Source: §F.2 · p. 36
  • The agent retrieved four vendor-review emails, updated four Pipefy statuses and phases, marked all four messages as read, and sent four confirmation emails.Source: §F.2 · p. 36
  • After an initially empty title lookup, the agent found the vendor records in tbl_ops, selected NorthWind LLC’s card_905, and left the similarly named Northwind card_902 unmoved.The records also mapped Summit to card_903, Apex to card_904, and BlueYard to card_906.Source: §F.2 · pp. 36, 41
  • All 4 confirmation emails were sent successfully to procurement.The recorded responses show four sent messages.Source: §F.2 · p. 47
  • Pipefy cards were updated: Summit and Apex approved, NorthWind LLC rejected, and BlueYard set to Pending Docs.NorthWind LLC’s decision cited a failed background check; BlueYard’s was conditional pending insurance documents.Source: §F.2 · pp. 46–47
  • All 4 related emails were marked as read by removing their UNREAD labels.Source: §F.2 · pp. 43, 44, 47

Related papers

Loading 2610.07332v1…