CERA-MoA: Co-Evolving Routing Mechanisms with Continually Learning LLM Agents

1 Introduction

• Existing routing does not adapt to agents’ evolving post-training capabilities, while manually partitioned data limits competence-based sample allocation and synergistic specialization.The paper says this leaves agents acting as generalists rather than domain experts.Source: §1 · p. 1
• CERA-MoA co-evolves dynamic routing and agent learning, using a predictive familiarity estimator and cumulative-threshold routing to select a tailored minimal agent subset.The estimator uses intermediate hidden states to assess query–agent expertise alignment before generation.Source: §1 · pp. 1–2
• The framework dynamically allocates training queries according to evolving agent capabilities, encouraging distinct problem-solving specialization through continual reinforcement learning.Its sample-level routing is designed to adapt as individual capabilities shift.Source: §1 · pp. 1–2
• CERA-MoA outperforms static-agent routing and fixed-workflow fine-tuning baselines, with reported performance gains across diverse problem domains.Source: §1 · p. 2

2 Related Work

• Multi-agent systems coordinate autonomous models through debate, majority voting, or mixtures of independent agents, with orchestration optimization and agent fine-tuning forming two broad research directions.Unlike token-level Mixture-of-Experts, these systems coordinate at the sample and semantic level.Source: §2 · p. 2
• Orchestration research spans routing, budget, diversity, confidence, and topology methods, but generally assumes candidate models remain static during orchestration training.The paper identifies a missing mechanism for realigning query allocation as models evolve and specialize.Source: §2 · p. 2
• Agent fine-tuning uses varied objectives and methods, yet collaborative pipelines commonly retain predefined workflows and manually partitioned or uniform data rather than routing queries by evolving competence.CERA-MoA integrates iterative routing into training to allocate semantic queries by competence and foster specialization.Source: §2 · p. 2

3.1 Overview

• During training, the router allocates each query to a tailored agent subset for independent reinforcement-learning updates; at inference, it routes to a minimal competent subset and aggregates responses by familiarity.For deterministic tasks, the aggregator uses familiarity-weighted majority voting; for open-ended tasks, it returns the most familiar agent’s completion.Source: §3.1 · pp. 2–3

3.2 Predictive Familiarity Estimator

• The predictive familiarity estimator maps fixed intermediate hidden-state features through trainable predictor heads and frozen target heads, using projection distance to produce reward-aligned familiarity scores.The shared backbone remains fixed for this objective; predictor heads adapt as policies and routed data evolve.Source: §3.2 · pp. 3–4

3.3 Cumulative-Threshold Adaptive Routing

• During training, exploration and historical-entropy terms rank agents, but the smallest ranked subset whose cumulative familiarity reaches threshold τ is activated; inference disables exploration.If the total familiarity remains below τ, the query is allocated to the entire population.Source: §3.3 · p. 4

3.4 Joint Optimization of Router and Agents

• The closed-loop protocol updates selected agents with DAPO and sequence-level importance sampling while updating the router from participating agents’ relative rewards to align familiarity with competence.Source: §3.4 · pp. 4–5
• The framework supports either independent LoRA adapters on a shared backbone or heterogeneous base models.Source: §3.4 · p. 5

4 Experiments

• The evaluation asks whether CERA-MoA beats routing and fixed-workflow baselines, adapts to heterogeneous models, improves query allocation, balances activation and performance, and induces specialization.The experiments assess both ID and OOD performance.Source: §4 · p. 5

4.1 Experimental Setup

• Experiments span math, code, instruction-following, and general-reasoning tasks, comparing routing and fine-tuning baselines using four-agent shared-model and three-model heterogeneous settings.OOD evaluation uses separate benchmarks without additional fine-tuning; the shared-model setup uses independent LoRA modules, while the heterogeneous ensemble combines three open-source models.Source: §4.1 · p. 6

4.2 Main Results (RQ1)

• CERA-MoA achieves the highest average performance across three base models on in-distribution and out-of-distribution tasks, including the heterogeneous ensemble.The reported OOD transfer scores are also highest; the supplied passages provide no numerical table values.Source: Table 2 · p. 6, §4.2 · p. 6, §4.3 · p. 6

4.3 Adaptation to Heterogeneous Models (RQ2)

• Across Qwen3-4B, Llama-3.2-3B, and Phi-4-mini, CERA-MoA outperforms baselines on both ID tasks and OOD transfer.The authors report gains on challenging reasoning datasets including DAPO-MATH, LogicBench, and OlympiadBench.Source: §4.3 · pp. 6–7

4.4 Ablation on Familiarity Score (RQ3)

• Direct reward regression and reward-driven multi-class classification both fall short of CERA-MoA, while familiarity scores evolve smoothly as estimated rewards oscillate.The authors attribute the difference to reward instability as agent policies change during training.Source: §4.4 · p. 7, Figure 2 · p. 7

4.5 Ablation on Adaptive Routing (RQ4)

• Compared with fixed Top-2 routing, CERA-MoA reduces average generation tokens by approximately 45% while retaining comparable performance.The router generally activates multiple agents for complex problems beyond an individual agent’s expertise; fixed Top-1 reduces overhead but incurs a performance drop.Source: §4.5 · p. 7

4.6 Capability Specialization Analysis (RQ5)

• The four Qwen3-4B agents differentiate into complementary roles, and queries routed to the same agent form clustered semantic neighborhoods.Agent 2 becomes the primary solver, while Agents 1, 3, and 4 specialize in concise math and logic, coding, and general reasoning, respectively.Source: §4.6 · p. 7, Figure 3 · p. 7

• CERA-MoA co-evolves query routing with continually learning agents, balancing efficiency and reasoning accuracy while inducing specialization and outperforming static-routing and fixed-workflow baselines.Its routing uses familiarity scores based on hidden-state semantics and cumulative thresholds.Source: §5 · p. 7
• Extending single-turn semantic routing to multi-turn interactions or multi-agent debate is identified as future work.The authors also propose upgrading familiarity-weighted voting to advanced answer aggregation, such as an adaptive meta-thinker.Source: §5 · p. 7

A.1 Co-Evolutionary Training Procedure

• The implementation supports shared-backbone LoRA agents or agents with distinct base models, while keeping the router backbone frozen.The shared-backbone setting uses four LoRA modules; heterogeneous agents use separate base models.Source: §A.1 · p. 10
• CERA-MoA extracts fixed mid-layer query features and uses trainable predictor heads against frozen target heads to estimate agent familiarity.The router backbone and target heads remain frozen; only agent policies and predictor heads are updated.Source: §A.1 · p. 10
• Agents are ranked with exploration-augmented scores, but subset activation tests raw familiarity against τ and activates all agents if the threshold is unreachable.Activated queries enter per-agent LIFO buffers for policy training.Source: §A.1 · pp. 10–11
• Each agent optimizes its policy on allocated samples, while the router updates from agents’ relative rewards using a bounded separation margin.Positive relative advantages pull predictor heads toward fixed anchors; negative advantages push them away, capped at margin m.Source: §A.1 · p. 11

A.2 Inference and Consensus Aggregation

• At inference, exploration bonuses are disabled and the router selects the minimal subset satisfying τ; deterministic tasks use familiarity-weighted voting, while open-ended tasks use the most familiar agent.Each selected agent generates one completion.Source: §A.2 · pp. 11–12

B.1 Hardware and Software Infrastructure

• Training requires approximately one H200 GPU day for 1,000 optimization steps, processing 16,000 queries with G = 8 rollouts per query.Memory use is managed with gradient checkpointing and co-located vLLM rollout engines.Source: §B.1 · p. 12

B.2 Hyper-Parameter Configuration

• The router samples 16 distinct prompts per orchestration round, while each agent updates only after its LIFO buffer fills a 16-query micro-batch.This makes per-agent update frequency adaptive to routed sample allocation.Source: §B.2 · p. 12

B.3 Prompt Design and Zero-Shot Task Instructions

• Prompts contain raw queries without role descriptors or behavioral hints, and task-family templates preserve verifier-defined instruction constraints.Code prompts support function-call and standard-input/output interfaces.Source: §B.3 · p. 12, Figure 4 · p. 14

C Task-Specific Reward Formulation

• Rollout rewards combine a task score with a length penalty that begins after γLmax and reaches η at the completion limit.Reported accuracy depends only on whether the unpenalized task reward equals 1.Source: C Task-Specific Reward Formulation · p. 13
• Math verification extracts the final boxed expression and checks normalized predictions against references using exact and symbolic equivalence tests.A timeout isolates symbolic checking; equivalent expressions can pass when symbolic parsing succeeds.Source: C Task-Specific Reward Formulation · p. 13
• Code-generation reward is the fraction of task tests passed by the extracted program; compilation failures, timeouts, malformed outputs, and missing code score zero.Evaluation supports function-call and stdin/stdout tasks, with execution isolated under a 10-second limit.Source: C Task-Specific Reward Formulation · p. 13
• Instruction-following reward is the fraction of verifier constraints satisfied, with a strict alternative requiring every constraint to pass.The verifiers are reconstructed from IFEval instruction identifiers and arguments.Source: C Task-Specific Reward Formulation · p. 13
• General-reasoning rewards use exact match after canonicalizing extracted answers, including prefixes, parentheses, choices, and whitespace.The procedure applies task-agnostic normalization before comparison.Source: C Task-Specific Reward Formulation · p. 13

D.1 Source Datasets

• The described math data include GSM8K, MATH, and DAPO-MATH-17K, covering grade-school word problems, competition problems, and verifiable symbolic or quantitative tasks.The cited dataset passages report both training and test splits.Source: §D.1 · p. 14
• Test-only evaluation examples include IFEval, HumanEval, AGIEval, ARC-Challenge, LogicBench, and OlympiadBench across instruction-following, code, general reasoning, and math or science reasoning.These cited benchmark passages describe evaluation splits without training data for the method.Source: §D.1 · p. 15

D.2 Construction of Training and In-Distribution Test Sets

• The training corpus contains 32,000 examples across math, code generation, instruction following, and general reasoning, sampled with a fixed seed.The examples are converted into a unified schema with task-specific verifier metadata.Source: §D.2 · p. 15
• In-distribution results use held-out test splits from the nine source datasets used for training, evaluated separately by dataset.The specified families include math, code generation, instruction following, and BIG-bench Hard general reasoning.Source: §D.2 · p. 15

D.3 Out-of-Distribution Evaluation

• Six benchmarks excluded from training test out-of-distribution generalization while retaining the trained router, adapters, prompts, and aggregation rule.The benchmarks cover instruction following, code generation, general reasoning, and mathematical or scientific reasoning.Source: §D.3 · p. 15

E.1 Router Computational Overhead Analysis

• The CERA-MoA router averages 11.87 ms per query, less than 1% of native generation time, because familiarity evaluation extracts intermediate hidden states with one partial forward pass.Table 6 profiles computational overhead on a single NVIDIA H200 GPU.Source: §E.1 · p. 15, Table 6 · p. 15

E.2 Co-Evolutionary Learning Dynamics

• System reward rises and converges stably as agent workloads differentiate by domain, while familiarity scores rise within specialized clusters.Agent 2 becomes the primary reasoning engine, Agent 3 specializes in code, and Agents 1 and 4 supplement other task areas; all continue to evolve.Source: §E.2 · p. 16, Figure 5 · p. 16, Figure 6 · p. 16

Cumulative Threshold τ

• ID 61.2 and OOD 71.5 are the peak results reported for a population trained with τ = 0.8, whose over-activation diluted specialization feedback.The authors report that the stricter training threshold prevented sharp domain experts from emerging.Source: Table 8 · p. 16, §E.3 · p. 16

Familiarity Temperature λ

• ID 55.3, OOD 59.8, and Avg. Tokens 463.72 at λ = 1.0 compare with ID 63.2, OOD 72.8, and Avg. Tokens 367.77 at λ = 2.0.Figure 8 tracks familiarity scores; at λ = 1.0 they reach τ = 0.7 too early, limiting exploration.Source: §E.3 · p. 16, Figure 8 · p. 17

Separation Margin m

• ID 59.2, OOD 70.7, and Avg. Tokens 364.74 are reported for the increased separation margin m = 2.0.Figure 9 is the familiarity progression under that margin; the authors report at least one agent’s familiarity fails to grow effectively.Source: Figure 9 · p. 17, §E.3 · p. 17
• Without a separation margin, familiarity scores fluctuate violently and fail to grow; the authors attribute this instability to unbounded negative-gradient shifts.Figure 10 shows progression without a protective separation margin.Source: Figure 10 · p. 17, §E.3 · p. 17

E.4 Ablation on Routing and Exploration Mechanisms

• The ablations test whether competence-based allocation and active exploration can avoid fixed-size routing and policy starvation.The quantitative results for routing allocation and exploration are summarized in Table 9.Source: §E.4 · p. 17, Table 9 · p. 17
• Fixed Top-3 training achieves comparable accuracy but costs 817.97 tokens per query, regardless of prompt difficulty.Under this setup, familiarity scores oscillate rather than grow steadily, impairing confidence calibration and inference routing.Source: §E.4 · p. 17, Figure 11 · p. 17
• Without UCB exploration and historical entropy, performance degrades across all metrics and routing collapses onto Agent 3, starving Agents 1 and 2.The ablation results are summarized in Table 9.Source: §E.4 · p. 17, Table 9 · p. 17

E.5 Ablation on Hidden State Extraction

• The final-layer variant suffers a notable degradation in both ID and OOD accuracy, accompanied by an unjustified surge in token generation.Figures 13, 14, and 15 visualize internal routing metrics under the final-layer configuration, diagnosing this failure mode.Source: §E.5 · p. 18
• At the same threshold (τ = 0.7), slower-growing final-layer familiarity scores force the router to activate a larger agent subset during training.This larger subset dilutes targeted specialization feedback.Source: §E.5 · p. 18, Figure 13 · p. 18, Figure 14 · p. 18
• Final-layer features form entangled rather than cleanly separable expertise clusters, which the authors attribute to token-specific predictive distributions and autoregressive interference.The authors contrast these features with mid-layer states that preserve broad, structurally rich semantics (Skean et al. 2025).Source: §E.5 · p. 18, Figure 15 · p. 18

F Limitations

• Familiarity is assessed from the initial prompt, limiting the current routing setup to single-turn or fixed-trajectory interactions when expertise may shift in longer workflows.The proposed extension evaluates updated-context hidden states at each interaction step.Source: F Limitations · p. 18
• For open-ended benchmarks such as MBPP and HumanEval, the protocol returns the most familiar agent’s response rather than combining outputs from multiple activated experts.This reduces computational overhead but bypasses multi-agent generation; diverse candidate synthesis could yield a more robust output.Source: F Limitations · pp. 18–19
• A generative aggregator, such as an LLM-based meta-thinker, is proposed to synthesize selected experts’ completions and is described as architecturally compatible.This is an extension, not the current aggregation protocol.Source: F Limitations · pp. 18–19