PaperlayerView exact revision on arXiv

Source-linked AI summary

ACLArena: Agent Continue Learning in Multi-stage Post-training

Haixin Wang, Xiaoxuan Wang, Junkai Zhang, Han Zhang, Renliang Sun, Alexander K Taylor, Yidan Shi, Haoran Deng, Chenguang Wang, Jason Cong, Yizhou Sun, Wei Wang

arXiv:2609.23989v1cs.AI

TL;DR

Agents trained across multiple stages must acquire new capabilities while retaining earlier ones, yet ACL lacks a well-established recipe and clear understanding of integration trade-offs. ACLArena studies sequential post-training, compares capability-integration strategies, and proposes routed LoRA experts specialized through RL. Its analyses identify persistent trade-offs and complementary SFT and RL roles, while the proposed approach substantially mitigates forgetting and improves capability integration.

  • Problem

    Agent continual learning lacks a well-established recipe and a clear understanding of trade-offs among approaches for integrating capabilities acquired at different training stages.

    Source: §1 · p. 2

  • Method

    ACLArena analyzes sequential post-training at model and token levels, compares distillation and model merging strategies, and proposes SDFT with routed LoRA experts refined through RL.

    Source: §1 · p. 2, §8 · p. 13

  • Results

    The analyses reveal partially aligned task updates, persistent capability trade-offs, and complementary SFT and RL roles; MLE substantially mitigates forgetting and consistently improves capability integration.

    Source: §1 · pp. 2–3

  • Takeaways & Limitations

    MLE combines task-specific adaptation with expert routing in one model, and the paper reports competitive performance across heterogeneous tasks with improved capability integration.

    Source: §1 · pp. 2–3

    Experiments focus on Qwen3-8B-Base and a fixed four-stage curriculum, leaving generalization across model families and training orders unclear.

    Source: §7 · p. 13

Abstract

from arXiv · show

Building general-purpose agents for industrial deployment requires integrating multiple capabilities, each typically acquired at a distinct stage of training. Yet there is currently no well-established recipe for Agent Continual Learning (ACL), with little understanding of the trade-offs among existing integration paradigms. To address this gap, we introduce ACLArena, a framework for comprehensively studying, analyzing, and evaluating ACL. We first build a sequential training pipeline and conduct an in-depth analysis that explains the mechanisms of forgetting and generalization from two complementary perspectives, the model level and the token level. Guided by these analyses, we systematically compare multi-teacher on-policy distillation, self-distilled fine-tuning, and model merging to assess their ability to recover previously learned capabilities while preserving newly acquired ones. Through extensive experiments, we develop a detailed understanding of how capabilities transfer across stages. Finally, we propose a new ACL recipe that combines offline replay over high-quality trajectories with a routed network of multiple LoRA experts each specialized via RL, substantially improving the agent's ability to learn across multiple domains. Comprehensive experiments on four reasoning and agentic tasks, evaluated under both in-domain and out-of-domain settings, demonstrate the value of our analysis and the effectiveness of our approach.

1 Introduction

Multi-stage post-training makes agent continual learning a practical problem: agents must gain capabilities in new environments while retaining earlier ones, but the field lacks a realistic testbed and controlled comparisons of practical strategies. ACLArena analyzes sequential training and compares integration approaches, then proposes a routed LoRA-expert recipe guided by findings about task interference and the complementary roles of SFT and RL.

1 Introduction

  • Agents face a continual-learning problem because they must develop capabilities in new environments while retaining capabilities acquired earlier.The paper frames this in the context of capabilities such as chain-of-thought reasoning, tool use, and instruction following acquired sequentially.Source: §1 · p. 2
  • ACLArena addresses the lack of a realistic testbed and controlled comparisons by constructing a multi-stage pipeline, diagnosing forgetting and generalization, and comparing practical ACL strategies.Its contribution also includes a unified method that performs competitively across heterogeneous tasks and consistently improves capability integration.Source: §1 · pp. 2–3
  • The study asks how sequential post-training changes capabilities, what drives forgetting and transfer, and whether prior models can be reused to mitigate forgetting and promote transfer.The questions cover both parameter-space reuse and behavior-space reuse through generated trajectories.Source: §1 · p. 2
  • Sequential training is the diagnostic baseline, followed by model-wise and token-wise analyses and comparisons of on-policy distillation, self-distilled fine-tuning, and model merging.The three strategies reuse prior models online in behavior space, offline through filtered specialist trajectories, and directly in parameter space, respectively.Source: §1 · p. 2
  • Task-specific post-training produces partially aligned updates and persistent capability trade-offs, while SFT establishes valid task behavior and RL makes smaller, policy-local refinements.These findings motivate reconciling task-specific adaptation with capability consolidation while reducing cross-task interference.Source: §1 · p. 2
  • MLE combines rapid behavior learning through SDFT with RL-specialized LoRA experts and routing, serving multiple stages within one model while substantially mitigating forgetting.Each stage yields its own experts, which the routing mechanism composes for tasks across stages.Source: §1 · p. 2

2 Related Work

The related work covers three ways of consolidating specialist capabilities: multi-teacher on-policy distillation, self-distilled fine-tuning, and model merging. They differ in whether they reuse behavior online, replay filtered trajectories offline, or combine model weights.

2.1 Multi-teacher OPD

  • Recent multi-teacher OPD approaches use student-generated trajectories for dense supervision from stronger teachers, with examples spanning capability consolidation and refinement.Examples include MiMo-V2, GLM-5, Nemotron-Cascade 2, and DeepSeek-V4; Qwen3 uses student-generated responses with KL-divergence minimization against a teacher.Source: §2.1 · p. 3

2.2 Self-distilled Fine-tuning

  • SDFT consolidates specialist capabilities by replaying filtered teacher-generated traces and then applying alignment-oriented RL, offering a simpler and more stable recipe than OPD.It relies more heavily on data-mixture design and provides weaker on-policy correction than OPD.Source: §2.2 · p. 3

2.3 Model Merging

  • Rio 3.5 Open 397B illustrates a merge-then-distill pipeline, combining model merging in weight space with subsequent OPD from a stronger teacher.Its documentation describes merging Nex-N2-Pro with Qwen3.5-397B-A17B before the distillation stage.Source: §2.3 · p. 3

3 Diagnosing Baseline: Sequential Training

The sequential-training baseline follows a fixed Math→Search→E-commerce→IF curriculum and evaluates how performance changes across stages. It reveals positive early transfer alongside substantial, task-dependent forgetting; model- and token-level analyses link these patterns to partially aligned updates and uncertainty-sensitive prediction changes.

3.1 Multi-stage Task Curation

  • The fixed curriculum proceeds from mathematical reasoning to single-tool search, then E-commerce with up to fifteen tools, and finally instruction following.Math is used to strengthen reasoning before agentic tasks, and IF is introduced last to improve instruction adherence and alignment.Source: §3.1 · p. 4

3.2 Sequential Training Formulation

  • Each stage trains the current policy on its own prompt distribution and reward, with an environment added for agentic stages; the sequential process yields successive checkpoints.The formulation distinguishes the stage-specific input distribution from the agentic environment.Source: §3.2 · p. 4

3.3.1 Main Results

  • E-commerce training reduces AIME from 23.33 to 6.04 and NQ from 45.2 to 14.6; the final Seq-IF checkpoint recovers NQ to 33.5 and reaches 84.8 on IF-Eval.Multi-hop search drops from 37.4 to 9.4 during E-commerce training and recovers to 25.0 after Seq-IF; earlier Seq-Math training raises NQ from 13.3 to 22.0 before search-specific training.Source: Table 1 · p. 4, §3.3.1 · p. 5

3.3.2 Capability Dynamics: Forgetting and Transfer

  • Every capability degrades after its own training stage, but the severity varies: search falls sharply, math declines and then collapses, and E-commerce capability is comparatively durable.The final checkpoint remains below earlier peaks for AIME26, NQ, and τ 3-Retail, and no capability improves monotonically after its instilling stage.Source: §3.3.2 · p. 5

3.3.3 Model-level Diagnosis: Parameter Displacement.

  • The single-task oracle updates occupy distinct, only partially aligned parameter directions, and accumulated cross-stage interference makes later checkpoints harder to align with any one task oracle.Each stage moves weights toward its corresponding oracle, but no checkpoint aligns perfectly with it.Source: §3.3.3 · p. 6

3.3.4 Token-level Diagnosis: Prediction Stability across Stages

  • Under fixed-prefix evaluation, most low-entropy token positions remain stable across later stages, while prediction changes concentrate in the smaller high-entropy minority.The analysis forces subsequent checkpoints to score the same token sequence decoded by Seq-Math and measures whether each checkpoint still ranks its token first.Source: §3.3.4 · p. 6
  • Task-specific training induces partially aligned updates, so sequential reconciliation of those updates leads to uneven transfer and forgetting across tasks.Source: §3.3.4 · p. 6

4 Method

The section presents three routes for consolidating capabilities across post-training stages: multi-teacher on-policy distillation, self-distilled fine-tuning, and model merging.

4 Method

  • The three integration routes reuse prior models in behavior space online, behavior space offline, or parameter space.The section introduces these as complementary technical routes for model integration.Source: §4 · p. 6

4.1 Multi-teacher On-policy Distillation

  • Mixed OPD retains reverse KL for sharp imitation but adds top-q-truncated forward KL when teacher uncertainty is high, targeting decision points where mode-seeking can prune viable strategies.The forward-KL term is evaluated over the teacher’s top-q tokens to avoid noisy low-probability-tail supervision while keeping overhead negligible.Source: §4.1 · pp. 6–7
  • Multi-teacher OPD trains a student initialized from the last checkpoint on a balanced mix of stage prompts, routing each rollout to its domain-owning teacher for token-level scoring.Frozen teachers score student-generated trajectories; the additional cost is a constant number of forward passes per trajectory, independent of the number of capabilities consolidated.Source: §4.1 · p. 7

4.2 Self-distilled Fine-tuning

  • SDFT trains task-specific oracle policies from the base model, filters their high-quality trajectories, and uses a task-mixing distribution to train a consolidated student.Its effectiveness depends on domain mixing and data order: imbalance can suppress under-represented capabilities, while strictly ordered data can reintroduce forgetting.Source: §4.2 · pp. 7–8

4.3 Model Merging

  • Model merging combines specialized checkpoint weights through task-vector displacements from a shared base, with coefficients that can be tuned to balance capabilities.The section compares uniform, grid-searched, and capability-balanced weighting schemes.Source: §4.3 · pp. 8–9

5 Exploration of Different ACL Paradigms

The section compares practical ACL strategies and analyzes how they affect capability retention, parameter updates, and the distinct roles of SFT and RL.

5.1 Experiment Setup

  • The evaluation covers in-domain and out-of-domain math, search, E-commerce, and instruction-following tasks, using independently trained per-task oracle experts and several consolidation implementations.Experiments use Qwen3-8B-Base; task-specific benchmark assignments and implementation details are described in the setup passages.Source: §5.1 · p. 9

5.2 Main Results

  • MMOPD improves AIME26 from 10.21 to 21.25, NQ from 33.5 to 45.2, single-hop search from 44.9 to 53.7, and multi-hop search from 25.0 to 32.8 versus Sequential.It largely preserves IF performance, but slightly lowers MMLU and IF-Bench; the final sequential checkpoint also retains later-stage performance while AIME26 falls to 10.21.Source: §5.2 · p. 9
  • No consolidation method improves every domain: SDFT recovers search but degrades E-commerce and IF, while merging's MMLU result of 80.3 is strongest but lags on most E-commerce and IF benchmarks.The comparisons indicate that recovering one capability can interfere with others in a shared checkpoint.Source: §5.2 · p. 9

5.3.1 Single-teacher OPD

  • Single-teacher OPD substantially restores forgotten search capability, surpasses the original Search teacher, and largely preserves E-commerce capability.The study selects Search because it exhibits the highest degree of forgetting.Source: §5.3.1 · p. 10, Table 3 · p. 10

5.3.2 SDFT Replay Variant

  • SDFT initialized from the sequential checkpoint yields better overall learning than base-model initialization, with the largest differences on the two agentic tasks.Math and instruction-following differences are marginal; the comparison uses oracle trajectories collected from the corresponding policy.Source: §5.3.2 · p. 10, Table 4 · p. 10

5.3.3 SDFT Data Recipe

  • Balanced-Mix's uniform task sampling stabilizes gradients across differently directed tasks, reduces loss faster and lower, and achieves better average performance than Random-Mix.Random-Mix samples in proportion to trajectory-set size.Source: Table 4 · p. 10, §5.3.3 · p. 11

5.4.1 Magnitude and Direction of Different Post-Training Pipeline

  • SFT updates are typically 3–7× RL and MMOPD updates; SDFT moves farther from the sequential checkpoint than MMOPD and has cross-initialization update cosine 0.77.The figure compares parameter-space positions and relative update magnitudes; unrelated SFT tasks have cosine similarity 0.05.Source: §5.4.1 · p. 11, Figure 3 · p. 11

5.4.2 Complementary Roles of SFT and RL

  • SFT establishes valid task behavior, while RL builds on it to improve reasoning, decision making, and task competence; direct RL without prior SFT lowers valid-format rate and increases truncation.SFT teaches the task interface, valid output format, and basic action space.Source: §5.4.2 · p. 12

6 Ours - Mixture of Low-Rank Experts

MLE combines a shared trajectory-trained capability substrate with stage-specific RL LoRA experts, preserving specialization while sharing most model parameters.

Motivation

  • MLE is motivated by partially misaligned task updates: it consolidates transferable behaviors in a shared backbone while preserving interfering stage-specific residuals separately.The motivation draws on SDFT's consolidation of heterogeneous trajectories and RL's smaller, more policy-local updates.Source: §6 · p. 12

Architecture

  • MLE starts from an SDFT checkpoint trained on multi-domain trajectories, freezes that backbone, and trains one environment-specific LoRA expert with RL for each stage.Each stage policy is represented as θk = θshared + ∆θk, with a low-rank adapter parameterizing the residual.Source: §6 · p. 12
  • At inference, observable environment context selects the expert without benchmark identities or ground-truth task labels, and adding a stage introduces only a small additive parameter cost.The separate low-rank subspaces preserve incompatible specializations rather than seeking one shared-weight compromise.Source: §6 · p. 12

Results

  • MLE approaches independently trained task experts across in-domain benchmarks and reports the highest evaluated scores on GPQA, single-hop search, multi-hop search, and the mock domain out of domain.It addresses complementary weaknesses in which MMOPD performs well on E-commerce but moderately on search, while SDFT favors search over E-commerce.Source: §6 · p. 12

7 Limitations & Future Work

The evaluation's scope and MLE's routing assumptions leave open how broadly the findings and method generalize.

Limitations & Future Work

  • Experiments focus on Qwen3-8B-Base and a fixed four-stage curriculum, so generalization of the observed forgetting patterns and method trade-offs across model families and training orders remains unclear.The authors identify additional environments, longer sequences, and alternative curricula as future tests.Source: §7 · p. 13
  • MLE selects experts using observable environment context; effectiveness with ambiguous or changing context, or when one episode requires multiple specializations, remains untested.The authors propose learned routing and dynamic expert composition as future directions.Source: §7 · p. 13

8 Conclusion

The conclusion synthesizes ACLArena's analysis of cross-stage forgetting and transfer with MLE, a combination of SDFT and LoRA-based RL experts intended to preserve task-specific adaptation.

8 Conclusion

  • The study identifies partially aligned task updates, persistent capability trade-offs, and complementary SFT and RL roles, then proposes MLE to preserve task-specific adaptation and reduce interference.MLE combines SDFT with LoRA-based RL experts.Source: §8 · p. 13

H More Details on Token-level Analysis

The appendix specifies the multi-stage setting, training conventions, and task environments, then compares OPD variants and explains why vanilla OPD can collapse. It also describes the shared RL backbone and the different ways consolidation methods reuse prior models.

A ACL Notations

  • At deployment, system instructions and tool schemas are observable, but benchmark identities and ground-truth task labels are not provided.The setting studies preserving and composing capabilities across sequential post-training stages, rather than inferring latent task identity.Source: A ACL Notations · p. 17

B Training Details

  • Per-task experts train independently from the base, sequential checkpoints inherit earlier stages, and MLE adds domain-specific RL-trained LoRA experts to SDFT.MMOPD and SDFT start from the final sequential checkpoint; model merging averages the four sequential checkpoints without training.Source: Table 5 · p. 18

B.1 Tool-calling Format

  • Tasks use a unified Qwen3 tool-call format: JSON-schema tools in prompts and JSON invocations and responses within dedicated tags.Source datasets are converted to the schema, with malformed or unparseable instances filtered out.Source: §B.1 · p. 18

C.1 Failure of Vanilla OPD

  • OPD’s truncate ratio reaches 100%, while its valid ratio and final test-set performance fall to 0; mixed OPD remains stable and sharpens critical decisions.The comparison concerns distilling Search back into E-commerce; reverse-KL OPD collapses, while forward-KL OPD is stable but has limited distillation capacity.Source: Figure 4 · p. 19, §C.1 · p. 21

B.2 Policy Optimization Algorithms

  • GSPO clips one length-normalized importance ratio per response, aligning importance sampling with trajectory-level rewards to stabilize long-horizon RL.The ratio aggregates per-token importance ratios; all tokens in a trajectory share the same clipped update.Source: §B.2 · p. 19

B.3 Task & Environment Details

  • Training covers integer-answer math, retrieved question answering, retail-domain API dialogue, and verifiable instruction following.Search uses three retrieved passages; e-commerce training uses retail while telecom and mock are held out, and instruction-following rewards rely on executable verifiers.Source: §B.3 · pp. 20–21

C.2 Implementation Details

The implementation details report trajectory sampling for SDFT and a weight sweep for linear model merging. The merge results show that emphasizing one checkpoint shifts capability trade-offs rather than removing interference.

C.2 Implementation Details

  • OPD uses 3,000 prompts per domain and stops at 100 steps after the three task rewards converge.Its setup uses γ = 0.5, rollout batch size 32, group number 8, global batch size 256, entropy threshold 0.3, and top-q 16.Source: §C.2 · p. 21

D More details on SDFT

  • SDFT samples 16 rollouts per prompt, filters them into high-quality trajectories, and trains until loss convergence at 1k steps.The resulting sets contain 156,285 math, 157,750 search, 19,573 cs, and 225,681 IF trajectories.Source: D More details on SDFT · p. 21
  • Figure 6 reports pairwise cosine similarities between domain task vectors, defined as checkpoint displacements from the shared base model.The supplied passage identifies the measurement but does not report its values.Source: Figure 6 · p. 22

E More details on Model Merging

  • Uniform merging has the highest average performance at 31.63, while Search-heavy merging reaches 44.5 on search and E-commerce-heavy merging reaches 17.56 on e-commerce.IF-heavy merging reaches 79.5 on IF; no single linear weighting scheme consistently preserves strong performance across all tasks.Source: E More details on Model Merging · p. 22

F Evaluation Details

Evaluation uses consistent inference settings across models, with task-specific decoding and interaction limits. Retrieval-based search returns three passages, and τ 3-Bench uses the basic split and a bounded interaction budget.

F Evaluation Details

  • Search allows five interaction turns and 2,048 new tokens per turn, using E5 and FAISS over Wikipedia-18 to return the top three passages.For τ 3-Bench, evaluation uses the basic split, up to 200 interaction steps, and GLM-4.7-Flash user simulation.Source: F Evaluation Details · p. 22
  • Most models use greedy decoding at temperature 0; MMLU-Redux uses chain-of-thought prompting at temperature 0.7, while GPQA-Diamond uses temperature 0.AIME 2026 samples 16 responses per problem at temperature 1.0 and top-p 0.7, reporting avg@16.Source: F Evaluation Details · p. 22
  • Generation limits are 16,384 tokens for MMLU-Redux, 30,000 for GPQA-Diamond and IF evaluations, and 8,192 for τ 3-Bench.Models are served with SGLang using mem-fraction-static=0.85 and the Qwen tool-call parser.Source: F Evaluation Details · p. 22

G More Details on Model-level Analysis

The appendix probes task-vector geometry and token-level prediction consistency across sequential training. It reports that task-specific updates differ, while token consistency varies with entropy and declines along the training sequence.

H.2 Splitting Positions

  • Figure 7 shows that low-entropy majority positions remain almost exactly preserved, while the high-entropy minority degrades monotonically across training stages on all three benchmarks.The split is at H = 0.5; consistency stays above 0.97 below 0.2, is around 0.92 in [0.2, 0.5), then falls steeply in higher-entropy bins.Source: Figure 7 · p. 23, §H.2 · p. 24

G.1 Stronger Evidence

  • Figure 6 compares pairwise cosine similarities of domain task vectors, each defined as an endpoint’s displacement from the shared base checkpoint.The vectors are cumulative along the sequential training chain, so their pairwise similarity must be interpreted alongside the isolated-increment comparison.Source: §G.1 · p. 23
  • After removing the shared sequential prefix, isolated domain RL increments are mutually near-orthogonal (|cos| ≤0.004), despite high cosine values between some cumulative vectors.The cumulative E-commerce and IF vectors have cos = 0.98 because IF is trained atop the service checkpoint; these cosines reflect accumulated training geometry, not intrinsic task similarity.Source: §G.1 · p. 23

G.2 Beyond a Particular Curriculum

  • Independent oracle experts also show heterogeneous cross-domain transfer, indicating that capability conflict is not solely due to training order.The sequential curriculum determines how interference manifests, but the cited results provide order-independent evidence of distinct and potentially conflicting optimization effects.Source: §G.2 · p. 23

H.1 Measurement Protocol

  • Token consistency is measured by running checkpoints over identical token-id prefixes generated by the math checkpoint, then counting unchanged top-1 predictions.The token-weighted low-entropy positions make up 95.2% of AIME, 93.6% of GPQA, and 91.9% of IF-Eval positions.Source: §H.1 · p. 24, §H.2 · p. 24

H.3 Reading the Per-bin Table

  • Table 7 shows consistency falling both as entropy rises and along the training order, from Seq-SeaSFT ≈ Seq-Search to Seq-IF.The shaded aggregates plotted in Figure 7 are token-weighted over the bins above them; bins marked † contain fewer than 500 positions and are unreliable.Source: §H.3 · p. 24, Table 7 · p. 25
Loading 2609.23989v1…