PaperlayerView exact revision on arXiv

Source-linked AI summary

BACAM: Behavior-Aware Continual Agent Merging for Multi-Turn Interaction

Shuaitong Li, Baochen Xiong, Xiaoshan Yang, Xizhe Zheng, Yifan Xu, Jianhao Huang, Changsheng Xu

arXiv:2610.04966v1cs.AI

TL;DR

Continual agent merging must integrate experts as they arrive while acquiring incoming behavior and preserving previously integrated interactive capabilities. BACAM learns parameter-wise gates from candidate-generated trajectories, with task- and tensor-level controls for acquisition and retention. Across four interactive tasks, it achieves a 62.82% average success rate, exceeding the strongest evaluated baseline by 21.69 percentage points in the default four-expert order.

  • Problem

    Existing agent-merging methods typically require all experts to be available together, and parameter- or feature-space merging does not ensure incoming behavior is acquired on merged-model trajectories while preserving earlier interactive behavior.

    Source: §1 · p. 2

  • Method

    BACAM learns parameter-wise merging gates from candidate-generated trajectories, using incoming-expert supervision for acquisition and current-model constraints and tensor-level budgets to limit interference.

    Source: §1 · p. 2, §3.1 · p. 3

  • Results

    62.82% average success rate in the default four-expert order, exceeding the strongest baseline by 21.69 percentage points across four interactive tasks.

    Source: §5 · pp. 9–10

  • Takeaways & Limitations

    The results support the promise of behavior-guided merging for interactive agents, while identifying robustness to expert merging order as a direction for future work.

    Source: §5 · p. 10

    The setting assumes experts share an architecture and tokenizer and retains task instances and environments for earlier tasks, although earlier experts are not retained.

    Source: §3.1 · p. 3

Abstract

from arXiv · show

Model merging offers a way to integrate the capabilities of specialized experts, but existing agent merging methods typically require all of them to be available at once. We study continual agent merging, which integrates incoming experts sequentially without retaining previously merged experts. Yet merging in parameter space or feature subspaces does not ensure that the merged model acquires an incoming expert's behavior on interaction trajectories. Moreover, updates toward a new expert can disrupt the merged model's previously integrated interactive behavior. Therefore, we propose Behavior-Aware Continual Agent Merging (BACAM), which learns parameter-wise merging gates from candidate-generated trajectories using expert-guided behavioral supervision. Task-level stability-plasticity control and tensor-level conflict-aware update budgets limit interference with existing capabilities while allowing new ones to be acquired. The learned gates are folded into the model weights without additional inference-time parameters. Across four interactive tasks - web shopping, tool use, information retrieval, and embodied interaction - BACAM achieves an average success rate of 62.82%, exceeding the strongest evaluated merging baseline by 21.69 percentage points. Our code is publicly available at https://github.com/shuaitongli/BACAM.

1 INTRODUCTION

Continual agent merging integrates incoming experts sequentially, but must acquire their interactive behavior while preserving capabilities already incorporated. BACAM learns parameter-wise gates with behavioral supervision and task- and tensor-level controls; it achieves the highest average success rate at each default-sequence stage and exceeds WUDI-Merging by 21.69 percentage points after four experts.

1 INTRODUCTION

  • Specialized agents exist for tool use, information search, embodied interaction, and web shopping, while current LLM-based agents often show limited cross-environment robustness.Joint multi-task reinforcement learning can train across environments but requires substantial engineering and computation.Source: §1 · p. 1
  • Continual agent merging integrates incoming experts into the current model without retaining earlier experts, unlike batch merging, which combines all experts at once.The paper formulates this setting for multi-turn interactive tasks and evaluates acquisition and retention over complete interaction trajectories.Source: §1 · p. 2, Figure 1 · p. 2
  • Parameter- or feature-space merging does not ensure that the merged model acquires incoming behavior on its own trajectories or preserves established interactive behavior.Because actions determine subsequent observations, parameter changes can redirect trajectories and disrupt previously integrated behavior.Source: §1 · p. 2
  • BACAM learns parameter-wise gates from candidate-generated trajectories, using task-level behavioral constraints and tensor-level conflict-aware budgets to balance acquisition and retention.The incoming expert guides new-task acquisition, while the current merged model constrains drift on earlier tasks.Source: §1 · p. 2
  • BACAM achieves the highest average success rate at each stage of the default merging sequence and exceeds WUDI-Merging by 21.69 percentage points after four experts.The evaluation covers four interactive tasks and tracks capability acquisition and retention across four arrival orders.Source: §1 · p. 2

2 RELATED WORK

Prior merging work addresses parameter combinations, agent-specific merging, and sequential model integration, but does not resolve continual merging for agents evaluated through full interactions. The paper identifies preservation of earlier interactive capabilities as an open issue for sequential methods.

2 RELATED WORK

  • Parameter-merging methods combine task updates through arithmetic, conflict handling, sparsification, decorrelation, or optimized coefficients, but do not directly assess multi-turn behavioral retention.The approaches discussed include Task Arithmetic, TIES-Merging, DARE, TSV-Merge, WUDI-Merging, AdaMerging, and Expert Merging.Source: §2 · p. 3
  • Agent-merging methods selectively aggregate task-specific regions, transfer role-conditioned neurons, or project new task vectors into behavior-critical null spaces, but assume jointly available source models.The paper discusses RAM, ARM, and M2A as examples.Source: §2 · p. 3
  • Continual model-merging methods integrate models sequentially, but primarily evaluate non-interactive prediction tasks, leaving preservation over full agent interactions unclear.The cited approaches include OPCM, DOP, MINGLE, and NUFILT; changed actions can redirect subsequent observations and decisions.Source: §2 · p. 3

3 BEHAVIOR-AWARE CONTINUAL AGENT MERGING

BACAM merges an incoming expert with the current model by optimizing parameter-wise gates on candidate-generated interaction trajectories. Task-level controls guide new-task acquisition while constraining drift on earlier tasks, and tensor-level budgets restrict updates where those goals conflict.

3.1 PROBLEM DEFINITION

  • The continual objective is to acquire the incoming expert’s capability while retaining the current merged model’s capabilities on every earlier task.The setting assumes ordered tasks, experts with a shared architecture and tokenizer, and retained unlabeled task instances and interaction environments even though earlier experts are not retained.Source: §3.1 · p. 3

3.2 FRAMEWORK OVERVIEW

  • BACAM constructs a candidate from the current merged model and incoming expert, then optimizes parameter-wise gates that interpolate between their weights.The current model and incoming expert remain frozen; each gate value selects a point between retaining the old-model parameter and selecting the incoming expert’s parameter.Source: Figure 2 · p. 4, §3.2 · p. 4

3.3 TASK-LEVEL STABILITY–PLASTICITY CONTROL

  • On candidate-generated histories, NPC uses forward KL to guide acquisition from the incoming expert, while OSC uses reverse KL to constrain behavioral drift from the current model on earlier tasks.The candidate interacts with the environment; teacher distributions are evaluated on the same candidate-generated prefix. Loss aggregation weights critical behavior-token groups and trajectories to avoid ordinary tokens or longer interactions dominating.Source: Figure 2 · p. 4, §3.3 · pp. 4–5

3.4 TENSOR-LEVEL CONFLICT-AWARE PLASTICITY BUDGETING

  • CAPB uses local gradient probes to estimate tensor conflicts and assigns smaller plasticity budgets where updates toward the incoming expert conflict with old-task stability.Budgets combine conflict direction with gradient magnitudes and old-task sensitivity; positive gate updates toward the new expert are restricted, while negative updates are not.Source: §3.4 · pp. 5–6
  • After gate optimization, BACAM folds the learned gate into model parameters without adding inference-time parameters or computation.The method applies the tensor budgets during gate updates before folding the gate into the model.Source: §3.4 · p. 6

4 EXPERIMENTS

Experiments compare BACAM with merging baselines across four interactive tasks and assess performance, order effects, parameter adaptation, and component ablations. BACAM leads the evaluated methods in average success rate, while results vary by task and merging order.

4.1 EXPERIMENTAL SETUP

  • The evaluation uses WebShop, tool use, information retrieval, and embodied interaction experts, with WTSA as the default merging order.Success rate is the primary metric; ALFWorld is evaluated on IID and OOD splits, and the comparison includes parameter-only, unlabeled-data, batch, and continual baselines.Source: §4.1 · pp. 6–7

4.2 PERFORMANCE COMPARISON

  • 62.82% Avg SR exceeds the strongest measured baseline, WUDI-Merging, by 21.69 percentage points after four experts.Table 1 reports final success rates; Avg averages five task columns.Source: §4.2 · p. 7, Table 1 · p. 7

4.3 CONTINUAL MERGING ANALYSIS

  • Final Avg SR ranges from 56.02% in SAWT to 62.82% in WTSA, with order effects varying by task.WebShop varies by 20.40 points across orders, while Search varies by 1.68 points; the initial expert’s own-task capability is largely preserved, but later-added WebShop and ALFWorld can retain gaps to expert scores.Source: §4.3 · pp. 8–9
  • Under TWAS, gate movement falls across successive integrations from 0.035 to 0.025 to 0.013, and MLP tensors account for 72–77% of total movement.LayerNorm receives the tightest plasticity budgets; adaptation also varies by tensor type and task, with WebShop protected more strongly than Tool in middle-layer value projections during ALFWorld integration.Source: §4.3 · p. 9

4.4 ABLATION ON EACH COMPONENT

  • 62.82% Avg SR with BACAM exceeds 60.89% with forward KL and 62.14% with reverse KL; uniform token weighting gives 61.40%.Uniform KL directions produce task trade-offs, while uniform token weighting has its largest decreases on Tool and ALFWorld OOD.Source: §4.4 · p. 9
  • Removing the tensor-level conflict budget reduces Avg SR from 62.82% to 61.58%, with Tool SR falling from 38.00% to 34.00%.Search and ALFWorld SR also decline, while WebShop rises slightly from 77.20% to 77.40%.Source: §4.4 · p. 9

5 CONCLUSION

BACAM sequentially merges incoming experts into the current model, using candidate-generated interaction behavior to balance new-task acquisition against retention. In the default four-expert order, it reaches 62.82% average success rate, 21.69 percentage points above the strongest baseline.

5 CONCLUSION

  • 62.82% average success rate in the default four-expert order exceeds the strongest baseline by 21.69 percentage points.The conclusion identifies robustness to expert merging order as a direction for future work.Source: §5 · pp. 9–10

A.1 CONTINUAL MERGING SETUP

  • Continual merging processes ordered tasks with shared-architecture experts, using the current merged model and incoming expert to acquire new capability while retaining current capability.The current merged model, rather than discarded historical experts, is the stability reference; degradation can accumulate along the sequence.Source: §A.1 · pp. 12–13

A.2 POLICY-INDUCED INTERACTION TRAJECTORIES

  • Candidate-generated trajectories connect parameter updates to interactive capability because an early action can change later observations and decisions.An incorrect action can redirect the subsequent interaction history.Source: §A.1 · p. 13, §A.2 · p. 13

A.3 GATE AND DUAL UPDATES

  • Each refresh round holds candidate trajectories and teacher distributions fixed during gate updates, while the local conflict probe estimates tensor budgets without updating optimizer or dual variables.The gate is restored before actual updates.Source: §A.3 · p. 13

A.5 PARAMETER INJECTION AND LOCAL CONFLICT CONTROL

  • A gate movement measures movement along the available expert-difference direction, not absolute parameter displacement or task importance.Coordinates where the two models agree cannot change through gate adjustment.Source: §A.5 · p. 15
  • Tensor budgets reduce only gate-update components that increase the incoming expert’s contribution; clipping keeps parameters between the two frozen models.Negative gate updates remain available for stability recovery, and the restriction bounds parameter injection rather than guaranteeing improved task success.Source: §A.5 · p. 15

A.4 DERIVATION OF THE BEHAVIORAL OBJECTIVES

The behavioral objectives use incoming-expert supervision for acquisition and current-model supervision for preserving earlier-task behavior. Their losses aggregate token and trajectory contributions with explicit weighting, while compressed distributions approximate the full-vocabulary losses.

A.4 DERIVATION OF THE BEHAVIORAL OBJECTIVES

  • Forward KL makes candidate probabilities track the incoming expert’s soft token targets on candidate histories, encouraging acquisition of the expert’s behavior.At the logit level, descent increases a token’s logit when its candidate probability is below the expert’s probability.Source: §A.4 · p. 14
  • Reverse KL uses the current merged model as the old-task teacher and discourages behavioral drift, weighting discrepancies differently from forward KL.The candidate probabilities also weight the discrepancy.Source: §A.4 · p. 14
  • Token aggregation gives critical and remaining token groups equal weight when both are present, and gives each trajectory equal total weight regardless of its length.The compressed distributions retain 32 tokens plus one remaining-vocabulary probability, so they approximate rather than exactly reproduce full-vocabulary losses.Source: §A.4 · p. 14

B.1 EVALUATION PROTOCOL

The evaluation protocol uses fixed held-out sets and benchmark-specific success criteria, with interaction length interpreted alongside task performance. The merging configurations distinguish BACAM’s candidate-trajectory training from baselines’ data access and optimization signals.

B.1 EVALUATION PROTOCOL

  • WebShop uses 500 synthetic goals over 1,000 products, BFCL v3 uses 100 held-out multi-turn tasks, MuSiQue uses 2,017 questions, and ALFWorld uses 140 IID and 134 OOD games.The same fixed evaluation sets are used across models.Source: §B.1 · p. 16, Table 3 · p. 16
  • Success rate reflects benchmark-specific criteria: full WebShop completion, complete BFCL instances, matched Search answers, and completed ALFWorld goals.For WebShop, task score credits partial constraint satisfaction, whereas success rate requires full completion.Source: §B.1 · p. 16
  • Round averages interaction steps across WebShop and ALFWorld episodes, including failures, so it must be read with success rate and, for WebShop, task score.A shorter Round can reflect early termination rather than successful efficiency.Source: §B.1 · p. 16
  • The main four-expert average weights five success-rate columns equally, including ALFWorld IID and OOD separately, rather than pooling their instances.Pooling would give Search more weight because its evaluation set is larger.Source: §B.1 · p. 16

B.2 MERGING CONFIGURATIONS

  • BACAM training states are decision points from candidate-generated trajectories; inputs are unlabeled and teachers supply token-distribution targets.The measured runs share optimization settings, including a Top-32 representation and distillation temperature 1.0.Source: §B.2 · p. 17, Table 4 · p. 17
  • Baseline implementations vary in task-instance access and optimization signal; no ground-truth response labels are used, and calibration data are separate from held-out evaluation.Data-collection and update schedules are method-specific, not budget-matched.Source: §B.2 · p. 17, Table 5 · p. 17

C.1 TWO- AND THREE-EXPERT RESULTS

The section compares success rates after merging two- and three-expert subsets, separating smaller expert-group results from the final four-expert comparison.

C.1 TWO- AND THREE-EXPERT RESULTS

  • Table 6 reports success rates after merging the WebShop–Tool and WebShop–Tool–Search expert subsets.The subset results distinguish combining fewer experts from the final four-expert comparison.Source: §C.1 · p. 18, Table 6 · p. 18
  • Continual methods integrate each subset’s experts sequentially in the stated order, whereas batch methods merge the same subset jointly.Continual runs follow WebShop → Tool → Search.Source: §C.1 · p. 18, Table 6 · p. 18
  • Table 6 reports SR (%) and computes each average within its expert group; its markings identify the highest and second-highest measured results per column.Ties share the same mark.Source: Table 6 · p. 18

C.2 TASK SCORES AND INTERACTION LENGTHS

The section reports task score and interaction length alongside success rate, and presents BACAM’s stage-wise scores across measured merging orders.

C.2 TASK SCORES AND INTERACTION LENGTHS

  • Table 7 complements success rate with WebShop task score and average interaction steps, with Round including successful and failed episodes.Task score captures partial shopping-task completion, while success rate requires full completion; interaction length should be considered with task performance.Source: §C.2 · p. 18, Table 7 · p. 19
  • Round comparisons alone do not establish performance gains because shorter interaction can reflect early termination and longer interaction can reflect repeated unsuccessful actions.Interpret Round with success rate and, for WebShop, task score.Source: §C.2 · p. 18

C.3 STAGE-WISE SCORES AND CAPABILITY ACQUISITION

  • Table 8 reports BACAM’s success rates after each integration for the measured orders, with WebShop task score in its final column.A dash marks a task whose expert has not yet been integrated.Source: §C.3 · p. 18, Table 8 · p. 19

D.1 TWAS ORDER

The diagnostics characterize gate movement and plasticity budgets across two expert-arrival orders. In TWAS, movement declines across integrations and budgets constrain updates toward incoming experts, while SAWT reports corresponding layer- and tensor-level views.

D.1 TWAS ORDER

  • Under TWAS, gate movement declines across integrations, MLP tensors account for most movement, and attention gains share as the sequence progresses.Budgets vary by capability and stage and restrict updates toward incoming experts; these diagnostics describe optimization behavior, not tensor importance for task success.Source: §D.1 · p. 20
  • During ALFWorld integration in TWAS, WebShop has lower final-refresh budgets than Tool across highlighted value-projection layers 14–19.Lower budgets restrict updates toward the incoming expert.Source: Figure 4 · p. 20
  • TWAS diagnostics show per-refresh gate movement declining across stages, movement dominated by MLP tensors, and more persistent movement than budgets across stages.The layer-wise and tensor-level plots show movement alongside minimum plasticity budgets, where lower budgets impose tighter update limits.Source: Figure 5 · p. 20, Figure 6 · p. 21, Figure 7 · p. 21

D.2 SAWT ORDER

  • SAWT reports the same layer-wise and tensor-level diagnostics, with ALFWorld having lower final-refresh budgets than Search across highlighted value-projection layers 14–19 during WebShop integration.The plots compare gate movement and minimum plasticity budgets by layer and weight type or by tensor; lower budgets restrict updates toward the incoming expert.Source: Figure 8 · p. 22, Figure 10 · p. 23, Figure 11 · p. 23
  • Across SAWT integrations, gate movement declines across stages, MLP tensors dominate movement, and movement is more persistent across stages than budgets.The incoming-expert order is ALFWorld, WebShop, then Tool.Source: Figure 9 · p. 22

E QUALITATIVE EXAMPLES

Matched evaluation examples contrast BACAM with WUDI-Merging at decisive interaction steps. BACAM follows remaining task constraints and environment feedback, while WUDI-Merging diverges before completion.

E QUALITATIVE EXAMPLES

  • After noting the 52-by-84-inch size remained unmet, the action selected it and purchased successfully, scoring 1.00; another cited outcome omitted size and scored 0.80.Source: E QUALITATIVE EXAMPLES · p. 24
  • For a request about Quasar Ltd.’s latest trading details, BACAM resolves the ticker before retrieving stock information and is correct under the BFCL evaluator.Source: E QUALITATIVE EXAMPLES · p. 24
  • In a transaction-history example, the agent treats the request as asking for past transactions, but the tool response does not match the request.Source: E QUALITATIVE EXAMPLES · p. 24
  • For the multi-hop park clue, BACAM resolves Tanzania and identifies Ruaha National Park; WUDI-Merging assumes the United States and answers Yellowstone incorrectly.Source: E QUALITATIVE EXAMPLES · p. 24
  • In ALFWorld, BACAM finds the lamp at the dresser and succeeds in 4 steps; WUDI-Merging repeats invalid commands and fails within 50 steps.Source: E QUALITATIVE EXAMPLES · p. 24
  • Across matched examples, BACAM follows remaining constraints and environment feedback at decisive steps, whereas WUDI-Merging diverges before completion.The examples cover web shopping, tool use, information retrieval, and embodied interaction.Source: E QUALITATIVE EXAMPLES · p. 24
Loading 2610.04966v1…