Source-linked AI summary
Testing Interchangeability in LLM Agent Teams
Jianxin Gao, Tianyi Yu, Linna Deng, Runze Li, Zining Wang
TL;DR
Production multi-agent systems assume that agents occupying the same role are interchangeable, but persistent team histories may encode partner-specific coordination. This paper independently forms teams, swaps role-matched agents, and compares held-out performance against controlled roster-change conditions. Swaps preserve much of task outcome while increasing communication costs, with larger effects as teams accumulate longer, more divergent histories.
Problem
It is unclear whether LLM agents with matched capability, role, and experience are interchangeable when persistent team histories encode partner-specific coordination.
Method
The study forms independent teams from one base model, maintains separate task and partner notebooks, swaps role-matched agents, and evaluates held-out tasks against a placebo.
Results
Task score moves little after swaps, while communication per unit of progress rises 16 to 63 percent; swap effects are larger after longer formation histories.
Takeaways & Limitations
In these settings, agents are more fungible in task outcome than in coordination efficiency, because partner-specific conventions remain measurable after formation.
Takeaways & Limitations
The evidence is dyadic and covers only the studied formation horizons; ρ was still rising at twenty episodes, and longer or larger teams remain untested.
Abstract
from arXiv · showhide
Production multi-agent systems replace agents constantly, on the assumption that an agent filling a role is interchangeable with any other agent that can do the job. We test that assumption. Eight teams per setting are formed independently from one base model on the same tasks, each agent keeping a private notebook across ten formation episodes; we then trade role-matched agents between teams and measure what changes on held-out tasks. Against a placebo that reproduces the disruption of a roster change without changing who occupies the seat, a swap costs little in task score but raises the communication a team spends per unit of progress by 16 to 63 percent, and in Hanabi a swapped agent is more expensive than an inexperienced one, consistent with interference from conventions learned with its former partner. In Collab-Overcooked, when the agent that sets the agenda is replaced, most of the extra communication comes from the agent that stayed. Three ablations, over base models, decoding temperature and formation length, move the swap penalty alongside one other quantity: how far independently formed teams drift apart. Greedy decoding lowers both; doubling a team's history raises both. In these settings, agents are more fungible in task outcome than in coordination efficiency, with larger swap effects after longer formation histories.
1. Introduction
The paper asks whether LLM agents filling the same role are interchangeable after forming team-specific knowledge. It introduces a controlled swap test to separate generic task experience from partner-specific coordination, motivated by evidence that team history may encode conventions about teammates.
- Production systems assume role-matched agents are interchangeable, despite persistent team memories that may encode partner-specific knowledge.
- Existing evidence does not isolate partner identity while holding model, role, and experience fixed.
- The swap test independently forms teams, trades role-matched agents, and evaluates them on held-out tasks against a placebo roster disruption.
- The design separates generic experience from partner-specific residue by comparing agents with matched model, role, prompt, and experience but different formation partners.
- Task score and coordination cost can diverge: swaps cost little in score but increase communication per unit of progress by 16 to 63 percent.
2. Related Work
Related work frames the study through ad hoc teamwork, zero-shot coordination, LLM convention formation, and human team familiarity. Prior studies leave open whether partner identity affects otherwise matched LLM agents with persistent team history.
- Ad hoc teamwork and zero-shot coordination: Ad hoc teamwork studies agents joining unfamiliar teams, while zero-shot coordination uses self-play versus cross-play gaps to expose arbitrary conventions.
- Ad hoc teamwork and zero-shot coordination: Hanabi and Overcooked provide canonical settings for conventional and embodied coordination, respectively.
- Coordination in LLM agent teams: Prior LLM coordination studies often mix capability differences, use homogeneous teams, or omit persistent memory between games.
- Conventions and drift among LLMs: LLM populations can converge on naming conventions, settle into partner-shaped attractor states, and spread behavioral properties across networks.
- Team familiarity in humans: Human team-familiarity research motivates separating individual experience from knowledge about how specific teammates coordinate.
3. Evaluation
Figure 1 depicts the swap test: independently formed teams trade role-matched agents and evaluate them on held-out tasks, while provenance and notebook sections track what travels.
- Teams form independently from one base model on a shared task pool before role-matched agents are traded between teams.
- The exchanged agents carry private notebooks divided into task knowledge κ and partner knowledge π.
- After the trade, both teams are evaluated on held-out tasks, with seat superscripts distinguishing initiators from responders.
3. Fungibility and the Swap Test
The evaluation operationalizes fungibility by separating task knowledge from partner knowledge in persistent notebooks and comparing roster-change conditions on held-out tasks. Its metrics quantify the task-score and coordination-cost residue attributable to mismatched partner history.
- Teams, formation, and notebooks: Agents share a frozen base model and role prompt; their persistent notebook separates task notes κ from partner notes π.Task notes record environment facts, while partner notes record teammate behavior, hand-offs, and agreements.
- Teams, formation, and notebooks: Each cohort forms K independently seeded teams for E episodes, producing both task knowledge and partner knowledge before the swap test.
- Swap conditions: All non-intact conditions receive the same roster-change announcement before held-out evaluation, controlling for personnel-change disturbance.
- Swap conditions: The conditions compare intact teams, placebo resets, swaps, cleared partner notes, amnesia, and naive fresh replacements.
- Swap conditions: A naive replacement is a fresh same-model agent with an empty notebook, providing the inexperienced-replacement baseline.
- Metrics: Task score T is normalized to [0, 1], while coordination cost C measures communication effort per unit of task progress.
- Metrics: The ratio ρ measures the share of formation gains that remains nonportable after subtracting what an equally experienced stranger knows.ρ = 0 indicates portability; ρ = 1 indicates that all formation gains are tied to the pairing, with higher values possible under mismatched conventions.
- Metrics: The additional contrasts test whether deleting partner notes reproduces the swap residue and whether outdated partner information is worse than none.
4. Experiments
Across the experiments, swapping a role-matched agent changes task score little but substantially increases coordination cost, especially in more coupled settings. The loss is partner-specific, uneven across seats, and partly reflected in learned protocols and notes.
- Experimental design: Eight-team cohorts use role-matched swaps, placebo controls, and pair-level bootstrap intervals to separate roster disruption from agent identity.Each cohort contains K = 8 teams, with swaps applied as four disjoint pairs.
- Task outcome versus coordination: Task score changes little after a swap, while coordination cost rises 16 to 63 percent across the evaluated settings.In low coupling, score falls from 91.6 under placebo to 90.0 under swap, while messages per sub-task rise from 3.33 to 3.85; Hanabi loses 2.0 points against 4.6 points attributable to experience.
- Task outcome versus coordination: In Hanabi, a swapped agent costs more coordination effort than an inexperienced replacement, with ρC = 1.11.Former-partner conventions can create mismatches that require detection and repair, whereas a fresh agent has fewer partner-specific expectations.
- What the extra messages do: Extra communication reflects repair and clarification: high coupling assigns 38 percent of added traffic to correction after failed hand-offs, while low coupling includes reissued requests and clarification.The high-coupling swap raises messages 51 percent, compared with 16 percent at low coupling, while score barely moves in either setting.
- Where the residue sits: Partner-specific residue increases with coordination demand but is only partly localized to partner notes, and independently formed teams develop recognizably similar protocols.ρ is 0.18, 0.26, and 0.44 across the three settings; deleting intact teams’ partner notes reproduces 81 and 64 percent of the swap penalty in the coupled settings.
- Which seat carries the loss: Replacing the initiating seat is more damaging than replacing the responding seat, and about seven tenths of the extra messages then come from the incumbent.In high coupling, initiator and responder swaps cost 7.5 and 2.5 progress-completeness points, respectively; protocol movement is also larger when the newcomer occupies the lead seat.
5. Discussion
Repeatedly interacting teams develop modest partner-specific behavior alongside mostly portable task knowledge. Thus, agent rotation can preserve task outcomes while reducing coordination efficiency, and clearing partner-specific notes is suggested as an intervention.
- Partner-specific behavior is measurable but modest; most gains from ten episodes concern task knowledge that an equally experienced same-model agent already has.
- Swap penalties affect communication more than task accuracy, so success-rate monitoring can make a reshuffled team appear recovered prematurely.
- Clearing partner-specific notes is proposed as a simple intervention suggested by the ablations, not tested directly here.
6. Conclusion
Equally experienced agents can usually replace one another without much task-score loss, but coordination efficiency is less interchangeable. Swap effects are larger when coordination demands, team age, or decoding freedom increase.
- Task score moves little after swapping, while communication per unit of progress rises 16 to 63 percent and recovers at half the speed.
- In Collab-Overcooked, swap losses concentrate in the seat that initiates plans, and most extra communication comes from the agent that stayed.
- Partner specificity is measurable, modest, and larger when tasks demand more coordination, teams are older, or decoding leaves more freedom to invent.
7. Limitations and Future Work
The study’s evidence is bounded by dyadic benchmarks, short formation histories, and a notebook design that may favor partner-specific effects. Future work should test longer histories, larger teams, and cleaner protocol interventions.
- 7. Limitations and Future Work: The study is dyadic, so its benchmarks do not address whether larger teams route around replaced members or retain pairwise versus collective residue.
- 7. Limitations and Future Work: Ten formation episodes provide a short horizon, while ρ still rises at twenty, leaving longer-lived team behavior unresolved.
- 7. Limitations and Future Work: The notebook design may favor partner-specific effects, and deleting sections changes context length without length-matched or neutral-text controls.
- 7. Limitations and Future Work: Future studies should extend formation to a hundred episodes, test teams of three or more agents, and fix protocols to separate decoder effects from conventions.