Competence-Aware Orchestration for LLM-Based Multi-Agent Systems

• Richer specialization and denser communication may help difficult cases, but can add token cost, latency, and redundant interactions.Source: §1 · p. 1
• Existing automated methods adapt structures to task semantics, but give less attention to conditioning collaboration scale and structure on the backbone’s competence for the query.Source: §1 · p. 1
• The method first estimates task-conditioned competence for a backbone, then uses that representation to guide role activation and communication graph construction.Source: §1 · p. 2
• The competence representation combines a reusable backbone capability fingerprint with query-specific task information.Source: §1 · p. 2
• Orchestration adapts team composition through role activation and coordination topology through communication graph generation.Source: §1 · p. 2
• Experiments show improved performance across benchmarks and backbones, lower collaboration cost on stronger backbones, larger gains on weaker ones, and the same trend on unseen backbones.Source: §1 · p. 2

• Earlier systems used manually designed sequential or debate templates, while later methods searched, optimized, or pruned structures before deployment (Du et al., 2024; Zhang et al., 2025d; Khattab et al., 2024; Zhuge et al., 2024; Qian et al., 2025; Zhang et al., 2025b; Wang et al., 2025b).Source: §2 · p. 2
• Dynamic orchestration methods adapt agent interaction at inference time to the current query or runtime setting, including query-specific graph generation and routing (Zhang et al., 2025c; Yue et al., 2025).Source: §2 · p. 2
• Prior dynamic methods mainly adapt to task semantics or runtime context, whereas this work conditions role participation and communication on query-specific backbone competence.Source: §2 · p. 3

3.1 OVERVIEW

• The two-stage pipeline learns task-conditioned competence from a reusable capability fingerprint and query information, then adapts role activation and communication graph construction.The competence representation controls both stages of multi-agent orchestration.Source: §3.1 · p. 3

3.2.1 CAPABILITY FINGERPRINTING

• Each backbone’s reusable capability fingerprint is a 4 × 12 binary matrix, with capability domains as rows and shared easy-to-hard probes as columns.The same fixed probe set is evaluated once per backbone, and the resulting fingerprint is reused for later queries.Source: §3.2.1 · pp. 3–4

3.2.2 TASK-CONDITIONED COMPETENCE MODELING

• Query-relevant domain weights aggregate the fingerprint, which is fused with task semantics into a competence representation trained using repeated-trial query success rates as proxy supervision.The domain weights reflect the query’s demands, and the fingerprint’s probe columns encode increasing difficulty.Source: §3.2.2 · p. 4

3.3 STAGE II: COMPETENCE-AWARE MAS ORCHESTRATION

• Stage II uses task-conditioned competence to set role activations and determine how activated roles communicate.Role activation assesses role suitability; graph construction specifies communication among activated roles.Source: §3.3 · p. 4

3.3.1 ROLE ACTIVATION

• Role activation combines task–role relevance with competence-conditioned modulation; semantic relevance and a success-rate-derived soft budget provide weak supervision.Higher standalone success rates encourage lower mean activation, while lower rates encourage higher mean activation.Source: §3.3.1 · pp. 4–5

3.3.2 GRAPH-BASED MAS GENERATION

• The graph encoder conditions edge predictions on role semantics, task information, competence context, and activation strength; its objective balances graph utility against communication cost.The cost term weights expected active edges using the query’s empirical success rate, reducing the relative communication penalty when standalone success is lower.Source: §3.3.2 · p. 6

4.1 EXPERIMENTAL SETTING

• On MMLU, the evaluation covers six benchmarks and uses accuracy for five and Pass@1 for HumanEval, comparing single-agent, fixed-pattern, and dynamic-structure baselines.Multi-agent evaluations use a summarizer agent, and the orchestration modules are trained sequentially offline.Source: §4.1 · pp. 6–7

4.2 PERFORMANCE COMPARISON

• 93.81% average performance across six benchmarks is the method’s best result under the same backbone setting as all baselines.Table 1 reports the six-benchmark comparison.Source: Table 1 · p. 7, §4.2 · p. 7

4.3 ABLATION STUDY

• Removing task conditioning costs 2.55 average points and 16.84% more tokens; removing the capability prior costs 3.02 points and 154.64%, while descriptions cost 3.10 points and 100.67%.The three comparisons respectively remove the task representation, remove the backbone-specific capability fingerprint, or replace that fingerprint with textual descriptions.Source: §4.3 · p. 8

4.4 CROSS-BACKBONE PERFORMANCE AND GENERALIZATION TO UNSEEN MODELS

• For performance-oriented backbones, the average rises from 87.22 to 89.97 with 36.26% fewer tokens; for efficiency-oriented backbones, it rises from 74.07 to 78.65, with gains also on unseen backbones.The performance-oriented comparison is against G-Designer; the unseen-backbone results include GLM 5.1 and Gemini 3.1 Pro.Source: Table 3 · p. 8, §4.4 · pp. 8–9, Table 4 · p. 9

4.5 CASE STUDY

• With Llama-3.1-8B, the method adds roles and answers a harder professional-psychology question correctly; with GPT-4o, it uses fewer roles, preserves correctness, and cuts token cost by 42.2%.The case studies vary the task at fixed Llama-3.1-8B and vary the backbone at fixed task.Source: Figure 4 · p. 9, §4.5 · p. 9

5 CONCLUSION

• The paper studies how to adapt multi-agent collaboration to a backbone’s competence on the current query.This is the central problem stated in the conclusion.Source: §5 · p. 9
• A task-conditioned competence representation, learned under success-rate supervision, guides role activation and communication topology.The framework uses this representation to orchestrate the multi-agent structure.Source: §5 · p. 9
• Across six benchmarks and multiple backbones, the method improves performance, reduces collaboration cost on stronger backbones, and yields larger gains on weaker and unseen backbones.Source: §5 · p. 9

Dataset-specific splits

• MMLU uses its development split for training queries and validation split for evaluation; the other five benchmarks use disjoint, dataset-specific index sets.Evaluation selects only the designated evaluation indices.Source: §A.1 · p. 13

Design goals

• The probe design targets clear difficulty resolution, multi-domain coverage, and limited one-time registration overhead, yielding a compact, structured set.Candidate items are partitioned into four coarse domains according to their source datasets.Source: §A.1 · p. 13

Difficulty calibration

• Four progressively stronger anchors define four monotone difficulty levels; non-monotone and all-fail items are discarded as ambiguous or uninformative.The retained patterns range from items solved by all anchors to items solved only by the strongest anchor, creating a shared empirical difficulty scale.Source: §A.1 · p. 13

Probe-set sampling

• 48 probes come from 16 domain–difficulty cells, with 3 items per cell and 12 probes per domain, forming a 4 × 12 easy-to-hard layout.The pool contains 40 randomly sampled training-side queries from each of six benchmarks; both probe construction and supervision data collection draw from this pool.Source: §A.1 · pp. 12–13

Supervision rationale

• Empirical query-level success serves as a proxy for competence because repeated success is treated as evidence of greater competence on that query.The resulting supervision trains task-conditioned competence modeling and supports downstream role activation.Source: §A.2 · p. 14

Data partitioning

• The 240-query pool contributes 48 probes for fingerprint registration and 192 non-probe queries for estimating query-level competence targets.The probe set is used only for registration, while the remaining samples build proxy supervision data.Source: §A.2 · p. 14

Proxy target construction

• Each proxy target is the empirical success rate from 10 repeated runs at temperature 0.2; HumanEval targets estimate pass@1, while other benchmarks estimate query accuracy.HumanEval uses benchmark unit-test pass/fail outcomes; the other benchmarks use a standard correctness indicator.Source: §A.2 · p. 14

Training-set construction

• Supervision spans 14 varied LLM backbones, and distribution-aware sampling over target success rates yields 1,060 training instances.The backbones include general-purpose, code-specialized, math-specialized, and reasoning-oriented models; GPT-4o is a probe-construction anchor, not a supervision backbone.Source: §A.2 · p. 14, Table 7 · p. 15

Auxiliary prediction-head evaluation

• On 153 backbone–query pairs across 28 held-out queries, predicted success rates reach pooled Spearman correlation 0.629 ± 0.022 and MAE 0.234 ± 0.010.Evaluation is query-disjoint, and these figures are means ± standard deviations across three training seeds; MAE is on the [0, 1] scale.Source: §A.2 · p. 15

Cost scope

• The amortization calculation concerns one-time registration, not offline supervision collection or model training.Registration evaluates a new backbone on the fixed 48-item probe set, after which its fingerprint can be reused.Source: §A.3 · p. 15

Online token usage

• G-Designer uses 3,953.94 tokens/query on average, versus 2,520.26 tokens/query for the proposed method.The comparison reports average online token usage; the associated saving is averaged over the three performance-oriented backbones.Source: §A.3 · p. 15

Cost recovery

• 27.79 queries on average recover the one-time cost of obtaining a backbone’s capability fingerprint.The average one-time fingerprinting cost across three performance-oriented backbones is 39,840 tokens.Source: §A.3 · p. 15

Evaluation scope

• The evaluation uses Gemma4 on the unseen datasets GPQA Diamond and BBEH-mini.The trained model is evaluated without additional training data for these benchmarks.Source: §A.4 · pp. 15–16

Results

• Accuracy improves over G-Designer by 3.03 points on GPQA and 2.18 points on BBEH-mini, with tokens reduced by 33.6% on both benchmarks.The table reports results on harder unseen benchmarks with Gemma4; accuracy is in percent and token savings are relative to G-Designer for the same benchmark and backbone.Source: Table 9 · p. 16, §A.4 · p. 16

• An anonymized code repository is available for the paper.Source: §A.5 · p. 16