PaperlayerView exact revision on arXiv

Source-linked AI summary

From Multi-Agent to Single-Agent: When Is Skill Distillation Beneficial?

Binyan Xu, Dong Fang, Haitao Li, Kehuan Zhang

arXiv:2604.01608v5cs.AI

TL;DR

The paper asks which components of multi-agent workflows should transfer into single-agent skills while reducing orchestration overhead. It distinguishes capability resources from pipeline guidance, introduces Behavior-Outcome Freedom to guide selective transfer, and proposes AdaSkill, which achieves strong performance with substantially lower deployment overhead across 11 datasets.

  • Problem

    The paper addresses how to distill multi-agent workflows into single-agent skills while deciding which workflow components should survive transfer.

  • Method

    AdaSkill preserves validated capability resources, removes runtime orchestration, and uses Behavior-Outcome Freedom with a calibrated rule to conditionally inherit source-pipeline guidance.

  • Results

    Across 11 datasets, AdaSkill combines strong task performance with substantially lower deployment overhead; it wins five of six performance aggregates against the native source-workflow deployment reference and is 2.4–17.7× faster.

  • Takeaways & Limitations

    Capability resources and pipeline guidance should be treated as different transfer functions, with pipeline guidance selected according to behavior–outcome geometry.

  • Takeaways & Limitations

    The conclusions are limited to structured data-science workflows with explicit pipeline control and objective evaluation, and extending the framework to interactive or human-scored tasks requires corresponding behavior representations and outcome models.

Abstract

from arXiv · show

Multi-agent systems (MAS) for structured data-science tasks externalize analytical control through workflows spanning stages, tools, shared state, verification, and repair. Distilling such workflows into a single-agent skill can reduce orchestration overhead, but it remains unclear which workflow components should cross the control boundary. We distinguish capability resources, which expand what an agent can do, from pipeline guidance, which constrains which solutions it explores. On the same causal-estimation instances, adding task-qualified source pipeline guidance to a capability-matched skill changes normalized utility by +19.6 points under method-selection accuracy but -10.3 points under numerical error. To explain this reversal, we introduce Behavior-Outcome Freedom (F), a pre-synthesis diagnostic of signed behavior-outcome rank mismatch, and formalize its candidate-conditional role through Signed Anchor-Rank Transfer. Motivated by this mechanism, we propose AdaSkill, which preserves validated capability resources, removes runtime orchestration, and conditionally inherits pipeline guidance using a calibrated rule over F. Across 16 capability-matched interventions, the native-scale Full-minus-Discard effect decreases across the continuous F scale (r = -0.80, p < 0.001), while a 15-treatment atomic sweep localizes the reversal to pipeline guidance. Across 11 datasets spanning four structured data-science task families, AdaSkill combines strong task performance with substantially lower deployment overhead.

1 Introduction

The paper asks which parts of multi-agent workflows should transfer into single-agent skills, distinguishing capability resources from pipeline guidance. It introduces Behavior-Outcome Freedom and AdaSkill to make structural transfer conditional while reducing orchestration overhead.

  • Motivation: Multi-agent workflows can perform strongly but externalize analytical control into repeatedly executed runtime orchestration.Skill distillation seeks to transfer reusable workflow competence into a single tool-using agent while preserving analytical competence.
  • Transfer distinction: Capability resources expand what an agent can access, whereas pipeline guidance constrains which reachable solutions it explores.The paper argues these components should not share a single inheritance rule.
  • Diagnostic: Behavior-Outcome Freedom (F) measures signed rank mismatch between pairwise behavioral distance and score distance.Low F indicates positively ordered geometry; values approaching or exceeding one indicate weakening or anti-concordant ordering.
  • Proposed solution: AdaSkill preserves validated capability resources, removes runtime orchestration, and conditionally inherits source-pipeline guidance using a development-calibrated rule over F.The policy is evaluated through scorer-controlled, atomic, matched, and workflow-excluded interventions across 11 datasets.

2 Background and Related Work

Prior work either orchestrates agents through explicit workflows or internalizes workflow behavior into single-agent models and skills. This paper studies selective transfer of explicit workflow control rather than treating agent count as the central object.

  • Orchestrated Agentic Workflows: Orchestrated agentic systems organize complex tasks through ordered stages, shared artifacts, tool interfaces, and feedback loops.Examples span general multi-agent systems and structured data-science workflows such as text-to-SQL and causal analysis.
  • Scope: The paper’s object of study is explicit workflow control, while agent count is treated as an implementation detail of the source system.Adjacent transfer, auditing, and deployment literatures define the boundary of this constructive workflow-to-skill study.
  • From External Orchestration to Internalized Control: MAS-distillation methods compress interaction graphs, debate dynamics, or execution trajectories into a single student model.These approaches focus on internalizing source interaction logic or protocols.
  • Skill Discovery and Evolution: Inference-time skill methods synthesize, verify, attribute, refine, or evolve reusable skills from workflows and execution experience.Related systems optimize skills through failure analysis, exploration, experience consolidation, or reinforcement learning.

3 Methodology

The methodology decomposes workflow transfer into capabilities, structural guidance, and orchestration, then uses Behavior-Outcome Freedom to characterize when structural contraction can preserve outcome information. AdaSkill operationalizes this diagnostic through selective conversion and matched interventions.

  • Workflow transfer: Workflow-to-skill distillation relocates selected stages, role-conditioned calls, shared state, validation, and feedback from runtime orchestration into an inference-time skill.Capability components make actions and information available; structural components prioritize or prohibit solution paths.
  • Candidate qualification: A task-qualified source is published, high-performing, evaluated for the same analytical objective, metric-mapped, and interface-valid.Qualification defines the candidate class, while transfer sign remains the intervention estimand.
  • Formal account: Structural transfer depends on outcome relief, induced shift strength, and alignment with high-value regions, so F alone cannot identify an unexecuted candidate’s lift direction.The formal account separates pre-synthesis geometry F, candidate contraction Δπ, and realized native-scale lift.
  • Estimation: F is computed from pairwise distances between behavioral representations and score differences across repeated Base Agent runs.The population formulation uses two independent runs, behavioral distance U, and score-distance V, with rank correlation defined when both variables vary.
  • Estimation: The rank statistic is scale-invariant across categorical, set-valued, and continuous behaviors with ties, and orders geometric support for transfer within qualified candidates.Operational routing uses Fout first and Ftrace only when the output representation is unidentifiable.
  • Behavior-Outcome Freedom: Behavior-Outcome Freedom F measures signed rank fidelity between behavioral distance and outcome distance.Low F indicates tightly coupled positive ordering; values above one correspond to anti-concordance and reverse the implication of behavioral contraction.
  • Adaptive converter: AdaSkill preserves validated capability resources, removes runtime orchestration, and makes pipeline guidance the sole adaptive object.The converter emits a skill for direct deployment by a tool-using agent and validates retained or discarded clauses.

4 Experiments

Across 11 datasets and four task families, the experiments evaluate whether AdaSkill preserves strong performance while reducing deployment overhead, and test how Behavior-Outcome Freedom predicts structural-transfer value. Results show scorer-conditioned reversals, pipeline-specific instability, and a robust negative relationship between F and matched pipeline effects.

  • Experimental setup: 11 datasets span Text-to-SQL, Causal Estimation, Causal Discovery, and Feature Engineering, comparing source workflows, base agents, AdaSkill, EvoSkill, and Workflow Compiler.Table 2 reports dataset-level performance and domain-level efficiency, with T2SQL cells listing EX, NSF, and SRR.
  • Task-level performance and efficiency: AdaSkill wins five of six performance aggregates against the native source-workflow reference and is 2.4–17.7× faster.Its mean T2SQL EX is 67.1 versus 68.5 for APEX-SQL, with 38% lower measured cost and 3.6× lower latency.
  • Scorer-conditioned effects: +19.6 normalized points under MSA and −10.3 under MRE are the Textbook Full-minus-Discard effects, showing scorer-conditioned structural value.On Real, the corresponding effects are +25.0 and −2.1 while task, workflow, capability layer, and instances remain fixed.
  • Component ablation: Pipeline guidance reverses with F (r = −0.74), whereas tools and knowledge rise with F (r = 0.67 and 0.52), localizing instability to structural transfer.The complete component set preserves a negative profile (r = −0.61), while capability-only composition strengthens the positive profile (r = 0.95).
  • Atomic interventions: Across 15 atomic Pipeline-only interventions, normalized pipeline lift decreases with F (Pearson r = −0.623, p = 0.013; Spearman ρ = −0.568, p = 0.027).The sweep isolates one structural component against the Base Agent before matched capability-controlled testing.
  • Matched structural interventions: Across 16 matched interventions, δpipe has a −9.04 utility-point slope per unit F, with a 95% interval of [−13.23, −4.85], p < 0.001, and ROC AUC 1.000.The native-scale Pearson correlation is r = −0.797 (p = 0.0002), and family-excluded refits retain negative slopes.
  • Deployment efficiency: AdaSkill’s conversion cost can amortize after 3–382 deployed queries, combining a measured diagnostic charge with lower per-query costs than matched alternatives.For CE, AdaSkill costs $0.207 per query versus $0.223 for Full and $0.211 for Discard, alongside a $6.12 Freedom probe.
  • Budget sensitivity: The adaptive converter uses calibrated diagnostic budgets, with MSA stabilizing at N ≥5 while MRE remains on the discard side across the tested grid.For CE, the selected knee is M=N=6; increasing M beyond six produces only marginal movement.

5 Limitations

The paper’s conclusions are scoped to structured data-science workflows with explicit pipeline control and objective evaluation. Its diagnostic is conditional on the agent, metric, representation, protocol, and meaningful outcome variation.

  • Scope: The conclusions are limited to structured data-science workflows with explicit pipeline control and objective evaluation.Extending the framework to interactive or human-scored tasks requires corresponding behavior representations and outcome models.
  • Diagnostic assumptions: Behavior-Outcome Freedom is metric-, agent-, representation-, and protocol-conditioned and requires meaningful outcome variation across Base Agent runs.These conditions constrain how broadly the diagnostic can be interpreted.

6 Conclusion

The paper reframes workflow distillation as selective transfer: preserve capability resources, remove runtime coordination, and conditionally inherit pipeline guidance using behavior–outcome geometry. Across 11 datasets, AdaSkill combines strong performance with substantially lower deployment overhead.

  • Conclusion: Capability resources expand what an agent can do, while pipeline guidance acts as a structural intervention whose value depends on behavior–outcome geometry.This distinction determines which workflow components should cross the control boundary.
  • Conclusion: AdaSkill preserves validated capability, removes runtime coordination, and selectively inherits pipeline guidance using Behavior-Outcome Freedom.The design treats pipeline guidance as the adaptive object rather than applying one inheritance rule to all workflow content.
  • Conclusion: Across 11 datasets spanning four structured data-science task families, AdaSkill achieves strong performance with substantially lower deployment overhead.This is the paper’s stated system-level conclusion.

A Theoretical Scope and Proofs

The theory decomposes structural lift into intervention strength and behavior–outcome alignment, then bounds when Behavior-Outcome Freedom can predict transfer direction and native-scale lift. Its scope is limited when interventions alter outcome mechanisms, representations, protocols, or metric calibration.

  • Theory overview: The appendix proves Structural-Lift Decomposition, geometric determinacy bounds for F, Signed Anchor-Rank Transfer, and sign, scale, and support impossibility results.A calibration corollary connects ordinal transfer to continuous metric-scale lift, while binary successful-anchor transfer is an exact special case.
  • Structural-lift decomposition: Structural lift equals intervention strength times behavior–outcome alignment, with absolute lift bounded by the product of their norms.The decomposition assumes a fixed conditional outcome kernel and represents the intervention through a conditional density ratio.
  • Geometric determinacy: F constrains rank predictability: 0 ≤ρS(U, V )^2 ≤Γrank ≤1 and 1 −Γrank ≤2F −F^2.The bound distinguishes Spearman rank association from the optimal conditional prediction of outcome-distance rank from behavioral distance.
  • Scope and limitations: The structural-lift theorem applies when pipeline effects are mediated by B; altered conditional means, unrestricted capability expansion, and omitted off-support effects require an additional remainder.The native-scale corollary is also bounded by the chosen metric calibration and protocol-defined rank geometry.
  • Geometric determinacy: F≈1 indicates weak signed monotone coupling, whereas F>1 denotes anti-concordance that can remain predictive but reverses the implication of behavioral contraction.The unsigned strength 1−|ρS| cannot route transfer because it discards whether the orientation is positive or negative.
  • Anchor-rank transfer: Signed Anchor-Rank Transfer is exact on the ordinal score-distance scale, while continuous metric lift additionally requires a calibrated residual model with positive slope.The continuous corollary applies to bounded higher-is-better metrics and abstains when bH≤0; a large residual limits raw-scale inference.

B.3 Data Separation and Inference

The study separates diagnostic, conversion, and final-evaluation data roles, then estimates Behavior-Outcome Freedom from repeated Base Agent behavior using output and trace representations. Routing applies identifiability criteria and retains validated capabilities while omitting source-pipeline clauses when diagnostics are insufficient.

  • Data separation: Diagnostic, conversion, and final evaluation use disjoint data roles fixed before skill synthesis.Freedom is computed from repeated Base Agent runs, while final results use locked evaluation sets.
  • Freedom estimation: The diversity planner generates methodologically distinct strategies to expose task-valid behavioral alternatives without entering conversion inputs.Planner prompts and run traces are excluded from conversion, while deterministic distance, Mantel, aggregation, and bootstrap computations produce the diagnostic.
  • Inference: Bootstrap reconstruction preserves dependence among pairwise entries, while per-question analyses resample questions after computing identifiable-question estimates.Per-question tasks include only questions with observed score variation, so the diagnostic reports both identifiable fraction and conditional coupling.
  • Freedom estimation: F is computed from pairwise distances among repeated Base Agent runs in output and trace spaces.Fout uses task-specific structured-output dissimilarities; Ftrace uses response-free process-trace embeddings and scaled cosine dissimilarity.
  • Routing and audit: Per-question metrics require at least three identifiable questions for pipeline retention; otherwise routing falls back to alternative representations or emits no mandatory source-pipeline clauses.CE-Synthetic-MSA and sparse Spider NSF/SRR remain audit entries, while CE Real MRE has one representation mismatch.

C.2 Operational AdaSkill Protocol

AdaSkill converts source workflows into single-agent skills by separating capabilities, knowledge, pipeline guidance, and orchestration, then applying a frozen binary rule to pipeline inheritance. Matched analyses show that structural value declines with F and is localized to pipeline guidance rather than shared capabilities.

  • Component decomposition: The component boundary classifies callable tools and declarative facts as capabilities or knowledge, while action policies and runtime coordination are pipeline or orchestration.Mixed fact-and-directive clauses are classified as pipeline regardless of headings or wording.
  • Conversion protocol: AdaSkill retains validated tools and knowledge, removes runtime orchestration, and conditionally retains metric-specific pipeline guidance using a frozen threshold over F.The converter does not receive probe traces, question identities, skill outcomes, validation feedback, or final-test data.
  • Conversion protocol: Metric modules are composed by the decision they score, preventing retained guidance for one metric from imposing structure on another metric’s subproblem.Shared tools and knowledge are deduplicated, while pipeline guidance remains scoped to its metric-specific decision.
  • Structural results: Pearson r = −0.623 (p = 0.013) in the 15-treatment atomic sweep localizes the reversal to pipeline-only interventions.The sweep excludes the Dia Workflow Compiler proxy and uses neither Full nor Discard outcomes.
  • Structural results: −9.04 utility points per unit F was the matched 16-intervention slope, with a 95% interval of [−13.23, −4.85] and p < 0.001.The strict-identifiability subset also remained negative, with slope −9.63 and p < 0.001.
  • Structural results: −F achieved ROC AUC 1.000 for distinguishing Full > Discard across all 16 interventions and the strict-identifiability subset.Perfect ordering persisted after leave-one-regime deletions and removal of up to six smallest contrasts.

C.5 Sensitivity to M and N

The sensitivity analysis identifies a moderate diagnostic budget as an empirical operating point: estimates stabilize with enough runs and questions, while diversity planning improves exploratory coverage at smaller budgets. The selected M=6, N=6 setting preserves binary routing separation at substantially lower cost.

  • Representation scope: Figure 9 measures Fout, whereas Figure 11 measures Ftrace, so their estimates need not coincide at the same M and N.The two sweeps therefore assess complementary behavioral representations rather than interchangeable numerical estimates.
  • Budget sensitivity: For Ftrace MSA, estimates stabilize once N ≥5, while increasing M beyond 6 produces only marginal shifts.For Ftrace MRE, N=3 already exceeds the calibrated threshold range; additional runs reduce variation without changing the binary decision.
  • Operating point: M=6, N=6 costs $6.12, consumes 30% of the maximum budget, and preserves binary separation.This operating point is the reported empirical knee of the accuracy–cost curve and costs 2.8× less than M=N=10.
  • Planner ablation: At N=6, planner-conditioned Ftrace MSA is 52 versus 42 without the planner, compared with a large-budget reference of approximately 53.At M=N=20, planner and no-planner protocols recover nearly identical values and ordering.
  • Planner ablation: The planner is intended to expose plausible task-valid alternatives that low-temperature independent runs may undersample.It generates methodologically distinct strategies without changing sampling temperature or top-p.

C.7 F Robustness across Datasets

Across causal-estimation datasets, the MSA–MRE ordering of F remains stable despite numerical variation, supporting source-dataset structural ranking as an initial diagnostic. The main cross-dataset discrepancy is localized to one trace-space MRE comparison, while planner ablation preserves the ordering at larger budgets.

  • Scope and audit: The retained Synthetic-MSA diagnostic has one identifiable question and is excluded from the distributional comparison.This exclusion matches the finite-sample identifiability audit.
  • Distribution: For MRE, Fout ranges from 0.638 to 1.012, while Real and Synthetic differ by Δµ = 0.059.The reported variation preserves the separation between MSA and MRE across identifiable CE comparisons.
  • Cross-dataset comparison: MRE/Fout for Real versus Synthetic has Δµ = 0.059 and p = 1.00, whereas the main discrepancy occurs in Textbook versus Synthetic MRE/Ftrace.That trace-space comparison has p = 0.026 with non-overlapping intervals, and the table labels the comparison descriptive.
  • Cross-dataset stability: The MSA–MRE ordering remains stable across identifiable causal-estimation datasets despite numerical variation.A source-dataset estimate can therefore provide an initial structural ranking followed by target-dataset re-estimation or held-out validation.
  • Planner robustness: At M=N=20, planner and no-planner trace estimates are 0.55 versus 0.53 for MSA and 0.85 versus 0.85 for MRE.The planner reaches the same scorer ordering at the operating budget without creating the reversal.

D.3 Feature Engineering

Feature Engineering is evaluated in a high-freedom regime where AdaSkill favors minimal structure, achieving strong AUC performance while reducing reported cost relative to the source workflow.

  • Feature Engineering: F out = 1.037–1.337 places Feature Engineering in a high-freedom regime.Table 16 reports AUC, selected feature counts, latency, and cost for this setting.
  • Efficiency: Removing the source evolutionary loop sharply improves deployment efficiency, while the remaining differences are small.
  • Feature Engineering: AdaSkill achieves the best mean AUC on Taobao and matches the source FELA workflow on Dia at substantially lower reported cost.The FELA workflow cost is estimated from its recorded operating range.

E.1 Performance and Cost Pareto Analysis

AdaSkill occupies the non-dominated performance–cost frontier across all six task-level aggregates, while retaining selected task-specific pipeline guidance without the source workflow’s global orchestration.

  • Pareto analysis: AdaSkill lies on the non-dominated frontier in all six panels of the performance–cost comparison.The comparison uses performance and cache-adjusted deployment cost for the selected final AdaSkill.
  • Pareto analysis: The T2SQL EX panel trades 1.4 points of mean EX for 60% higher measured cost under the workflow reference.
  • Breakdown: Figure 14 reports selected AdaSkill performance across 11 datasets and efficiency summaries for T2SQL, CE, CD, and FE.T2SQL and CE use task-average efficiency, whereas CD and FE report per-dataset efficiency.
  • Architecture transformation: For CE, AdaSkill retains callable tools, descriptive domain facts, and a metric-scoped low-F method-selection policy while discarding global ordering and shared-state orchestration.The transformation contrasts the original CAIS four-stage, eight-micro-tool workflow with the resulting skill-guided agent.
Loading 2604.01608v5…