PaperlayerView exact revision on arXiv

Source-linked AI summary

IF-MAP: Influence-Function-Inspired Agent Prioritization in Multi-Agent Systems

Jie Mao, Pingchen Lu, Zikun Qu, Jiahang Sun, Xiangyi Wang, Zhiwei Shang, Zhiyong Wang, Guojun Xiong, Zhongxiang Dai

arXiv:2610.24383v1cs.AIcs.MA

TL;DR

Limited resources make it important to identify which agent is most promising to improve, since failure responsibility alone does not reveal likely system-level gains. IF-MAP estimates priorities from end-to-end responses to semantic prompt perturbations. Across six workflows, its priorities yield larger selected gains than uniform selection for prompt optimization and model upgrades.

  • Problem

    Improving every agent can exceed available resources, while failure responsibility does not uniquely indicate which agent can produce the largest system-level gain.

    Source: §1 · pp. 1–2

  • Method

    IF-MAP perturbs each agent’s prompt, measures end-to-end loss changes, and uses a regularized local response model to assign black-box priority scores.

    Source: §1 · pp. 2–3

  • Results

    3.81 pp versus 1.85 pp for uniform selection under prompt optimization; across five model-upgrade targets, selected gains are 7.62–16.42 pp versus 3.81–8.62 pp.

    Source: §1 · p. 3

  • Takeaways & Limitations

    Across six workflows, IF-MAP priorities guide prompt optimization and model upgrades toward agents with larger selected gains than uniform selection.

    Source: §1 · p. 3, §6 · p. 9

    Realized gains depend on the embedding model, local linearity, probe coverage, and alignment between available interventions and the fitted loss-reducing direction.

    Source: §6 · p. 10

Abstract

from arXiv · show

Improving every agent in an LLM-based multi-agent system can be prohibitively costly, making it important to identify which agent should be prioritized to achieve the greatest system-level gain with limited resources. Failure attribution can inform this decision, but an agent's responsibility for failures does not directly reveal the system-level gains achievable by improving it. We therefore introduce IF-MAP, an influence-function-inspired method for identifying agents with high potential for system-level improvement. For each agent, IF-MAP measures end-to-end loss changes under controlled semantic prompt perturbations, fits a regularized local response model in a low-dimensional prompt-embedding subspace, and uses the estimated response norm as a priority score. This black-box procedure requires no access to model parameters or gradients and produces agent priorities that can guide subsequent improvement effort. We evaluate IF-MAP across six multi-agent workflows spanning code generation, question answering, and mathematical reasoning, using the same priorities for prompt optimization and model upgrades. Optimizing the selected agents' prompts achieves a macro-average gain of 3.81 percentage points, compared with 1.85 for uniform agent selection. Across five target models, upgrading the selected agents yields macro-average gains of 7.62-16.42 percentage points, compared with 3.81-8.62 for uniform agent selection. Despite its probing overhead, IF-MAP reduces Qwen3-8B and DeepSeek-V3.2 token use by 74.2% and 35.2%, respectively, relative to optimizing every agent, and reduces target-model tokens by 63.6-64.9% relative to upgrading every agent. These results support the use of IF-MAP to identify promising agents and direct improvement effort toward greater end-to-end performance gains.

1 INTRODUCTION

Choosing which agent to improve matters because system-wide upgrades can be costly, and responsibility for failures does not uniquely indicate improvement potential. IF-MAP estimates agent priorities from controlled prompt perturbations, and its rankings yield larger selected gains than uniform selection across six workflows.

1 INTRODUCTION

  • The prioritization objective is to select an agent expected to yield the largest system-level gain without fully improving every candidate.Improving all agents may exceed development resources or make each workflow execution prohibitively expensive.Source: §1 · pp. 1–2
  • Failure responsibility alone does not identify improvement potential: an agent associated with errors may offer little room to improve, while another may improve downstream behavior.The paper frames IF-MAP as a complementary estimate of promising targets for system-level improvement.Source: §1 · p. 2
  • IF-MAP perturbs one agent’s prompt at a time, measures end-to-end loss changes, and estimates a local response whose norm ranks improvement priorities.The method uses semantic prompt perturbations and a regularized local model, without requiring model parameters or gradients.Source: §1 · pp. 2–3, Figure 1 · p. 2
  • 3.81 pp versus 1.85 pp for uniform selection under prompt optimization; model-upgrade selected gains range from 7.62–16.42 pp versus 3.81–8.62 pp.The evaluation spans six workflows in code generation, question answering, and mathematical reasoning.Source: §1 · p. 3

2 AGENT PRIORITIZATION IN MULTI-AGENT SYSTEMS

The section defines agent prioritization as choosing, under a limited budget, an agent whose improvement is expected to yield the greatest end-to-end gain. It describes how priorities are selected and evaluated against observed improvement outcomes.

2 AGENT PRIORITIZATION IN MULTI-AGENT SYSTEMS

  • An improvement procedure changes one agent while keeping the workflow and other agents’ configurations fixed, and its expected utility is the target of prioritization.Procedures can refine prompts, upgrade or fine-tune models, or improve tools.Source: §2 · p. 4
  • A prioritization method scores each agent and selects the highest-scoring one, aiming for high expected utility without running the full improvement procedure for every agent.The workflow’s dependencies mean changing one agent can affect downstream agents as well as its own outputs.Source: §2 · p. 4
  • Evaluation compares selected-agent test gains with zero gain, uniform selection, and the empirical best, and uses Kendall’s τb to assess ranking agreement.The test outcomes from improving every agent serve as an offline reference and are not inputs to prioritization.Source: §2 · p. 4

3 IF-MAP: INFLUENCE-FUNCTION-INSPIRED AGENT PRIORITIZATION

IF-MAP prioritizes agents by estimating how controlled prompt changes affect end-to-end workflow loss, then uses the resulting scores to direct prompt optimization or model upgrades. Its response model combines black-box probe results with prompt-embedding directions in a regularized, low-dimensional fit.

3.1 PROBING AGENTS THROUGH SEMANTIC PROMPT PERTURBATIONS

  • Each probe changes one agent’s prompt while fixing the rest of the workflow, and evaluates the full system so measured effects include downstream interactions.Perturbations retain the agent’s role and required output interface; the pool is capped at ten formal changes.Source: §3.1 · p. 4

3.2 ESTIMATING LOCAL RESPONSE STRENGTH

  • IF-MAP fits a regularized local response model from prompt-embedding directions and normalized loss responses, using a low-dimensional subspace because few probes make unrestricted estimates unreliable.The fitted response summarizes which directions tend to increase or decrease system loss and how strongly.Source: §3.2 · p. 5
  • For a common intervention radius, the priority ranks agents by maximum first-order loss reduction predicted within the fitted subspace.The method compares G-norm with I-abs and MeanAbs, which summarize probe-effect magnitudes without using prompt-change directions and sizes in the same way.Source: §3.2 · pp. 5–6
  • Table 1 reports mean selected gain and mean Kendall τb per dataset and as six-dataset macro means.The table reports means; complete mean±SE results are in Appendix Table 11, and uniform-agent values are reference averages rather than a selector.Source: Table 1 · p. 6, §3.2 · p. 6

3.3 FROM AGENT PRIORITIES TO SYSTEM IMPROVEMENT

  • The same priorities select one agent for prompt optimization or a stronger-model upgrade, with the rest of the workflow fixed and final configurations evaluated on the test set.Prompt optimization revises the selected agent using development-set evidence; model upgrades retain its original prompt.Source: §3.3 · p. 6

4 EXPERIMENTS

Across six workflows, IF-MAP’s selected agents outperform uniform selection for prompt optimization and model upgrades. The section also examines the costs and ablations of prioritization.

4.1 EXPERIMENTAL SETUP AND BASELINES

  • The evaluation uses six AFlow-derived workflows spanning code generation, question answering, and mathematical reasoning, and compares IF-MAP with six adapted attribution, debugging, or weak-agent localization methods.Experiments use Qwen3-8B for workflow execution, DeepSeek-V3.2 for prompt operations, and five upgrade targets.Source: §4.1 · pp. 6–7

4.2 MAIN RESULTS

  • Prompt optimization yields a 3.81 pp macro-average selected gain, 1.96 pp above uniform selection; upgrade gains are 1.86–2.05 times those of uniform selection across five targets.For upgrades, the macro-average selected gains are 12.52, 12.48, and 11.91 pp for the three Qwen targets, DeepSeek-V3.2, and GPT-5 nano, respectively; corresponding uniform references are 6.57, 6.58, and 5.81 pp.Source: §4.2 · pp. 7–8, Table 2 · p. 7

4.3 COST ANALYSIS

  • Compared with optimizing every agent, IF-MAP uses 25.82% of Qwen tokens and 64.78% of DeepSeek tokens; for upgrades, FullSwap uses 2.75–2.85 times as many target-model tokens as IF-MAP.Semantic probing uses more Qwen3-8B executions than most attribution baselines, while using 1.5–16.6% of their DeepSeek tokens.Source: §4.3 · p. 8

4.4 ABLATION STUDY

  • With identical probes and outcomes, G-norm has the highest selected gains: 3.812 pp for prompt optimization and 12.394 pp for model upgrades.It exceeds I-abs by 0.536 and 2.998 pp, and MeanAbs by 0.236 and 3.304 pp, respectively.Source: §4.4 · pp. 8–9
  • MeanAbs has slightly higher prompt-ranking agreement (0.774 versus 0.741) while yielding lower selected gain, showing that full-ranking agreement need not identify the best top-ranked intervention target.Source: §4.4 · p. 9
  • Increasing the formal probe budget from three to five raises selected gains from 2.452 to 2.903 pp for prompt optimization and from 6.744 to 7.806 pp for upgrades.Up to ten probes further raise gains to 3.812 and 12.394 pp; per-workflow effects are not monotonic, and reduced variants retain auxiliary adaptive evaluations.Source: §4.4 · pp. 8–9

5 RELATED WORK

Related work covers failure attribution and debugging, workflow and prompt optimization, and local response estimation. IF-MAP focuses on selecting which agent to improve using workflow-level intervention outcomes.

Attribution, debugging, and improvement

  • Failure-attribution and debugging methods localize agents or steps associated with unsuccessful outcomes, while intervention studies distinguish responsibility from repair benefit.DoVer argues for validating attribution through realized interventions; CICA reports that the module assigned greatest responsibility is not necessarily the one whose repair yields greatest benefit in its setting.Source: §5 · p. 9

Optimizing prompts and workflows

  • Prompt and workflow optimization methods improve prompts, compound computations, workflows, or components, while IF-MAP targets the preceding decision of which agent to improve.Some coalition or removal approaches require a semantics-preserving executor for workflows with absent agents, which is not consistently available for the studied workflows.Source: §5 · p. 9

Local response estimation

  • Local sensitivity, finite-difference, and surrogate methods characterize input–output behavior; IF-MAP applies this principle to semantic agent responses propagated through downstream workflows.The resulting task-level utility signal prioritizes agents, and the paper evaluates its transfer to prompt optimization and model upgrades.Source: §5 · p. 9

6 CONCLUSION

The conclusion presents IF-MAP as a black-box way to direct limited improvement effort in fixed multi-agent systems. Its priorities yield larger selected gains while reducing downstream calls, subject to factors that affect realized gains.

Conclusion

  • IF-MAP estimates agent-level improvement potential from end-to-end responses to semantic prompt perturbations and a regularized local response model.Across six workflows, its priorities guide prompt optimization and model upgrades, yielding larger selected gains than uniform selection while reducing downstream calls required by full-agent interventions.Source: §6 · pp. 9–10
  • Realized gains depend on the embedding model, local linearity, probe coverage, and alignment between available interventions and the fitted loss-reducing direction.Future work should translate continuous response directions into discrete improvements through direction-conditioned generation, iterative edit-and-measure procedures, adaptive probing, and uncertainty-aware selection.Source: §6 · p. 10

AI USE STATEMENT

The statement describes generative AI assistance in draft preparation and distinguishes those uses from LLMs used in experiments.

AI USE STATEMENT

  • Generative AI assistants supported literature retrieval, drafting, code assistance, table construction, and automated consistency checks; authors remain responsible for verifying evidence, references, and the submission.The experiments separately use LLMs for workflow execution, prompt generation, and diagnostic judgments; no missing measurements were synthesized as results.Source: AI USE STATEMENT · p. 11

ETHICS STATEMENT

The ethics statement identifies safeguards for code execution, sensitive deployment data, and research integrity. The appendix describes the scope of its experimental protocol.

ETHICS STATEMENT

  • Generated code should run in an appropriately restricted environment, and only authorized data should be sent to external prompt-generation or judgment services.Execution transcripts may contain sensitive material in deployment settings.Source: ETHICS STATEMENT · p. 11
  • Research integrity requires retaining infrastructure failures, distinguishing missing from zero results, and disclosing adaptations of published baselines.Source: ETHICS STATEMENT · p. 11

A EXPERIMENTAL PROTOCOL AND DETAILS

A.1 DATA ROLES AND WORKFLOW NODES

The appendix specifies data roles and benchmark workflows, alongside model configurations and repetition procedures. Development data support prioritization and adaptive candidate work, while test data evaluate frozen interventions.

A.1 DATA ROLES AND WORKFLOW NODES

  • The test set evaluates the final intervention and is excluded from agent prioritization and candidate generation.Optimization and validation sets are used adaptively and treated as development data; model-upgrade experiments use probing-set priorities and test-set gains without optimization or validation sets.Source: §A.1 · p. 15
  • Table 4 gives probing, optimization, validation, and test-set sizes; ordinary optimization and validation sets are capped within separate thirds, while MATH uses reduced performance-experiment sizes.The full-sample row applies only to the cost scenario.Source: Table 4 · p. 15
  • The development split assigns examples to probing, optimization, and validation blocks, with seed 42; MATH prioritization uses a fixed 80-example probing subset.The optimization and validation blocks are not replenished with unused examples above their caps.Source: §A.1 · p. 15
  • The six workflows cover code generation, short factual and multi-hop question answering, discrete passage reasoning, and competition-style mathematics.HumanEval and MBPP use execution/testing; NQ, HotpotQA, DROP, and MATH use workflow-specific scoring.Source: §A.1 · p. 15

A.2 MODELS, CONTEXTS, AND REPETITIONS

  • The appendix specifies model contexts, token limits, and temperatures, and reports three prioritization runs plus five intervention runs per agent and dataset.The five model-upgrade targets are Qwen3-32B, Qwen3-235B-A22B-Instruct-2507, Qwen3.7-Plus, DeepSeek-V3.2, and GPT-5 nano; intervention gains use means and standard errors across five runs.Source: §A.2 · p. 16

B THE TEN-SLOT PERTURBATION SPECIFICATION

The appendix details formal prompt perturbations, a shared prompt-optimization procedure, and the statistical and reference-data conventions used to report results. These procedures include development-set acceptance checks and uncertainty summaries across repeated runs.

B THE TEN-SLOT PERTURBATION SPECIFICATION

  • The formal perturbation pool contains up to ten distinct prompts per node, combining ordinary mechanisms, sectionwise rephrasing, and adaptive candidates regardless of measured-effect sign.Adaptive construction can generate up to 36 auxiliary candidates, then considers eligible priors to produce distinct prompts; structural checks preserve placeholders and protected content, and failed measurements remain unavailable.Source: B THE TEN-SLOT PERTURBATION… · pp. 16–17, Table 6 · p. 16

C COMMON PROMPT-OPTIMIZATION INTERVENTION

  • The shared optimizer ranks eligible directions by absolute probing-set mean effect, generates four local revisions per direction, selects by optimization-set mean loss, and evaluates the final accepted prompt on test data.It holds the original prompt and prioritization evidence fixed, and considers each direction at most once.Source: C COMMON PROMPT-OPTIMIZATION… · p. 17

C.1 ACCEPTANCE RULE

  • Prompt acceptance uses optimization and validation gains, allowing validation improvement to offset a small optimization decline; the final exported prompt need not have the best optimization-set score alone.The validation comparison also guards against both the original prompt and the previous accepted state; generalization is evaluated after the prompt is fixed for test evaluation.Source: §C.1 · p. 18

C.2 REPRESENTATIVE GENERATION INSTRUCTION

  • The representative generator instruction supplies the original and current prompts, measured perturbation and its signed effect, and probing examples when available; it requests four distinct local revisions.The instruction says to incorporate supported beneficial changes and avoid copying harmful differences, while preserving required placeholders and protected fragments.Source: §C.2 · p. 18

D STATISTICAL UNITS AND RECOMPUTATION

  • Selected-gain summaries average gains over top-tied nodes, and Kendall τb compares score rankings with mean intervention gains; macro statistics weight the six workflows equally.Prioritization summaries aggregate three runs, while intervention means and standard errors use five repeated gains; standard-error bars summarize variability rather than serve as standalone significance tests.Source: D STATISTICAL UNITS AND RECOMPUTATION · pp. 18–19

E ALL-NODE GAIN REFERENCES AND COMPLETE UNCERTAINTY TABLES

  • All-node intervention gains are offline reference values excluded from selector inputs; the appendix tables report per-node gains and uncertainty conventions for prompt optimization and model upgrades.Table 7 lists per-node prompt-optimization and five-target model-upgrade mean gains; Table 8 expands IF-MAP uncertainty across all five upgrade targets.Source: E ALL-NODE GAIN REFERENCES AND… · pp. 19, 21, Table 7 · p. 19, Table 8 · p. 20

F EXTERNAL SELECTORS AND ADAPTATIONS

The external selectors are adapted to rank workflow agents from development trajectories, and their rankings are reused for prompt optimization and model upgrades. Coalition and removal variants are omitted because the workflows lack a common semantics-preserving removal operator.

F EXTERNAL SELECTORS AND ADAPTATIONS

  • Six external selectors use the same frozen development trajectories, node inventory, and downstream gain matrices, and their resulting rankings are reused unchanged for both interventions.Diagnostic judgments use DeepSeek-V3.2, task execution uses the base workflow model, and each selector is run independently three times.Source: F EXTERNAL SELECTORS AND ADAPTATIONS · p. 24
  • Who&When-AAO ranks agents by votes from valid agent-step attributions, while A2P aggregates votes from its final agent-step diagnosis.The adapted locator instead divides each trajectory's vote uniformly among the weak agents it returns.Source: F EXTERNAL SELECTORS AND ADAPTATIONS · p. 24
  • AgentRx votes for the owner of a responsible step from its violation log; CHIEF votes for its final supported agent, and FAMAS votes for the owner of its unique highest-scoring anchor triple.A FAMAS top-score tie yields an abstention.Source: F EXTERNAL SELECTORS AND ADAPTATIONS · p. 24
  • Coalition/removal variants are excluded because the six workflows lack a common semantics-preserving removal operator; in five chain workflows, the only executable coalition gives every node equal value.MATH allows additional graph-valid coalitions, but does not define the inputs downstream nodes receive when required parents are absent.Source: F EXTERNAL SELECTORS AND ADAPTATIONS · pp. 24–25

G SCORING AND PERTURBATION-BUDGET ABLATIONS

The section compares scoring-rule and perturbation-budget variants, accounts for the costs of prioritization and downstream interventions, and describes reproducibility materials and future topology studies.

G SCORING AND PERTURBATION-BUDGET ABLATIONS

H COST ACCOUNTING

  • Cost accounting covers measured execution costs over six datasets, including token counts, request counts, and costs for prioritization, prompt optimization, and model upgrades.Token counts include input and output tokens; cells report absolute cost and percentages relative to IF-MAP in the corresponding setting.Source: H COST ACCOUNTING · p. 26, Table 16 · p. 26, Table 17 · p. 27

I ARTIFACT PROVENANCE AND REPRODUCIBILITY

  • The artifact release materials include prompt and evaluation records, per-call cost accounting, a public artifact, and evaluations on independently constructed workflows.Future topology studies are proposed to test whether node-response scores remain useful when interaction graphs change information routing and correction opportunities.Source: I ARTIFACT PROVENANCE AND… · p. 28

Related papers

Loading 2610.24383v1…