PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond Prompt or Skill? Attribution-Guided Optimization of Modular LLM Programs

Haoran Shou, Haoyue Liu, Yu Huo, Kun Zeng, Xiaoying Tang

arXiv:2609.32492v1cs.AI

TL;DR

Modular language programs leave open which component should change when task feedback reveals a failure. Sparo uses counterfactual attribution to select targeted updates across prompts, skills, and routing; it achieves the highest point estimate in all 25 model–task settings, exceeding the strongest competitor by 4.4 points on average.

  • Problem

    End-to-end task feedback does not identify whether a failure comes from an underspecified prompt, missing or harmful procedural knowledge, or incorrect skill activation.

    Source: Introduction · p. 1

  • Method

    Sparo evaluates counterfactual program variants, derives a probabilistic responsibility distribution over the prompt, skill library, and router, then samples a component for targeted mutation.

    Source: Introduction · p. 2

  • Results

    Across 25 model–task settings, Sparo has the highest point estimate in every setting and exceeds the strongest competing method by 4.4 points on average.

    Source: Experimental Results · p. 5

  • Takeaways & Limitations

    The findings support treating the prompt, skill library, and routing rules as jointly optimized components rather than isolated targets.

    Source: Conclusion · p. 7

    Sparo operates in a black-box setting with a fixed worker model and cannot update its parameters.

    Source: Problem Setup · p. 2

Abstract

from arXiv · show

Large language models can solve increasingly diverse reasoning tasks, yet their performance remains highly sensitive to task prompts, intermediate instructions, and the way reusable problem-solving knowledge is incorporated. Existing optimization methods usually focus on only one part of this design space: they either optimize a monolithic prompt, or separately induce and refine skills from model traces. As a result, they lack a principled mechanism for deciding which component should be updated when failures occur, and they rarely optimize prompts, skills, and skill-use policies in a unified framework. We propose SPARO (Skill, Prompt, And Routing Optimization), a framework that jointly optimizes task instructions, reusable skill blocks, and routing rules. It performs controlled counterfactual evaluations, converts examples' effects into a probabilistic responsibility distribution over prompt, skill, and routing components, samples one component from that distribution, and applies the corresponding targeted mutation. This design moves language-program optimization beyond global prompt rewriting toward structured, reusable, and selectively activated task knowledge. Across five benchmarks and five worker models, SPARO consistently outperforms both prompt-centered and skill-centered optimization baselines. These results suggest that effective language-program optimization depends not only on discovering useful task knowledge, but also on deciding where that knowledge should be stored and when it should be activated.

Introduction

Modular language programs create a credit-assignment problem: task feedback does not identify whether a failure calls for changing the prompt, reusable skills, or skill routing. Sparo addresses this by attributing failures across components and applying targeted updates, outperforming prompt-centered and skill-centered optimizers in the reported evaluation.

Introduction

  • Modularity creates a credit-assignment problem because end-to-end feedback does not reveal whether a failure stems from the prompt, procedural knowledge, or skill activation.The optimizer uses a cluster-balanced cross-round pool and relabels historical examples under the current program, since examples can change from failures to successes as the program evolves.Source: Introduction · pp. 1–2
  • Sparo evaluates controlled counterfactual variants, converts their effects into a responsibility distribution over the prompt, skill library, and router, then samples a component for targeted mutation.It retrieves diverse historical evidence and evaluates candidates on a validation set, using Pareto-aware program-pool selection.Source: Introduction · p. 2
  • Across five heterogeneous reasoning and question-answering benchmarks and five instruction-tuned and multimodal worker models, Sparo obtains the highest score in every evaluated model–task setting.Source: Introduction · p. 2
  • The paper frames modular program optimization as assigning responsibility across global prompts, reusable skills, and routing mechanisms.Its contributions also include preserving diverse historical evidence and using probabilistic counterfactual attribution to select which component to update.Source: Introduction · p. 2

Related Work

Prior language-program optimization methods often focus on prompts or reusable skills, while structured-program work addresses modular optimization in different ways. Sparo’s distinction is attribution across heterogeneous components, including a persistent skill library and input-dependent routing.

Black-box prompt optimization

  • Black-box prompt optimizers search over instructions or prompt programs using scores, feedback, reflection, or evolving candidate populations, but do not choose skills or routing as alternative update targets.Examples include APE, OPRO, ProTeGi, TextGrad, Self-Refine, PromptBreeder, and GEPA.Source: Related Work · p. 2

Skill learning and reusable procedures

  • Skill-learning methods retain feedback, trajectories, or reusable procedures, but do not determine whether an error calls for changing skill content, the global prompt, or skill activation.The passage names Reflexion, ExpeL, Voyager, Trace2Skill, SkillOpt, and SkillGen.Source: Related Work · p. 2

Structured language programs and routing

  • Structured-program methods optimize instructions, demonstrations, prompt factors, or multi-stage programs; Sparo instead assigns credit across heterogeneous components of one modular program.Its skill library persists across inputs, and its router selects skills at inference time, so missed and harmful activations become counterfactuals; candidate selection uses GEPA’s validation-based Pareto mechanism.Source: Related Work · p. 2

Optimization

Sparo optimizes a fixed-worker modular language program by assigning responsibility for failures to editable components and targeting mutations accordingly. It gathers cross-round evidence, tests component-specific counterfactuals, and selects candidate programs using validation-based pool selection.

Problem Setup

  • With the worker model fixed, Sparo seeks the best program under a fixed optimization budget using validation-set performance as its objective.The optimizer may query the worker LLM but cannot update its parameters; the integration template is held fixed.Source: Problem Setup · pp. 2–3

Modular Program Composition

  • A program combines a global prompt, reusable skill blocks, a router, and an integration template; the prompt sets the shared task contract, while selected skills supply local procedures.The router uses positive and negative triggers plus representative queries and applicability descriptions; its metadata governs when a skill is selected, while the procedure governs how to handle the recurring subproblem.Source: Problem Setup · p. 3, Modular Program Composition · p. 3

Cluster-Balanced Evidence Retrieval

  • Sparo maintains a cross-round pool that is relabeled under the current program and retrieves related failures alongside successes from corresponding clusters.Clustering over input and error features helps provide consistent evidence rather than isolated mistakes.Source: Cluster-Balanced Evidence Retrieval · p. 3

Counterfactual Attribution and Component Choice

  • Sparo evaluates one-component-at-a-time counterfactuals, aggregates their example-level contributions into component responsibilities, and samples a component for a targeted mutation.Skill attribution distinguishes repair of an existing skill from proposing a new one; routing attribution tests adding missed skills and removing incorrectly activated ones.Source: Counterfactual Attribution and… · pp. 3–4

Targeted Mutations from Relabeled Evidence

  • PromptEdit changes global instructions, SkillEdit creates or revises reusable blocks, and RoutingEdit changes triggers or routing rules when skills are selected incorrectly.The cluster-balanced pool supplies broader evidence to the optimizer, with success examples serving as anchors against edits that break solved cases.Source: Targeted Mutations from Relabeled… · p. 5

Candidate Evaluation and Program Pool Selection

  • Sparo retains candidates on the validation-based Pareto frontier or through the pool’s quality–diversity criterion, then returns the candidate with the highest validation score.Candidate evaluation follows GEPA’s selection mechanism; the paper identifies attribution-guided candidate generation, not a new selection rule, as its contribution.Source: Candidate Evaluation and Program Pool… · p. 5

Experiments

Across five benchmarks and five worker models, Sparo outperforms prompt-centered and skill-centered optimizers. Its component choices vary with task and worker, while routing remains part of the selected updates.

Experimental Setup

  • The evaluation covered FinQA, CLUTRR, ChartQA, ProofWriter, and SearchQA across five worker models, using 12 optimization steps and shared train/validation/test splits.SearchQA used 1,400 test examples; the other benchmarks used 400/200/1,000 train/validation/test examples.Source: Experimental Setup · p. 5

Experimental Results

  • Sparo led all 25 model–task settings, beating the strongest competitor by 4.4 points on average and by as much as 15.1 points.These are single-split-seed results; on Qwen3.5-4B, the average margin across five tasks was 5.7 points.Source: Experimental Results · p. 5, Table 1 · p. 6
  • On ChartQA and SearchQA, Sparo improved over the stronger skill-only method by 3.7 points on average across ten worker–task pairs, with a 9.6-point gain on GLM-4.6V-Flash ChartQA.The gains were consistent across all ten pairs.Source: Experimental Results · p. 5
  • On FinQA, CLUTRR, and ProofWriter, Sparo exceeded the strongest prompt-centered baseline by 9.1 points on average across 15 worker–task pairs.On Qwen3.5-4B CLUTRR, Sparo scored 69.0 versus 32.3 for the strongest prompt-centered method.Source: Experimental Results · p. 5

Component Selection Patterns Across Tasks

  • Skill mutations were selected most often on CLUTRR and FinQA, while prompt mutations dominated on ChartQA and led on ProofWriter and SearchQA for Qwen; routing held 11.1%–33.3% of selections.The component-selection distributions varied by task and worker, and routing was never the most frequent choice.Source: Component Selection Patterns Across… · p. 6

Ablation Study

Removing skills and counterfactual attribution caused the largest average performance losses in the reported ablations. On CLUTRR, Sparo also outperformed random, round-robin, and prompt-only component selection.

System-Level Component Ablation

Attribution Strategy Analysis

  • On CLUTRR, Sparo scored 69.0 versus 41.0 for random selection, 38.0 for round-robin selection, and 28.0 for prompt-only optimization.The variants differed only in how they selected the component to edit.Source: Figure 4 · p. 7, Attribution Strategy Analysis · p. 7

Parameter Sensitivity

On CLUTRR with Qwen3.5-4B, Sparo performed best with an interior memory-capacity setting and balanced router weights. Tilting the router allocation toward one signal reduced accuracy in the tested sweep.

Parameter Sensitivity

  • A memory capacity of 24% of the training set performed best; smaller capacities lacked historical evidence, while larger ones could add redundant or outdated experiences.The sensitivity runs held other hyperparameters fixed.Source: Parameter Sensitivity · p. 7
  • Balanced router weights (0.55, 0.25, 0.40) scored 69.0, ahead of description-heavy (0.40, 0.45, 0.30) at 57.1, positive-heavy (0.70, 0.20, 0.30) at 44.5, and negative-heavy (0.50, 0.20, 0.60) at 40.8.The sweep rebalanced rather than removed a signal, so it does not isolate each term.Source: Parameter Sensitivity · p. 7

Conclusion

SPARO jointly optimizes an LLM program’s prompt, skill library, and routing rules. Cross-round experience and counterfactual attribution guide where to revise, and it improves over prompt- and skill-oriented optimizers across five benchmarks and five worker models.

  • SPARO jointly optimizes an LLM program’s prompt, skill library, and routing rules.Source: Conclusion · p. 7
  • A cross-round experience pool and counterfactual attribution help SPARO identify where a failure originates before revising the program.Source: Conclusion · p. 7
  • SPARO improves over recent prompt- and skill-oriented optimizers across five benchmarks and five worker models.Source: Conclusion · p. 7

Related papers

Loading 2609.32492v1…