PaperlayerView exact revision on arXiv

Source-linked AI summary

More Is Not Always Better: Cross-Component Interference in LLM Agent Scaffolding

Ming Liu

arXiv:2605.05716v1cs.AIcs.CL

TL;DR

Agent scaffolding is often assembled under the assumption that more components improve capability, but the effects of combining components are not systematically understood. This paper evaluates subsets of five components across tasks and models, finding that the best subset depends on task and scale. The best proper subset matches or outperforms All-In in all six tested conditions.

  • Problem

    Scaffolding frameworks offer little systematic guidance on which component subsets to include, despite the common assumption that more components are better.

    Source: §1 · p. 1

  • Method

    The study evaluates all subsets of five scaffolding components and defines interference as a performance drop when adding a component.

    Source: §3.1 · p. 3, §3.2 · p. 3, §10 · p. 9

  • Results

    The best proper subset matches or outperforms All-In in all six tested conditions, with the gap narrowing from 32% at 8B to 19% at 70B and approximately 0% on Claude Haiku.

    Source: §10 · p. 9

  • Takeaways & Limitations

    The optimal component count varies by task and scale, so the all-in configuration is not consistently the best-performing choice.

    Source: §10 · p. 9

    The evaluation covers two benchmarks rather than multi-step agentic tasks, and differing tool-use components confound cross-task comparisons.

    Source: Limitations · p. 9

Abstract

from arXiv · show

LLM agent systems are built by stacking scaffolding components (planning, tools, memory, self-reflection, retrieval) assuming more is better. We study cross-component interference (CCI): degradation when components interact destructively. We run a full factorial experiment over all 2^5=32 subsets of five components on HotpotQA and GSM8K with Llama-3.1-8B/70B (96 conditions, up to 10 seeds). The All-In system is consistently suboptimal: on HotpotQA, a single-tool agent surpasses All-In by 32% (F1 0.233 vs 0.177, p=0.023); on GSM8K, a 3-component subset beats All-In by 79% (0.43 vs 0.24, p=0.010). Optimal component count is task-dependent (k*=1-4) and scale-sensitive: at 70B, combinations that hurt at 8B provide gains, though All-In still trails the best subset. We fit a main-effects regression (R^2=0.916, adj-R^2=0.899, LOOCV=0.872), compute exact Shapley values, and find 183/325 submodularity violations (56.3%), showing greedy selection is unreliable. A three-body synergy among Tool Use, Self-Reflection, and Retrieval (INT_3=+0.175, 95% CI [+0.003,+0.351]) is reported as exploratory. CCI replicates across model families (Qwen2.5) and is robust to prompt paraphrasing. Our findings suggest maximally-equipped agent defaults should be replaced by task-specific subset selection via interaction-aware analysis.

1 Introduction

The paper challenges the assumption that adding agent-scaffolding components is always beneficial, and frames their interactions as measurable cross-component interference. It evaluates component subsets and relates the findings to prior work that did not map these interactions systematically.

1 Introduction

  • On HotpotQA with Llama-3.1-8B, Tool Use alone scores F1=0.233 versus All-In's F1=0.177, a 32% degradation.The paired t-test reports p=0.023.Source: §1 · p. 1
  • The best scaffold is task- and scale-dependent: at 70B, added components help on HotpotQA, but All-In still trails the best subset by 19%.On GSM8K, the optimal 8B configuration has three components and accuracy 0.43.Source: §1 · p. 2
  • The study evaluates all 32 subsets of five agent components across benchmarks and model families to expose their interaction landscape.The introduction reports 118 configurations and 32,000+ evaluations.Source: §1 · p. 2
  • On Llama-8B/HotpotQA, 5/6 expansions from the best scaffold degrade performance significantly, and the directional pattern replicates across model families and prompt paraphrases.The reported pattern is not a context-length artifact: a length-matched control retains a 6–9× gap.Source: §1 · p. 2
  • A main-effects model explains 91.6% of variance, while 183/325 submodularity violations exhibit sign-flipping that makes greedy selection unreliable.Shapley decomposition attributes 70% of scaffold value to Tool Use and finds Planning significantly negative.Source: §1 · p. 2

2 Related Work

  • Prior work examined component value or isolated interactions, while prompt optimizers treated prompts atomically; this study maps combinations and interactions across scaffolding components.Related work also identifies prior studies of instruction interference as examining one interaction at a time.Source: §2 · pp. 2–3

3 Problem Setup

The setup treats agent scaffolding as a selectable subset of five components and defines interference through a drop in performance after adding a component. The experiment compares these configurations across models and benchmarks.

3.1 Scaffolding Components

  • The five toggled modules are planning, tool use, memory, structured reasoning, and reflection, and each configuration activates a subset.Five binary components yield 32 possible configurations.Source: §3.1 · p. 3

3.2 CCI Definition

  • Cross-Component Interference occurs when adding a component to an existing configuration reduces performance; this operational definition does not specify a causal mechanism.The paper calls CCI widespread when it holds across many component-configuration pairs.Source: §3.2 · p. 3

3.3 Experimental Protocol

  • The experiments cover HotpotQA and GSM8K with Llama, Qwen, and Claude models, evaluating all 32 subsets and reporting 32,000+ evaluations.Key HotpotQA configurations are replicated across 10 seeds; the 70B model uses 4-bit NF4 quantization.Source: §3.3 · pp. 3–4

4 Main Result: CCI in the Combinatorial Space

On HotpotQA, adding components to Tool Use degrades performance, while Shapley values identify Tool Use as the strongest contributor and Planning as negative; on GSM8K, Retrieval is positive and part of the optimal subset.

4.1 Tool Use Dominates; Adding Components Hurts

  • With Tool Use, F1 ranges from 0.142 to 0.284 (mean 0.204), versus 0.000 to 0.099 (mean 0.043) without it.In the seed-42 sweep, every multi-component configuration containing Tool Use scores below Tool Use alone; adding 1–4 components degrades performance by 26%–33%.Source: §4.1 · p. 4

4.2 Multi-Seed Validation

  • F1=0.233 for Tool Use alone versus F1=0.177 for All-In, a 32% degradation across 10 seeds.The comparison is reported in the multi-seed HotpotQA validation.Source: Table 2 · p. 4, §4.2 · p. 4
  • Every measured expansion from Tool Use degrades performance; 5 of 6 comparisons are significant at uncorrected p<0.05 and 4 of 6 survive Holm–Bonferroni correction.Effect sizes are uniformly large (Cohen’s dz=0.87–1.44).Source: §4.2 · p. 4

4.3 Shapley Decomposition

  • Tool Use has Shapley value +0.177 and 70% of absolute Shapley mass; Planning is negative at −0.029 (95% CI [−0.055, −0.003]).Shapley values average each component’s marginal contribution across coalitions.Source: §4.3 · p. 4, Table 3 · p. 5
  • Planning and Memory impose a 22% CCI tax on Tool Use and Self-Reflection’s positive value; on GSM8K, Retrieval has Shapley value +0.065 and is part of the k*=3 optimum.The GSM8K attribution is consistent with Retrieval’s role in the optimal subset.Source: §4.3 · p. 5

5 CCI Varies by Task and Scale

The best component count and the direction of component effects vary by task and scale. The best proper subset matches or outperforms All-In across the tested model–benchmark conditions, though the evidence varies in strength.

5.1 Task-Dependence: GSM8K

  • On GSM8K, T+SR+R scores 0.430 with k*=3, while HotpotQA’s optimum is Tool Use alone (k*=1).Self-Reflection and Retrieval help on math tasks but hurt on retrieval QA.Source: §5.1 · p. 5
  • The GSM8K best subset outperforms All-In by 79% (McNemar p=0.010); the penalty comes from adding Planning and Memory to the effective three-component core.Source: §5.1 · p. 5

5.2 Capacity-Dependence: 70B Scale

  • Adding Retrieval to Tool Use shifts from −0.072 at 8B to +0.072 at 70B; at 70B, adding components helps Tool Use.Source: Table 5 · p. 5, §5.2 · p. 5
  • The best-versus-All-In gap narrows from 32% at 8B to 19% at 70B, where the comparison is single-seed and p=0.13.The authors caution that the two models also differ in architecture and training compute, and two data points cannot establish a scaling law.Source: §5.2 · p. 5

5.3 Cross-Family Replication

  • The best subset matches or outperforms All-In in all six tested model×benchmark conditions, but only Llama-8B has multi-seed validation; Claude Haiku’s configurations cluster within 0.25 percentage points.The reported Claude Haiku F1 values range from 0.395 to 0.398, with fully overlapping confidence intervals.Source: §5.3 · pp. 5–6

6 Analysis Framework

The analysis finds that main effects generalize better than pairwise interactions, while component gains can reverse across scaffold contexts. A three-component synergy is reported as exploratory, with a wide confidence interval and task-specific effects.

6.1 Component Interaction Regression

  • LOOCV R^2 was 0.872 for the six-parameter main-effects model versus 0.748 for the pairwise model, and BIC favored main effects by ΔBIC=25.3.The main-effects model's superior generalization indicates CCI is primarily additive in these data.Source: §6.1 · p. 6

6.2 Non-Submodularity

  • 183/325 triples (56.3%) violated submodularity, but the violation-rate confidence interval overlapped 50%, so the authors rely on the marginal-ratio evidence instead.The median submodularity ratio was γ=0.52, with a bootstrap 95% CI of [0.23, 0.71], below the threshold of 1.Source: §6.2 · p. 6
  • Among the 20 largest violations, 18 (90%) showed sign-flipping; adding Self-Reflection to {T} hurt (−0.068), but adding it to {P, R, T} helped (+0.094).The sign reversal was +0.161; none of the top 20 violations was individually significant after Bonferroni correction.Source: §6.2 · p. 6
  • Greedy selection from Tool Use stopped at K=1 on GSM8K (accuracy 0.220), missing the T+SR+R optimum (accuracy 0.430), a 95% improvement.The authors call greedy selection empirically unreliable on this benchmark suite, not provably suboptimal in general.Source: §6.2 · pp. 6–7

6.3 Exploratory: Three-Body Catalysis

  • INT3 for Tool Use, Self-Reflection, and Retrieval was +0.175 (BCa 95% CI [+0.004, +0.352]), an exploratory result with a wide interval and task-specificity.The triplet's performance was 0.271 versus a pairwise prediction of approximately 0.145; INT3 was near zero on GSM8K (+0.01).Source: §6.3 · p. 7

7 Robustness

Robustness checks find that component effects are not explained by token count or a single prompt template, though fine-grained configuration rankings vary with wording. On Claude Haiku, configurations cluster within a narrow band, and no tested ordering dominates.

7.1 Length-Matched Control

  • The reported finding is that CCI is driven by semantic content, not token count: real components outperform length-matched meaningless padding by 6–9×.Source: §7.1 · p. 7

7.2 Template Paraphrase Robustness

  • Tool Use beat All-In in all three template variants, with gaps of +3%, +20%, and +13%, while fine-grained configuration rankings were template-sensitive.Across variant pairs, Spearman ρ was 0.14–0.49 for the six tested configurations; component-level attribution remained robust.Source: §7.2 · p. 7

7.3 Closed-Source API Validation: Capability Saturation

  • On Claude Haiku, all K=1–5 configurations fell within F1 ∈[0.395, 0.398]; Tool Use alone scored 0.396 and All-In 0.398, a 0.2% difference.The confidence intervals fully overlapped, and the authors describe the difference as far below the noise floor.Source: §7.3 · p. 7

7.4 Component Ordering

  • Across five tested orderings and five configurations, ordering effects reached 62% for P+T, but no single ordering dominated and optimal ordering only partly recovered CCI losses.The preliminary test used 200 HotpotQA tasks.Source: §7.4 · pp. 7–8

8 Error Analysis

Error analysis finds that interference varies by task and component pairing rather than having one universal culprit. The paper recommends subset evaluation while describing task- and scale-dependent patterns as hypotheses that need broader validation.

8 Error Analysis

  • Some Tool Use pairs worked while others collapsed on 68% of analyzed CCI tasks, and no component was a universal culprit: Planning disrupted 84%, Memory 68%, Retrieval 68%, and Self-Reflection 58%.On 13 of 19 tasks, at least one component could be safely added while another could not.Source: §8 · p. 8

9.1 Implications for Agent Design

  • The authors recommend evaluating subsets per setting: optimal k* shifts from 1 on HotpotQA to 3 on GSM8K, and the best-vs-All-In gap narrows from 32% at 8B to 19% at 70B.Greedy selection is unreliable in these results: on GSM8K it stops at Tool Use alone (accuracy 0.220), missing T+SR+R (accuracy 0.430).Source: §9.1 · p. 8

9.2 Relation to Multi-Task Learning

  • Unlike multi-task-learning interference from training-time gradient conflict, CCI is described as inference-time interference through the shared context window.The authors term this mechanism context-window crowding.Source: §9.2 · p. 8

9.3 Hypothesized CCI Patterns (Exploratory)

  • The proposed regimes are bandwidth competition (HotpotQA/8B, k*=1), signal dilution (GSM8K/8B, k*=3), and capacity-resolved synergy (HotpotQA/70B).Each regime is supported by only one task–model cell, so the labels are descriptive rather than predictive.Source: §9.3 · p. 8

Limitations

The evaluation is limited to prompt-based scaffolding, two benchmarks, and mostly single-seed experiments, while several implementation and mechanism questions remain open.

Limitations

  • Only Llama and Qwen open-weight models up to 70B plus one closed-source API model are covered; larger frontier models may behave differently.The Haiku pattern may shift toward component complementarity on stronger instruction-following models.Source: Limitations · pp. 8–9
  • Tool Use includes the Finish[answer] protocol, so its Shapley attribution reflects both tool capability and protocol access.Configurations without Tool Use lack that submission protocol, which partially explains their zero-performance baseline.Source: Limitations · p. 9
  • Hand-written, independently composed prompts may leave CCI open to attenuation through joint prompt optimization.Directional CCI persisted across three stylistically diverse paraphrases and four model checkpoints, arguing against a single-template artifact.Source: Limitations · p. 9
  • Only 8B HotpotQA has 10-seed validation; 70B, Qwen, and GSM8K use single seeds.The reported 70B result is explicitly non-significant (p=0.13).Source: Limitations · p. 9
  • HotpotQA and GSM8K do not cover agentic benchmarks such as SWE-bench or WebArena, and tool-use components differ across the two tasks.All experiments also use four reasoning steps, leaving longer-horizon patterns unresolved.Source: Limitations · p. 9
  • The study examines static prompt-based scaffolding; non-prompt tools or external memory may show different patterns.Source: Limitations · p. 9
  • The regression finds CCI primarily additive at aggregate level, but a definitive causal account of component interference remains open.Source: Limitations · p. 9

Ethical Considerations

The paper reports its ethical scope and concludes that prompt-based scaffolding can interfere when components are combined beyond a task-optimal subset. It recommends evaluating subsets for each task and model.

Ethical Considerations

  • The work studies publicly available techniques on open-weight models and established benchmarks; it introduces no new capabilities, collects no human data, and poses no foreseeable dual-use risks.The authors note that their findings may reduce unnecessary computation from suboptimal configurations.Source: Ethical Considerations · p. 9

10 Conclusion

  • Across two benchmarks, three model families, five scales, and 32,000+ evaluations, prompt-based scaffolding showed Cross-Component Interference beyond task-optimal subsets.Source: §10 · p. 9
  • The best proper subset matches or outperforms All-In in all six tested conditions, while the optimal component count varies by task and scale.The reported gap narrows from 32% at 8B to 19% at 70B and approximately 0% for Claude Haiku.Source: §10 · p. 9
  • A 6-parameter main-effects model captures 91.6% of variance, while 183/325 submodularity violations exhibit sign-flipping and make greedy selection unreliable.Source: §10 · p. 9
  • The qualitative pattern replicates across model families, extends to a closed-source API model with capability-dependent attenuation, and is robust to template variants.Source: §10 · p. 9
  • The authors recommend evaluating scaffolding subsets per task and model instead of assuming that more components are better.Source: §10 · p. 9

A Full 32-Configuration Results

This appendix presents the full configuration tables and evaluates whether pairwise interaction terms improve the model's fit and prediction.

A Full 32-Configuration Results

  • The pairwise model fits an intercept, five main effects, and ten pairwise couplings to 32 observations using ±1 component-presence encoding.Source: C Pairwise Interaction Model Details · p. 11
  • Table 8 covers all 32 configurations on HotpotQA, using 100 tasks, Llama-3.1-8B, and F1.Source: Table 8 · p. 11
  • Pairwise terms raise raw R2 to 0.937 but reduce LOOCV R2 to 0.748, indicating overfit; information criteria favor the main-effects model.The pairwise model is retained for interpretive value about component-pair synergy and conflict, not for better out-of-sample prediction.Source: C Pairwise Interaction Model Details · pp. 11–12
  • Table 9 covers all 32 configurations on HotpotQA with Llama-3.1-70B-4bit and F1.Source: Table 9 · p. 12

D Component Prompt Templates

The appendix specifies the prompt blocks used for agent components and reports several negative mechanistic analyses.

D Component Prompt Templates

E Negative Mechanistic Results

Loading 2605.05716v1…