PaperlayerView exact revision on arXiv

Source-linked AI summary

The Decomposition Tax: LLM Pipelines Lose Up to 40 Accuracy Points at Their Own Interfaces

Tianqi Bu, YuXuan Peng, Junteng Tu, Henghui Xiao

arXiv:2609.32825v1cs.AIcs.CLcs.SE

TL;DR

Prior work either changes call structure along with stage visibility or studies interface failures without pricing the interface alone. The paper compares pipeline arms that differ in whether stages see the original problem, holding model, stages, prompts, and completion budget fixed. The decomposition tax reaches 40.5 accuracy points, and 70 of 118 primary-family tests survive Benjamini-Hochberg correction.

  • Problem

    Prior work either changes call structure along with stage visibility or studies interface failures without pricing the interface alone.

    Source: §1 · p. 1

  • Method

    The paper compares pipeline arms that differ in whether stages see the original problem, holding model, stages, prompts, and completion budget fixed.

    Source: §1 · p. 1, §2.1 · p. 2

  • Results

    The decomposition tax reaches 40.5 accuracy points, and 70 of 118 primary-family tests survive Benjamini-Hochberg correction.

    Source: §1 · p. 2

  • Takeaways & Limitations

    With one lossy interface, re-grounding the stage after it beats re-grounding the stage before it on 7 of 7 models on both benchmarks.

    Source: §1 · p. 2

    The prescriptions are measured on two mathematical-reasoning benchmarks with a fixed pipeline family, and transfer to other pipeline types is unknown.

    Source: Ethics and broader impact · p. 11

Abstract

from arXiv · show

A four-stage LLM pipeline gives up as much as 40.5 accuracy points at its own interfaces (gemma-3-12B on MATH-500, Holm-corrected p = 1.66e-19; the largest tax in the primary family). We hold model, problem, stages, stage prompts and completion budget fixed, vary only whether each stage can still see the original problem, and call the accuracy difference the decomposition tax. Across 21 open-weight models from nine organisations, on GSM-Hard and MATH-500 at n = 200 paired items per cell, 70 of 118 primary-family tests survive Benjamini-Hochberg correction and 54 survive Holm. On GSM-Hard, a placebo recovers nothing: it carries at least 60% of the extra tokens and at most one word of the problem. Builders design a pipeline one stage at a time, and its bill arrives at the interfaces between stages. Rewriting one stage's instruction moves gemma-3-12B's tax from 4.5 to 36.5 points, and adding "every relationship stated between them" to a stage that lists the numerical quantities lowers the tax on 9 of 9 models on MATH-500. Re-grounding, which shows a stage the original problem again, belongs after the loss. With one lossy interface, re-grounding the stage after it beats re-grounding the stage before it on 7 of 7 models on both benchmarks; on MATH-500 the earlier repair is worse than none on 7 of 7. Newer models still pay: gemma-4-12B gives up 37.0 points, and the repair holds on all three of the newest models we test. A sealed held-out test refuted a stronger rule we registered, which predicted the paying stage from the interface and receiver types, so we locate the tax by measuring one stage at a time. The prescription has two parts: re-ground the stage after the lossy interface, and if a stage must list the quantities, tell it to keep the relationships.

1. Introduction

Prior work prices stage prompts or catalogs interface failures, but does not isolate the cost of an interface. This paper measures what re-grounding the original problem buys back while holding the pipeline otherwise fixed.

Introduction

  • The closest prior interface-cost measurement changes both call structure and stage visibility, whereas this paper prices the interface alone.The paper frames this gap against pipeline frameworks that let builders rewrite stage prompts and failure studies that catalogue and repair interface problems.Source: §1 · p. 1
  • On MATH-500, gemma-3-12B solves 15.0% of problems in the strict arm and 55.5% in the full arm; gemma-4-12B solves 86.5% in one call and gives up 37.0 points.In the full arm, each stage also sees the original problem.Source: §1 · pp. 1–2
  • Two 2×2 squares vary sender content and form or sender instruction and receiver behavior, while mixed-order pipelines separate interface loss from stage position.The decomposition tax is TAX = acc(full) − acc(strict), paired per item.Source: §1 · pp. 1–2

Contributions

  • 40.5 accuracy points is the largest reported decomposition tax; 70 of 118 primary-family tests survive Benjamini-Hochberg correction and 54 survive Holm.On the four-stage headline pipeline, bootstrap intervals exclude zero for 17 of 20 models on MATH-500 and 16 of 20 on GSM-Hard.Source: §1 · p. 2
  • Rewriting one stage’s instruction moves gemma-3-12B’s tax from 4.5 to 36.5 points, while preserving stated relationships lowers the tax on 9 of 9 MATH-500 models.On GSM-Hard, a placebo carrying at least 60% of re-grounding’s extra tokens and at most one word of the problem recovers nothing.Source: §1 · p. 2

2. Setup

The setup compares pipeline arms that differ in which stages see the original problem, while keeping stage prompts fixed. It varies sender content and form, the whole sender instruction, and receiver behavior to study interface effects.

2.1 The arms differ only in what each stage sees

  • Re-grounding prepends the original problem to a stage’s input while leaving its prompt byte-identical; later stages otherwise read only the previous stage’s output.Each stage is a single LLM call with a fixed prompt, and the final stage emits an answer in a fixed format.Source: §2.1 · p. 2
  • CONTENT varies whether the sender passes the quantities and question alone or also preserves every relationship; FORM varies a clean list versus a prose restatement.EDGE combines content and form in the sender’s whole instruction, while RECEIVER varies whether the next stage is committed to output expressions or may re-derive them.Source: §2.1 · pp. 2–3
  • MIX_LR places a condensing interface before a restating one, while MIX_RL reverses their order to locate loss independently of stage position.A build-time check enforces that the square’s four cells differ only in the specified prompt strings, with other stages byte-identical.Source: §2.1 · p. 3
  • The benchmarks are GSM-Hard and MATH-500, with 200 items per cell; the primary family contains 118 greedy full-versus-strict tests.The 21 open-weight models span 1B to 24B parameters; A_asset excludes OLMo-2-13B because its short context window requires a smaller completion budget.Source: §2.1 · p. 3

3. Results

The results show a substantial, widespread accuracy tax at pipeline interfaces, with the cost varying by stage instructions and the placement of re-grounding. The tax persists in newer models, although its make-up can shift.

3.1 The tax reaches 40.5 points (Figure 2)

  • The largest primary-family tax was +0.4050 for gemma-3-12B on MATH-500 A_asset; 70 of 118 tests survived Benjamini-Hochberg and 54 survived Holm.On A_asset, bootstrap intervals excluded zero for 17 of 20 models on MATH-500 and 16 of 20 on GSM-Hard; paired within-model comparisons showed condensing cost more in 34 of 39 pairs.Source: Figure 2 · p. 4, §3.1 · pp. 4–5

3.2 One stage’s instruction moves the tax from 4.5 to 36.5 points (Figure 3)

  • On gemma-3-12B, changing one stage’s instruction moved the depth-3 MATH-500 tax from +0.0450 to +0.3650, while adding the relationships clause lowered it on 9 of 9 models.The clause also lowered the tax on 3 of 4 GSM-Hard models; a committed receiver raised the tax on 10 of 11 models on each benchmark.Source: §3.2 · pp. 5–6, Figure 3 · p. 6

3.3 With one lossy interface, the repair belongs after it (Figure 4)

  • Re-grounding the stage after a single lossy interface beat re-grounding the stage before it on 7 of 7 models on both benchmarks.On MATH-500, re-grounding before the loss did worse than no repair on 7 of 7 models; in MIX_LR, the two downstream stages helped about equally.Source: §3.3 · p. 7, Figure 4 · p. 7

3.4 Newer models still pay, and the repair still holds (Figure 5)

  • All three 2026-cohort models still paid on MATH-500 A_asset, and the repair passed on all three, while gemma-4-12B’s tax make-up shifted toward the receiver.At a matched 4096-token budget, the 2026 models’ mean tax was +0.2317 versus +0.3383 for the comparison group; gemma-4-12B’s receiver effect was +0.1425 and edge effect +0.0125.Source: §3.4 · p. 8

4. Related work

Related work studies decomposition, interface failures, prompt format and context, and test-time compute. This paper distinguishes its measurement by varying what one stage can see while holding the rest of the pipeline fixed.

Decomposition and its cost

  • Prior decomposition work measures what its design buys, while Radhakrishnan et al. (2023) measure isolated-context costs with call structure and visibility changing together.DnDScore checks decomposed subclaims against source context; the paper uses its full arm as an instrument.Source: §4 · pp. 8–9

Failures at the interface

Format, context and compute

  • Unlike studies that vary whole-prompt format, information position, or compute budgets, this work varies one interface’s content with the rest fixed.A placebo separates missing content from missing length.Source: §4 · p. 9

5. Conclusion

The conclusion reports substantial interface costs and recommends re-grounding after a lossy interface, while qualifying the scope of its evidence and rejecting a stronger registered prediction rule.

5. Conclusion

  • The prescription is to re-ground the stage after the lossy interface and, when listing quantities, instruct the stage to preserve their relationships.Source: §5 · p. 10
  • The tax reaches 40.5 accuracy points, and 70 of 118 primary-family tests survive Benjamini-Hochberg correction.Source: §5 · p. 10
  • A sealed test refuted the registered rule predicting the paying stage from interface and receiver types; the conclusion instead locates the tax by measuring one stage at a time.Source: §5 · p. 10

Reproducibility statement

The reproducibility statement describes automated checks and data-generation safeguards, and makes code, logs, and result files available on request.

Reproducibility statement

  • Predictions were registered in a dated file, with 24 amendments available with the code on request, including amendments recording a failed prediction and withdrawal of the original confirmatory family.Source: Reproducibility statement · p. 10
  • Nineteen checks run on every build, and 17 include committed negative tests demonstrating that they detect their target defects.Two checks report inventory rather than pass or fail a claim and are marked accordingly.Source: Reproducibility statement · p. 10
  • Three named checks trace manuscript values and citations to verified sources and prevent a label from taking different values.The checks are claims_trace.py, cite_trace.py, and consistency.py.Source: Reproducibility statement · p. 10
  • The results pipeline rejects files missing sample sizes or bootstrap intervals, or whose reported delta differs from recomputed arm accuracy; 12 interrupted files were refused.Source: Reproducibility statement · p. 10
  • Code, per-run logs, and 1,025 result files are available from the corresponding author on request, with cell manifests enabling pooling assumptions to be re-checked.Cells ran on four GPU types; the registered cross-architecture control covers Ada but not the two Ampere cards used for 10 result files.Source: Reproducibility statement · p. 10

Ethics and broader impact

The work uses public benchmarks and released open-weight models without human subjects, personal data, or new training. Its practical claims are bounded by the mathematical-reasoning evaluations, while the reported result may reduce compute relative to pursuing accuracy with more stages or sampling.

Ethics and broader impact

  • The work uses public benchmarks and released open-weight models for local evaluation only, with no human subjects, personal data, or newly trained model.Source: Ethics and broader impact · p. 11
  • The practical evidence is limited to two mathematical-reasoning benchmarks and a fixed pipeline family; transfer to retrieval, planning, code, or tool use is unknown.On the one non-mathematical task measured cleanly, the tax was smaller in 4 of 4 paired comparisons over 2 models, which the authors characterize as directional rather than a result.Source: Ethics and broader impact · p. 11
  • The authors say the result is more likely to reduce compute than increase it by recovering accuracy otherwise pursued with more stages or sampling.Source: Ethics and broader impact · p. 11

H Limitations

The reported decomposition tax has important boundaries: the repaired pipeline can still trail a single call, and several comparisons are limited by benchmark scope, controls, or measurement resolution.

H Limitations

  • Across 444 three-arm cells, the re-grounded pipeline averages 0.0701 lower accuracy than a single call and wins 127; on A_asset it averages 0.1201 lower and wins 17 of 104.The monolith baseline is also constrained by completion-budget truncation; the tax itself compares two identical-depth pipelines and excludes the monolith.Source: H Limitations · pp. 20–21
  • The tax’s edge-and-receiver decomposition is scoped to the 2024 cohort; its pattern in the 2026 cohort may not be stable, and broader current-generation testing is needed.Both registered predictions about tax size in the 2026 cohort pass, but the decomposition into edge and receiver factors does not carry over on MATH-500.Source: H Limitations · pp. 21–22
  • MIX_LR and MIX_RL are not perfectly matched, so the claim that earlier loss costs more is descriptive, not a clean single-factor comparison.The upstream re-grounding sign remains negative on 7 of 7 MATH-500 models and 3 of 7 GSM-Hard models.Source: H Limitations · p. 21
  • HO2_formal’s MATH-500 results fail a registered gate: its middle stages collapse 200 problems into 122–172 distinct messages in the full arm.On GSM-Hard, three of four models pass.Source: H Limitations · p. 21
  • At n = 200, one cell resolves about ±0.0550, leaving FORM (+0.0517) and the content-by-form interaction (+0.0550) below that threshold; larger per-cell samples are needed.The authors also identify a matched-budget monolith for a non-thinking model and another current-generation family as needed follow-up tests.Source: H Limitations · pp. 21–22
  • No compute-matched control tests whether the strict arm could use the full arm’s token budget for more reasoning, another attempt, or a longer answer.The GSM-Hard placebo recovers nothing despite carrying at least 60% of the added tokens, but it does not answer that alternative-budget question.Source: H Limitations · p. 21
  • The synthetic CASCADE tasks do not measure decomposition because their apparent effect is instruction/task mismatch.They remain in the exploratory corpus as evidence about the measure when the task is wrong.Source: H Limitations · p. 21
  • The non-mathematical comparison is limited to two models on LastLetter, where the tax is smaller in all four comparisons; the other benchmark tasks are mathematical reasoning.Six models ran on LastLetter, but some lacked a matched MATH-500 pair or failed the task or output-format criteria.Source: H Limitations · pp. 21–22

I Related work in full

Prior work studies decomposition, context loss, and failures in multi-agent systems, but varies different objects or asks different questions. This paper’s related work positions its contribution as pricing a fixed pipeline interface and using re-grounding to localize its cost.

I Related work in full

  • Least-to-most prompting (Zhou et al., 2023) carries earlier answers forward, while Decomposed Prompting (Khot et al., 2023) gives handlers only sub-queries; the paper’s strict arm is a particularly lossy regime.Unlike least-to-most, the strict arm also removes earlier-stage outputs, so its tax includes loss of history as well as loss of the original problem.Source: I Related work in full · p. 22
  • Radhakrishnan et al. (2023) report an 3.8-point drop from CoT decomposition to factored decomposition, but change call structure and visibility together and include no re-grounded pipeline arm.Their reported accuracies are 86.0, 85.6, and 81.8 across four QA tasks on one Claude-family model; the re-grounded pipeline here still trails a single call.Source: I Related work in full · p. 23
  • AgentAsk (Lin et al., 2026) repairs injected inter-agent message errors, Decomposition Dilemmas measures a decomposition trade-off, and DnDScore verifies claims in their source context.This paper instead uses re-grounding as an instrument to price the cost of an interface without injected faults.Source: I Related work in full · p. 23
  • Sclar et al. (2024) find large prompt-format effects, while this paper varies format at one interface and crosses it with content; the two effects compound in the 2024 models.The interaction is positive on all six 2024-cohort models and +0.0550 across all nine.Source: I Related work in full · p. 23
  • Failure-taxonomy and injected-fault studies, including MAST, MAS-FIRE (Jia et al., 2026), and Huang et al. (2025), examine observed or injected failures rather than isolating what the original problem contributes.Konstantinou et al. (2026) report fault detection falling from 25% to 14% when tests follow generated code, a structurally similar effect in software engineering.Source: I Related work in full · pp. 23–24
  • Lost in the Middle (Liu et al., 2024) concerns positional context degradation; this paper’s placebo and dose-response arms instead track whether problem content is missing.DisasterBench (Chen et al., 2026) infers failure origins from trajectories, whereas per-stage intervention measures recovery to attribute the tax.Source: I Related work in full · p. 24
  • Test-time compute studies vary budgets to optimize their use, while this paper holds the pipeline design and budget fixed and varies one interface; statistical corrections are assigned by configuration-defined families.The edge-level cost is presented as information a search over pipeline edges would need.Source: I Related work in full · pp. 24–25
Loading 2609.32825v1…