Source-linked AI summary
The Regression Tax: Decomposing Why Skills Help and Hurt LLM Agents
Darshan Tank, Baran Nama
TL;DR
Average task-success improvement does not show how many previously solved tasks skills break. The paper compares agents with and without skills across benchmarks and model–harness stacks, decomposing paired outcomes and examining execution traces. It finds that regressions offset a substantial share of gains and identifies grounding and verification as recurring sources of errors.
Problem
Average improvement in task success can hide tasks that skills cause an agent to fail despite solving them without skills.
Source: §1 · p. 1Method
The study compares paired agents with and without skill libraries across 5,832 runs, two office-automation benchmarks, and three model–harness stacks, then uses traces to examine regression mechanisms.
Source: §1 · p. 1Results
324 regression transitions offset 59% of 553 gross gain transitions, and many residual failures occur at grounding and verification.
Source: §7 · p. 7Takeaways & Limitations
Evaluating skills by decomposing net effects into gains and regressions reveals which solved tasks they break, while trace analysis points to grounding and verification as important error stages.
Source: §1 · p. 1, §7 · p. 7Both benchmarks concern office automation, model and harness effects are coupled, and the mechanism labels were coded by one author.
Source: §6.2 · p. 7
Abstract
from arXiv · showhide
Adding procedural skills to an LLM agent is typically evaluated by average improvement in task success. However, this metric hides an important cost: skills can also make agents worse. We measure both sides by comparing agents with and without skills across nearly 6,000 runs spanning two office automation benchmarks and three model harness stacks. This allows us to distinguish two outcomes. A regression is a task solved without skills but failed after skills are added. A residual failure is a task that fails both with and without skills. We find that regressions are substantial enough that the best performing skills outperform others primarily by regressing less, not by gaining more. We identify three causes of regression: (i) skill description osmosis, a skill changes an agent's behavior simply by being present in context, even when it is never invoked; (ii) grounding displacement, a skill's prescribed procedure overrides how the agent interprets its inputs; and (iii) verification displacement, where the procedure suppresses checks the agent would otherwise perform on its outputs. Analysing persistent failures reveals the same underlying pattern. Existing skills overemphasize procedural guidance the stage least often responsible for failure while under supporting grounding and verification, the dominant sources of remaining errors. After correcting evaluation artifacts and studying traces, we find many regressions and persistent failures recoverable through better grounding and verification. Procedural skills should be evaluated by decomposing their net effect into gains and regressions, not by aggregate improvement alone. We identify three regression modes skills should avoid, and find that reliability depends more on grounding and verification than on procedural skill choice.
1 Introduction
Average task-success improvement misses an important cost: skills can break tasks an agent previously solved. This study decomposes those outcomes and traces how regressions arise, including when skill descriptions are present but their bodies are not invoked.
1 Introduction
- Average improvement alone is incomplete because two libraries can have the same average gain while differing in how many previously solved tasks they break.A regression is a task solved without skills but failed after skills are added.Source: §1 · p. 1
- Prior harmful-skill methods identify skills to drop but do not explain why they regress tasks or measure effects of descriptions that remain present without invocation.The skill description stays in the system prompt at every step and can change behavior without the body being called.Source: §1 · p. 1
- Across 5,832 runs on two office-automation benchmarks and three model–harness stacks, paired comparisons isolate the effect of changing the skill library.The pass-rate difference is decomposed into tasks newly solved and tasks newly broken.Source: §1 · p. 1
- The study examines three trace-suggested regression mechanisms: skill-description osmosis, grounding displacement, and verification displacement.It also locates many residual failures at grounding and verification.Source: §1 · p. 1
- 324 regression transitions offset 59% of 553 gross gain transitions, and the study re-grades 226 treatment task-conditions that the original spreadsheet grader failed.These are task-condition transitions, not unique tasks.Source: §1 · p. 1
2 Related Work
Prior work generates skills and evaluates average gains, while other research studies context distraction or screens skills for harmful effects. This paper instead asks why regressions occur and measures influence from skills that are present but never retrieved.
2 Related Work
- Skill-generation systems focus on creating skills and average gains, leaving the losses skills can cause unexamined.Examples include Trace2Skill, EvoSkill, SkillOpt, and SkillOS.Source: §2 · p. 1
- Huang [4] finds gains and regressions in roughly equal numbers across tested conditions, but treats conflicts with task-specific instructions as diagnostics rather than explaining mechanisms.The study examines regressions in full agent stacks and traces how they arise, including description-only effects.Source: §2 · p. 1
- Prior context studies find that irrelevant sentences, distractors, or buried relevant information can hurt reasoning; a skill description is always present and may affect behavior without invocation.The paper later measures this presence-only effect as osmosis.Source: §2 · pp. 1–2
- Existing skills mainly target the method stage, while this study places regressions and residual failures at grounding and verification.Grounding means reading the right inputs, method is the procedure, and verification checks the output.Source: Figure 1 · p. 2
- Regression-focused systems select, mask, or drop harmful skills, whereas this study asks why regressions happen and measures influence when a skill is never retrieved.These methods include randomized masking, regression-budget screening, and updating only when held-out performance does not drop.Source: §2 · p. 2
3 Experimental Setup
The experiments pair agents’ outcomes with and without skill libraries while holding the task and stack fixed. They span two office-automation benchmarks, three model–harness stacks, and libraries produced from shared failure signals by three different creators.
3 Experimental Setup
- Each comparison holds everything fixed except the skill library in context and pairs with-skill and no-skill outcomes on the same tasks.The setup covers benchmarks, stacks, libraries, outcome definitions, and significance testing.Source: §3 · p. 2
3.1 Benchmarks and Tasks
- The benchmarks cover financial-document question answering and spreadsheet manipulation; OfficeQA-Pro emphasizes grounding in long reports, while SpreadsheetBench covers cell- and sheet-level edits.The OfficeQA-Pro subset focuses on hard cases, and spreadsheet tasks are retained when they can be graded by exact cell value.Source: §3.1 · p. 2
3.2 Model–Harness Stacks
- Three model–harness stacks are tested, with each harness paired to its native model; the stack, rather than model and harness separately, is the analysis unit.The authors note that model and harness effects are therefore not separated.Source: §3.2 · p. 3
3.3 Skill Libraries and How They Are Produced
- Three skill creators turn shared analyst-derived failure signals into libraries, using a benchmarked eval loop, a one-pass structural check, or self-critique with reuse.Within a stack, libraries range from 3 to 23 skills; the same signals yield differently sized libraries, suggesting wording affects outcomes.Source: §3.3 · p. 3
3.4 Conditions and the Controlled Comparison
- Each stack and benchmark has a no-skill condition and three skill-library conditions; comparisons between libraries built from the same signals support mechanism attribution.Libraries differ in size and wording, so treatment-versus-treatment contrasts are used only for task outcomes traceable to specific skill content.Source: §3.4 · p. 3
3.5 Outcomes and Metrics
- Paired task outcomes are classified as gains, regressions, residual failures, or retained passes, and net effect is gains minus regressions.The net effect is the main scalar reported, with regressions also broken down by mechanism.Source: §3.5 · p. 3
3.6 Scale and Statistical Protocol
- The study runs 5,832 paired task-condition runs without seed replication and tests net effects using the exact two-sided McNemar test on discordant pairs.Table 2 reports pass rates, gain and regression counts, net effect, paired pass-rate difference with a 95% confidence interval, and the exact p-value.Source: §3.6 · p. 3, Table 2 · p. 4
4 Gains, Regressions, and Net Effect
Across 18 library conditions, regressions offset much of the gains, and paired counts reveal effects hidden by aggregate pass rates. After multiple-comparison correction, significant net effects remain confined to one stack and benchmark.
4 Gains, Regressions, and Net Effect
- Pass-rate difference equals gains minus regressions divided by task count; paired counts show which tasks libraries newly solve and which they break.Source: §4 · p. 4
4.1 The Regression Tax
- 324 regressions offset 59% of 553 gain transitions across 18 conditions, leaving a net of 229; every library regressed at least once.These are task-condition transitions, not unique tasks.Source: §4.1 · p. 4
4.2 Why the Decomposition Changes Interpretation
- On Claude Code · sonnet-4.6 and OfficeQA-Pro, Anthropic gained the fewest tasks but had the largest net effect (+8), reversing the gains-only ranking.This pattern was not universal: on SpreadsheetBench, leading libraries often gained more as well as regressed less.Source: §4.2 · p. 4
4.3 Statistical Uncertainty
- Three of 18 conditions survive Bonferroni correction, all on Claude Code · sonnet-4.6 and SpreadsheetBench, with corrected p<.001.The authors report raw transition counts as observed and reserve stronger causal interpretations for trajectory evidence and cross-library contrasts.Source: §4.3 · pp. 4–5
5 Mechanisms of Regression
Paired trajectories classify regressions into distinct mechanisms: grounding dominates OfficeQA-Pro, while verification is especially consequential on SpreadsheetBench. Residual failures also cluster at grounding or verification rather than the procedural middle.
5 Mechanisms of Regression
- Osmosis requires an uninvoked skill body plus influence evidence; grounding errors misread inputs, verification errors suppress output checks, and Other covers unsupported flips or unreadable traces.Labels come from paired trajectories and cross-library contrasts and are observational, not a controlled causal test.Source: §5 · p. 5
- Mechanism labels compare paired trajectories, body engagement, the first input- or output-stage divergence, and cross-library changes; one author assigned the labels.OfficeQA-Pro supports full coding, while SpreadsheetBench’s grader artifacts and cell answers limit the analysis to a coarser breakdown.Source: §5 · p. 5
- Of 81 OfficeQA-Pro regressions, 59 are grounding displacement, 14 osmosis, 3 mixed grounding and verification, and 5 Other.SpreadsheetBench receives a coarser breakdown: 70 osmosis, 46 body-engaged, 32 grader-artifact, and 95 Other regressions among 243.Source: §5 · p. 5, Table 4 · p. 5
5.1 Skill-Description Osmosis
- Of sonnet-4.6’s 13 OfficeQA-Pro regressions, 9 are osmosis; an uninvoked skill library shifts UID0096 from a passing to a failing answer.The shift tracks words in skill descriptions, although the bodies were never read.Source: §5.1 · p. 5
5.2 Grounding Displacement
- Grounding displacement accounts for 59 of 81 OfficeQA-Pro regressions, as invoked procedures can redirect an agent from a correct reading of its inputs.A separate cross-library contrast is consistent with grounding benefit, but does not isolate skill content because libraries differ in wording, size, and instructions.Source: §5.2 · p. 6
5.3 Verification Displacement
- On SpreadsheetBench, 663 failing tasks contain formulas; re-grading finds 226 already-correct outputs, 396 genuinely wrong values, and 41 formulas the engine cannot evaluate.Execution-based checks recover the 226 correct outputs; the value-only grader penalized stacks differently according to the formulas they produced.Source: §5.3 · p. 6
- Grounding dominates OfficeQA-Pro regressions, while a concrete execution check recovers 226 correct SpreadsheetBench formula outputs the value-only grader had missed.The analysis also identifies osmosis as a channel invisible to invocation-based methods.Source: §5.3 · p. 6
5.4 Residual Failures
- OfficeQA-Pro residual failures are input-grounding errors, while SpreadsheetBench residual formula failures are output-stage cases, including 226 already-correct formulas missed by the value-only grade.The two benchmarks place residual failures at opposite ends of the pipeline, not in the procedural middle emphasized by existing libraries.Source: §5.4 · p. 7
6 Discussion
The discussion recommends measuring gains and regressions separately and distinguishing description presence from body invocation. It points skill design toward grounding and executable verification, while noting limits to generalization and mechanism validation.
6.1 Designing and Measuring Skill Libraries
- Report gains and regressions alongside net pass-rate change, since identical net effects can conceal different success-and-regression tradeoffs.One library may add successes without breaking prior ones, while another reaches the same net effect through both gains and regressions.Source: §6.1 · p. 7
- Compare no library, descriptions only, and descriptions plus available bodies; retrieval masking leaves descriptions in context and cannot measure presence-only effects.Source: §6.1 · p. 7
- Skill design should supply specific grounding information and use executable output checks as a complementary way to strengthen verification.Source: §6.1 · p. 7
6.2 Limitations
- The evidence covers two office-automation benchmarks, couples each model with its harness, and uses one-author mechanism coding; results may not transfer where method is the main bottleneck.Three of five nominally significant cells survive the stated Bonferroni correction, all on the Claude Code · sonnet-4.6 SpreadsheetBench stack.Source: §6.2 · p. 7
7 Conclusion
Skills change which tasks agents solve, and their regressions offset a substantial share of gross gains; the conclusion recommends reporting paired outcomes and separately testing description effects, grounding, and output checks.
7 Conclusion
- Across 5,832 task-condition runs, 324 regression transitions offset 59% of 553 gross gain transitions.The conclusion argues that paired gain and regression counts reveal task-level trade-offs hidden by net pass rate.Source: §7 · p. 7
- On OfficeQA-Pro, most coded regressions involve grounding displacement, while some occur without body invocation and are consistent with an always-present description channel.Verification displacement is rare on OfficeQA-Pro and cannot be cleanly separated from other mechanisms in SpreadsheetBench cells.Source: §7 · p. 7
- The authors recommend reporting paired gains and regressions, testing description-in-context separately from body invocation, and probing grounding and output checks through controlled interventions.They caution against inferring these effects directly from observational traces.Source: §7 · p. 7
A Reproducibility Details
The worked cases distinguish grounding and unchecked-output faults from mistakes in the stated computation, while the reproducibility details identify the model–harness stacks used.
A Reproducibility Details
- The three model–harness stacks pair OpenCode with minimax-m2.7, Codex with gpt-5.4-mini, and Claude Code with sonnet-4.6.The text refers to each stack by harness and model.Source: A Reproducibility Details · p. 8
B Worked Cases: Correct Method, Wrong Grounding or Verification
- The worked cases hold the visible arithmetic or spreadsheet logic fixed to illustrate faults at grounding and verification.The examples include two OfficeQA-Pro regressions and one SpreadsheetBench validation regression.Source: B Worked Cases: Correct Method, Wrong… · p. 8
B.1 Grounding: a correct computation on a misread figure
- In UID0025, the skill run misread the 1934 figure as 949 rather than 549, so correct subtraction produced the wrong answer.Both runs used the same method and subtracted correctly; the skill-invoked navigation led to the wrong figure, a grounding displacement.Source: §B.1 · p. 8
B.2 Verification: a correct magnitude with an unchecked output
- In UID0086, the skill run returned −4.816 instead of +4.815: the magnitude and computation were correct, but the requested absolute change was positive.The case is coded as grounding, illustrating an unchecked-output fault rather than establishing a separate verification mechanism.Source: §B.2 · p. 8
B.3 Validation regression: a correct output the value grade cannot recompute
A SpreadsheetBench treatment formula passed in a full engine but failed value-only grading because the grader could not recalculate its modern array function; re-grading excluded a different, genuinely incorrect output.
B.3 Validation regression: a correct output the value grade cannot recompute
- The value-only verifier scored the baseline INDEX/MATCH formula 9/9 but the treatment AGGREGATE formula 0/9 because gnumeric could not evaluate AGGREGATE.A full engine reproduced all nine golden values, showing that this transition was caused by the validation toolchain rather than an incorrect agent answer.Source: §B.3 · pp. 8–9
- The re-grade counted only already-correct outputs: a treatment that used AGGREGATE but mis-anchored its range and left E9:E12 empty was not recovered.Section 5.3 reports 226 treatment task-conditions whose correct outputs had been scored wrong by value-only grading.Source: §B.3 · p. 9
C Worked Cases: Osmosis, a Grounding Contrast, and a Double Fault
- The worked cases include an osmosis negative case, a grounding positive contrast, and a grounding-plus-verification double fault.The cases are based on paired run artifacts.Source: C Worked Cases: Osmosis, a Grounding… · p. 9
C.1 Osmosis: a presence-only flip with no body read
A task’s answer shifted identically across three libraries even though no skill body was invoked; the authors attribute the supported presence-only effect to description wording rather than a one-off flip.
C.1 Osmosis: a presence-only flip with no body read
- The three treatments added skill descriptions to context, and no treatment run invoked a body, isolating descriptions as their only addition over baseline.The passage reports zero invoked skills under anthropic, openai, and Ours.Source: §C.1 · p. 9
- The shift was reproducible across separately authored libraries and tracked description vocabulary naming a revised reading of the customs figure.The case was excluded from the noise bin; four other OfficeQA flips lacking the two supporting signals were labeled Other.Source: §C.1 · p. 9
- With no skill body read or invoked, the baseline answered 37.708% while all three libraries returned 38.757%.The task asked for the centered moving average of the customs-duty rate for FY1939–1941; the same incorrect value appeared under every library.Source: §C.1 · p. 9
C.2 Osmosis on SpreadsheetBench: descriptions tip an interpretation
On SpreadsheetBench, descriptions alone can redirect task interpretation even when skills are not invoked. A separate case shows a skill library can supply a reading the baseline missed.
C.2 Osmosis on SpreadsheetBench: descriptions tip an interpretation
- On task 31628, all three libraries return serial 45301 instead of golden day 10 despite zero skill invocations, illustrating description-only osmosis.The paper links this channel to 45 minimaxm2.7 osmosis regressions in Table 4.Source: §C.2 · pp. 9–10
- The descriptions cue a serial-number, number-format, or integer frame echoed in traces, while the description-free baseline reads the instruction literally.Source: §C.2 · p. 9
C.3 Grounding: a library that supplies the reading the baseline missed
- On UID0117, only the anthropic library reaches ground truth −1.063; the no-skills baseline and two other libraries return −1.182.The grounding benefit is not isolated experimentally because the libraries also differ in wording and size.Source: §C.3 · p. 10
C.4 Grounding and verification together: a diverged input and an unchecked output
In a regression task, the no-skills run passes while two skill-library runs regress through input grounding and output verification problems. The paper also reports that LLMs served as both agents and skill sources, but only polished the paper's prose.
C.4 Grounding and verification together: a diverged input and an unchecked output
- The no-skills baseline passes with [2.81, 0.030, 8.705], while Anthropic fails to verify a final triple and OpenAI returns [2.87, 0.030, 8.706].Anthropic diverges on the sourced year-over-year figure; OpenAI mis-sources that term while preserving the slope and intercept.Source: §C.4 · p. 10
- The case is coded grounding-plus-verification because the errors occur at the input-reading and output-checking ends, not in the regression itself.Source: §C.4 · p. 10
D The Use of Large Language Models (LLMs)
- LLMs were both the agents and the source of tested skills; for this paper, they were used only to polish prose, not generate new text from scratch.Source: D The Use of Large Language Models… · p. 10