PaperlayerView exact revision on arXiv

Source-linked AI summary

Does Your Generalist Skill Optimizer Really Incentivize Agentic Tasks? Rethinking Skill Optimization via Bipartite Attribution

Jiawei Zhou, Bo Peng, Yuxuan Huang, Ming Zhang, Youbang Sun, Xiang Wang, Chaochao Lu

arXiv:2610.37201v1cs.AI

TL;DR

Small, noisy dev sets make it difficult to tell whether skill edits help, while success scores omit their input-token cost. BiPA replays selection and traces tokens through turns and context; across the evaluated skills, test success was nearly unchanged while input-token use rose.

  • Problem

    On long-horizon tasks, stochastic success and small dev sets make reported gains uncertain, and success scores do not reveal the cost of extra turns.

    Source: §1 · p. 1

  • Method

    BiPA replays the optimizer’s acceptance rule on unchanged candidates and graphs agent turns against the text they read to attribute extra input tokens.

    Source: §1 · p. 2

  • Results

    Across sixteen learned skills, test success was 0.1 points above the seed, while the skills read about 1.16 times the seed’s input tokens over more turns.

    Source: §1 · p. 2, §7 · p. 9

  • Takeaways & Limitations

    The authors recommend testing gains against selection alone, charging for added turns, allowing rule deletion and merging, and showing editors the costs of earlier rules.

    Source: §1 · p. 2

    BiPA cannot decompose success over turns, context components, or rules because a black-box model’s state is not observable for defining the needed value function.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Skill optimizers adapt an agent to long-horizon tasks, such as repairing a terminal environment, by rewriting its skill, a natural-language procedure in its context, and keeping the edits that raise success on a small dev set. On such tasks success is noisy, so a gain on a small dev set does not show that an edit helps, and a success score does not show what the edit costs. We introduce BiPA, which replays an optimizer's acceptance rule on candidates identical to its seed to measure what selection alone reports, and records each episode as a bipartite graph of turns and the text they read to trace extra input tokens to the behavior behind them. Across three optimizers, three agentic benchmarks and two models, the learned skills succeed about as often as their hand-written seed, 0.1 points higher on average. Their dev gains vanish on test: keeping the best of several candidates on at most 30 tasks reports gains this large even when none changed. They read about 16% more input tokens: the optimizers almost always add rules, half to re-check work and some to act one call at a time, and the agent follows them as extra turns, each re-sending the whole conversation. Four principles follow: accept an edit only if its gain exceeds what selection alone reports; charge it for the turns it adds, not for its length; let the editor delete and merge rules; show it what earlier rules cost. Code is available at https://github.com/valleysprings/BiPA.git.

1 Introduction

Skill optimizers rewrite natural-language procedures for long-horizon agent tasks, but noisy small development sets make gains hard to interpret and success scores omit cost. BiPA measures selection effects and traces added input tokens to agent behavior.

1 Introduction

  • BiPA replays acceptance on candidates identical to the seed and represents turns and the text they read as a bipartite graph to trace extra input tokens.It also splits token increases into more tool calls, the same calls over more turns, more text per turn, and longer skill text.Source: §1 · pp. 1–2
  • The study applies BiPA to GEPA, SkillOpt, and SkillAdam across three agentic benchmarks using DeepSeek-V4.1-Flash and GPT-6-Luna.The optimizers start from one generic hand-written seed procedure.Source: §1 · p. 2
  • Across sixteen learned skills, test success was only 0.1 points above the seed, while skills read about 16% more input tokens over more turns.They gained about as many tasks as they lost; added checks were carried out as extra turns that each re-sent the conversation.Source: §1 · p. 2
  • Unchanged candidates could report dev gains of the same size as the optimizers’ gains when the gate kept the best of several candidates on at most 30 tasks.Long-task success is stochastic, so a small dev set can rank skills inconsistently; success scores also omit the input-token cost of extra turns.Source: §1 · pp. 1–2
  • The authors propose accepting edits only when gains exceed selection-only reports, charging for added turns, allowing rule deletion and merging, and exposing earlier rules’ costs.These principles follow the finding that added rules can create extra turns, each of which re-sends the conversation.Source: §1 · p. 2

2 Related Work

Prior work adapts prompts and agent skills using scored edits, feedback, search, or trajectory-derived procedures; related evaluation and cost-accounting studies motivate BiPA’s focus on what skills change in agent behavior.

2.1 Text Optimization for Agents

  • Prompt and skill optimizers edit scored text or learn procedures from outcomes, while cost-aware methods target prompt length, inference tokens, pass rate and cost, or skill compression.GEPA reflects on run logs and draws edits from skills best on at least one dev task; cost-aware rewriting has found that shorter skills can cost more.Source: §2.1 · pp. 2–3

2.2 Agent and Harness Evaluation

  • Agent and harness evaluations study held-out success, validation differences, and selection bias, while cost-accounting work tracks variable token use and attributes costs or failures to steps.BiPA adds tracing of skill cost to the turns its rules add, each of which re-sends history.Source: §2.2 · p. 3

3 Problem Formulation

The paper defines skill-optimization outcomes and paired cost measures, then asks whether learned skills outperform their seed and what behavior accounts for any extra input tokens.

3 Problem Formulation

  • A skill is context text loaded at each episode’s start; an episode records a task attempt’s binary outcome, turns, tool calls, and input tokens.Input tokens sum what each turn sends to the model, including cached tokens.Source: §3 · p. 3
  • An optimizer edits a seed or earlier candidate, and its gate keeps an edit when its acceptance rule accepts the candidate against its parent on a dev set.The returned skill is selected by dev success, and the reported dev gain sums the margins of kept skills from the seed.Source: §3 · p. 3
  • Test success is averaged over k episodes per task, while input-token cost is paired task by task against the seed using tasks with records for both skills.The per-task ratio weighs long and short tasks alike; the paper asks whether test gains exceed seed-run spread and dev gains exceed selection-only reports, and what content and behavior account for extra tokens.Source: §3 · p. 3

4 Methodology

BiPA evaluates skill edits without changing the optimizer: it replays acceptance on unchanged candidates and attributes extra input tokens to the turns and context they read. Its methodology separates selection effects from token costs by content and behavior.

4.1 Selection Replay

  • A replay applies the optimizer’s acceptance rule to unchanged candidates, measuring the dev gain that selection alone can report.A large share of replays reporting at least the observed gain means that gain is not evidence of improvement.Source: §4.1 · p. 4

4.2 Episodes as Bipartite Graphs

  • Each episode becomes a bipartite graph linking turns to the prompt, messages, and tool outputs they read, with tool calls recorded on call edges.The graph’s component totals identify which content accounts for extra input tokens.Source: §4.2 · p. 4

4.3 Turn Elasticity

  • With uncompacted history, an extra turn re-sends the prefix and conversation, adding at least 1% input tokens per 1% more turns and nearly 2% in long episodes.The turn elasticity increases with the history share; without history, it is 1.Source: §4.3 · pp. 4–5

4.4 The Four-Term Split

  • The four-term split attributes input-token changes to more actions, turn-splitting, content per turn, and skill text, separating behavior changes from reading costs.The turn terms sum to the extra-turn cost; content per turn can offset some turn-splitting cost, and the split is an accounting of associated behavior.Source: §4.4 · p. 5

5 Experiments

Across three skill optimizers, three benchmarks and two models, the experiments compare learned skills with a shared hand-written seed and trace their effects on success, behavior, and input cost. The results show little average test-success change, dev gains consistent with selection alone, and higher input-token use linked to added rules and turns.

5.2 Main Results

  • Test success is only 0.1 points above the seed across sixteen learned skills, while DeepSeek skills average 1.4 points below it.The nine DeepSeek skills range from −11.1 to +2.7 points; Luna’s seven average +1.6 points.Source: §5.2 · p. 6
  • Unchanged candidates reach each of nine reported DeepSeek dev gains with probability 0.25 to 0.94, while test differences do not follow those gains (Figure 3).Source: §5.2 · p. 6, Figure 3 · p. 7
  • The learned skills cover 57 tasks the seed does not but lose 62 it covers, and episodes agree more where success rises.Tool sequences are more alike than under the seed on 68% of tasks whose success rose, versus 49% of unchanged tasks.Source: §5.2 · p. 7
  • All nine DeepSeek skills read 1.08 to 1.43 times the seed’s input tokens per task and take 1.03 to 1.22 times its turns.Across all sixteen skills, the average input-token ratio is 1.16.Source: §5.2 · p. 7
  • Most transferred skills score below the seed, but their input-token costs largely travel across models and benchmarks.On the other benchmark, five of six score 2.5 to 9.3 points below the seed; transferred skills read 1.00 to 1.49 times the seed’s input tokens.Source: §5.2 · p. 7

5.3 Additional Analysis

  • Of 337 rules added under DeepSeek, half ask the agent to verify and 18 ask it to serialize, while none asks it to read less.The editors add 37 rules on average, up to 105; 58 of 59 scored edits lengthen the skill.Source: §5.3 · p. 7
  • On AutoBench, learned skills serialize writes to one per turn, and extra turns account for 73% of the summed log-token increase across DeepSeek skills.The excess input is mostly tool outputs re-read on extra turns, with skill text contributing 21%.Source: §5.3 · p. 8
  • Within tasks, 10% more turns correspond to about 14 to 16% more input tokens, while re-billing with only the last turn’s history brings the slope to about 1.The fitted turn slopes are 1.35 to 1.58; output tokens and tool calls rise at slopes of at most 1.03 under DeepSeek.Source: §5.3 · p. 8, Figure 5 · p. 8

6 Design Principles for Skill Optimizers

The design principles target the observed gap between noisy dev success scores and the costs of rules that add turns and input tokens. They recommend accounting for selection noise and turn costs while giving editors the ability and information to control rule accumulation.

6 Design Principles for Skill Optimizers

  • Keep the parent unless a candidate’s gain exceeds what selection alone reports; when success cannot separate skills, let cost decide only beyond run-to-run token noise.A repeated dev win accepts 6 to 17% of unchanged candidates and retains 55 to 94% of edits that truly solve two more dev tasks in simulation.Source: §6 · p. 8
  • Charge an edit for the turns it adds, not its length, by comparing extra turns with its dev gains.Across 59 DeepSeek edits, length change explains 10% of variance in input-token change, versus 93% for turn change.Source: §6 · p. 8
  • Let the editor delete and merge rules, including rules that serialize calls across turns.The three DeepSeek editors shorten the skill in only 1 of 59 edits; batching independent reads and returning only what the next turn needs are also proposed.Source: §6 · pp. 8–9
  • Show the editor how many extra turns and input tokens its earlier rules caused, because both kept and rejected edits often raise input tokens.73% of kept and 76% of rejected edits raise input tokens over their parent; only 5 of 59 keep dev success without adding tokens.Source: §6 · p. 9

7 Conclusion

Across the studied optimizers and settings, learned skills sharpen behavior without expanding it, while offering little average test-success gain and reading more input tokens. The authors attribute the extra cost to accumulated rules becoming extra turns that re-read conversation history.

7 Conclusion

  • Across sixteen learned skills, test success is 0.1 points above the seed, while skills cover about as many new tasks as they lose and read 1.16 times its input tokens.The conclusion links extra cost to accumulated rules, extra turns, and the harness re-sending history.Source: §7 · p. 9

Limitations

BiPA measures optimizer edits after episodes are recorded, but cannot predict which rules add turns or explain why the model chooses a behavior. Its evaluation covers three optimizers from one shared seed, two models, one harness, and three long-horizon benchmarks.

Scope

  • The evaluation applies three published optimizers from one shared seed, using two models in one harness on three long-horizon benchmarks.The paper says BiPA applies unchanged to other optimizers, seeds, models, harnesses, and task families.Source: Limitations · p. 9

Post-hoc measurement

  • BiPA cannot establish that a particular rule caused a behavior; that link rests on matching optimizer edits to the behaviors they name.Controlled rewrites of single rules could test the link causally, while a model of it could estimate edit costs before runs.Source: Limitations · p. 9

Harnesses that compact their history

  • For harnesses that compact history or discount cached history, the graph and cost split still apply, but turn elasticity must be measured from episodes.Re-billing recorded episodes under a compacted history with turns held fixed is described as a first step.Source: Limitations · p. 9

Decomposing success

  • Success cannot be decomposed over turns or rules behind a black-box API because the model’s state is unobservable and no value function can be defined.The paper identifies this missing rule-level credit as the credit a gate would need to judge single rules.Source: Limitations · p. 9

A Experimental Setup

Across the tested skills, success was not reliably higher than the seed, while learned skills generally read more input tokens. The section describes the experimental setup, the turn-elasticity proof, and results on selection, task variability, and run-to-run noise.

A Experimental Setup

  • The evaluation used three task benchmarks, two models in OpenCode, and three released skill optimizers initialized from one shared sample of 160 training episodes.The benchmarks were split into training, development, and test sets; test sizes were 18, 75, and 120 tasks, respectively.Source: A Experimental Setup · p. 14

B Proof

  • With history present, the proof gives turn elasticity e_R > 1, approaching 2 as R grows; it is below 2 exactly when h < 2F.Without history, h = 0 and e_R = 1.Source: B Proof · p. 14

C Selection and Significance

  • Only GPT-6-Luna’s SkillAdam on AutoBench exceeded the seed’s run-to-run spread on test, by 6.4 points versus a 5.8-point spread.The reported development gains did not carry over to test; unchanged candidates could report gains of the same size under the selection rule.Source: C Selection and Significance · p. 14

C.1 Task Model and Test Intervals

  • Only 1 of 18 TB2, 16 of 75 τ 3-bench, and 22 of 120 AutoBench test tasks were solved in some but not all seed episodes.The fitted Beta prior’s expected shares of tasks in none, some, or all runs were close to the observed shares for DeepSeek-V4.1-Flash.Source: §C.1 · p. 15, Figure 6 · p. 15
  • −1.4 points over the nine DeepSeek skills and +0.1 over all sixteen skills were the pooled test differences; neither shows learned skills succeeding more often than the seed.The one-sided 95% upper bounds were +0.6 and +1.6, respectively.Source: §C.1 · p. 15

C.2 Run-to-Run Noise

  • Seven of nine DeepSeek skills differed from the seed by less than its benchmark’s test-run spread, while eight of nine had every run read more input tokens than any seed run.For GPT-6-Luna, only SkillAdam on AutoBench exceeded the seed’s spread; only the three AutoBench skills consistently read more tokens than any seed run.Source: §C.2 · p. 15, Figure 7 · p. 16

C.3 Replaying Selection with Unchanged Candidates

Replaying the optimizers’ acceptance rules on unchanged candidates shows that selection can produce the reported dev gains, which do not reliably carry over to test. The paper also tests stricter acceptance rules, a three-word seed, and whether learned skills cost more on the same tasks.

C.3 Replaying Selection with Unchanged Candidates

  • Unchanged candidates reach the reported dev gains with probability 0.25 to 0.94, while test gains and losses are generally similar in size.For SkillAdam, unchanged candidates reach its gains with probability 0.25, 0.87, and 0.88 on TB2, τ 3-bench, and AutoBench; across ten SkillOpt and GEPA skills, test newly solves 8.9% and newly fails 6.9% of tasks.Source: §C.3 · pp. 16–17

C.4 Acceptance Rules

  • Requiring a dev win to repeat accepts 6.4–17.1% of unchanged candidates and retains 55–94% of true two-task gains; a cost-aware rule would refuse 29 of 30 kept edits.Of the 43 DeepSeek edits with a same-run parent, only one changed success beyond noise, while nine changed input tokens beyond noise.Source: §C.4 · pp. 18–19, Figure 10 · p. 19

C.5 Starting from a Three-Word Seed

  • From a three-word seed, Luna’s AutoBench test success rises 18.3 and 19.4 points, matching runs from the hand-written seed; τ 3-bench shows no improvement.On AutoBench the skills score 43.3% and 44.4%, versus 37.2% for the hand-written seed; on τ 3-bench they score 60.0% and 68.0%, versus 69.3%.Source: §C.5 · p. 19, Figure 11 · p. 20

D.1 Extra Cost on the Same Tasks

  • On the same tasks the seed and learned skills all solve in every test run, each DeepSeek skill reads 1.08 to 1.29 times the seed’s input tokens.For seven of nine skills, the paired bootstrap interval of ρ lies above 1.Source: §D.1 · p. 21

D.2 Where the Extra Tokens Come From

Extra input-token cost is driven mainly by additional turns, which re-read prior context, rather than by longer skill text. Learned skills add many rules, especially verification rules, and remain more costly under cache pricing and compacted-history accounting.

D.2 Where the Extra Tokens Come From

  • Turn-related terms account for 73% of the summed log-token increase across nine DeepSeek skills, versus 21% for skill text.Under GPT-6-Luna, turn terms add +0.86 to log input tokens across six skills, compared with +0.04 from skill text.Source: §D.2 · p. 21
  • Tool outputs are the largest part of extra input tokens for seven of nine DeepSeek skills, which read 5 to 32% more input tokens per task than the seed.Extra turns re-read earlier tool outputs; on TB2, skill text leads for GEPA, while agent and user messages lead for SkillOpt.Source: §D.2 · p. 21

D.3 Cost under Cache Pricing and Compacted History

  • Even with cached-input pricing or compacted history, every learned skill costs more than the seed: 1.05 to 1.27 times under cache pricing and 1.05 to 1.36 under compaction.Compacted-history re-billing flattens the mean turn slope from 1.47 to 1.04.Source: §D.3 · p. 22

E.1 Skill Growth and Rules

  • Among 337 added rules across nine DeepSeek skills, 52% ask for verification, 18% for serialization, and none for reading less.The nine skills add 37 rules on average; under Luna, 41% of added rules ask for verification and none limit reading.Source: §E.1 · p. 23

E.2 Tool Calls per Turn

Learned skills serialize AutoBench writes, and that behavior accounts for nearly all the reduction in estimated token cost when calls are packed into turns like the seed. Optimizer selection does not distinguish edits by their input-token cost, and transferred skills mostly score below the seed while retaining much of that cost.

E.2 Tool Calls per Turn

  • On AutoBench, at most 0.3% of writes under each learned skill share a turn, versus 56% under the seed; packing calls as densely as the seed halves mean log ρ from 0.17 to 0.08.Nearly all the reduction comes from serialized writes, which the rules request because concurrent writes can overwrite changes to the simulated world.Source: §E.2 · pp. 23–24

E.3 Kept and Rejected Edits

  • Kept and rejected edits raise log input tokens by 0.100 and 0.128 on average, respectively, with no detected difference between them (p = 0.46).Only five of 59 edits both preserve dev success and avoid adding tokens.Source: §E.3 · p. 24

F.1 Skill Transfer

  • Transferred skills mostly score below the seed while retaining extra input-token cost: 1.08 to 1.38 times under Luna for DeepSeek τ 3-bench skills, and 1.24 to 1.49 times on AutoBench for TB2 skills.AutoBench skills run on TB2 read 1.00 to 1.22 times the seed’s input tokens.Source: §F.1 · p. 24, Figure 15 · p. 24

F.2 Trajectory Analysis

Trajectory analysis examines how learned skills shape agent behavior and use: they sharpen rather than expand behavior, with policy-first procedures that keep actions within the request and proceed in verified steps. Skill-loading route affects whether a skill is loaded, but not its token cost once loaded.

F.2 Trajectory Analysis

  • Across six model–benchmark pairs, the seed gained 36 tasks and lost 13 versus no-skill, while learned skills gained 57 and lost 62 versus seed across optimizers.When a learned skill raised success, episodes were 1.61 times as likely to agree more than when success was unchanged (Mantel–Haenszel odds ratio; p = 0.03).Source: §F.2 · p. 25

F.3 Skill Loading

  • SkillOpt’s τ 3-bench skill read 1.226 times the seed’s input tokens always resident and 1.230 on demand; on demand raised both skills’ cost 6 to 7%.Without an activation instruction, the seed and SkillOpt τ 3-bench skills never loaded (0 of 75 episodes), while GEPA’s τ 3-bench skill loaded in 60 of 75 episodes.Source: §F.3 · pp. 25–26, Figure 17 · p. 26

G.1 Text We Supply to the Optimizers / G.2 The Seed Skill

  • The optimizers used their upstream templates unchanged; the study supplied GEPA’s objective and SkillAdam’s five short prompt sections, and most runs began from a shared verification-before-completion seed skill.The seed applied one rule—run a fresh, complete check before claiming completion—to every task; only Appendix C.5 runs started elsewhere.Source: §G.1 · p. 26, §G.2 · p. 27

G.3 Skills Learned under DeepSeek

  • All nine DeepSeek optimizer runs ended with a skill other than the seed, and the paper releases those skills in its skills/ directory.The shown SkillAdam TB2 skill adds four lines, removes two, and contains 654 words versus 580 in the seed.Source: §G.3 · p. 28

G.4 Skills Learned from the Three-Word Seed

  • The GPT-6-Luna Appendix C.5 runs started from a three-word seed and returned four skills, with word counts including front matter.The listed examples include SkillOpt and GEPA skills for AutoBench and a SkillOpt skill for τ 3-bench.Source: §G.4 · pp. 28–29
  • The skills match actions to the exact requested records, preserve unrelated settings, and explain when an operation would exceed the request’s scope.They distinguish completed, unavailable, and unresolved parts rather than proceeding with an overbroad operation.Source: §G.4 · p. 30
  • Troubleshooting proceeds one actionable step at a time, uses the user’s result to guide further checks, and retests the original issue after a fix.For MMS, the skill names relevant causes to check and requires a reboot after resetting APN settings.Source: §G.4 · p. 30
  • The GEPA τ 3-bench skill prescribes a policy-first workflow: authenticate, inspect authoritative guidance, resolve the actual request, and verify outcomes before reporting completion.It also requires following prescribed transfer procedures and trying applicable supported actions before transferring when policy permits.Source: §G.4 · p. 30
  • The skill requires one tool call per step, explicit confirmation before policy-governed database changes, and inspection of the final state after acting.It also distinguishes agent tools from user tools and requires exact identifiers and arguments when asking users to act.Source: §G.4 · p. 30

Related papers

Loading 2610.37201v1…