CIPO: Counterfactual Imagination Policy Optimization for Adaptive Tool Granularity Selection

1. Introduction

• Successful tool trajectories contain reusable local routines, but skill use requires state-dependent choices because skills can hide useful observations or execute unnecessary steps.The paper frames the needed training signal as comparing atomic and skill choices from the same state.Source: §1 · pp. 1–3
• CIPO mines recurring tool-use patterns under a budget, then instantiates them as executable skills with parameter binding, execution traces, and soft interruption.Atomic tools remain available for fine control alongside the composite skills.Source: §1 · p. 3
• CIPO improves task success and decision efficiency over baselines by learning state-appropriate choices between atomic tools and composite skills.The authors report that gains do not come simply from invoking skills more often.Source: §1 · p. 3

Experience and Workflow Reuse

• Memory and workflow reuse retain prior experience as context or guidance, leaving open when reusable procedures should become actions rather than remain guidance.The paper notes that context-level reuse does not reduce low-level tool decisions.Source: §2 · p. 4

Tool Composition and Skill Construction

• Tool composition and skill-construction methods build higher-level execution units, but often leave skill-library ambiguity and atomic-versus-skill selection underexplored.Examples include LLMCompiler, LLM-Tool Compiler, AWO, SkillRL, SKILL0, and Trace2Skill.Source: §2 · p. 4

Policy Optimization for Tool-Use Agents

• Tool-use policy optimization and tool-agent systems commonly keep individual invocations as actions or assume fixed action granularity, unlike CIPO's focus on granularity choice.The paper contrasts CIPO with ReAct, Toolformer, ToolLLM, Gorilla, GRPO, and related policy-optimization work.Source: §2 · pp. 3–4

3.1. Problem Formulation

• Adaptive tool granularity selection chooses between a single atomic operation and a composite skill according to the current history and subtask.Atomic actions offer intermediate observations or fine control; a skill suits a matching reusable procedure.Source: §3.1 · pp. 4–5

3.2. Budget-constrained Skill Mining

• CIPO abstracts successful tool calls by tool name and parameter-key signature, then mines recurring patterns with BPE-style merging under explicit candidate and length constraints.The output is a compact symbolic candidate set, not yet executable skills.Source: §3.2 · p. 5

3.3. Executable Skill Instantiation

• CIPO turns mined sequences into callable skills with compact parameter interfaces, runtime selection control, internal execution traces, and soft interruption on failure.Internal parameters can flow from earlier outputs; unresolved parameters remain exposed to the policy.Source: §3.3 · pp. 5–6

3.4. CIPO: Counterfactual Imagination Policy Optimization

• The paired outcome difference supplies a bounded supplementary reward, while the original task reward remains the main policy-optimization signal.The adjusted rollout group is then optimized with the standard GRPO objective.Source: §3.4 · pp. 7–8
• For each base rollout, CIPO branches at the first eligible decision, replacing the chosen action with a feasible action at the other granularity from the same history.The same policy completes the branch, and only one branch is made per base rollout to limit rollout growth.Source: §3.4 · p. 7
• At inference, the policy selects directly from the augmented action space; counterfactual branching is used only during training.CIPO fixes the rollout budget per instruction and fills unused branch slots with additional base rollouts.Source: §3.4 · pp. 7–8

4.1. Experimental Setup

• Evaluation covers TOOLATHLON, TOUCAN, and TRAJECT-Bench with Qwen2.5-7B and Llama3.1-8B, using Tool F1, TSR, and decision compression metrics.TSR averages five independent judge evaluations; a smaller Comp./Raw ratio indicates stronger compression.Source: §4.1 · pp. 8–9

4.2. Main Results

• CIPO achieves the strongest overall performance across three benchmarks and two backbones, with the best Tool F1 and TSR in all six settings.The main-results table reports Tool F1 and TSR across the benchmark-backbone settings.Source: §4.2 · pp. 9–10, Table 1 · p. 9
• Avg.Comp./Raw of 0.845 ± 0.03 corresponds to about 15.5% fewer policy-level decisions after skill expansion, while CIPO maintains the strongest overall TSR and competitive Tool F1.This compression improvement over composition and skill-based baselines does not come simply from invoking more skills.Source: §4.2 · p. 10

4.3. Skill Usage and Granularity Selection

• CIPO increases training-time skill usage toward the coverage-derived reference, whereas GRPO + Skills remains near its initial level.The reference estimates library-level coverage and is not a state-level optimal policy.Source: §4.3 · p. 10, Figure 3 · p. 10

4.4. Ablation Studies

• Averaged across benchmarks and backbones, CIPO improves F1, TSR, and decision compression over GRPO + Skills, while removing the mining budget weakens performance.The reported ablation comparisons support benefits beyond simply adding skills and highlight the value of a compact, controlled library.Source: §4.4 · p. 11, Table 2 · p. 11
• CIPO-first achieves the best average F1, TSR, and decision reduction compared with CIPO-random under the same rollout budget.CIPO-first branches at the first eligible granularity decision; CIPO-random samples among eligible decisions.Source: §4.4 · p. 11

5. Conclusion

• Across three benchmarks and two backbones, CIPO improves tool trajectory quality, task success, and decision compression by learning when to use composite skills versus atomic control.The conclusion attributes these gains to budget-constrained skill construction and paired outcome differences from the same decision context.Source: §5 · p. 11

A.1. Input Corpus and Token Abstraction

• Successful tool-call sequences are abstracted into tokens that preserve tool names and argument keys, then mined by repeatedly merging frequent adjacent pairs into longer patterns.Concrete argument values are omitted to reduce spurious variants and expose recurring tool structures.Source: §A.1 · p. 17, §A.2 · p. 17

A.3. Budget Constraints

• Mining limits merge rounds, pair frequency, and skill length, then prunes low-use or too-short candidates; retained libraries contain at most K candidates, each no longer than Lmax.The stated bounds follow because each merge creates at most one candidate and accepted merges obey the maximum-length constraint.Source: §A.3 · pp. 17–18, §A.4 · pp. 18–19

A.5. Computational Complexity

• With at most K merge rounds, mining takes O(KN) time and O(N + K) memory, where N is the initial corpus token count.The per-round scans are linear in the current corpus, which only shrinks as merges proceed.Source: §A.5 · p. 19

A.6. Semantic Skill Naming and Output

• Retained symbolic sequences are registered with tool schemas and metadata, then converted into callable skills with explicit parameters, dataflow, selection control, traces, and soft interruption.This registration output feeds skill instantiation, turning mined sequences into executable actions.Source: §A.6 · p. 19

B. Details of Executable Skill Instantiation

• Executable skills expose parameters that require policy control, pipe values resolved from earlier outputs, support candidate selection, and return traces when execution is interrupted.This lets composite actions reduce repeated decisions without becoming fixed, opaque pipelines.Source: B. Details of Executable Skill… · pp. 20–21

C. CIPO Training Algorithm

• CIPO fixes the rollout budget at G = 4, attempts at most one counterfactual branch per base rollout, and uses valid base-branch pairs for supplementary rewards before a group-relative update.It samples additional base rollouts when needed to fill the four-rollout group; all four rollouts enter the update.Source: C. CIPO Training Algorithm · pp. 21–22

D. Counterfactual Replacement Rules

• For each base rollout, CIPO branches at the first feasible alternative-granularity action, preserving the prefix and rejecting replacements whose arguments cannot be resolved.For atomic-to-skill replacements, it selects the shortest feasible skill, using frequency statistics to break ties.Source: D. Counterfactual Replacement Rules · p. 23
• CIPO accepts a replacement only when its arguments are executable from the decision context, and otherwise rejects it and scans later actions.When feasible, a branch replaces a skill with its first internal atomic tool or tests a candidate skill, then returns control to the policy after the replacement step.Source: D. Counterfactual Replacement Rules · pp. 24–25

E. Reward Composition

• The base reward combines task progress, skill behavior, and execution efficiency; task reward reflects reference-tool coverage and order, while skill reward favors relevant successful calls and penalizes irrelevant or repeated calls.The total skill bonus is clipped to discourage calling skills solely for reward.Source: E. Reward Composition · pp. 25–26
• Efficiency bonuses reward shorter decision trajectories only when task progress is sufficient, while very short rollouts with low coverage receive an under-exploration penalty.The efficiency bonus is gated by task completion and coverage, preventing high reward for stopping early without solving the task.Source: E. Reward Composition · p. 26
• The bounded counterfactual adjustment supplements, rather than replaces, task reward, comparing paired rollouts that share a prefix but differ in one granularity choice.Both rollouts continue under the current policy, giving feedback about which action worked better in the matched decision context.Source: E. Reward Composition · pp. 26–27

F. Training Dynamics

• CIPO’s logged group-level training reward combines task progress, skill behavior, efficiency, and a bounded counterfactual adjustment.The curves are used to analyze optimization, not to report final evaluation performance.Source: F. Training Dynamics · p. 27
• Reward generally increases during training across TOOLATHLON, TOUCAN, and TRAJECT-Bench, suggesting the counterfactual branch signal can be optimized with task reward.Source: Figure 4 · p. 27
• On TRAJECT-Bench, both models improve rapidly before rewards fluctuate around a higher plateau amid diverse trajectories and varying rollout difficulty.Source: F. Training Dynamics · p. 27
• Rewards increase or stabilize at higher levels across all benchmark-backbone combinations, while final evaluation results are reported separately.The curves cover TOOLATHLON, TOUCAN, and TRAJECT-Bench with Qwen-7B and Llama-8B.Source: F. Training Dynamics · pp. 27–28
• Local reward fluctuations are expected because online updates sample different tasks, base rollouts, and counterfactual branches.Branches can expose better or worse granularity choices under the current policy, varying group rewards.Source: F. Training Dynamics · p. 27

G. Derivation of the Coverage-derived Skill Usage Reference

• The coverage-derived reference estimates the expected skill-call share after compressing atomic calls covered by the mined skill library.It assumes uncovered calls remain atomic and covered calls are replaced by skills of frequency-weighted average length m.Source: G. Derivation of the Coverage-derived… · p. 28
• The reference ratio is c/m divided by c/m + (1 − c), where c is coverage and m is average skill length.Higher coverage raises the reference ratio, while longer skills and uncovered atomic calls lower it.Source: G. Derivation of the Coverage-derived… · p. 28
• This library-level reference is computed for the TOOLATHLON library with Qwen2.5-7B and is not an optimal policy or state-level reference.Figure 3 measures online training rollouts, whereas Table 12 reports final evaluation trajectories, so their absolute usage ratios need not match.Source: G. Derivation of the Coverage-derived… · p. 28

Component effects

• CIPO further improves F1, TSR, and decision compression beyond GRPO + Skills, while removing counterfactual reward weakens performance.GRPO + Skills itself improves those metrics over the reported SFT baseline; removing the mining budget also weakens performance relative to full CIPO.Source: H. Full Ablation Results · p. 29

Skill construction and instantiation

• Removing budgeted mining increases skill-library redundancy, while removing executable instantiation also hurts performance.Reliable skills require parameter interfaces, internal dataflow, selection control, execution traces, and soft interruption.Source: H. Full Ablation Results · p. 30

Branch-point selection

• CIPO-random and CIPO-late are weaker than CIPO-first on average under the same rollout budget, supporting first-eligible branching as a cleaner comparison.The variants select eligible branch points randomly or prefer later ones, respectively.Source: H. Full Ablation Results · p. 30
• CIPO-all uses extra computation and has lower average F1 and TSR than CIPO-first, despite stronger compression on TOOLATHLON and TOUCAN.Its compression is worse on TRAJECT-Bench, so branching at every eligible decision is not automatically better.Source: H. Full Ablation Results · p. 30

Reward sensitivity

• The default counterfactual-reward setting has the highest overall F1 and TSR among tested variants, balancing a useful granularity signal against task reward.Reducing reward scale weakens the signal; excessive scale or a wide clipping range can make it too dominant, while narrow clipping suppresses useful differences.Source: H. Full Ablation Results · p. 30

I. Hyperparameters and Training Details

• CIPO’s five-stage pipeline mines patterns, instantiates executable skills, performs skill-enabled SFT, optimizes with counterfactual branches, and evaluates without branching.The same pipeline is used for Qwen-7B and Llama-8B.Source: §I · p. 30
• Skill mining uses budget-constrained BPE-style patterns from successful trajectories, abstracting tool names and parameter-key signatures while removing task-specific argument values.Source: §I · p. 30
• Mined candidates become executable skills with parameter schemas, execution plans, internal dataflow maps, traces, and interruption behavior.Source: §I · p. 31
• Policy optimization uses G = 4 rollouts per instruction, including two base rollouts and up to two valid counterfactual branches; only valid pairs receive supplementary reward.All four rollouts contribute to group-relative normalization, with extra base rollouts when fewer than two valid branches are available.Source: §I · pp. 31–32
• At inference, CIPO selects from the augmented tool library without counterfactual branching, using deterministic decoding and a fixed execution budget.Tool F1 is order-aware; TSR uses an LLM judge, and Comp./Raw compares compressed decisions with expanded atomic calls.Source: §I · p. 33

• Comp. and Raw report compressed policy decisions and expanded atomic calls; Avg. Comp./Raw is the mean of six setting-level ratios.Source: J. Additional Results and Uncertainty… · p. 34
• Skill Use varies across benchmarks, with lower usage on TRAJECT-Bench consistent with its weaker decision-compression gain.Source: J. Additional Results and Uncertainty… · p. 34
• Table 12 adds trajectory-quality, skill-use, reliability, and decision-efficiency metrics, including tool precision, recall, next-tool accuracy, progress rate, and skill coverage.Source: J. Additional Results and Uncertainty… · pp. 33–34, Table 12 · p. 35
• Tool F1 and Comp./Raw uncertainty use episode-level bootstrap 95% confidence intervals, while TSR reports mean and standard deviation across five judge runs.Source: J. Additional Results and Uncertainty… · p. 34, Table 13 · p. 35

K. Dataset Statistics

• The benchmarks cover long-horizon dependencies in TOOLATHLON, large-scale tool selection in TOUCAN, and coherent, ordered tool-use trajectories in TRAJECT-Bench.TOOLATHLON averages about 13.5 tool invocations per episode, while TRAJECT-Bench averages about 6.4.Source: K. Dataset Statistics · pp. 35–36

L. Task Success Evaluation

• Each benchmark–backbone–method setting evaluates 100 episodes in five independent judge runs, using the task, reference and actual tool chains, and final response.Source: L. Task Success Evaluation · p. 36
• The LLM judge assesses tool-use strategy rather than simulator artifacts, accepts functionally equivalent valid paths, and allows partial completion on long tasks.Source: L. Task Success Evaluation · p. 36
• Task Success Rate (TSR) counts episodes judged yes or partial, while strict success counts only yes.Source: L. Task Success Evaluation · pp. 36–37

L.1. Strict and Execution-Based Task Success

• Strict judge-based evaluation counts only yes labels across six settings, while execution-based verification checks final environment states on TOOLATHLON tasks only.The strict evaluation uses gpt-4o; the execution-based evaluation covers the same 100 TOOLATHLON tasks across three training seeds and two backbones.Source: §L.1 · pp. 38–39
• CIPO exceeds GRPO + Skills by 6.28 percentage points in yes-only strict success and by 7.50 percentage points in the TOOLATHLON execution-based comparison.Source: Table 15 · p. 39, §L.1 · p. 39

M. Effect of the Skill-mining Budget

• TSR rises from K = 10 to K = 50, then declines at K = 100 and 200; K = 50 gives the best average TSR.A small budget can miss reusable routines, while a larger one can add redundant skills and ambiguity.Source: M. Effect of the Skill-mining Budget · pp. 39–40, Figure 6 · p. 40

N. Task-level Decision Compression

• Figure 7 plots policy-level decisions on the horizontal axis and expanded atomic tool invocations on the vertical axis; the diagonal marks no compression.Source: N. Task-level Decision Compression · p. 40, Figure 7 · p. 41
• More CIPO episodes lie above the no-compression diagonal than GRPO + Skills episodes, showing that policy decisions can trigger multiple atomic operations through skills.The pattern appears across many episodes rather than only a few isolated cases.Source: N. Task-level Decision Compression · p. 40, Figure 7 · p. 41

O. SFT Initializations and GRPO Baselines

• Standard GRPO uses atomic actions, whereas GRPO + Skills and CIPO use atomic and skill actions; CIPO additionally uses matched counterfactual branches and paired reward adjustments.Source: O. SFT Initializations and GRPO… · pp. 41–42
• GRPO + Skills is the skill-enabled baseline for evaluating CIPO beyond access to executable skills; standard GRPO is the atomic-only baseline.Standard GRPO versus GRPO + Skills also differs in action space and SFT preparation.Source: O. SFT Initializations and GRPO… · p. 41

P. Baseline Implementation Details

• All methods share the same evaluation tasks, backbones, simulator, decoding budget, and metrics; auxiliary resources are built only from training trajectories.Test episodes are excluded from memory construction, workflow induction, skill mining, and hyperparameter selection.Source: P. Baseline Implementation Details · pp. 42, 44
• Memory and workflow baselines provide retrieved procedures or guidance in prompts, while ReAct and ToolLLM operate over atomic tools only.ExpeL adds retrieved examples and insight rules; MemP, Mem0, AWM, and AutoManual provide procedural guidance.Source: P. Baseline Implementation Details · pp. 42–43
• Skill and tool-composition baselines expose reusable or compiled procedures, but do not train atomic-versus-skill choices with matched counterfactual branches.Adaptations include skill banks, internalized skill descriptions, callable subsequences, meta-tools, and compiled execution structures.Source: P. Baseline Implementation Details · p. 43
• Internal GRPO variants distinguish atomic-only training, skill-enabled action spaces, less-controlled mining, and first-versus-random branch selection.CIPO-random uses group size G = 4 and samples among all eligible granularity decisions.Source: P. Baseline Implementation Details · p. 43

Q. Broader Impact

• CIPO’s broader-impact discussion says executable skills can reduce repeated low-level decisions and help complete complex workflows with fewer policy-level actions.Suggested settings include research assistance, software workflows, document processing, and API-based data analysis.Source: Q. Broader Impact · p. 44
• Opaque skills may hide intermediate steps; CIPO uses trace-aware interruption for inspection and recovery, but its experiments do not cover high-stakes deployment.The authors identify human approval, sandboxing, tool-permission control, logging, and domain-specific checks as safeguards for such settings.Source: Q. Broader Impact · p. 44