PaperlayerView exact revision on arXiv

Source-linked AI summary

Small agents should know what their tools do

Alexander Kovalenko, Artem Redchych

arXiv:2610.35981v1cs.SE

TL;DR

Small open models can call tools but often fail to account for the effects of their calls, limiting evaluation of safety and efficiency. EffectWorld makes tool effects explicit and uses replay verification, executable tool programs, and trace optimization to generate and assess training data. The paper reports gains across in-distribution and held-out settings, while finding that positive examples—not negative objectives—carry the learning.

  • Problem

    Small open models call tools without reliably accounting for their effects, while common benchmarks largely score whether calls are well formed.

    Source: §1 · p. 1

  • Method

    EffectWorld declares tool effects, idempotence, and resource footprints, then uses replay verification, tool programs, and trace optimization for training and evaluation.

    Source: §1 · p. 2, §3.1 · p. 5

  • Results

    Across settings, SFT improves task success and targeted positives shift cue-specific behavior; filtering refusal templates restores BFCL non-live accuracy to 84.8.

    Source: §1 · p. 2, §8 · pp. 16–17

  • Takeaways & Limitations

    Positive examples transfer along the cue they demonstrate, whereas suppression and preference objectives on verified negatives do not improve over the SFT control.

    Source: §8 · p. 17

    EffectWorld covers nine task kinds in one training domain, uses template utterances and a scripted user, and does not evaluate BFCL multi-turn or τ 2-bench end to end.

    Source: §7 · p. 16

Abstract

from arXiv · show

Small open language models can call tools, but they do not reason about what a call does: they delete without asking, repeat writes, re-read state they already hold, and issue one call per turn where a short program would do. We introduce EffectWorld, an executable world of effect-typed tools in which every tool declares its effect, whether it is idempotent, and which resources it reads and writes. These declarations let a replay verifier decide whether a trace is effect-safe without labels or judge models, let an executor run tool programs of several dependent calls in one turn, and let a compiler-style optimizer turn wasteful traces into verified training signal; together they make EffectWorld a benchmark of effect safety and call efficiency. Low-rank adapters trained on this data deliver large gains: on Qwen3-4B, they raise task success from 53% to 99% in distribution, from 52% to 81% on a held-out split with unseen task kinds and distractor tools, and from 74% to 83% on an unseen domain, with about a third of the output tokens. The gains carry over to other model families. On the Berkeley Function-Calling Leaderboard (BFCL), the adapter matches the base model's live accuracy and raises multi-turn accuracy from 21.9 to 26.6, and a recipe with near-miss declines lifts the BFCL average above the base model's while matching its irrelevance detection. Along the way we uncover and fix a pitfall that in-domain evaluation alone would miss: refusal templates in widely used public tool-use corpora teach a small model not to call tools, cutting BFCL non-live accuracy from 88.0 to 31.4, and filtering them restores 84.8. We also find that positive examples, not negative ones, carry the learning: suppression and preference objectives on the verified negatives add nothing over an SFT-only control, while targeted positives move exactly the behaviour they show.

1 Introduction

The paper makes tool effects explicit across data generation, training, and evaluation, using them to check safety and optimize call traces. It reports that data and positive examples matter: refusal templates harm tool calling, while negative-training objectives do not improve on the SFT control.

EffectWorld

  • EffectWorld makes each tool’s effect, idempotence, and resource footprint explicit so replay verification can check safety and optimization can identify unnecessary calls.The optimizer turns inefficient traces into labelled negative examples.Source: §1 · p. 2

Negative training

  • Efficiency negatives are calls the optimizer verifies as unnecessary, while rule-based and policy negatives target incorrect or unsafe actions at their shared prefix.The method adapts LUNE’s span-localized LoRA recipe, replacing its objective with unlikelihood or negative preference optimization.Source: §1 · p. 2

Benchmark and data contributions

  • The paper introduces an executable world with a hard split for unseen task kinds and distractor tools, plus a test-only commerce domain.Its contributions also include tool programs for dependent calls and a footprint-based optimizer with 19 negative classes.Source: §1 · p. 2

Objective comparison

  • With matched positive-data exposure, UL, NPO, and DPO do not beat the SFT control on any held-out split.The comparison also documents three loss-scaling defects that made an initial round fail silently.Source: §1 · p. 2

Findings about data

  • In a small model, refusal templates in public tool-use data collapse willingness to call tools on BFCL despite perfect in-domain metrics, and filtering those templates fixes the problem.Positive examples transfer along demonstrated cues: declining when a tool is absent does not teach declining off-topic requests.Source: §1 · p. 2

2 Related work

Prior tool-use work builds call-capable models and benchmarks increasingly complex tool interactions, but much of it checks calls or outcomes without declaring what each call does. The paper connects effect declarations to compiler-style optimization and verifier-derived training data.

Tool-use data

  • Toolformer and ReAct established self-annotation and interleaved reasoning with actions; later corpora scale synthesis, while none of these corpora records what a call does to the world.The paper uses three public corpora, whose refusal-style turns are examined in §6.1.Source: §2 · p. 2

Evaluation

  • BFCL evaluates call structure, relevance, irrelevance, and multi-turn interaction, while other benchmarks test nested calls, policies, final state, or implicit dependencies.Unlike task-written checks, this verifier derives checks from tool declarations, which also drive its optimizer.Source: §2 · pp. 2–3

Efficient tool use

  • LLMCompiler and CodeAct support broader executable plans, whereas EffectWorld’s programs allow only references to earlier-call results, keeping them within native tool-call formats.Footprint-based static analysis identifies dead, redundant, or reorderable calls.Source: §2 · p. 3

Safety and tool interfaces

  • Prior work studies side effects, reversibility, runtime guards, and effect-aware interfaces; this paper trains adapters to use declared effects, though confirmation habits can remain tied to tool names.The cited interface work notes that protocol annotations are advisory and may be too coarse to specify boundary behavior.Source: §2 · pp. 3–4

Learning from negative examples

  • Unlike prior negative-training approaches built around failed trajectories or preference pairs, this method localizes labelled negatives to one turn and uses verifier-derived examples offline.In this setting, the paper reports that these negatives do not improve over SFT with the same positives.Source: §2 · p. 4

3 Data

EffectWorld combines effect-typed tools, executable call programs, generated tasks, replay-based scoring, and an optimizer that turns wasteful traces into labelled training pairs. Its evaluation spans workspace tasks, unseen task kinds, and an unseen commerce domain.

3 Data

  • EffectWorld is an executable world of effect-typed tools, with training data combining generated traces, optimizer-derived negatives, and public tool-use data.Its tools carry effects, idempotence flags, and footprints that name the resources they read and write.Source: §3 · p. 4, §3.1 · p. 4

3.1 EffectWorld: effect-typed tools and effect safety

  • Four effect types distinguish pure computation, state reads, reversible writes, and destructive changes, while idempotence and footprints independently characterize repeated calls and accessed resources.These declarations let the verifier check confirmation, duplicate-write, and stale-state rules without labels or a judge model.Source: §3.1 · pp. 4–5, Table 1 · p. 5

3.2 Tool programs

  • A tool program places multiple dependent native tool calls in one assistant turn, using $ref to pass earlier results; the executor orders calls by dependencies and footprint conflicts.Destructive calls require a prior user confirmation, which acts as a barrier.Source: §3.2 · pp. 5–6, Figure 2 · p. 6

3.3 Tasks, verifier and scripted user

  • Generated tasks include a reference solution; replay scoring checks goal predicates, required answer facts, collateral changes, and effect safety, while recording call and turn excess.Naive test plans average 4.52 calls per task versus 2.11 for optimal plans; the scripted user responds deterministically.Source: §3.3 · p. 6

3.4 Evaluation splits

  • Evaluation uses separate seeds across a test split, a hard split with distractor tools and unseen task kinds, and a 547-task commerce split with tools absent from training.The commerce setting tests whether efficiency and effect-safety behavior transfers to unseen tools.Source: §3.4 · pp. 6–7

3.5 Trace optimizer and negative classes

  • The optimizer applies constant folding, redundant-read and dead-call elimination, batching, and scheduling, accepting rewrites only when replay preserves final state without new tool errors.It reduces every naive trace to the optimal plan’s call count; its changes become good/bad action pairs at the first differing turn, alongside rule-generated mistake classes.Source: §3.5 · pp. 7–8

3.6 Public data and training mixes

  • The raw SFT mix contains 5,976 gold world trajectories and 15,001 public trajectories; the clean mix removes 1,751 public trajectories containing refusal templates.The negative variants use capped world-class pairs, with refusal pairs added to the clean mix.Source: §3.6 · p. 8

4 Experimental setup

The setup compares shared supervised fine-tuning with recipes that alter training data or add negative objectives, then evaluates tool-world behavior and BFCL under a common serving protocol. It also tests whether findings transfer across model backbones.

4.1 Training objectives

  • SFT is the shared first stage; negative-objective variants suppress the divergent bad-action span while retaining good-action likelihood and matching the control’s positive-data exposure.The compared objectives are UL, NPO, and DPO, continued from the SFT adapter for 300 steps.Source: §4.1 · pp. 8–9

4.2 Recipes

  • Recipe variants change the training mix or add a second-stage objective, with SFTclean serving as the reference recipe for the other variants.SFTdecline removes the needed tool, SFTeffects annotates effect and idempotence, and SFTnear-miss offers related but unsuitable tools.Source: §4.2 · p. 9, Table 2 · p. 9

4.3 Backbones

  • Three additional backbones—Qwen3-1.7B, Phi-4-mini-instruct, and Falcon3-3B-Instruct—use the SFTclean and SFTdecline recipes.Each model is trained and evaluated in its own tool-call format; some Falcon3 examples exceeding 4,096 tokens are dropped.Source: §4.3 · pp. 9–10, §4.2 · p. 9

4.4 Evaluation and compute

  • Evaluations use a shared serving stack; EffectWorld tasks run in program and stepwise modes, while BFCL v4 covers offline groups and stateful multi-turn settings.World runs allow 12 model turns; BFCL uses model-specific official handlers and reports its own accuracies per group.Source: §4.4 · p. 10
  • Reported model differences use paired bootstrap intervals over task or prompt sampling, and each job runs on one A100 40 GB.The intervals do not cover seed variation; a second training seed is reported in §7.Source: §4.4 · p. 10

5 Results

The SFT recipes achieve near-perfect in-distribution performance and improve success on the hard and commerce splits, but transfer of effect safety is incomplete. On BFCL, the clean recipe matches live accuracy and improves multi-turn accuracy, while negative-objective variants add no consistent held-out gains over the SFT control.

5.1 In-distribution test split

  • SFTraw solves 99.4% of in-distribution tasks in program mode and all stepwise, with no excess calls or violations; SFTclean reaches 99.2% and 100%.Program mode uses 41% fewer action turns, 25% fewer model calls, and 23% fewer prompt tokens than stepwise mode at the same success.Source: §5.1 · p. 10

5.2 Hard split

  • SFTraw raises hard-split success from 52% to 80–81% in program mode; SFTclean reaches 81.6% and 29.8 points above the base.The split includes unseen task kinds and distractor tools, but both recipes retain failures involving ambiguous contacts and incorrect entity choices.Source: §5.2 · pp. 10–11

5.3 Commerce split

  • On 547 commerce tasks with 17 unseen tools, SFTraw and SFTclean raise success from 74% to 79% and 83%, but unconfirmed destructive calls rise from 7 to 24–37.The adapters also skip the required order read when changing a delivery address.Source: §5.3 · p. 11

5.4 BFCL

  • SFTclean matches the base on BFCL live accuracy at 76.5 and raises multi-turn accuracy to 26.6 against 21.9, while irrelevance detection falls.Non-live irrelevance detection falls from 88.8 to 75.0, and live irrelevance detection from 81.1 to 74.4.Source: §5.4 · pp. 11–12

5.5 NESTful

  • On NESTful, SFTclean raises tool-name F1 to 0.52 and argument match to 17.3%, but exact sequence match remains near zero across models.Models rarely chain results through $refs because training references identifiers and addresses, not computed values.Source: §5.5 · p. 12

5.6 Variants trained with negatives

  • UL and NPO do not fix targeted hard-split failures and reduce commerce success; unlikelihood lowers hard-split program success by 3.5 points and NPO by 3.9.On commerce, suppression costs 20.8 points under unlikelihood and 12.4 under NPO; the multi-item cart task falls from 96% to 30% under unlikelihood.Source: §5.6 · p. 12
  • Negative objectives alter call propensity rather than consistently improving performance: unlikelihood raises BFCL live and multi-turn accuracy but lowers irrelevance detection, while DPO reduces violations yet loses success.Removing refusal pairs recovers irrelevance detection but removes live and multi-turn gains; using only refusal pairs sharply lowers irrelevance detection.Source: §5.6 · pp. 12–13
  • Negative-objective variants lose to the control on hard and commerce splits; the first round also produced repeated tokens or unclosed tool-call blocks.That first round had three objective-definition defects, including suppressing closing tokens and scaling the loss against the wrong reference.Source: §5.6 · pp. 12–13

5.7 Recipes that change the data

SFTnear-miss is the first recipe to match the base model on BFCL irrelevance detection and exceed its average, but commerce confirmation worsens. Across other backbones, SFT gains transfer to the hard split more reliably than to unseen commerce, and recipe effects vary by model.

5.7 Recipes that change the data

  • SFTdecline adds missing-tool decline tasks, while SFTeffects exposes effect declarations under renamed tools to encourage confirmation beyond a memorized tool name.The recipes change one element of SFTclean each; neither reaches the base on its targeted BFCL behavior alone.Source: §5.7 · p. 13
  • SFTdecline leaves BFCL irrelevance detection unchanged from SFTclean and lowers live accuracy by 8.0 points, though it leads the hard split by 2.2 program-mode points and 4.1 stepwise.It reduces calls by 0.37 per commerce task but increases unconfirmed destructive calls from 24 to 42 and loses 4.8 points in commerce success.Source: §5.7 · p. 13
  • SFTeffects reduces unconfirmed destructive calls on commerce from 24 to 19 and raises purchases made after confirmation from 32% to 50%, without a significant violation reduction.It also loses 5.0 points on BFCL multi-turn; the passage reports the best NESTful name F1 as 0.55.Source: §5.7 · p. 13
  • 89.6 BFCL irrelevance detection matches the base model’s 88.8, and SFTnear-miss raises the BFCL average to 70.0 versus the base’s 68.7.On commerce, its success falls by 8.8 points as unconfirmed destructive calls rise from 24 to 64; refunds of delivered orders fall from 86% to 5%, and purchases after confirmation from 32% to 7%.Source: §5.7 · pp. 13–14

5.8 Other backbones

  • SFTclean raises in-distribution success to 99% for Qwen3-1.7B and Falcon3-3B and 69% for Phi-4-mini, with hard-split gains of 54.7, 32.0, and 52.1 points, respectively.Commerce gains are 41.0, 25.8, and 27.2 points, but success remains 35%–54%, below Qwen3-4B’s 83%.Source: §5.8 · p. 14
  • On BFCL, Phi-4-mini reaches 74.5 non-live and 63.8 live accuracy, Falcon3-3B gains 26.5 non-live and 12.7 live points, while Qwen3-1.7B loses live accuracy from 74.5 to 67.1.Phi-4-mini’s gains include better call formatting and structure; its irrelevance detection falls from 87.5 under lenient decoding to 73.8.Source: §5.8 · p. 14
  • SFTdecline reduces commerce effect-safety violations to 0.026, 0.009, and 0.000 per task for Qwen3-1.7B, Phi-4-mini, and Falcon3-3B, respectively, while calling less.Falcon3-3B falls from 6.20 to 0.01 calls per commerce task and from 35.4% to 17.4% commerce success.Source: §5.8 · p. 14

6 Analysis

The analysis traces BFCL declines and gains to training-data cues: refusal templates shift models away from calling, while near-miss examples teach the topic-match/function-mismatch cue needed for irrelevance detection. BFCL gains concentrate in behaviors represented in EffectWorld, but BFCL does not assess the effect-safety properties the paper targets.

6.1 Public data teaches refusal

  • SFTraw answers 374 of 400 BFCL simple-Python prompts without a call, although the base calls on all 400; 255 responses claim the model lacks the tool it was given.About 780 sampled Nemotron trajectories contain refusal phrases, and about 16% of raw-mix examples begin without a call; in-domain tasks did not reveal the refusal pattern.Source: §6.1 · pp. 14–15
  • SFTclean removes public refusal signals and restores calls on simple prompts, but it answers 33 more of 240 irrelevance prompts with a call than the base model.Refusal negatives push further toward calling, while missing-tool decline positives do not restore BFCL irrelevance detection.Source: §6.1 · p. 15

6.2 When fine-tuning beats the base model on BFCL, and when it does not

  • Only SFTnear-miss beats the base on BFCL average; SFTclean answers 157 prompts correctly that the base misses, but answers 272 incorrectly that the base gets right.Half of those 272 errors are irrelevance prompts where SFTclean calls instead of declining.Source: §6.2 · p. 15
  • SFTclean solves 35.5% of BFCL multi-turn conversations versus 26.0% for the base, and scores 22.5% versus 15.5% on turns withholding a required parameter.It also reaches 82.6% versus 78.3% on live simple calls; these behaviors correspond to carrying state, asking for missing information, and calling when one function fits.Source: §6.2 · p. 15
  • All 14,999 public trajectories in the clean mix contain a call, and none of the training no-call settings offers a plausible but functionally mismatched tool.SFTnear-miss supplies that cue and raises irrelevance detection from 75.0 to 89.6, level with the base, although the same contrast costs confirmation on unseen tools.Source: §6.2 · p. 16
  • BFCL scores call form, not whether a call needed confirmation, repeated an earlier call, or overwrote unread state, so it cannot measure the targeted effect properties.Other remaining call errors include parallel-multiple prompts, argument types absent from JSON-only training, and canonical-value mismatches.Source: §6.2 · p. 16

7 Limitations

The paper’s evaluation and conclusions are bounded by a small, templated task world, limited user simulation, and narrow model and seed coverage.

7 Limitations

  • EffectWorld has nine task kinds, one training domain, one test-only domain, and template utterances without generated paraphrases.Its conclusions therefore concern a limited set of task and domain settings.Source: §7 · p. 16
  • The scripted user never changes its mind and confirms every recognized proposal, including one targeting the wrong entity.The verifier treats any proposal the scripted user confirms as confirmed, which can mis-score a later substitute after an entity is rejected.Source: §7 · p. 16
  • BFCL multi-turn and τ 2-bench are not evaluated end to end, and detailed analyses focus on Qwen3-4B-Instruct-2507.Other model families are evaluated on the same benchmarks but analyzed less deeply; all models have at most 4B parameters.Source: §7 · p. 16
  • SFT models use one training seed, and the reported intervals measure task sampling only.NPO is unstable across seeds, while the second-seed unlikelihood conclusions hold against that seed’s control.Source: §7 · p. 16

8 Conclusion

EffectWorld makes tool-call safety and efficiency checkable, and its training improves task success across tested splits and model families. The findings also show that public refusal templates and cue-mismatched examples can limit transfer, while the study leaves broader settings for future work.

8 Conclusion

  • 53% to 99% in-distribution, 52% to 81% on unseen task kinds, and 74% to 83% on an unseen shopping domain, with about a third of output tokens, for Qwen3-4B SFT.Across Qwen3, Phi-4, and Falcon3 models from 1.7B to 4B parameters, SFT raises in-distribution success to 69%–99% and adds 26–55 points on held-out splits.Source: §8 · p. 16
  • Refusal templates in public tool-use data cut BFCL non-live accuracy from 88.0 to 31.4 despite perfect in-domain metrics.The authors report that only an external benchmark exposed this effect.Source: §8 · p. 16
  • Neither suppression nor DPO on verified negatives beats SFT on held-out splits, while targeted positive examples transfer along the cue they demonstrate.Near-miss positives restore BFCL irrelevance detection to the base model’s level and lift the BFCL average above it, but can reduce confirmation on unseen tools.Source: §8 · p. 17
  • On the shopping domain, base models make 1 to 7 unconfirmed destructive calls, while their SFT adapters make 6 to 64, showing that confirmation did not transfer to unseen tools.The authors report that efficiency transfers to new tools, while effect discipline remains tied to the tools on which it was learned.Source: §8 · p. 17
  • The authors recommend declaring tool effects, auditing public data for refusal templates, and using positive examples for decisions such as declining, asking, and confirming.They also identify untested settings including longer cross-domain tasks, larger adapters or full fine-tuning, and models above 4B parameters.Source: §8 · p. 17

A Task kinds and negative classes

The appendix identifies workspace training task kinds by their generator names and describes the negative-class groupings used in the training pairs.

A Task kinds and negative classes

  • Table 8 lists workspace training task kinds by generator name and identifies “irrelevant” as the main text’s no-tool kind.The passage supplies no individual task-kind values beyond that mapping.Source: Table 8 · p. 20
  • The negative-class table groups pair-production methods with the BFCL categories they are expected to affect.Its pairs cover 17 world classes and refusal.Source: Table 9 · p. 20
  • Serialized_calls produces no training pairs because every turn its scheduling pass merges is already attributed to another optimizer class.Source: Table 9 · p. 20

B Detailed results

The detailed-results appendix points readers to full metrics for in-distribution and commerce evaluations and defines the measures reported in the accompanying tables.

B Detailed results

  • Table 10 covers 498 in-distribution test tasks and reports per-task efficiency, call, token, safety-violation, and collateral-change measures.It distinguishes program mode, which allows references, from stepwise mode.Source: Table 10 · p. 21, B Detailed results · p. 21
  • Table 10 defines excess calls as calls beyond the optimal plan and violations per task as unconfirmed destructive, duplicate-write, or stale-state actions.The caption also defines collateral as the share of tasks changing an entity outside the goal.Source: Table 10 · p. 21
  • Table 11 covers 547 tasks and 17 unseen tools in online shopping, reporting success in program and stepwise modes and other measures in program mode.Its unconfirmed-call measure concerns destructive pay, cancel, or refund calls issued without a confirmed proposal.Source: Table 11 · p. 21, B Detailed results · p. 21

C Example traces

The flowcharts use color and box shape to distinguish user turns, assistant calls, tool results, assistant text, and rejected or failing steps.

C Example traces

  • Grey boxes mark user turns, blue boxes assistant tool calls, green boxes tool results, and violet boxes assistant text.Source: C Example traces · p. 21
  • A blue box with several calls represents one program-mode turn, and $1.x refers to the first call’s result in that turn.Source: C Example traces · p. 21
  • Orange boxes mark a rejected training-pair turn or the step where a held-out task fails, with the negative class labelled.Source: C Example traces · p. 21

D Paired differences

Paired differences are reported with bootstrap intervals, alongside examples of trace optimization, policy and efficiency pairs, correctness pairs, and held-out adapter behavior.

D Paired differences

  • 95% paired bootstrap intervals use 2,000 resamples, with significance defined by an interval that excludes zero.Tasks are paired by task and mode, or BFCL prompts; Tables 12 and 13 list the differences discussed in the text.Source: D Paired differences · p. 21, Table 12 · p. 24
  • Figure 3 shows the optimizer removing two reads and combining a write with a scheduled e-mail after the confirmation barrier.Each removed call becomes a redundant_read pair at its prefix.Source: Figure 3 · p. 22
  • Figure 4 depicts policy and efficiency pairs at a shared prefix, distinguishing the chosen turn from a rejected turn.Solid arrows mark the prefix and chosen turn; dashed arrows mark the rejected turn at that prefix.Source: Figure 4 · p. 22
  • Figure 5 presents correctness pairs from rule perturbations and a refusal class built on filtered public prefixes.Source: Figure 5 · p. 23
  • Figure 6 contrasts held-out behaviors, including a contact-selection failure, an unsafe substitute proposal, a confirmed unseen-domain purchase, and repeated orders.The scripted user confirms every proposal; repeated orders follow dissatisfaction with the answer wording.Source: Figure 6 · p. 23

E Reproducibility

The project lists its code, data generators, configurations, and cluster jobs, while Table 13 continues the paired-difference results for data recipes and other backbones.

E Reproducibility

  • The project’s RESULTS.md lists code, data generators, configurations, and every cluster job behind each number.Source: E Reproducibility · p. 24
  • Table 13 continues the paired differences for data recipes and other backbones.Source: Table 13 · p. 25
  • Table 13 presents continued paired-difference results for data recipes and other backbones, and explicitly states that it uses the same conventions as Table 12.Source: Table 13 · p. 25

Related papers

Loading 2610.35981v1…