PaperlayerView exact revision on arXiv

Source-linked AI summary

A Matter of TASTE: Improving Coverage and Difficulty of Agent Benchmarks

Tomer Keren, Nitay Calderon, Asaf Yehudai, Yotam Perlitz, Michal Shmueli-Scheuer, Roi Reichart

arXiv:2605.28556v2cs.AI

TL;DR

Tool-agent benchmarks are costly to construct, and coverage of structurally diverse tool-use patterns has received comparatively little attention. TASTE builds tasks around sampled tool sequences; across 11 agent–user pairs, performance on τ c-Bench was consistently lower than on τBV, with relative drops of −5% to −80%.

  • Problem

    Tool-agent benchmarks are costly to construct, while coverage of structurally diverse tool-use patterns has received comparatively little attention.

    Source: §1 · p. 2

  • Method

    TASTE reverses task construction by sampling diverse tool sequences and synthesizing complete benchmark tasks around them.

    Source: §1 · p. 2

  • Results

    Relative drops of −5% to −80% were observed across 11 agent–user pairs and three domains on τ c-Bench compared with τBV.

    Source: §5 · p. 7

  • Takeaways & Limitations

    The authors conclude that high τBV scores may reflect benchmark saturation rather than robust task-solving ability.

    Source: §5 · p. 7

    The evaluation uses final-state reward alone, so it does not assess trajectory-level efficiency, intermediate rewards, or interaction quality.

    Source: §2 · p. 3

Abstract

from arXiv · show

As agent capabilities advance, existing benchmarks, such as $τ^2$-Bench, are becoming increasingly saturated. Yet constructing new benchmark tasks remains complex, costly, and labor-intensive. Moreover, the standard approach, in which scenarios are first written in natural language and then mapped to tool sequences, captures only a narrow subset of the tool-use patterns agents exercise. In this paper, we address these problems by reversing the task construction process. We propose TASTE: Task Synthesis from Tool Sequence Evolution, an automatic method that generates challenging tasks with broader tool-use coverage. TASTE utilizes an Adaptive Contrastive $n$-gram model trained on LLM-judged validity signals. This enables sampling valid tool sequences that cover a vast range of tool combinations. TASTE then selects representative sequences from the pool via clustering, instantiates them into complete benchmark tasks, and refines them through iterative difficulty evolution. Using TASTE, we construct $τ^c$-Bench, a challenging extension of the three domains of $τ^2$-Bench. We evaluate $11$ agent/user LLM pairs and find that models nearly saturating $τ^2$-Bench suffer severe performance drops on our tasks (e.g., Gemini-3-Flash falls from $0.82\!-\!0.94$ to $0.28\!-\!0.61$). Beyond increasing difficulty, our generated tasks more than double the number of unique tool combinations agents must execute. Our results suggest high scores on existing benchmarks often reflect saturation rather than robust task-solving ability. By automating the generation of difficult, high-coverage benchmarks, TASTE enables continuous, scalable evaluation of future agents.

1 Introduction

TASTE reverses benchmark construction by sampling diverse tool sequences and synthesizing tasks around them, targeting validity, difficulty, and coverage. The resulting τ c-Bench is harder than τ 2-Bench and covers more tool-use patterns.

1 Introduction

  • TASTE operationalizes benchmark validity, difficulty, and coverage, defining coverage through diversity in ordered tool sequences leading to target states.Tasks are valid when automatically verifiable and their gold final state is reachable; difficulty should separate capability levels, while coverage spans structurally diverse tool-use patterns.Source: §1 · pp. 2–3
  • TASTE samples tool sequences with an Adaptive Contrastive n-gram model whose online validity judgments refine probabilities to explore valid tool combinations.An LLM judges sequences as plausibly or implausibly valid, and the model contrasts separate n-gram count tables.Source: §1 · pp. 2–3
  • TASTE clusters sampled sequences with weighted Levenshtein distance, instantiates selected sequences as tasks, verifies solvability, then evolves difficulty through ambiguity and decoy records.K-medoids selects K representative sequences; a hint-assisted verifier agent cooperates with a simulated user to reach the target state.Source: §1 · p. 3
  • Across 11 agent/user LLM pairs, τ c-Bench scores are substantially lower than τ 2-Bench, with Gemini-3-Flash falling from 0.82−0.94 to 0.28−0.61.The benchmark more than doubles unique tool combinations; weighted edit distance rises by up to 124%, TTR by up to 111%, and tool-frequency entropy by 35%.Source: Figure 1 · p. 2, §1 · p. 3

2 Evaluating Conversational Tool-Using Agents

The section defines conversational tool-agent evaluation through interactions between an evaluated agent and a simulated user, with task success determined by the final world state. It then characterizes benchmark quality through validity, coverage of tool-use structures, and difficulty that differentiates agent capabilities.

2 Evaluating Conversational Tool-Using Agents

  • Conversational tool-agent evaluation follows trajectories jointly shaped by the evaluated agent, simulated user, and environment, and judges success by the resulting world state [7].Many different trajectories can be correct if they reach the predefined target state induced by a gold tool-call sequence.Source: §2 · p. 3
  • The benchmark defines success as a binary final-state match to the target induced by a gold tool-call sequence, while distinguishing tool calls with arguments from their ordered tool-name sequence.The study adopts final-state reward alone for simplicity and consistency with standard τ-Bench reporting, although this omits trajectory-level properties such as efficiency, intermediate rewards, and interaction quality.Source: §2 · pp. 3–4

2.1 Desiderata for Agent Benchmarks

  • A task is valid when its success is automatically verifiable and its gold final state is correct and achievable under the task specification.Invalid tasks can penalize agents for benchmark defects rather than capability limitations.Source: §2.1 · p. 4
  • Coverage measures whether tasks represent structurally different, common gold tool sequences rather than near-duplicates, arbitrary combinations, or isolated outliers.It can be quantified using tool-edit distance or the number of unique tool n-grams, abstracting away from linguistic variation in instructions and initial states.Source: §2.1 · p. 4
  • Difficulty should meaningfully differentiate capability levels, balancing structural demands of tool sequences against interaction demands from instructions, initial states, and user behavior.Structural factors include sequence length and tool types; interaction factors include ambiguity, distractors, dependencies, stale information, and cooperative or adversarial users.Source: §2.1 · p. 4

3 TASTE: Task Synthesis from Tool Sequence Evolution

TASTE automatically constructs benchmark tasks from tool sequences through sampling, representative-sequence selection, and task generation with difficulty evolution. Its stages target validity, coverage, and difficulty.

3.1 Tool Sequence Sampling with an Adaptive Contrastive n-gram Model

  • The Adaptive Contrastive n-gram sampler learns from LLM plausibility labels, combining positive and negative n-gram evidence to generate diverse candidate tool sequences under desired constraints.Online updates refine sampling as sequences are generated and validated; negative evidence penalizes patterns associated with implausible sequences.Source: §3 · p. 4, §3.1 · p. 5

3.2 Clustering and Selection of Representative Tool Sequences

  • From N=2000 candidates, TASTE selects K representative sequences with K-medoids and semantically weighted Levenshtein distance, producing clusters with similar functional structure.Invalid medoids are replaced by the next-closest cluster member; if a cluster has no valid members, clustering is rerun with valid medoids fixed.Source: §3.2 · pp. 5–6

3.3 Generating Tasks from Tool Sequence and Evolving the Difficulty

  • Each selected sequence is instantiated as a scenario, user instruction, and database state, then checked for structural validity and solvability by a hint-assisted verifier.The verifier receives shuffled gold tool calls with some arguments removed, so it must infer the plan and recover missing information rather than replay the solution.Source: §3.3 · p. 6
  • Three LLM calls evolve base tasks through adversarial strategy analysis, decoy-record insertion, and scenario rewriting while preserving the gold tool-call sequence.Evolved tasks are revalidated; failed variants are simplified before the pipeline falls back to the base task.Source: §3 · p. 5, §3.3 · p. 6

4 Experimental Setup

The experiments extend three τ²-Bench domains using corrected task sets, apply TASTE with specified generation models and settings, and evaluate agent–user pairs under the benchmark harness. The section also reports generation and evaluation costs.

Datasets

  • The evaluation covers Airline, Retail, and Telecom (N = 50/114/114) using τ-Bench-Verified corrected task sets; Telecom gold sequences are restricted to write-type tools.The restriction reflects that Telecom gold sequences encode only write actions.Source: §4 · p. 7

TASTE and τc-Bench

  • TASTE trains a trigram model for 3,000 iterations and samples sequences up to length 15, using Gemini-3-Flash for validation and instantiation and Gemini-3-Pro for task evolution.Sampled sequence lengths follow a skew-normal distribution with µ=7, σ=5, α=2; Gemini-3-Flash also serves as the hint-assisted verifier agent.Source: §4 · p. 7

Evaluated agents

  • Six agent LLMs are paired with Gemini-3-Flash and GPT-5.2 user simulators; results report pass^1 with final-state reward and additionally pass^3 for Airline.Evaluation uses the τ²-Bench harness with default settings and vendor-default reasoning effort settings.Source: §4 · p. 7

Costs

  • Stages 1 and 2 cost approximately $10 per domain, Stage 3 approximately $2.50 per task, τc-Bench generation $725 total, and evaluation $520 total.Evaluation costs vary by agent–user pair.Source: §4 · p. 7

5 Results

TASTE’s generated tasks are harder and cover more varied tool sequences than τBV, and its results justify design choices for sequence sampling, verification, and difficulty evolution.

5 Results

  • Across 11 agent–user pairs, performance on τ c-Bench was 5%–80% lower than τBV; Gemini-3-Flash fell from 0.82–0.94 to 0.28–0.61 across domains, with the stated Airline-user exception.The authors interpret the high τBV scores as potentially reflecting benchmark saturation rather than robust task-solving ability.Source: §5 · p. 7
  • Across all three domains, τ c-Bench increased WED by up to 124%, TTR by up to 111%, and tool-frequency entropy by 35% versus τBV.WED captures structural dissimilarity between tool sequences, while TTR measures the fraction of unique n-grams; the reported gains also hold for sequences from successful simulations.Source: Figure 2 · p. 8, §5 · p. 8

5.1 Method Analysis

  • The full adaptive contrastive n-gram model achieved 86.7% validity versus 6.7% with uniform tool sampling, an approximately 13× improvement.Without adaptive training, even initialization from τBV n-grams reached only 16.7%; negative evidence improved validity by 10 points at k = 400 and 20 points at k = 800, while λneg added 10 points for the final model at k = 3000.Source: §5.1 · p. 8, Figure 3 · p. 9
  • Evolved tasks reduced success by 36–55% with Gemini-3 and 16–37% with GPT-5.2, leading TASTE to select Gemini-3 for evolution.TASTE can also control difficulty through sequence length and read-to-write ratio: longer tasks had 30% lower performance on average and write-heavy tasks 43% lower across agent–user pairs and domains.Source: Figure 3 · p. 9, §5.1 · p. 9
  • The verifier achieved precision of 1.0 and 0.97 and recall of 0.75 and 0.83 in Airline and Retail, respectively; all 15 tasks with no successful evaluated pair were manually judged valid.Its validation set compared original τ2-Bench tasks with τ-Bench-Verified entries: 50 Airline tasks (8 invalid) and 86 Retail tasks (9 invalid), excluding tasks with minor fixes.Source: §5.1 · pp. 8–9

6 Related Work

The paper situates TASTE among tool-use benchmarks and synthetic task-generation methods, distinguishing its goal of exploring tool-sequence coverage for benchmark construction. It defines coverage by the diversity of gold tool sequences rather than available tools or schemas.

6 Related Work

  • Existing benchmarks span single-turn API use, persistent multi-turn environments, and conversational agent–simulated-user tasks, but rely heavily on costly, error-prone manual authoring [12] [22] [23] [24] [14] [25] [10] [11] [26].The paper focuses on conversational benchmarks that evaluate the resulting world state after interaction.Source: §6 · p. 9
  • ToolGrad [38] and Trajectory2Task synthesize tasks from trajectories, whereas TASTE starts from tool sequences to explore tool combinations for benchmarks rather than training data.ToolGrad incrementally proposes, executes, and selects API calls before instruction instantiation; Trajectory2Task replaces calls and evolves instructions toward challenging intents.Source: §6 · p. 9
  • Synthetic task generation reduces annotation cost, ranging from instruction bootstrapping [27–29] and tool-call generation [1] [12] [30] [31] to multi-turn pipelines using blueprints, simulated environments, role-playing, and trajectory feedback [32] [33] [34] [35] [36] [37].The tool-call methods use insertion filtering, tool descriptions, or post-hoc verification.Source: §6 · p. 9
  • Coverage is measured over gold tool sequences using diversity measures, capturing procedural patterns that domain, tool, schema counts, and capability tags do not [12] [24] [25].Sequence-level coverage indicates whether a benchmark tests a broad range of agent behaviors or repeatedly tests similar patterns.Source: §6 · pp. 9–10

7 Conclusions

TASTE automatically generates difficult agent-benchmark tasks with broad tool-use coverage; its τ c-Bench extension of τ 2-Bench exposes performance drops for models that nearly saturate the original benchmark.

7 Conclusions

  • TASTE generated τ c-Bench, a challenging extension of τ 2-Bench, where models nearly saturating the original benchmark suffer performance drops.The method generates tasks designed to be difficult and cover a broad range of tool-use patterns.Source: §7 · p. 10
  • Although focused on conversational tool-using agents, TASTE operates on tool sequences and environment specifications and can adapt to single-turn and non-conversational settings.Source: §7 · p. 10
  • Future work could use TASTE to generate training data and extend coverage and difficulty definitions to additional axes.Source: §7 · p. 10

A Additional Results

Additional analyses show that golden-sequence length and write intensity track task accuracy, τc-Bench spreads tool usage more evenly, and weighted edit distance yields more semantically coherent clusters than regular edit distance.

Coverage Desiderata

  • Table 2 compares gold and GoldSim sequences using length, write/read balance, cluster diversity, n-gram entropy and richness, and type-token ratio.GoldSim selects the shortest successful agent sequence per task from pooled runs, falling back to the gold sequence when none succeeds, so each cell retains one sequence per task.Source: Table 2 · p. 15, A Additional Results · p. 15

Difficulty Desiderata

  • 13–20 pp accuracy gaps separate short and long tasks across all three domains; write-heavy tasks show 16–31 pp gaps over read-heavy tasks in airline and retail.Telecom is excluded from the write-ratio comparison because every task’s evaluation criteria encode only write actions, making its per-task write-ratio 1.0.Source: Figure 4 · p. 16, A Additional Results · p. 16

Tool-Usage Distributions Across Domains

  • τc-Bench spreads tool usage more evenly across the tool vocabulary than τBV, raising normalized entropy across the three domains.The telecom comparison is restricted to write actions (21 of 43 tools), with both distributions renormalized over that subset; the airline counterpart appears in Figure 2.Source: Figure 5 · p. 16, A Additional Results · p. 16

Qualitative Comparison: Weighted vs. Regular Edit Distance

  • Weighted edit distance (WED) produces semantically coherent airline clusters, while regular edit distance (ED) yields nearest neighbors that drift from medoid intent.WED inter-medoid distances (3.66–4.66) exceed the visible intra-cluster neighbor distances; under ED, inter- and intra-cluster scales are closer.Source: A Additional Results · p. 17

B Implementation Details · B.1 Stage 1: Tool Sequence Sampling with an Adaptive Contrastive n-gram Model

TASTE’s implementation runs three artifact-linked stages using domain-specific environment resources and original tasks; Stage 1 trains an adaptive trigram sampler through plausibility judgments over candidate tool sequences.

B Implementation Details

  • The three stages run as separate processes linked by artifacts; domain-specific inputs are the environment definition and original tasks, and LLM calls default to Gemini-3-flash.The environment definition comprises the policy document, tool specification, and database schema; model-choice ablations are reported in Section 5.Source: B Implementation Details · p. 17

B.1 Stage 1: Tool Sequence Sampling with an Adaptive Contrastive n-gram Model

  • Stage 1 initializes a trigram sampler with λ0 = 0.1, λneg = 1, empty positive and negative count tables, and tool-identifier tokens plus reserved BOS and EOS.BOS and EOS are used for padding and termination but are never emitted by the sampler.Source: §B.1 · p. 17
  • The sampler’s temperature follows exponential decay from T0 = 3.0 with τdecay = 1500, while its iteration counter advances once per generation attempt.The decay schedule is instantiated with those parameters.Source: §B.1 · p. 17
  • Candidate sequence lengths come from a skew-normal distribution with location 7, scale 5, and skewness 2.0, then are clipped to [1, 15] and rounded.Lengths are sampled as integers after hard clipping.Source: §B.1 · p. 17
  • Generation prepends two BOS tokens and samples autoregressively until the chosen length; an LLM validator judges plausibility and identifies problematic indices for invalid candidates.The validator uses a domain-specific template filled with the candidate sequence and environment.Source: §B.1 · p. 17
  • Plausible sequences increment every extracted BOS-padded n-gram in C+, while implausible sequences increment only windows ending at problematic indices in C−; tuple-equality deduplication skips duplicates.Skipped duplicates update neither table, but the generation-attempt counter still advances.Source: §B.1 · p. 17
  • Before the 3,000-attempt loop, original-task gold tool sequences are validator-judged and ingested; Stage 1 outputs the trained sampler and final count tables.Seed sequences are ingested according to the validator’s verdict.Source: §B.1 · p. 18

B.2 Stage 2: Clustering and Selection

Stage 2 samples tool sequences, clusters them with a tool-aware distance, validates cluster medoids, and emits the plausible medoid sequences.

Pool generation

  • The sampler draws 2,000 unique tool sequences at temperature 1.5, using the same skew-normal length distribution as Stage 1.Source: §B.2 · p. 18

Weighted edit distance

  • Levenshtein distance assigns insertion and deletion cost 1, and substitution costs 0, 0.33, 0.66, or 1 for exact matches, shared type and group, shared type only, or different types.Tool types are READ, WRITE, or GENERIC, with GENERIC treated as READ for type comparisons; a tool group is its name prefix before the first underscore.Source: §B.2 · p. 18
  • Within-cluster distances were 1.0–3.98 and the smallest between-medoid distance was 5.31, with clusters aligning to distinct intents while absorbing cosmetic variants.The illustration uses three airline-domain clusters for booking, modifying, and cancelling reservations.Source: §B.2 · p. 18

K-medoids

  • K equals the number of seed tasks; k-medoids++ initialization and PAM-style assignment and updates continue until medoids stabilize or 100 iterations are reached.K is 50 for airline and 114 for retail and telecom.Source: §B.2 · p. 18

Medoid validation and replacement

  • Each medoid is plausibility-validated; an implausible medoid is replaced by the nearest plausible member of its cluster, and clusters without one are marked unusable.Source: §B.2 · p. 18

Re-clustering of unusable clusters

  • Re-clustering freezes previously accepted medoids, draws a fresh pool, and selects free medoids for unusable clusters; remaining unusable clusters are dropped after three rounds.Frozen centers remain in nearest-medoid assignment but are excluded from medoid updates; benchmark size is reduced when clusters are dropped.Source: §B.2 · p. 18

Output

  • For each plausible cluster, Stage 2 outputs its medoid sequence.Source: §B.2 · p. 18

B.3 Stage 3: Task Generation and Evolution

Stage 3 turns each plausible medoid into a validated base task, then attempts to evolve it into a harder adversarial variant. The stage emits one task per cluster in the τ^2-Bench schema, retaining the base task when evolution cannot be validated.

B.3 Stage 3: Task Generation and Evolution

  • Each Stage 2 medoid is instantiated as a base task through scenario construction and database initialization, then validated before adversarial evolution.One LLM call writes a coherent real-world scenario and verbose user instruction; a second generates required database entities merged into the initial state.Source: §B.3 · p. 19
  • The validity pipeline combines rule-based, LLM, and simulation checks, while a modified hinted verifier tests solvability without making tool execution a transcription exercise.It checks specified arguments, entity existence, schema conformity, gold-sequence success, and policy coherence, short-circuiting on failure; the verifier shuffles gold calls and drops 30% of their arguments, and tasks must earn reward 1.0.Source: §B.3 · p. 19
  • Failures trigger feedback-driven repair or retries; if adversarial evolution still fails, a lite rewrite is tried before retaining the unchanged base task.Unrepairable base tasks are regenerated, with repeated failures prompting replacement by another cluster candidate; the graduated fallback aims to enhance as many tasks as possible while keeping the final set complete.Source: §B.3 · pp. 19–20
  • Three evolution steps analyze strategies and decoys, add corresponding database records, and rewrite the instruction while preserving the gold sequence and arguments.The evolved task is validated with rule-based checks skipped because the gold sequence, its arguments, and the non-decoy environment are preserved by construction.Source: §B.3 · pp. 19–20
  • Evolution selects adversarial patterns for each write action from three categories: database-grounded misdirection, policy enforcement, and conversational adversity.Examples include plausible but wrong database entities, pressure to perform policy-forbidden actions, and withholding information.Source: §B.3 · p. 20
  • The stage emits one task per cluster in the τ^2-Bench task schema, in evolved form when evolution succeeds.If both full and lite evolution fail validation, the original base task is retained.Source: §B.3 · p. 20

B.4 Consolidated Hyperparameter Table · B.5 Pseudo-codes · C Examples

These appendices report run hyperparameters, map model names to API identifiers, specify the TASTE procedures in pseudocode, and show generated task examples across three domains.

B.4 Consolidated Hyperparameter Table

  • Stage 2 uses a 2,000-sequence pool at temperature 1.5 and up to 100 K-medoids iterations, while Stage 3 specifies redaction p=0.3, two simulation attempts, and three Lite retries.Hyperparameter values are constant across domains unless noted.Source: §B.4 · p. 20, Table 3 · p. 20
  • Table 4 maps paper model names to exact API identifiers and reports average USD cost per task for each model.Source: Table 4 · p. 21

B.5 Pseudo-codes

  • Algorithm 1 initializes positive and negative n-gram sets from validator judgments, then repeatedly samples unseen sequences and adds their n-grams according to plausibility.Source: §B.5 · p. 21
  • Algorithm 2 samples a sequence pool, clusters it into K medoids, and validates members in distance order until it finds plausible medoids or reaches K.Source: §B.5 · p. 21
  • Algorithm 3 builds a base task for each validated medoid by generating a scenario, initializing a database, and checking the resulting task; failed medoids are marked for re-clustering.Source: §B.5 · p. 22
  • For each valid base task, Algorithm 3 tries full evolution, then Lite evolution if needed, and keeps the base task if both evolved versions fail validation.Source: §B.5 · p. 22

C Examples

  • Figure 6 shows one end-to-end task from each of the three τ²-Bench domains, pairing the gold tool-call sequence with base and evolved variants for comparison across Known Info and Task Instructions.The task purpose appears above the gold sequence and is shared by both variants.Source: C Examples · p. 22

Airline task#1 … 2. Limitations

The section presents supporting materials for TASTE, including an airline tool-sequence cluster, prompts and checklists for validating sequences and tasks, constructing scenarios, and initializing databases, and identifying the paper’s desiderata, pipeline, three-domain instantiation, and where limitations are discussed.

Airline task#1

  • Cluster A’s airline medoid is a six-action sequence from user lookup and flight search through booking, flight updates, and passenger updates.The cluster is titled “Search, Book, and Update.”Source: Airline task#1 · p. 26

D Prompts

  • The domain-agnostic validation prompt checks sequence structure, logical flow, and benchmark viability, while allowing uncommon plausible and multi-intent sequences.Its checks include read-before-write and causal ordering; the same prompt template is used across airline, retail, and telecom, with domain context supplied through placeholders.Source: D Prompts · pp. 26–27
  • Task-generation instructions require forced action ordering and a solvable, policy-compliant scenario with consistent IDs, arguments, and user intent.They also specify message boundaries and require escalation to be justified when the sequence includes a handoff.Source: D Prompts · p. 28
  • The database prompt requires schema-conformant, referentially consistent initial data that satisfies action preconditions and reconciles policy-dependent totals.It also specifies exact enums and numeric types, excludes runtime-created entities from initialization, and requires dual-database consistency where applicable.Source: D Prompts · pp. 29–30
  • The reviewer prompt treats only defects that make a perfect score impossible as critical, distinguishing them from reorderable actions and missing but inferable information.Critical defects include forbidden actions, impossible database preconditions, scenario-action contradictions, missing prerequisites, unverified writes when policy requires a read, and actions after terminal handoff.Source: D Prompts · p. 30
  • The review prompt checks status preconditions, dependency flow, database and device-state impossibilities, scenario contradictions, and operations disallowed by policy.Examples include an ineligible initial status, a missing issue step before payment, nonexistent entities, mismatched device state, once-per-entity violations, and hard-cap breaches.Source: D Prompts · p. 31
  • The adversarial-strategy prompt targets every golden-sequence write with tactics such as eligibility pressure, argument misdirection, factual lies, decoys, withholding, escalation, or forbidden extra writes.It requires at least one global technique and two real conditional branches, while keeping golden arguments fixed and avoiding nonexistent or reserved runtime entities.Source: D Prompts · pp. 31–33
  • The adversarial-scenario prompt requires every planned trap to remain solvable by a policy-compliant agent, with certain resolutions, exact facts, and real adversarial branches.It also requires the user to accept corrected values explicitly and wait for confirmation that requests are complete.Source: D Prompts · pp. 33–34

1. Claims

  • The claims checklist says the paper formalizes benchmark desiderata, proposes the tool-sequence-first TASTE pipeline, and instantiates it across three τ-Bench domains.It locates these contributions in Sections 2.1 and 3, with quantitative criteria results in Section 5 and Table 1.Source: §1 · p. 34

2. Limitations

  • The limitations checklist reports that limitations are discussed in the Method and Results sections, Sections 3 and 5.Source: §2 · p. 34

3. Theory assumptions and proofs … 6. Experimental setting/details

The paper reports no theoretical results, but describes reproducibility materials and experimental settings across the main text and appendices. It says anonymized code and task sets for all three domains are provided with reproduction instructions.

3. Theory assumptions and proofs

  • The paper includes no theoretical results, theorems, lemmas, or formal proofs; it validates the Adaptive Contrastive n-gram model empirically.Source: §3 · p. 35

4. Experimental result reproducibility

  • Section 3 specifies the full pipeline, while Appendix B gives implementation details and hyperparameters and Appendix D provides the prompts used at every stage.Source: §4 · p. 35
  • Section 4 reports the evaluated agent/user model pairs, τ-Bench harness configuration, and per-domain task counts.Source: §4 · p. 35

5. Open access to data and code

  • The authors provide the pipeline implementation and generated τ⋆-Bench task sets for all three domains as anonymized supplementary material.Source: §5 · p. 36
  • The supplementary materials include instructions for reproducing the main experimental results, alongside full hyperparameter settings and the exact prompts used at each pipeline stage.Source: §5 · p. 36, §6 · p. 37

6. Experimental setting/details

  • The experimental details include per-domain task counts of 50/114/114, costs, and the reward signal used in each domain.Source: §6 · p. 37

7. Experiment statistical significance … 16. Declaration of LLM usage

The paper reports limits on statistical uncertainty and compute reproducibility, and documents its ethics, impacts, released assets, human-subjects status, and LLM use. It identifies LLMs as core components of TASTE and says prompts are provided in Appendix D.

7. Experiment statistical significance

  • Table 1 reports pass^1 and airline pass^3 point estimates without explicit error bars or confidence intervals; no additional seeds were run because of API costs.The pass^k metric requires success on all k independent trials and captures variability across trials for each agent/user pair.Source: §7 · p. 37

8. Experiments compute resources

  • Experiments use external LLM APIs rather than local GPU training, so API cost is the relevant resource reported for reproduction.Section 4 provides benchmark-creation costs per domain and generator model, evaluation costs per task for each evaluated model, totals for the 50/114/114-task sets, and notes additional ablation runs.Source: §8 · p. 38

9. Code of ethics; 11. Safeguards

  • The authors say the work conforms to the NeurIPS Code of Ethics and that no safeguards are needed because only synthetic benchmark tasks are released.They report no human subjects, personally identifiable or scraped data, or prohibited application areas; released assets do not include pretrained models or other high-misuse-risk materials.Source: §9 · p. 38, §11 · p. 39

10. Broader impacts

  • The authors mark broader impacts N/A, describing the work as evaluation methodology and a benchmark rather than a deployed model or capability-increasing system.They cite more reliable and trustworthy tool-using agents as a general positive impact and characterize negative downstream uses as indirect and generic.Source: §10 · p. 38

12. Licenses for existing assets

  • Existing assets are credited and used under their stated terms: τ-Bench, τ²-Bench, and τ-Bench Verified are under MIT License, while evaluated LLMs use providers’ public APIs.The authors state that assets are cited at first use and that API access follows each provider’s terms of service.Source: §12 · p. 39

13. New assets

  • The generated τ⋆-Bench task sets and pipeline implementation are released as anonymized supplementary material with documentation.Source: §13 · p. 40

14. Crowdsourcing and research with human subjects

  • The paper reports no crowdsourcing or human-subjects research, so participant instructions, compensation, and IRB approval are not applicable.Task generation and validation use LLMs and rule-based checks; the only manual step was internal author inspection of 15 tasks on which no evaluated agent-user pair succeeded.Source: §14 · p. 40

16. Declaration of LLM usage

  • LLMs serve as TASTE’s plausibility judges, generators, verifier, and adversarial-evolution modules; the specific models and prompts are listed in Section 4 and Appendix D.The modules judge plausibility during n-gram training and clustering validation, generate scenarios, databases, and user instructions, provide a hint-assisted verifier agent, and perform strategy, perturbation, and scenario rewriting.Source: §16 · p. 41

Related papers

Loading 2605.28556v2…