PaperlayerView exact revision on arXiv

Source-linked AI summary

APIFlow-Bench: Measuring Whether Agents Survive Long, Dependent API Workflows

Zelin Wan, Arash Nourian, Xiaoxiao Li, Nihar Nandan, Kamalakannan Nandagopal

arXiv:2608.29128v1cs.AIcs.LGcs.SE

TL;DR

APIFlow-Bench addresses the limited diagnostic value of single-bit workflow evaluation by introducing an auditable benchmark for dependent REST-API workflows. It validates generated tasks, grades both provenance-linked state and typed final answers, and finds that long-chain success declines while reliability and delivery failures distinguish models.

  • Problem

    Single workflow-level pass/fail metrics do not distinguish important API-agent failures, including recovery from authentication errors and incorrect final delivery after correct backend execution.

  • Method

    APIFlow-Bench combines validated synthetic API worlds, seven capability axes, provenance-gated deterministic grading, and typed answer-card verification.

  • Results

    Across 19 models, success declines from 93% on individual subtasks to 74% on clean 20-subtask chains, while the headline findings show stronger model separation in reliability than best-case performance.

  • Takeaways & Limitations

    Long-horizon API evaluation should distinguish reach from reliability and verify that final answers originate from the correct API data flow.

  • Takeaways & Limitations

    The benchmark uses deterministic REST mocks and a 19-model slate, so its scores cover modeled failure modes rather than arbitrary production systems.

Abstract

from arXiv · show

Tool-using agents are commonly evaluated by a single bit: whether an end-to-end workflow completed. This metric fails to distinguish failures that matter in production, such as expired credentials, malformed payloads, or correct execution followed by incorrect final delivery. We introduce APIFlow-Bench, a fully auditable benchmark for long-horizon, dependent REST-API workflows that decomposes performance into seven engineering capabilities and requires agents to produce answers supported by the actual call path. We generate synthetic API worlds forward, subtask by subtask; each subtask is admitted only after a zero-LLM self-test triad verifies its grader and an oracle establishes solvability, and an adversarial audit identified and fixed six grader exploits. Grading is deterministic and provenance-sensitive: a state check traces a mock-minted canary through the API data flow to the response the answer must originate from, and a typed answer card is verified field by field. We release all answer keys and 44,362 unredacted execution transcripts. Across 19 frontier and open-weight models under one neutral scaffold, we find: (1) longer dependency chains degrade success, from 93% on individual subtasks to 74% on clean 20-subtask chains and 61% when including the 8% of chain trials that a model-consensus screen flags as passed by no model; (2) reliability separates models more than best-case capability, with best-of-five spanning seven points but all-five-of-five reliability spanning 44 points; (3) the independent-error account of compounding failure does not fit the data: pass rates on 20-subtask chains are 33 percentage points above the product of subtask-level rates, and on the clean slice 77% of failing runs reached the correct final state and failed only at delivery.

1 Introduction

APIFlow-Bench addresses the limits of single workflow-level pass/fail metrics by evaluating distinct capabilities and requiring both correct backend state and correct final delivery. It combines auditable generation and provenance-sensitive grading for dependent REST-API workflows.

  • Motivation: A single workflow-level outcome cannot distinguish authentication recovery from error-free execution or correct backend state from incorrect final delivery.This motivates measuring the failure surface beyond end-to-end completion.
  • Benchmark contribution: APIFlow-Bench evaluates authentication, discovery, schema repair, multistep execution, error recovery, pagination, and statefulness separately.Tasks can be composed into cumulative workflows of up to 20 dependent subtasks within one evolving synthetic API world.
  • Validation: Generated subtasks enter the bank only after a zero-LLM grader self-test triad and an oracle solvability gate, while assembled chains undergo golden replay.An independent adversarial audit also targeted grader exploits.
  • Evaluation: The evaluator requires both a provenance-traced state check and a field-by-field typed answer-card check.The state check follows a mock-minted canary through the API data flow to the required response.
  • Benchmark contribution: 44,362 unredacted execution transcripts accompany a public benchmark spanning 467 tasks across 13 generated REST API worlds.The release also includes a content-hash-pinned manifest.

2 Related Work

APIFlow-Bench extends tool-use and stateful-environment evaluation toward dependent REST workflows whose correct calls must compose into reliable execution. Its generated-bank validation and model-panel screening address solvability, grader soundness, and shortcut risks.

  • Positioning: APIFlow-Bench measures completion of dependent REST workflows rather than isolated tool selection or argument construction.Its unit of analysis is stateful workflow execution in which correct calls may still be insufficient.
  • Positioning: The benchmark complements stateful environments such as AppWorld and τ-bench by requiring dependent API execution rather than only final-world comparison.The cited related work evaluates interactive environments with persistent state and task outcomes.
  • Reliability: Repeated-trial evaluation distinguishes whether a model can succeed from whether it succeeds reliably under the same task conditions.Best-of-five measures reach, while all-five-of-five measures consistent success.
  • Long-horizon analysis: Controlled forward-growing chains compare observed long-chain success with predictions derived from subtask-level rates.Chain length changes without intentionally changing the tool surface, task distribution, or labeling rule.
  • Benchmark integrity: Generated interactive benchmarks require evidence that tasks are solvable, graders are sound, and evaluation shortcuts are unavailable.APIFlow-Bench uses a four-safeguard validation stack and a model-panel screen for cells no model passes.

3 APIFlow-Bench: Interface, Tasks, and Composition

APIFlow-Bench uses a fixed seven-tool contract and failure-first tickets to evaluate agents across typed API interactions and dependent task composition. Passing requires both the produced backend state and the returned answer to be correct.

  • Interface: The fixed action space contains seven polymorphic tools over five typed entity kinds, keeping the measured interface identical across adapters.The tools are read, write, edit, search, execute, clarify, and report_blocked.
  • Tasks: Failure-first tickets begin with a broken call, a truthful error hint, and a requested job outcome.The running example starts with a 401 from an expired token and asks for a live reading and reference identifier.
  • Grading: A pass requires both the backend state produced by the calls and the answer returned to the user to satisfy their respective checks.The state check and answer card therefore assess execution and delivery as separate surfaces.
  • Composition: Each storyline position becomes a subtask, while cumulative chains preserve earlier state and dependencies within one evolving world.Planted identifiers make earlier segments load-bearing where coupling exists.

4 Building a Verifiable Generated Bank

The benchmark constructs synthetic API worlds forward while certifying grader behavior, subtask solvability, chain dependencies, and discoverability before evaluation. These safeguards aim to make generated tasks reproducible and resistant to invalid shortcuts.

  • Generation: Synthetic generation provides difficulty control and a regenerable private split, but generated tasks must demonstrate fairness and solvability.Unlike a hand-authored bank, generated tasks require explicit validity evidence.
  • Forward construction: A world is initialized once and propagated through accepted subtasks, with deterministic seeds fixing axis, difficulty, and coupling sequences.Each accepted subtask emits the next world state.
  • Validation gates: The zero-LLM grader triad requires blank and sabotaged exams to fail while the answer key passes.This tests whether the validator is load-bearing before models encounter the subtask.
  • Validation gates: An oracle attempts each subtask ten times in the scoring harness, admitting it only when pass@10 ≥3.Tasks that fail the gate are revised or chains are truncated rather than padded with uncertified subtasks.
  • Chain validation: Assembled chains require forward golden replay to pass and negative replay of dependency-sensitive shortcuts to fail.Static checks also block publication when a seed-workspace value leaks into an answer card.
  • Task validity: Every success-path gate must be discoverable from the served specification or truthful error signals.Otherwise calibrated abstention could be punished while guessing is rewarded.

5 Grading: Certify the Labor, Then the Delivery

APIFlow-Bench separates successful backend execution from correct final delivery using two independent grading surfaces. Its deterministic evaluator is provenance-sensitive, and an adversarial audit exposed six implementation exploits.

  • A substring check is rejected because it can both reject equivalent values and accept lucky coincidences.
  • G1 requires the required calls to execute and traces a distinctive canary from the correct API response into the final workspace.
  • G2 compares a typed answer card field by field, preserving normalized formatting while requiring strict answer structure.
  • Provenance gating permits unredacted release of answer keys and 44,362 transcripts but does not prevent training on released solutions.
  • Six grader exploits succeeded in an independent audit, and the audit’s single-pass, uninstrumented coverage does not establish that other axes are exploit-free.

6 Experimental Setup

The experiment evaluates 19 frontier and open-weight models on a frozen benchmark under a shared, unmodified React scaffold. Results use completed trials, while uncertainty accounts for repeated epochs clustered within worlds.

  • 19 frontier and open-weight models each ran 467 tasks for 5 epochs at temperature 1.0 under Inspect AI’s stock react() loop.
  • 44,362 of 44,365 grid trials ran, and 44,343 completed without harness error; reported results use completed trials.
  • 90% confidence intervals use cluster bootstraps over worlds, keeping each model’s five epochs together.
  • At the headline slice’s true n of eleven worlds, intervals are wide and overlap, so pooled levels and full-bank statistics support the defensible comparisons.

7 Screening the Bank for Cells No Model Passes

The bank is screened for world-by-length cells that no model passes, separating potentially underspecified assembled chains from ordinary difficulty. The screen materially changes aggregate rates, but its threshold and specificity have important limits.

  • 18 of 226 chain cells received zero passes across 19 models, covering 1,710 of 21,455 chain trials (8.0%).
  • The flagged cells contain assembled grader checks for mid-chain steps omitted from compressed instructions, although reference solutions pass their forward golden replays.
  • The screen catches all-zero cells after assembly that the unit-level validation pipeline structurally misses.
  • The screening threshold has escapes: w16@19 and w08@19 each pass 1 of 95, while w08@19 and nine further 1–3-pass cells remain unadjudicated.
  • Root-cause analysis covered only positive screen results, so the method makes no specificity claim and needs controlled precision-recall evaluation.
  • The frozen bank leaves both identified consequences unrepaired to preserve reproducibility.
  • 77.0% pooled chain pass rate with flagged cells rises to 83.7% without them, while the headline slice moves from 43.6–72.7% to 53.3–88.9%.
  • Axis-level slices can be misleading because two of three length-20 authentication worlds are flagged, leaving the survivor at 17.9%.

8 Results

Longer dependent chains reduce success while exposing delivery and consistency failures that single-task or best-case metrics obscure. The benchmark also finds that observed long-chain success exceeds independent-error predictions, with many clean-slice failures reaching the correct state before failing at final delivery.

  • Chain length: 92.9% on solo tasks falls to 74.4% on clean 20-subtask chains and 60.9% when screened cells are included.The comparison holds the world set, tool surface, and difficulty labeling fixed; screened cells account for the lower inclusive figure.
  • Model comparisons: Point-estimate spreads increase from 19.8 points on solo tasks to 29.1 on chain-20, but all cluster-bootstrap intervals overlap.On the log-odds scale the spreads are 2.1 and 1.2, so the robust effect is level collapse rather than resolved model separation.
  • Final delivery: 77% of clean-slice failures reached the correct final state and failed only the typed answer card.Among 219 clean-slice failures, 169 failed only delivery while their state checks remained green, often because of a wrong value, missing field, or undeclared card.
  • Reliability: 44 points separate all-five-of-five reliability, compared with seven points for best-of-five capability.Best-of-five spans 85.4–92.3%, whereas all-five-of-five spans 42.2–86.1%; the latter is therefore the more discriminating consistency measure.
  • Failure composition: 74.4% observed pass versus 41.4% predicted at 20 subtasks yields a 33-point gap above the independent-error model.The null uses model- and world-specific solo rates; at two subtasks, observed and predicted pooled rates are 84.4% and 84.3%.
  • Failure composition: Length samples more of the failure surface rather than taxing chains multiplicatively under the benchmark’s grading regime.The paper attributes the gap to selective chain grading and correlated within-session competence, with failures concentrating around accumulated state and final delivery.

9 Conclusion, Limitations, & Future Work

APIFlow-Bench finds that long-horizon reliability declines substantially and that failures are correlated rather than independent. The benchmark’s conclusions are tempered by possible generator-family entanglement, deterministic REST-only scope, and unresolved all-zero cells.

  • Key findings: Reliability separates models more than best-case capability, with a 7-point versus 44-point spread.The paper identifies reliability separation as a key finding, alongside the level collapse and correlated long-horizon failure.
  • Limitations: A generator-family advantage remains possible because no cross-family re-gating control or gate-yield statistics were retained.Deterministic validators, golden replay, and a non-ranking verifier mitigate some risks, but do not eliminate this entanglement concern.
  • Limitations: The benchmark models REST workflows under a seven-tool contract rather than arbitrary production systems.GraphQL and MCP adapters ship without task banks, and deterministic mocks trade realism for replayability and difficulty control.
  • Key findings: The observed chain-pass gap above the independent-failure baseline grows with chain length.The figure’s comparison uses solo-rate products as the null model and observed chain pass rates as the empirical curve.
  • Key findings: The benchmark’s taxonomy decomposes prior API evaluations into seven capability axes, while auth refresh and schema repair remain least targeted.The comparison is presented as coverage of engineering capabilities, not as a quality ranking of prior benchmarks.

E Per-Model Results on Every Task Set

Per-model results compare cost, pass rates, and variability across task sets and chain lengths. The figures caution that small samples, uneven worlds, screened cells, and cost differences complicate simple model rankings.

  • Cost and pass rate: Below $0.01 per trial, the cost–pass-rate frontier is entirely open-weight.The prices are release-pinned list prices and should not be read as serving-efficiency measurements.
  • Per-model comparisons: Cost spans two orders of magnitude at comparable accuracy across the 19-model panel.Table 3 reports all five task sets, 90% cluster-bootstrap intervals on the headline slice, and mean trial cost.
  • Chain-length trends: Pass-rate descent across chain lengths is real but non-monotone, with uneven world contributions and small per-length samples.Flagged cells can drive specific lengths sharply downward, so pooled trends are more informative than individual points.
  • World and model variation: Variation across worlds exceeds variation across models in the 20-subtask slice.Two all-zero rows require diagnosis because reference solutions clear the assembled grader despite no model passing.
  • Evaluation integrity: All reported numbers follow an adversarial audit that found six exploits in the completed grader and fixed them.The table documents the six exploits and their fixes; reported results are post-fix.

H Datasheet, Licensing, and Reproduction

The release provides a procedurally generated, privacy-preserving REST benchmark with public artifacts, transcripts, and reproducibility tooling. Reproducing the leaderboard still requires provider API keys, while local bank and replay checks do not require network access or credentials.

  • Datasheet: The bank contains 467 tasks across 13 procedurally generated worlds, comprising 241 solo subtasks and 226 cumulative chains.It uses no human subjects, user traffic, or personally identifiable information.
  • Licensing and distribution: The harness, frozen bank, transcripts, and leaderboard are publicly released through the project repositories.The artifact URLs identify separate locations for the harness and bank, transcripts, and leaderboard.
  • Licensing and distribution: The harness and scoring artifacts are Apache-2.0 licensed, and 44,362 raw trial transcripts are released.Model-level bundles are attached to repository releases, and the authors state that the license permits the stated use.
  • Reproduction: Local bank verification and golden replay run without network access or credentials, but leaderboard reproduction requires provider API keys.The published 467 × 5 grid cost roughly $8 to $730 in total for each model under release-pinned prices.
  • Reproduction: Versioned releases pin the bank path, content hash, epoch count, and price-table version to preserve reproducibility over time.Golden replay re-verifies equivalence to the evaluated bank after annotation stripping, while flagged cells remain listed rather than silently repaired.
  • Safe use: Third-party task banks should be treated as executable code because task code runs inside the harness process.The warning specifically covers validator.py and mock_overrides.py.
Loading 2608.29128v1…