PaperlayerView exact revision on arXiv

Source-linked AI summary

Multi-Step Tool-Calling over Korean Open Public APIs: A Benchmark and a Data-Synthesis Recipe

Dain Kim, Eungi Cho, Kyumin Kim, Shinyeong Noh, Kyuseong Lim

arXiv:2609.05395v1cs.AIcs.CL

TL;DR

Open-source models underperform on multi-step tool calling over live Korean public APIs, while existing benchmarks do not measure this setting. The paper introduces KOPA-BENCH and EDGE, which uses live execution to verify dependencies and synthesize executable trajectories. GRPO training on EDGE data substantially improves small models on KOPA-BENCH and BFCL, with the 9B model nearly matching a much larger model.

  • Problem

    Open-source models underperform on multi-step tool calling over live Korean public APIs, where tasks require dependent lookups and high-cardinality response handling.

  • Method

    The paper builds KOPA-BENCH and uses EDGE to verify tool dependencies through live execution and synthesize executable multi-step trajectories.

  • Results

    GRPO training on EDGE data substantially improves small open-source models on KOPA-BENCH and extends gains to BFCL.

  • Takeaways & Limitations

    KOPA-BENCH and EDGE provide a foundation for evaluating and training tool-calling agents over live public APIs.

  • Takeaways & Limitations

    Because the benchmark and dataset use live APIs, endpoint schemas, availability, and records may drift, so exact reproduction depends on public-service stability.

Abstract

from arXiv · show

Data-sovereignty regulations increasingly require public institutions to deploy open-source, on-premise LLM agents that chain multiple tool-calls across live government APIs. However, open-source models consistently underperform in this multi-step setting, and no existing benchmark measures the gap. We introduce the Korean Open Public API Benchmark (KOPA-Bench), comprising 145 real-world tasks. To close this gap, we present EDGE, an Execution-grounded Dynamic Graph for tool-calling data synthEsis driven by live execution. EDGE builds a graph of how each tool's output can feed another's input, keeps only the links that succeed when actually called against the live APIs, and traverses these verified links to synthesize executable multi-step trajectories. Fine-tuned via GRPO on the resulting dataset, our 9B model nearly matches the untuned 27B model from the same family, improving substantially not only on KOPA-Bench but also on the BFCL benchmark.

1 Introduction

Open-source models underperform on live Korean public APIs because multi-step tasks require prerequisite lookups and careful handling of multi-record responses. KOPA-BENCH and EDGE address this gap by benchmarking and synthesizing executable tool-call trajectories.

  • Public institutions need open-source agents that operate over laws, corporate disclosures, and transportation data exposed through live public APIs.
  • Entity code lookups and high-cardinality responses create dependent chains that open-source models frequently mishandle.Models may skip prerequisite lookups or answer prematurely from partial results.
  • KOPA-BENCH contains 145 multi-step tasks across six domains of live Korean public APIs, with one tool’s output supplying the next tool’s input.
  • Existing tool-calling datasets do not target Korean public APIs, live endpoint failures, or high-cardinality handoffs.
  • EDGE verifies tool dependencies against live APIs and synthesizes trajectories that resolve high-cardinality outputs through bounded fan-out or deterministic reduction.
  • GRPO training on EDGE data raises KOPA-BENCH pass@1 by +13pp for 4B and +10pp for 9B models, with gains extending to BFCL.

2 Benchmark Construction

KOPA-BENCH evaluates multi-step function calling on real Korean public APIs across diverse domains and platforms. Its tasks use validated tool chains and score both outcomes and the actions producing them.

  • KOPA-BENCH comprises 145 tasks across 10 platforms and six domains, grounded in real Korean public APIs.
  • The benchmark selects platforms for domain coverage, API interconnectivity, and licensing that permits derivative works under KOGL.Tasks require five tool-calls on average, with up to 14 calls.
  • Each platform exposes typed MCP function signatures backed by clients that handle authentication, sessions, errors, retries, and task environment state.
  • Domain experts create chainable tool sets and queries annotated with golden actions, optimal trajectories, and target responses or environment states.
  • Two-stage validation audits every task and repeatedly executes corrected golden trajectories until all 145 tasks pass.The audit initially found errors in 7 tasks, or 4.8%, and 80% passed the first execution check excluding model failures.
  • Evaluation separates RESPONSE and ENVIRONMENT outcome criteria from ACTION scoring of the executed tool-call trajectory.

3 Methods

EDGE constructs a live-execution-validated dependency graph, then synthesizes trajectories whose junction types handle single values, bounded fan-out, and high-cardinality reductions. It generates and validates natural-language queries and answers from these executions.

  • EDGE overview: EDGE uses two phases: Phase A retains successfully executing dependency edges, while Phase B assembles typed trajectories and generates Korean queries and answers.
  • Phase A: Graph construction: The tool inventory contains 2,318 live tools whose responses may range from zero to hundreds of thousands of records.
  • Phase A: Graph construction: A dense retriever limits candidate dependencies to 15 same-domain and 10 cross-domain neighbors before LLM feasibility scoring and parameter binding.
  • Phase A: Graph construction: Admitted edges receive feasibility-anchored Beta priors, then live execution updates posteriors and topology through Thompson sampling with ε-greedy exploration.
  • Phase A: Graph construction: Edges are pruned when posterior evidence indicates insufficient success probability, yielding a refined graph G⋆ for trajectory synthesis.
  • Phase B: Trajectory synthesis: Cardinality typing assigns SEQ for one value, FAN for 2–5 values, and DRV above five, using bounded enumeration or deterministic reduction for downstream calls.
  • Phase B: Trajectory synthesis: EDGE also synthesizes semantic-parallel, comparison, and conditional trajectories, retaining only successfully executed paths.
  • Phase B: Trajectory synthesis: An LLM generates each Korean query and derives its answer from cached execution results, followed by three-stage validation.

4 Experiments

The experiments evaluate EDGE-trained models on KOPA-BENCH and BFCL, then isolate the contributions of the training objective, trajectory composition, filtering, and execution-grounded graph refinement. Across these analyses, filtered and execution-verified data consistently support stronger tool-calling performance.

  • Main results: KOPA-BENCH and BFCL evaluate whether EDGE training improves Korean public-API tool use and generalizes beyond the synthesis distribution.The evaluation reports pass@1 and pass@4 using four rollouts per task, with additional confidence-interval analysis over eight seeds.
  • Main results: 0.18 to 0.31 (+13pp) and 0.33 to 0.43 (+10pp) are the KOPA-BENCH pass@1 gains for the 4B and 9B models, respectively.The 9B model reaches 0.43, approaching Qwen3.5-27B at 0.45 with one-third as many parameters.
  • Main results: BFCL multi-turn performance improves by +4.04pp for 4B and +5.87pp for 9B, exceeding the corresponding single-turn gains.The models improve in aggregate over their base counterparts despite training data being predominantly single-turn.
  • Training objective: SFT alone raises pass@1 by +9.7pp and Action by +9.0pp, while GRPO adds a further consistent gain across all three metrics.Both methods use the identical 1,781-task corpus; GRPO reaches 0.4690 pass@4 from 0.3586 under SFT.
  • Trajectory composition: 0.2327, 0.3080, and 0.3094 pass@1 are achieved by pure-sequential, parallel + MIXED, and full trajectory subsets, respectively.The full subset reaches 0.4690 pass@4, while parallel + MIXED reaches 0.4000; MIXED trajectories comprise 36.7% of the corpus.
  • Data filtering: 0.242 to 0.309 (+6.7pp) is the pass@1 increase after removing synthesized tasks with anomalies such as malformed structures and stale labels.The filtering improvement extends consistently across all metrics.
  • Execution-grounded graph: 62.7% of G⋆ edges execute successfully versus 50.2% of skeleton edges and 14.8% of pruned edges.Among pruned edges, 70.5% never succeed in any trial, compared with 27.7% among retained G⋆ edges; residual zero-success edges in G⋆ reflect limited exploration.

5 Conclusion

The paper introduces KOPA-BENCH to expose limitations in multi-step tool-calling over live Korean public APIs and presents EDGE to synthesize verified trajectories. GRPO training on the resulting data substantially improves small open-source models, nearly matching larger architectures and generalizing to BFCL.

  • 5 Conclusion: KOPA-BENCH measures multi-step tool-calling over Korean public APIs, while EDGE synthesizes verified trajectories through execution-grounded dependency graphs.The benchmark reveals existing performance limitations, and the framework retains tool dependencies validated against live APIs.
  • 5 Conclusion: GRPO-trained small open-source models achieve substantial gains, nearly matching much larger architectures and generalizing to BFCL.The conclusion positions KOPA-BENCH and EDGE as a foundation for deploying LLM agents in public applications.

Limitations

The benchmark and EDGE dataset depend on live Korean public APIs, limiting reproducibility when services change and leaving transfer beyond Korean public-sector settings unresolved. Comparisons also isolate execution grounding within EDGE rather than benchmarking complete alternative synthesis systems.

  • Reliance on live endpoints: Live API schemas, availability, and returned records can change over time, so exact reproduction depends on public-service stability outside the authors’ control.The authors filter real-time-data tasks and compare environment states by hash, but these mitigations do not eliminate endpoint drift.
  • Beyond Korean public APIs: KOPA-BENCH and EDGE cover Korean public-sector APIs across six domains, leaving transfer to other languages, private APIs, and countries open.The paper leaves extension to these broader settings for future work.
  • Scope of comparison: The study compares EDGE with controlled variants rather than running an existing synthesis system end to end on the same tool inventory.This design isolates execution grounding while leaving system-level comparisons with prior synthesis pipelines open.
  • Scope of comparison: Prior benchmarks and synthesis methods use related function-calling or simulated environments, but the supplied discussion does not establish direct system-level comparability with EDGE.The related-work passages describe existing benchmarks and synthesis approaches without reporting an end-to-end comparison under this study’s tool inventory.

B.1 Task Example

A representative Finance task requires chained lookups and computation under runtime-injected state, then evaluates the model’s answer, actions, and resulting environment state against annotated ground truth. The benchmark therefore tests both solving the query and executing an appropriate trajectory.

  • Task setup: A Finance task asks for the percentage difference between a stock’s latest closing price and a convertible bond’s refixing floor.The query refers to Lightron Fiber-optic Devices’ January 2026 convertible-bond issuance and uses the current date.
  • Task execution: The expected trajectory resolves a corporate code, retrieves the bond’s refixing floor, fetches the prior closing price, and computes the percentage gap.The example yields corp code 00367482, refixing floor 532, closing price 1680, and a computed result of 215.78....
  • Runtime state: Runtime system state combines static values such as current_date with dynamically resolved values such as an API key.PredefinedSystemState supplies static entries, while ActionSystemState executes a function at the start of each episode.
  • Evaluation workflow: The task pairs a natural-language query with injected system state and evaluates tool calls and the final answer against golden actions, expected response, and resulting environment state.The evaluation workflow checks multiple aspects of the episode rather than only the final text.
  • Response evaluation: Response evaluation supports string, number, and judge-based answer types, using normalization or symbolic comparison before falling back to LLM-as-a-Judge where specified.Open-ended Law tasks rely solely on LLM-as-a-Judge, while string and number tasks use format-specific extraction.
  • Trajectory evaluation: Environment evaluation compares the final isolated server state with the golden state using SHA-256 hashes, while ACTION evaluation measures trajectory correctness and efficiency.ACTION correctness matches required calls and parameters, and efficiency penalizes duplicates or exceeding the allowed-call budget.

C Statistical Reliability of Main Results

Because KOPA-BENCH contains only 145 tasks, the authors re-evaluate models across eight independent seeds using Student’s t confidence intervals. The resulting intervals are narrow and non-overlapping between fine-tuned models and their base counterparts, supporting statistically robust improvements.

  • Methodology: Eight independent random seeds are used to estimate 95% confidence intervals for pass@1, addressing uncertainty from the benchmark’s modest 145-task size.The main-text results use four-run averages, while the appendix reports revised eight-seed estimates that remain consistent.
  • Methodology: Student’s t intervals are used instead of a normal approximation because n=8 is small, yielding more conservative uncertainty estimates.The interval is based on the mean and standard deviation of pass@1 across seeds.
  • Results: The reported confidence intervals are narrow, with widths of roughly 0.02–0.05 in pass@1, indicating stable results across seeds.The paper reports stability despite the limited number of benchmark examples.
  • Results: The fine-tuned and base-model intervals do not overlap: [0.2762, 0.3100] versus [0.1400, 0.1807] for 4B, and [0.4040, 0.4235] versus [0.3277, 0.3775] for 9B.The paper interprets these non-overlapping intervals as statistically robust improvements rather than artifacts of benchmark size.

D Benchmark Statistics

KOPA-BENCH covers 10 platforms across six domains and measures task structure through sequential steps, tool calls, efficiency headroom, available tools, and parallel execution. EDGE’s graph procedure updates and prunes tool dependencies using execution outcomes and posterior viability estimates.

  • Benchmark coverage: The benchmark spans 10 platforms across six domains, with task statistics computed from golden actions and metadata.The supplied passages identify the benchmark’s platform and domain breadth but do not enumerate the per-domain values.
  • Task statistics: Parallel actions share a step, so the mean tool-call count is 4.92 while the mean step count is 3.32.Tool calls count golden actions, whereas steps count sequential reasoning stages after grouping parallel actions.
  • Task statistics: Efficiency headroom is max_allowed_calls/Nopt; a value of 1.0 makes ACTION efficiency zero after the first redundant call, while larger values permit more tolerance.Available tools and parallel percentage respectively capture selection difficulty and the fraction of tasks containing parallel calls.
  • Graph updates: EDGE classifies edge executions into structural or environmental outcomes, updating Beta posteriors more heavily for structural failures than environmental noise.Successes increment αe and failures increment βe, with the update magnitudes determined by outcome type.
  • Graph pruning: An edge is pruned only after enough trials when its posterior mass above viability threshold τ falls below confidence level ε.The rule uses Pr[θe > τ] = 1 − Iτ(αe, βe) together with the edge’s trial count.
  • Graph updates: EDGE initializes admitted-edge Beta priors from LLM feasibility scores, samples paths with Thompson sampling, updates outcomes, and prunes edges iteratively.An edge enters the skeleton only when its feasibility score reaches s_min, with κ controlling prior commitment.

E.3 Graph Convergence Analysis

EDGE progressively concentrates its graph on executable dependencies by using live execution to update edge reliability and prune failures. Both path-level and step-level execution success rise during refinement, while pruning decisions follow execution-grounded posteriors rather than LLM priors.

  • Execution-driven convergence: +31pp trajectory pass rate and +28pp step success rate accrue over 100 iterations as unreliable edges are pruned.The loop removes structurally failing edges and increasingly routes execution toward edges with demonstrated success.
  • Execution-driven convergence: Execution-grounded posteriors can move edges away from their LLM feasibility scores in both directions.With κ = 2.0, high-prior edges often fall below the diagonal while some low-prior edges rise above it.
  • Execution-driven convergence: Retained and pruned edges separate near the posterior threshold τ rather than along the prior axis.This indicates that pruning uses observed execution outcomes, including rejecting highly rated edges that live APIs contradict.
  • Data filtering: A three-stage cascading pipeline removes execution flaws, textual anomalies, logical leaks, and ground-truth errors from synthesized data.Rule-based filters remove unreproducible, erroneous, redundant, and malformed instances; later stages use LLM judgments and independent re-solving.

G Training Data Analysis

The EDGE training corpus is structurally demanding because Korean public APIs produce frequent one-to-many and long-range tool-call dependencies. Its construction and filtering target these characteristics rather than simple one-to-one tool use.

  • Training-data structure: 64.9% of single-junction sequential trajectories use FAN or DRV one-to-many junctions, versus 35.1% using one-to-one SEQ junctions.High-cardinality API fields make one-to-many junctions the norm in the 1,781-task corpus.
  • Training-data filtering: The synthesis pipeline filters execution flaws, textual anomalies, and logical leaks before training-data use.The supplied passages identify the filtering taxonomy and its representative examples, while the hyperparameter table records EDGE settings.
  • Training-data structure: 4.13 tool calls occur per task on average, with up to 36 calls, and 57.6% of tasks require at least four sequential hops.These chains require models to sustain dependent calls across multiple steps.
  • Comparison with prior datasets: 74.1% of corpus calls return at least two records, while 81.2% of chained calls are one-to-many.Median chained cardinality is 27 and reaches 224,958, far exceeding prior datasets’ single-digit cardinalities.

H Data Contamination Audit

The contamination audit separates intended sharing of the public-API tool pool from direct task or dependency leakage. Evaluation includes held-out platforms and entirely unseen dependency structures, supporting a non-contaminated test of generalization.

  • Tool and query overlap: 0 of 145 evaluation queries appear verbatim in training, while tool-universe overlap is 62.2% (135 of 217 gold functions).The overlap reflects the shared API pool rather than direct query contamination.
  • Held-out platforms: 31 evaluation tasks use three platforms whose tools never enter synthesis, making the held-out unit the platform rather than an individual function.The held-out platforms are Seoul Open Data Plaza, DART, and KRX.
  • Held-out platforms: +22.6pp is the fine-tuning gain on held-out platforms, from 0.2903 to 0.5161, exceeding the full-benchmark gain of +15.9pp.This improvement is reported for Qwen3.5-4B pass@4 on the 31 held-out-platform tasks.
  • Audit definitions: The audit distinguishes function-pair dependency edges from adjacency edges and excludes API-key acquisition edges.Function-pair edges capture output-to-input dependencies, while adjacency links consecutive calls regardless of dependency.
  • Dependency-structure overlap: Dependency edges have 0.0 Jaccard similarity, and all 178 evaluation dependency-edge occurrences are unseen in training.Even under the looser adjacency definition, 95.6% of evaluation edge occurrences remain unseen.
Loading 2609.05395v1…