PaperlayerView exact revision on arXiv

Source-linked AI summary

SCCGBench: Executability-Oriented Evaluation for Real-World Web Service-Call Code Generation

Shuo Wang, Guodong Fan, Ming Zhu, Jing Li, Chun Wang

arXiv:2610.12920v1cs.SE

TL;DR

Existing benchmarks do not directly evaluate whether generated Web-service code reaches its intended endpoint, and LLM-completed comments can leak interface details. SCCGBench builds a traceable benchmark with redacted intents and execution-grounded metrics. Models often look plausible but fail in intent-only generation, while gold documentation substantially raises Mock success.

  • Problem

    Existing benchmarks rarely establish whether generated HTTP requests hit the correct external endpoint, and LLM-completed comments can leak hosts or routes.

    Source: §I · p. 1

  • Method

    SCCGBench builds a provenance-preserving benchmark of real GitHub service-call functions, uses redacted intents, and evaluates requests with Mock, Robust, and interface-level diagnostics.

    Source: §I · p. 2

  • Results

    Models often look right but fail to execute in intent-only generation, while gold documentation raises Mock to approximately 89–90% for most open-source models and about 82% for DeepSeek-V4-Pro.

    Source: §I · p. 2

  • Takeaways & Limitations

    The large RAG–Oracle gap indicates that reliable service-call automation should use a retrieve–select–generate pipeline, with API recommendation or reranking as an independently optimizable component.

    Source: §VI · p. 10

    Construct validity is limited because Mock and Robust use virtual rather than full real-network execution, and external validity is limited to Python and the RapidAPI ecosystem.

    Source: §VI · p. 10

Abstract

from arXiv · show

LLM-generated service-call code is increasingly important for service computing, where developers rely on large language models (LLMs) to compose services and invoke third-party services. However, current approaches have not yet evaluated the invocation and execution of services in the actual code environment. To support execution-grounded evaluation, we present the Service-Call Code Generation Benchmark (SCCGBench), a benchmark of 3,135 traceable and decontamination-filtered RapidAPI call functions from real GitHub repositories. SCCGBench uses redacted-intent generation to reduce circular leakage and introduces Mock, Robust, ParamF1, and host-, endpoint-, method-, and parameter-level diagnostics. Mock denotes a virtual-execution hit that simultaneously matches the gold host, endpoint, and HTTP method, while Robust further requires status checking and exception handling. We evaluate six code LLMs and one non-neural retrieval baseline under intent-only, retrieval-augmented, and gold-documentation Oracle settings. Results show that models often generate syntactically correct or superficially similar HTTP-request code but achieve low Mock success in the intent-only setting; the strongest evaluated model obtains only 3.76% Mock. Under the gold-documentation Oracle settings (C3 and C5), Mock reaches 88.73%--90.52% for the open-source models evaluated in RQ2 and 76.47%--81.86% for DeepSeek-V4-Pro, whereas sparse and dense retrieval remain far below this oracle level. The findings indicate that service/API grounding is the main bottleneck: Mock is dominated by host, endpoint, and method alignment, while ParamF1 separately reveals parameter-alignment errors. SCCGBench provides an execution-grounded benchmark for Web service invocation, API recommendation, service-interface retrieval, endpoint grounding, and reliable service-call automation.

I. INTRODUCTION

Existing benchmarks and metrics do not directly establish whether generated Web-service code reaches the intended endpoint, while comment generation can leak interface details. SCCGBench addresses these issues with traceable real-world examples, redacted intents, and execution-grounded evaluation.

I. INTRODUCTION

  • Generated requests must align with the intended host, endpoint route, HTTP method, and parameters; plausible, syntactically complete code can still target the wrong interface.The paper frames service-call generation as both service-interface grounding and code synthesis.Source: §I · p. 1
  • Existing benchmarks emphasize static similarity or unit tests, which cannot establish whether a request hits the correct endpoint and are difficult to apply at scale to live services.Public sources may also contain repetitive wrappers and templates that do not reflect diverse in-the-wild API use.Source: §I · p. 1
  • LLM-completed comments may reveal hosts, endpoints, parameter keys, or URL fragments, turning interface grounding into answer extraction.SCCGBench masks these details before generating intents for samples without valid comments.Source: §I · p. 2
  • Models remain below 3.8% Mock in intent-only generation, while gold documentation raises Mock to approximately 89–90% for most open-source models and about 82% for DeepSeek-V4-Pro.BM25 and dense retrieval still leave many correct endpoints outside the candidate set.Source: §I · p. 2
  • SCCGBench contains 3,135 samples, 850 APIs, and 2,082 repositories, with provenance preservation and redacted intent generation to reduce contamination and circular leakage.Paired validation and group-wise ablations show no evidence that redacted intents provide an execution advantage over human comments.Source: §I · p. 2
  • The evaluation framework combines Mock and Robust with host-, endpoint-, method-, and parameter-level diagnostics to measure service-interface grounding.The paper identifies API documentation retrieval and endpoint grounding as central bottlenecks for reliable invocation.Source: §I · p. 2

II. BACKGROUND AND RELATED WORK

Prior work spans service recommendation, LLM-based tool use, and increasingly realistic code benchmarks. The paper positions SCCGBench as addressing whether generated code constructs a request that reaches a concrete external endpoint.

II. BACKGROUND AND RELATED WORK

  • Service-call code generation shifts binding to a concrete host, route, method, and parameter schema from human-written code to model-generated code.The task is related to Web service discovery, API recommendation, service selection, and service management.Source: §II · p. 2
  • Tool/API studies show the importance of tool selection and documentation grounding, while SCCGBench evaluates complete executable request construction at the code level.The cited examples include ToolLLM, API-Bank, RestGPT, and Gorilla –.Source: §II · p. 3
  • Existing code benchmarks improve realism and contamination control but rarely test whether an HTTP request reaches the correct external endpoint or address circular leakage from LLM-completed comments.SCCGBench addresses this gap through real provenance, redacted intents, and endpoint-level mock execution.Source: §II · p. 3

III. CONSTRUCTION AND EVALUATION OF SCCGBENCH

SCCGBench combines a benchmark-construction pipeline with metrics that compare generated requests against hidden interface ground truth. Its evaluation distinguishes complete interface hits, request construction, parameter alignment, and robustness.

III. CONSTRUCTION AND EVALUATION OF SCCGBENCH

  • SCCGBench’s pipeline covers RapidAPI documentation crawling, GitHub code retrieval, AST-based decoupling, multi-source alignment, template decontamination, redacted intent generation, API-level splitting, and executability evaluation.The overview is shown in Fig. 2.Source: §III · p. 3

A. Problem Formalization

  • Endpoint@Acc is host-independent, so a matching route can score even when the host is wrong; ExecReq separately measures whether the code constructs a structurally complete virtual request.Host, endpoint, and method accuracies provide component-level grounding diagnostics.Source: A. Problem Formalization · p. 3
  • Mock requires the generated host, endpoint, and HTTP method to match the gold interface simultaneously, while ParamF1 measures request-parameter alignment against documentation-derived parameters.The evaluation compares the captured request in virtual execution with the hidden ground-truth interface rather than comparing generated code text with a reference implementation.Source: A. Problem Formalization · p. 3
  • Robust further requires status checking and exception handling in addition to satisfying Mock.This distinguishes a correct interface hit from a call that also includes these reliability checks.Source: A. Problem Formalization · p. 3

B. Data Acquisition and Multi-Source Alignment

The data-acquisition and alignment process combines RapidAPI documentation with GitHub request code, then records the strength of each interface match and sample provenance.

B. Data Acquisition and Multi-Source Alignment

  • RapidAPI crawling supplies endpoint descriptions, parameter schemas, response definitions, and normalized metadata for hosts, routes, methods, parameter keys, and type constraints.These metadata serve as ground truth for alignment and evaluation.Source: B. Data Acquisition and Multi-Source… · p. 3
  • GitHub code retrieval uses AST parsing and a strict-to-loose hierarchy of matching signals, from exact endpoint URLs through route-and-host and host matches to headers, request libraries, and function definitions.Earlier levels provide stronger endpoint-grounding evidence; later levels improve coverage for long-tail interfaces.Source: B. Data Acquisition and Multi-Source… · p. 3
  • Alignment labels record whether documentation matches code by full URL, route plus host, or host alone, while repository metadata and code-hash deduplication preserve traceability and avoid duplicate implementations.The alignment granularity also supports micro-level diagnostic stratification.Source: B. Data Acquisition and Multi-Source… · pp. 3–4

C. Data Processing and Decontamination

SCCGBench cleans service-call functions and reduces circular leakage by combining structural extraction, template filtering, and redacted intent generation. It also extracts dependency and parameter-schema information for evaluation.

C. Data Processing and Decontamination

  • AST-based extraction isolates target request functions, removes irrelevant context, validates syntax, and formats code for atomicity and readability.Source: C. Data Processing and Decontamination · p. 4
  • Template-style samples are filtered using signatures such as fixed placeholder keys and repetitive request structures, followed by audits.Source: C. Data Processing and Decontamination · p. 4
  • Redacted intents are generated from code after replacing hosts, URLs, endpoint paths, parameter keys and values, and code keys with placeholders; leakage checks trigger one stricter retry.Few-shot human comments guide the style while the generated intent describes the user goal rather than implementation details.Source: C. Data Processing and Decontamination · p. 4
  • AST imports and request construction are used to extract dependencies and parameter definitions, which support execution checks and ParamF1 against documentation schemas or observed call-code parameters.Source: C. Data Processing and Decontamination · p. 5
  • Paired tests on 291 human-commented samples found no significant Mock or ES advantage for redacted intents over human comments; unredacted transcriptions made the task significantly easier.The paired tests reported p=0.68 for Mock and p=0.65 for ES; unredacted comments yielded Mock 7.9 vs. 3.4, p=0.012.Source: C. Data Processing and Decontamination · p. 5

D. Data Packaging, Quality, and Statistics

SCCGBench packages traceable service-call examples with provenance and quality metadata, and reports repository, API, and dependency characteristics. The experimental setup evaluates intent-only generation, retrieval augmentation, oracle documentation, and benchmark reliability.

D. Data Packaging, Quality, and Statistics

  • The dataset contains 3,135 traceable call functions from 2,082 repositories, with a 96% precise mapping rate in a manual audit of 50 complex samples.Average stars and forks are 35.83 and 8.39; samples average 36.2 effective lines of code, with a median of 28.Source: D. Data Packaging, Quality, and… · p. 5
  • Samples preserve implementation, API-alignment, repository, and comment-source metadata; Table II presents dataset statistics.Source: D. Data Packaging, Quality, and… · p. 5, Table II · p. 5
  • The dataset covers 850 APIs with a long tail: 418 appear once, while the top 20% account for 64% of samples.About 79% of repository updates fall in 2024–2026; 71% of samples are from services with at least 1,000 subscriptions, and 84% involve FREE/FREEMIUM interfaces.Source: D. Data Packaging, Quality, and… · p. 5

IV. EXPERIMENTAL SETUP

  • The experimental questions examine intent-only executability and static similarity, whether retrieval and gold documentation improve executability, and reliability across contamination, temporal leakage, metric trustworthiness, and decoding diversity.Source: §IV · p. 5

A. Model Selection

The evaluation combines one retrieval baseline and six code LLMs with shared generation prompts and multiple diagnostic metrics. Virtual execution captures requests without contacting real services.

A. Model Selection

  • One non-neural retrieval baseline and six code LLMs are evaluated; BM25 copies the nearest training code, and generation uses temperature 0, top-p=1, and greedy decoding.DeepSeek-V4-Pro is evaluated in Non-think mode for protocol consistency.Source: A. Model Selection · p. 5, Table III · p. 6

B. Task Design and Prompt Template

  • Models fill valid parameter values and synthesize a complete Python request function; Oracle C5 adds status checking and try-except handling to the gold-grounding C3 setting.Intent-only, RAG, and C3 share a prompt without a fixed error-handling template.Source: B. Task Design and Prompt Template · p. 6

C. Evaluation Metrics

  • Metrics cover static similarity, syntax and dependency validity, host/endpoint/method accuracy, ParamF1, request constructability, Mock, Robust, retrieval quality, and robustness agreement.Mock and Robust are defined elsewhere in the paper; this passage lists the evaluation metrics.Source: C. Evaluation Metrics · p. 6

D. Virtual Execution Environment and Mock Judgment

  • The virtual environment parses generated code, replaces network libraries with fakes, captures request details, and returns a placeholder response without making network calls.Timeouts and exception handling isolate failures, while module restoration avoids evaluation side effects.Source: D. Virtual Execution Environment and… · p. 6

E. Evaluation Strategy: From Macro to Micro

On the 612-sample test set, static similarity did not predict service-call executability: intent-only generations often constructed plausible requests but targeted the wrong interface. Macro-to-micro analysis locates the dominant failure at API grounding, especially host selection.

E. Evaluation Strategy: From Macro to Micro

V. EXPERIMENTAL RESULTS

  • The 612-sample test supports three findings: static similarity diverges from executability, gold documentation improves executability by an order of magnitude, and reliability analyses support the benchmark’s decontamination and metrics.Source: §V · p. 6

A. RQ1: Benchmark Performance and Executability Diagnosis

  • BM25 leads static CodeBLEU, Edit Similarity, and Recall, while DeepSeek-V4-Pro reaches CodeBLEU 25.18; Qwen-32B has Token Precision 66.75 and Recall 40.52.DeepSeek-Coder-1.3B has the lowest scores and 31.9% syntax correctness.Source: A. RQ1: Benchmark Performance and… · pp. 6–7, Table IV · p. 7
  • 3.76% Mock was the strongest evaluated model’s intent-only score; larger neural models had near-perfect syntax and 80–92% request constructability but at most 3.8% Mock.Host accuracy was 3–8% and endpoint accuracy 12–19% for these larger models; BM25 had the highest static similarity but Mock remained 0.Source: A. RQ1: Benchmark Performance and… · p. 7
  • For Qwen-32B, Mock rises from 0% with host-only documentation to 4.6% with a complete URL, while Method@Acc and ExecReq stay near 88%.Endpoint@Acc rises from 0.9% to 24.9% across those strata.Source: A. RQ1: Benchmark Performance and… · p. 7
  • Across 3,672 intent-only generations, host errors account for 69.8% of failures, compared with 11.9% syntax errors and 13.6% request-construction failures.Endpoint and method errors after a correct host account for 2.6% and 0.1%, respectively.Source: A. RQ1: Benchmark Performance and… · p. 7
  • Representative failures include falling back to a familiar public API or confusing similarly named RapidAPI services, despite successful syntax and request construction.Source: A. RQ1: Benchmark Performance and… · p. 7

B. RQ2: Information Augmentation and the Retrieval Bottleneck

The RQ2 evaluation tests whether retrieval and gold documentation improve service-call executability. Oracle settings produce the strongest Mock results, while retrieval—especially endpoint retrieval—remains a bottleneck.

B. RQ2: Information Augmentation and the Retrieval Bottleneck

  • Qwen2.5-Coder-32B Mock rises from 2.9% intent-only to approximately 89% under Oracle, while DeepSeek-V4-Pro rises from 3.8% to 76.5–81.9%.For Qwen2.5-Coder-32B, BM25 RAG reaches 20.3% and dense RAG 25.7–28.3%; for DeepSeek-V4-Pro, the corresponding ranges are 20.1% and 22.7–26.8%.Source: B. RQ2: Information Augmentation and… · p. 8
  • BM25 reaches 33.99% Endpoint@10, while dense retrieval reaches 52.45–54.41%; average Mock remains about 25.5% with dense retrieval versus 86.3% for Oracle C3.The reported dense-retrieval-to-Oracle C3 Mock gap is about 61 points; ParamF1 under RAG stays in the low single digits.Source: B. RQ2: Information Augmentation and… · p. 8
  • Dense retrieval mitigates but does not remove the API-documentation retrieval and endpoint-grounding bottleneck.With gold documentation, most open-source models reach about 89–90% Mock; C5 adds status-checking and exception-handling constraints after endpoint grounding.Source: B. RQ2: Information Augmentation and… · p. 8

C. RQ3: Reliability and Bias Analysis

The reliability analyses find no evidence that redacted intents or the temporal split create an unfair execution advantage, and grounding metrics align closely with human judgments. Sampling diversity yields only marginal executability gains.

C. RQ3: Reliability and Bias Analysis

  • Redacted-intent and human-comment groups show no significant Mock difference in the matched or group-wise analyses, while unredacted transcription comments yield Mock 7.9 vs. 3.4 (p=0.012).The matched tests report p=0.68 for Mock and p=0.65 for ES; group-wise examples are Qwen-32B 2.92 vs. 2.96 and DeepSeek-V4-Pro 3.8 vs. 3.7.Source: C. RQ3: Reliability and Bias Analysis · p. 8
  • The June 2024 temporal split shows no significant Mock degradation after the cutoff; DeepSeek-V4-Pro post-cutoff Mock rises from 3.11 to 3.99.The authors treat the cutoff as a coarse boundary, not a guaranteed knowledge cutoff for every model.Source: C. RQ3: Reliability and Bias Analysis · pp. 8–9
  • Grounding metrics achieve human–machine Cohen’s κ of 0.86–1.0, but broad judgments of whether code “can run successfully” reach κ=0.05.The human counterpart of Mock has κ=0.90; Robust’s error-handling judgment has moderate agreement at κ=0.32.Source: C. RQ3: Reliability and Bias Analysis · p. 9, Table IX · p. 9
  • Qwen2.5-Coder-7B pass@5 reaches 1.31%, up from pass@1 = 0.56%, so sampling diversity alone does not overcome the executability bottleneck.The paper attributes the main issue to failure to ground intent to the correct interface, rather than the decoding strategy.Source: C. RQ3: Reliability and Bias Analysis · pp. 9–10

VI. DISCUSSION AND THREATS TO VALIDITY

The discussion frames service-call generation as an interface-grounding problem and recommends separating retrieval, selection, and generation. It also qualifies virtual-execution metrics and the benchmark’s scope.

VI. DISCUSSION AND THREATS TO VALIDITY

  • The large RAG–Oracle gap motivates a retrieve–select–generate pipeline with API recommendation or reranking as an independently optimizable component.Mock evaluates host, endpoint, and method matching, while ParamF1 separately reports parameter alignment.Source: §VI · p. 10
  • In a bounded real-call check, Mock-positive requests return 2xx in 91.3% of effective attempts (95/104), supporting positive predictive validity under reachable-service conditions, not general equivalence to real execution.Five of nine eligible Mock-negative requests also return 2xx because they target different but accepting endpoints on the same host.Source: §VI · p. 10
  • Construct validity is limited by virtual execution, and external validity is limited to Python and the RapidAPI ecosystem.The authors identify JavaScript, other API ecosystems, and larger scales as future extensions.Source: §VI · p. 10

VII. CONCLUSION

SCCGBench evaluates real-world service-call code with traceable RapidAPI functions, redacted intents, and execution-grounded metrics. The paper concludes that documentation retrieval and endpoint grounding are central bottlenecks, and provides the benchmark and code.

VII. CONCLUSION

  • The conclusion reports 3,135 traceable, decontamination-filtered RapidAPI functions and finds that intent-only Mock is at most 3.8%, versus about 89–90% for most open-source models under Oracle.BM25 Endpoint@10 is only 34% and dense retrieval is about 54%; reliability analyses find no evidence that redacted intents or older samples create an unfair execution advantage.Source: §VII · p. 10

DATA AND CODE AVAILABILITY

  • The SCCGBench dataset, construction pipeline, and evaluation code are released with train/validation/test splits, redacted-intent scripts, virtual execution, dense retrieval, and pass@k experiments.Third-party credentials are redacted.Source: DATA AND CODE AVAILABILITY · p. 10

Related papers

Loading 2610.12920v1…