Automated Test Generation for REST APIs: No Time to Rest Yet

1 INTRODUCTION

• The study identifies four contributions: systematic tool selection, benchmark-code analysis, in-depth failure analysis, and concrete future research directions.The authors describe these as extensions beyond concurrent comparison efforts, which they characterize as limited in scope.Source: §1 · p. 1
• The benchmark consists of 20 Java/Kotlin open-source services selected from related work and GitHub, with excessive reliance on external resources excluded.The study selected tools that used commonly used REST API standards, generated actual test cases, and could handle the benchmark; the resulting set contains one white-box and nine black-box tools.Source: §1 · pp. 1–2
• The evaluation applies 10 selected tools to 20 services and measures code coverage and several kinds of fault-detection ability.Measures include lines, branches, and methods exercised, plus generic errors, unique failure points, and unique library failure points.Source: §1 · p. 2
• All tools achieved relatively low line and branch coverage on many benchmarks; weak handling of input constraints and endpoint dependencies limits effectiveness.The study identifies dependency-aware testing as key to effective REST API testing, while existing tools omit dependencies or infer them with weak heuristics.Source: §1 · p. 2
• The study recommends using specification and server-log information to improve generated input parameters and exploring static analysis or natural-language processing to detect dependencies.A proof-of-concept evaluation reports the feasibility of these suggested directions.Source: §1 · p. 2
• The paper provides an artifact containing tools and benchmarks to support replication and further research.Its contributions also include analysis of tool strengths and weaknesses, suggestions for improvement, and discussion of future research implications.Source: §1 · p. 2

2.1 REST APIs

• A REST API operation combines a resource endpoint and HTTP method; requests may also carry headers and a body, and services return status-coded responses.The paper notes that 5xx responses indicate server errors and that studies often count 500 responses as bugs.Source: §2.1 · p. 2

2.2 OpenAPI Specifications

• OpenAPI is a standard format for specifying service operations, input and output data types, and possible return codes.Figure 1's example describes a GET operation with a required productName path parameter and a JSON response.Source: §2.2 · p. 2
• A producer-consumer dependency exists when one operation returns data needed as input to another operation.The example links GET /products/{productName} to a DELETE operation that needs a constraint id returned by the GET request.Source: §2.2 · p. 3

3.1 Tools Selection

• The search identified 19 publicly available tools, then excluded tools that did not work or could not run on the 20-service benchmark, leaving 10 popular tools.The search covered REST API research published since 2011, selecting techniques that use established API standards and produce actual test cases.Source: §3.1 · p. 3

3.2 Tools Description

• EvoMaster's white-box mode uses source code and an evolutionary algorithm to target uncovered branches, while its black-box mode generates random requests.The white-box tool also samples or mutates tests and can account for operation dependencies when creating stateful requests; both modes use an oracle for 5xx failures.Source: §3.2 · p. 3
• RESTler infers producer-consumer dependencies and searches for stateful request sequences, while RestTestGen matches input and output field names to infer dependencies and order operations.RESTler uses execution feedback to prune invalid sequences; RestTestGen uses retrieved response values, random/default/example values, and nominal or error tests.Source: §3.2 · pp. 3–4
• RESTest uses specified inter-parameter dependencies and constraint solving for nominal tests, while Schemathesis performs property-based negative testing and can follow specification links for stateful tests.RESTest also generates faulty tests; Schemathesis uses multiple response-schema oracles and can use user-provided examples.Source: §3.2 · p. 4
• Dredd validates responses using specification sample and dummy values, while Tcases models input characteristics and uses each-choice testing to generate cases.Tcases can alternatively build an example-based model and generate tests from specification samples.Source: §3.2 · p. 4
• bBOXRT mutates valid inputs into invalid ones for robustness testing, while APIFuzzer generates random requests from the specification and mutates request values.bBOXRT requires manual service-behavior classification and behavior tags; APIFuzzer reports test results in JUnit XML.Source: §3.2 · p. 4

• The study asks how much coverage tools achieve, how many error responses they trigger, and what the findings imply for future techniques.Source: §4 · p. 4
• Coverage is measured through line, branch, and method coverage to assess how much service code the tools exercise.Source: §4 · p. 4
• Fault-finding measures group 500 errors by stack trace, then by topmost method-line failure point, distinguishing service failures from third-party-library failures.Source: §4 · pp. 4–5
• The study assesses tool strengths and weaknesses and discusses implications for developing REST API testing techniques.Source: §4 · p. 5

• The authors selected services from prior benchmarks and GitHub searches, then filtered out candidates lacking OpenAPI specifications, compiling unreliably, or relying heavily on rate-limited external services.Source: §4.1 · p. 5
• The final benchmark contains 20 Java/Kotlin open-source REST services, selected in part to enable evaluation of the white-box tool EvoMasterWB.Source: §4.1 · p. 5
• Table 2 lists each service’s name, total size, Java/Kotlin code size, and number of operations.Source: §4.1 · p. 5

• Experiments ran on Google Cloud e2-standard-4 machines with Ubuntu 20.04, four 2.2GHz Intel-Xeon processors, and 16GB RAM.Source: §4.2 · p. 5
• Tools used recommended configurations; RESTest received inter-parameter dependency specifications, while EvoMasterWB used custom instrumentation and database-injection drivers.Source: §4.2 · p. 5
• Each tool ran for one hour in 10 trials and once for 24 hours; coverage and error data were collected every 10 minutes during the shorter runs.Source: §4.2 · p. 5

Overall coverage achieved

• After one hour, EvoMasterWB achieved less than 53% line, less than 37% branch, and less than 53% method coverage; EvoMasterBB led black-box tools at 45.41%, 28.21%, and 47.17%, respectively.Source: §4.3 · p. 6, Figure 2 · p. 6
• Low coverage reflects difficulty generating constrained parameter values and identifying request dependencies, while mismatches between API specifications and implementations can also hinder testing.Source: §4.3 · p. 6

Coverage increase over time

• Coverage often reached its highest level within 10 minutes, but continued increasing on six services whose inputs were generally simpler and less constrained.Source: Figure 2 · p. 6, §4.3 · p. 7
• On services with domain-specific or constrained parameter values, tools tended to hit a coverage wall relatively soon because they could not generate inputs satisfying those requirements.Source: §4.3 · p. 7

4.4 RQ2: Error Responses Triggered

• EvoMasterWB detected the most unique errors, failure points, and library failure points by a wide margin, followed by Tcases, EvoMasterBB, RESTler, and Schemathesis.EvoMasterWB also had the highest median 500-error count, although its spread was larger.Source: §4.4 · p. 7
• There were 3 to 7 times fewer unique failure points than unique errors, and approximately half of failure points occurred in library methods.Multiple 500 errors could arise at the same program point along different call chains.Source: §4.4 · p. 7
• Library failure points can threaten robustness or security, as unchecked values passed to libraries led to costly gas loss, database connection failures, and parsing errors.Service-code failures usually followed parameter checks, whereas failures outside service code often involved accepted erroneous requests and could be severe.Source: §4.4 · p. 7
• Coverage and 500-error counts had a strong positive correlation of approximately 0.7881, consistent across line, branch, and method coverage.Tools with higher coverage usually also triggered more failures.Source: §4.4 · p. 7
• Varying parameter combinations was particularly effective at triggering failures, and Tcases and Schemathesis outperformed other tools with this strategy.Generating requests with different input types also seemed to help reveal faults.Source: §4.4 · p. 7

4.5.1 Implications for Techniques and Tools Development.

• The authors propose extracting sample values from parameter descriptions and using server logs and specifications to improve generated inputs.They also suggest combinatorial testing for black-box tools and symbolic execution when white-box search cannot derive a useful gradient.Source: §4.5.1 · p. 8
• A prototype using NLP on OpenAPI descriptions and server messages found 8 of 12 manually identified inter-parameter dependencies and useful parameter values.The prototype parsed descriptions and messages for token dependencies; Figure 4 illustrates parsed graphs for two Languagetool parameters.Source: §4.5.1 · p. 8, Figure 4 · p. 9
• More sophisticated dependency inference could account for dynamic responses and differently typed fields while avoiding false links between operations sharing parameters.The authors also suggest textual similarity and machine learning as possible ways to identify producer-consumer relationships.Source: §4.5.1 · p. 9

4.5.2 Analytical Comparison of White-Box and Black-Box Tools.

• EvoMasterWB covered SCS branches using values “male”/“dr” and “female”/“prof” that black-box tools could not generate.Figure 5 shows the relevant code; EvoMasterWB’s evolutionary search uses branch distance to find the inputs.Source: Figure 5 · p. 9, §4.5.2 · p. 9
• Among the tools considered, EvoMasterWB achieved approximately 53% line coverage, 36% branch coverage, and 53% method coverage, outperforming the other tools in the comparison.Source: §4.5.2 · p. 9
• Schemathesis covered a Features-Service line missed by EvoMasterWB by ordering dependent operations and reusing specification sample values.The sequence creates a product, adds configurations, then retrieves them.Source: §4.5.2 · pp. 9–10, Figure 6 · p. 10
• RESTler’s small input dictionary and RestTestGen’s inaccurate dependency heuristics constrain behavioral variety and stateful testing; Dredd’s examples and dummy values omit uncommon behaviors.Other tools’ random-based approaches may also fail to produce valid cases needed for input mutation.Source: §4.5.2 · p. 10
• Black-box tools generally had low coverage, relying heavily on random testing or limited sample data; Schemathesis and EvoMasterBB performed better than the others.EvoMasterBB uses evolutionary request generation, while Schemathesis uses specification examples and reuses values in request sequences.Source: §4.5.2 · p. 10

• Because the evaluation uses 20 RESTful services, its results may not generalize to other services.The benchmark includes diverse services, including ones used in prior evaluations.Source: §4.6 · p. 10
• Running services locally may produce behavior different from real-world deployments.For ERC-20, the study used a private Ethereum network rather than the main network, though the authors judged significant effects unlikely.Source: §4.6 · p. 10
• OpenAPI specifications may be incomplete and limit coverage or bug-finding; almost 99% of over 1,000 specifications checked had some mismatch.The authors considered the benchmark specifications representative of what tools encounter in practice.Source: §4.6 · p. 10
• RESTest required manually added inter-parameter dependency annotations, and specification quality may have affected its performance.The authors say this is an unavoidable threat when using RESTest.Source: §4.6 · p. 10
• EvoMasterWB requires a test driver for each service, and the study team built drivers for 10 services.The authors report this took only a few minutes per service, but creating drivers entails manual effort.Source: §4.6 · p. 10

• Compared with Corradini et al., this study includes six additional tools and evaluates code coverage and unique failures rather than API coverage and status-code presence alone.It also analyzes testing strategies and implications for tool development.Source: §5 · p. 11
• Against Martin-Lopez, Segura, and Ruiz-Cortés [59], the study compares 10 tools on 20 APIs rather than three tools on four APIs, and finds EvoMasterWB ahead of RESTest.The authors attribute the differing outcome partially to the smaller, RESTest-benchmark-only service set in the other study and possible configuration effects.Source: §5 · p. 11
• Hatfield-Dodds and Dygalo [38] focused on Schemathesis, without an in-depth analysis of coverage or errors beyond counting triggered errors.The paper distinguishes that focus from its comparison of existing tools.Source: §5 · p. 11
• The study excludes metamorphic testing because it requires testers to provide low-level details, including problem-specific metamorphic relations.It also excludes approaches that require non-trivial user-provided specifications, formal specifications, or target specific problems.Source: §5 · p. 11
• It excludes general API tools without automated testing, traffic-fuzzing tools requiring recorded usage, and SOAP/WSDL testing techniques.Two tools proposed after the study was finalized were not included in the comparison.Source: §5 · p. 11

6 CONCLUSION AND FUTURE WORK

• The empirical study compared 10 REST API testing techniques on 20 services using code coverage and unique failures.The authors analyzed the techniques’ strengths and weaknesses and discussed research implications.Source: §6 · p. 11
• The experiment infrastructure, data, and results are publicly available.Source: §6 · p. 11
• Future work will use mutation to further evaluate fault detection and investigate better input generation and operation-dependency handling.Planned approaches include extracting values from specifications and logs, symbolic analysis, and NLP-based inference of producer-consumer dependencies.Source: §6 · p. 11