PaperlayerView exact revision on arXiv

Source-linked AI summary

Testing RESTful APIs: A Survey

Amid Golmohammadi, Man Zhang, Andrea Arcuri

arXiv:2212.14604v1cs.SE

TL;DR

Testing RESTful APIs is challenging because it involves network messages and may require database setup and external-service mocking. This paper synthesizes 92 articles, reporting broad research activity and potential usefulness for practitioners while noting unresolved challenges.

  • Problem

    Testing RESTful APIs requires network messages and may also require setting up database data and mocking interactions with external services.

    Source: §1 · p. 1

  • Method

    The authors survey and categorize 92 scientific articles on RESTful API testing, using a literature review and snowballing to gather papers.

    Source: §1 · p. 1, §4.4 · p. 7

  • Results

    Research interest has increased since 2017, with black- and white-box techniques, open-source prototypes, evaluations on real APIs that found faults, and challenges both addressed and unresolved.

    Source: §10 · p. 30

  • Takeaways & Limitations

    The survey reports that research prototypes evaluated on real APIs and finding real faults indicate potential usefulness for industry practitioners.

    Source: §10 · p. 30

    Relevant articles may have been missed despite database searches and forward and backward snowballing, and manual selection leaves room for human error.

    Source: §9 · p. 29

Abstract

from arXiv · show

In industry, RESTful APIs are widely used to build modern Cloud Applications. Testing them is challenging, because not only they rely on network communications, but also they deal with external services like databases. Therefore, there has been a large amount of research sprout in recent years on how to automatically verify this kind of web services. In this paper, we review the current state-of-the-art on testing RESTful APIs, based on an analysis of 92 scientific articles. This review categories and summarizes the existing scientific work on this topic, and discusses the current challenges in the verification of RESTful APIs.

1 Introduction

The introduction frames RESTful APIs as common in enterprise backends and public services, then outlines a survey organized around publication trends, testing approaches, tools, and challenges.

1 Introduction

  • REST APIs are common choices for web and enterprise backends, including in microservice architectures.The introduction names Netflix, Uber, Airbnb, eBay, Amazon, Twitter, and Nike as examples of companies using them.Source: §1 · p. 1
  • RESTful APIs are also widely available as public services; ProgrammableWeb lists more than 24,000 Web APIs, with RESTful APIs the most common.Google, Amazon, Twitter, Reddit, and LinkedIn are cited as providers of online REST APIs.Source: §1 · p. 1
  • The publication-status questions ask about yearly paper counts, publication venues, and the frequency of contribution types.These are RQ1–RQ3.Source: §1 · p. 1
  • The remaining questions cover testing metrics and techniques, automated testing kinds and evaluation artifacts, research and non-research tools, prototype features, and research challenges.They span RQ4–RQ12.Source: §1 · p. 2
  • The survey groups its 12 research questions into publication status, automated testing approaches, available tools, and addressed or open challenges.The paper’s roadmap assigns these topics to Sections 5–8.Source: §1 · pp. 1–2

2 Background

The background explains REST’s resource-oriented style and HTTP-based interactions, then introduces OpenAPI as a schema format that supports understanding and testing APIs.

2.1 Hypertext Transfer Protocol and REST

  • REST is an architectural style for accessing and manipulating resources over HTTP, not a protocol or a prescribed implementation pattern.Its design constraints include client–server separation, statelessness, caching, a uniform interface, and multilayeredness; code-on-demand is optional.Source: §2.1 · p. 2
  • REST APIs represent resource state to callers, while HTTP verbs such as GET, POST, PUT, PATCH, and DELETE specify resource actions.GET retrieves resources, POST adds one, PUT replaces, PATCH modifies, and DELETE removes one.Source: §2.1 · p. 3

2.2 OpenAPI

  • OpenAPI provides a language-independent interface description that people and machines can use without source-code access or network-traffic analysis.Automated tools can use specifications for documentation, code generation, and testing.Source: §2.2 · p. 3
  • An OpenAPI document describes endpoint URIs, HTTP verbs, parameters, and possible responses, and OpenAPI is widely used to define REST API schemas for fuzzing.Other schema-description languages include RAML and API Blueprint, but the paper says they are less commonly used, especially for fuzzing.Source: §2.2 · p. 3

3 Related Work

The related-work section distinguishes this survey from earlier web-service and REST API surveys by emphasizing its broader coverage of recent research.

3 Related Work

  • Earlier surveys on service-oriented architecture testing date from 2009 and 2013 and predate the growth in REST API testing research activity from 2017 onward.The authors use this age difference to motivate an updated account of the field.Source: §3 · p. 4
  • A 2022 survey of RESTful API testing analyzed 16 papers and asked about unit-test generation challenges, code coverage, available solutions, and authentication-enabled testing.The present survey positions its own coverage as broader than these four questions.Source: §3 · pp. 4–5
  • The survey covers 92 articles, compared with 16 in the 2022 REST API survey, and addresses more questions across testing metrics, testing kinds, fuzzers, and challenges.All 16 articles covered by the earlier survey are included in this analysis.Source: §3 · p. 5

4 Research Method

The authors conduct a systematic literature review structured around 12 questions, using database searches, selection criteria, snowballing, and checked data extraction.

4.1 Research Questions

  • The review’s 12 research questions span publication status, automated testing approaches, available tools, and addressed or open challenges.They cover publication trends and venues, testing metrics and techniques, evaluation artifacts, tool support, and research challenges.Source: §4.1 · p. 5

4.2 Database and Search Queries

  • Searches used seven databases, queries tailored to REST API testing, and a March 1, 2022 cutoff, with repository search limitations affecting query precision.For example, ScienceDirect limited Boolean operators and did not support wildcards; Springer lacked advanced section-based searches.Source: §4.2 · pp. 5–6

4.3 Paper Selection Criteria

  • Eligible papers were English-language studies related to testing REST APIs, including broader web- or cloud-service studies; theses and existing surveys were excluded.A study did not need to focus exclusively on REST to qualify.Source: §4.3 · p. 6, Table 2 · p. 7

4.4 Snowballing

  • Forward and backward snowballing added 52 papers to the 40 found in the initial search, yielding 92 papers in total.The authors cite post-cutoff publications, sources outside the initial search, and potentially missed search results among the reasons for the larger snowballing yield.Source: §4.4 · p. 7

4.5 Data Extraction

  • Two authors extracted data using predefined categories that were refined during paper review, and another author double-checked the results.Discrepancies were discussed and settled by the third author.Source: §4.5 · p. 7

5 Status of Publications in REST API Testing

The survey finds a sharp rise in REST API testing publications since 2017, spread across many venues. Most papers propose automated approaches, while others analyze, compare, or demonstrate tools and propose future work.

5.1 RQ1: How many papers were published per year?

  • Since 2017, REST API testing publications increased sharply, reaching 24 papers in 2021; the 2022 count covers only half the year.Before 2017, yearly counts fluctuated between 0 and 2.Source: §5.1 · p. 8

5.2 RQ2: In which venues were the papers published?

  • The 92 selected studies appeared in 63 venues, including 48 conference venues, 14 journals, and one open-access repository.There were 66 conference papers and 20 journal articles; ICSE and ICST had 7 and 6 papers, respectively, and TOSEM was the most frequent journal.Source: §5.2 · p. 9

5.3 RQ3: What contributions to testing RESTful APIs have been made by the papers in this area, and what are their frequencies?

  • Automated-approach papers include tools, algorithms, frameworks, and methods; examples include EvoMaster extensions and ARTE's realistic-input generation integrated into RESTest.Analysis contributions include coverage metrics, fault taxonomies, security metrics, and a proposed mocking framework; empirical studies compare existing approaches, while tool demonstrations and proposals without conducted studies are separate categories.Source: §5.3 · pp. 9–10
  • Five contribution categories emerged; 66% of papers proposed a new automated approach or extension, while 18% offered a new analysis approach and potential solution.Empirical studies of automated approaches made up 3% of papers.Source: §5.3 · p. 10

6 Approaches to Testing REST API

The survey frames automated REST API testing as using API schemas and test criteria to generate request sequences and evaluate system behavior. Tests may also need to account for authentication, databases, and external services.

6 Approaches to Testing REST API

  • The survey aims to identify existing automated REST API testing approaches and organizes its review around metrics, techniques, kinds of testing, and evaluation artifacts.A test is a sequence of HTTP requests with concrete parameter values; schemas can guide test generation and metrics, while authentication and external services are also testing concerns.Source: §6 · p. 11

6.1 RQ4: What metrics are used to evaluate the effectiveness of the testing?

The survey groups evaluation metrics into coverage criteria, fault detection, and performance. It describes schema- and code-based coverage, several fault-oracle types, and performance measures for REST API tests.

6.1 RQ4: What metrics are used to evaluate the effectiveness of the testing?

  • The survey classifies REST API testing metrics as coverage criteria, fault detection, and performance measures.It reports metric categories and their frequencies in Figures 5 and 6.Source: §6.1 · p. 11

(1) schema related coverage

  • Schema-based coverage measures whether tests exercise elements such as paths, operations, input parameters, status codes, and responses.Martin-Lopez et al.'s 10 schema-based metrics include parameter, content-type, and response-body-property coverage.Source: (1) schema related coverage · pp. 13–14

(2) code coverage

  • Code coverage is used to evaluate both black-box and white-box REST API testing, but automated runtime collection depends on implementation access and language-specific instrumentation.Pythia requires a pre-manual step to extract code information; EvoMaster supports runtime coverage for JVM and JavaScript.Source: (2) code coverage · p. 14

(3) other metrics

  • Other metrics include grammar, dependency, model, graph, and security measures, as well as valid-call and error-type coverage.Examples include RESTler's grammar coverage and model-based state or transition coverage.Source: (3) other metrics · p. 14

(1) service error

  • HTTP 5xx responses serve as potential fault indicators, but a 500 response can also be correct when an external dependency such as a database is unavailable.The survey notes that 500 may act as a fault oracle, but not every such response reflects a software fault.Source: (1) service error · p. 15

(2) violation of schema

  • A response that differs from its API-schema specification can be treated as a fault, with tools using OpenAPI-based checks or reported inconsistency counts.Examples include RestTestGen, QuickRest, and EvoMaster.Source: (2) violation of schema · p. 15

(3) violation of defined rules

  • Rule-based oracles test security, behavior, properties, regression, and resource relationships, including idempotency, hypermedia links, and metamorphic relations.Security checks can flag access to a resource after deletion; regression approaches compare behavior across API versions.Source: (3) violation of defined rules · pp. 15–16

6.1.3 Performance Metrics

  • Performance metrics include average response time, latency, slow responses, and request amplification, sometimes assessed against configured thresholds.Some approaches measure response time across request-generation strategies or send requests from different geographic locations.Source: §6.1.3 · p. 16

6.2 RQ5: What techniques are used for automatically testing RESTful APIs?

The survey finds that black-box testing dominates automated REST API testing, while white-box work mainly uses search-based methods. It also reviews property-, model-, and graph-based techniques and other applications.

6.2.2 Search-based Testing

  • Search-based testing reformulates test generation as optimization; EvoMaster supports black- and white-box modes, with white-box heuristics using instrumentation, testability transformations, and SQL handling.MIO is EvoMaster's default white-box algorithm, with extensions including adaptive hypermutation, smart sampling, and resource-based techniques.Source: §6.2.2 · p. 18

6.2.3 Property-based Testing

  • Property-based approaches test schema conformity, resource states, connectedness, and semantic or metamorphic properties of REST APIs.Some approaches derive properties from documented responses or generate finite-state models of resource changes.Source: §6.2.3 · p. 19

6.2.4 Model-based Testing

  • Model-based testing uses models of API states or behavior to generate tests or check whether observed behavior conforms to a model.Examples include ε-NFA models for hypermedia behavior, colored Petri Nets for architecture constraints, and OpenAPI-derived test-suite models.Source: §6.2.4 · p. 19

6.2.5 Others

  • Other techniques use graphs to generate tests, record service interactions for regression testing, compare API versions, or analyze impacts of migration to microservices.Graph traversal such as breadth-first search can generate tests from representations of API structures and relationships.Source: §6.2.5 · pp. 19–20

6.2.1 Black-box and White-box Testing

  • 73% of surveyed automated REST API testing approaches are black-box; white-box approaches mainly use search-based techniques, while property- and model-based testing are widely employed in black-box work.A hybrid approach combining RESTest and EvoMaster performed best on four selected APIs compared with the isolated black-box and white-box solutions.Source: §6.2.1 · pp. 17–18, §6.2.5 · p. 20

6.3 RQ6: What kind of testing has been automated for RESTful APIs?

The review identifies eight types of testing automated for RESTful APIs, with system testing the most common. The studies also address security, component-level testing, regression, robustness, architecture conformance, and acceptance criteria.

6.3 RQ6: What kind of testing has been automated for RESTful APIs?

  • Security testing includes a semi-automatic approach that uses OpenAPI specifications to identify details about security flaws.The review also notes reported API incidents involving vulnerabilities, bot/scraping, denial of service, and authentication problems.Source: §6.3 · p. 20
  • Integration testing checks whether component collections work as expected; one cited approach automates it with a focus on hypermedia testing.Unit testing is also represented by MockRest, a mock framework for consistent responses when the real REST API is down.Source: §6.3 · p. 20
  • Two papers address specific REST API regression-testing problems, and fuzzer-generated tests can also be reused for regression testing.EvoMaster automates system testing, and its generated test cases can be used for regression testing.Source: §6.3 · p. 21
  • Robustness testing includes checking whether REST services handle invalid inputs using constraint information from an OpenAPI specification.The review cites one work applying this approach.Source: §6.3 · p. 21
  • Architecture design testing is represented by one paper that checks API conformance to REST architecture constraints.The study aims to enhance system design quality.Source: §6.3 · p. 21
  • Acceptance testing appears in one paper, which applies it to web service choreographies.It assesses whether a system satisfies acceptance criteria and gives the client the option to accept it.Source: §6.3 · p. 21
  • 72 of 92 papers focus on system testing, which is more frequent than testing requiring component or source-code access.Some papers combine system testing with other types.Source: §6.3 · pp. 20–21

6.4 RQ7: What kind of artifacts are used for conducting empirical evaluations?

Empirical evaluations use varied API case studies, most commonly local open-source projects and APIs available on the internet. Some studies use artificial artifacts or no case studies, while internet-based experiments may be difficult to repeat.

6.4 RQ7: What kind of artifacts are used for conducting empirical evaluations?

  • 16% of papers use artificial APIs; local closed-source APIs account for 7%, and locally run mixes of open- and closed-source APIs account for 4%.Local closed-source studies typically require academia-industry collaboration.Source: §6.4 · p. 22
  • 28% of case studies use internet APIs, where evaluations typically use black-box techniques and may not be repeatable if APIs change or disappear.The review gives RestTestGen's use of 87 internet APIs as an example.Source: §6.4 · p. 22
  • 14% of papers conduct no empirical evaluations and use no case studies.This includes proposal-or-idea papers as well as other contribution types.Source: §6.4 · p. 23
  • 30% of papers use local open-source projects, while 28% use APIs available on the internet as case studies.The review identifies these as the two most common groups.Source: §6.4 · pp. 22–23

7 Tools for Testing RESTful APIs

The survey identifies the research and non-research tools used for REST API testing and compares the capabilities reported for open-source research prototypes.

7.1 RQ8: Which research tools are open-source?

  • 16 open-source research tools were identified; EvoMaster appeared in 19 papers, while RESTest and RESTler each appeared in 8.The counts include papers presenting or extending tools and papers using them for comparison.Source: §7.1 · pp. 23–24
  • Java and Python were each used by 6 open-source tools, making them the most common implementation languages.The survey counts programming languages used by the open-source research tools.Source: §7.1 · p. 24

7.2 RQ9: Which non-research tools are used/compared?

  • The non-research tools include libraries and toolkits; Postman tests can seed fuzzing, while Fuzz-lightyear maintains state between requests.Other listed toolkits include Tcases for black-box model-based testing based on OpenAPI and Dredd for sample-value-based testing and response validation.Source: §7.2 · p. 24
  • The survey identified 20 non-research tools, and RestAssured was the most used or compared, appearing in 12 papers.RestAssured facilitates HTTP calls to REST APIs and response validation, and is mainly used in tests generated by fuzzers.Source: §7.2 · p. 24

7.3 RQ10: Which features are supported by the research prototypes?

  • The feature inventory was based on published tool papers and online documentation, but some feature information was unclear or missing.The survey notes that documentation sometimes says a tool generates test cases without specifying their language or format.Source: §7.3 · p. 25
  • OpenAPI was supported as an API schema format by 12 of the 16 open-source tools.Tools use the specification to identify REST API endpoints.Source: §7.3 · p. 25
  • Test cases were the most common prototype output, with JUnit the most employed test format.EvoMaster also supports tests in C# and JavaScript, including xUnit and Jest.Source: §7.3 · p. 25
  • Only EvoMaster supports clearing the database between tests and automated code instrumentation; 8 tools support testing authenticated or authorized APIs.Database clearing resets state so test cases can be independent; EvoMaster uses instrumentation to collect code coverage during test generation.Source: §7.3 · p. 25

8 Research Challenges

The survey separates challenges already addressed from open research needs. Addressed work commonly targets resource and inter-parameter dependencies and automated oracles, while open work emphasizes tool support, broader evaluation, and additional testing types.

8.1 RQ11: Which research challenges are addressed?

  • Resource dependencies affect API-call sequences, while inter-parameter dependencies constrain which input combinations form valid calls.One approach models dependency patterns in new test cases; another specifies inter-parameter dependencies with IDL.Source: §8.1 · p. 27
  • Oracle research extends fault detection beyond HTTP 500 statuses and schema mismatches, including security rules and metamorphic relations.Metamorphic testing checks whether multiple executions satisfy a relation when a single execution’s expected outcome is unclear.Source: §8.1 · p. 27
  • Other addressed challenges include database-state handling, mocking unavailable services, and black-box coverage criteria for remote APIs.The survey reports database state can improve code coverage and fault finding; mocking aims to provide reliable responses, and remote APIs cannot use code-coverage metrics.Source: §8.1 · p. 27
  • Testability transformations modify SUT source code to improve search fitness, including by detecting query parameters absent from an OpenAPI schema.The cited REST API approaches use these transformations during the search.Source: §8.1 · p. 27
  • Resource and inter-parameter dependencies, along with automated oracles, are among the most common addressed challenges.Resource dependency was addressed in 8 articles, and inter-parameter dependency in 6.Source: §8.1 · pp. 26–28

8.2 RQ12: Which open research challenges are identified?

  • Tool support is the most common open objective, mentioned in 25 papers, with prototypes needing broader applicability and practitioner usability.Future improvements include support for OpenAPI v3 and authenticated endpoints.Source: §8.2 · p. 28
  • More REST API case studies are the second most frequent open challenge, appearing in 13 papers; networked evaluations are time-consuming and API setup can be costly.Internet APIs can change and lack code-coverage access, while local experiments require setup such as databases and authentication.Source: §8.2 · p. 28
  • Open work also includes security and other non-functional testing, external-service handling, database support beyond SQL, and resource relations for non-hierarchical schemas.Security testing is cited in 10 papers; external services can increase flakiness or require running many services, while non-hierarchical schemas make endpoint relations harder to infer.Source: §8.2 · p. 29

9 Threats to Validity

The survey’s validity is bounded by possible omissions and human error in selecting and extracting studies. The authors describe screening and cross-checking steps intended to reduce these risks, but acknowledge that different selections or interpretations remain possible.

Threats to Validity

  • Relevant studies may have been missed because database search results and publication coverage are imperfect, despite forward and backward snowballing.Manual paper selection also leaves room for human mistakes, and other researchers might have made different selections.Source: §9 · p. 29
  • Article data extraction required manual effort and expertise and could be affected by human mistakes.At least two authors checked each selected paper to reduce this risk.Source: §9 · p. 29
  • The authors analyzed 16 of their own 92 surveyed articles but acknowledge possible misinterpretation of other researchers’ papers.They state confidence in their analysis of their own work while retaining this risk for other articles.Source: §9 · pp. 29–30

10 Conclusion

The survey synthesizes 92 articles on RESTful API testing, a research area whose activity has grown since 2017. It finds varied black-box and white-box techniques, released prototypes, and empirical studies reporting real faults, while noting that challenges remain.

10 Conclusion

  • The survey analyzes 92 articles and reports an exponential increase in research interest beginning in 2017.It covers both black-box and white-box techniques and identifies challenges that remain unresolved.Source: §10 · p. 30
  • Several prototypes have been released as open-source projects, and empirical studies on real-world APIs have found real faults.The survey says this research has potential usefulness for practitioners in industry.Source: §10 · p. 30
  • The survey provides a detailed snapshot of the current research state and a starting point for new research directions.The paper characterizes the field as growing.Source: §10 · p. 30

Related papers

Loading 2212.14604v1…