PaperlayerView exact revision on arXiv

Source-linked AI summary

RESTCov: A Tool for Structural Coverage Analysis of REST APIs

Tolgahan Bardakci, Serge Demeyer

arXiv:2608.28114v1cs.SE

TL;DR

RESTCov addresses the lack of structural coverage visibility for REST APIs when implementations are accessible only through black-box execution. It maps OpenAPI specifications to observed HTTP traffic, reports multiple coverage dimensions, and provides machine-readable and HTML outputs for examining gaps and mismatches.

  • Problem

    REST API testers may lack visibility into which documented API elements are exercised when source-code instrumentation is unavailable or impractical.

  • Method

    RESTCov parses an OpenAPI specification and observed request/response logs, maps traffic to documented API elements, and reports structural coverage across several dimensions.

  • Results

    RESTCov processed the OpenAPI specifications used across five production-API studies and Petstore successfully, while some tested coverage tools did not.

  • Takeaways & Limitations

    Machine-readable JSON and human-readable HTML outputs support test-suite assessment, coverage-gap inspection, and unmatched-request diagnosis without implementation access.

  • Takeaways & Limitations

    The paper demonstrates RESTCov on one API and does not present a multi-API evaluation; its coverage also does not verify functional correctness.

Abstract

from arXiv · show

REST APIs are widely used in modern software systems, but developers and testers often lack visibility into which parts of an API specification are exercised by a test suite. Traditional coverage analysis usually relies on source-code instrumentation, which is impractical for REST APIs that are distributed, externally maintained, and hence accessible only through black-box execution. This paper presents RESTCov, a lightweight tool that computes structural REST API coverage from an OpenAPI specification and observed HTTP request/response logs, reporting coverage across paths, operations, parameters, media types, status codes, and status classes. RESTCov produces both machine-readable results and a human-readable HTML report, helping users inspect coverage gaps, diagnose specification-log mismatches, and evaluate REST API test suites without requiring access to the implementation. Screencast: https://youtu.be/mNz2P43OyUc Repository: https://github.com/2tolgahan2/RESTCov

I. INTRODUCTION

RESTCov addresses the need for black-box visibility into which documented REST API elements are exercised when specifications and traffic logs are available but source-code instrumentation is impractical. It computes structural coverage across multiple API dimensions and provides outputs for diagnosing mismatches and inspecting gaps.

  • RESTCov produces machine-readable results and a human-readable HTML report for inspecting coverage gaps and unmatched requests.
  • RESTCov computes structural REST API coverage from an OpenAPI specification and observed HTTP traffic without source-code instrumentation.
  • Coverage spans paths, operations, parameters, media types, status codes, and status classes.
  • Unmatched request samples help diagnose specification-log mismatches, including traffic that may fall outside the documented API.
  • The reusable artifact includes source code, example inputs, documentation, and expected outputs for practical use and future experimentation.

II. RELATED WORK AND POSITIONING

Prior REST API testing research primarily develops methods for generating or executing tests from API specifications. RESTCov is positioned within coverage analysis, making API-level exercise visible without source-code access.

  • RESTTESTGEN and RESTest generate black-box REST API test cases from API specifications.
  • Other approaches include LLM-based test amplification, stateful fuzzing, system-level test generation, property-based generation, and schema-aware or model-based testing.
  • These related tools primarily focus on generating or executing REST API tests rather than reporting structural coverage.

B. REST API Coverage Analysis

REST API coverage analysis measures how thoroughly tests exercise API specifications without requiring source-code access. RESTCov distinguishes documented coverage from traffic outside the specification by explicitly collecting unmatched requests.

  • Restats and RESTCov share the goal of making API-level coverage visible without source-code access.
  • RESTCov collects traffic outside the documented specification into one explicit unmatched-request list shown in both JSON and HTML outputs.
  • The unmatched-request list exposes specification-log mismatches caused by specification-implementation drift or client-side artifacts such as malformed or encoded paths.
  • RESTCov reports whether documented specification elements were observed, but does not verify functional correctness or testing adequacy.

B. Inputs

RESTCov parses an OpenAPI specification and paired request/response logs, reconstructs observed traffic, and matches requests to documented operations. It then reports coverage across structural dimensions while preserving unmatched cases caused by strict matching.

  • B. Inputs: RESTCov operates on an OpenAPI specification and paired request/response log files, supporting different OpenAPI versions.
  • B. Inputs: The tool extracts documented paths, operations, parameters, response codes, and media types, then reconstructs corresponding fields from observed traffic.
  • B. Inputs: Documented path templates are compiled into regular expressions with path parameters replaced by wildcard segments before matching requests.
  • B. Inputs: RESTCov requires exact path matching for casing, trailing slashes, and encoding, while comparing HTTP methods case-insensitively.
  • B. Inputs: Coverage dimensions include paths, method-path operations, parameters, request and response content types, individual status codes, and status classes.
  • B. Inputs: Reporting each dimension reveals gaps that remain when a test suite exercises many operations but misses parameters, media types, or response outcomes.

E. Outputs

RESTCov provides both machine-readable and human-readable coverage outputs, while unmatched request samples help diagnose specification-log mismatches.

  • RESTCov writes computed metrics and observations to coverage.json for further analysis, test-suite comparison, or CI/CD integration.
  • RESTCov presents the same coverage information in report.html, an inspectable HTML report that includes unmatched request samples.
  • Unmatched samples can indicate specification-implementation drift, although malformed or encoded paths may instead reflect client-side artifacts.

IV. DEMONSTRATION

The Petstore demonstration shows RESTCov processing an OpenAPI specification and recorded HTTP logs, aggregating matched observations into coverage metrics and reports. The observed traffic covered documented paths, operations, and parameters, but not all media types or response outcomes.

  • Demonstration setup: The demonstration uses the Petstore API with recorded request/response logs to show RESTCov processing inputs and producing coverage results.
  • Demonstration scope: RESTCov has also processed five production APIs, including Google Drive and Spotify, although this paper presents no multi-API evaluation.
  • Command-line execution: RESTCov runs from the command line by specifying the OpenAPI specification, log directory, and output directory.
  • Processing pipeline: After execution, RESTCov parses specifications, reconstructs HTTP interactions, matches requests to documented operations when possible, and writes JSON and HTML reports.
  • Coverage results: Table I reports distinct documented-element totals for paths and operations, while other dimensions aggregate counts across operations.
  • Coverage results: All documented paths, operations, and parameters were covered, but request and response content types, status codes, and status classes had lower coverage.

B. Output Interpretation

RESTCov’s JSON and HTML outputs distinguish matched coverage metrics from unmatched traffic. The report makes it possible to inspect mismatches and determine whether they reflect documentation problems or client-side path artifacts.

  • JSON interpretation: The total requests field counts only observed requests successfully matched to documented operations; unmatched requests are reported separately.
  • JSON interpretation: The generated JSON report includes selected coverage metrics and an unmatched_samples list containing requests excluded from the matched-request count.
  • HTML interpretation: The HTML report summarizes coverage by dimension and lists unmatched samples that may represent undocumented endpoints, malformed paths, encoded paths, or other mismatches.
  • Mismatch diagnosis: The unmatched path /pet%2F%2FuploadImage contains a double-encoded slash and is likely a client-side artifact rather than an undocumented or uncovered operation.

C. Insights from the Report

RESTCov’s report supports diagnosis beyond a single coverage number by connecting coverage dimensions and unmatched traffic to testing and maintenance decisions. The Petstore demonstration shows gaps between exercised endpoints and exercised response behavior alongside specification-log mismatches.

  • Interpreting coverage gaps: Low parameter coverage may indicate tests exercise only default or minimal requests, while low status code coverage may indicate missed error-handling behavior.
  • Interpreting mismatches: Unmatched request samples help users decide whether to add tests, improve the specification, or inspect the logging format.
  • Maintenance implications: By surfacing mismatches from traffic, RESTCov supports ongoing API maintenance and evolution rather than only a one-time release coverage check.
  • Demonstration insight: RESTCov surfaces differences between exercised endpoints and exercised response behavior, while exposing specification-log mismatches through unmatched samples.

V. AVAILABILITY AND REUSE

RESTCov is distributed as a reusable GitHub artifact with source code, documentation, examples, expected outputs, and HTML-report screenshots. Users can apply it to another REST API by supplying an OpenAPI specification, HTTP logs, and output directory.

  • The GitHub repository includes RESTCov’s source code, documentation, example inputs, expected outputs, and HTML-report screenshots.
  • The root directory contains the analysis script restcov.py, dependency file requirements.txt, README.md documentation, and LICENSE.
  • The Petstore example contains an OpenAPI specification, recorded HTTP request/response logs, and expected outputs.
  • The screenshots directory provides views of RESTCov’s HTML report.
  • To reuse RESTCov with another REST API, users provide an OpenAPI specification, a request/response log directory, and an output directory.The README supplies dependency and execution instructions.

VI. LIMITATIONS

RESTCov’s coverage results have important scope boundaries: the paper demonstrates the tool on one API, measures observed specification-element coverage rather than behavioral correctness, and uses paired request/response text files.

  • The paper demonstrates RESTCov using a single Petstore API and does not present a multi-API evaluation.The tool was also used across five production APIs elsewhere, but those studies are not evaluated here.
  • RESTCov reports whether documented specification elements appeared in traffic, not whether endpoint behavior was correctly exercised.Its coverage measure should therefore be interpreted as observed structural coverage.
  • The current version stores each HTTP interaction as paired request/response text files, which may become cumbersome for large-scale log analysis.Consolidated formats such as HAR or JSONL are future work rather than current requirements.
  • JSON output supports automated analysis, while the HTML report helps inspect coverage gaps and unmatched requests.Together, these outputs support test-suite assessment without access to the service implementation.
Loading 2608.28114v1…