PaperlayerView exact revision on arXiv

Source-linked AI summary

Design, Monitoring, and Testing of Microservices Systems: The Practitioners' Perspective

Muhammad Waseem, Peng Liang, Mojtaba Shahin, Amleto Di Salle, Gastón Márquez

arXiv:2108.03384v2cs.SE

TL;DR

Microservices practice lacks sufficient empirical evidence on how systems are designed, monitored, and tested. This mixed-methods study examines practitioners’ approaches and identifies recurring practices, challenges, and needs for dedicated solutions.

  • Problem

    Empirical evidence is limited on how industry practitioners design, monitor, and test microservices systems, despite the importance of these activities for successful systems.

  • Method

    The study uses mixed methods to investigate practitioners’ architecting activities, quality attributes, and design, monitoring, and testing practices.

  • Results

    Practitioners commonly use established decomposition, architecture, monitoring, and testing practices, while microservices complexity creates challenges across all three activities.

  • Takeaways & Limitations

    Researchers and practitioners should develop dedicated solutions for the reported design, monitoring, and testing challenges.

  • Takeaways & Limitations

    The survey may be limited by insufficiently described constructs and by the answer options available in mostly closed-ended questions.

Abstract

from arXiv · show

Context: Microservices Architecture (MSA) has received significant attention in the software industry. However, little empirical evidence exists on design, monitoring, and testing of microservices systems. Objective: This research aims to gain a deep understanding of how microservices systems are designed, monitored, and tested in the industry. Method: A mixed-methods study was conducted with 106 survey responses and 6 interviews from microservices practitioners. Results: The main findings are: (1) a combination of domain-driven design and business capability is the most used strategy to decompose an application into microservices, (2) over half of the participants used architecture evaluation and architecture implementation when designing microservices systems, (3) API gateway and Backend for frontend patterns are the most used MSA patterns, (4) resource usage and load balancing as monitoring metrics, log management and exception tracking as monitoring practices are widely used, (5) unit and end-to-end testing are the most used testing strategies, and (6) the complexity of microservices systems poses challenges for their design, monitoring, and testing, for which there are no dedicated solutions. Conclusions: Our findings reveal that more research is needed to (1) deal with microservices complexity at the design level, (2) handle security in microservices systems, and (3) address the monitoring and testing challenges through dedicated solutions.

1. Introduction

The study addresses limited empirical knowledge about how industry designs, monitors, and tests microservices systems. It uses mixed quantitative and qualitative evidence to examine practitioners’ practices, challenges, and solutions.

  • Microservices applications consist of independently developable, deployable, and testable business-driven services communicating through an API layer.The architecture follows a share-as-little-as-possible approach and commonly uses separate data storage per microservice.
  • Industry evidence remains limited on how microservices systems are designed, monitored, and tested.These aspects are important for succeeding with microservices systems and require strategies suited to their characteristics.
  • The study combines a survey of 106 practitioners with six interviews to investigate microservices practices in industry.The research aims to provide a comprehensive understanding of design, monitoring, and testing in practice.
  • Domain-driven design combined with business capability is reported as the most effective application-decomposition strategy.The study also identifies security and availability as significant concerns for practitioners.
  • The research contributes quantitative and qualitative evidence, identifies practitioner challenges and solutions, and discusses implications for researchers and practitioners.Its scope covers the design, monitoring, and testing of microservices systems.

2. Background

The background frames microservices as independently operable services and organizes the study around their design, monitoring, and testing. It defines the investigated activities, practices, metrics, strategies, tools, and industry challenges.

  • Microservices systems comprise small, autonomous services that collectively implement an application, unlike monoliths deployed as a single unit.Each service accomplishes a business capability and contains the data needed for independent execution.
  • Design: The design investigation covers application decomposition, implementation strategies, architecting activities, description methods, patterns, quality attributes, skills, and design challenges.The study also examines how design challenges affect monitoring and testing.
  • Design: Architecting activities examined for microservices include architectural analysis, synthesis, evaluation, implementation, and maintenance and evolution.The study focuses on practitioners’ agreement and disagreement about these activities.
  • Monitoring: Monitoring research addresses dynamically evolving service interactions through metrics, practices, tools, challenges, and solutions.The study identifies resource usage and system behavior metrics, alongside practices such as log management, exception tracking, health checks, audit logging, and distributed tracing.
  • Testing: Microservices testing requires extra effort because of polyglot repositories, inter-process communication, service dependencies, cloud infrastructure, and third-party components.The study investigates testing strategies, tools, and required skills because empirical evidence from practitioners has been limited.

3. Research method

The study used a mixed-methods design combining a broad practitioner survey with follow-up interviews to examine microservices design, monitoring, and testing in practice.

  • Study design: The researchers first conducted a survey and then used in-depth interviews to investigate practitioners’ perspectives on microservices systems.The interviews were intended to deepen understanding of key survey findings.
  • Survey: Survey participants were recruited through personal and professional networks, social platforms, and GitHub-related practitioner contacts.The study targeted practitioners whose organizations had adopted or planned to adopt MSA.
  • Data quality: The researchers applied inclusion and exclusion criteria to identify valid responses and removed meaningless, inconsistent, or logically senseless answers.The criteria evaluated participants’ responsibilities, MSA usage, and experience with microservices systems.
  • Sampling: The survey used non-probabilistic convenience and snowball sampling because microservices practitioners were difficult to identify through random selection.Participants could access the survey through personal contacts and social or professional platforms and could invite other eligible practitioners.
  • Survey: The survey used an online questionnaire covering practitioner demographics and microservices design, monitoring, and testing.The instrument contained 39 questions across these areas, including both closed-ended and open-ended formats.
  • Interviews: Six practitioners were recruited for follow-up interviews, with interviews planned to last approximately 40–55 minutes.Interviewees came from IT companies in five countries and were selected to explore key survey findings in greater depth.

3.4. Data analysis

The study analyzed closed-ended survey responses statistically and qualitative responses and interviews through coding and comparison, with subgroup analyses for practitioner and organizational characteristics.

  • Survey analysis: Closed-ended survey responses were analyzed using descriptive statistics, while open-ended responses were analyzed with open coding and constant comparison from Grounded Theory.The analysis linked survey questions, interview questions, methods, and research questions.
  • Interview analysis: Interview transcripts were filtered for relevance and then coded to identify reasons underlying key survey findings.The transcripts were translated, transcribed, reviewed for irrelevant information, and coded after extraneous material was removed.
  • Statistical testing: The researchers tested group differences at a 95% confidence level using independent-sample t-tests, Bonferroni correction, and effect sizes.The tests compared means between selected participant groups and quantified the magnitude of differences.

4. Results

The results section presents findings from the collected survey and interview responses and addresses the study’s research questions.

  • The study reports its mixed-methods results according to the collected responses and the research questions defined earlier.

4.1. Demographics

The survey covered practitioners across diverse countries, roles, organizations, experience levels, and development contexts. Most respondents worked with MSA in organizations that had adopted it, but many had limited microservices experience or no professional training.

  • Participants: 106 survey responses came from 29 countries across five continents, with the largest groups from Pakistan, China, and Germany.The study also included six interviews with practitioners from six organizations in six countries.
  • Participants: Application developers, architects, and DevOps engineers were the most common participant responsibilities.Application developers accounted for 69 respondents, architects for 45, and DevOps engineers for 21; participants could report multiple responsibilities.
  • MSA adoption: 86% of respondents reported that their organizations had adopted MSA for at least some applications.Among 106 respondents, 63 used MSA for specific applications and 28 used it for all applications.
  • Experience and training: 59.4% of respondents had 0–2 years of experience developing microservices systems, while 38.7% had 3–5 years of IT-industry experience.Three respondents reported more than 10 years of microservices experience, which the authors accepted after reviewing their other responses and backgrounds.
  • Experience and training: 74.5% of respondents had received no professional training for developing microservices systems.Only 27 of 106 respondents reported such training, excluding higher education courses from the training definition.

4.2. Design of microservices systems (RQ1)

Practitioners commonly combine domain-driven design with business capability to decompose applications, while favoring high-level, informal or semi-formal architectural descriptions. Architecture evaluation and implementation receive substantial use, but design work remains uneven across organizations and activities.

  • Organizational practice: 36.7% (39 out of 106) reported no dedicated architect or architecture team in their organizations.Most of these respondents came from small companies or small teams that possibly could not afford a software architect.
  • Application decomposition: 42.3% (44 out of 104) used a combination of business capability and DDD to decompose applications into microservices.All interviewees confirmed that this combination reflects industry practice, and five of six were unfamiliar with another decomposition approach.
  • Design depth: 46.7% (49 out of 105) modeled only the high-level design before writing code, compared with 30.5% (32 out of 105) who modeled detail design.Practitioners reported that high-level designs visualize microservices, fundamental functionality, communication or orchestration, and service boundaries.
  • Architecting activities: Four of six interviewees said they focus mainly on solution-space activities because scenarios are easier to evaluate than problem-space designs.One example uses reference architectures and simulated loss of nodes or data centers to evaluate alternatives.
  • Architecting activities: Over half of participants agreed to use Architecture Evaluation and Architectural Implementation when designing microservices systems.Architecture Evaluation was used to assess necessity, benefits, deployment, availability, fault tolerance, scalability, and added complexity.
  • Design description: Practitioners primarily use informal Boxes and Lines or semi-formal UML, while formal ADLs and DSLs are not widespread.Almost 51% frequently used Boxes and Lines, 41.5% frequently used UML, and 84.7% never, sometimes, or rarely used ADLs.

4.3. Monitoring of microservices systems (RQ2)

Monitoring in microservices systems relies mainly on infrastructure-oriented metrics and complementary operational practices. Practitioners also report substantial challenges in collecting data across distributed systems and adapting monitoring to varied technologies.

  • Monitoring metrics: Resource usage, load balancing, and availability are the most frequently used monitoring metrics.Resource usage was reported by 66.9% of participants, load balancing by 53.7%, and availability by 52.8%.
  • Monitoring practices: Log management, exception tracking, and health check API are the most frequently employed monitoring practices.Participants used log management, exception tracking, and health check API more often than distributed tracking and audit logging.
  • Monitoring tools: Jira and Datadog Kafka Dashboard were the most commonly reported monitoring tools.Jira was reported by 39.6% of participants and Datadog Kafka Dashboard by 38.6%; other tools included Spring Boot Actuator, Grafana, and Zipkin.
  • Challenges and solutions: Practitioners recommended monitoring tools and in-house frameworks, but the reported solutions were largely tool-based.The study identified ten recommended tools and one in-house framework, including Prometheus, Elastic Stack, Jaeger, and Dynatrace.
  • Challenges and solutions: The main monitoring challenges involve collecting metrics and logs from containers and performing distributed tracing.Interviewees linked these challenges to communication among many services, nonstandard runtime infrastructure, heterogeneous technologies, and differing log and data formats.

4.4. Testing of microservices systems (RQ3)

Testing practice is dominated by established strategies and tools rather than techniques designed specifically for microservices. Practitioners nevertheless report challenges associated with manual testing, integration, debugging, and the distributed structure of microservices systems.

  • Testing strategies: Unit testing, E2E testing, and integration testing are the most frequently used testing strategies.Unit testing was used very often or often by 63.2% of participants, E2E testing by 55.6%, and integration testing by 59.4%.
  • Testing strategies: Practitioners use E2E testing to assess the correctness of the whole microservices system and understand customer use.Interviewees described the top testing strategies as fundamental across MSA-based, SOA-based, and other systems.
  • Testing tools: JUnit, JMeter, and Mocha are the most commonly used testing tools.JUnit was reported by 51.9% of participants, JMeter by 27.3%, and Mocha by 21.7%.
  • Challenges and solutions: Exactly 50.0% of participants identified creating and implementing manual tests as challenging.Other prominent challenges included integration testing and debugging microservices deployed on container platforms.
  • Challenges and solutions: Testing challenges arise from the number of microservices, multiple connection points, and the knowledge required across services and domains.Practitioners also reported that many challenges had short-term or minimal impact, while less experienced participants rated some challenges as more critical.
  • Challenges and solutions: Practitioners suggested test automation and behavior-driven development as approaches for addressing testing challenges.Automation was proposed for unit, integration, E2E, and consumer-driven contract testing; BDD was suggested for automated and integration testing.

5. Discussion

The study examines the state of practice for designing, testing, and monitoring microservices systems using survey and interview data. The discussion situates the findings within broader comparisons with non-MSA systems and related implications.

  • Study scope: The empirical study investigates the state of practice of designing, testing, and monitoring microservices systems.
  • Study scope: The study combines an online survey of 106 practitioners from 29 countries with interviews of 6 practitioners from four countries.
  • Discussion structure: The discussion covers implications for researchers, comparisons with non-MSA-based systems, and additional discussion of the survey findings.

5.1. Analysis of the results

Practitioners commonly combine domain-driven design with business capability for decomposition, emphasize solution-oriented architecting activities, and rely on informal description methods and several recurring design patterns. Security, availability, performance, and scalability are prominent quality concerns, while pattern selection and architectural activities remain areas for further study.

  • Application decomposition strategies: DDD and business capability are complementary decomposition strategies that practitioners use together to break monolithic applications into microservices.The paper recommends empirical guidelines for applying both strategies, particularly for inexperienced teams.
  • MSA architecting activities: Practitioners are more likely to use Architectural Evaluation and Architectural Implementation when designing microservices systems.Detailed design clarifies microservice functionality and interfaces and enables evaluation of design options before implementation.
  • MSA description methods: Informal and semi-formal methods are preferred over formal approaches for describing microservices architectures.The paper calls for research on the advantages and disadvantages of informal descriptions and on adapting formal methods such as DSLs and ADLs.
  • Quality attributes: Security, availability, performance, and scalability are the most important quality attributes considered in microservices design.Reliability, usability, maintainability, and compatibility are also commonly rated important or very important.
  • MSA design patterns: API gateway, Backend for frontend, and Access token are the most often used MSA design patterns.API gateway and Backend for frontend address client access to individual services, while Access token addresses security.
  • MSA design patterns: Service isolation, common communication patterns, and multiple data-management patterns are frequently used to support operational and architectural needs.Practitioners report isolated deployment patterns, Messaging or Remote procedure invocation, and patterns including Database per service, Sagas, Shared database, and CQRS.

5.2. Comparing the survey results with non-MSA-based systems

The comparison found limited evidence for several design and monitoring topics, but identified shared concerns and practices between MSA-based and non-MSA-based systems. Unit testing and common testing tools appear across both contexts, while integration testing remains a shared challenge.

  • Design comparison: Insufficient evidence prevented comparison of several design topics, including decomposition, high-level design, architecting activities, description, and design challenges.Quality attributes were nevertheless identified as a major concern in both MSA-based and non-MSA-based systems.
  • Quality attributes: Performance is a quality attribute of common interest across MSA-based and non-MSA-based systems, alongside practitioner interest in availability and security.These attributes concern system response, recovery from failures, and recovery from security incidents.
  • Monitoring comparison: Insufficient evidence also limited comparison of monitoring metrics, practices, tools, challenges, and solutions between MSA-based and non-MSA-based systems.The paper identified only one literature survey on runtime monitoring instrumentation for non-MSA-based systems.
  • Testing comparison: Unit testing is commonly used as a de facto strategy for both MSA-based and non-MSA-based systems.JUnit, JMeter, and Selenium are also commonly used testing tools across both contexts.
  • Testing comparison: Integration testing is a shared testing challenge in MSA-based and non-MSA-based systems.The comparison also reports overlap in commonly used testing tools and strategies.

5.3. Implications for practitioners

The practitioner implications emphasize targeted guidance for architecture ownership, decomposition, and design-pattern selection. The paper also identifies organizational and methodological topics requiring further empirical investigation.

  • Architecture responsibility: 36.7% of organizations reported having no dedicated architecture team or architect responsible for microservices architecture.The paper proposes studying effects on system quality and the countermeasures used in practice.
  • Application decomposition: 42.3% of organizations use DDD and business capability together to decompose monolithic applications into microservices.The paper recommends guidelines explaining how to apply both strategies together, especially for inexperienced teams.
  • Development practices: Future studies should examine how Scrum and DevOps practices are used to design, develop, and maintain microservices systems.The paper highlights the need to investigate challenges arising when MSA is used with these development methods.
  • Design-pattern selection: Decision models could help developers and architects select MSA design patterns by considering quality-attribute effects across design areas.The proposed scope includes communication, service discovery, monitoring, and logging.

6. Threats to validity

The survey addresses threats to validity in dissemination, participant selection, measurement, response rate, generalizability, and analysis. Despite mitigation procedures, the findings may not represent all microservices practitioners, particularly given the geographic concentration of respondents.

  • Internal validity: Participant expertise could not be guaranteed because respondents may have lacked the required experience to answer the survey.The researchers used personal contacts, relevant platforms, explicit survey criteria, and an explanatory preamble to reduce this risk.
  • Internal validity: Most questions were closed-ended, creating a risk that available answer options would be insufficient.Options were based on existing literature, and free-text responses were allowed for additional answers.
  • Internal validity: The 39-question survey may have reduced response rates: 135 responses were received from nearly 9000 emails, including 106 valid responses.The authors reduced open-ended questions and informed participants about the survey length to mitigate this threat.
  • Construct validity: Construct validity was threatened by inadequate explanation of survey constructs, although literature grounding, expert review, and a pilot with ten practitioners were used as mitigations.The authors also revised several questions after pilot feedback.
  • External validity: The study mitigated dissemination risk by selecting groups whose discussions addressed microservices techniques, tools, and technology issues.Researchers reviewed existing group discussions before posting the survey invitation.
  • External validity: 106 valid responses came from 29 countries, but 56.6% of respondents were from Pakistan and China, limiting generalizability to all practitioners.The paper recommends another study with more diverse practitioners to increase external validity.
  • Conclusion validity: Contradictory responses and possible differences in interpreting inclusion and exclusion criteria could affect the study’s conclusions.Multiple-author review, predefined criteria, and discussion sessions were used to reduce conclusion-validity threats.

7. Related work

Prior studies examined microservices adoption, technologies, design areas, migration, and debugging, but this survey addresses underreported architecting, monitoring, and testing evidence from practice.

  • Prior empirical studies: Existing surveys examined microservices drivers, technologies, challenges, advantages, design areas, migration, and debugging practices.These studies included surveys and interviews with practitioners, including 122 practitioners, 10 experts, and 16 practitioners.
  • Prior empirical studies: Participants in earlier work identified monitoring and logging as the most important microservices design area.
  • Research gap: The compared literature did not report MSA architecting activities from practice or discuss microservices patterns and quality attributes.
  • Research gap: No identified study reported practice-based evidence about monitoring and testing microservices systems.

8. Conclusions

This mixed-methods study reports how industry practitioners design, monitor, and test microservices systems. It identifies commonly used strategies, patterns, metrics, practices, tools, and persistent design challenges, and calls for dedicated solutions.

  • Study scope: The study combined survey and interview evidence to characterize practitioners’ design, monitoring, and testing of microservices systems.
  • Study scope: 106 practitioners from 29 countries participated in the survey, while six practitioners from four countries participated in interviews.
  • Design findings: 42.3% reported combining domain-driven design and business capability strategies to decompose applications into microservices.
  • Design findings: API gateway, Database per service, Backend for frontend, and Access token were the most often used MSA design patterns.
  • Monitoring findings: Resource usage, load balancing, availability, and database connections were frequent monitoring metrics, while log management, exception tracking, and health check API were common practices.
  • Testing findings: Unit testing, E2E, and integration testing were the top three testing strategies, with Junit, Jmeter, and Mocha the most popular tools.
  • Implications: The study recommends dedicated solutions for challenges in designing, monitoring, and testing microservices systems.

Appendix A. Abbreviations used in this study

Appendix A contains a table of abbreviations used in the study.

  • Appendix contents: Table 24 is titled “Abbreviations used in this study.”

Appendix B. Survey and interview questionnaires

The appendices provide the survey and interview materials used to study microservices design, monitoring, and testing, including participant questions, design practices, quality attributes, patterns, challenges, and solution-oriented questions.

  • Questionnaire structure: The survey covers participant background, MSA design, monitoring, and testing.
  • Participant background: The questionnaire asks about organization domains, size, team size, work experience, and development methods.
  • Design questions: Design questions address architectural evaluation, architectural implementation, maintenance and evolution, description methods, diagrams, components, and quality attributes.
  • Design questions: The questionnaire lists MSA patterns including API gateway, Backend for frontend, Database per service, Access token, service discovery, service mesh, and deployment options.
  • Challenges and solutions: Respondents are asked about design challenges such as defining service boundaries, managing complexity, security, scalability, data management, communication, and team alignment.
Loading 2108.03384v2…