PaperlayerView exact revision on arXiv

Source-linked AI summary

Data Management in Microservices: State of the Practice, Challenges, and Research Directions

Rodrigo Laigner, Yongluan Zhou, Marcos Antonio Vaz Salles, Yijian Liu, Marcos Kalinowski

arXiv:2103.00170v2cs.DBcs.SE

TL;DR

Research had not deeply characterized data-management practices and challenges in microservices. The paper combines a literature review, open-source application analysis, and practitioner survey to investigate them. It identifies ad hoc data-management designs and proposes database features to address microservice requirements by design.

  • Problem

    In-depth evidence about microservice data-management practices and practitioner challenges was lacking, despite growing adoption and the recognized importance of data management.

    Source: §1 · pp. 1–2

  • Method

    The study reviews representative literature, analyzes open-source microservice applications, and surveys experienced practitioners to cross-validate findings.

    Source: §1 · p. 2

  • Results

    The study finds that practitioners combine heterogeneous systems and application-level mechanisms ad hoc, and proposes database features to address microservice requirements by design.

    Source: §8 · p. 12

  • Takeaways & Limitations

    The authors conclude that meeting microservice data-management requirements by design calls for candidate database-system features that can help DBMSs play a central role.

    Source: §8 · p. 12

    Existing benchmarks such as DeathStarBench fail to incorporate asynchronous and event-driven microservice properties, complicating testing of new solutions.

    Source: §6 · p. 10

Abstract

from arXiv · show

Microservices have become a popular architectural style for data-driven applications, given their ability to functionally decompose an application into small and autonomous services to achieve scalability, strong isolation, and specialization of database systems to the workloads and data formats of each service. Despite the accelerating industrial adoption of this architectural style, an investigation of the state of the practice and challenges practitioners face regarding data management in microservices is lacking. To bridge this gap, we conducted a systematic literature review of representative articles reporting the adoption of microservices, we analyzed a set of popular open-source microservice applications, and we conducted an online survey to cross-validate the findings of the previous steps with the perceptions and experiences of over 120 experienced practitioners and researchers. Through this process, we were able to categorize the state of practice of data management in microservices and observe several foundational challenges that cannot be solved by software engineering practices alone, but rather require system-level support to alleviate the burden imposed on practitioners. We discuss the shortcomings of state-of-the-art database systems regarding microservices and we conclude by devising a set of features for microservice-oriented database systems.

1 INTRODUCTION

Microservices decompose applications into independently built, deployed, and scaled services, often with service-specific databases. The paper investigates data-management practice and challenges because existing research offers limited evidence and database systems often provide storage without a holistic view.

1 INTRODUCTION

  • Microservices communicate through remote calls and can be redeployed independently, while faults in individual services are isolated to limit propagation.Services may use databases suited to their own data formats and workloads, but application transactions must be broken down across the decomposed architecture.Source: §1 · p. 1
  • Service-specific databases support different data models and workloads, but the architecture makes transactions across services harder than transactions across modules in a monolith.The Products service may use a document model for schema evolution while Stock uses relational constraints.Source: §1 · p. 1
  • Research had not adequately characterized which database technologies, patterns, consistency semantics, and data-exchange mechanisms practitioners use in microservices.Database systems are often relegated to storage and lack a holistic view of dataflow and interactions among services.Source: §1 · p. 2
  • The investigation combines a systematic review of 10 representative articles, analysis of 9 open-source applications, and an online survey of more than 120 experienced participants.The survey cross-validates findings from the literature and application analyses.Source: §1 · p. 2
  • Practitioners face challenges enforcing safety across services, including distributed constraints, event interleavings, weak concurrency isolation, replication semantics, and consistency across storage technologies.They often combine heterogeneous systems ad hoc, leaving substantial data-management logic in the application layer.Source: §1 · p. 2
  • The paper characterizes practice and challenges, then proposes database-system features intended to address microservice data-management requirements by design.The authors argue that the observed problems cannot be resolved through code refactorings alone.Source: §1 · p. 2

2 RELATED WORK

The paper positions its work as an in-depth study of how microservice developers interact with database systems, addressing technical data-management issues that related software-engineering work does not capture.

2 RELATED WORK

  • Prior microservices research largely examined migration from monoliths and general software-engineering concerns, while in-depth data-management analysis remained lacking.The authors describe their study as a first step toward understanding developers’ interactions with database systems.Source: §2 · p. 3
  • Related studies and architectural-smell work did not capture consistency models or database issues such as replication and constraint enforcement.The paper presents an in-depth characterization of developers’ application-tier data-management challenges and DBMS limitations.Source: §2 · p. 3
  • The paper distinguishes its contribution from prior work by examining database-system limitations in light of the challenges found in microservice architectures.It identifies shared persistence and architectural anti-patterns as related but insufficient treatments of these technical issues.Source: §2 · p. 3

3 STATE OF THE PRACTICE

The study finds that microservice adoption is closely tied to data management, especially functional decomposition, isolation, and schema evolution. It also describes database deployment patterns and heterogeneous technology stacks used in practice.

3.1 Motivating factors

  • 57% of survey respondents and the reviewed literature identify functional decomposition as a major adoption driver for scalability and high data availability.The paper connects this pattern to functional scaling: grouping data by function across databases.Source: §3.1 · p. 3
  • Schema evolution is a major adoption driver, while practitioners cite polyglot persistence roughly 3.5 times less often than schema evolution.The authors infer that support for diverse data models is less pressing than schema evolution.Source: §3.1 · p. 3
  • Almost 15% of participants cite event-driven data management as a primary motivation, and literature and repositories report asynchronous messaging.The authors suggest asynchronous events should be considered in data-management technology design.Source: §3.1 · p. 3
  • Functional decomposition, fault isolation, schema evolution, and event-driven architecture are identified as primary data-management motivations for adopting microservices.The authors relate these motivations to decentralized data management, particularly a database or schema per microservice.Source: §3.1 · p. 3

3.2 Database systems and deployment patterns

  • Most practitioners prefer a managed database server per microservice, and practitioners, literature, and repositories report container-based deployment.The reported drivers include loose coupling, independent data-layer scalability, and fault isolation; separate containers allow independent scaling and limit faults to container boundaries.Source: §3.2 · p. 4
  • Practitioners report heterogeneous DBMS combinations, including relational databases with MongoDB, and a stack combining relational DBMSs, Redis, MongoDB, and ElasticSearch.Commonly, relational or document databases serve services, Redis caches recurring requests, and event-driven replication to ElasticSearch supports online analytical queries.Source: §3.2 · p. 4

4 DATA MANAGEMENT LOGIC IN PRACTICE

The reported data-management tasks span cross-microservice business transactions, online queries, event-driven computing, and stream processing. Across these activities, the findings point to asynchronous coordination and application-level mechanisms, while practitioners also use orchestration and streaming tools.

4 DATA MANAGEMENT LOGIC IN PRACTICE

  • The task categories identified from participant responses were business transactions, event-driven computing, online queries, and stream processing.Figure 2 presents the types of data-management logic identified from those responses.Source: §4 · p. 4, Figure 2 · p. 4

4.1 Cross-microservice business transactions

  • Microservices often perform operations spanning services despite their autonomy, and the reported coordination patterns generally provide only eventual consistency.The paper defines a business transaction as any unit of work across microservices, not necessarily under ACID guarantees.Source: §4.1 · p. 4
  • Asynchronous, event-based coordination is common in repositories and literature, while orchestration-like mechanisms are prominent in industry and 2PC is uncommon.The paper reports custom-made orchestration engines among 51.2% of participants; 78% of respondents describing use cases reported application-code workflows and application-level validations.Source: §4.1 · p. 5

4.2 Online queries

  • Participants reported cross-microservice queries, served through service composition, replication, replication to a database, stream processing, or shared-schema views.The paper reports 27 participants (40.3%) using a mechanism for aggregating data from multiple microservices; views were the least cited practice.Source: §4.2 · pp. 5–6

4.3 Stream processing

  • Microservice developers use ad-hoc data-processing pipelines, while the static dataflow abstraction can be difficult to use for expressing their computations.Reported uses include data transformations, anomaly detection, replication, and materialized views.Source: §4.3 · p. 6

4.4 Event-driven computing

  • 45 of 67 participants (67.16%) reported event-driven computations, commonly used for event-based and asynchronous workflows.26 participants (38.81%) provided a brief use-case description.Source: §4.4 · p. 7

5 MICROSERVICES THROUGH THE LOOKING GLASS

Building on decentralized, application-level data management, this section details challenges developers face and unmet needs for database-community support.

5.1 Cross-microservice validations

  • Cross-service inventory checks can become stale before order processing, causing excess events, resource trashing, and compensation logic under contention.Coordination is needed to ensure correctness under conflicting reads and writes.Source: §5.1 · p. 7

5.2 Implicit cross-microservice associations

  • Without cross-service foreign-key enforcement, deleting a product can leave dependent records assuming it still exists, bringing the system into an inconsistent state.The analyzed applications showed no evidence of such enforcement.Source: §5.2 · pp. 7–8

5.3 Cross-microservice queries

  • Application-level queries across services leave developers to handle fractured reads and failures, without a consistent view of the entire application at one point in time.Failed requests can produce missing records and incomplete results.Source: §5.3 · p. 8

5.4 Feral ordering

  • Asynchronous event interleavings make it difficult to reason about consistency requirements; eventual event arrival can determine whether a price change affects an order.The paper notes that applications may require either outcome, while some enforce no constraint.Source: §5.4 · pp. 8–9

5.5 Replication hell

  • Ad-hoc replication lacks ordering guarantees for updates, and queries may aggregate replicated data without reflecting a system-wide snapshot.The paper notes that not all applications relying on state queries use eventual consistency.Source: §5.5 · p. 9

5.6 Non-transactional queuing

  • External message brokers without database interoperability prevent 2PC, while non-transactional queuing leaves isolation and fault-handling gaps that challenge correctness.The observed practice lacked message persistence queues integrated with the database.Source: §5.6 · p. 9

6 DELVING INTO DEVELOPERS’ PAINS

Practitioners report intertwined challenges in enforcing correctness, handling asynchronous consistency, querying across services, and testing realistic deployments. Survey responses broadly align with earlier findings while adding schema evolution and data cleaning as concerns.

Eventual consistency and schema evolution

  • Asynchronous releases can leave dependent services with conflicting assumptions about message structures and invariants, while eventual consistency complicates reasoning about distributed state.Respondents identified eventual consistency (15.68%) as a challenge; asynchronous replication and workflow-oriented transactions were cited as drivers of non-converged state.Source: §6 · p. 10

Consistency across heterogeneous systems

  • Some applications require stronger write guarantees across heterogeneous database systems, beyond eventual consistency.14.70% of survey responses indicated a need for stronger guarantees over writes in different database systems.Source: §6 · p. 10

Ad hoc inconsistency handling

  • Developers address inconsistencies with ad hoc measures, including manual database intervention and custom data-management logic.An asynchronous-write example describes a cache expiring before the corresponding information is persisted in the database.Source: §6 · p. 10

Consistent querying

  • Online querying lacks a de facto approach, and practitioners also seek a consistent global view that reflects current changes across data sources.Ad hoc mechanisms were reported for online queries (8.82%).Source: §6 · p. 10

Functional partitioning

  • Poorly considered functional partitioning can burden developers with data inconsistencies, and participants linked unnecessary service separation to consistency problems.The paper notes that further exploration of partitioning with real-world systems is necessary.Source: §6 · p. 10

Benchmark limitations and survey synthesis

  • Existing benchmarks such as DeathStarBench do not incorporate asynchronous, event-driven microservice behavior, making new solutions harder to test; survey perceptions otherwise align with the findings and add schema evolution and data cleaning concerns.The paper identifies the lack of benchmarks reflecting real-world deployments as a barrier to effective experimentation and reasoning.Source: §6 · pp. 9–10

7 TOWARDS MICROSERVICE-ORIENTED DATABASE SYSTEMS

The paper argues that microservice data-management needs exceed what current database systems and application-level workarounds adequately support. It proposes database features and implementation paths that account for cross-service events, constraints, replication, and queries while preserving service autonomy.

Required DBMS features

  • The proposed features include database awareness of cross-service events and event-based constraints, plus coordination abstractions balancing loose coupling with consistency.These features are intended to capture data exchange and support application safety.Source: §7 · p. 11
  • The proposed system should support cross-microservice associations and cache freshness semantics, and offer abstractions for replication with varied consistency models.The paper also notes that cross-service constraints must account for data in underlying caches.Source: §7 · p. 11
  • Native consistent cross-microservice online queries should respect data sovereignty, with optimization options including preprocessing, pre-joining, or proactive replication.Simply querying private states across tenants is insufficient for consistent views; the DBMS must capture interactions among microservices.Source: §7 · pp. 11–12

Why web-scale architectures are not enough

  • BASE principles do not provide a full-fledged data-management solution for microservices, whose requirements include cross-service validations, replication, queries, and interleaving events.Pushing constraint enforcement to the application level is described as often leading to consistency problems.Source: §7 · p. 11

Shortcomings of current DBMSs

  • Multi-tenant systems force a trade-off between isolated service states and shared states, while either approach can jeopardize microservice autonomy or require application-level coordination.Sharing a tenant can create schema and failure dependencies; separate tenants impede cross-service constraints, cache consistency, and replication.Source: §7 · p. 12

Realizing the features

  • The paper proposes extending stateful middle-tier abstractions or bringing database components into the application tier, alongside abstractions that expose application data-management logic to the DBMS.The discussed approaches have limitations: the paper notes missing needed features in initial middle-tier efforts and the lack of a formalism and implementation for the application-tier proposal.Source: §7 · p. 12

8 CONCLUSION

The paper observes that microservices lack a holistic data-management solution, leading practitioners to use ad hoc designs and application-level mechanisms. It proposes candidate database features to address these needs.

Conclusion

  • Without a holistic data-management solution, practitioners resort to ad hoc designs that combine heterogeneous services and application-level mechanisms.The paper says the resulting problems cannot be resolved through code refactorings alone.Source: §8 · p. 12
  • The paper proposes candidate database features intended to make database systems central to data management in the microservice paradigm.Source: §8 · p. 12
  • The described problems arise in a setting where practitioners weave heterogeneous services together and rely on application-level data-management mechanisms.Source: §8 · p. 12

Related papers

Loading 2103.00170v2…