Source-linked AI summary
Overload Control for Scaling WeChat Microservices
Hao Zhou, Ming Chen, Qian Lin, Yong Wang, Xiaobin She, Sifan Liu, Rui Gu, Beng Chin Ooi, Junfeng Yang
TL;DR
Large-scale microservices need overload control that handles complex dependencies and changing workloads without relying on service-specific logic. DAGOR monitors load per microservice and coordinates adaptive load shedding with upstream services. The paper reports similar request success rates across request types under overload and more than five years of WeChat deployment.
Problem
Complex microservice dependencies and changing workloads make service-specific overload control difficult to design and can make independent shedding harmful to the system.
Source: §1 · pp. 1–2Method
DAGOR monitors each microservice’s load and adaptively adjusts priority thresholds, while informing upstream services so requests can be shed earlier.
Source: §1 · p. 2Results
DAGOR manifests roughly the same success rate for different request types under overload, unlike CoDel, which favors simple overload over subsequent overload.
Source: §5.4 · p. 11Takeaways & Limitations
DAGOR is service agnostic and has run in the WeChat business system for more than five years.
Source: §1 · p. 2, §7 · p. 12
Abstract
from arXiv · showhide
Effective overload control for large-scale online service system is crucial for protecting the system backend from overload. Conventionally, the design of overload control is ad-hoc for individual service. However, service-specific overload control could be detrimental to the overall system due to intricate service dependencies or flawed implementation of service. Service developers usually have difficulty to accurately estimate the dynamics of actual workload during the development of service. Therefore, it is essential to decouple the overload control from service logic. In this paper, we propose DAGOR, an overload control scheme designed for the account-oriented microservice architecture. DAGOR is service agnostic and system-centric. It manages overload at the microservice granule such that each microservice monitors its load status in real time and triggers load shedding in a collaborative manner among its relevant services when overload is detected. DAGOR has been used in the WeChat backend for five years. Experimental results show that DAGOR can benefit high success rate of service even when the system is experiencing overload, while ensuring fairness in the overload control.
1 INTRODUCTION
Large-scale microservice systems need overload control that can handle dynamic workloads and complex dependencies without relying on service-specific logic. DAGOR monitors load at each microservice and coordinates priority-based load shedding with upstream services.
1 INTRODUCTION
- Service-specific overload control is difficult to design for intricate dependencies and workload dynamics, motivating a service-agnostic system-level approach.The paper argues overload control should be decoupled from service logic because developers struggle to estimate actual workload dynamics during service development.Source: §1 · p. 1
- Unmonitored microservices can become overload points whose effects ripple upstream, while fast-evolving deployments make centralized monitoring difficult.The resulting cascading overload can leave affected services with high delay.Source: §1 · p. 2
- Independent shedding can waste resources on partially processed failed requests and drive throughput toward zero when many required microservices reject requests.With K required microservices each rejecting independently with probability p, the passage gives complete-request success expectation as (1 −p)K.Source: §1 · p. 2
- DAGOR assigns business and user priorities at entry services, monitors each microservice’s load, adaptively sheds load, and informs upstream services of threshold changes.Its adaptive algorithm attempts to shed half the load when overload is detected.Source: §1 · p. 2
- DAGOR uses a small set of thresholds and marginal coordination, and is designed to accommodate service changes and external environments.The supplied passage describes the scheme as service agnostic and deployed across WeChat microservices with diverse business logic.Source: §1 · p. 2
- The paper presents DAGOR, shares WeChat operating experience, and evaluates its capability as a practical solution for operational microservice systems.The authors distinguish this focus from prior work on shedding load inside a network path.Source: §1 · p. 2
2 BACKGROUND
WeChat’s backend combines thousands of services in a rapidly changing architecture with substantial workload fluctuations. Its service graph and deployment patterns help explain why overload control must accommodate scale and change.
2 BACKGROUND
- WeChat supports more than 3000 mobile services, including instant messaging, social networking, mobile payment, and third-party authorization.Source: §2 · p. 2
2.1 Service Architecture of WeChat
- The service architecture is a DAG of basic sink services and leap services that invoke downstream services; user requests start at entry services and end at basic services.The architecture is layered, with entry services at the top, basic services at the bottom, and most leap services between them.Source: §2.1 · p. 2
- WeChat receives 10^10–10^11 entry-service requests daily, which trigger enough downstream work to require handling hundreds of millions of service requests per second.Source: §2.1 · p. 3
2.2 Deployment of WeChat Services
- The backend runs more than 3000 services on over 20000 machines, and averaged almost a thousand service changes per day from March to May 2018.The authors report that centralized or SLA-based overload control can hardly support service changes at this scale and pace.Source: §2.2 · p. 3
2.3 Dynamic Workload
- Peak request volume is typically about 3 times the daily average and can reach around 10 times during occasions such as Chinese Lunar New Year.The passage describes over-provisioning for such fluctuations as uneconomic and runtime-adaptive overload control as more practical.Source: §2.3 · p. 3
3 OVERLOAD IN WECHAT
WeChat overload can propagate along service call paths, and repeated or multiple overloaded calls make successful completion difficult. The system’s scale and lack of a single entry point complicate coordinated load shedding.
3 OVERLOAD IN WECHAT
- Overload may arise from load surges, capacity degradation, network outages, configuration changes, software bugs, or hardware failures.The passage describes three basic forms as building blocks for more complex overload scenarios.Source: §3 · p. 3
3.1 Overload Scenarios
- Subsequent overload occurs when more than one service is overloaded, or when a single overloaded service is invoked multiple times by associated upstream services.In WeChat, Forms 1 and 2 dominate overload cases, while Form 3 is relatively rare; random load shedding could lead to close-to-zero client request success when requesting services are overloaded.Source: §3.1 · pp. 3–4
- When a service is overloaded, random shedding can sharply reduce task success under subsequent overload because a task must succeed at every overloaded invocation.In the example, two calls to a service accepting half of requests yield a 25% success rate for the upstream task.Source: §3.1 · p. 4
3.2 Challenges of Overload Control at Scale
- WeChat has no single request entry point, and variable call paths make it impossible to precisely determine which requests to drop when a service overloads.Source: §3.2 · p. 4
- Coordination can reduce wasted work and latency from excessive aborts, but maintaining effective coordination across a complex, continuously evolving service DAG is costly.The challenge is especially severe during subsequent overload.Source: §3.2 · p. 4
4 DAGOR OVERLOAD CONTROL
DAGOR provides service-agnostic overload control by combining service-level admission decisions with coordinated request shedding across service dependencies.
4 DAGOR OVERLOAD CONTROL
- DAGOR is service agnostic and system-centric, controlling overload at individual machines while coordinating related services to protect upstream success rates.Its design also aims for efficient and fair shedding: abort tasks early to limit wasted computation and preserve roughly equal saturated throughput across upstream services.Source: §4 · pp. 4–5
4.1 Overload Detection
- DAGOR detects server overload using average request queuing time, which reflects local processing capacity without the downstream effects that can distort response time.In WeChat, monitoring refreshes every second or 2000 requests, and the overload threshold is 20 ms against a default 500 ms service-task timeout.Source: §4.1 · p. 5
4.2 Service Admission Control
- After detecting overload, DAGOR uses service admission control, combining priority-based strategies with adaptive and collaborative control.Source: §4.2 · p. 5
4.2.1 Business-oriented Admission Control.
- Business-oriented admission control assigns each entry-service action a business priority, sheds lower-priority requests first, and propagates that priority along the downstream call path.Priorities are stored in a replicated action-priority hash table; unlisted actions receive the lowest priority.Source: §4.2.1 · pp. 5–6
4.2.2 User-oriented Admission Control.
- Hourly changes to a user-ID hash assign requests from the same user a consistent priority within each hour while rotating high priorities among users over time.User priority supplements business priority to shed only some requests at the admission boundary without randomly disrupting subsequent requests in a call path.Source: §4.2.2 · pp. 6–7
4.2.3 Adaptive Admission Control.
- DAGOR combines tens of business-priority admission levels with 128 user-priority levels each, yielding tens of thousands of compound levels; one-by-one search requires validation time and struggles to meet real-time adaptive-control requirements.Source: §4.2.3 · p. 7
4.2.4 CollaborativeAdmission Control.
- Servers piggyback their current admission levels on responses so upstream servers can locally shed requests likely to be rejected downstream, reducing wasted network and server resources.Admission decisions remain independently determined by each server, while actual shedding is performed collaboratively by upstream servers.Source: §4.2.4 · pp. 8–9
4.3 Workflow of Overload Control
- At system entry, the entry service assigns business and user priorities, which subsequent downstream requests inherit in their request headers.Source: §4.3 · p. 9
- Services admit incoming requests by priority and current admission level, then upstream services locally gate downstream requests using the stored level.Each service periodically adjusts its admission level according to its load status.Source: §4.3 · p. 9
- Downstream services piggyback their current admission levels on responses, which upstream services extract to update their local records.Service requests and responses are delivered through message passing.Source: §4.3 · p. 9
- DAGOR separates admission decisions from business logic while coordinating individual services and related upstreams, enabling scalable, efficient, and fair overload control.Requests on the same call path share business and user priorities, allowing upstream services to sustain saturated throughput despite downstream overload.Source: §4.3 · p. 9
5 EVALUATION
The evaluation tests DAGOR’s overload detection, service admission control, and fairness using workloads that vary in the number of invocations to an overloaded service. DAGOR’s admission control improves success rates under subsequent overload, while its detection and fairness results support service-agnostic load management.
5 EVALUATION
- DAGOR has been deployed in the WeChat business system for more than five years and is evaluated against state-of-the-art load-management techniques.Experiments run on an in-house cluster with Intel Xeon E5-2698 CPUs and 64 GB DDR3 memory per node.Source: §5 · p. 9, §5.1 · p. 9
5.1 Experimental Setup
- The stress test uses M1–M4 tasks that invoke service M one to four times through service A; M is deployed on three servers and saturates at about 750 QPS.Twenty application servers generate the workloads; tasks succeed only if all invocations succeed, with rejected invocations retried up to three times.Source: §5.1 · p. 9
5.2 Overload Detection
- At 630 QPS, response-time detection starts shedding, while queuing-time detection postpones shedding until 750 QPS in M1.Response time also produces false positives; its threshold is difficult to tune because it includes service-specific processing time, unlike queuing time.Source: Figure 6 · p. 10, §5.2 · p. 10
5.3 Service Admission Control
- In M2, DAGOR achieves around 50% higher success rate than CoDel and SEDA; for M3 and M4, its success rate remains close to optimal.For M1 simple overload, all techniques perform roughly the same.Source: Figure 7 · p. 10, Figure 8 · p. 11, §5.3 · p. 11
5.4 Fairness
- Under mixed M1–M4 workloads, DAGOR maintains roughly the same success rate across request types, while CoDel favors M1 over M2–M4.The experiment varies feed rate from 250 to 2750 QPS and assigns business and user priorities randomly within a fixed range.Source: §5.4 · p. 11
6 RELATED WORK
Prior overload-control research largely targets other systems or simpler service architectures. The related work contrasts DAGOR’s microservice focus and service-agnostic design with approaches that depend on centralized control, session properties, or static service partitions.
6 RELATED WORK
- DAGOR is presented as the first overload-control approach specifically addressing large-scale microservice architectures, unlike earlier work focused on databases, stream processing, sensor networks, web services, or SOA.The SOA techniques reviewed use admission control or resource scheduling.Source: §6 · p. 11
- Coflow-based Varys requires prior flow-size knowledge and centralized control, while Baraat avoids centralized control at a performance cost; DAGOR is service-agnostic and flow-independent.The paper characterizes DAGOR as non-invasive for microservice architecture.Source: §6 · p. 11
- Session-based admission control and QoS-aware session control favor long-lived sessions, whereas DAGOR uses user-oriented rather than session-oriented admission.The cited session-based approaches also rely on a centralized module, which can become a scalability bottleneck; DAGOR is decentralized.Source: §6 · pp. 11–12
- Staged overload control depends on service-semantic partitioning and static resource quotas, which the paper says do not suit WeChat’s complex, dynamic service architecture.DAGOR is described as service-agnostic and adaptable to an evolving microservice architecture.Source: §6 · p. 12
7 CONCLUSION
DAGOR is a service-agnostic overload-control scheme run in the WeChat backend for over five years. Its design principles emphasize decentralized control, feedback, and profiling actual workloads.
7 CONCLUSION
- DAGOR has run in the WeChat business system for more than five years, and the authors regard its design principles as insightful for other microservice systems.The paper describes DAGOR as service agnostic, independent but collaborative, efficient, and fair.Source: §7 · p. 12
- Large-scale microservice overload control should be decentralized and autonomous in each service, rather than relying on centralized resource scheduling, to scale with evolving systems.Source: §7 · p. 12
- Overload-control algorithms should use varied feedback mechanisms rather than relying solely on open-loop heuristics; DAGOR's collaborative admission control is one example.Source: §7 · p. 12
- Effective overload-control design should profile actual workload processing behavior, informing DAGOR's use of request queuing time for overload detection and two-tier priority-based admission control.Source: §7 · p. 12