PaperlayerView exact revision on arXiv

Source-linked AI summary

MCRL2: Multi-resource Cross-attention-based Representation Learning-augmented Reinforcement Learning for Cloud Microservice Scheduling

Tiangang Li, Shi Ying, Xiangbo Tian, Chuan Shi, Ding Xiao

arXiv:2609.13048v1cs.LG

TL;DR

Microservice scheduling must handle dynamic resource imbalance, nonlinear coupling among resource dimensions, and heterogeneous service demands, while existing DRL approaches lack structured state representations for these interactions. MCRL2 combines multi-resource cross-attention representations with soft actor–critic reinforcement learning. On real-world production traces, it consistently outperforms baselines in load balancing, scheduling success rate, and average completion time.

  • Problem

    Microservice scheduling faces dynamic node imbalance, nonlinear cross-resource coupling, and heterogeneous instance demands, while existing DRL approaches lack structured representations of these system characteristics.

    Source: §1 · pp. 1–2

  • Method

    MCRL2 augments soft actor–critic reinforcement learning with MCRL, a multi-resource cross-attention approach that represents interactions among nodes, resources, and microservices.

    Source: §1 · p. 3, §4.2 · p. 6

  • Results

    MCRL2 consistently outperforms baselines on real-world production traces in load balancing, scheduling success rate, and average completion time.

    Source: §7 · p. 14

  • Takeaways & Limitations

    The reported results support MCRL2 as an effective scheduling approach across the evaluated production-trace workload patterns.

    Source: §1 · p. 3, §7 · p. 14

    The formulation considers containerized microservice instances and schedules newly created instances onto nodes as autoscaling changes their number.

    Source: §3.1 · pp. 3–4

Abstract

from arXiv · show

Efficient microservice scheduling is crucial for maintaining load balance across nodes in data centers and ensuring high quality of service. However, achieving this in practice remains challenging due to dynamic resource imbalance under fluctuating workloads, nonlinear coupling across multiple resource dimensions, and the heterogeneity of microservice resource demands. While reinforcement learning-based approaches have shown promise, they struggle to capture the complex interdependencies among heterogeneous resources and neglect the importance of learning informative system representations. To address these limitations, we propose MCRL2, a novel reinforcement learning approach augmented with multi-resource cross-attention-based representation learning for microservice scheduling. Specifically, we first propose MCRL, a novel representation learning approach that captures structured and informative interactions among nodes, resources, and microservices via a multi-resource cross-attention mechanism. Then, MCRL2 augments reinforcement learning through MCRL-enhanced actor-critic architecture combined with a maximum entropy objective, improving system state expressiveness and leading to more stable and effective scheduling decisions. Extensive experiments on real production cluster traces demonstrate that MCRL2 significantly outperforms existing baselines in load balancing, scheduling success rate and average completion time across diverse workload patterns.

1. INTRODUCTION

Microservice schedulers must contend with shifting node loads, changing interactions among resource dimensions, and heterogeneous instance demands. MCRL2 pairs structured cross-attention representations with actor–critic reinforcement learning, and the introduction reports better baseline performance across workload patterns.

1. INTRODUCTION

  • Node resource utilization varies over time, and co-located instances can intensify imbalance, requiring scheduling decisions to adapt to changing workloads.The paper describes these trends in production-cluster data.Source: §1 · p. 1
  • Resource-dimension coupling can shift between negative, positive, or weak correlations, creating nonlinear dependencies and local contention.The paper notes that co-location can intensify these dependencies.Source: §1 · pp. 1–2
  • Microservice demands differ across instances and fluctuate over time, challenging adaptation across spatiotemporally heterogeneous workloads.The paper identifies both spatial variation among instances and temporal variation within an instance.Source: §1 · p. 2
  • MCRL2 augments reinforcement learning with MCRL representations in a dual-stream soft actor–critic design targeting heterogeneous states, value-estimation bias, and training stability.The paper also reports extensive evaluation on real production-cluster traces.Source: §1 · pp. 2–3
  • MCRL uses cross-attention to represent interactions among nodes, resource dimensions, and microservices, including dynamic cross-node and cross-resource relationships.Its components aggregate node-level features and fuse them with microservice-instance features.Source: §1 · pp. 2–3
  • MCRL2 outperforms baselines under different workload patterns in experiments on a real production-cluster microservice dataset.The introduction does not give metric-specific values for this comparison.Source: §1 · p. 3

2. RELATED WORK

The reviewed work spans heuristic placement and scheduling methods as well as deep reinforcement learning approaches. These studies pursue varied objectives and use different state representations.

2. RELATED WORK

  • Heuristic approaches include greedy, genetic, and sunflower whale optimization methods for placement, allocation, and QoS-aware scheduling.The cited studies report goals including throughput, response time, delay, resource consumption, and cost.Source: §2 · p. 3
  • DRL-based approaches use methods including PPO, A2C, DQN, and SAC for load balancing, deployment, routing, and instance selection.The reviewed work includes objectives such as latency, communication cost, delay, and energy consumption.Source: §2 · p. 3
  • The comparison of DRL schedulers covers their algorithms, optimization objectives, and state-representation strategies.Source: §2 · p. 3

3. PROBLEM DEFINITION AND FRAMEWORK OVERVIEW

The paper formulates cloud microservice scheduling as a resource-constrained assignment problem with load balancing and scheduling success rate as objectives. Its framework monitors system state, places instances, and uses resulting rewards and experience trajectories to train a DRL scheduler.

3.1 Problem Definition

  • Scheduling is formulated as a multi-objective, resource-constrained assignment problem whose solution space grows exponentially with the numbers of instances and nodes.The paper characterizes the problem as NP-hard.Source: §3.1 · p. 4
  • Assignments must respect each node’s multidimensional resource capacity, and every microservice instance must be assigned to exactly one node.Source: §3.1 · p. 4
  • The optimization objectives include cluster load balance and microservice-instance scheduling success rate.The formulation defines load balancing by resource dimension and includes a weight for each dimension.Source: §3.1 · p. 4

3.2 Framework Overview

  • System monitoring supplies application- and resource-layer telemetry for state construction; placement decisions produce updated-state rewards and experience trajectories for training the actor–critic scheduler.The framework places submitted instances on cluster nodes as user requests change autoscaled instance counts.Source: §3.1 · p. 4, §3.2 · p. 5

4. MULTI-RESOURCE CROSS-ATTENTION MECHANISM-BASED REPRESENTATION LEARNING

MCRL uses cross-resource attention to represent dynamic resource dependencies, then aggregates resource, node, and instance information into features for microservice scheduling.

4.1 Multi-resource Cross-Attention Mechanism

  • MCAM independently projects each resource embedding into query, key, and value vectors, then uses directional cross-resource attention to fuse resource values with residual connections.The resulting representations retain resource utilization information while incorporating cross-resource dependencies.Source: §4.1 · pp. 5–6

4.2 Multi-resource Cross-Attention Mechanism-based Representation Learning

  • MCRL separately embeds each resource dimension, applies MCAM to encode dynamic cross-resource correlations and coordination policies, and stacks the outputs as global resource representations.These representations provide fine-grained resource interaction features.Source: §4.2 · p. 6
  • MCRL averages fused resource features at each node, then combines node-level features with microservice-instance information into final scheduling representations.The joint feature provides contextual information about nodes and instances within the cluster.Source: §4.2 · p. 6

5. MULTI-RESOURCE CROSS-ATTENTION-BASED REPRESENTATION LEARNING-AUGMENTED REINFORCEMENT LEARNING FOR MICROSERVICE SCHEDULING

Section 5 formulates microservice scheduling as an MDP with a workload-sensitive reward, specifies an MCRL-based soft actor-critic, and proves convergence under stated assumptions.

5.1 Representation Learning-augmented Reinforcement Learning Model for Microservice Scheduling

  • The MDP state comprises node-, microservice-, and instance-specific telemetry; the agent selects actions and transitions between environmental states.The passage gives the state size as |N| + 2 + J.Source: §5.1 · p. 7
  • Reward penalizes failed placements and adjusts to node-versus-cluster workload deviation, while maximizing discounted cumulative return sets the policy’s optimization direction.Failed scheduling receives a reward of −1.0.Source: §5.1 · p. 7

5.2 Implementation Details of Multi-resource Cross-Attention-based Representation Learning-augmented Reinforcement Learning

  • RL-Actor-Critic pairs MCRL-based state encoders with policy and Q-value networks, using dual Q-networks to mitigate value overestimation and target networks to stabilize training.The actor outputs an action distribution from its representation; the critic evaluates state-action values.Source: §5.2 · p. 8
  • The policy and value networks are updated independently to prevent training instabilities attributed to parameter coupling.The actor and critic use independent MCRL-based representation-learning parameters.Source: §5.2 · p. 8
  • The maximum-entropy objective balances reward and policy entropy, while representation-guided exploration prioritizes critical nodes.The temperature parameter adjusts the reward-maximization versus entropy-regularization trade-off.Source: §5.2 · pp. 8–9

5.3 Convergence Proof of MCRL2

  • With stable representations and finite state and action spaces, soft policy evaluation converges to a unique Q-function, and policy improvement converges to the maximum-entropy optimal policy.The policy objective is non-decreasing during improvement and bounded under the stated conditions.Source: §5.3 · pp. 9–10

6. PERFORMANCE EVALUATION

The evaluation compares MCRL2 with five baselines across training convergence, load balancing, scheduling success, completion time, and decision time using Alibaba production-cluster traces.

6.1 Experimental Setup

  • The experiments use Alibaba production-cluster traces, with 2,000 instances for training and subsequent 2,000-instance and 4,000-instance evaluation patterns.The simulated cluster has 15 heterogeneous nodes and three VM types with distinct resource configurations.Source: §6.1 · pp. 10–11, Table 3 · p. 11
  • The evaluation tracks cumulative and real-time load balancing, scheduling success rate, and average instance completion time.Lower DCLB indicates better cumulative load balancing, while higher MSSR indicates improved system QoS.Source: §6.1 · p. 11

6.2 RQ1: Effectiveness for training convergence.

  • MCRL2 converges fastest and reaches the highest final reward, while SAC has comparable initial convergence but a lower final reward.DDQN and PPO converge more slowly and reach lower rewards; PPO also has notable reward fluctuations.Source: §6.2 · p. 11, Figure 6 · p. 12

6.3 RQ2: Effectiveness for load balancing.

  • MCRL2 achieves comprehensive cumulative load-balancing superiority, especially for memory, and ranks first most often; its cumulative CPU imbalance differs from PPO by only 5.33 under MS-4000.The 5.33 difference is reported for cumulative CPU load imbalance under the MS-4000 pattern.Source: §6.3 · p. 12, Figure 9 · p. 13, Figure 10 · p. 13
  • Across production-trace patterns, MCRL2 significantly outperforms all baselines in load balancing, maintaining lower CPU and memory imbalance over time.The evaluation covers MS-2000 and MS-4000 workload patterns.Source: §6.3 · p. 12, Figure 7 · p. 12, Figure 8 · p. 12

6.4 RQ3: Effectiveness for microservice scheduling success rate.

  • MCRL2 achieves scheduling success rates of 0.995 and 0.988, improving by 5.85% and 5.56% over the best baseline in MS-2000 and MS-4000, respectively.The figure compares scheduling success rates across the two patterns.Source: §6.4 · p. 13, Figure 13 · p. 13

6.5 RQ4: Effectiveness for average completion time.

  • MCRL2 reduces average completion time to 603.33 seconds and 607.28 seconds, improving by 5.48% and 5.29% over the best baseline, respectively.It has the lowest average completion time in both evaluation patterns.Source: §6.5 · pp. 13–14, Figure 14 · p. 14

6.6 RQ5: Efficiency for scheduling decision time.

  • MCRL2 incurs higher cumulative scheduling decision time than DRL-based baselines, but the added delay remains relatively small and is not a performance bottleneck.Reported cumulative decision times are averaged over five experimental runs.Source: §6.6 · p. 14, Table 4 · p. 14

7. CONCLUSION

MCRL2 combines multi-resource cross-attention-based representation learning with reinforcement learning to address dynamic imbalance, cross-resource coupling, and heterogeneous microservice demands. On production traces, it outperforms baselines in load balancing, scheduling success rate, and average completion time.

7. CONCLUSION

  • On real-world production traces, MCRL2 consistently outperforms baselines in load balancing, scheduling success rate, and average instance completion time.The conclusion highlights its practical value in dynamic microservice environments.Source: §7 · p. 14
  • MCRL uses multi-resource cross-attention to capture fine-grained interdependencies among nodes, resources, and microservices.Its learned representations are described as informative and as improving scheduling effectiveness.Source: §7 · p. 14
  • The approach targets dynamic resource imbalance, nonlinear cross-resource coupling, and multidimensional heterogeneity in microservice requests.MCRL2 integrates the representation learning approach with soft actor-critic for stable and efficient decision-making.Source: §7 · p. 14

Related papers

Loading 2609.13048v1…