PaperlayerView exact revision on arXiv

Source-linked AI summary

A Comparison of Reinforcement Learning Techniques for Fuzzy Cloud Auto-Scaling

Hamid Arabnejad, Claus Pahl, Pooyan Jamshidi, Giovani Estrada

arXiv:1705.07114v1cs.DCcs.AI

TL;DR

Cloud auto-scaling must decide when and how to resize resources while meeting QoS requirements, a challenge for threshold-based approaches that leave threshold-setting to users. The paper combines a fuzzy controller with SARSA and Q-learning and reports that both approaches scale resources while reducing costs, with SARSA improving solution quality for periodic workloads.

  • Problem

    Threshold-based autoscaling requires users to set threshold conditions, while choosing when and how to resize resources to maintain acceptable QoS remains difficult.

    Source: §I · p. 1

  • Method

    The paper combines a fuzzy logic controller with SARSA and Q-learning as reinforcement-learning approaches for dynamic resource allocation.

    Source: §I · p. 1

  • Results

    Both approaches efficiently scale cloud resources to meet QoS requirements while reducing provider costs, and the SARSA experiment reports improved solution quality for periodic workloads.

    Source: §VII · p. 10

  • Takeaways & Limitations

    The authors identify fuzzy reinforcement-learning autoscaling as a potential alternative to traditional threshold-based approaches across workload patterns.

    Source: §VII · p. 10

    The fuzzy-rule antecedent membership functions are assumed fixed; incremental clustering is proposed for enabling dynamic changes.

    Source: §VII · p. 10

Abstract

from arXiv · show

A goal of cloud service management is to design self-adaptable auto-scaler to react to workload fluctuations and changing the resources assigned. The key problem is how and when to add/remove resources in order to meet agreed service-level agreements. Reducing application cost and guaranteeing service-level agreements (SLAs) are two critical factors of dynamic controller design. In this paper, we compare two dynamic learning strategies based on a fuzzy logic system, which learns and modifies fuzzy scaling rules at runtime. A self-adaptive fuzzy logic controller is combined with two reinforcement learning (RL) approaches: (i) Fuzzy SARSA learning (FSL) and (ii) Fuzzy Q-learning (FQL). As an off-policy approach, Q-learning learns independent of the policy currently followed, whereas SARSA as an on-policy always incorporates the actual agent's behavior and leads to faster learning. Both approaches are implemented and compared in their advantages and disadvantages, here in the OpenStack cloud platform. We demonstrate that both auto-scaling approaches can handle various load traffic situations, sudden and periodic, and delivering resources on demand while reducing operating costs and preventing SLA violations. The experimental results demonstrate that FSL and FQL have acceptable performance in terms of adjusted number of virtual machine targeted to optimize SLA compliance and response time.

I. INTRODUCTION

The introduction frames cloud auto-scaling as a challenge of resizing resources to meet QoS without relying on user-set thresholds. It presents a fuzzy controller combined with SARSA and Q-learning, implemented and evaluated in OpenStack.

I. INTRODUCTION

  • The paper compares fuzzy SARSA learning and fuzzy Q-learning, with SARSA using on-policy updates and Q-learning learning independently of the current policy.The comparison also considers performance and scalability under different workload patterns.Source: §I · p. 1
  • The fuzzy logic controller supports higher-level reasoning while reinforcement learning adjusts the auto-scaling controller.The paper motivates fuzzy logic as a way to represent many states with fewer fuzzy states.Source: §I · p. 1
  • The contributions include integrating fuzzy logic and reinforcement learning in a real cloud controller and implementing the approaches in OpenStack.The paper also describes a review of cloud auto-scaling approaches.Source: §I · p. 1
  • The authors report extensive experimentation and evaluation across a wide range of workload patterns.The stated comparison covers different workload patterns.Source: §I · p. 1
  • The approaches handle varied traffic and deliver resources on demand while reducing costs; experiments report promising resource adjustment for SLA compliance and response time.The introduction describes reductions in infrastructure and management costs.Source: §I · pp. 1–2

II. BACKGROUND AND RELATED WORK

The section reviews threshold-based, feedback-control, and forecasting approaches to cloud auto-scaling. Their decisions rely respectively on configured thresholds, system feedback, or predictions from historical data.

A. Threshold-based rules

  • Threshold-based autoscaling uses performance-metric rules, but its performance depends on the quality of user-set thresholds.Its simplicity makes it easy for cloud providers and clients to set up.Source: A. Threshold-based rules · p. 2

B. Control theory

  • Control-theory autoscaling compares system output with a reference, using feedback to align performance with the targeted SLA.Related approaches include PI controllers, queueing-based estimation, online resource models, and Kalman filters.Source: B. Control theory · p. 2

C. Time series analysis

  • Time-series methods forecast future values from historical observations, but prediction accuracy depends on the application, workload pattern, metric, history window, interval, and technique.Methods reviewed include AR, MA, ARMA, smoothing, pattern matching, signal processing, and neural networks.Source: C. Time series analysis · p. 2

D. Reinforcement learning (RL)

The section introduces reinforcement learning for auto-scaling and describes its implementation in OpenStack. The controller learns scaling actions from observed states and rewards, while Heat and Ceilometer support orchestration and monitoring.

D. Reinforcement learning (RL)

  • RL initializes Q(s, a), selects actions using an ϵ-greedy policy, then updates the last state-action value from the observed next state and reward.For auto-scaling, the agent is the auto-scaler, actions scale resources up or down, and the target is the application.Source: D. Reinforcement learning (RL) · pp. 2–3
  • RL can learn from observations without prior knowledge of the target scenario, but may converge slowly on large problems and requires good Q-function initialization.The paper identifies convergence time and initialization as drawbacks of RL approaches.Source: D. Reinforcement learning (RL) · p. 3

III. OPENSTACK ORCHESTRATION

  • OpenStack provides interrelated services for cloud resources; Nova manages VMs, Ceilometer collects telemetry, and Heat orchestrates application infrastructure.Heat templates define resource stacks and relationships, while Heat with Ceilometer can adjust allocations using monitoring alerts.Source: §III · p. 3
  • Heat templates define minimum and maximum instance counts and scaling actions; Ceilometer checks alarms at intervals and triggers the configured policy when thresholds are passed.The checking and scaling actions run automatically during the application lifecycle.Source: §III · p. 3

IV. ON-POLICY AND OFF-POLICY RL AUTO-SCALING

The paper extends fuzzy auto-scaling with SARSA-based on-policy FSL and Q-learning-based off-policy FQL. Its FRL controller combines fuzzy rules with reinforcement learning to adapt resource provisioning against application goals.

IV. ON-POLICY AND OFF-POLICY RL AUTO-SCALING

  • The paper extends a fuzzy controller with on-policy SARSA-based FSL and relates it to off-policy Q-learning-based FQL.The approaches are presented as distinct learning strategies for fuzzy auto-scaling.Source: §IV · pp. 3–4

A. Reinforcement Learning (RL)

  • In RL, the auto-scaler applies scaling actions to the application, observes the resulting state and reward, and learns actions through trial and error.Actions depend on the current state and environmental parameters such as workload or performance.Source: A. Reinforcement Learning (RL) · p. 4

B. Fuzzy Reinforcement Learning (FRL)

  • FRL uses fuzzy rules to guide resource provisioning in a MAPE-K loop that monitors application characteristics, checks goals, and adapts allocation.The goals—SLA, cost, and response time—are reflected in the reward function.Source: B. Fuzzy Reinforcement Learning (FRL) · p. 4
  • The fuzzy controller partitions workload and response-time inputs into fuzzy sets and applies IF-THEN rules to produce VM scaling actions.For example, high workload and bad response time map to an action that increments the deployed-node count by two; fuzzy processing proceeds through fuzzification, reasoning, and defuzzification.Source: B. Fuzzy Reinforcement Learning (FRL) · p. 4

C. Fuzzy SARSA Learning (FSL)

FSL combines fuzzy control with SARSA to adapt scaling decisions from application feedback. Its on-policy updates incorporate the action actually taken, while fuzzy rules jointly contribute to action values.

C. Fuzzy SARSA Learning (FSL)

  • FSL uses the fuzzy controller with SARSA to learn scaling decisions from application feedback rather than relying on static thresholds.The state is modeled as workload, response time, and VM count.Source: C. Fuzzy SARSA Learning (FSL) · p. 4
  • The fuzzy controller forms a control action as a weighted average of rule consequences, with each rule weighted by its firing degree.The method initializes q-values to zero and uses ε-greedy action selection, reducing ε over time to end exploration.Source: C. Fuzzy SARSA Learning (FSL) · p. 5
  • Because fuzzy actions can combine multiple rules, FSL estimates an action’s Q-value from the q-values of applicable rules and their firing levels.The action-value function represents how desirable an action is in a state based on its return.Source: C. Fuzzy SARSA Learning (FSL) · p. 5
  • The reward reflects both acquired resources, which determine cost, and SLO violations.The controller uses current VM count and response time when calculating the reward.Source: C. Fuzzy SARSA Learning (FSL) · p. 5
  • As an on-policy method, SARSA updates an action’s value using the next action actually selected; γ discounts future rewards and η controls the impact of new rewards.FSL repeats action selection, reward observation, and q-value updates until convergence.Source: C. Fuzzy SARSA Learning (FSL) · p. 5

D. Fuzzy Q-Learning (FQL)

FQL combines fuzzy control with off-policy Q-learning, updating values independently of the policy currently followed. Its example considers finite states and five scaling actions, while the described design can extend to other state and action spaces.

D. Fuzzy Q-Learning (FQL)

  • Unlike SARSA, FQL updates q-values using the largest possible reward from the next state, independently of the agent’s current policy.Its error signal uses the value of the new state rather than the value of the next action actually chosen.Source: D. Fuzzy Q-Learning (FQL) · p. 5
  • The example uses a finite state space and five scaling actions, but the design is described as applicable to other state and action spaces.Convergence is detected when consequent-function changes become negligible in each learning loop.Source: D. Fuzzy Q-Learning (FQL) · pp. 5–6
  • FQL selects an action with ε-greedy exploration, observes the next state and reward, then updates q-values until convergence.The algorithm uses the same fuzzy action and action-quality calculations as FSL.Source: D. Fuzzy Q-Learning (FQL) · p. 6

V. IMPLEMENTATION

The implementation runs the fuzzy RL autoscaler in OpenStack, using Heat and Ceilometer resources alongside a dedicated controller VM. The controller gathers platform information, chooses horizontal scaling actions, and accesses OpenStack APIs through cURL.

V. IMPLEMENTATION

  • The OpenStack implementation uses Heat to orchestrate resources and Ceilometer to collect measurements, with autoscaling groups, scaling policies, and alarms supporting the scaling process.The autoscaling group encapsulates the resources to scale, the policy defines the scaling effect, and alarms specify triggering conditions.Source: §V · p. 6
  • Each autoscaling-group VM runs a simple web server that listens on port 80 and returns an HTML page.The server is configured as user data to run when the instance boots; the implementation uses a CirrOS test image.Source: §V · p. 6
  • The scaling-policy template supports absolute or percentage adjustments, including changing capacity, setting exact capacity, or changing capacity by a percentage.Its properties specify the autoscaling group, adjustment type, and adjustment size.Source: §V · p. 6
  • A dedicated Ubuntu controller VM replaces OpenStack’s threshold-based autoscaling component and enacts scaling decisions made by FQL or FSL.The native approach uses alarms on Ceilometer metrics; the implemented load balancer distributes requests using ROUND_ROBIN.Source: §V · pp. 6–7
  • The controller gathers load-balancer, Ceilometer, and web-server state before choosing scale direction; scaling up launches a server and adds it to the load-balancer pool.The FQL and FSL algorithms run on the controller VM.Source: §V · p. 7
  • The controller uses cURL to authenticate with Keystone and call OpenStack APIs, because direct API access from the controller VM was unavailable.The authenticated request combines an Auth-Token with the Computing Service API Endpoint.Source: §V · p. 7

VI. EXPERIMENTAL COMPARISON

The experimental evaluation is intended to assess FQL and FSL effectiveness, compare their differences, and demonstrate cost improvement for the cloud provider.

VI. EXPERIMENTAL COMPARISON

  • The evaluation aims to assess the effectiveness of FQL and FSL.The passage states the evaluation goal but supplies no measured effectiveness result.Source: §VI · p. 7
  • The evaluation also aims to identify differences between the two approaches.The passage does not specify the differences found.Source: §VI · p. 7
  • The evaluation is presented as demonstrating cost improvement for the cloud provider.No amount or metric for the cost improvement is provided in this passage.Source: §VI · p. 7

A. Experimental setup and benchmark

The study compares FQL and FSL in OpenStack under synthetic workload patterns and real website traces, using fixed-VM baselines to benchmark adaptive scaling.

A. Experimental setup and benchmark

  • FQL and FSL were implemented and tested in OpenStack with the VM range set to 1–5 to examine behavior under heavy request traffic.The authors note that larger VM limits could also be used.Source: A. Experimental setup and benchmark · p. 7
  • Tests cover predictable bursting, variations, and ON&OFF workloads, each with 10–100 concurrent users per second.The study also validates the approaches on Wikipedia and FIFA World Cup access traces.Source: A. Experimental setup and benchmark · p. 7
  • Siege generated N requests per second for 10 minutes at each concurrent-user count to measure performance, including average response time.The traces represent requests or users accessing the websites over time.Source: A. Experimental setup and benchmark · p. 7
  • Learning used η = 0.1 and γ = 0.8, with exploration reduced after convergence to a minimum ϵ of 0.2.The approaches begin with exploration, then enter a balanced exploration–exploitation phase.Source: A. Experimental setup and benchmark · pp. 7–8
  • The proposed approaches were compared with fixed minimum- and maximum-permitted VM baselines, VM#1 and VM#5, reflecting under- and over-provisioning; Figures 10–13 show observed response times and VM-use percentages for FSL and FQL.Source: A. Experimental setup and benchmark · p. 8, Figure 10 · p. 8, Figure 11 · p. 8

B. Comparison of effectiveness

Both controllers adapt resources to workload and response-time demands, but their relative scaling quality varies by workload pattern. Results on two real traces also favor FSL, with different margins and learning states.

B. Comparison of effectiveness

  • Both FQL and FSL dynamically adjust VM counts to meet response-time QoS expectations, unlike fixed-VM baselines.The response-time comparisons are shown in Figures 10 and 11.Source: B. Comparison of effectiveness · p. 9
  • For predictable bursts FSL finds a significantly better solution, FQL is superior for variations, and their ON&OFF solutions are broadly similar.The authors attribute FSL’s bursting advantage to faster on-policy learning; FSL’s faster convergence may leave its policy less suitable for highly fluctuating, non-periodic variations.Source: B. Comparison of effectiveness · p. 9
  • Expert initial knowledge improves results over non-expert knowledge across the presented cases.Each algorithm was run several times with both knowledge types, represented by different colors in the figures.Source: B. Comparison of effectiveness · pp. 8–9
  • On the Wikipedia trace FSL performs slightly better than FQL; on the bursty, unpredictable FIFA trace FSL has better measured response time while FQL remains in exploration/exploitation.Figures 14 and 15 show the respective trace response times.Source: B. Comparison of effectiveness · p. 9, Figure 14 · p. 9, Figure 15 · p. 9

C. Comparison of cost-effectiveness of scaling

Both approaches scale VM counts with workload, and the reported average maximum VM-use percentages indicate resource use while meeting QoS requirements.

C. Comparison of cost-effectiveness of scaling

VII. CONCLUSION

The conclusion presents fuzzy rule-based FQL and FSL as autoscaling approaches for cloud workloads where threshold selection is challenging. Their suitability is demonstrated in OpenStack, with performance differences across workload patterns and convergence times.

VII. CONCLUSION

  • Setting useful thresholds for traditional autoscaling is challenging, motivating machine-learning approaches that complement or replace expert knowledge for workload fluctuations.Source: §VII · p. 10
  • The paper compares fuzzy rule-based FQL and FSL, reporting that both can scale resources to meet QoS requirements while reducing provider costs through improved utilization.The conclusion notes that SARSA’s per-step reward improves solution quality for periodic workloads.Source: §VII · p. 10
  • The authors identify an autonomic autoscaler combining fuzzy logic and reinforcement learning, tested on different workloads in an open-source cloud platform, as a promising direction beyond threshold-based approaches.Source: §VII · p. 10
  • The study assumes antecedent membership functions remain fixed once specified; future work includes partial observability, incremental clustering for dynamic membership functions, and other resource types such as containers.Source: §VII · p. 10
Loading 1705.07114v1…