Source-linked AI summary
BestConfig: Tapping the Performance Potential of Systems via Automatic Configuration Tuning
Yuqing Zhu, Jianxun Liu, Mengying Guo, Yungang Bao, Wenlong Ma, Zhuoyue Liu, Kunpeng Song, Yingchun Yang
TL;DR
Many systems expose numerous parameters whose best settings depend on workloads, making configuration tuning costly and difficult for users. BestConfig automates this task with an extensible architecture, divide-and-diverge sampling, and recursive bound-and-search within a resource limit. Across six systems and workloads, it reports substantial throughput improvements and runtime reductions from configuration adjustment alone.
Problem
Configuration tuning is costly and difficult because heterogeneous systems and workloads create complex parameter interactions that users must optimize with limited samples.
Method
BestConfig combines an extensible closed-loop architecture with divide-and-diverge sampling and recursive bound-and-search optimization.
Results
BestConfig improved throughput across Tomcat, Cassandra, and MySQL and reduced Hive and Spark join runtimes compared with default settings.
Takeaways & Limitations
BestConfig provides an open-source system for tuning cloud systems across varied deployments and workloads within a specified resource limit.
Takeaways & Limitations
BestConfig requires a resource limit with enough samples relative to the number of parameters and cannot improve systems whose tuned parameters do not affect performance.
Abstract
from arXiv · showhide
An ever increasing number of configuration parameters are provided to system users. But many users have used one configuration setting across different workloads, leaving untapped the performance potential of systems. A good configuration setting can greatly improve the performance of a deployed system under certain workloads. But with tens or hundreds of parameters, it becomes a highly costly task to decide which configuration setting leads to the best performance. While such task requires the strong expertise in both the system and the application, users commonly lack such expertise. To help users tap the performance potential of systems, we present BestConfig, a system for automatically finding a best configuration setting within a resource limit for a deployed system under a given application workload. BestConfig is designed with an extensible architecture to automate the configuration tuning for general systems. To tune system configurations within a resource limit, we propose the divide-and-diverge sampling method and the recursive bound-and-search algorithm. BestConfig can improve the throughput of Tomcat by 75%, that of Cassandra by 63%, that of MySQL by 430%, and reduce the running time of Hive join job by about 50% and that of Spark join job by about 80%, solely by configuration adjustment.
1 INTRODUCTION
BestConfig addresses costly, expertise-intensive configuration tuning for diverse systems and workloads by automating search within resource limits. Its experiments report substantial performance gains or runtime reductions through configuration adjustment alone.
- Motivation: Configuration complexity overwhelms users and can cause errors or poor performance under atypical workloads.Hadoop alone has more than 180 configuration parameters.
- Motivation: More than 11 times performance gain can result from changing MySQL’s query_cache_type from zero to one for an application workload.The gain is especially significant for recurring workloads.
- Challenges: Configuration tuning must handle varied systems, deployments, performance goals, and application workloads.Examples include Hadoop, Spark, MySQL, and Tomcat, with goals such as throughput, latency, or running time.
- Challenges: Irregular performance surfaces and limited samples make general configuration tuning difficult, while existing solutions are not generally applicable.The paper contrasts MySQL, Tomcat, and Spark performance surfaces and notes that related models do not cover complicated cases.
- Approach: BestConfig combines an extensible architecture with divide-and-diverge sampling and recursive bound-and-search optimization under a resource limit.The system uses a sample-test-optimize process in a closed loop.
- Results: 75% higher Tomcat throughput, 63% higher Cassandra throughput, and 430% higher MySQL throughput were achieved versus the default configuration.The same evaluation reduced Hive join runtime by about 50% and Spark join runtime by about 80%.
2 BACKGROUND AND MOTIVATION
The paper motivates automatic configuration tuning by showing that workload-specific optima, complex interactions, and high-dimensional spaces make manual tuning difficult. It frames the central challenge as achieving useful optimization with limited real-system samples across heterogeneous environments.
- Background and Motivation: A configuration setting that performs best for one workload may not be optimal for another, so default settings generally cannot optimize all workloads.For MySQL, query_cache_type matters under uniform-read but has no obvious performance relation under a Zipfian read-write workload.
- Background and Motivation: Manual tuning requires heuristics, repeated configuration changes, and workload tests, and can consume substantial time.Tuning MySQL once took five junior employees about half a year to find an appropriate setting.
- Background and Motivation: Even experienced developers face irregular parameter interactions that are difficult to reason about or express with simple models.The paper cites interacting PostgreSQL cache-size and buffer-size parameters as an example.
- Challenges: Automatic tuning must support varied systems, deployment environments, workloads, and performance goals, including goals that must be maximized or minimized.Examples include reducing Spark running time and increasing MySQL throughput without worsening memory usage.
- Challenges: High-dimensional parameter spaces require broad coverage, but collecting enough real performance samples is costly and large-scale simulation is infeasible.The tuning method must therefore produce results from limited samples.
3 BESTCONFIG DESIGN
BestConfig uses a flexible closed-loop architecture to tune deployed systems against workload-specific goals within a resource limit. It combines sampling and performance optimization so each informs the other while configurations are tested on the target system.
- Design Overview: BestConfig automatically finds a configuration setting that optimizes a deployed system’s performance under a specific workload and resource limit.The system under tune is called the SUT.
- Performance Metric: Multiple performance goals are combined into a scalar maximization objective through a user-defined utility function.Examples include throughput divided by latency and throughput constrained by memory usage.
- Architecture: Loosely coupled components connect system manipulation, workload generation, sampling, and performance optimization for different deployed systems and workloads.The architecture is designed to require only minor changes when reused.
- Sampling and Optimization: BestConfig combines DDS sampling with RBS optimization so sampling covers less-explored subspaces and optimization exploits sampling information during local search.The two methods are designed as a complete solution rather than separate stages.
- Tuning Process: The closed loop generates constrained settings, tests them on the SUT, converts results through the utility function, and searches for the best observed configuration.Additional loops run when the resource limit permits more tests.
4 DDS & RBS IN COOPERATION
BestConfig combines DDS, which spreads samples across a high-dimensional parameter space under a resource limit, with RBS, which recursively searches bounded regions around promising configurations. Together, they balance coverage, scalability, and progressively finer optimization.
- 4.1 DDS: Divide & Diverge Sampling: DDS divides parameter ranges into intervals and aligns interval permutations so each parameter interval is represented once across k diverse samples.This avoids the exponential sample count of full gridding while spreading points across the space.
- 4.1 DDS: Divide & Diverge Sampling: DDS scales sampling directly with the resource limit by setting k to the number of allowed tests and increasing coverage as more tests become available.Repeated sampling can reuse the initial division and explore previously unconsidered subspaces.
- 4.2 RBS: Recursive Bound & Search: RBS starts from the best point in an initial sample set, bounds a region around it, and recursively samples around each newly best point.The bounds use neighboring represented parameter values around the current best configuration.
- 4.2 RBS: Recursive Bound & Search: RBS allocates a test budget across sample-set sizes and search rounds, enabling finer searches in promising bounded subspaces when more tests are available.If a round finds no improvement, RBS can restart sampling over the complete parameter space to avoid remaining in a sub-optimal bounded region.
- 4.3 Why Combining DDS with RBS Works: Under uniform sampling, DDS is expected to find a point outperforming the default after about n = 1/(1 − ϕ_Di(y0)) samples in subspace D_i.The guarantee is asymptotic: more samples eventually produce a point with performance greater than the default configuration.
5 EVALUATION
BestConfig is evaluated against model-based methods and across diverse systems, workloads, sampling strategies, and resource settings. The experiments show broad performance gains, while also identifying parameter count, sampling, rounds, and deployment conditions as important boundaries.
- 5.1 Infeasibility of Model-based Methods: COMT and GPR produce inaccurate or noncompetitive configuration recommendations because their assumptions do not hold for highly irregular system performance surfaces.COMT assumes linear parameter-performance relations, while GPR assumes a differentiable performance function; increasing samples does not reliably improve their results.
- 5.2 Automatic Configuration Tuning Results: BestConfig improves performance across all evaluated tuning experiments, including Hive+Hadoop with 109 parameters.The evaluation covers Cassandra, MySQL, Tomcat, Spark, and Hive+Hadoop using a sample set of 100 and one round in the reported experiments.
- 5.2 Automatic Configuration Tuning Results: 50%: BestConfig reduces the running time of the Hive+Hadoop Join job when the sample-set size is increased to 500.The result is reported for the HiBench Join workload and demonstrates improved tuning with a larger sample set.
- 5.2 Automatic Configuration Tuning Results: 75% Tomcat throughput, about 25% Cassandra throughput, 430% MySQL throughput, and about 50% and 80% lower Hive and Spark Join running times are reported from configuration adjustment.These figures are summarized relative to the paper’s reported tuning comparisons, solely through configuration adjustments.
- 5.3 Sampling and Search: DDS samples points with the best performance as much as three times more often than gridding and uniform sampling in the bound-and-sample round.DDS with RBS also makes progress in earlier rounds than LHS with RBS, while RBS+DDS outperforms RRS+LHS under the same resource limit.
- 5.4 Varied Sample-Set Sizes & Rounds: BestConfig usually tunes better with more resources and rounds, but the required sample budget depends on the number of parameters and outcomes vary with system, deployment, workload, and parameter set.The paper recommends resource limits proportional to parameter count and reports that unrelated parameters, untuned bottlenecks, or an already optimal default can prevent improvement.
6 USE CASE: TOMCAT FOR CLOUD+
BestConfig was deployed on fully loaded Tomcat servers in Huawei’s Cloud+ applications, where configuration adjustment improved performance without changing CPU utilization.
- Deployment Context: The deployment ran virtual machines with eight cores, including four assigned to network communication.Under the default setting, network-serving cores were fully loaded while the other four operated at about 80% utilization.
- Deployment and Results: 4% higher overall throughput was achieved while CPU utilizations remained the same.Stability tests demonstrated that the improvement was maintained stably.
- Deployment and Results: BestConfig improved a fully loaded system simply by adjusting its configuration setting.The result expanded understanding of the deployed system through automatic configuration tuning.
7 RELATED WORK
Prior configuration-tuning methods target particular systems, rely on restrictive assumptions, or require costly sampling, motivating BestConfig’s general-system design.
- Sampling and Search: BestConfig’s DDS provides wider coverage and scalability than LHS by remembering sampled subspaces and resampling toward broader parameter-space coverage.
- System-Specific Methods: Earlier Web-system methods either tune few parameters or require many initial tests, making large-scale sampling costly.The related approaches include hill climbing, simulated annealing, genetic algorithms, and reinforcement learning.
- System-Specific Methods: Database methods such as iTuned assume smooth performance surfaces, an assumption that may not hold for Tomcat or some MySQL configurations.Otter-Tune additionally uses feature selection to reduce the number of parameters.
- System-Specific Methods: Hadoop-specific methods depend on strong system knowledge or large databases of costly samples, limiting their applicability to general systems.
8 CONCLUSION
BestConfig automatically tunes deployed systems under resource limits using an extensible architecture, DDS sampling, and RBS optimization, with improvements observed across six widely used systems.
- BestConfig finds configuration settings that optimize a deployed system’s performance for a specific workload within a given resource limit.
- BestConfig combines a flexible architecture with the scalable DDS sampling method and scalable RBS optimization algorithm.The system is available as an open-source package for developers to use and extend.
- Performance improvements were observed after tuning six widely used systems.
- BestConfig improved a fully loaded Tomcat deployment by adjusting configuration settings alone.