PaperlayerView exact revision on arXiv

Source-linked AI summary

DB-BERT: a Database Tuning Tool that "Reads the Manual"

Immanuel Trummer

arXiv:2112.10925v1cs.DBcs.AIcs.CL

TL;DR

Tuning hundreds of database parameters is difficult, and automated methods can depend on costly benchmark iterations and expert-selected inputs. DB-BERT analyzes tuning documents with BERT and reinforcement learning to guide configuration choices. Its experiments report significant benefits from text analysis across benchmarks, metrics, and database systems.

  • Problem

    Automated database tuning faces hundreds of parameters, costly iterations, and a need for experts to select knobs and value ranges.

    Source: §1 · p. 1

  • Method

    DB-BERT extracts parameter-value hints from text and trains with DBMS feedback to guide tuning configurations.

    Source: §1 · pp. 1–2

  • Results

    Across TPC-C and TPC-H, throughput and latency, and MySQL and Postgres, experiments show DB-BERT benefits significantly from text analysis.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Experiments show that extracting tuning hints from text documents leads to significantly better tuning results.

    Source: §9 · p. 13

    DB-BERT currently optimizes metrics such as latency or throughput that are easy to measure; extending it to difficult-to-measure objectives remains future work.

    Source: §9 · p. 13

Abstract

from arXiv · show

DB-BERT is a database tuning tool that exploits information gained via natural language analysis of manuals and other relevant text documents. It uses text to identify database system parameters to tune as well as recommended parameter values. DB-BERT applies large, pre-trained language models (specifically, the BERT model) for text analysis. During an initial training phase, it fine-tunes model weights in order to translate natural language hints into recommended settings. At run time, DB-BERT learns to aggregate, adapt, and prioritize hints to achieve optimal performance for a specific database system and benchmark. Both phases are iterative and use reinforcement learning to guide the selection of tuning settings to evaluate (penalizing settings that the database system rejects while rewarding settings that improve performance). In our experiments, we leverage hundreds of text documents about database tuning as input for DB-BERT. We compare DB-BERT against various baselines, considering different benchmarks (TPC-C and TPC-H), metrics (throughput and run time), as well as database systems (Postgres and MySQL). In all cases, DB-BERT finds the best parameter settings among all compared methods. The code of DB-BERT is available online at https://itrummer.github.io/dbbert/.

1 INTRODUCTION

DB-BERT uses natural-language analysis of tuning documents to guide database configuration, addressing the difficulty of tuning hundreds of parameters without expert-selected knobs and value ranges. Its experiments show significant benefits from text analysis across multiple benchmarks, metrics, and database systems.

1 INTRODUCTION

  • Tuning hundreds of DBMS parameters manually is difficult, while machine-learning tuning can require hundreds of iterations and expert-selected knobs and value ranges.The paper aims to replace that expert input with information gained automatically from text documents.Source: §1 · p. 1
  • DB-BERT extracts hints that recommend parameter values, including absolute or relative values and explicit or implicit parameter references.Relative recommendations may require system properties, such as RAM, to translate into concrete values.Source: §1 · p. 1
  • A fine-tuned BERT model identifies hint-containing snippets, translates hints, may deviate from proposed values, and assigns weights to conflicting hints.The choices for these hint-processing tasks are represented with text labels.Source: §1 · p. 1
  • DB-BERT uses Double Deep Q-Network reinforcement learning to select configurations from benchmark rewards, and trains independently of any particular system or benchmark.Training rewards admissible settings and penalizes rejected settings or settings that significantly reduce performance on a simple workload.Source: §1 · p. 2
  • Unlike the prior text-only approach, DB-BERT uses performance measurements to adapt recommendations to particular benchmarks and metrics.The prior approach requires manually labeled hint translations, whereas DB-BERT uses unlabeled input text.Source: §1 · p. 2
  • Across TPC-C and TPC-H, throughput and latency, and MySQL and Postgres, experiments show DB-BERT benefits significantly from text analysis.The paper compares DB-BERT with prior text-based work and database-tuning methods without text.Source: §1 · p. 2
  • DB-BERT combines natural-language documents with benchmark runtime feedback to guide database tuning and evaluates the approach across multiple benchmarks, metrics, and systems.The documents are mined from large collections using relevant Google queries.Source: §1 · p. 2

2 BACKGROUND AND RELATED WORK

The section situates DB-BERT among pre-trained language models, natural-language database interfaces, reinforcement learning, and database tuning. It distinguishes DB-BERT through its handling of multiple hints, conflicts, and tuning feedback without annotated hints.

2.1 Pre-Trained Language Models

  • Pre-trained language models use readily available training tasks such as masked language modeling, reducing the training data and computational overhead needed for other NLP tasks.The section identifies the Transformer architecture as enabling massively parallel training of large models.Source: §2.1 · p. 2

2.2 Natural Language Query Interfaces

  • Unlike text-to-SQL systems that translate one sentence into one query, DB-BERT extracts multiple hints from multi-sentence passages and prioritizes conflicting hints from multiple sources.DB-BERT also does not assume labeled training samples, unlike most prior text-to-SQL work.Source: §2.2 · pp. 2–3

2.3 Reinforcement Learning

  • DB-BERT combines reinforcement learning with NLP for database parameter tuning, a performance-tuning application not considered by prior work on instruction following.The section identifies Double Deep Q-Networks as the reinforcement-learning algorithm DB-BERT uses.Source: §2.3 · p. 3

2.4 Database Tuning

  • Prior NLP-enhanced database tuning uses supervised learning and fixed recommendations without runtime feedback, while DB-BERT supports implicit hints and needs no annotated tuning hints.The paper compares the approaches experimentally in Section 8.Source: §2.4 · p. 3
  • DB-BERT extends database parameter tuning by adding natural-language documents to the optimization input.The section places parameter tuning among database optimization problems addressed with machine learning.Source: §2.4 · p. 3

3 PROBLEM MODEL

The problem model formalizes tuning as selecting valid parameter values to optimize a benchmark and extends it with text documents and system properties. DB-BERT uses those documents to identify and translate tuning hints, while assuming access to a DBMS instance.

3 PROBLEM MODEL

  • A configuration assigns valid values to selected DBMS parameters, with unspecified parameters retaining their defaults.Database tuning seeks such assignments to optimize a benchmark's workload and performance metric.Source: §3 · p. 3
  • NLP-enhanced tuning adds a collection of hint-bearing documents and system properties to the benchmark, using text analysis to tune DBMS parameters.The implementation considers integer, numeric, and Boolean parameters and assumes access to a DBMS instance to validate parameter names and types.Source: §3 · p. 3
  • A tuning hint is modeled as text, a parameter, and a value; the parameter may be implicit, and one snippet may contain multiple hints.The suggested value may require translation rather than being the final concrete parameter value.Source: §3 · pp. 3–4
  • A translated hint maps a suggested value to a parameter formula that may use system properties, such as RAM, to compute a concrete setting.For the example recommending 25% of available RAM for shared_buffers, the formula evaluates to 2 GB on a system with 8 GB RAM.Source: §3 · p. 4

4 SYSTEM OVERVIEW

DB-BERT uses database text to generate tuning hints and combines them with benchmark feedback to search for DBMS configurations. Its reinforcement-learning process translates, adapts, and weights hints while iteratively evaluating configurations.

4 SYSTEM OVERVIEW

  • DB-BERT takes the benchmark, text documents, and hardware properties as inputs; RAM, core count, and disk space support translating relative recommendations.It can also process arbitrary parameters when their recommended values are absolute.Source: §4 · p. 4
  • DB-BERT uses benchmark run-time feedback as well as text, so it requires a connection to the DBMS instance being tuned.Accepted configurations with improved performance receive higher rewards; rejected settings are penalized.Source: §4 · pp. 4–5
  • DB-BERT iteratively builds configurations from prioritized hints, tests them on the benchmark, and uses rewards based on performance to improve later decisions.The search continues until the user interrupts or the specified optimization time limit is reached.Source: §4 · p. 4
  • For each hint, DB-BERT decides how to translate its text, whether to adjust the recommended value, and how to weight it against conflicting hints.It aggregates the weighted hints into a small set of configurations for trial runs.Source: §4 · p. 4
  • DB-BERT uses BERT to estimate the value of choices from tuning-hint text paired with natural-language labels, refining model weights from rewards.The labels express the semantics of candidate choices.Source: §4 · p. 5

5 EXTRACTING CANDIDATE HINTS

DB-BERT creates candidate tuning hints from text snippets, including implicit parameter references and possible values. It then orders hints to consider frequently mentioned parameters early without exhausting one parameter's hints first.

5 EXTRACTING CANDIDATE HINTS

  • Candidate hints combine parameter references with possible values, including numbers found in text and default Boolean values 0 and 1.Because this Cartesian product can pair unrelated parameters and values, the translation stage must distinguish actual hints from erroneous candidates.Source: §5 · p. 5
  • Implicit parameter references are identified by selecting the parameter name whose BERT encoding has the smallest cosine distance from the text encoding.Explicit parameter references are extracted as well.Source: §5 · p. 5
  • Hints for parameters mentioned more often are prioritized, but DB-BERT considers only a fixed number per parameter before moving to the next.It groups hints by parameter and cycles through parameters in decreasing order of mention frequency, adding up to l hints each pass.Source: §5 · pp. 5–6, Figure 3 · p. 6

6 TRANSLATING SINGLE HINTS

DB-BERT translates each tuning hint into a formula through reinforcement-learning decisions, using DBMS feedback to reward admissible translations that improve performance. A BERT-based agent selects actions from natural-language descriptions of their choices.

6 TRANSLATING SINGLE HINTS

  • Hint translation is modeled as sequential decisions over partially specified formulas, with DBMS feedback penalizing inadmissible values and rewarding performance improvements.The decisions specify parts of a formula that maps a hint to a parameter value.Source: §6 · p. 6

6.1 Learning Environment

  • The learning environment starts each hint with an empty formula and first decides whether it is erroneous and whether its value is relative or absolute.For relative values, the formula multiplies the text value by a system property; setting S_1 = 1 treats absolute values in the same framework.Source: §6.1 · p. 7
  • A later decision selects a multiplicator to adapt the proposed value; rejected settings end with negative reward, while admissible settings are benchmarked against defaults.The benchmark reward uses run-time delta for OLAP workloads such as TPC-H and throughput delta for OLTP workloads such as TPC-C.Source: §6.1 · p. 7

6.2 Learning Agent

  • The learning agent frames each action choice as a multiple-choice question, comparing the hint text with natural-language labels describing the options.BERT model weights are fine-tuned using received rewards.Source: §6.2 · pp. 7–8
  • During generic training, DB-BERT masks parameter names so the agent does not learn system-specific names that do not generalize across systems.For system- and benchmark-specific tuning, parameter names are not masked.Source: §6.2 · p. 8

7 AGGREGATING HINTS

DB-BERT aggregates batches of potentially conflicting hints by assigning learned weights, then evaluates configurations assembled from the weighted recommendations. It uses performance feedback and penalties for rejected configurations to guide tuning.

7 AGGREGATING HINTS

  • Each episode starts from the default configuration, and the number of hints considered limits how many settings can change.The paper notes that tuning a small number of knobs is typically sufficient for near-optimal performance.Source: §7 · p. 8
  • DB-BERT assigns each admissible hint a learned weight, then uses weighted hints to assemble configurations for benchmark evaluation.Algorithm 6 translates candidate hints and assigns weights; Algorithm 7 combines weighted hints and evaluates configurations.Source: §7 · pp. 8–9
  • When selecting parameter values, DB-BERT greedily minimizes the maximum weighted distance from hint-proposed values to selected values.It chooses a limited number of values per parameter to cover the recommendations; some knobs also permit only specific values, so proposed values cannot simply be averaged.Source: §7 · p. 9
  • The value-selection heuristic has a 1 − e^-1 approximation guarantee because its distance reduction is non-negative, monotone, and submodular.The passage attributes the guarantee to the greedy algorithm of Nemhauser.Source: §7 · p. 9
  • Configurations that the DBMS rejects receive a penalty, while accepted configurations are rewarded according to benchmark performance.The weighted-hint evaluation returns the highest reward obtained by any evaluated configuration.Source: §7 · p. 9

8 EXPERIMENTS

The experiments evaluate DB-BERT on Postgres and MySQL across TPC-H and TPC-C, comparing it with tuning approaches with and without NLP. DB-BERT finds better solutions than the compared methods, while further analyses examine document choice and design variants.

8.1 Experimental Setup

  • The experiments cover Postgres 13.2 and MySQL 8.0, hundreds of numerical and Boolean parameters, TPC-H and TPC-C, and five runs with 25 minutes of tuning per run.The setup tunes 232 Postgres and 266 MySQL parameters; TPC-H uses scaling factors one and ten, and TPC-C uses scaling factor 20.Source: §8.1 · p. 9

8.2 Tuning Text Documents

  • The Postgres and MySQL tuning collections include manuals, blog entries, and forum discussions, with parameter mentions and proposed values distributed non-uniformly.Most parameters and value assignments appear only once, while buffer-size parameters feature prominently among frequently mentioned parameters.Source: §8.2 · p. 10

8.3 Training

  • DB-BERT trains without annotated tuning hints, using database feedback for valid settings and performance, with masked parameter names to focus learning on hint sentence structure.Training uses the opposite system’s document collection and a synthetic benchmark whose performance reward is scaled down to avoid specialization to it.Source: §8.3 · p. 11

8.4 Comparison with Baselines

  • Across the tested benchmarks, DB-BERT finds significantly better solutions on average than the text-based baselines and solutions of higher quality than DDPG++ within the tuning time.The comparison covers TPC-H execution time and TPC-C throughput; DB-BERT also reaches a near-optimal solution faster than Prior-Main analyzes all documents.Source: §8.4 · p. 11, Figure 7 · p. 12, Figure 8 · p. 12

8.5 Further Analysis

  • Both processing hints in document order and discarding implicit hints degrade DB-BERT’s TPC-H tuning performance.Document-order processing delays attention to the most relevant parameters, while discarding implicit hints reduces the available hint set.Source: Figure 9 · p. 12, §8.5 · p. 12
  • For Postgres TPC-H, benchmark-specific text helps DB-BERT converge to near-optimal solutions faster and find a slightly better solution than generic Pg100 text.Prior-Main instead extracts no hints from the single specialized blog post and performs worse with the smaller input.Source: Figure 10 · p. 13, §8.5 · p. 13

9 CONCLUSION AND OUTLOOK

DB-BERT extracts tuning hints from text documents, and experiments show these hints lead to significantly better tuning results. Its current optimization scope is limited to easily measured metrics, with broader objectives left for future work.

9 CONCLUSION AND OUTLOOK

  • DB-BERT extracts tuning hints from text documents, and experiments show these hints lead to significantly better tuning results.Source: §9 · p. 13
  • Current optimization is limited to metrics such as latency or throughput that can be easily measured.Source: §9 · p. 13
  • Future work will extend DB-BERT to difficult-to-measure objectives, including performance settings that must account for risks such as data loss.The authors propose extracting information about such metrics and risks from manuals.Source: §9 · p. 13
Loading 2112.10925v1…