PaperlayerView exact revision on arXiv

Source-linked AI summary

Transfer Learning for Performance Modeling of Configurable Systems: An Exploratory Analysis

Pooyan Jamshidi, Norbert Siegmund, Miguel Velez, Christian Kästner, Akshay Patel, Yuvraj Agarwal

arXiv:1709.02280v1stat.MLcs.PFcs.SE

TL;DR

Performance models can be costly to build for highly configurable systems, and it is unclear when their knowledge transfers across environments. The authors empirically examine relatedness across systems and environmental changes, finding that small changes often allow linear transfer while severe changes can preserve other useful knowledge.

  • Problem

    It is unclear why and when transfer learning works for performance modeling, despite the cost of learning models across highly dimensional configuration spaces.

    Source: §I · p. 1

  • Method

    The authors analyze hypotheses about relatedness between source and target performance models across 36 environmental changes in four subject systems.

    Source: B. Methodology · p. 4, §IV · p. 5

  • Results

    Small environmental changes often support linear model transfer, while larger changes can preserve influential-option, interaction, or invalid-configuration knowledge.

    Source: §I · p. 2, B. Implications for transfer learning… · p. 8, §X · p. 9

  • Takeaways & Limitations

    Transfer learning can draw on diverse forms of shared performance knowledge, including partial knowledge that may guide sampling after severe changes.

    Source: §I · p. 2, §X · p. 9

Abstract

from arXiv · show

Modern software systems provide many configuration options which significantly influence their non-functional properties. To understand and predict the effect of configuration options, several sampling and learning strategies have been proposed, albeit often with significant cost to cover the highly dimensional configuration space. Recently, transfer learning has been applied to reduce the effort of constructing performance models by transferring knowledge about performance behavior across environments. While this line of research is promising to learn more accurate models at a lower cost, it is unclear why and when transfer learning works for performance modeling. To shed light on when it is beneficial to apply transfer learning, we conducted an empirical study on four popular software systems, varying software configurations and environmental conditions, such as hardware, workload, and software versions, to identify the key knowledge pieces that can be exploited for transfer learning. Our results show that in small environmental changes (e.g., homogeneous workload change), by applying a linear transformation to the performance model, we can understand the performance behavior of the target environment, while for severe environmental changes (e.g., drastic workload change) we can transfer only knowledge that makes sampling more efficient, e.g., by reducing the dimensionality of the configuration space.

I. INTRODUCTION

The paper asks when performance knowledge transfers across environments, since a changed environment may require costly model building from scratch. Its exploratory study finds that small changes often permit linear transfer, while severe changes can still preserve useful partial knowledge.

I. INTRODUCTION

  • Because configuration spaces grow exponentially and interactions and constraints complicate modeling, a changed environment may require a new model learned from scratch.That limits reuse of existing performance models across environments.Source: §I · p. 1
  • The study compares performance behavior across workload, hardware, and software-version changes, testing hypotheses about shared knowledge across 36 changes in four configurable systems.The selected changes included simple hardware changes and severe desktop-to-cloud changes.Source: §I · p. 2
  • For small environmental changes, performance models can often be transferred linearly; severe changes may preserve influential-option or invalid-region knowledge useful for guided sampling.The authors report these patterns across the environmental changes they explored.Source: §I · p. 2
  • The authors formulate hypotheses spanning obvious relationships such as correlation and subtler possibilities such as stable influential options or invalid regions.They present the findings as opportunities for transfer approaches based on more than correlation.Source: §I · p. 2

II. INTUITION

The paper frames transfer learning as reusing performance knowledge when a configurable system moves to a new environment. Such reuse may avoid rebuilding a model from scratch when relevant performance behavior remains stable.

II. INTUITION

  • Performance models support debugging, tuning, design-time evolution, and runtime adaptation; transfer learning can reuse prior-environment knowledge instead of remeasuring configurations from scratch.The paper links reuse to cheaper, faster, and more accurate performance models for changing environments.Source: §II · p. 2

A. Environmental changes

  • An environment is determined by hardware, workload, and software version, although the study limits its scope to these common performance-affecting changes.The paper notes that other changes, such as a JVM upgrade, are possible.Source: A. Environmental changes · p. 2

B. Preliminary concepts

  • A configuration space is the Cartesian product of option domains; the paper describes binary options while noting that the definitions generalize to finite domains.A configuration selects a value for every option.Source: 1) Configuration and environment space · p. 3

1) Configuration and environment space:

This section establishes the paper’s notation for configurations and environments, performance measurements, and several forms of performance knowledge that may be compared across environments.

1) Configuration and environment space:

  • An environment change can vary workload while holding hardware and software version fixed, using the notation ec: [h, w1→w2, v].An environmental condition is represented by hardware, workload, and version variables.Source: 1) Configuration and environment space · p. 3

2) Performance model:

  • A performance model maps a configuration and environment to performance, while its training data pairs measured configurations with their performance values.The number of measurements is denoted by n.Source: 2) Performance model · p. 3

3) Performance distribution:

  • A performance distribution represents a probability distribution over performance measures for a system’s environmental conditions, estimated from measured values using kernel density estimation.The authors use this representation as a relaxed alternative to comparing performance models.Source: 3) Performance distribution · p. 3

4) Influential option:

  • An influential option is one with a statistically significant effect on system performance.The paper examines whether such options remain influential across environments.Source: 4) Influential option · p. 3

5) Options interaction:

  • An option interaction is a nonlinear effect in which two options’ combined influence differs from the sum of their individual influences.The paper uses interactions to capture effects that individual option influences do not compose linearly.Source: 5) Options interaction · p. 3

6) Invalid configuration:

The paper treats transferable knowledge broadly, from models and distributions to option effects and invalid configurations. Its research questions examine how much of this knowledge remains related after environmental changes.

6) Invalid configuration:

C. Transferable knowledge

  • Transferable knowledge is source-environment information that can contribute to a faster, cheaper, more accurate, or more reliable target model.Examples include learned models, dependency graphs, and option interactions; configuration knowledge can also guide target-environment sampling.Source: C. Transferable knowledge · p. 3, A. Research questions · p. 4

A. Research questions

  • The research questions test consistency in overall behavior, influential options, option interactions, and invalid configurations across environments.The first question considers linear shifts and then weaker forms of relatedness; the others ask whether option influence, interactions, and invalidity are preserved.Source: A. Research questions · p. 3

B. Methodology

The study compares performance behavior across 36 environmental changes using measurements from four configurable systems. It evaluates progressively weaker assumptions about how source and target environments are related, across hardware, workloads, and versions.

B. Methodology

  • Performance was measured across configurations, then compared over 36 changes involving hardware, workloads, and software versions.Source: B. Methodology · p. 4
  • The analysis tests progressively relaxed hypotheses about source–target relatedness; stronger supported assumptions imply more informative knowledge is available to transfer.Source: B. Methodology · p. 4
  • Expected change severity was recorded in advance, from small processor changes to replacing a desktop with a cloud virtual machine.Source: B. Methodology · p. 4

C. Subject systems

  • SPEAR measurements covered all 16,384 configurations, four SAT-problem workloads, three hardware systems, and two solver versions.Source: C. Subject systems · p. 4
  • x264 used 4,000 randomly sampled configurations, tested on three videos, two hardware systems, and three versions.Source: C. Subject systems · p. 4
  • SQLite covered 1,000 configurations across two hardware platforms, two versions, and four query workloads; SaC covered 71,267 configurations and 10 demo programs.Source: C. Subject systems · p. 4

IV. PERFORMANCE BEHAVIOR CONSISTENCY (RQ1)

Performance behavior was often linearly related across environments, especially for hardware and lower-severity workload changes, but other transferable relationships remained when linear correlation was weak. Stability depended on the kind of environmental change and, for hardware, measurement stability.

H1.1

  • Very high linear correlations appeared in about one-third of changes, especially hardware and many low-severity workload shifts; version-change correlations varied by system.Source: §IV · p. 5

Hardware change

  • Hardware changes often had near-perfect correlations, but unstable cloud measurements coincided with weak correlations; the source-to-target noise-variance ratio was 33.39 in ec2 versus 1.51 in ec1.Source: §IV · p. 5

Workload change

  • SQLite’s write-heavy sequential-to-batch change had correlation 0.96, while its read-heavy random-to-sequential change had correlation 0.5.The passage attributes the difference to SSD read/write properties and the database’s different internal functions for inserting and retrieving data.Source: §IV · p. 5

H1.2

  • Some changes without strong linear correlation still had similar performance distributions, indicating potential for nonlinear transfer; Fig. 2 contrasts Dec1 KL = 25.02 with Dec13 KL = 0.32.Source: §IV · p. 5, Figure 2 · p. 6

H1.3

  • Configuration rankings were generally stable across hardware and small workload changes; SPEAR ec2 also retained high rank correlation despite low linear correlation amid measurement noise.Source: §IV · p. 6

H1.4

  • Top/bottom performers commonly matched across hardware and small workload changes, while the section’s insight singles out hardware as preserving top configurations.Source: §IV · p. 6

V. SIMILARITY OF INFLUENTIAL OPTIONS (RQ2)

The study finds that influential options and their relative importance are often preserved across environmental changes. This continuity can support sampling focused on a smaller set of options, even where overall performance correlations are low.

H2.1

  • Slightly more than half of options were influential in either environment, and a very high percentage of those influential options were shared across environments.Source: §V · p. 6
  • Sampling can fix options found non-influential in the source and focus measurements on influential options, though preserved option importance does not rule out changed interactions.Source: §V · pp. 6–7

H2.2

  • Option-importance correlations were generally high and tended to be higher for less severe changes, including some changes with low overall performance correlation.Source: §V · p. 7

VI. PRESERVATION OF OPTION INTERACTIONS (RQ3)

Pairwise interactions were often preserved across environments, although only a minority of possible interactions affected performance. The findings point to targeted configuration measurements as a way to reduce measurement effort.

H3.1

  • Mean influential pairwise-interaction percentages were 25% for SPEAR, 28% for x264, 10% for SQLite, and 6% for SaC.The reported common-interaction percentages were 96% for SPEAR, 81% for x264, 85% for SQLite, and 72% for SaC.Source: §VI · p. 7
  • Interactions often stayed consistent across environmental changes, which the authors identify as transferable knowledge for reducing measurement effort through targeted configuration measurements.Source: §VI · p. 7

H3.2

  • Interaction-effect correlations were very high, sometimes perfect, even for several changes where earlier metrics found little evidence of transferable knowledge.The paper notes that linear transfer of the whole performance function may fail for severe changes while a more complex transfer function may still exist.Source: §VI · p. 7

VII. INVALID CONFIGURATIONS SIMILARITY (RQ4)

Invalid regions can remain reusable even when performance behavior changes substantially. Across environmental changes, the transferable knowledge ranges from linear model transformations for small changes to invalid-configuration information that can guide sampling under severe changes.

VII. INVALID CONFIGURATIONS SIMILARITY (RQ4)

  • A considerable share of configurations is typically invalid in both environments, but common invalid configurations were low (≤10%) for some SaC workload changes.SaC had approximately 18% invalid sampled configurations; effects of some options varied across programs.Source: §VII · p. 7
  • For SPEAR and x264, almost all invalid configurations were common across environments, and their invalidity classifiers had almost perfect coefficient correlations.For two SPEAR cases where other metrics found no reusable knowledge, invalid regions still allowed avoiding almost half the target space.Source: §VII · pp. 7–8

A. Lessons learned

  • For small changes, correlated response functions support simple linear transfer; for large changes, option or interaction influences may persist while target coefficients need relearning.The discussion draws on analyses of 36 environmental changes.Source: A. Lessons learned · p. 8, §VIII · p. 8
  • Even under severe changes, invalid configurations can remain common, providing information that may help avoid invalid regions during sampling.The authors identify this as a transferable form of knowledge for severe environmental changes.Source: A. Lessons learned · p. 8, §VII · p. 8

B. Implications for transfer learning research

The discussion maps transferable knowledge to change severity, while emphasizing that severe-change techniques remain an open research need. The authors also qualify generalization by the selected systems, sampled configurations, operationalizations, and learners.

B. Implications for transfer learning research

  • Small changes support linear model transformations, large changes may preserve option or interaction influences for non-linear mapping, and severe changes may permit transfer of invalid-region knowledge.These findings suggest that different environmental changes call for different forms of transferable knowledge.Source: B. Implications for transfer learning… · p. 8
  • Users may estimate change severity and test transferability quickly with a few measurements before selecting a transfer-learning strategy.The authors note that this judgment may be possible without deep knowledge of system implementations or configuration spaces.Source: B. Implications for transfer learning… · p. 8
  • For severe changes, techniques still need to be designed and evaluated against current sampling and learning strategies.The authors identify sampling that uses relatedness or avoids invalid configurations, and learning mechanisms that exploit cross-environment associations, as research opportunities.Source: B. Implications for transfer learning… · p. 8
  • The findings also motivate work on transferring test cases, detecting invalid configurations, and using option and interaction importance in tuning and optimization.The paper also points to transfer-learning applications in performance modeling, including tuning queuing-network model parameters.Source: B. Implications for transfer learning… · pp. 8–9

1) External validity:

  • The study used diverse systems and purposefully selected environmental changes, so generalization to other systems or changes requires caution.Additional measurements on the same systems were excluded because they were consistent with the presented data and added no new insights.Source: 1) External validity · p. 9

2) Internal and construct validity:

  • Only one system was measured exhaustively; sampling the others may have missed effects in unmeasured parts of their configuration spaces.The authors report no surprisingly different observation in the exhaustively measured SPEAR dataset.Source: 2) Internal and construct validity · p. 9
  • Informal metric comparisons and ad-hoc thresholds leave results dependent on operationalization, although the authors do not expect the overall picture to change.They used only partial statistical testing and generally selected simple, reliable metrics.Source: 2) Internal and construct validity · p. 9
  • The study used step-wise linear regression, regression trees, and multinomial logistic regression, while other learners may differ in accuracy and cost.Benchmark noise could not be eliminated; measurements were repeated three times on dedicated systems, with repeats for unusually large deviations.Source: 2) Internal and construct validity · p. 9

IX. RELATED WORK

Prior configurable-system performance research developed sampling, optimization, and machine-learning approaches, while related work also studied environmental changes and transfer learning. This paper distinguishes itself by broadly investigating why and when transfer across hardware, workload, and software versions works.

A. Performance analysis of configurable software

  • Configurable-system performance analysis uses experimental-design sampling, optimization methods, and machine-learning models to learn or search within high-dimensional configuration spaces.These approaches aim to gain information from sparse samples or find optimal configurations under limited sampling budgets.Source: A. Performance analysis of… · p. 9
  • Unlike work comparing models, optimization, or sampling strategies, this study focuses on transferring performance models across hardware, workload, and software versions.The authors describe transfer learning as orthogonal to those approaches and potentially able to make them more efficient.Source: A. Performance analysis of… · p. 9

B. Performance analysis across environmental change

  • Earlier studies examined environmental change in contexts including MapReduce, anomaly detection, hardware variants, and parameter dependencies..Source: B. Performance analysis across… · p. 9
  • Prior software-engineering transfer-learning work included performance prediction and hardware transfer, while this study investigates more broadly why and when transfer applies across environmental changes.The related work also includes transfer in defect prediction and effort estimation.Source: B. Performance analysis across… · p. 9

X. CONCLUSIONS

The study finds diverse transferable knowledge across environments, with the useful form depending on the environmental change. Small changes can support linear model transfer, while more severe changes can still yield knowledge useful for guided sampling.

X. CONCLUSIONS

  • Performance models are related across environments through overall response, distributions, influential options and interactions, and invalid configurations.These distinct forms of shared knowledge may support different transfer strategies.Source: §X · p. 9
  • Small environmental changes can permit simple linear transfer, while severe changes may still support sampling by reusing knowledge about invalid configurations.The conclusion describes these as different ways to exploit cross-environment relationships.Source: §X · p. 9
  • Table II summarizes several forms of knowledge that can be transferred across environments for use in transfer learning.Source: §X · p. 9, Table II · p. 10

Related papers

Loading 1709.02280v1…