Automatic Construction and Natural-Language Description of Nonparametric Regression Models

James Robert Lloyd; David Duvenaud; Roger Grosse; Joshua B. Tenenbaum; Zoubin Ghahramani

Source-linked AI summary

Automatic Construction and Natural-Language Description of Nonparametric Regression Models

James Robert Lloyd, David Duvenaud, Roger Grosse, Joshua B. Tenenbaum, Zoubin Ghahramani

arXiv:1402.4304v3Feb 18, 2014stat.MLcs.LG

TL;DR

The paper addresses the limited automation of flexible, interpretable statistical modeling. It introduces ABCD, which searches a compositional Gaussian-process model language and automatically describes discovered components in reports. The system achieves state-of-the-art extrapolation performance on 13 time series while producing interpretable model explanations.

Problem
Flexible but interpretable statistical models have seen little automatic construction compared with simple models such as linear regression.
Method
ABCD searches an open-ended compositional language of Gaussian process models and translates model components into natural-language descriptions.
Results
ABCD achieves state-of-the-art extrapolation performance on 13 time series while generating reports that highlight interpretable features.
Takeaways & Limitations
The procedure has the potential to make powerful statistical model-building techniques accessible to non-experts.
Takeaways & Limitations
The system excludes autoregressive and moving-average models that regress on previous time-series values.

Abstract

from arXiv · show

This paper presents the beginnings of an automatic statistician, focusing on regression problems. Our system explores an open-ended space of statistical models to discover a good explanation of a data set, and then produces a detailed report with figures and natural-language text. Our approach treats unknown regression functions nonparametrically using Gaussian processes, which has two important consequences. First, Gaussian processes can model functions in terms of high-level properties (e.g. smoothness, trends, periodicity, changepoints). Taken together with the compositional structure of our language of models this allows us to automatically describe functions in simple terms. Second, the use of flexible nonparametric models and a rich language for composing them in an open-ended manner also results in state-of-the-art extrapolation performance evaluated over 13 real time series data sets from various domains.

1 Introduction

The paper proposes ABCD, a system that automatically constructs interpretable regression models and generates reports explaining their components. It combines an open-ended model language, efficient search, principled evaluation, and natural-language descriptions, achieving state-of-the-art extrapolation performance on 13 time series.

Results: 71.5% of the residual variance is explained by one component, increasing total variance explained from 72.8% to 92.3%.The component is approximately periodic with a 10.8-year period and applies until 1643 and from 1716 onwards.
Contributions: ABCD uses an open-ended compositional language of Gaussian process models to capture modeling assumptions and composition techniques used by human statisticians.Its language represents flexible regression functions through properties such as smoothness, trends, periodicity, and changepoints.
Contributions: The system searches the model space efficiently and evaluates candidate models using marginal likelihood and the Bayesian Information Criterion.These criteria balance model fit with model complexity.
Contributions: ABCD automatically translates model components into natural-language descriptions and visual reports that expose factors underlying the data.Examples include reports describing an approximately 11-year sunspot cycle and its disappearance during the Maunder minimum.
Results: ABCD achieves state-of-the-art extrapolation performance against existing model-construction techniques on 13 real time series.The paper evaluates predictive performance at extrapolation while also demonstrating interpretable automatically generated reports.

2 A language of regression models

The regression-model language uses Gaussian processes whose kernels encode interpretable function properties, then composes them with addition, multiplication, and changepoint constructions. The expanded and reparameterized language represents a broad range of regression structures, including models unavailable in earlier formulations.

Gaussian processes provide an analytically tractable framework for representing both simple parametric and complex nonparametric regression functions.
Base kernels encode uncorrelated noise, constant, linear, smooth, and periodic functions, while addition and multiplication create richer structures.
Changepoints are incorporated through kernel combinations with sigmoidal functions, enabling different component behaviors across input regions.
The language expands and reparameterizes base kernels to support automatic descriptions and a larger set of common regression models.
Table 1 identifies common regression models expressible in the language, including models marked as unavailable in Duvenaud et al. (2013).

3 Model Search and Evaluation

ABCD searches the regression-model space greedily, optimizing each proposed model and evaluating it with BIC. BIC balances marginal likelihood against model complexity through a parameter-count penalty.

ABCD explores the regression-model space with a greedy search using operators that include changepoint constructions.
After each proposal, kernel parameters are optimized by conjugate gradient descent before model evaluation.
BIC(M) = −2 log p(D | M) + |M| log n evaluates optimized models using marginal likelihood and parameter count.
BIC trades off model fit and complexity by penalizing the number of kernel parameters, implementing Bayesian Occam’s Razor.

4 Automatic description of regression models

ABCD converts compositional Gaussian-process kernels into sums of interpretable components and generates descriptions by assigning head nouns, modifiers, parameters, and data-derived refinements. Components are ordered by their contribution to cross-validated prediction error.

ABCD simplifies kernel expressions into sums of products so each additive function can receive a separate natural-language description.
Independent Gaussian-process functions with kernels k1 and k2 sum to a Gaussian process with kernel k1 + k2, supporting separate descriptions of additive components.
Within each product, one kernel supplies the noun and the remaining kernels become modifiers, producing descriptions of the form Determiner + Premodifiers + Noun + Postmodifiers.
Description refinements can incorporate kernel parameters, qualitative parameter effects, and information calculated from the data.
The description heuristics choose head nouns and refinement parameters by author judgment, leaving objective learning from expert statisticians for future work.
Additive components are ordered by the component that most reduces 10-fold cross-validated mean absolute error.

5 Example descriptions of time series

ABCD generates concise descriptions of diverse time-series structures and identifies meaningful components in solar and airline data. In the solar example, it recovers the Maunder minimum, an approximately 11-year cycle, and long-term trends, while comparison with Eureqa exposes limitations of sinusoidal approximations.

Examples: ABCD provides automatically generated reports for 13 data sets and discovers varied time-series patterns with descriptions.
Solar activity: Solar irradiance contains an approximately 11-year activity cycle and a low-variance interval from 1645 to 1715 corresponding to the Maunder minimum.
Solar activity: 92.3% of solar-data variation is explained by the first 4 additive components, rising to 99.7% with the first 6 components.
Solar activity: The solar model identifies a constant component, a smooth component, and an approximately periodic component with a period of 10.8 years outside the Maunder-minimum interval.
Solar activity: The fourth solar component identifies the 11-year cycle, while the third captures long-term trends and the second corresponds to the Maunder minimum.
Solar activity: Adding one solar component explains 37.4% of residual variance and reduces cross-validated MAE by 31.97%, from 0.33 to 0.23.
Airline passengers: For airline passengers, ABCD represents approximately periodic behavior with linearly increasing amplitude and expresses heteroscedasticity through a white-noise kernel multiplied by a linear kernel.
Comparison with equation learning: The Eureqa solar function captures a trend and periodic component but heavily approximates the annual cycle and does not capture heteroscedasticity.

6 Designing kernels for interpretability

ABCD adapts its kernel language to keep composite Gaussian-process models interpretable, separating distinct behaviors and avoiding unnecessary posterior ambiguities. The resulting components describe trends, periodicity, changing amplitude, and heteroscedastic noise in simple terms.

6 Designing kernels for interpretability: ABCD uses base kernels selected to improve interpretability relative to related kernel languages.The language separates medium-term trends from short-term deviations rather than combining them in one rational-quadratic component.
6 Designing kernels for interpretability: A discovered component captures approximately yearly periodicity with smoothly varying shape, linearly increasing amplitude, and a six-week typical within-period lengthscale.This component explains 89.9% of residual variance and reduces cross-validated MAE by 63.45%, from 34.03 to 12.44.
6 Designing kernels for interpretability: Another component models uncorrelated noise whose standard deviation increases linearly.It explains 100.0% of residual variance but leaves cross-validated MAE unchanged at 9.10, suggesting very short-term structure, noise, or a model/search artefact.
6 Designing kernels for interpretability: Subtracting constants from periodic and linear kernels prevents convoluted product descriptions and inflated component credible intervals.Without subtraction, products expand into several qualitatively different additive functions, while constant terms can induce posterior anti-correlation.

7 Related work

Related work spans manually constructed and algorithmically searched kernel spaces, spectral and multiple-kernel learning, equation learning, and open-ended model search. ABCD distinguishes itself by automatically interpreting the selected nonparametric models and generating natural-language descriptions.

7 Related work: ABCD follows work on manually constructed composite Gaussian-process kernels but automatically builds a similar carbon-dioxide model.Prior genetic-algorithm approaches searched similar model spaces without interpreting the resulting models.
7 Related work: Its kernel language includes sparse-spectrum and spectral-mixture kernel classes that can approximate stationary covariance functions.These classes correspond to sums of cosine terms and products of squared-exponential and cosine terms, respectively.
7 Related work: Multiple-kernel learning can optimize in polynomial time when component kernels and parameters are pre-specified; ABCD compares against a Bayesian restriction of its language.For MKL, the ABCD greedy search corresponds to forward selection.
7 Related work: Unlike equation-learning systems that learn parametric functions, ABCD learns a parametric covariance form that can model functions without simple parametric forms.Its open-ended compositional search is related to prior searches over matrix decompositions and graph structures.
7 Related work: The paper presents automatic description of nonparametric statistical models as a novel natural-language output capability.The procedure automatically interprets the chosen model, making results accessible to non-experts.

8 Predictive Accuracy

ABCD is evaluated on interpolation and extrapolation across 13 real time series against several regression and model-construction methods. It outperforms competing methods on average, while the interpretability-oriented variant trades some predictive performance for simpler descriptions.

8 Predictive Accuracy: ABCD outperforms the other model-building methods on average for both interpolation and extrapolation.Interpolation results are reported as similar to extrapolation results.
8 Predictive Accuracy: The comparison excludes autoregressive and moving-average models because the evaluation was restricted to regression algorithms.Constructing a language for that class of time-series model is identified as future work.
8 Predictive Accuracy: ABCD-interpretability distributes products over addition during search so BIC penalizes models with many additive components more heavily.The aim is to reduce the interpretability cost of nested products of sums, though the resulting models can sacrifice predictive accuracy relative to ABCD-accuracy.
8 Predictive Accuracy: On standardized extrapolation RMSE across 13 time series, ABCD-accuracy outperforms ABCD-interpretability, while both have lower quartiles than every other method.RMSE is computed by training on the first 90% and predicting the remaining 10%, then normalizing each dataset's best performance to 1.
8 Predictive Accuracy: Unpenalized marginal likelihood caused overfitting in the authors' experiments.The benchmark also used the same inference method across algorithms for comparability, despite more advanced methods typically being used for changepoint modeling.
8 Predictive Accuracy: Greater model capacity follows the reported performance ordering ABCD > trend-cyclical-irregular > Bayesian MKL > squared exponential.Eureqa performs relatively poorly because few datasets are parsimoniously explained by a parametric equation.

9 Conclusion

The paper presents ABCD as a system that constructs models from an open-ended language and automatically generates reports describing discovered time-series patterns. It reports state-of-the-art interpolation and extrapolation performance relative to existing model-construction techniques.

9 Conclusion: ABCD constructs models from an open-ended language and generates detailed reports describing patterns captured by those models.The reports are intended to make powerful statistical model-building techniques accessible to non-experts.
9 Conclusion: ABCD achieves state-of-the-art extrapolation and interpolation performance compared with existing model-construction techniques on time series.The paper also demonstrates discovery and description of varied patterns across several time series.

Appendices

The appendices define ABCD’s kernel language and its greedy construction procedure, including compositional search operators and several experimentally motivated extensions.

Base kernels: The appendices define base kernels including white noise, constant, linear, squared exponential, and periodic kernels, with parameters specified for scalar inputs.The white-noise definition uses the Kronecker delta function.
Search operators: Changepoint and changewindow operators use sigmoid-based gating to restrict kernels outside or within selected regions.Changewindow operators replace the sigmoid with a product of two sigmoids.
Model construction: ABCD starts with a noise kernel, proposes new expressions through search operators, optimizes parameters by likelihood, and selects candidates using BIC in a greedy search.Ten random restarts initialize parameters, and searches run to depth 10.
Search operators: The search language composes kernels through addition, multiplication, and base-kernel replacement.These operations act on kernel subexpressions and base kernels within an expression.
Search operators: Several additional search operators were introduced from typical search paths, but their inclusion is not rigorously justified.This marks an explicit methodological limitation of the search language.

C Predictive accuracy

ABCD achieves strong interpolation performance across 13 datasets, while changepoint parameterization can reduce accuracy near transitions and Eureqa misses fine-scale structure.

Interpolation: All quartiles of standardized RMSE are lower for both ABCD variants, and ABCD-accuracy has median 1 while performing best on 7 datasets.Standardized RMSE divides each dataset’s RMSE by its smallest RMSE, so the best performance equals 1.
Interpolation: Changepoints performs slightly worse than MKL despite being strictly more general, because sigmoid changepoints introduce parametric forms into the regression models.The resulting interpolation degradation occurs at locations of change points.
Interpolation: More robust changepoint modeling would require more flexible changepoint shapes or improved inference over changepoint location and shape.The passage gives fully Bayesian inference as an example of improved inference.
Interpolation: Eureqa performs poorly because its parametric forms capture broad trends but fail to explain fine details.This limitation concerns its suitability for the interpolation task.
Interpolation: The supplementary results compare ABCD variants with spectral kernels, TCI, Bayesian MKL, Eureqa, changepoints, squared exponential, and linear regression.Raw interpolation results are reported in table 4.