PaperlayerView exact revision on arXiv

Source-linked AI summary

State-of-the-art in selection of variables and functional forms in multivariable analysis -- outstanding issues

Willi Sauerbrei, Aris Perperoglou, Matthias Schmid, Michal Abrahamowicz, Heiko Becher, Harald Binder, Daniela Dunkler, Frank E. Harrell, Patrick Royston, Georg Heinze

arXiv:1907.00786v1stat.ME

TL;DR

The paper examines unresolved decisions about functional forms and variable inclusion in multivariable modelling. It illustrates weaknesses in modelling approaches and concludes that many open issues remain, with insufficient evidence to identify state-of-the-art approaches.

  • Problem

    Selecting functional forms and deciding which variables to include are crucial but complicated parts of multivariable model building.

  • Method

    The paper discusses methodological issues and uses examples of potentially suboptimal approaches to illustrate weaknesses in medical-data analyses.

  • Results

    Many open issues remain, and determining which approaches are state-of-the-art requires much more research.

  • Takeaways & Limitations

    The evidence gap remains substantial when determining which approaches should be considered state-of-the-art.

  • Takeaways & Limitations

    Data-dependent final models are partly determined by chance, while ignoring model-building uncertainty can bias parameter estimates and underestimate their variances.

Abstract

from arXiv · show

How to select variables and identify functional forms for continuous variables is a key concern when creating a multivariable model. Ad hoc 'traditional' approaches to variable selection have been in use for at least 50 years. Similarly, methods for determining functional forms for continuous variables were first suggested many years ago. More recently, many alternative approaches to address these two challenges have been proposed, but knowledge of their properties and meaningful comparisons between them are scarce. To define a state-of-the-art and to provide evidence-supported guidance to researchers who have only a basic level of statistical knowledge many outstanding issues in multivariable modelling remain. Our main aims are to identify and illustrate such gaps in the literature and present them at a moderate technical level to the wide community of practitioners, researchers and students of statistics. We briefly discuss general issues in building descriptive regression models, strategies for variable selection, different ways of choosing functional forms for continuous variables, and methods for combining the selection of variables and functions. We discuss two examples, taken from the medical literature, to illustrate problems in the practice of modelling. Our overview revealed that there is not yet enough evidence on which to base recommendations for the selection of variables and functional forms in multivariable analysis. Such evidence may come from comparisons between alternative methods. In particular, we highlight seven important topics that require further investigation and make suggestions for the direction of further research.

1. Introduction

This paper examines whether available methods for selecting variables and functional forms provide enough evidence to define state-of-the-art methodology for descriptive multivariable models. It reviews existing approaches, illustrates modelling problems, and identifies unresolved issues requiring further research.

  • Scope and motivation: Descriptive modelling aims to capture data structure parsimoniously rather than directly optimize predictive performance.Interpretability, transportability, and general usability are important criteria for descriptive models.
  • Scope and motivation: Data-dependent derivation of a final model introduces bias, while model complexity and variance must be weighed against the intended modelling purpose.The paper notes that estimates and test statistics can be affected by data-dependent model building.
  • Scope and motivation: Selection of variables and functional forms for continuous variables are central, interrelated challenges in building good descriptive multivariable models.The paper focuses on statistical model building for descriptive analysis.
  • Evidence gap: Traditional selection methods have existed for over 50 years, while newer techniques have expanded the available toolkit without establishing clear usage guidance.Analysts may have access to many approaches, but it is not straightforward which method to use or under what circumstances.
  • Evidence gap: Functional-form choices commonly default to linearity or use categorization, despite available fractional-polynomial and spline-based methods and limited comparative guidance.Combining variable selection with functional-form selection further increases modelling complexity.
  • Paper contribution: The review surveys variable-selection methods, four approaches to continuous-variable functional forms, and topics requiring further research to support SOTA arguments.It focuses on observational studies of human health and aims to provide evidence-supported arguments for multivariable models with continuous variables.

2. General issues in building descriptive regression models

Building descriptive regression models requires decisions about candidate variables and functional forms, with subject-matter knowledge often insufficient in observational studies. These choices affect interpretability, usability, bias, and the credibility of the resulting model.

  • Model-building considerations: The intended use of a model determines the balance between interpretability, transportability, usability, predictive accuracy, and complexity.Descriptive models prioritize interpretability and general usability, whereas prediction may favor more variables and flexible functions.
  • Model-building considerations: Observational studies often lack sufficiently reliable subject-matter knowledge to pre-specify all confounders and their functional forms.Pre-specifying every aspect of the model is described as almost invariably unrealistic in observational studies.
  • Variable and function choices: Variable selection concerns which candidate variables enter the final model, and the modelling of one variable may depend critically on how other variables are represented.The information cost of selected variables also affects practical usability.
  • Model-building considerations: Data-dependent model building can introduce replication instability, selection bias, omitted-variable bias, underestimated variance, and overestimated predictive ability.These problems are especially important for weak risk factors, whose meaning depends on sample size.
  • Variable and function choices: Continuous variables are often categorized or assumed linear before selection, although known weaknesses of categorization and underused flexible smoothers motivate better choices.Fractional polynomials and several spline methods are available, but methods addressing variable and function selection jointly are not often used.
  • Scope: The paper deliberately excludes interactions and time-dependent effects from its discussion of model-building issues.These are identified as additional analysis issues outside the paper’s scope.

3. Variable selection: strategies

Variable-selection strategies range from stepwise procedures and univariate screening to best-subset, information-criterion, and change-in-estimate approaches. Their trade-offs involve model size, selection uncertainty, bias, predictive error, and practical purpose.

  • Traditional strategies: Traditional stepwise procedures iteratively add or remove variables using significance thresholds, with smaller thresholds producing fewer selected variables.Backward elimination begins with a plausible model, while stepwise procedures combine forward and backward steps.
  • Traditional strategies: Backward elimination is widely regarded by biostatisticians as preferable to other described stepwise procedures because it starts from a plausible model.Selection nevertheless sets some coefficients exactly to zero and intentionally moves away from the maximum-likelihood solution.
  • Consequences of selection: Selection can reduce variance through implicit shrinkage for weak predictors, but selected weak predictors may have coefficients seriously biased away from zero.Increased bias can raise mean squared error, while uncertainty in the selection decision contributes further MSE inflation.
  • Evidence gaps: Resampling-based measures for selection-induced bias and root-MSE inflation have been proposed but still require validation in simulation studies.This limits the evidential basis for judging the advantages and disadvantages of competing strategies.
  • Alternative strategies: Best-subset selection compares all possible models using an information criterion, but its greater flexibility can increase selection uncertainty and spurious predictor selection.Combining it with functional-form selection would substantially enlarge the candidate-model space.
  • Alternative strategies: AIC targets expected predictive performance, whereas BIC targets selection of true predictors under assumptions about the model space and sample size.AIC corresponds to a p-value threshold of 0.157 for a one-degree-of-freedom comparison; BIC becomes stricter as sample size increases.
  • Alternative strategies: Univariate selection can be misleading because unadjusted and adjusted effects may differ in either direction.Univariate screening remains popular in high-dimensional bioinformatics and genetics applications.
  • Purpose-driven strategies: Change-in-estimate procedures retain variables when removing them materially changes the exposure coefficient, while augmented backward elimination aims to reduce bias relative to conventional backward elimination.A commonly cited relevance threshold is a change exceeding 10% in the exposure coefficient.

3.3 Modern variable selection strategies

Modern variable-selection strategies combine penalization, shrinkage, boosting, and resampling to select predictors while controlling model complexity. Their behavior depends on scaling, collinearity, tuning, and dependencies among selected variables.

  • Penalized selection: The Lasso penalizes absolute coefficients, setting some exactly to zero and thereby performing variable selection.Its penalty strength λ is commonly estimated by cross-validation.
  • Penalized selection: The Lasso is not invariant to covariate transformations because standardization choices can change its results.Standardized covariates are typically used so coefficients share a common scale.
  • Penalized selection: Elastic Net addresses grouped and collinear predictors by combining L1 and L2 penalties and removing the Lasso’s limit on selected variables.It contains Lasso and Ridge penalization as special cases.
  • Boosting: Component-wise boosting updates one coefficient by a small fraction at each step and uses early stopping for selection and shrinkage.The number of boosting steps is commonly optimized by cross-validation, and the method is useful in high-dimensional settings.
  • Resampling: Resampling-based procedures use bootstrap inclusion frequencies as a model-selection criterion, but correlated variables can make individual frequencies misleading.Dependencies among inclusion fractions have been proposed to address this problem.
  • Resampling: Simple summaries of resampled models may select a bad model when inclusion of variables depends on other variables.Further practical experience and consideration of dependencies remain necessary.

3.5 Bias and the role of shrinkage methods

Shrinkage methods reduce coefficient magnitudes after or alongside variable selection and may lessen selection-induced overestimation bias. However, uncertainty after selection remains difficult to quantify, and empirical evidence for newer procedures is limited.

  • Shrinkage methods: Shrinkage methods introduce bias toward zero into regression coefficients to reduce prediction error.They can be combined with traditional variable-selection methods using global or parameter-specific factors.
  • Shrinkage methods: Backward elimination with parameter-specific shrinkage factors produced models similarly or more accurate than modern approaches in linear-regression comparisons.The construction was not motivated primarily by bias correction.
  • Open issues: A systematic investigation of shrinkage’s possible bias-correction side effect is still lacking.The proposed leave-one-out cross-validation estimation method can also be questioned because other resampling procedures have been preferred elsewhere.
  • Post-selection inference: Traditional post-selection confidence intervals and p-values ignore uncertainty from model selection in predictions, effect estimates, and variance.This underestimation of uncertainty has been confirmed in numerous simulation studies.
  • Post-selection inference: Selective-inference procedures can estimate confidence intervals for Lasso-selected regression parameters, but practical evidence on their performance remains limited.These procedures were developed only recently.
  • Open issues: Parameterwise or joint shrinkage factors have been proposed for selection of variables and functional forms, while variance correction remains unclear.The usefulness of sandwich variance estimators for approximately valid intervals is unresolved.

4. Continuous variables – to categorize or to model?

Categorization simplifies presentation but discards information and can produce unstable or implausible step-function relationships. Flexible approaches such as fractional polynomials and splines model continuous effects without categorization.

  • Categorization: Categorizing continuous predictors creates issues involving category number, cutpoints, coding, power loss, local bias, and invalid inference from data-dependent cutpoints.Conventional linear representations likewise often lack justified or assessed assumptions.
  • Flexible modelling: Fractional polynomials and spline methods provide flexible alternatives for modelling continuous effects in multivariable analyses.The reviewed approaches include fractional polynomials, restricted regression splines, penalized regression splines, smoothing splines, and generalized additive models.
  • Categorization: Categorization can make nonlinearity undetectable and results can vary when different cutpoints are used.Dichotomization produces the greatest information loss among grouping choices.
  • Categorization: Data-dependent minimum-p-value cutpoints are partly chance-driven, non-reproducible, and strongly overestimate group differences.Uncorrected multiple testing can raise the type I error to around 40%.
  • Categorization: Different cutpoints across studies hinder comparison and synthesis of results.A review of S-Phase fraction identified nineteen cutpoints, several from optimal-cutpoint analyses.
  • Flexible modelling: Fractional-polynomial models extend power transformations, with FP1 and FP2 functions usually sufficient and exponents drawn from a specified set.Repeated powers use the form β1x^p + β2x^p log x.

5. Combining variable and function selection

Selecting variables and their functional forms jointly is difficult because inclusion decisions and shape choices can influence one another. Proposed procedures include MFP, spline-based algorithms, boosting, and penalized smooth-term selection, but comparative evidence remains limited.

  • Joint selection: Variable inclusion and functional-form decisions can depend on one another through complex relationships involving variables and confounders.A variable’s apparent significance may change with the chosen functional form.
  • Joint selection: A linear BMI effect showed no evidence of association with CHD mortality, whereas a GAM revealed a non-monotone J-shaped association.The models used the same adjustment covariates.
  • MFP: MFP combines backward elimination with functional-form selection and is intended to produce relatively interpretable, reportable models.Extensions include modelling interactions, time-dependent effects, and meta-analytic functions.
  • MFP: MFP significance settings control model complexity: an extremely low FP threshold usually selects only linear functions, making MFP effectively backward elimination.The backward-elimination and FP components jointly determine whether simpler or more complex functions are selected.
  • Bias correction: Data-dependent selection of variables and functions is likely to produce overestimation bias, motivating parameter-specific or joint shrinkage factors.Systematic investigations of joint shrinkage in combined selection have not been reported.
  • Penalized smooths: Penalizing smooth terms can control wiggliness and, with null-space penalization, shrink a smooth term to zero so it is effectively dropped.These methods require appropriate fitting and smoothing-parameter optimization routines.
  • Open issues: Further research is needed to compare penalization methods with stepwise GAM procedures and to evaluate newer selection approaches in practice.Estimated smoothness parameters also make associated p-values approximate.
  • Alternative approaches: Boosting, GAMSEL, COSSO, SpAM, and related penalized approaches offer alternatives for selecting nonlinear effects or choosing between linear and nonlinear fits.Many of these methods have not been evaluated in practice, so their properties remain uncertain.

6. Examples illustrating the problems

Published observational analyses illustrate practical problems in variable selection, reporting, functional-form choice, and interpretation. A categorized and spline-based analysis of physical activity produced numerically discrepant hazard ratios, with no explanation for the discrepancy.

  • Ramaiola et al (2015): In an 86-patient study, 12 explanatory variables were screened using bivariate tests before backward elimination in a multivariable model.The starting set comprised variables significantly associated with Pfn-1 at α=0.05.
  • Ramaiola et al (2015): Duration of ischaemia and patient age remained independent factors for Pfn-1 levels, both with p-values lower than 0.001.Time of ischaemia was not mentioned as a candidate variable in the statistical methods section.
  • Ramaiola et al (2015): The analysis reported regression coefficients without standard errors or measurement scales and mentioned no assessment of assumptions, nonlinearities, interactions, or model stability.These omissions limit interpretation and leave the stability of the selected model unexamined.
  • Arem et al (2015): For physical activity, categorized and spline analyses both suggested benefit, but their numerical results did not agree well.The categorized analysis reported hazard ratio 0.8 (95% confidence interval 0.78 to 0.83) between 12 minutes and 7.5 MET hours per week, whereas the spline analysis changed smoothly from 1 to about 0.82.
  • Arem et al (2015): The paper gave no explanation for the discrepancy, and its claim that modest activity provides substantial mortality benefit was not fully supported by the two analyses.The authors suspect that the spline analysis may have ignored the spike at zero in MET, which was 8.0%.

7. Towards state of the art – research required!

The paper identifies seven research priorities for developing evidence-supported guidance on variable and functional-form selection. It emphasizes head-to-head empirical comparisons across realistic modelling settings, while noting that current knowledge remains limited and assumptions constrain guidance.

  • Research strategy: Evidence-supported guidance will generally require extensive, better-designed simulation studies because theoretical results often rely on strong assumptions.The paper stresses that guidance is obtainable only under specific assumptions, such as sufficiently large samples and absence of interactions.
  • Issue 1: Traditional and modern variable-selection strategies need head-to-head comparison across explanatory modelling situations.Studies should assess inclusion and exclusion, model stability and complexity, coefficient accuracy, and vary candidate variables, distributions, correlations, effect strengths, and sample size.
  • Issue 2: Spline procedures require comparison in univariable and multivariable settings, including their dependence on knots, function complexity, usability, and variable inclusion.Practical studies should use mixed variable types, splines for continuous variables, varied functions, and varied sample sizes.
  • Issue 3: Approaches for variables with a spike at zero need comparison, including two-step fractional-polynomial and one-step spline-based procedures.Four strategies have been suggested for two spike-at-zero variables, but current knowledge is limited.
  • Issue 4: Multivariable procedures that select models and functions should be evaluated for accuracy, efficiency, transportability, implementation, and interpretability.The large number of recently proposed penalized and spline-based approaches motivates further performance studies.
  • Issues 5–6: Further work is needed on post-selection shrinkage, inference after selection, and robust variance estimation in explanatory modelling.Open questions include shrinkage-factor choice, cross-validation procedures, confidence intervals after forward selection or Lasso/LARS, and variance estimators robust to misspecification.
  • Issue 7: Very large datasets require adapted procedures for interpretable explanatory models, including reconsidered tuning parameters, variable clustering, and integration of subject-matter knowledge with statistical learning.The paper highlights settings with n > 100,000 and p > 1,000, whereas existing procedures were developed for much smaller n and p.

8. Discussion

The discussion identifies seven key issues in selecting variables and functional forms for multivariable descriptive models, emphasizing that evidence remains insufficient for state-of-the-art recommendations. It calls for systematic comparisons, better model-uncertainty assessment, and accessible guidance for analysts with limited statistical training.

  • Seven key issues were identified as priorities for building evidence-supported guidance on multivariable model development.
  • The scope focuses on descriptive models in observational studies, while prediction, causal inference, initial data analysis, survival data, and high-dimensional data are addressed elsewhere.
  • Data-dependent model building can produce biased parameter estimates and underestimated variances because model-building uncertainty is ignored.
  • Model derivations should be complemented by investigations of model stability, although such investigations are not yet standard practice.
  • Simulation studies are needed to compare competing methods, but comparisons can favor new methods and are often poorly designed, analyzed, or reported.
  • STRATOS aims to provide accessible and accurate guidance, including guidance for analysts with lower statistical knowledge, but further research is still needed to establish state-of-the-art procedures.

Tables

Table 1 summarizes the relevant issues involved in deriving evidence-supported state-of-the-art guidance.

  • Table 1 summarizes issues considered most relevant for deriving evidence-supported state-of-the-art guidance.

1. Methods based on spline functions

Spline methods represent continuous variables through flexible polynomial bases or penalized fits, but their choices differ in knot specification, smoothness control, and computational demands. Despite well-understood mathematical properties, thorough empirical comparisons remain sparse.

  • Regression splines use polynomial bases and require choices about the number and placement of knots.
  • Allowing knot positions to be estimated can increase flexibility but may cause computational cost and instability, while automatic knot selection is difficult to implement.
  • With enough knots, precise knot placement is not crucial; suggested counts range from three for small datasets to five for larger datasets.
  • Smoothing splines place knots at observations and use roughness penalties, whereas P-splines use many equidistant B-spline knots with discrete difference penalties.
  • Thin-plate regression splines avoid specifying knot number, knot placement, or basis functions and can use automatic penalty optimization.
  • Although spline mathematics is well understood, systematic comparisons among spline procedures and against alternatives remain limited.

2. Fractional polynomials

Fractional polynomials provide a compact family of globally defined transformations for continuous covariates, while the function selection procedure tests progressively simpler forms. Their flexibility and interpretability come with limits, including reduced responsiveness to local data features.

  • Fractional polynomials model continuous covariates using power transformations selected from a restricted set, with 0 denoting log X.
  • Model simplicity is favored for descriptive aims because interpretability, transportability, and general usability matter.
  • FP2 functions combine two powers and allow repeated powers represented by β1Xp + β2 Xp log X.
  • The FP1 and FP2 classes contain 8 and 36 functions, respectively, offering varied functional forms despite their small size.
  • Unlike splines, fractional polynomials are globally defined and cannot identify or respond to local features of the data.
  • Its sequential tests assess overall association, nonlinearity, and whether a simpler or more complex function is warranted.
Loading 1907.00786v1…