Source-linked AI summary
The autofeat Python Library for Automated Feature Engineering and Selection
Franziska Horn, Robert Pack, Michael Rieger
TL;DR
Autofeat addresses the tension between transparent but less accurate linear models and difficult-to-explain nonlinear models by automatically engineering and selecting nonlinear features for linear prediction. Its multi-step process improves linear-model performance while retaining a transparent model for decision-making, though its scope is especially oriented toward heterogeneous scientific data rather than relational or time-series applications.
Problem
Nonlinear models can be difficult to fit and explain, whereas linear models are efficient and interpretable but generally less accurate on nonlinear data.
Method
Autofeat generates a large pool of nonlinear features, filters and selects a small subset, and trains scikit-learn-style linear regression or classification models.
Results
The library improves linear-regression performance and can sometimes outperform other nonlinear machine-learning models while retaining transparent, traceable results.
Takeaways & Limitations
Combining a linear model with selected nonlinear features provides accurate predictions with interpretability suitable for business decisions by non-statisticians.
Takeaways & Limitations
Autofeat is especially intended for heterogeneous scientific datasets and is not positioned as a competitor to featuretools for relational data or tsfresh for time series.
Abstract
from arXiv · showhide
This paper describes the autofeat Python library, which provides scikit-learn style linear regression and classification models with automated feature engineering and selection capabilities. Complex non-linear machine learning models, such as neural networks, are in practice often difficult to train and even harder to explain to non-statisticians, who require transparent analysis results as a basis for important business decisions. While linear models are efficient and intuitive, they generally provide lower prediction accuracies. Our library provides a multi-step feature engineering and selection process, where first a large pool of non-linear features is generated, from which then a small and robust set of meaningful features is selected, which improve the prediction accuracy of a linear model while retaining its interpretability.
1. Introduction
The paper addresses the trade-off between interpretable linear models and more accurate but harder-to-explain nonlinear models. It proposes autofeat as an accessible general-purpose library that automatically engineers and selects nonlinear features for linear prediction.
- 1. Introduction: Nonlinear models such as neural networks can be difficult and time-consuming to fit and challenging to explain to non-statisticians.For physical-system data, plausible models may matter more than achieving the smallest prediction errors.
- 1. Introduction: Linear models are efficient and intuitive, but usually lose accuracy because original inputs often relate nonlinearly to the target.This motivates transforming the original inputs before applying a linear predictor.
- 1. Introduction: Autofeat automatically generates tens of thousands of nonlinear features and selects informative additions for a linear model.The framework is intended to produce accurate, transparent predictions and is publicly available as a Python implementation.
- Related Work: Existing feature-construction methods differ between exhaustive generation followed by selection and iterative feature expansion, with trade-offs in memory use and feature discovery.Autofeat follows the exhaustive feature-pool strategy, while prior approaches include beam search, complex selection, and meta-learning.
- Related Work: The authors identify a lack of an easy-to-use general-purpose open-source library combining automated feature engineering and selection.Existing tools often target polynomial, relational, or time-series features rather than heterogeneous scientific datasets.
2. Automated Feature Engineering and Selection with autofeat
Autofeat uses a scikit-learn-style workflow to generate a large nonlinear feature space, remove invalid or redundant features, and select a compact subset for linear prediction. Its selection procedure combines correlation filtering, L1-regularized models, noise filtering, and repeated selection to retain useful features while preserving interpretability.
- Automated Feature Engineering and Selection with autofeat: Autofeat models automatically generate and select nonlinear features before training a linear regression or classification model.AutoFeatRegressor and AutoFeatClassifier expose a familiar scikit-learn-style interface.
- Feature Engineering: Feature generation alternates user-selected transformations and pairwise operators, causing the feature space to grow exponentially.With three original features, successive steps produce about 20, 750, and over 4000 features.
- Feature Engineering: SymPy simplifies generated expressions, while physical units restrict the library to physically legal feature combinations.Pint is used to represent units and prevent nonsensical operations such as subtracting temperature from volume.
- Feature Selection: Feature selection first removes highly correlated features, then uses a multistep procedure based largely on L1-regularized linear and logistic models.The library also exposes this selection component independently through the FeatureSelector class.
- Feature Selection: Repeated selection runs, noise features, correlation filtering, and final refitting address failures caused by large pools of interrelated engineered features.After selection, typically only a few dozen of several thousand engineered features remain for prediction.
3. Experimental Results
Across five regression datasets, autofeat improved on standard ridge regression while preserving interpretability, though it generally remained below random-forest performance and could overfit with deeper engineering.
- The experiments compared autofeat regression models using one, two, or three engineering steps across five regression datasets.
- Autofeat clearly outperformed standard linear ridge regression but generally did not match random-forest regression, while retaining interpretability.
- With one feature-engineering step, autofeat generated 2–11 additional features across datasets; with two or three steps, it produced 31 additional features on average.
- Most selected features were ratios or products of transformed features, rather than only standard polynomial expansions.
- Three engineering steps could overfit training data because complex features captured noise as well as signal.This issue was serious for diabetes and boston, where over 30k and 50k features were generated from fewer than 500 data points.
4. Conclusion
The autofeat library combines automated nonlinear feature generation and selection with linear models to improve accuracy while preserving transparent, traceable results. Its demonstrated gains are strongest relative to linear regression, with occasional advantages over other nonlinear models, within a scope focused on heterogeneous scientific datasets.
- Autofeat generates many nonlinear features and iteratively selects a small subset to improve linear-model accuracy while retaining traceable results.
- Autofeat targets heterogeneous scientific datasets and is not intended to compete with featuretools for relational data or tsfresh for time series.
- AutoFeatRegressor significantly improves linear-regression performance and sometimes outperforms other nonlinear ML models across several datasets.