PaperlayerView exact revision on arXiv

Source-linked AI summary

An Empirical Analysis of Feature Engineering for Predictive Modeling

Jeff Heaton

arXiv:1701.07852v2cs.LG

TL;DR

Feature engineering is manual and time-consuming, while different machine-learning models may respond differently to engineered feature types. The paper empirically tests this by generating datasets designed around engineered expressions and evaluating four model types’ ability to synthesize them. The results show that model types perform differently across engineered-feature classes, supporting recommendations about feature selection and model ensembles.

  • Problem

    Feature engineering requires substantial manual effort, and evidence is needed about which engineered-feature types different model types can synthesize themselves.

  • Method

    The study evaluates four regression model types on datasets designed to test their ability to synthesize specific engineered features.

  • Results

    The studied model types perform differently across engineered-feature classes, with some classes synthesized more successfully by particular models.

  • Takeaways & Limitations

    The findings support recommendations about engineered-feature choices for particular model types and about combining model types in ensembles.

Abstract

from arXiv · show

Machine learning models, such as neural networks, decision trees, random forests, and gradient boosting machines, accept a feature vector, and provide a prediction. These models learn in a supervised fashion where we provide feature vectors mapped to the expected output. It is common practice to engineer new features from the provided feature set. Such engineered features will either augment or replace portions of the existing feature vector. These engineered features are essentially calculated fields based on the values of the other features. Engineering such features is primarily a manual, time-consuming task. Additionally, each type of model will respond differently to different kinds of engineered features. This paper reports empirical research to demonstrate what kinds of engineered features are best suited to various machine learning model types. We provide this recommendation by generating several datasets that we designed to benefit from a particular type of engineered feature. The experiment demonstrates to what degree the machine learning model can synthesize the needed feature on its own. If a model can synthesize a planned feature, it is not necessary to provide that feature. The research demonstrated that the studied models do indeed perform differently with various types of engineered features.

I. INTRODUCTION

Feature engineering transforms existing inputs into new features, but designing these transformations is labor-intensive and model-dependent. This paper empirically examines which engineered-feature types different regression models can synthesize and evaluates four popular model types.

  • Feature engineering is labor-intensive because data scientists spend substantial effort designing preprocessing pipelines and data transformations.
  • Feature engineering adds mathematical transformations of existing features, such as ratios, differences, BMI, and wind chill.These calculated fields are intended to expose interactions among existing features.
  • Prior work applied hand-created feature engineering successfully in competitions and across regression, rules-learning, and text-classification settings.
  • The experiments create datasets containing inputs and target expressions designed to test models’ ability to synthesize engineered features.The tested feature classes include distance between quadratic roots, power ratios, ratios of products, and roots of ratios.
  • The study compares four regression model types selected for their popularity and differing approaches: deep neural networks, gradient boosted machines, support vector regression, and random forests.Each experiment was repeated five times, using the best run for comparison.

A. Counts

The count feature represents how many input values satisfy a threshold condition. The dataset construction encodes sampled integer counts across 50-dimensional input vectors.

  • A count feature counts the elements of a feature vector that satisfy a specified condition, such as exceeding zero.
  • Equation 1 assigns 1 when x_i > t and 0 otherwise, producing indicator values for threshold comparisons.
  • The resulting y is an integer equal to the number of input values above threshold t.The count outcome is uniformly sampled from integers in [1, 50] when generating corresponding inputs.
  • Algorithm 1 generates rows with random integer outcomes and 50-column input vectors constructed to sum to each outcome.
  • Table I illustrates count-transformation rows in which y_1 records how many features x_1 through x_50 exceed zero.

B. Differences and Ratios

The differences-and-ratios dataset tests whether models can synthesize simple differences, ratios, and related mathematical transformations. Inputs are sampled uniformly, with denominators restricted to avoid division by zero.

  • The dataset samples x observations uniformly from [0, 1] and generates y using differences and ratios of those observations.
  • Denominator observations are sampled from [0.1, 1] to avoid division by zero.
  • The tested transformations include simple difference, simple ratio, and power ratio expressions.The passage begins listing the selected equations but is truncated after the power-ratio reference.

C. Distance Between Quadratic Roots

This experiment tests synthesis of the distance between the roots of a quadratic equation. It generates inputs uniformly over a bounded interval and discards invalid results.

  • The target feature is the distance between quadratic roots, calculated as the difference between the two quadratic-formula outputs.
  • The dataset samples x values uniformly from [−10, 10] and discards invalid results.

D. Distance Formula

The paper tests whether models can synthesize distance-based and transformation-based engineered features from generated datasets. It uses uniformly sampled inputs for distance, logarithmic, power, and root transformations.

  • D. Distance Formula: The distance formula contains a ratio inside a radical and computes Euclidean distance between two points.Inputs are x1, x2, x3, and x4, sampled uniformly from [0, 10].
  • D. Distance Formula: Distance-formula inputs are sampled uniformly from [0, 10] to generate the corresponding outcome.
  • E. Logarithms and Power Functions: The transformations are evaluated as engineered features that models may synthesize from their input data.The paper frames these functions as examples of mathematical transformations used in feature engineering.
  • E. Logarithms and Power Functions: The paper also examines logarithm, second-power, and square-root transformations of a single input.These transformations are motivated by established uses in regression and other model types.
  • E. Logarithms and Power Functions: Logarithm and root datasets sample x uniformly from [1, 100].Each uses one x1 observation to generate one y1 observation.
  • E. Logarithms and Power Functions: The second-power dataset samples x uniformly from [1, 10].The sampled x1 values produce y1 by applying the second-power transformation.

F. Max of Inputs

The paper generates datasets for maximum-value and mathematical-expression features using controlled random sampling. These experiments test synthesis of maxima, polynomials, rational combinations, and ratio-power expressions.

  • F. Max of Inputs: The max-of-inputs feature takes ten uniformly sampled inputs from [1, 100] and returns their maximum.Equation 12 defines the calculated outcome.
  • Polynomial Features: Polynomial experiments test models’ ability to synthesize features following the polynomial in Equation 13.
  • Polynomial Features: Polynomial data are generated by uniformly sampling x from real numbers in [0, 2).The y1 value is calculated from x1 using Equation 13.
  • Rational Features: Rational-feature experiments combine differences and polynomials, and also include a ratio-power equation analogous to BMI.
  • Rational Features: Rational-difference datasets sample four observations from [1, 10], while rational-polynomial datasets sample one observation from [1, 10].

IV. RESULTS ANALYSIS

The experiments compare model performance across engineered-feature datasets using normalized error scores. Results show strong variation by model and feature class, with neural networks performing consistently well except on ratios of differences.

  • Evaluation: NRMSD normalizes error across datasets whose y-value ranges differ considerably.RMSE is in the same units as y, so normalization enables cross-dataset comparison.
  • Overall Results: Some model types perform much better than others on particular classes of engineered features.This is the paper’s main cross-model result.
  • Overall Results: All four models easily learned single-feature log, polynomial, power, and root transformations.
  • Overall Results: None of the models successfully learned the ratio-difference feature.Table III reports scores for each equation type and model.
  • A. Neural Network Results: The deep neural network performed well on every equation type except the ratio of differences and consistently outperformed the other three models on the remaining types.Its layered weighted transformations provide insight into this performance.

B. Support Vector Machine Results

The SVM experiments used a Gaussian-kernel SVR with normalized inputs and found that several engineered transformations were difficult to synthesize.

  • Model setup: The experiments evaluated nine SVMs spanning three C values and three γ values, reporting the best combination for each feature type.The tested C values were 0.001, 1, and 100; γ values were 0.1, 1, and 10.
  • Model setup: SVM inputs were normalized to [0,1], adding calculations that made this feature-engineering comparison less pure than those for other models.The SVM used a Gaussian kernel.
  • Findings: SVMs found max, quadratic, ratio-of-differences, polynomial-ratio, and ratio features difficult to synthesize.All other feature experiments were within a low NRMSD level.
  • Interpretation: An SVR produces its final output through a decision function using an input vector, learned coefficients, a nonlinear kernel, and an intercept.The paper used a radial basis function kernel based on the Gaussian function.

C. Random Forest Results

The random forest experiments used ensembles of 100 decision trees and found greatest difficulty synthesizing standard deviation, ratio-of-differences, and sum features.

  • Model setup: Random forests were built as ensembles of 100 classifier trees generated from randomly sampled training data.The ensemble generally outperformed individual trees.
  • Findings: Random forests had the most difficulty synthesizing standard deviation, ratio-of-differences, and sum features.This result came from the model’s attempts to synthesize the engineered features.
  • Comparison: GBM had the same greatest difficulties as random forests: standard deviation, ratio-of-differences, and sum.GBM used gradient-based optimization to produce combinations of trees.

V. CONCLUSION & FURTHER RESEARCH

The experiments show that different machine-learning model types benefit from different synthesized features, while neural networks/SVMs and random forests/GBMs show similar feature preferences within their groups. The study uses these findings to recommend model-feature choices and ensemble pairings, but notes that limited hyperparameter tuning constrains individual results.

  • Conclusion: Different model types benefit from different sets of synthesized features.Neural networks and support vector machines generally favor similar engineered features, as do random forests and gradient boosting machines.
  • Conclusion: Recommendations cover both feature types for particular models and model combinations for ensembles.
  • Conclusion: Model type strongly influences which engineered features should be considered.The experiments found that ratio-of-differences features were not synthesized well by any explored model.
  • Conclusion: Neural networks or support vector machines might ensemble well with random forests or gradient boosting machines.The paper attributes this potential complementarity to the two groups synthesizing different features.
  • Limitations: The study did not spend significant time tuning models separately for each dataset.Additional hyperparameter tuning might improve results for individual models and datasets.
  • Further research: Future research will examine additional engineered features and a wider set of machine-learning models.Features composed of multiple input features are identified as a logical focus.
Loading 1701.07852v2…