PaperlayerView exact revision on arXiv

Source-linked AI summary

Maximum Relevance and Minimum Redundancy Feature Selection Methods for a Marketing Machine Learning Platform

Zhenyu Zhao, Radhika Anand, Mallory Wang

arXiv:1908.05376v1stat.MLcs.LG

TL;DR

Large marketing machine-learning systems must select useful features from rich feature sets while controlling computational, maintenance, overfitting, and interpretation costs. This paper extends mRMR with nonlinear redundancy and model-based relevance measures, evaluates variants across synthetic and real-world marketing datasets, and implements the selected method in production. The study reports robust or model-specific performance for different variants and describes successful platform deployment with business impact from an online experiment.

  • Problem

    Large feature sets create computation, overfitting, maintenance, and interpretation challenges, while selecting a useful nonredundant subset is important for marketing machine learning.

  • Method

    The paper extends mRMR with nonlinear redundancy and model-based relevance measures, evaluates eight methods on synthetic and real-world marketing datasets, and implements the selected method in an automated platform.

  • Results

    FCQ shows robust performance and high computation efficiency across classification models, while RFCQ and RFRQ perform optimally for Random Forests and competitively for other models.

  • Takeaways & Limitations

    The FCQ implementation made model training and prediction more scalable, feature pipelines easier to manage, and model interpretation more straightforward in the marketing platform.

Abstract

from arXiv · show

In machine learning applications for online product offerings and marketing strategies, there are often hundreds or thousands of features available to build such models. Feature selection is one essential method in such applications for multiple objectives: improving the prediction accuracy by eliminating irrelevant features, accelerating the model training and prediction speed, reducing the monitoring and maintenance workload for feature data pipeline, and providing better model interpretation and diagnosis capability. However, selecting an optimal feature subset from a large feature space is considered as an NP-complete problem. The mRMR (Minimum Redundancy and Maximum Relevance) feature selection framework solves this problem by selecting the relevant features while controlling for the redundancy within the selected features. This paper describes the approach to extend, evaluate, and implement the mRMR feature selection methods for classification problem in a marketing machine learning platform at Uber that automates creation and deployment of targeting and personalization models at scale. This study first extends the existing mRMR methods by introducing a non-linear feature redundancy measure and a model-based feature relevance measure. Then an extensive empirical evaluation is performed for eight different feature selection methods, using one synthetic dataset and three real-world marketing datasets at Uber to cover different use cases. Based on the empirical results, the selected mRMR method is implemented in production for the marketing machine learning platform. A description of the production implementation is provided and an online experiment deployed through the platform is discussed.

I. INTRODUCTION

The paper studies mRMR feature selection for large-scale marketing machine learning, where rich feature sets create computational, maintenance, overfitting, and interpretation challenges. It extends and evaluates mRMR variants before selecting one for implementation in an automated marketing platform.

  • Motivation: Large marketing models can use hundreds or thousands of features, but directly using all available features may cause inefficiency, over-fitting, maintenance workload, and interpretation difficulty.
  • Motivation: Feature selection reduces these costs by accelerating computation, improving prediction accuracy, lowering feature-pipeline maintenance, and simplifying model interpretation and diagnosis.
  • mRMR rationale: mRMR selects features using both predictive relevance and redundancy, addressing the limitation that the best m individual features may not form the best subset of m features.
  • Contributions: The paper contributes two mRMR extensions: nonlinear association for redundancy and model-based feature importance for relevance.
  • Study design: The study evaluates existing and extended mRMR methods on one synthetic dataset and three real-world marketing datasets, measuring both model performance and computation speed.
  • Contributions: The best-performing method is implemented and deployed in a large-scale automated machine learning platform for marketing.

A. mRMR Framework

The mRMR framework greedily selects unselected features by balancing their relevance to the response against redundancy with already selected features. Its variants differ in how they measure relevance, redundancy, and the way those terms are combined.

  • Core framework: For a candidate feature Xi, the mRMR criterion combines mutual-information relevance to the class label with average mutual-information redundancy against selected features.
  • Core framework: At each step, the feature with the highest mRMR importance score is added to the selected feature set.
  • Discrete-feature variants: MID uses a difference between relevance and redundancy, whereas MIQ uses a quotient to balance the two terms.
  • Continuous-feature variants: For continuous features, FCD and FCQ avoid expensive density estimation by using F-statistics for relevance and Pearson correlation for redundancy.
  • Continuous-feature variants: FCD combines relevance and redundancy through a difference, while FCQ uses a quotient scheme.

III. PROPOSED MRMR EXTENSIONS

The paper extends mRMR to better represent nonlinear feature relationships and to align relevance scoring with a known downstream model. These extensions replace Pearson correlation with RDC for redundancy and F-statistics with embedded model importance for relevance.

  • Extension rationale: The extensions target nonlinear associations between features and the response variable that Pearson correlation may not capture.
  • Redundancy extension: RDC replaces Pearson correlation as the redundancy measure in the FCQ and FCD variants.
  • Relevance extension: When the downstream model is known, an embedded feature-importance score can provide a model-related relevance measure.
  • Relevance extension: The proposed model-based extension replaces the F-statistic with feature importance scores such as those from random forests.

A. Non-linear Association Extension for Redundancy

The paper extends mRMR by using nonlinear dependence for redundancy and model-based importance for relevance, with quotient criteria accommodating differing measurement scales.

  • Non-linear redundancy measure: RDC measures nonlinear dependence through copula transformations, random nonlinear projections, and the largest canonical correlation.The resulting RDC score replaces Pearson correlation in redundancy terms such as FCQ and FCD.
  • Non-linear redundancy measure: FRQ replaces correlation-based redundancy with RDC in an F-test-based quotient criterion.The paper defines FRQ after describing RDC as the nonlinear association score.
  • Model-based relevance measure: Model-based feature importance is introduced as the relevance measure for downstream tree-based classification models.The extension is motivated by marketing applications using random forests and gradient boosting trees.
  • Criterion design: The quotient scheme is chosen because model-based relevance and correlation-based redundancy have different scales.The paper states that quotient criteria are relatively immune to scale differences compared with difference criteria.
  • Model-based relevance measure: RFCQ uses Random Forest importance for relevance and correlation for redundancy in a quotient-form mRMR criterion.The Random Forest importance score is denoted IRF(Y, Xi).

IV. EMPIRICAL EVALUATION

The empirical evaluation compares eight feature-selection methods across one synthetic and three real marketing datasets, measuring computation speed and downstream classification accuracy under a shared setup.

  • Evaluation scope: The study evaluates eight feature-selection methods across one synthetic dataset and three real marketing datasets covering multiple business use cases.The methods include seven mRMR variants and Random Forest feature importance as a benchmark.
  • Evaluation criteria: Performance is assessed along two dimensions: feature-ranking computation speed and downstream model accuracy.Accuracy is measured using AUC and F1-score.
  • Evaluation criteria: Three downstream classifiers—Naive Bayes, Logistic Regression, and Random Forest—represent probabilistic, regression-based, and tree-based model families.Using different classifier families makes the evaluation representative across classification settings within the study’s scope.
  • Evaluation procedure: The procedure splits data, ranks selected features, trains models with increasing feature counts, predicts test probabilities, and averages AUC and F1-score.Synthetic data uses repeated generation trials, while real data uses four-fold cross-validation.
  • Evaluation scope: 96 empirical result sets arise from 8 feature-selection methods × 3 classifiers × 4 datasets.The evaluation uses a shared algorithm implementation and computation environment across datasets.

B. Synthetic Data Example

The synthetic classification data contains known informative, redundant, and irrelevant features, enabling controlled evaluation of mRMR variants. Across accuracy, redundancy, and computation measures, model-based relevance variants and selected mRMR methods perform strongly, particularly with compact feature subsets.

  • Data generation: Independent informative features are generated by nonlinear spline transformations, while redundant features derive from random subsets of informative features through linear combinations or spline transformations.Irrelevant features are sampled independently from N(0, 1).
  • Data generation: The synthetic dataset contains 70 features: 10 independent informative, 20 linear redundant, 20 nonlinear redundant, and 20 irrelevant.Its controlled construction makes the feature types and relationships known for evaluation.
  • Evaluation results: AUC and F1 Score show similar relative performance across feature-selection methods, so the discussion focuses mainly on AUC while generalizing conclusions to F1 Score.The evaluation results are presented in Figure 2 and Table II.
  • Evaluation results: Random Forest generally outperforms the other classifiers, while RFRQ reaches optimal Random Forest performance with as few as 6 features.Feature selection also improves Naive Bayes over using all 70 features and gives other models comparable performance with fewer features.
  • Evaluation results: FCQ, RFCQ, and RFRQ perform well, with RFCQ and RFRQ reaching the RF method’s performance using fewer features than RF selection.The model-based relevance extensions RFCQ and RFRQ appear successful, whereas nonlinear association via RDC does not show significant improvement.
  • Redundancy and efficiency: mRMR-selected top features are less correlated than RF-selected features, reducing redundancy and improving interpretability especially for small feature subsets.FCQ, RFCQ, and RFRQ show lower correlations among top features, particularly among the top 6.
  • Redundancy and efficiency: FCQ and FCD are fastest, followed by RF and RFCQ, while MID, MIQ, FRQ, and RFRQ are relatively computationally expensive.Overall, FCQ and RFCQ show good performance across both accuracy and computation-time metrics.

C. Real Data Examples

Three real-world Uber marketing datasets spanning multiple use cases and regions were used to compare feature-selection methods across classification models. RFCQ, RFRQ, and FCQ generally offered strong accuracy, while FCQ and FCD were fastest.

  • Datasets: The evaluation used three Uber marketing datasets spanning multiple product use cases, regions, and datasets containing approximately 100–1,000 features.The datasets contained 100,000 to 1 million user-level observations.
  • Performance patterns: AUC increased rapidly when the feature set was small, particularly at five or fewer features, with F1-score patterns consistent with AUC.This pattern was reported across the real-data evaluation and compared with the synthetic example.
  • Classification models: Within Random Forests, RFCQ and RFRQ selected strong feature sets, while FCQ performed close to optimal as a model-free method.For Logistic Regression and Naive Bayes, model-based methods did not show a consistent advantage, whereas FCQ performed well across datasets.
  • Computation time: FCQ and FCD each ran in under one minute in all evaluated scenarios, whereas MID, MIQ, FRQ, and RFRQ took substantially longer.The longer runtimes were associated with the large feature set in Dataset 1 and large sample size in Dataset 3.
  • Overall selection: Considering accuracy and computation efficiency together, RFCQ was suitable for Random Forest downstream models, while FCQ offered robust accuracy across models and outstanding runtime.For Random Forest hyperparameter comparisons on Dataset 2, RFRQ, RFCQ, and FCQ consistently performed well.

V. IMPLEMENTATION IN PRODUCTION

The production marketing platform connects Uber’s user data to separate training and prediction pipelines. Its feature-selection module narrows generated features using empirical evaluation results, with FCQ implemented in the first production phase.

  • Platform: Uber’s marketing platform enables marketing teams to develop and deploy machine-learning models in a quick, easy, and scalable fashion.It connects to Uber’s rich user-data sources and uses separate training and prediction pipelines.
  • Training pipeline: The training pipeline architecture includes modules for feature and label generation, feature transformation, feature selection, sampling, and model training.These modules are presented in sequence in the architecture figure.
  • Production feature selection: After feature engineering generates a large feature set, the Feature Selection module narrows it to a relevant and diversified subset for modeling.The first production phase implemented FCQ based on the empirical evaluation results.
  • Production feature selection: The production implementation is described through the platform and feature-selector architectures.The paper presents these architectures as the focus of the following subsections.

A. Architecture

The architecture places feature selection between feature transformation and downstream sampling and model training. The selector accepts configuration and data inputs, then returns selected features and mRMR importance scores, with feature count chosen by AUC.

  • Pipeline architecture: The training pipeline orders Feature and Label Generator, Feature Transformer, Feature Selector, Data Sampler, and Model Training.The architecture is illustrated as the training pipeline structure.
  • Selector interface: The Feature Selector takes the full feature dataframe and label, selection method, maximum feature count, and pipeline type as inputs.The pipeline type distinguishes training from prediction.
  • Selector interface: The selector outputs a dataframe of selected features, the selected-feature list, and feature-importance scores represented by mRMR scores.These outputs support both modeling and inspection of the selected subset.
  • Feature-count selection: The number of selected features is the lowest n achieving the highest AUC when Random Forest models are evaluated over increasing feature counts.The search can range from one feature to a sufficiently large count such as n = 50.

B. Implementation

The feature-selection variants were developed and evaluated in Python, then implemented for production in Scala Spark. The implementation choice reflects performance, memory, concurrency, framework, and interoperability considerations.

  • Implementation transition: The mRMR variants were first developed in Python for the empirical evaluation and then implemented for the production platform.The production implementation used Scala Spark.
  • Implementation rationale: Scala Spark was chosen because JVM execution, concurrency primitives, and data-processing capabilities support large-dataset performance and memory management.The paper also cites Scala’s frameworks, libraries, and language features as factors in the decision.
  • Implementation rationale: Scala integrates well with upstream services implemented in Java.This interoperability was one of the stated reasons for selecting Scala Spark.

C. Challenges and Optimization

The initial mRMR implementation was too slow on real datasets, prompting optimizations that preserved accuracy while supporting scalable platform use and online validation.

  • Challenges: The first mRMR module ran considerably longer than acceptable on real datasets, requiring optimization and heuristic approximations while retaining accuracy.
  • Optimization: Optimization replaced loops with Scala map operations and combined Spark RDD and Dataframe APIs for more efficient processing.RDDs provided low-level control, while Dataframes supported higher-level operations, space savings, and faster execution.
  • Optimization: The selector was placed after downsampling so it processed a smaller representative sample rather than the entire dataset.
  • Optimization: Error handling defaulted to keeping all features when mRMR failed to finish in time.
  • Evaluation: Across synthetic and three real-world marketing datasets, FCQ showed robust performance and high efficiency, while RFCQ and RFRQ were strongest for Random Forests.
  • Production Impact: FCQ was implemented in Scala Spark, and the platform implementation supported scalable training and prediction, easier pipeline maintenance, and clearer model interpretation.An online experiment using a deployed cross-sell model demonstrated business impact.
  • Future Extensions: The proposed extensions can be generalized by substituting other nonlinear association measures or model-based feature-importance scores.
Loading 1908.05376v1…