Automated machine learning: AI-driven decision making in business analytics

1. Introduction

• The paper frames AI-enabled decision-making as important amid competition and technological disruption across industries adopting advanced analytics.Source: §1 · p. 1
• Business analytics combines machine learning, statistics, information systems, operations research, and management science, and includes descriptive, predictive, and prescriptive analytics.Source: §1 · p. 1
• AutoML automatically tunes and compares algorithms to find hyperparameter combinations, while preprocessing and feature engineering are only partly supported.The paper identifies automation of the complete predictive-modeling process as a longer-term goal.Source: §1 · p. 1
• The study tests whether off-the-shelf AutoML can match or outperform manually trained ML models, addressing the expertise required to tune models and the shortage of analytics talent.The paper presents faster development cycles and a possible response to the ML skill gap as motivations for this comparison.Source: §1 · pp. 1–2
• The paper argues that human oversight remains relevant, presenting augmentation rather than pure automation as a focus for intelligent systems.Source: §1 · p. 2
• The paper benchmarks H2O AutoML against a manually created ML model to compare predictive ability and ease of use, then discusses managerial implications and future research.Source: §1 · p. 2

2.1. AutoML

• AutoML can include preprocessing and feature engineering, but the paper focuses on hyperparameter optimization and model choice because current solutions handle preprocessing poorly.The passage describes H2O as one of the most mature frameworks and cites benchmark studies reporting strong classification and regression performance.Source: §2.1 · p. 2
• Random grid search samples hyperparameter values uniformly at random, and H2O can stop the search after a set model count or runtime.Source: §2.1 · p. 2
• H2O AutoML creates candidate models across GLM, random forest, gradient boosting, and deep learning, then uses stacking to form two super learners.One combines all candidate models; the other combines the best model from each family, with a non-negative GLM meta-learner.Source: §2.1 · p. 2

2.2. Experimental design

• The study's main comparison is H2O AutoML versus a manually trained super-learning ensemble on real-world datasets.Source: §2.2 · p. 2
• The benchmark covers three business domains: credit risk, insurance claims, and marketing.Source: §2.2 · p. 2
• The experiment uses real-world datasets to evaluate the competing modeling approaches.Source: §2.2 · p. 2

2.2.1. Datasets

• Three publicly available datasets from UCI or Kaggle contain 23, 57, and 16 features, respectively, and binary response variables for default, insurance claims, or marketing sales.Table 1 is identified as a summary of the case-study datasets.Source: §2.2.1 · p. 2, Table 1 · p. 3

2.2.1.1. Credit risk.

• The credit-risk dataset contains 30,000 Southeast Asian credit-card clients, with 23 features and binary default or non-default outcomes.It includes historical payment and demographic information.Source: §2.2.1.1 · p. 2

2.2.1.2. Claims prediction.

• The insurance dataset contains 595,212 automotive policyholders, including 573,518 non-filed claims and 21,694 filed claims.Source: §2.2.1.2 · p. 2
• For the claims-prediction case, H2O AutoML trains base learners and stacks them into two super learners, then ranks outcomes by the selected evaluation metric.One super learner uses all trained classifiers; the other uses the best classifier from each ML family.Source: Figure 1 · p. 3

2.2.2. Preprocessing

• The section introduces the preprocessing steps required before the experiment.The following steps cover random undersampling, categorical encoding, and training/test splitting.Source: §2.2.2 · p. 3

2.2.2.1. Random undersampling.

• Random undersampling balances class distributions by removing observations from the majority class, which loses some information.The datasets were brought to a balanced state using this approach.Source: §2.2.2.1 · p. 3

2.2.1.3. Marketing.

• The marketing dataset contains 45,211 direct-campaign observations: 39,922 unsuccessful and 5,289 successful outcomes.Success denotes a conversion or final sale; the data include 16 features and a binary response.Source: §2.2.1.3 · p. 3
• Oversampling the minority class was an alternative, but it was set aside because it would enlarge the dataset and increase training time.The passage says this was especially problematic for datasets already containing substantial information.Source: §2.2.1.3 · p. 3

2.2.2.2. Encoding.

• H2O’s one_hot_explicit setting converts a categorical feature with N levels into N + 1 numeric columns.ML models require numeric inputs; one-hot encoding or, for ranked categories, ordinal encoding are described as options.Source: §2.2.2.2 · p. 3

2.2.2.3. Training/Test split.

• The 80:20 split reserves 20% of the data for testing classifier generalization, while cross-validation divides the 80% training portion into training and validation sets.The same cross-validation setup is used for the later fusion of base learners into the meta-learner.Source: §2.2.2.3 · pp. 3–4

• The comparison uses GLM, Random Forest, Gradient Boosting Machine, and Deep Feedforward Neural Network base classifiers, stacked into super learners and assessed with four metrics.The metrics are AUC, Accuracy, F-score, and LogLoss; the best model is automatically selected using a chosen evaluation measure.Source: §2.2.3 · p. 4

• RStudio hosted preprocessing, fitting, and evaluation; the H2O R package set up AutoML and the Random Forest, Gradient Boosting Machine, and Deep Learning baselines.H2O is an open-source Java platform that can scale from desktop environments to larger-scale settings and integrate with RStudio through a REST API.Source: §2.2.4 · p. 4

• The study benchmarked H2O AutoML against a manually optimized super learner using AUC, Accuracy, F-score, and LogLoss, with Accuracy and F-score reported at a 0.5 threshold.The base models were trained using grid search, random search over predefined parameter ranges, and manual adjustments.Source: §3 · p. 4
• Gradient Boosting had the highest overall base-classifier performance, Random Forest followed, and Deep Learning scored lowest across all three datasets.Table 2 presents the optimized base-classifier results for all three case studies.Source: §3 · p. 4, Table 2 · p. 4
• Using Random Forest, Gradient Boosting Machine, and Deep Learning together gave the best stacked-ensemble performance across the three case studies.The study tested all three combinations of the baseline models.Source: §3 · p. 4
• AUC difference of 0.002 favored the stacked ensemble over H2O AutoML across all three datasets.The cases covered credit risk, insurance claims, and marketing.Source: §3 · p. 4
• Credit-risk differences were 0.003 Accuracy, 0.003 F-score, and 0.002 LogLoss; insurance differences were 0.004 Accuracy, 0.002 F-score, and 0.001 LogLoss.The passage reports these as performance differences between the stacked ensemble and AutoML.Source: §3 · p. 4
• AutoML slightly outperformed the stacked ensemble on marketing Accuracy (−0.001) and F-score (−0.002), while the stacked ensemble led in most other metric comparisons.Source: §3 · p. 4

4.1. Discussion of results

• Across three real-world case studies, H2O AutoML did not outperform the manually tuned classifier, and its underlying models failed to match manual prediction accuracy.The reported comparisons cover credit risk, insurance, and marketing; Table 3 presents the final comparison.Source: §4.1 · pp. 4–5, Table 3 · p. 4
• H2O does not test whether smaller candidate-model subsets perform better, while weaker models can dilute a stacked ensemble; extending runtime did not significantly improve output.The authors note that careful hyperparameter tuning remains the route to maximum prediction accuracy, while AutoML can speed prototyping; the practical choice depends on whether a small performance edge warrants the extra tuning time.Source: §4.1 · p. 5
• The authors caution that AutoML cannot yet replace expert teams in its current state, and model performance can vary substantially with the training dataset.Business context also matters: credit decisions depend only partly on probability-of-default accuracy, since insurance or collateral can reduce loss given default.Source: §4.1 · p. 5

4.2. Managerial implications

• AutoML could support ML adoption by helping fill the talent gap and enabling fast prototyping, while preprocessing and turning predictions into actionable prescriptions remain unresolved.The authors describe AutoML as a step toward an end-to-end decision engine, but say corporations still need experts or consultants until those gaps are addressed.Source: §4.2 · p. 5

4.3. Future research

• Further research should address preprocessing, prescriptive decisions under uncertainty, and human augmentation rather than assuming fully off-the-shelf automation is the ideal endpoint.The authors suggest that integrating intelligent systems with human expertise may be more desirable than complete automation.Source: §4.3 · pp. 5–6, §4.3.2 · p. 6, §4.3.3 · p. 6

4.3.1. Preprocessing

• Preprocessing can consume 80–90% of data-science time, and machines may not categorize semantically related features as well as domain experts.The authors identify automated feature engineering as a future research direction.Source: §4.3.1 · p. 6

4.3.2. Prescriptive analytics

• Moving from prediction to concrete managerial action remains poorly understood, particularly because decision-making must account for uncertainty.The authors identify research on deep reinforcement learning for prescriptive analytics and managerial decisions in uncertain environments as a possible direction.Source: §4.3.2 · p. 6

4.3.3. Human in the loop

• A human-in-the-loop approach could combine AI and human capabilities, improving control, feedback, and understanding of model predictions.The authors present augmentation, rather than full automation, as a promising direction for future research.Source: §4.3.3 · p. 6

• H2O AutoML does not reach the prediction accuracy possible through careful manual model adjustment, although the study finds it can still be a powerful tool.Source: §5 · p. 6
• Digitalization has increased demand for ML and AI experts, contributing to a skill shortage that has slowed adoption in business analytics.Source: §5 · p. 6
• AutoML can provide ML experts a prototyping baseline to help accelerate development and deployment, while making ML models more accessible to non-experts.Source: §5 · p. 6
• AutoML is a step toward an end-to-end business analytics decision engine and could foster human empowerment through an augmented workforce.Source: §5 · p. 6