Source-linked AI summary
Deep learning in business analytics and operations research: Models, applications and managerial implications
Mathias Kraus, Stefan Feuerriegel, Asil Oztekin
TL;DR
Business analytics and operations research have limited research adoption of deep learning despite its promise for predictive modeling and growing practical importance. This overview reviews the field, examines three real-data operations-research case studies, and proposes a deep-embedded architecture. The case studies support deep learning's feasibility and effectiveness, while customized architectures improve performance over default designs.
Problem
Operations research remains in its infancy in adopting deep learning, despite the technology's expected prediction-performance improvements and importance to business analytics.
Method
The paper reviews deep learning from an operations-research perspective, compares it across three real-data case studies, and proposes a tailored deep-embedded network architecture.
Results
The case studies suggest deep learning is feasible and effective, while replacing a default architecture with the proposed deep-embedded architecture improves performance by 1.56% to 21.73% across all metrics.
Takeaways & Limitations
Researchers, managers, and practitioners receive recommendations and implications for using deep neural networks in operations research and business analytics.
Takeaways & Limitations
Accountability and interpretability are widely regarded as weaknesses of deep learning in neural networks.
Abstract
from arXiv · showhide
Business analytics refers to methods and practices that create value through data for individuals, firms, and organizations. This field is currently experiencing a radical shift due to the advent of deep learning: deep neural networks promise improvements in prediction performance as compared to models from traditional machine learning. However, our research into the existing body of literature reveals a scarcity of research works utilizing deep learning in our discipline. Accordingly, the objectives of this overview article are as follows: (1) we review research on deep learning for business analytics from an operational point of view. (2) We motivate why researchers and practitioners from business analytics should utilize deep neural networks and review potential use cases, necessary requirements, and benefits. (3) We investigate the added value to operations research in different case studies with real data from entrepreneurial undertakings. All such cases demonstrate improvements in operational performance over traditional machine learning and thus direct value gains. (4) We provide guidelines and implications for researchers, managers and practitioners in operations research who want to advance their capabilities for business analytics with regard to deep learning. (5) Our computational experiments find that default, out-of-the-box architectures are often suboptimal and thus highlight the value of customized architectures by proposing a novel deep-embedded network.
1. Introduction
Deep learning is gaining importance in business analytics because deep neural networks can model complex relationships and exploit large datasets, yet operations research has adopted the technology only sparsely. This overview explains its foundations, applications, performance benefits, architectural choices, and managerial implications.
- Deep neural networks can model highly complex, non-linear relationships between predictor and outcome variables.
- Large datasets and scalable optimization routines make deep learning increasingly suitable for business analytics applications.Traditional methods such as support vector machines and random forests can often not be applied to very large datasets.
- Deep learning has achieved strong results in speech recognition, translation, image-related tasks, and recommendation systems.Examples include speech transcription approaching professionally trained human accuracy and translation performance approaching that of humans.
- Operations research remains in its infancy regarding deep-learning adoption, with a literature review finding only three relevant papers after 12 of 15 matches were discarded.
- The overview presents deep-learning concepts, three operations-research case studies, architecture recommendations, and implications for managers and practitioners.It compares prediction and operational performance and proposes directions for future research.
- Default, out-of-the-box architectures are often insufficient, motivating the proposal of a novel deep-embedded network architecture.
2. Mathematical background: From neural networks to deep learning
Predictive analytics learns mappings from input features to outcomes by minimizing a task-specific loss, while deep neural networks increase representational flexibility through multiple layers. This added flexibility can model complex nonlinear relationships but also raises computational and data requirements.
- Predictive analytics: Classification predicts discrete labels, whereas regression predicts real-valued outcomes.The paper frames both tasks as learning a mapping f: x 7→y.
- Predictive analytics: Predictive models map input features x to unknown outcomes y by selecting parameters w that minimize a loss over samples.The loss depends on the prediction task and objective, including classification or regression and the relative costs of errors.
- Deep neural networks: Greater model flexibility increases computational requirements and creates a need for many input–outcome pairs because of the curse of dimensionality.The paper also identifies overfitting prevention and architecture selection as central practical challenges.
- Neural networks: Artificial neural networks consist of connected units that transform weighted inputs through nonlinear activation functions into outputs.A single-layer perceptron combines inputs using weights and a bias before applying an activation function such as ReLU.
- Neural networks: Neural-network optimization is difficult because nonlinear activations make the problem non-convex and without a direct closed-form global solution.Gradient descent is used to find local optima, with reported solutions shown to be close to the global optimum.
- Deep neural networks: Deep neural networks stack multiple layers, increasing parameter count and flexibility for representing highly nonlinear functions.Practical experience suggests deeper models can better reduce generalization error, although architecture depth varies substantially across applications.
2.3. Model estimation
Deep neural-network estimation minimizes loss through gradient-based optimization and backpropagation. Because these models are computationally demanding and prone to overfitting, practical training uses minibatches, learning-rate controls, optimization variants, and regularization.
- Gradient-based optimization: Deep-network training updates parameters across all layers by backpropagating loss derivatives from the output toward the input.Backpropagation uses the chain rule and reuses computations, while matrix operations can be parallelized on GPUs.
- Estimation challenges: Deep-network optimization is computationally challenging because of many free parameters, and optimizing a neural network is NP-hard even when two-thirds of training outputs must be correct.This complexity creates a trade-off between expressive capacity and overfitting risk.
- Gradient-based optimization: Deep-network optimization can remain slow because each parameter update may require predictions for all training samples.Stochastic gradient descent reduces this burden by approximating the full loss with a smaller minibatch, typically recommended at 32–256 samples.
- Gradient-based optimization: The learning rate controls optimization step size: excessive values cause oscillations, whereas insufficient values slow optimization.Early stopping terminates training when validation performance fails to improve for a specified period.
- Optimization variants: Momentum improves stochastic-gradient training when gradient directions change by accumulating a moving average of past gradients.Common deep-learning optimizers include Adam, Adagrad, Adadelta, and RMSProp, but which performs best remains unresolved.
- Regularization: Regularization methods address overfitting by penalizing large weights, randomly dropping neurons, or normalizing layer outputs.Weight decay smooths decision boundaries, dropout limits co-adaptation, and batch normalization can permit higher learning rates with less careful initialization.
2.4. Advanced architectures
Convolutional networks exploit local spatial structure, whereas recurrent networks process variable-length sequences by accumulating information in hidden states. Gated variants regulate information flow to improve recurrent optimization.
- Convolutional neural network: Convolutional networks use local neighborhoods and shared filters to transform spatially structured inputs.Their layered structure progresses from locally identified concepts toward higher-level concepts.
- Convolutional neural network: Convolutional neurons connect densely within restricted neighborhoods rather than across the entire input.This design exploits spatial dependencies such as neighboring pixels.
- Recurrent neural networks: Traditional machine learning assumes fixed-dimensional input vectors, making it poorly suited to sequences with varying lengths.Recurrent neural networks are designed to handle sequential inputs of arbitrary length.
- Recurrent neural networks: Recurrent networks reuse the same weights while combining each current element with the previous hidden state.Information is passed onward and accumulated into a state encoding the sequence.
- Gated recurrent architectures: GRUs use update and reset gates, while LSTMs add a cell with forget, input, and output gates to control stored information.The gates determine which values are retained, discarded, written, or emitted.
- Gated recurrent architectures: GRUs have fewer parameters than LSTMs, while practical evidence reports comparable performance across varied tasks.Long sequences can nevertheless make recurrent optimization numerically unstable.
3. Methods and materials
Deep learning can process raw heterogeneous data and reduce reliance on manual feature engineering. The proposed architecture embeds categorical variables, combines them with numerical inputs, and supports multiple neural-network families.
- Motivation: Deep neural networks can handle raw pixels, characters, words, or time series without manual feature engineering.Traditional approaches instead construct rules and extract specific representations from the data.
- Categorical variables: Embeddings replace sparse one-hot vectors with low-dimensional dense representations that can be optimized jointly with the neural network.This avoids optimization difficulties associated with large one-hot vectors.
- Proposed architecture: The architecture targets heterogeneous data containing both categorical and numerical inputs.Dense categorical representations support numerically stable optimization even with many categories.
- Proposed architecture: The proposed architecture maps each categorical variable to a dense embedding and concatenates all embeddings with numerical variables.The resulting vector supplies input to MLP, RNN, or CNN layers.
- Regularization and flexibility: The architecture is flexible across MLPs, CNNs, and RNNs, with batch normalization and dropout used to counteract overfitting.The case studies instantiate deep-embedded DNN and deep-embedded LSTM variants.
Algorithm 1
The experimental design jointly trains embeddings and neural-network components, compares them with traditional and embedding-based baselines, and evaluates performance using prediction and cost metrics.
- Algorithm 1: The deep-embedded architecture jointly trains categorical embeddings with the downstream neural network instead of using pre-training.This design is intended for operations-research settings with scarce data.
- Algorithm 1: A deep-embedded LSTM integrates jointly trained embeddings with recurrent sequence modeling.The authors describe this as a distinction from prior embedding approaches.
- Algorithm 1: Dropout-based embeddings are used to address the overfitting risk introduced by joint training.Naive embeddings serve only as a baseline.
- Experimental setup: Traditional models use 10-fold cross-validation, whereas deep learning uses 10% of training samples for validation.Validation splits are random for case study 1 and chronological for the remaining time-series predictions.
- Experimental setup: The experiments cover three case studies with public and proprietary industry data and compare traditional models, naive embeddings, and deep networks.Benchmarks include linear models, random forests, support vector machines, and single-layer neural networks.
- Evaluation: Regression is evaluated primarily with mean squared error, classification with area under the curve, and all models through operations-research cost metrics.Percentage errors are not used because they do not measure costs.
4. Numerical results
Across service-request forecasting, sales forecasting, and insurance classification, deep-embedded networks outperform traditional and default neural baselines, with improvements also reflected in statistical and cost evaluations.
- Service-request forecasting: 3.75% improvement over the best baseline was achieved by the deep-embedded architecture for hourly service-request forecasting.The deep-embedded model outperformed all baseline models, including the default neural network.
- Service-request forecasting: 29,658.0 monetary units in savings were estimated for the service-request forecasting application.The cost evaluation accounts for prediction errors associated with idle workers or unsatisfied customers.
- Insurance classification: 0.01 AUC improvement, equivalent to 1.56%, was achieved by the deep-embedded DNN in insurance-score classification.The deep-embedded architecture outperformed all baseline models, while deeper architectures did not improve performance.
- Insurance classification: Deep learning improved predictive accuracy over traditional and out-of-the-box DNN models in the insurance application.The authors connect this improvement to more precise identification of insurance scores and premiums.
- Data-size sensitivity: Random forests outperformed deep neural networks with up to 10% of training observations, but the deep network performed favorably with the complete dataset.This experiment supports the need for large datasets to train deep neural networks.
- Managerial interpretation: The proposed embedding visualization combines dimensionality reduction with marginal effects to support store comparison and location planning.It can also inform store-disposal decisions by displaying trade-offs between sales and similarity to the core business area.
5. Discussion
The discussion presents deep learning as a feasible approach for business analytics and operations research, while emphasizing that effective adoption requires customized architectures, substantial data, and careful implementation. It also identifies unresolved challenges involving configuration, uncertainty quantification, interpretability, and deployment.
- Empirical findings: Deep learning can consistently outperform traditional machine-learning models in prediction and operational performance across the case studies.The discussion characterizes deep learning as feasible and effective for the examined operations-research applications.
- Empirical findings: Deep neural networks can identify previously unknown, useful patterns more accurately than random forests, artificial neural networks, and support vector machines.The discussion attributes this advantage partly to DNNs’ larger parameter space and fewer strong mathematical assumptions.
- Customized architectures: 1.56%–21.73% performance improvements were obtained across all metrics by replacing the default architecture with the proposed deep-embedded architecture.This result supports customizing architectures rather than relying on default, out-of-the-box designs.
- Requirements and caveats: Effective deep learning requires extensive datasets, because firms need large-scale data for the technique to learn better predictions.The discussion also notes that deep learning introduces complex architectures requiring thorough understanding and careful implementation.
- Managerial implications: Deep neural networks reduce reliance on feature engineering by learning data-driven representations, which is especially useful for sequential data such as time series and natural language.Raw sequential data can be fed directly into the network, while embeddings can be customized to domain-specific applications.
- Limitations and future research: Operations-research adoption remains limited, and practitioners face challenges in identifying valuable use cases, configuring networks, quantifying uncertainty, and interpreting results.The discussion points to AutoML, probabilistic or Bayesian variants, and further interpretability research as possible remedies.
6. Conclusion
The conclusion frames deep learning as increasingly important for business analytics as organizations generate more data and seek better predictive support. Across diverse operations-research case studies, DNNs improve operational performance, while customized architectures can outperform default designs.
- Business analytics uses data from varied sources to improve understanding of business processes, operations, and systems.
- As data generation accelerates, advances in predictive analytics become increasingly pertinent for decision-making and business competition.Even minor prediction-accuracy improvements can increase revenues.
- Case studies from different operations-research areas were selected to validate that DNNs improve operational performance.
- Customized network architectures are often beneficial for improving operational performance.
- 1.56% to 21.73% performance improvements were achieved across all metrics over a default, out-of-the-box architecture.
Appendix A. Runtime analysis
Appendix A reports the runtime required to optimize machine-learning-model parameters across the paper’s case studies. The appendix identifies the runtime table and its measurement unit, but the supplied passages do not report the individual runtimes.
- Table A.1 reports the runtime for optimizing parameters inside the machine-learning models.
- The supplied appendix material identifies an arXiv version dated 12 September 2019.
- Runtime is measured in minutes and seconds for the paper’s case studies.
Appendix B. Ex post interpretation
The appendix uses partial dependence and Shapley-value analyses to interpret model behavior and feature effects on predicted sales. The supplied results highlight calendar timing, assortment, store type, weekdays, and school holidays.
- Partial dependence plots and Shapley values are used to interpret model behavior.
- Increasing sales are associated with around the 20th calendar week and especially at the end of the year.
- Assortment and specific store types (= 0) are associated with decreasing effects on predicted sales for the upper sample.
- The middle of the week, represented by day of week equals three, and school holidays lead to increased predicted sales.
- Figure B.1 shows the marginal effect of the scaled week-of-year variable on the prediction outcome.
Appendix C. Hyperparameter tuning
Appendix C documents the grid search used for hyperparameter tuning and notes two modeling choices affecting evaluation and computation. The supplied passages do not list the searched parameter values.
- Table C.2 reports the tuning parameters used in the grid search.
- ARMA and ARIMA models are evaluated only for time-series prediction tasks.
- A kernel approximation is used to reduce computation time.
- The appendix presents hyperparameter tuning as a grid-search procedure.
Appendix D. Loss curves
Figure D.3 shows training and validation loss across epochs for LSTM and GRU networks used in load and sales forecasting. The curves indicate GRU overfitting after about 40 epochs in load forecasting, motivating additional regularization.
- Figure D.3 tracks training and validation loss by epoch for LSTM and GRU networks in load and sales forecasting.One epoch is a complete pass of all training samples through optimization.
- One epoch represents a complete pass of all training samples through the optimization process.
- Around 40 epochs, the GRU overfits in the load forecasting study, indicating that additional regularization is needed to stabilize optimization.