PaperlayerView exact revision on arXiv

Source-linked AI summary

Profit Driven Decision Trees for Churn Prediction

Sebastiaan Höppner, Eugen Stripling, Bart Baesens, Seppe vanden Broucke, Tim Verdonck

arXiv:1712.08101v1stat.MLcs.LGstat.AP

TL;DR

Churn models are commonly selected for accuracy, but retention campaigns require attention to the costs of errors and benefits of correct classifications. ProfTree uses an evolutionary algorithm to optimize EMPC directly in decision-tree construction. In benchmarks on six real-life churn datasets, it was the overall most profitable model, with the best MPC results on all six datasets.

  • Problem

    Accuracy-based churn model selection does not account for misclassification costs and correct-classification benefits, despite their importance to retention-campaign profit.

    Source: §1 · p. 2, §2.3 · p. 8

  • Method

    ProfTree uses an evolutionary algorithm to optimize EMPC directly during decision-tree construction.

    Source: §1 · p. 3, §6 · p. 27

  • Results

    ProfTree was the overall most profitable classifier by EMPC and MPC, and was best by MPC on all six real-life churn datasets, compared with six other tree-based methods.

    Source: §5.3 · p. 26, §6 · p. 27

  • Takeaways & Limitations

    Decision trees offer an interpretable model that can help explain why customers defect.

    Source: §6 · p. 27

    The case study does not report C4.5 results because its implementation cannot restrict tree depth.

    Source: §4 · p. 17

Abstract

from arXiv · show

Customer retention campaigns increasingly rely on predictive models to detect potential churners in a vast customer base. From the perspective of machine learning, the task of predicting customer churn can be presented as a binary classification problem. Using data on historic behavior, classification algorithms are built with the purpose of accurately predicting the probability of a customer defecting. The predictive churn models are then commonly selected based on accuracy related performance measures such as the area under the ROC curve (AUC). However, these models are often not well aligned with the core business requirement of profit maximization, in the sense that, the models fail to take into account not only misclassification costs, but also the benefits originating from a correct classification. Therefore, the aim is to construct churn prediction models that are profitable and preferably interpretable too. The recently developed expected maximum profit measure for customer churn (EMPC) has been proposed in order to select the most profitable churn model. We present a new classifier that integrates the EMPC metric directly into the model construction. Our technique, called ProfTree, uses an evolutionary algorithm for learning profit driven decision trees. In a benchmark study with real-life data sets from various telecommunication service providers, we show that ProfTree achieves significant profit improvements compared to classic accuracy driven tree-based methods.

1. Introduction

Churn models are often selected for accuracy, although retention campaigns need models that account for profit, including the costs of errors and benefits of correct classifications. ProfTree integrates the expected maximum profit measure for customer churn (EMPC) into decision-tree construction using an evolutionary algorithm.

1. Introduction

  • Retention campaigns seek to identify potential churners among large customer bases and target those most profitable to retain.Companies use historical and behavioral information to estimate churn propensity and offer incentives to customers they seek to retain.Source: §1 · p. 2
  • EMPC-based selection addresses accuracy metrics’ neglect of profit, including differing misclassification costs and benefits from correct classifications.Verbraken et al. (2013) proposed EMPC to identify the most profitable churn model and reported discrepancies between EMPC and AUC assessments.Source: §1 · pp. 2–3
  • ProfTree optimizes EMPC during training by using an evolutionary algorithm to select decision-tree split rules.The authors choose trees for interpretability and fast computation; they contrast evolutionary global optimization with the local optima of CART and C4.5.Source: §1 · p. 3

2. Preliminaries

The preliminaries frame churn prediction as binary classification and introduce tree optimization, profit-based evaluation, evolutionary search, and related churn-modeling approaches.

2.1. Customer Churn Prediction Formulated as a Classification Problem

  • Churn prediction assigns customers to churn or no-churn classes from observed features; decision trees are interpretable and can handle nonlinearities and categorical features without complex transformations.Examples include contract length, average revenue, complaints, and customer region.Source: §2.1 · pp. 3–4
  • A tree partitions feature space into regions and estimates churn scores from the proportion of churners in each terminal node.Trees are represented by split variables and rules; the resulting scores can be thresholded to assign predicted classes.Source: §2.1 · pp. 4–5

2.2. General Fitness Function of Decision Trees

  • Decision-tree fitness balances predictive loss against complexity, commonly penalizing trees with more terminal nodes.CART and C4.5 search one split at a time, which can yield local optima; evolutionary algorithms offer a global-search alternative.Source: §2.2 · pp. 5–6

2.3. Profit-based Classification Performance Evaluation

  • Accuracy-based measures omit classification costs and benefits, an unrealistic assumption for imbalanced churn problems where false negatives lose customers and false positives incur campaign costs.Average profit instead offsets costs against benefits across classifications and depends on the classification threshold.Source: §2.3 · pp. 7–8
  • The EMPC framework evaluates expected maximum profit across uncertainty in offer acceptance probability γ, while η estimates the customer-base fraction to target for maximum profit.The framework also defines the deterministic maximum profit measure (MPC); its EMPC approach models γ with a beta distribution.Source: §2.3 · pp. 9–10

2.4. Evolutionary algorithms

  • Evolutionary algorithms evolve a population of candidate solutions using fitness-guided selection, crossover, and mutation to search complex optimization problems.Their candidate solutions may include classification and regression trees, and they can return globally optimal solutions when properly defined and not prematurely terminated.Source: §2.4 · pp. 10–11

2.5. Related work

  • ProfLogit directly maximizes EMPC with a genetic algorithm, whereas Bahnsen et al.’s example-dependent churn framework models individual costs without retained-customer benefits (Stripling et al., 2015, 2017; Bahnsen et al., 2015b).Bahnsen et al. (2015a) apply their cost-based framework to decision trees; other churn research includes support-vector machines and ensemble methods.Source: §2.5 · pp. 11–12
  • Churn research spans multiple predictive techniques, and acquiring new customers in saturated markets can cost five to six times more than preventing churn.The literature includes support-vector machines and ensemble methods applied to churn prediction.Source: §2.5 · p. 12

3. Profit Maximizing Classification Trees

ProfTree uses an evolutionary algorithm to optimize churn-classification trees for EMPC, with a penalty for tree complexity. The section describes its search, regularization tuning, and profit-based customer-list measures.

3. Profit Maximizing Classification Trees

  • ProfTree uses an evolutionary algorithm to optimize tree splits for EMPC, aiming to produce the most profitable classifier.Its fitness function penalizes tree complexity, measured by the number of terminal nodes and weighted by λ.Source: §3 · p. 12, §3.1 · p. 12

3.1. Fitness Function of ProfTree

  • The fitness function maximizes EMPC while penalizing tree complexity, yielding a globally EMPC-optimal tree with minimized complexity.Complexity is measured by the number of terminal nodes, weighted by λ.Source: §3.1 · p. 12

3.2. The ProfTree algorithm

  • The evolutionary search initializes random trees, modifies them with variation operators, and selects the fittest as survivors until termination.The returned tree has the highest fitness according to the stated objective.Source: §3.2 · p. 13

3.3. Variation operators

  • Each iteration modifies trees using four mutation operators and one crossover operator.The mutation operators include splitting, pruning, and major or minor split-rule changes.Source: §3.3 · pp. 13–14

3.4. Termination criteria

  • The search runs for at least 1,000 iterations and stops after 100 iterations without improvement in the best 5% of trees, or at the user-set limit.The default user-set limit is 10,000 iterations.Source: §3.4 · p. 14

3.5. Control parameters

  • Users can constrain node sizes and maximum tree depth; λ controls terminal-node count, and population size defaults to 100.Each of the five variation operators has a 20% selection probability by default.Source: §3.5 · p. 14

3.6. Tuning the Regularization Parameter

  • Five replications of twofold cross-validation estimate each λ value’s average EMPC, and λopt is the grid value with the highest average.The grid should be dense; preliminary studies found λopt typically in [0, 1].Source: §3.6 · pp. 14–15

3.7. Performance Measures based on the Expected Profit Maximizing Fraction for Customer Churn

  • The η-based precision, recall, and F1 assess the customer list selected using ηempc, while EMPC remains the criterion for selecting the final model.ηempc identifies the top fraction of predicted would-be churners to target for maximum profit.Source: §3.7 · pp. 15–16

4. Case study

The Korean telecom case study compares interpretable, depth-limited tree classifiers on churn prediction, tuning EvTree and ProfTree for EMPC. ProfTree achieves the strongest reported profitability among the tree classifiers, while its search can produce different tree structures with similar fitness.

4. Case study

  • The case study uses 889 customers, 277 recorded churners, and 10 explanatory variables, with trees limited to three split levels.The compared methods are CART, ctree, EvTree, and ProfTree; trees also share minimum terminal- and internal-node sizes of 10 and 20 observations.Source: §4 · p. 16
  • C4.5 results are omitted because its implementation cannot restrict tree depth.The shared maximum-depth constraint was used for the case-study comparison.Source: §4 · pp. 16–17
  • CART and ctree use only active_months and contract_period, while ProfTree adds revenue_avg, region, and overdue_amt to its tree.For customers with contracts of at least 18 months, ProfTree uses average revenue and overdue amount to assess churn likelihood.Source: §4 · pp. 17, 21
  • Grid search with 5 × 2-fold cross-validation selects αopt = 0.14 for EvTree and λopt = 0.16 for ProfTree using EMPC.Both trees are then grown on the full data set at their selected parameter values.Source: §4 · p. 17, Figure 2 · p. 19
  • 0.894 EUR per customer is ProfTree’s profit gain over EvTree; ProfTree also has the highest EMPC and MPC among the tree classifiers.It also has the highest AUC, η-precision (52%), and η-recall (94.9%).Source: Table 3 · p. 21, §4 · p. 21
  • The case study presents ProfTree as balancing profitability and complexity by searching a larger space of potential trees.Its fitness function penalizes tree size through a λ-weighted complexity term.Source: §4 · p. 21
  • With default settings, all 10 ProfTree runs stabilize by 600 iterations, yet produce different solutions with very similar fitness values.In the case-study setup, restricting maximum tree depth yielded a unique solution, though stochastic search may otherwise return different structures.Source: §4 · p. 22, Figure 4 · p. 22

5. Empirical evaluation

The benchmark evaluates ProfTree against other tree classifiers on real-life telecom churn data using repeated twofold cross-validation. ProfTree generally leads on profit measures, while its results on accuracy-related measures show a trade-off with profit optimization.

5.1. Experimental setup of the benchmark study

  • The study compared ProfTree with EvTree, CART, C4.5, and ctree, including CART and C4.5 variants with and without pruning, on real-life telecom churn data.It used five replications of twofold cross-validation; observations with missing values were removed before analysis.Source: §5 · p. 23, §5.1 · p. 23

5.2. Results of the experiment

  • ProfTree ranked first for EMPC in five of six data sets (average rank 1.17) and first for MPC across all six (average rank 1).Its EMPC profit gains ranged from 0.001 to 0.24e per customer; it was 4% better than ctree on UCI and 0.6% worse than EvTree on KorPer.Source: §5.2 · p. 24, Figure 6 · p. 26
  • Despite its profit lead, ProfTree ranked worst on MER (average rank 6.5) and low on AUC (4.33), while leading η-precision (2.67), η-recall (2.00), and η-F1 (2.00).The authors associate the discrepancy with ProfTree maximizing EMPC rather than minimizing misclassification error.Source: §5.2 · pp. 24, 26, Figure 6 · p. 26, §5.3 · p. 27

5.3. Discussion

  • The authors conclude that accuracy-based model selection can yield less profitable models, as ProfTree combines the highest overall profitability with the worst MER values.Their discussion links this pattern to ProfTree maximizing profit during construction rather than minimizing misclassification error.Source: §5.3 · pp. 26–27
  • η-precision gives retention teams lists with the highest share of true would-be churners, while η-recall captures the largest proportion of them.The discussion states that all η-based performance metrics are optimized for maximum profit.Source: §5.3 · p. 27

6. Conclusions and future work

ProfTree uses an evolutionary algorithm to optimize EMPC while constructing interpretable decision trees. Across real-life churn data sets, it was the most profitable model among the compared tree-based methods, and the authors propose combining it with random forests as future work.

6. Conclusions and future work

  • ProfTree uses an evolutionary algorithm to optimize EMPC during decision-tree construction, producing profit-driven trees that the authors describe as easy to interpret.The method uses a regularized EMPC fitness function.Source: §6 · p. 27
  • Across six real-life churn data sets, ProfTree was the most profitable model among six other tree-based methods; profits were significantly higher in almost all cases, with relatively small losses in the worst case.The evaluation used five replications of twofold cross-validation.Source: §6 · pp. 27–28
  • ProfTree had the highest hit and recall rates, and its high η-precision supports targeting customers who are both likely to churn and profitable to retain.Source: §6 · p. 28
  • Future work proposes ProfForest, which would aggregate a collection of profit-induced trees.The authors intend to combine ProfTree with random forests.Source: §6 · p. 28
Loading 1712.08101v1…