Source-linked AI summary
Interpretable Predictions of Tree-based Ensembles via Actionable Feature Tweaking
Gabriele Tolomei, Fabrizio Silvestri, Andrew Haines, Mounia Lalmas
TL;DR
The paper tackles the difficulty of understanding and changing predictions from machine-learned models when some input features are actionable. It exploits tree-based ensemble internals to generate feature-tweaking recommendations, demonstrating them on low- and high-quality advertisements; 57.3% of recommendations were judged helpful and likely to improve ad quality if implemented.
Problem
The paper addresses the need to understand model predictions and identify adjustable feature changes that can alter them.
Method
The authors propose an algorithm that uses the internals of tree-based ensemble binary classifiers to recommend feature tweaks for changing predictions.
Results
57.3% of recommendations were judged helpful and likely to improve ad quality if implemented, using advertisements from Yahoo Gemini.
Takeaways & Limitations
The approach provides advertisers with actionable suggestions for changing low-quality advertisements toward positive model predictions.
Takeaways & Limitations
The method may be unsuitable for high-dimensional datasets such as text, images, or videos because its worst-case complexity is O(2^n).
Abstract
from arXiv · showhide
Machine-learned models are often described as "black boxes". In many real-world applications however, models may have to sacrifice predictive power in favour of human-interpretability. When this is the case, feature engineering becomes a crucial task, which requires significant and time-consuming human effort. Whilst some features are inherently static, representing properties that cannot be influenced (e.g., the age of an individual), others capture characteristics that could be adjusted (e.g., the daily amount of carbohydrates taken). Nonetheless, once a model is learned from the data, each prediction it makes on new instances is irreversible - assuming every instance to be a static point located in the chosen feature space. There are many circumstances however where it is important to understand (i) why a model outputs a certain prediction on a given instance, (ii) which adjustable features of that instance should be modified, and finally (iii) how to alter such a prediction when the mutated instance is input back to the model. In this paper, we present a technique that exploits the internals of a tree-based ensemble classifier to offer recommendations for transforming true negative instances into positively predicted ones. We demonstrate the validity of our approach using an online advertising application. First, we design a Random Forest classifier that effectively separates between two types of ads: low (negative) and high (positive) quality ads (instances). Then, we introduce an algorithm that provides recommendations that aim to transform a low quality ad (negative instance) into a high quality one (positive instance). Finally, we evaluate our approach on a subset of the active inventory of a large ad network, Yahoo Gemini.
1 INTRODUCTION
The paper addresses the difficulty of interpreting and changing predictions from machine-learned models by proposing actionable feature tweaks for tree-based ensembles. It demonstrates the approach by recommending changes that aim to convert low-quality advertisements into high-quality ones.
- Machine-learned models can be difficult to interpret, particularly when they fail or produce unexpected predictions.
- Feature engineering represents objects as fixed points in a feature space, making learned predictions appear irreversible.
- The research question is which feature-vector changes can modify a model’s prediction.
- The proposed algorithm tweaks input features on top of a tree-based ensemble binary classifier to transform true negative instances into positively predicted ones.
- The paper evaluates the framework in online advertising after training a Random Forest to distinguish low- and high-quality advertisements.
- Recommendations are assessed using advertisements served by Yahoo Gemini, with the intended aim of helping advertisers improve campaign return on investment.
2 PROBLEM STATEMENT
The paper formulates actionable prediction changes for tree-based ensembles: transform a true negative feature vector into a positively predicted one while minimizing transformation effort. It defines the ensemble, path-based constraints, and computational boundaries of this task.
- 2.1 Notation: The model is an ensemble of K tree-based classifiers whose individual outputs are combined into one prediction, potentially by majority voting.Each base classifier is represented by a decision tree, and the forest contains K such trees.
- 2.2 Enforcing Positive Prediction: The task transforms a true negative instance x into x′ such that the ensemble predicts +1, selecting the lowest-cost transformation under δ.The cost function can measure the number of affected features or Euclidean distance between the original and transformed vectors.
- 2.3 Positive and Negative Paths: Each tree path is a sequence of binary feature-threshold tests leading to a positive or negative leaf, so satisfying a positive path supplies a candidate transformed instance.The method focuses on positive paths in trees that currently vote negatively and constructs feature values satisfying their conditions.
- 2.4 Tweaking Input Features: The method restricts candidate feature changes using a tolerance ϵ, with each changed value positioned just across the relevant path threshold.A common global tolerance assumes features have been standardized; with z-score standardization, changes are interpreted relative to feature standard deviations.
- 2.4 Tweaking Input Features: The tweaking problem is NP-hard, although the single-tree case can be solved optimally.The general problem remains NP-hard even with the additional tolerance constraint, while practical tree-depth bounds limit the examined path space.
- 2.5 The Feature Tweaking Algorithm: With tree depth bounded by the n input features, the worst-case complexity is O(2^n), making the method potentially unsuitable for high-dimensional data.The authors report practical feasibility for average input sizes below 100 features because relatively few positive paths are typically examined.
3 USE CASE: IMPROVING AD QUALITY
The use case applies actionable feature tweaking to online advertising, using dwell time to classify ad quality and Random Forests to generate recommendations for improving low-quality ads.
- 3.1 Why Ad Qality?: The study targets improving Yahoo Gemini mobile native advertisements by shifting low-quality ads toward high-quality regions of an ad-quality feature space.The algorithm creates proposed high-quality ads and associates each transformation with a cost, prioritizing lower-cost actionable suggestions.
- 3.3.3 Labelled Dataset of Ads.: Ad quality is defined through landing-page dwell time, with high-quality ads exceeding threshold τ based on users’ average time on the page.The threshold is the average time users spend on at least 50% of other ad landing pages.
- 3.3.3 Labelled Dataset of Ads.: The binary classifier predicts ad quality from features describing the ad creative, landing page, or both.Feature categories include landing-page, creative, and combined sources; historical features improve prediction but are excluded from tweaking because they cannot be altered.
- 3.3.4 Offline Evaluation.: Decision Trees, Gradient Boosted Decision Trees, and Random Forests are trained because the tweaking algorithm operates on tree-based ensemble classifiers.The dataset is split into 80% training and 20% held-out test data, with 10-fold cross-validation used for model selection.
- 3.3.4 Offline Evaluation.: Random Forest is the best-performing model, using an ensemble of 1,000 base trees with maximum depth 16 and generalizing best on unseen examples.Evaluation on Dtest uses F1 and Matthews Correlation Coefficient, while model selection uses ROC AUC.
4 EXPERIMENTS: AD FEATURE RECOMMENDATIONS
The experiments examine how tolerance and cost choices affect actionable ad transformations, then assess their availability, feature patterns, and human-rated helpfulness. Recommendations are evaluated on Yahoo Gemini advertisements, with model and data limitations noted.
- Recommendation generation: Each valid ϵ-transformation yields feature-level directives whose vector differences specify the magnitude and direction of recommended ad changes.The transformation x′ is compared with the original x component-wise, and nonzero differences identify features to modify.
- Hyperparameter effects: Testing ϵ values from 0.01 to 1 found the highest ad coverage at ϵ = 0.5.The tested values represent multiples of a feature-specific standard-deviation unit.
- Hyperparameter effects: Larger ϵ values make transformation counts more evenly distributed across ads, while coverage later decreases between ϵ = 0.5 and 1.The distributions remain skewed, with many transformations concentrated on a few ads.
- Hyperparameter effects: The study compares five tweaking costs: tweaked feature rate, Euclidean, cosine, Jaccard, and Pearson correlation distance.These costs quantify effort using feature changes or distances between original and transformed vectors.
- Recommendation evaluation: Using ϵ = 0.05 and cosine distance, around 91.0% of landing pages had three transformations, while 7.5% had two and 1.5% had one.This setting was selected as the best trade-off between ad coverage and average cost.
- Recommendation evaluation: 57.3% of recommendations were rated helpful, with 60.4% inter-agreement and 0.4% classified as non-actionable.Among the 42.3% non-helpful recommendations, about 25% were considered neutral.
5 RELATED WORK
The paper situates its approach within actionable knowledge discovery and post-processing research, emphasizing an exact ensemble-transformation algorithm and practical recommendation evaluation.
- Actionable knowledge research has largely focused on accuracy and related model properties, while this challenge remains comparatively unexplored.
- Prior post-processing work includes pruning, summarizing, and matching learned rules, alongside domain-driven data-mining frameworks for actionable knowledge.
- Existing decision-tree studies propose actions for maximizing expected group profit or otherwise supporting actionability, but differ from this paper’s instance-level ensemble focus.
- Like Cui et al., this work transforms predictions for individual instances using an ensemble of trees and introduces an algorithm finding an exact solution.
- The paper addresses NP-hardness through model feedback, constrains feature changes with ϵ, evaluates five transformation costs, ranks recommendations by feature importance, and studies practical effects.
- LIME instead explains selected predictions by fitting an interpretable local model around them and evaluates explanations across text and image classifiers.
6 CONCLUSIONS
The paper develops feature-tweaking recommendations for tree-based ensemble classifiers despite the problem’s NP-hardness. In Yahoo Gemini online advertising, creative strategists judged 57.3% of recommendations helpful, while future work targets broader settings.
- The paper addresses the need to understand and alter predictions by shifting true negative instances in feature space toward positive classifications.
- The approach remains computationally feasible by bounding the number of feature changes and using model feedback to avoid exploring the entire exponential search space.
- 57.3% of recommendations were rated helpful by creative strategists in the Yahoo Gemini advertising evaluation.
- Future work will extend the approach to multi-class settings, other learning models, and reinforcement learning.