Source-linked AI summary
Adaptive Protection for Evolutionary Feature Construction in Symbolic Regression with Application to Credit Classification
Hengzhe Zhang, Qi Chen, Bing Xue, Lean Yu, Wolfgang Banzhaf, Mengjie Zhang
TL;DR
Evolutionary feature construction can lose valuable constructed features when genetic operators disrupt them, and existing protection mechanisms are not universally applicable. This paper introduces importance-guided adaptive protection and finds improved search effectiveness across regression benchmarks and credit classification datasets.
Problem
Genetic operators can disrupt valuable constructed features, while existing feature-importance-aware operators lack a simple mechanism applicable to arbitrary genetic operators.
Method
The paper restores selected constructed features after genetic operations using importance-weighted protection, preserving important features while allowing less important features to change.
Results
Protection-based methods substantially outperform no-protection baselines across 98 regression datasets and improve search effectiveness on two credit classification datasets.
Takeaways & Limitations
Adaptive protection improves evolutionary feature-construction search across regression and credit classification settings, with robustness to different importance metrics and base learners.
Takeaways & Limitations
The protection mechanism operates at the feature level rather than dynamically identifying and protecting useful substructures.
Abstract
from arXiv · showhide
Evolutionary feature construction has shown strong promise in symbolic regression by automatically discovering informative transformations of input features that enhance a simple base learner. However, existing approaches often lack explicit mechanisms to preserve important constructed features discovered during evolution, and valuable genetic material can be lost when genetic operators disrupt effective features. This paper introduces an adaptive protection mechanism that leverages feature importance metrics to selectively preserve constructed features during evolution. The mechanism provides stronger protection for more important constructed features while still allowing less important features to be modified and to incorporate useful building blocks from more important features. We evaluate the approach using multiple feature importance calculation methods and demonstrate its robustness across different base learners. Experimental results on 98 regression benchmark datasets show that the proposed mechanism consistently improves solution quality over baseline approaches, and experiments on two credit classification datasets demonstrate that the method also extends effectively to improve search effectiveness beyond symbolic regression.
1 Introduction
The paper introduces an importance-guided adaptive protection mechanism for evolutionary feature construction, preserving valuable constructed features while allowing less important ones to change. It evaluates multiple importance metrics across model types, symbolic-regression benchmarks, and credit-classification datasets.
- Evolutionary feature construction uses genetic programming to evolve informative feature transformations for predictive models.
- Genetic mutation and crossover can disrupt or remove valuable constructed features while generating new combinations.
- Feature importance metrics, including coefficients, marginal contribution, and Shapley values, can identify constructed features that contribute most to model performance.
- The proposed adaptive protection mechanism uses importance-weighted restoration, protecting more important constructed features more strongly while allowing less important features to be modified or replaced.The mechanism is embedded in a universal protection framework compatible with any genetic operator.
- The study compares multiple feature-importance methods across model types and evaluates the approach on symbolic-regression benchmarks and credit-classification datasets.The evaluation examines effectiveness across different importance metrics and base learners.
2 Related Work
Feature construction methods in evolutionary computation are categorized as wrapper, filter, or embedded, with this work focusing on wrapper-based approaches. Prior building-block research motivates protecting useful components through an operator-agnostic mechanism compatible with diverse evolutionary operators.
- Feature construction paradigms: Feature construction comprises wrapper, filter, and embedded paradigms, with wrapper methods evaluating constructed features through learner performance.Wrapper examples include M3GP, ITEA, MRGP, and Genetic Programming Gene-pool Optimal Mixing.
- Feature construction paradigms: Wrapper-based approaches have demonstrated strong performance in symbolic regression and classification, motivating their focus in this work.
- Building blocks: Building-block research spans structural and semantic program components, learned building blocks, and reuse across circuit synthesis, Boolean problems, and image analysis.
- Building blocks: Prior analyses identify useful components, but crossover and mutation often require redesign to exploit them effectively.
- Building blocks: An operator-agnostic protection mechanism is desirable because it can preserve useful parts while remaining compatible with traditional, semantic, and LLM-based operators.
3 Algorithm
The algorithm evolves populations of genetic-programming feature trees, evaluates them with task-specific learners and fitness measures, and adaptively protects important trees during genetic operations. It combines importance-based protection with elitism and an archive of top individuals to preserve useful genetic material and stabilize predictions.
- Evolutionary feature construction: The evolutionary process uses generational populations of individuals, each containing multiple feature-transformation trees built from mathematical operators, input variables, and constants.The workflow includes population initialization, solution evaluation, importance calculation, parent selection, offspring generation, and elitism.
- Solution evaluation: Fitness is measured by R2 for regression and AUC for classification, while epsilon lexicase selection uses per-example squared error or cross-entropy losses.Regression uses ridge regression or random decision trees; credit classification uses regularized logistic regression.
- Adaptive protection: Importance-weighted protection saves original trees before crossover and mutation, then restores them probabilistically so higher-importance trees are preserved more often while lower-importance trees remain modifiable.The protection coefficient mprotect controls protection strength, with zero disabling protection.
- Elitism and ensemble archive: The algorithm carries the best individual forward and maintains an archive of the top 100 evaluated individuals for final prediction, averaging predictions or class probabilities when applicable.Archived random-decision-tree predictions are averaged to reduce variance, while logistic-regression classification averages predicted class probabilities.
- Importance calculation: Feature importance is calculated using ridge coefficients, SHAP values, marginal contribution, tree impurity decrease, or absolute logistic-regression coefficients, depending on the base learner.The three ridge-regression metrics are systematically compared within the protection mechanism.
4 Experimental Setup
The experiments evaluate adaptive protection on 98 small regression benchmarks and two credit-classification datasets, comparing protection levels and feature-importance methods. Performance is assessed across repeated train-test runs using task-appropriate metrics and statistical significance testing.
- Datasets: 98 PMLB regression datasets with up to 2000 samples form the main evaluation, supplemented by Australian Credit and German Credit classification datasets.The sample limit reflects computational cost constraints.
- Experimental design: The experimental design compares different protection levels and feature-importance calculation methods.The protection coefficient’s impact is analyzed separately.
- Implementation: Categorical features use target encoding fitted on training data, and the function set includes arithmetic operations, mathematical functions, and the analytical quotient.For credit classification, evolution runs for 30 generations to avoid overfitting while other settings remain unchanged.
- Evaluation: Regression uses test-set R² and credit classification uses AUC, with 30 independent runs per experiment and an 80:20 train-test split.Results are averaged across runs, with statistical significance tested by the Wilcoxon signed-rank test at p < 0.05.
5 Results and Analysis
Across 98 regression datasets, adaptive protection improved test performance over no protection and reduced-operation baselines, while remaining robust across importance methods and base learners. It also transferred beyond symbolic regression, improving training search effectiveness on two credit datasets.
- Adaptive protection versus reduced operations: All protection-based methods consistently outperform low operation rate configurations, showing selective protection is more effective than globally reducing genetic operations.The reduced-operation baselines apply genetic operators to only 1 or 5 trees per offspring instead of all 10 trees.
- Importance methods: Most feature-importance methods perform similarly, with the vast majority of Built-In-versus-SHAP comparisons resulting in ties.Built-In is most efficient, while marginal contribution requires one additional refit per feature for each evaluated individual.
- Robustness across base learners: Adaptive protection maintains effectiveness with random decision-tree base learners using impurity-decrease importance, outperforming both no protection and reduced operation rates.Comparisons are made within the same base-learner setting to isolate the protection mechanism’s effect.
- Transfer to credit classification: On training data, the proposed method achieves higher AUC than both no protection and reduced genetic operations on both the Australian and German credit datasets.On test data, it matches no protection on both datasets.
6 Conclusions
The paper introduces adaptive, importance-weighted protection for preserving valuable constructed features during evolutionary feature construction. Evaluations on regression and credit-classification datasets show improved search effectiveness, while future work should protect useful substructures beyond the feature level.
- Contribution: The mechanism restores selected constructed features after genetic operations, protecting important features while allowing less important ones to be modified or replaced.Protection is guided by an importance-weighted protection score.
- Experimental results: 98 regression benchmark datasets show protection-based methods substantially outperform no-protection baselines, with adaptive protection more effective than simply reducing genetic operations.The mechanism also improves search effectiveness on two credit classification datasets and remains robust to different feature-importance calculation methods.
- Limitations and future work: A limitation is that protection operates at the feature level, motivating dynamic identification and protection of useful substructures through building-block analysis.The proposed direction applies throughout the evolutionary process.
A Feature Construction in Credit Classification
Credit classification is a suitable application for evolutionary feature construction because credit risk involves nonlinear attribute interactions and complex predictors that are labor-intensive to design manually. Prior work also indicates that engineered transaction, behavioral, and virtually generated features can improve fraud detection, fraud ranking, and robustness.
- Motivation: Credit classification suits evolutionary feature construction because credit risk often depends on nonlinear interactions among attributes.The passage identifies nonlinear attribute interactions as a central motivation for applying evolutionary feature construction to credit classification.
- Motivation: Complex combinations of raw features may produce useful predictors that are labor-intensive to design manually.Evolutionary construction can target combinations that would otherwise require substantial manual feature-engineering effort.
- Prior Work: Prior work reports that engineered transaction and behavioral features improve cost-sensitive fraud detection and cash-out fraud ranking.These examples connect feature engineering with improved performance in two fraud-related credit applications.
- Prior Work: Feature engineering with virtual sample generation has also been reported to improve robustThe supplied passage ends mid-sentence after “improve robust,” so the specific robustness outcome is incomplete.
B Further Analysis of Power Transformation and Protection Level · B.1 Power Transformation
The analysis evaluates power transformations of feature-importance values for adaptive protection, finding that linear scaling (p = 1) performs best overall. Higher powers overemphasize importance differences and can degrade protection and performance.
- B.1 Power Transformation: Power transformations |β_j|^p were evaluated for p ∈ {0.5, 1, 2, 5}.The investigation tests whether nonlinear importance transformations improve adaptive protection.
- B.1 Power Transformation: Power 1, corresponding to linear scaling, provides the best overall performance.This result is reported across the comparison in Figure 6 and Table 8.
- B.1 Power Transformation: Higher powers overemphasize differences in feature-importance values.The resulting rankings become extreme and underweight features of medium importance.
- B.1 Power Transformation: Extreme rankings from higher powers lead to inadequate protection for features of medium importance.The mechanism therefore protects the importance distribution less effectively under these transformations.
- B.1 Power Transformation: Higher powers are associated with performance degradation relative to the best-performing linear scaling.The passage attributes the degradation to overemphasized importance differences.
- B.1 Power Transformation: Figure 6 presents a comparison of feature-importance power values.It is used to assess the tested transformation settings.
- B.1 Power Transformation: Table 8 provides a statistical comparison of feature-importance power values.It complements the performance comparison of the tested powers.
B.2 Effective Protection Level
The study evaluates protection coefficients of 0, 1, 3, and 5 to balance preserving valuable trees with allowing further modifications. Protection consistently outperforms no protection, and with 10 trees per individual, larger coefficients perform better.
- B.2 Effective Protection Level: The comparison tests protection coefficients of 0, 1, 3, and 5, with 0 representing no protection.The evaluation examines how protection strength affects the trade-off between preserving valuable trees and permitting further modifications.
- B.2 Effective Protection Level: Protection consistently outperforms no protection across the statistical comparisons.The results are summarized in Figure 7 and Table 9.
- B.2 Effective Protection Level: With 10 trees per individual, larger protection coefficients perform better.This result indicates that stronger protection is more effective under the stated population configuration.
C Deterministic vs. Probabilistic Adaptive Protection
The section compares deterministic and probabilistic adaptive protection variants that preserve exactly five features per individual during genetic operations. The variants differ in whether they always retain the highest-importance features or sample features according to importance.
- C Deterministic vs. Probabilistic Adaptive Protection: Both adaptive protection variants implement partial elitism by keeping exactly K = 5 features per individual unchanged through genetic operations.The variants protect five features during genetic operations.
- C Deterministic vs. Probabilistic Adaptive Protection: The deterministic variant always preserves the 5 highest-importance features.Its selection rule is fixed rather than sampled.
- C Deterministic vs. Probabilistic Adaptive Protection: The probabilistic variant samples 5 features without replacement, with each feature drawn with probability proportional to its importance.Features with zero importance are never selected for protection.