Source-linked AI summary
LLM-FE: Automated Feature Engineering for Tabular Data with LLMs as Evolutionary Optimizers
Nikhil Abhyankar, Parshin Shojaee, Chandan K. Reddy
TL;DR
Automated feature engineering is costly, and traditional search and existing LLM approaches have limitations in using domain knowledge and prior discovery experience to link generation with measured performance. LLM-FE iteratively generates and evaluates feature programs through evolutionary search, and it consistently improves results across classification and regression benchmarks compared with evaluated baselines.
Problem
Traditional feature-engineering searches rely on predefined transformations, while existing LLM approaches do not leverage prior discovery experiments to connect generation meaningfully with data-driven performance.
Source: §1 · p. 1Method
LLM-FE uses LLMs as evolutionary optimizers, iteratively generating feature programs from domain knowledge and refining them using performance feedback and successful prior programs.
Source: §1 · pp. 2–3Results
Across classification datasets LLM-FE improves over raw-data models and achieves the lowest mean rank, and on regression datasets it outperforms all evaluated baselines with the lowest mean rank.
Source: §4.4 · p. 7Takeaways & Limitations
LLM-FE can improve predictive performance while reducing manual effort in resource-intensive feature-engineering settings.
Source: Impact Statement · p. 11LLM-FE may include serialized data samples in prompts, making privacy a consideration in sensitive domains.
Source: Impact Statement · p. 11
Abstract
from arXiv · showhide
Automated feature engineering plays a critical role in improving predictive model performance for tabular learning tasks. Traditional automated feature engineering methods are limited by their reliance on pre-defined transformations within fixed, manually designed search spaces, often neglecting domain knowledge. Recent advances using Large Language Models (LLMs) have enabled the integration of domain knowledge into the feature engineering process. However, existing LLM-based approaches use direct prompting or rely solely on validation scores for feature selection, failing to leverage insights from prior feature discovery experiments or establish meaningful reasoning between feature generation and data-driven performance. To address these challenges, we propose LLM-FE, a novel framework that combines evolutionary search with the domain knowledge and reasoning capabilities of LLMs to automatically discover effective features for tabular learning tasks. LLM-FE formulates feature engineering as a program search problem, where LLMs propose new feature transformation programs iteratively, and data-driven feedback guides the search process. Our results demonstrate that LLM-FE consistently outperforms state-of-the-art baselines, significantly enhancing the performance of tabular prediction models across diverse classification and regression benchmarks. The code is available at: https://github.com/nikhilsab/LLMFE
1 Introduction
Automated feature engineering is costly, and traditional search spaces often omit domain knowledge; existing LLM methods also do not use prior discovery experience to connect generation with measured performance. LLM-FE uses an LLM-guided evolutionary process with data-driven feedback and reports consistent improvements over state-of-the-art feature-engineering methods.
1 Introduction
- Traditional automated methods search manually predefined transformations, often without domain knowledge, despite feature engineering's importance and resource cost.Domain knowledge can reduce search complexity and support more interpretable, effective features.Source: §1 · p. 1
- Prior LLM-based methods rely mainly on direct prompting or validation scores, without using prior discovery experiments to connect generation with data-driven performance.This limits the link between generated features and measured results.Source: §1 · p. 1
- LLM-FE consistently outperforms state-of-the-art feature-engineering methods across XGBoost, TabPFN, and MLP predictors.The framework identifies contextually relevant features that improve downstream performance.Source: §1 · p. 2
- LLM-FE combines LLM domain knowledge, iterative evolutionary search, data-driven evaluation, and memory of successful programs to refine feature transformations.Its experimental contributions also report generalizability across predictors and LLM backbones, with ablations highlighting the roles of domain knowledge, feedback, search, and data samples.Source: §1 · pp. 2–3
2 Related Works
Prior automated feature engineering spans several search and learning strategies, while LLMs offer domain knowledge but may generate inconsistent outputs. LLM-FE combines that prior knowledge with evaluator-guided evolutionary refinement and maintains a diverse pool of promising programs.
Feature Engineering
- Traditional feature-engineering methods include tree-based exploration, transformation enumeration, and learning-based approaches, but often do not leverage domain knowledge.The cited approaches include Kanter & Veeramachaneni (2015), Khurana et al. (2016), Nargesian et al. (2017), and Zhang et al. (2023).Source: §2 · p. 3
LLMs and Optimization
- LLM optimization work couples generation with feedback, mutation, or crossover across tasks including prompt optimization, architecture search, heuristic discovery, and symbolic regression.LLM outputs can also be inconsistent or factually incorrect (Madaan et al., 2024; Zhu et al., 2023).Source: §2 · p. 3
LLMs for Tabular Learning
- Unlike tabular LLM feature-engineering approaches that often refine one candidate, LLM-FE searches a diverse pool of promising programs using mutation and crossover.Related approaches include FeatLLM, CAAFE, and OCTree; LLM-FE aims for predictive and human-interpretable transformations.Source: §2 · p. 3
3 LLM-FE Approach
LLM-FE casts feature engineering as search over transformation programs, with LLM proposals evaluated on validation data and refined using successful prior programs. Its prompting, sampling, evaluation, and multi-population memory components structure this iterative process.
3.1 Problem Formulation
- Feature engineering seeks a transformation program that augments the input space to improve downstream predictive performance; LLM-FE approximates the bilevel objective with iterative program evaluation and fitness feedback.The LLM samples candidate transformations conditioned on metadata, data samples, and high-scoring programs from previous iterations.Source: §3.1 · pp. 3–4
3.2 Feature Generation
- An LLM generates multiple feature-transformation programs using prior knowledge, reasoning, and in-context learning to explore the feature space.The search begins from an initial transformation program.Source: §3.2 · p. 4
3.2.1 Input Prompt
- Prompts combine task instructions, dataset metadata and representative samples, an initial program, an evaluation function, and successful prior demonstrations.Instructions ask for novel, complex features with reasoning; dataset details include feature names and descriptions.Source: §3.2.1 · p. 4
3.2.2 Feature Sampling
- At each iteration, temperature-based sampling produces candidate programs; invalid or over-time transformations are discarded before evaluation.Sampling balances exploration and exploitation, and execution limits support computational efficiency.Source: §3.2.2 · p. 5
3.3 Data-Driven Evaluation
- Generated features augment the dataset, and a prediction model is trained on transformed training data and assessed on transformed validation data to score feature quality.The objective is to maximize accuracy for classification or optimize error metrics for regression.Source: §3.3 · p. 5
3.4 Experience Management
- Multi-population memory stores scored programs in separate islands, and sampled in-context examples guide further search while balancing exploration and exploitation.Clusters are sampled with higher probability for higher scores; lower temperature concentrates selection on top-scoring clusters, while higher temperature spreads it more evenly.Source: §3.4 · pp. 5–6
4 Experimental Setup
LLM-FE was evaluated across classification and regression datasets, using multiple predictive-model architectures and established feature-engineering baselines. It achieved the lowest mean rank in both task settings, while ablations highlighted the roles of domain knowledge and evolutionary refinement.
4 Experimental Setup
- The evaluation used XGBoost, MLP, and TabPFN, and results indicated feature improvements across models and datasets.These predictors represent tree-based, neural, and transformer-based architectures, respectively.Source: §4 · p. 6
4.1 Baselines
- Baselines included OpenFE, AutoFeat, CAAFE, FeatLLM, and OCTree, with XGBoost and GPT-3.5-Turbo as defaults for comparisons.LLM-based methods had a fixed budget of 20 samples, and the reported result was the best-scoring program found within that budget.Source: §4.1 · p. 6
4.2 Datasets
- Datasets included 19 classification, 10 regression, and 8 large-scale high-dimensional classification tasks, plus five post-September 2021 classification datasets.The datasets included mixed categorical and numerical features and were evaluated over five random-seed train-test splits.Source: §4.2 · p. 6
4.3 LLM-FE Configuration
- LLM-FE used GPT-3.5-Turbo and Llama-3.1-8B-Instruct, generating three Python transformation programs per prompt with three islands.Each experiment used 20 LLM samples, and the top programs were sampled by validation score.Source: §4.3 · pp. 6–7
4.4 Results and Discussion
- LLM-FE achieved the lowest mean rank across 19 classification datasets and outperformed the raw-data model, with Table 2 presenting XGBoost accuracy comparisons.The results passage also reports lower computational cost than leading baselines.Source: §4.4 · p. 7, Table 2 · p. 7
- On 10 regression datasets, LLM-FE outperformed all compared baselines and achieved the lowest mean rank; Table 3 reports XGBoost RMSE comparisons.CAAFE and FeatLLM were excluded because available codebases lacked regression implementations.Source: §4.4 · p. 7, Table 3 · p. 7
4.5 Ablation Study
- Without evolutionary refinement, accuracy was 0.587; without domain knowledge, it was 0.626 on classification datasets with fewer than 10,000 samples.The ablation used XGBoost and GPT-3.5-Turbo; removing data examples caused only a slight performance drop.Source: §4.5 · p. 8
5 Analysis
LLM-FE offers a strong efficiency–performance trade-off while addressing simple-operator bias through evolutionary refinement. Analyses also report benefits from domain knowledge, gains on datasets designed to probe memorization, and generalization across predictor models and LLM backbones.
5.1 Efficiency Analysis
- LLM-FE lies on the Pareto frontier, with higher performance than OCTree at nearly the same runtime and substantially less runtime than CAAFE.The authors report the best efficiency–performance trade-off among evaluated methods on larger datasets.Source: §5.1 · p. 8, Figure 3 · p. 8
5.2 Bias Mitigation
- Evolutionary refinement more often retains complex operators, including groupbythenmean, groupbythenmin, residual, and sigmoid, than direct LLM generation.The authors attribute this to promoting diversity, empirical evaluation, and iterative refinement, which counteracts LLMs’ bias toward simple operators.Source: §5.2 · p. 9, Figure 4 · p. 9
5.3 Impact of Domain Knowledge and Evolutionary Refinement
- Domain knowledge yields the highest accuracy against both the base model and the variant without domain knowledge, while refinement improves accuracy across 20 iterations.Without refinement, validation accuracy improves early and then plateaus; with refinement, it continues improving. The qualitative example contrasts interpretable, clinically meaningful features with arbitrary transformations.Source: §5.3 · p. 9, Figure 5 · p. 10, Figure 6 · p. 10
5.4 Memorization in Feature Engineering
- LLM-FE delivers modest but consistent gains across datasets designed to probe memorization and datasets with pretraining exposure considered highly unlikely.The authors describe iterative selection, mutation, and evaluation by downstream performance as a filter for memorization-driven artifacts.Source: §5.4 · p. 10, Table 4 · p. 10
5.5 Generalizability Analysis
- LLM-FE improves downstream performance across XGBoost, MLP, and TabPFN with both GPT-3.5-Turbo and Llama-3.1-8B-Instruct.The reported generalizability analysis spans classification and regression tasks; TabPFN evaluations use 10,000 samples due to processing limits.Source: §5.5 · p. 11, Table 5 · p. 11
6 Conclusion
LLM-FE uses LLMs as evolutionary optimizers to discover features for tabular prediction. Experiments report consistent outperformance of state-of-the-art baselines, and the framework may extend to other tabular-pipeline stages.
6 Conclusion
- LLM-FE combines LLM-driven hypothesis generation, data-driven feedback, and evolutionary search to automate feature engineering for tabular prediction.The framework treats LLMs as evolutionary optimizers for discovering new features.Source: §6 · p. 11
- Across diverse tabular learning tasks and prediction models, LLM-FE consistently outperforms state-of-the-art baselines in predictive performance.Source: §6 · p. 11
- The framework could extend beyond feature engineering to data augmentation, data cleaning, and model tuning.Source: §6 · p. 11
Impact Statement
LLM-FE is framed as a way to improve predictive performance while reducing manual effort, but it relies on user-provided task and domain context. Compared with CAAFE and OCTree, it uses parallel multi-population evolutionary search rather than single-path refinement.
Impact Statement
- LLM-FE may include serialized samples in prompts, so privacy matters in sensitive domains; safety-critical outputs should remain expert-reviewable hypotheses.The authors note that locally deployed models, anonymized or synthetic samples, or omitting examples are possible mitigations.Source: Impact Statement · p. 11
- LLM-FE depends on task descriptions, feature metadata, and domain context, and is designed to augment rather than replace domain experts.Source: Impact Statement · p. 11
A Comparison with LLM-based Baselines
- LLM-FE explores multiple candidates across islands and keeps multi-population memory, whereas CAAFE and OCTree refine a single candidate or rule at a time.The paper reports that approximately 45% of LLM-FE features qualify as complex, versus negligible proportions for CAAFE and OCTree.Source: A Comparison with LLM-based Baselines · p. 15
B Implementation Details
LLM-FE prompts an LLM with dataset context and evaluates generated feature programs using validation performance, while an island-based memory supplies prior programs as examples. The section also describes traditional and LLM-based feature-engineering baselines.
B.1 LLM-FE
- LLM-FE combines dataset-specific prompts, validation-score evaluation, and island memory that supplies selected prior programs as in-context examples.Prompts include task and feature descriptions plus sampled data; each island retains hypotheses that exceed its current best score, with programs clustered by validation score.Source: §B.1 · pp. 15–16
- Prompts use dataset descriptions and serialized samples, with randomized instruction variants to encourage exploration of a wider range of operators.The alternate instructions direct the model toward complex combinations of operators rather than only simple operators.Source: §B.1 · p. 15, Figure 8 · p. 16
- Generated feature programs modify the data for model evaluation; programs exceeding 30 seconds or 2GB are disqualified and receive None scores.The LLM generates three outputs per prompt at temperature 0.8.Source: §B.1 · p. 15
B.2 Baselines
- Baselines include FeatLLM’s LLM-generated binary features, AutoFeat’s beam-search subsampling, sequential CAAFE and OCTree, and OpenFE’s feature boosting and pruning.OCTree is implemented only for classification datasets because its official implementation is limited to classification.Source: §B.2 · pp. 16, 18
C Dataset Details
The evaluation uses varied classification and regression datasets and examines LLM-FE alongside hyperparameter optimization. Even after HPO, LLM-FE provides further gains over the base model and OpenFE on three of five datasets.
C Dataset Details
- The evaluation covers binary classification, multiclass classification, and regression datasets from OpenML, UCI, and Kaggle, spanning varied sample sizes and feature dimensionalities.Datasets with merely numerical feature identifiers were excluded; some datasets have more than 50 features.Source: C Dataset Details · p. 18, Table 6 · p. 18
D.1 LLM-FE and Hyperparameter Optimization (HPO)
- HPO experiments use XGBoost and MLP on five classification datasets where baseline accuracy is below 0.8, with 100 Optuna trials and multiple dataset splits.MLP training runs up to 100 epochs with early stopping, retaining the checkpoint with the best validation score.Source: §D.1 · p. 19, Table 7 · p. 19, Table 8 · p. 19
- On three of five datasets, LLM-FE delivers further gains over the base model and OpenFE even after hyperparameter optimization.HPO improves the base model across all datasets.Source: §D.1 · p. 19, Table 9 · p. 19
D.2 Generalizability Analysis
LLM-FE improves performance across prediction models and, on most datasets, across additional LLM backbones and smaller predictors. Features generated with XGBoost also improve MLP and TabPFN performance when transferred across architectures.
D.2 Generalizability Analysis
- LLM-FE features generated with GPT-3.5-Turbo consistently improve XGBoost, MLP, and TabPFN performance across datasets over their unengineered base versions.Table 10 reports means and standard deviations over five splits; regression uses RMSE and classification uses accuracy.Source: §D.2 · p. 19, Table 10 · p. 20
- Across most datasets, LLM-FE outperforms base models with additional LLM backbones and with CatBoost or Logistic Regression predictors.The additional backbones tested are GPT-4o-mini, Qwen2.5-72B-Instruct, and Gemini-2.5-Flash.Source: §D.2 · p. 19, Table 11 · p. 20, Table 12 · p. 20
D.3 Transferability of Generated Features
- XGBoost-generated features improve MLP and TabPFN performance over their base versions, showing transfer across prediction architectures.The authors interpret this improvement as suggesting the features capture data characteristics useful across modeling paradigms.Source: §D.3 · p. 21
D.4 Statistical Analysis
Wilcoxon signed-rank tests find statistically significant LLM-FE improvements over base XGBoost on all regression datasets and on more than half of classification datasets, though significance is not universal.
D.4 Statistical Analysis
- LLM-FE significantly improves over base XGBoost on 10/10 regression datasets (p < 0.001) and 8/15 classification datasets (p < 0.05); two more classification datasets show marginal improvements.Tests use 25 paired observations per dataset from five seeds across five-fold cross-validation, with 95% confidence intervals.Source: §D.4 · p. 21
E.1 Interpretability Analysis
The analyses report that LLM-FE’s generated features contribute to prediction, remain effective under tested noise, and benefit from domain knowledge and evolutionary refinement. They also find that a moderate number of search islands balances exploration with refinement.
E.1 Interpretability Analysis
- 16.67% of generated features rank among the top 10 most impactful features, and over 60% appear within the top 50 by SHAP attribution.Generated programs are evaluated independently, and successful ones are stored for evolutionary refinement and composition into higher-order features.Source: §E.1 · pp. 21–22, Table 15 · p. 21
E.2 Robustness to Noise
- Across tested Gaussian-noise levels, LLM-FE maintains higher XGBoost accuracy and greater robustness than competing approaches on numerical classification datasets.The evaluation adds Gaussian noise at σ = 0, 0.01, 0.05, and 0.1, using GPT-3.5-Turbo for all LLM-based methods.Source: Figure 10 · p. 22, §E.2 · p. 22
E.3 Impact of Domain Knowledge
- Domain-informed transforms include Log_Cholesterol for heart-disease prediction and proliferation_activity for breast-cancer prediction, while the no-domain-knowledge variant uses less interpretable transformations.The passage attributes reduced outlier impact and variance stabilization to the logarithmic cholesterol transform.Source: §E.3 · p. 23
E.4 Impact of Multi-Island Evolution
- Moderate island counts provide the best balance between exploration and refinement; one island limits feature diversity, while too many reduce refinement depth.The study evaluates k = 1, 3, and 5 and reports overall performance as robust across island counts when the compute budget is appropriately distributed.Source: §E.4 · p. 23, Table 16 · p. 23
E.5 Impact of Evolutionary Refinement
- On PC1 and Balance-Scale, evolutionary refinement improves validation accuracy, while the non-refinement variant stagnates after seven and five iterations, respectively.The passage attributes the stagnation to local optima and says evolutionary refinement helps escape them.Source: §E.5 · p. 23, Figure 12 · p. 24