PaperlayerView exact revision on arXiv

Source-linked AI summary

SymboLLM-FE: LLM-Accelerated Symbolic Regression for Automated Feature Engineering on Tabular Data

Zi-Jian Cheng, Zi-Yi Jia, Zhi Zhou, Yu-Feng Li, Lan-Zhe Guo

arXiv:2608.28408v1cs.LG

TL;DR

Tabular data may lack informative features, while traditional and LLM-based AutoFE face interpretability, iteration-cost, hallucination, and bias challenges. SymboLLM-FE uses symbolic regression to discover mathematically grounded candidates and LLMs to refine them for interpretability. Across six real-world datasets and four Kaggle competitions, it outperforms existing AutoFE methods while using single-digit LLM calls.

  • Problem

    Tabular data often suffers from insufficient feature informativeness and implicit high-order interactions, while existing AutoFE methods have interpretability, iteration-cost, hallucination, and bias challenges.

  • Method

    SymboLLM-FE combines symbolic regression for mathematically grounded candidate formulas with LLM refinement using domain-specific priors and downstream validation.

  • Results

    SymboLLM-FE demonstrates superior performance and generalizability across six real-world datasets and four Kaggle competitions compared with current AutoFE methods.

  • Takeaways & Limitations

    Statistical prior-grounded LLM refinement and single-digit LLM calls address the paper’s targeted interpretability and iterative-experimentation challenges.

  • Takeaways & Limitations

    SymboLLM-FE still incurs prohibitive time overhead in resource-constrained or real-time scenarios, and its sliding-window strategy may miss non-adjacent feature synergies.

Abstract

from arXiv · show

Tabular data, as a core data format in machine learning, often lacks the discriminative power needed for high-performance modeling due to insufficient feature informativeness. Automated Feature Engineering (AutoFE) overcomes this by automating feature generation and selection, ensuring both model performance and operational efficiency. However, traditional AutoFE often yield features with poor interpretability because they rely on blind mathematical transformations, while large language models (LLM)-based AutoFE faces challenges in requiring costly multi-round iterations to generate high-utility features to effectively enhance model performance, compounded by inherent risks of bias and hallucination. In this paper, we combine symbolic regression with LLMs for feature engineering (SymboLLM-FE) to solve these challenges. We extract mathematically expressive formulas strongly correlated with the target via symbolic regression, which can enhance model performance, then refine them by LLMs with rich prior knowledge to ensure interpretability. Empirical results on six real-world datasets and four Kaggle competitions demonstrate that SymboLLM-FE outperforms existing AutoFE. SymboLLM-FE also addresses the dual challenges of poor interpretability and numerous iterations by employing a statistical prior-grounded LLM refinement mechanism and single-digit LLM calls.

1 Introduction

Tabular data often lacks informative features, while existing AutoFE methods trade off predictive performance, interpretability, efficiency, or reliability. SymboLLM-FE combines symbolic regression with LLM refinement to generate mathematically grounded, interpretable features and reports superior performance across broad evaluations.

  • Motivation: Tabular data often suffers from insufficient feature informativeness and implicit high-order interactions, contributing to suboptimal predictive performance.
  • Limitations of Existing AutoFE: Traditional AutoFE use predefined transformations or exhaustive search, but their complex mathematical features often lack semantic meaning and domain-specific interpretability.
  • Limitations of Existing AutoFE: LLM-based AutoFE provide semantic understanding but may require extensive downstream validation iterations and remain susceptible to hallucinations and implicit bias.
  • SymboLLM-FE: SymboLLM-FE combines symbolic regression for performance-enhanced feature discovery with LLM refinement for interpretability and business-logic alignment.
  • Evaluation: Evaluations on six real-world datasets and four Kaggle competitions demonstrate superior performance and generalizability compared with current AutoFE methods.
  • SymboLLM-FE: SymboLLM-FE uses statistical prior-grounded refinement and single-digit LLM calls to address poor interpretability and iterative experimentation.

2 Related Work

Related work describes LLMs as feature-engineering tools and symbolic regression as a source of interpretable mathematical relationships. The paper identifies a gap in using symbolic-regression equations as inputs for LLM-based feature engineering.

  • LLM-Based Tabular Learning: LLMs have been explored for tabular tasks and feature engineering because their prior knowledge and contextual understanding can improve data representation.
  • Traditional AutoFE: Traditional AutoFE use heuristic search or reinforcement learning but can face combinatorial explosion and weak alignment with domain-specific business logic.
  • Symbolic Regression: Symbolic regression discovers mathematical expressions that uncover latent data relationships while supporting feature interpretability and predictive accuracy.
  • Research Gap: Existing LLM–symbolic-regression work had not extended symbolic-regression equations into LLM-based feature engineering.

3 Problem Formulation and Analysis

The paper formulates AutoFE as maximizing downstream prediction performance through generated features, then contrasts traditional and LLM-based AutoFE bottlenecks. Traditional methods sacrifice interpretability, while LLM-based methods incur iteration costs and reliability risks.

  • 3.1 Problem Formulation: AutoFE generates m new features Cnew from the original feature set C, expanding the input dimension from n to n + m to improve predictor performance.
  • 3.1 Problem Formulation: The objective is to maximize predictor performance by changing the features generated by AutoFE.
  • 3.2 Analysis: Traditional and LLM-based AutoFE expose distinct bottlenecks in efficiency and feature quality.
  • 3.2 Analysis: Traditional AutoFE combines raw columns through blind mathematical operator stacks, producing large feature sets without semantic grounding or physical meaning.
  • 3.2 Analysis: The resulting traditional features provide poor actionable insight and generalization because their interpretability is severely compromised.
  • 3.2 Analysis: LLM-based AutoFE improves semantic interpretability through task background knowledge but struggles with high iteration costs.
  • 3.2 Analysis: LLM-generated features can lack robust discriminative power, requiring repeated trial-and-error validation and refinement that increases computational time.
  • 3.2 Analysis: Existing AutoFE therefore motivates a framework that avoids both LLM iteration overhead and traditional mathematical combinations lacking interpretability.

4 Method

SymboLLM-FE combines symbolic regression with LLM refinement in a staged pipeline. It narrows formula search, constructs a formula repository, generates and validates executable features, and iteratively prunes or refines them.

  • Method: SymboLLM-FE combines symbolic regression’s explicit formulas with LLM knowledge to refine candidate features and improve interpretability.
  • Dataset Split: Training data supports correlation-based subset construction and symbolic-model fitting, validation data guides LLM optimization, and test data is reserved for final evaluation.
  • Formula Generating: For each selected subset, symbolic regression produces a formula-based feature repository containing the subset, symbolic-model performance, and formula.
  • Feature Set Sampling: The expanding-sliding window applies Spearman correlation analysis and constructs feature subsets for separate symbolic regression models.
  • Feature Set Sampling: The window strategy reduces the feature-subset search from exponential O(2^n) to polynomial O(n^2).
  • Feature Set Sampling: Spearman-ranked features are processed with an expanding phase that increases window size from floor(n/2) to n and a sliding phase that shifts each window one feature at a time.
  • LLM-guided Code Generation: LLMs convert repository formulas into natural-language descriptions and executable Python pandas feature-engineering code using dataset context and prompts.
  • Iterative Feature Refinement: Generated features augment the dataset for downstream validation, whose performance feedback guides iterative refinement and pruning of harmful or redundant features.

5 Experiments

Experiments evaluate SymboLLM-FE across diverse datasets, downstream models, LLM backbones, Kaggle competitions, efficiency settings, and ablations. The framework reports stronger predictive performance while using a statistically guided symbolic-regression search and LLM refinement pipeline.

  • Experimental Setup: Experiments cover binary classification, multi-class classification, and regression datasets from finance, healthcare, OpenML, and Kaggle.The evaluation uses CatBoost, XGBoost, MLP, and TabPFN downstream models, with Accuracy, ROC-AUC, F1-score, RMSE, MAE, and R2 metrics.
  • Main Results: SymboLLM-FE achieves a statistically significant average gain of 1.23% over traditional AutoFE and approximately 1% higher accuracy than LLM-based AutoFE.The comparison uses TabPFN as the downstream predictor on real-world datasets.
  • Main Results: 2.5pp average score improvement is achieved by SymboLLM-FE+TabPFN over vanilla TabPFN and OpenFE+TabPFN across four Kaggle competitions.Figure 3 reports score comparisons for classification and regression competitions.
  • Efficiency and Generated Features: The framework reaches a model score of 80.02 with only 70 generated features and requires a mere 4 LLM calls.Restricting the LLM to refinement reduces search-space and convergence overhead.
  • Efficiency and Generated Features: Spearman-guided expanding-sliding windows reduce feature-search complexity from exponential O(2n) to polynomial O(n2).Candidate-formula counts follow derived arithmetic bounds, while local running time scales linearly with formulas and dataset size.
  • Ablation Study: Removing Spearman pre-sorting, expanding-sliding windows, or LLM-guided generation degrades performance, with LLM refinement addressing redundancy and suboptimal symbolic rules.The ablation study evaluates these components across multiple datasets using independent random seeds.

6 Conclusion

SymboLLM-FE combines symbolic regression with LLM refinement for automated feature engineering. The paper reports consistent advantages over current AutoFE, improved interpretability and efficiency, and effective integration with TabPFN for Kaggle tasks.

  • Conclusion: SymboLLM-FE combines symbolic regression’s mathematical rigor with LLMs’ refinement capabilities for AutoFE.The framework is presented as a hybrid feature-engineering approach.
  • Conclusion: The framework demonstrates consistent advantages over current AutoFE methods, enhanced interpretability, and greater efficiency.These conclusions summarize the paper’s reported evaluation findings.
  • Conclusion: Integrating SymboLLM-FE with TabPFN provides a feature-engineering module evaluated effectively on real-world Kaggle competitions.The paper presents this integration as a solution for real-world tasks.

Limitations

SymboLLM-FE remains constrained by computational overhead and by a sliding-window feature-subset strategy that can miss interactions among non-adjacent variables.

  • Limitations: Prohibitive time overhead limits SymboLLM-FE’s feasibility in resource-constrained or real-time scenarios.The limitation persists despite two mechanisms intended to improve scalability on high-dimensional datasets.
  • Limitations: Continuous sliding windows may overlook synergistic effects when implicitly correlated features are separated in the importance ranking.Non-adjacent variables may not be placed in the same constructed subset, causing their joint predictive contribution to be missed.

A Implementation Details of Symbolic Regression

The implementation uses genetic-programming symbolic regression to evolve interpretable formulas, penalizes expression complexity, and supplies selected rules and their performance to an LLM for code generation.

  • Algorithmic Framework: Symbolic regression evolves syntax-tree populations through initialization, fitness evaluation, genetic operations, and iterative evolution.The best individual from the final population is returned as the derived symbolic rule.
  • Algorithmic Framework: Fitness combines training-set MSE with a parsimony penalty weighted by Ω to balance predictive accuracy and expression complexity.The tree-size term penalizes bloat, with Ω controlling the accuracy–complexity trade-off.
  • Algorithmic Framework: Genetic operations use tournament selection, subtree crossover, mutation, reproduction, and repeated evolution until stopping criteria are met.These operations introduce variation and select higher-fitness expressions.
  • Operator Design: The primitive set contains 14 arithmetic, transcendental, and piecewise protected operators for expressive yet numerically stable formulas.Protected operations return safe defaults for undefined states such as division by zero.
  • Hyperparameters: A fixed symbolic-regression configuration uses a population size P = 20,000, crossover probability pcx = 0.9, and tournament size 100.The configuration is intended to provide broad search coverage while maintaining computational feasibility.
  • LLM Code Generation: The LLM prompt provides symbolic rules with test-set MAE and requests feature additions, optional column dropping, explanations, and sequential Python code blocks.Added columns follow the existing naming scheme, while dropped columns become unavailable to later blocks.

C.1 Datasets

The evaluation uses datasets spanning classification and regression across finance, healthcare, education, aerospace, and other settings, with multiple downstream model families and AutoFE baselines.

  • Datasets: The datasets cover binary classification, multi-class classification, and regression tasks across fields including finance and healthcare.
  • Datasets: The benchmark includes OpenML and Kaggle datasets such as Credit-g, Cmc, Ailerons, Spaceship, Academic, and Tesla.
  • Datasets: Credit-g contains 1,000 entries and 20 categorical or symbolic attributes for predicting good versus bad credit risk.
  • Datasets: Ailerons is a regression dataset with 12,250 samples and 33 features from an F16 aircraft-control problem.
  • Models and baselines: SymboLLM-FE is evaluated with CatBoost, XGBoost, MLP, and TabPFN, and compared against two traditional and five LLM-based AutoFE methods.
  • Evaluation: Classification is assessed using Accuracy and F1-score, regression using RMSE and R-squared, with results averaged over three random seeds.

D.2 Comparison with Other Feature Importance Metrics

Spearman is selected as the primary feature-importance metric because its rankings showed the strongest consistency with Pearson, SHAP, and mutual information rankings.

  • Metric comparison: Kendall’s τ consistency across Pearson, SHAP, mutual information, and Spearman identifies Spearman as the primary metric.Spearman achieved the highest consistency score, τ = 0.65.
  • Evaluation tables: The comparison tables evaluate AutoFE methods with CatBoost, XGBoost, MLP, and TabPFN downstream models.
  • Evaluation tables: The reported table metrics span Accuracy and ROC-AUC for classification and RMSE, MAE, and R2 for regression.

E Complete Experiments

Complete experiments report strong cross-model performance, statistical evidence, and feature-generation reliability, while also documenting implementation constraints and the bounded role of LLM refinement.

  • Complete experiments: SymboLLM-FE achieves the most best and second-best results across classification and regression datasets with tree-based and neural downstream models.The reported comparisons include CatBoost, XGBoost, MLP, and TabPFN.
  • Ablation stability: The complete framework attains the highest accuracy with the lowest standard deviation among ablation variants.The ablation study uses cross-validation and multiple independent random seeds.
  • Ablation significance: Paired t-tests find statistically significant improvements over all ablation baselines, including the strongest ablation variant.The confidence intervals show minimal overlap, particularly for variants missing key components.
  • Implementation limitations: The implementation uses heuristic code processing, including markup stripping and line filtering that can discard valid statements or retain invalid column references.The pipeline assumes prescribed output formatting and does not validate referenced column names before execution.
  • Reliability: Symbolic regression has a 100% acceptance rate, while LLM-generated feature-engineering code has a 92.6% success rate.
  • LLM refinement: An audit finds that 100% of LLM-generated rules are traceable to symbolic-regression outputs through interpretation or redundancy optimization.The LLM functions as a semantic interpreter and consolidation engine for mathematically validated expressions.
  • Scalability: SymboLLM-FE reduces the symbolic-regression search space from exponential O(2^n) to polynomial O(n^2) complexity.An expanding-sliding window and correlation-guided search constrain candidate generation.
  • Statistical analysis: Wilcoxon signed-rank tests show significant improvements over strong baselines on four of six real-world datasets, with p < 0.001 on Academic.Credit-g and Ailerons do not reach statistical significance.

J Comparison with Genetic Programming Methods

SymboLLM-FE outperforms or remains highly competitive with Genetic Programming baselines across six datasets while converting opaque symbolic expressions into domain-interpretable features. Its cases illustrate how LLM refinement adds semantic meaning to mathematically discovered relationships.

  • Performance comparison: SymboLLM-FE achieves superior or highly competitive performance across six datasets, with particularly significant gains on regression tasks and top rankings on most classification benchmarks.The comparison reports Accuracy (%) for Credit-g, Spaceship, Cmc, and Academic, and RMSE for Ailerons and Tesla.
  • Interpretability case studies: Eight case studies show how the framework transforms opaque symbolic regression expressions into semantically interpretable features with domain-grounded rationales.Each case presents the original formula, its interpretability challenge, the LLM-generated feature and rationale, and the resulting semantic interpretation.
  • Interpretability case studies: On Titanic, the opaque formula tan(−0.607 −tan(X5)) is refined around tan(X5), interpreted as an age-related nonlinear threshold effect on survival.The LLM links the repeated tan(X5) component to steep or periodic changes in the age–target relationship and rescue-priority dynamics.
  • Interpretability case studies: For CryoSleep, the LLM extracts X2 · tan(X2) and interprets it as multiplicative amplification of the protective effect when X2 = 1.The expression reframes repeated arithmetic and trigonometric interactions as a protection multiplier involving CryoSleep and other covariates.
  • Interpretability case studies: For HomePlanet and Destination, the LLM replaces an uninterpretable product of categorical codes with X4 −X1, a directed difference capturing their encoded directional relationship.The proposed feature is intended to represent how much Destination exceeds HomePlanet when the variables exert opposing effects on classification.
  • Interpretability case studies: On CMC, opaque education and child-count transformations are reframed as fertility–education trade-off features such as min(X3, X1), X3 −X1, or X3/(X1 + 1).The LLM connects these alternatives to a sociological interpretation of contraceptive decision-making.

K.6 Interpretability, Hallucination Mitigation, and Bias Analysis

SymboLLM-FE assigns symbolic regression the role of discovering effective mathematical structures and LLMs the role of injecting domain semantics. Analyses report high fidelity, reduced semantic drift, lower operator-distribution bias, and an integrator architecture designed for controlled automation.

  • Interpretability: Fidelity verification finds high mathematical, structural, and semantic similarity between symbolic-regression formulas and LLM-refined features on a hold-out set.The analysis presents refinement as logical simplification that preserves the original statistical signals rather than post-hoc explanation.
  • Hallucination mitigation: SymboLLM-FE reports a high comprehensive hallucination mitigation score and significantly lower semantic drift than baseline methods.The framework attributes this constraint to anchoring LLM refinement with symbolic-regression rules.
  • Interpretability: Symbolic regression discovers effective mathematical structures, while LLMs add domain-specific semantics, operational context, and theoretical grounding.The division of labor addresses both what mathematical structure exists and what that structure means in the application domain.
  • Architectural advantages: The framework uses the LLM as a deterministic feature integrator rather than an unconstrained generator, restricting synthesis to explicit logical compositions of existing features.This design targets semantic consistency, reduced combinatorial search, lower inference overhead, and greater controllability.
  • Deployment automation: The pipeline automates error recovery by capturing runtime exceptions, converting them into structured traces, and feeding corrective prompts back for iterative refinement.This closed-loop generate-execute-feedback protocol enables deployment without manual intervention.
  • Bias and contamination: The paper claims that schema-level metadata and symbolic inference rules decouple feature construction from historical samples and target variables, preventing data leakage and dataset-specific artifact fitting.The same architectural claim is presented as reducing propagation of historical annotation biases across heterogeneous deployments.
Loading 2608.28408v1…