PaperlayerView exact revision on arXiv

Source-linked AI summary

SIGMA: SHAP-Guided Implicit-Trajectory Generation for Metadata-Free LLM-Based AutoFE

Xuan Zheng, Kento Uchida, Shinichi Shirakawa

arXiv:2608.17948v1cs.LGcs.AI

TL;DR

LLM-based AutoFE is limited by unavailable semantic metadata and growing optimization trajectories. SIGMA addresses both with SHAP-guided generation and EXIT’s implicit trajectory, achieving comparable baseline performance while reducing duplicate generation.

  • Problem

    LLM-based AutoFE often assumes semantic information and expanding trajectories, limiting applicability when metadata is unavailable and risking context-length constraints.

  • Method

    SIGMA uses SHAP values for task-aware grouped feature generation and EXIT, which represents optimization history through exposed features rather than explicit trajectory tokens.

  • Results

    SIGMA achieves comparable performance to current LLM-based baselines and reduces duplicate generation from 36.6% without EXIT by 30%.

  • Takeaways & Limitations

    SIGMA provides a constant-context, metadata-free AutoFE framework that remains competitive with traditional AutoFE through efficient feature utilization.

  • Takeaways & Limitations

    SIGMA currently focuses only on classification, retains a duplicate rate near 7%, and exhibits validation-set overfitting with performance degradation.

Abstract

from arXiv · show

Recent research has leveraged Large Language Models (LLMs) to enhance Automated Feature Engineering (AutoFE) through semantic descriptions and trajectory-based prompting. However, there exist two challenges that limit their applicability and scalability in long-horizon optimization: (1) semantic metadata is unavailable in many practical settings, and (2) trajectory accumulation increases the risk of exceeding the context window, while without it, the generation process can become unstable, leading to becoming stuck in the local optima and a high duplicate rate of generated features. To this end, we propose a SHAP-enhanced Implicit-trajectory Generation for Metadata-free AutoFE (SIGMA), a scalable constant-context optimization framework. SIGMA leverages SHAP values to provide task-aware signals for guiding group feature generation instead of semantic information. In addition, we adopt an EXposed-feature Implicit Trajectory (EXIT) approach, where the exposed features in the prompt implicitly represent the trajectory. Empirical results demonstrate that SIGMA achieves performance comparable to the state-of-the-art (SOTA) LLM baselines with a nearly constant prompt length. Notably, EXIT significantly reduces the duplicate ratio of generated features from 37.2% to 6.8%. At the same time, SIGMA matches traditional SOTA performance with only 5.4 features on average, demonstrating substantial efficiency gains in feature utilization.

1. Introduction

SIGMA is a metadata-free, constant-context LLM-based AutoFE framework that uses SHAP importance signals and implicit exposed-feature trajectories to guide structured feature generation. It achieves comparable performance to current LLM-based baselines while reducing duplicate generation and using features efficiently.

  • Motivation: Semantic-information requirements limit existing LLM-based AutoFE methods in real-world scenarios, motivating SIGMA’s metadata-free design.Traditional AutoFE methods also rely on manually defined transformation spaces that can limit exploration and produce sub-optimal results.
  • EXIT: EXIT represents optimization history through the visible feature composition rather than enumerating an explicit trajectory in the prompt.This implicit trajectory approach addresses the overhead of expanding long-horizon optimization prompts.
  • SIGMA framework: SIGMA replaces semantic descriptions with SHAP-based importance signals and uses grouped generation for structured feature exploration.The framework targets metadata-free LLM-based AutoFE through task-aware feature-importance guidance.
  • EXIT: 37.2% to 6.8%: EXIT reduces the duplicate generation rate without requiring an explicit trajectory in the prompt.The reported reduction supports more diverse generation during long-horizon optimization.
  • Results: SIGMA achieves comparable performance to current LLM-based baselines and remains competitive with traditional AutoFE through efficient feature utilization.The introduction characterizes SIGMA as a scalable constant-context optimization framework for metadata-free LLM-based AutoFE.

2. Related Work

Related work spans traditional expansion–reduction AutoFE frameworks and newer LLM-based approaches that exploit semantic knowledge and in-context reasoning. A recent direction decouples transformation proposal from feature selection.

  • Traditional AutoFE Methods: Traditional AutoFE methods generally follow an expansion–reduction framework for generating and selecting features.DFS uses relational paths and mathematical primitives for cross-table feature generation, while ExploreKit combines original features and ranks candidates.
  • LLM-based AutoFE Methods: LLM-based AutoFE methods leverage Transformer-based in-context learning and reasoning capabilities to generate features or transformation rules from textual descriptions.CAAFE generates interpretable features using prior semantic knowledge, while FeatLLM generates rules that transform features into binary sequences.
  • LLM-based AutoFE Methods: Li et al. (2026) decoupled transformation-operation proposal from the selection process.

3. Methodology

SIGMA’s methodology combines SHAP-based feature grouping, intra- and cross-group feature generation, and EXIT-based context control. It uses noise-calibrated importance groups and implicit trajectory signals from masked features and operations.

  • Overview: SIGMA comprises SHAP-based Feature Grouping, Intra-Group and Cross-Group Generation, and EXIT-based context-size control.The overall procedure is summarized in Algorithm 1.
  • SHAP-based Feature Grouping: A Gaussian noise feature provides a SHAP-derived threshold for assigning normalized features to top, useful, and weak groups.The top group contains features ranked in the top 10%, with a minimum of two features; weak features fall below the noise threshold.
  • SHAP-based Feature Grouping: Noise-threshold grouping adapts LLM attention to dataset-specific feature-importance distributions instead of enforcing a fixed partition ratio.Datasets may contain mostly below-noise features or features that are all more important than noise.
  • Intra-Group and Cross-Group Generation: Each generation step produces one intra-group feature for deep transformation or same-group interaction and one cross-group feature.The intra-group objective is to uncover hidden patterns within a feature group and improve already influential features.
  • EXIT: EXIT tracks selected generation features and masks them from subsequent prompts, while temporarily forbidding the two most frequent operations.Only operations associated with failed generations are tracked and temporarily forbidden; continuously improving operations are rewarded.

4. Experiments and Results

Across 16 tabular classification datasets, SIGMA achieves competitive predictive performance without semantic metadata while maintaining near-constant prompt length through EXIT. It also uses feature capacity efficiently, reduces duplicate generation, and supports deeper reusable feature construction, with operation tracking introducing an error–duplicate trade-off.

  • Experimental setup: 16 public tabular classification datasets were evaluated with F1-score as the primary metric and XGBoost as the downstream model.Datasets were capped at 50,000 samples, split 8:2, and resampled with three random seeds.
  • LLM-based AutoFE comparison: SIGMA achieves competitive performance with semantic-based CAAFE and consistently outperforms metadata-free OCTree across most datasets.These results indicate that effective features can be generated without detailed semantic descriptions.
  • Prompt efficiency: SIGMA maintains a near-constant prompt length, whereas prior LLM-based AutoFE frameworks show escalating prompt-token trends during optimization.OCTree limits its trajectory to the top-7 performing features, while CAAFE imposes no upper bound.
  • Feature efficiency: 0.2% is SIGMA’s F1-score gap to OpenFE under feature budget K = 20, achieved with approximately 5 accepted features rather than the full budget.DFS and OpenFE generate averages of 724.8 and 1556 features per dataset when all generated features are retained.
  • Ablation analysis: 36.6% of generated features are duplicated without EXIT, while EXIT decreases the duplicate rate by 30%.The duplicate generation wastes generation opportunities and leads to a lower acceptance rate.
  • Ablation analysis: 5.6% is SIGMA’s code error rate, 4% higher than other variants, revealing a trade-off between safer code generation with more duplicates and lower duplication with more errors.Without predefined operations, 14.1% of generated features are redundant, reflecting limited operation diversity.

5. Conclusion

SIGMA is a scalable constant-context framework for metadata-free LLM-based AutoFE that uses SHAP-guided grouped generation instead of semantic information. It achieves comparable performance to LLM-based baselines and remains competitive with traditional AutoFE, while retaining limitations in duplicate generation, task coverage, and validation overfitting.

  • Contributions: SIGMA introduces constant-context optimization for metadata-free LLM-based AutoFE, using SHAP values and grouped generation for task-aware, structured feature exploration.The framework replaces semantic information with SHAP-guided signals and supports long-horizon optimization through grouped generation.
  • Results: SIGMA achieves comparable performance to current LLM-based baselines with nearly constant context and remains competitive with traditional AutoFE through efficient feature utilization.These empirical results support the framework’s scalability and efficiency claims.
  • Limitations: Nearly 7% duplicate generation remains because current operation restrictions are still weak.Reducing duplicate generation is identified as a future-work priority.
  • Limitations: SIGMA currently focuses only on classification, leaving regression as an additional task for future consideration.The conclusion explicitly identifies regression-task coverage as an unresolved limitation.
  • Limitations: Performance degradation reflects overfitting to the validation set and motivates future work on this issue.The conclusion lists validation-set overfitting as a remaining limitation.

Appendix A. Prompt Examples

Appendix A presents the overall prompt template for optimizing feature distributions to improve classifier performance, including feature organization, formatting, and required code-generation instructions.

  • Prompt framing: The prompt assigns an expert to optimize feature distribution for <CLS_MODEL> performance on an <N>-class classification problem.This framing appears at the beginning of the template.
  • Prompt structure: The template includes dedicated sections for feature organization and the required feature format.These sections are represented by <GROUPINGDESCRIPTION> and <FEATURE_FORMAT>.
  • Code generation: Code generation requires TWO separate Python functions, each creating ONE new feature.The template supplies operation information and separate titles, purposes, and requirements for both functions.

Appendix B. Implementation Details

Appendix B specifies the implementation settings for AutoFeat and DFS, while OpenFE, CAAFE, and OCTree use their official implementations. AutoFeat and DFS use explicit feature-engineering configurations, whereas OpenFE uses defaults and CAAFE uses XGBoost for fair comparison.

  • AutoFeat uses the official Python library with feateng steps=2 and featsel runs=3.
  • DFS uses the official Python library with seven transformation primitives and max depth=2.The primitives are add numeric, subtract numeric, multiply numeric, divide numeric, natural logarithm, square root, and absolute.
  • OpenFE and OCTree use their official implementations, while CAAFE uses the official Python implementation with XGBoost for fair comparison.

Appendix C. Additional Comparison Results of Different Metrics

Appendix C compares LLM-based and traditional AutoFE using overall accuracy (ACC) and AUC-ROC. The traditional-method comparison is conducted under a feature budget of 20.

  • Overall comparison: Tables 4 and 5 report overall comparisons between LLM-based AutoFE and traditional AutoFE.Table 4 covers LLM-based methods, while Table 5 covers traditional methods.
  • Evaluation metrics: The comparisons use overall accuracy (ACC) and AUC-ROC as evaluation metrics.Both metrics are identified in the captions for Tables 4 and 5.
  • Evaluation setting: The traditional-method comparison is conducted under a feature budget of 20.This budget condition accompanies Table 5’s comparison.

Appendix D. Impact of LLMs on different datasets

Appendix D examines how different LLM backbones affect F1-score across datasets. The table reports dataset characteristics using class count, feature count, and sample count.

  • LLM backbone impact: The appendix demonstrates the influence of different LLM backbones.The comparison is presented across datasets.
  • Evaluation setup: Table 6 evaluates backbone impact using F1-score for each dataset.C denotes the number of classes, F the number of features, and N the number of samples.
Loading 2608.17948v1…