PaperlayerView exact revision on arXiv

Source-linked AI summary

Large Language Models Can Automatically Engineer Features for Few-Shot Tabular Learning

Sungwon Han, Jinsung Yoon, Sercan O Arik, Tomas Pfister

arXiv:2404.09491v2cs.LG

TL;DR

Existing LLM-based tabular methods face per-sample inference, training-access, and prompt-length constraints. FeatLLM uses LLMs to generate rule-based features for a simple downstream predictor, and reports strong few-shot performance across 13 tabular datasets, outperforming contemporary baselines.

  • Problem

    LLM-based tabular methods can require an LLM inference per sample, fine-tuning for high accuracy, or prompts that become infeasible as tabular feature counts grow.

    Source: §1 · p. 1

  • Method

    FeatLLM prompts LLMs to infer class rules from prior knowledge and few-shot examples, converts rules into binary features, and fits a simple model to estimate class likelihoods.

    Source: §1 · p. 2

  • Results

    Across 13 tabular datasets in low-shot regimes, FeatLLM shows strong, robust performance and outperforms contemporary few-shot learning baselines.

    Source: §1 · p. 2

  • Takeaways & Limitations

    FeatLLM requires only API access without training, uses minimal inference cost, and avoids sending each sample through the LLM at inference.

    Source: §1 · p. 2, §5 · p. 9

    Runtime for models using API queries depends on the API’s status at the time of use.

    Source: §4.2 · p. 8

Abstract

from arXiv · show

Large Language Models (LLMs), with their remarkable ability to tackle challenging and unseen reasoning problems, hold immense potential for tabular learning, that is vital for many real-world applications. In this paper, we propose a novel in-context learning framework, FeatLLM, which employs LLMs as feature engineers to produce an input data set that is optimally suited for tabular predictions. The generated features are used to infer class likelihood with a simple downstream machine learning model, such as linear regression and yields high performance few-shot learning. The proposed FeatLLM framework only uses this simple predictive model with the discovered features at inference time. Compared to existing LLM-based approaches, FeatLLM eliminates the need to send queries to the LLM for each sample at inference time. Moreover, it merely requires API-level access to LLMs, and overcomes prompt size limitations. As demonstrated across numerous tabular datasets from a wide range of domains, FeatLLM generates high-quality rules, significantly (10% on average) outperforming alternatives such as TabLLM and STUNT.

1. Introduction

LLMs’ prior knowledge and reasoning could support tabular learning, but existing LLM-based methods face inference-cost, training-access, and prompt-length constraints. FeatLLM instead uses LLMs to generate features, then reports strong few-shot results across 13 tabular datasets.

1. Introduction

  • Existing LLM-based tabular methods can require an LLM inference per sample, fine-tuning, or prompts too long for high-dimensional data.These constraints challenge real-world application, particularly when training code or tuning APIs are inaccessible.Source: §1 · p. 1
  • FeatLLM uses LLMs as feature engineers, generating class rules that replace existing features for prediction.The rules express feature conditions associated with class outcomes; once features are discovered, a complex model is not needed for inference.Source: §1 · pp. 1–2
  • It converts rules into binary features, fits a low-complexity model to estimate class probabilities, and ensembles repeated feature-generation runs with feature bagging.Bagging the features and samples mitigates excessive prompt size.Source: §1 · p. 2
  • FeatLLM outperforms contemporary few-shot learning baselines across 13 tabular datasets in low-shot regimes.The authors report strong, robust performance and high-quality features drawing on both LLM prior knowledge and data knowledge.Source: §1 · p. 2

2. Related Work

Few-shot tabular learning has explored prior knowledge from unlabeled data, pretraining, and transfer across datasets, while heterogeneous tables make universal augmentation difficult. LLM-based approaches serialize data into prompts; FeatLLM instead extracts class rules and turns them into features for prediction.

2.1. Few-Shot Learning with Tabular Data

  • Few-shot tabular methods have used unlabeled data, task-agnostic pretraining, and transfer across datasets to add information or inductive biases.The review notes that heterogeneous tabular data makes universally applicable augmentation difficult, and augmentation strategies have not been notably effective in few-shot settings.Source: §2.1 · p. 2

2.2. Language-Interfaced Tabular Learning

  • LLM-based tabular methods serialize data, feature descriptions, and task descriptions into prompts, using either parameter-efficient tuning or in-context demonstrations.Figure 1 depicts FeatLLM extracting class rules, converting them into binary features, and fitting a linear layer for class likelihoods, with repeated runs for ensembling.Source: §2.2 · pp. 2–3, Figure 1 · p. 3
  • FeatLLM departs from per-sample LLM prediction by parsing class conditions into program code and using them to create features for prediction.The extracted rules use prior knowledge and few-shot sample information.Source: §2.2 · p. 3

3. Methods

FeatLLM uses few-shot examples and structured prompting to extract class rules, converts those rules into binary features, and trains simple models to estimate class likelihood. Bagging and ensembling combine repeated models to address noisy rules and prompt-size limits.

3. Methods

  • FeatLLM turns rules inferred from few labeled samples into binary features, then uses a simple model to estimate class likelihood without routing each prediction through the LLM.The method repeats this process across trials and averages class probabilities from the resulting models.Source: §3 · p. 3, §3.2 · p. 5

3.1. Prompt Design for Extracting Rules

  • The extraction prompt supplies task and feature descriptions with labeled demonstrations, then guides the LLM to infer feature–target relationships before deriving class-specific rules.It fixes the number of extracted rules at 10 per class and specifies response and rule formats to facilitate parsing.Source: §3.1 · pp. 3–4, Figure 2 · p. 4

3.2. Inferring Class Likelihood via Rules

  • FeatLLM asks an LLM to convert generated natural-language rules into executable code, which creates binary indicators of whether samples satisfy each rule.A linear model learns non-negative feature weights from few-shot samples, while bagging and ensembling combine trials to improve robustness and reduce prompt-size pressure.Source: §3.2 · pp. 4–5

4. Experiments

Across its low-shot tabular experiments, FeatLLM is a top or second-place performer against baselines, and the analyses examine its component contributions, runtime, feature quality, robustness, and hyperparameters.

4.1. Performance Evaluation

  • The evaluation spans 13 tabular datasets and compares conventional, unlabeled-data, pretrained, and LLM-based baselines.The datasets include binary and multiclass tasks, including newer and synthetic datasets.Source: §4.1 · pp. 5–7
  • On datasets with over 100 attributes, TabPFN and two LLM-based baselines were excluded after failing inference, while FeatLLM performed satisfactorily on high-feature datasets.Table 3 covers two datasets with over 100 attributes.Source: Table 3 · p. 7, §4.1 · p. 7
  • FeatLLM consistently ranks first or second against baseline models, and with 4 shots achieves performance comparable to conventional tabular baselines using 32 shots.Figure 4 summarizes average AUC across datasets and shot counts.Source: §4.1 · p. 7, Figure 4 · p. 8

4.2. Analysis & Discussion

  • All ablations degraded AUC; ensemble improved performance in every setting, while tuning mattered more with more shots and descriptions and reasoning mattered more with fewer shots.The ablation summaries average AUC changes across 13 datasets.Source: §4.2 · p. 8, Table 4 · p. 8
  • On Adult with 16 training shots, FeatLLM inference time was comparable to conventional methods and lower than other LLM-based approaches; training required 30 API queries.The runtime comparison used one A100 GPU by default, except TabLLM, which used four.Source: §4.2 · p. 8, Table 5 · p. 8
  • Most generated rules were relevant before tuning, and linear-model tuning filtered rules whose learned weights fell below the 0.1 threshold.Feature quality was assessed by the ratio of newly generated features with AUROC above 0.5.Source: §4.2 · p. 8, Table 6 · p. 8
  • With unrelated Adult columns added to Heart, FeatLLM had the least performance degradation and only 1–2% of rules used noisy columns, but related columns could still induce spurious correlations.The comparison used eight labeled shots and excluded methods requiring unlabeled data.Source: §4.2 · p. 9, Figure 5 · p. 9
  • Performance gains from adding ensembles converged around 20, while 10 rules was selected as a prompt-length trade-off despite no statistically significant difference across rule counts.More rules can increase information and input dimensionality, but may also increase low-shot overfitting.Source: §4.2 · p. 9, Figure 6 · p. 9

5. Conclusion

FeatLLM uses LLM-generated rules as features for few-shot tabular prediction, reducing inference-time reliance on LLMs. Its stated scope is low-shot learning, with broader feature crafting left for future work.

5. Conclusion

  • FeatLLM shifts LLMs from per-sample prediction to generating prediction criteria as features, then uses bagging and a simple model for inference.The framework needs API access without training and is intended to reduce inference cost and address feature-size constraints.Source: §5 · p. 9
  • FeatLLM is designed for low-shot learning; extending feature crafting to larger datasets and feature types remains future work.The authors also identify interpretability as a future direction for wider application.Source: §5 · p. 9

6. Broader Impact

FeatLLM may be useful where labels are costly and LLM prior knowledge is valuable, but that same knowledge can introduce bias or misinformation into predictions. The appendix also documents task descriptions and Heart-dataset examples of rule extraction and parsing.

6. Broader Impact

  • FeatLLM may benefit finance and healthcare practitioners when labels are difficult or costly to collect and relevant prior knowledge is available.The passage notes that LLMs are pretrained on finance- or healthcare-related text.Source: §6 · pp. 9–10
  • LLM prior knowledge may contain societal biases or misinformation that affect predictions, potentially dominating observed labels.The authors identify analysis of these effects as important future work toward wider adoption.Source: §6 · p. 10

A.1. Task Description for Each Dataset

  • The appendix lists dataset task descriptions used in prompts, with label information and possible answer classes shown in Table 7.Descriptions draw on original dataset links and prior work.Source: §A.1 · p. 13, Table 7 · p. 13

A.2. Example Prompt for Extracting Rules

  • The Heart-dataset extraction example uses a four-shot prompt; Figure 7 shows the prompt and Figure 8 its rule-extraction outcome.The outcome is presented as the result of submitting the preceding prompt.Source: §A.2 · p. 14, Figure 7 · p. 14, §A.3 · p. 15

A.4. Example Prompt for Parsing Rules

B. Hyper-parameter analysis

Performance improves as the number of ensembles increases, then gains begin to converge at around 20 ensembles.

B. Hyper-parameter analysis

C. Baseline Implementation Details

The appendix describes baseline evaluation and implementation choices, then examines rule diversity and the balance between LLM prior knowledge and training samples. Ablations associate temperature, bagging, and sample shuffling with more diverse rules, while label-flip trials show variability in which knowledge source rules prioritize.

C. Baseline Implementation Details

  • Baseline evaluation used a stratified 20% test split, balanced k-shot training samples, and repeated runs with different split and sample randomness.Conventional model parameters were selected by grid search with k-shot cross-validation.Source: C. Baseline Implementation Details · p. 18
  • For conventional machine-learning methods—LogReg, RandomForest, and XGBoost—parameters were determined using grid search with k-shot cross-validation, and each model’s search space is described in Tables 8–10.Source: C. Baseline Implementation Details · p. 18, Table 8 · p. 18, Table 9 · p. 18
  • Unlabeled-data baselines used remaining non-test, non-k-shot samples as unlabeled data; columns with over 20% missingness and rows with missing values were removed.TabPFN was reproduced from its official repository using default parameters.Source: C. Baseline Implementation Details · p. 18
  • LLM baselines used GPT-3.5 for in-context learning and T0 for TabLLM; GPT-3.5 API probabilities were averaged across three temperature-1 runs.For TabLLM, class probabilities came from the class-name token probability using its public checkpoint.Source: C. Baseline Implementation Details · p. 18

D. Diversity of Features Generated by FeatLLM

  • FeatLLM varied temperature, feature and sample bagging, and training-sample order to diversify generated rules; rule sets were compared using matched-feature correlations.The matching procedure used the Hungarian Maximum Matching Algorithm and averaged correlations across iteration pairs.Source: D. Diversity of Features Generated by… · p. 19
  • Rule-set average correlation was 0.368 with variance 0.341; removing temperature, bagging, or instance shuffling increased correlation, indicating less diversity.Lower correlation indicates more diverse extracted rules.Source: D. Diversity of Features Generated by… · p. 19, Table 11 · p. 19

E. Prior Knowledge vs. Training Samples.

F. Error Rates of Automated Parsing via LLM

FeatLLM’s LLM-derived parsing code sometimes fails, particularly on datasets with many categorical features. The authors suggest specialized code-generation or higher-performance models as possible ways to reduce these errors.

F. Error Rates of Automated Parsing via LLM

  • LLM-generated parsing code can fail, with higher failure rates often occurring on datasets containing many categorical features.The authors attribute this pattern to GPT-3.5 syntax errors when handling categorical columns in pandas, with error probability rising as more categorical rules are used.Source: F. Error Rates of Automated Parsing… · p. 20
  • The authors suggest that a code-generation-specialized or higher-performance model such as GPT-4 could significantly reduce parsing errors.Source: F. Error Rates of Automated Parsing… · p. 20
  • Table 13 reports automated-parsing error rates separately for each dataset.Source: Table 13 · p. 20

G. Handling missing data.

FeatLLM handles missing values in its rule-derived binary features using zero fill, 0.5 fill, or MICE imputation. Imputation showed the lowest performance degradation, while backbone comparisons found task-dependent differences.

G. Handling missing data.

  • FeatLLM compares zero fill, 0.5 fill, and MICE imputation for missing values in rule-created binary features.Zero fill treats the rule as unsatisfied; 0.5 represents uncertainty; MICE uses relationships among training-data rules.Source: G. Handling missing data · p. 20
  • Imputation produced the lowest performance degradation among the tested missing-value strategies.Table 14 summarizes average AUC changes with standard error across 13 datasets and different missing ratios.Source: G. Handling missing data · p. 20, Table 14 · p. 20

H. Varying LLM backbones

I. Additional Experiments on Effect of Adding Spurious Correlations

Adding columns from the related Myocardial dataset made FeatLLM’s performance unstable because many columns were relevant to the Heart task. With more training shots, the linear model can reduce the weight given to irrelevant columns.

I. Additional Experiments on Effect of Adding Spurious Correlations

  • At higher shot counts such as 64, the linear model can reduce instability by assigning less weight to irrelevant columns.Source: §I · p. 22
  • Other LLM-based methods also experienced a significant performance decrease when related-domain Myocardial columns were added.Source: §I · p. 22
  • FeatLLM showed unstable AUC when noisy Myocardial columns were added to Heart, as many such columns were relevant to the task.Rules based on noisy columns accounted for 7–8% of total rules, compared with 1–2% in the original Adult-dataset experiments.Source: Figure 12 · p. 22, §I · p. 22

J. Qualitative Analysis

The qualitative analysis found that the highest-weight rules intuitively aligned with task descriptions. The appendix also supplies full evaluation and ablation results across 13 datasets, reported in AUC.

J. Qualitative Analysis

  • The highest-weight rules for each dataset and answer class were observed to align intuitively with the task description.Table 16 ranks the top three rules by importance, measured using weights from a trained linear model.Source: J. Qualitative Analysis · p. 23, Table 16 · p. 23

K. Full Results

  • The appendix presents full results omitted from the main manuscript because of space constraints.Source: K. Full Results · p. 24

K.1. Main Evaluation

  • Table 17 reports AUC evaluation results across 13 datasets, with best performances bolded and second-best FeatLLM results underlined.Failed cases are marked as N/A.Source: Table 17 · p. 25

K.2. Ablation Study

  • Table 18 reports AUC results across 13 datasets for the ablation study.Source: Table 18 · p. 26

Related papers

Loading 2404.09491v2…