PaperlayerView exact revision on arXiv

Source-linked AI summary

KnowFeat: Knowledge-Guided Feature Engineering via LLM Agents

Chengsong You, Wangyue Li, Weiqiao Que, Qizhou Chen, Kunyan Wu, Wei Deng, Feng Zhu, Xiaofeng He

arXiv:2609.03529v1cs.DB

TL;DR

LLM-based feature engineering lacks structured domain knowledge and feature-level provenance. KnowFeat injects five types of domain knowledge, verifies candidate features, and ranks first among seven methods across twelve public datasets under leakage-free evaluation.

  • Problem

    LLM-based methods lack systematic injection of structured specialized-domain knowledge and provenance linking each feature to its supporting knowledge assets.

    Source: §I · p. 1

  • Method

    KnowFeat injects five types of domain knowledge into an LLM agent, verifies candidates through code, statistical, and model-effectiveness checks, and records feature provenance.

    Source: §I · pp. 1–2

  • Results

    Average rank 2.3 among seven methods across twelve public datasets, with a significant one-sided Wilcoxon test (p=0.017); verified features remain competitive with strong baselines on both AML datasets.

    Source: §I · p. 2

  • Takeaways & Limitations

    Provenance cards trace accepted features from knowledge assets through code to verification outcomes, and practitioner tests cut compliance review time from hours to minutes.

    Source: §V · p. 10

    KnowFeat adds the most value when baseline features leave headroom and the domain contains knowledge the LLM does not already encode; it is unnecessary on compact datasets like blood transfusion.

    Source: §V · p. 10

Abstract

from arXiv · show

Automated feature engineering with large language models (LLMs) can produce semantically meaningful features for tabular data, yet existing methods lack structured domain knowledge, rigorous verification, and explainable provenance. We propose KnowFeat, a knowledge-guided feature engineering framework that organizes domain knowledge into five types -- schema metadata, regulatory indicators, detection rules, expert opinions, and court document evidence -- and injects them as structured context into an LLM agent. A three-stage verification pipeline filters candidates through code execution, statistical quality checks, and model effectiveness evaluation. Every accepted feature carries a provenance record tracing its design to specific knowledge assets. Under a strict held-out protocol that eliminates feature-selection leakage, KnowFeat ranks first (avg. rank 2.3) across twelve public benchmarks among seven methods (one-sided Wilcoxon p=0.017), with a peak gain of +11.6 pp AUC on a telecom churn dataset. On a real-world Bitcoin anti-money laundering (AML) dataset (Elliptic) and a synthetic digital currency AML benchmark (SimECNY), KnowFeat maintains competitive detection performance with full provenance traceability.

I. INTRODUCTION

Existing LLM feature-engineering methods lack structured domain knowledge, rigorous verification, and feature-level provenance. KnowFeat addresses these gaps with knowledge-guided generation, staged checks, and traceable records, and ranks first among seven methods on public datasets under leakage-free evaluation.

I. INTRODUCTION

  • Traditional automated feature engineering searches predefined transformations without understanding feature meaning, while LLM-based methods also lack structured knowledge, sufficient verification, and provenance.In regulated domains, stakeholders need to know what a feature represents before deployment.Source: §I · p. 1
  • KnowFeat organizes domain knowledge into five types and injects it as structured context into the LLM agent.The types are schema metadata, risk indicators, detection rules, expert opinions, and court document evidence.Source: §I · p. 1
  • Each candidate passes code execution, statistical quality checks, and model-effectiveness evaluation, with expensive evaluation reserved for candidates that survive earlier stages.The model-effectiveness stage uses XGBoost across five seeds.Source: §I · pp. 1–2
  • KnowFeat ranks first (avg. rank 2.3) among seven methods under leakage-free evaluation, with a significant one-sided Wilcoxon test (p=0.017).Its largest gain is +11.6 pp AUC on a telecom churn dataset.Source: §I · p. 2
  • On both AML datasets, KnowFeat’s verified features remain competitive with strong baselines, while FeatLLM’s 1,124-feature set lowers F1 on SimECNY.The evaluations use a real-world Bitcoin AML dataset and a synthetic AML benchmark.Source: §I · p. 2
  • Each accepted feature carries provenance linking it to specific knowledge assets, supporting audit in regulated domains.The record includes the reasoning chain, computation steps, and verification outputs.Source: §I · p. 2

A. Traditional Automated Feature Engineering

The section contrasts syntactic feature-generation approaches with semantic LLM methods, identifies structured domain knowledge as an untapped resource, and introduces KnowFeat’s knowledge-guided framework.

A. Traditional Automated Feature Engineering

  • Traditional AutoFE applies predefined mathematical operators without semantic understanding, limiting its ability to create domain-specific features such as transaction velocity.Examples include Deep Feature Synthesis, ExploreKit, AutoFeat, and OpenFE.Source: A. Traditional Automated Feature… · p. 2

B. LLM-Based Feature Engineering

  • LLM methods generate semantic features, but none combines structured knowledge injection, multi-stage verification, and feature-level provenance tracking.The methods vary in their use of knowledge and validation; KnowFeat is described as the first to combine all three capabilities.Source: B. LLM-Based Feature Engineering · p. 2

C. Domain Knowledge Integration in Machine Learning

  • Regulatory, expert, and institutional AML knowledge includes risk indicators, transaction patterns, and detection rules, but automated feature engineering has not tapped this structured knowledge.Knowledge-infused learning and transfer learning target unstructured data rather than tabular features.Source: C. Domain Knowledge Integration in… · p. 2

III. METHODOLOGY

  • KnowFeat’s methodology comprises knowledge injection, an LLM agent with specialized tools, three-stage verification, intelligent search, and provenance output.These components are presented as the framework’s main methodological stages.Source: §III · p. 2

A. Framework Overview

  • Given a dataset and domain knowledge base, KnowFeat generates features to improve a target metric and returns verified features with provenance cards.The design includes noise-calibrated thresholds, candidate forward selection, SHAP-guided budget reallocation, and cross-dimension propagation.Source: A. Framework Overview · pp. 2–3, Figure 1 · p. 3

B. Multi-Type Knowledge Injection

KnowFeat injects five increasingly specific knowledge types into feature generation and equips its LLM agent with tools to explore data, register candidates with provenance, and respond to verification feedback.

B. Multi-Type Knowledge Injection

  • KnowFeat organizes domain knowledge into five types, from schema metadata to court-document evidence, for systematic injection into feature generation.The AML resources include 38 risk indicators, 15 detection rules, 12 expert opinions, and risk factors extracted from 309 court judgments.Source: B. Multi-Type Knowledge Injection · p. 3

C. LLM Agent with Specialized Tools

  • The LLM agent explores data in a sandbox and registers each candidate with code, an explanation, a reasoning chain, knowledge-asset identifiers, computation steps, and source columns.The agent must trigger verification immediately after each registration; batch submission is prohibited.Source: C. LLM Agent with Specialized Tools · p. 4
  • The agent selects relevant knowledge, explores data, proposes a feature grounded in knowledge items, then uses verification feedback to guide its next steps.The search procedure also uses candidate-pool forward selection and SHAP-guided budget reallocation across dimensions.Source: C. LLM Agent with Specialized Tools · p. 4, B. Multi-Type Knowledge Injection · p. 4

D. Three-Stage Verification Pipeline

The verification pipeline progressively screens candidate features through executable-code checks, statistical quality filters, and model-effectiveness evaluation.

D. Three-Stage Verification Pipeline

  • Each candidate passes through three progressively stricter verification stages, with earlier stages designed to be cheaper than later evaluation.The supplied pipeline description names L1 code execution, L2 statistical checks, and L3 model-effectiveness evaluation.Source: D. Three-Stage Verification Pipeline · p. 4

1) L1: Code Execution Sandbox:

  • L1 executes generated Python in an isolated sandbox and rejects candidates with code, shape, identifier-column, or numeric-output failures.A failed candidate receives up to two automatic repair attempts before permanent rejection.Source: 1) L1: Code Execution Sandbox · p. 4

2) L2: Statistical Quality Filtering:

  • L2 compares IV and KS against noise-calibrated thresholds, while also enforcing correlation below 0.7 and a missing rate no greater than 30%.The thresholds are calibrated from 100 random features using the 95th percentile by default; redundancy and missing-rate thresholds are fixed.Source: 2) L2: Statistical Quality Filtering · p. 4

3) L3: Model Effectiveness:

KnowFeat evaluates whether candidate features improve prediction, with task-specific acceptance criteria and progressively more costly verification. Its search strategy allocates exploration across semantic dimensions and reuses information from prior rounds.

3) L3: Model Effectiveness:

  • Candidates are tested with XGBoost using stratified 80/20 train-test splits across five seeds, then accepted according to task-specific performance criteria.Public classification requires ΔAUC > 0; AML requires ΔRecall > 0 and Precision ≥90%.Source: 3) L3: Model Effectiveness · pp. 4–5
  • The three verification stages rise from milliseconds to seconds to minutes per feature, so early filtering avoids costly evaluation of defective candidates.The passage describes L1 as cheap, L2 as moderate, and L3 as expensive.Source: 3) L3: Model Effectiveness · p. 5

E. Intelligent Search Strategy

  • The search divides the feature space into four AML dimensions: Amount & Frequency, Time & Geography, Behavior Sequence, and Entity Association.Each dimension starts with a default budget of three candidates per round.Source: E. Intelligent Search Strategy · p. 5
  • After each round, mean absolute SHAP values guide inverse-SHAP budget reallocation, steering exploration toward underrepresented dimensions.Each dimension initially receives candidates, and the minimum per-dimension budget defaults to one.Source: E. Intelligent Search Strategy · p. 5
  • The agent receives the ten most recent failures for its current dimension, including rejection reasons, to avoid regenerating rejected ideas and guide novel proposals.The default number of failures supplied is K = 10.Source: E. Intelligent Search Strategy · p. 5
  • Greedy forward selection tests L2-passed but L3-failed candidates in quality-score order, retaining additions that improve the target metric.This step is intended to capture features that contribute positively in combination.Source: E. Intelligent Search Strategy · p. 5
  • The top five features by SHAP importance are shared across dimensions as inspiration prompts for adapting successful patterns.The default cross-dimension sharing count is Kcross = 5.Source: E. Intelligent Search Strategy · p. 5

F. Explainable Provenance Output

KnowFeat records provenance for each generated feature, connecting its design and verification to domain knowledge. The experiments are organized around public-dataset performance, AML detection, knowledge contributions, and feature quality and generalization.

F. Explainable Provenance Output

  • Each generated feature receives a provenance card that records the knowledge assets, reasoning chain, computation steps, and verification results behind it.Verification records include L1/L2/L3 outcomes such as IV, KS, redundancy, and ΔAUC/ΔRecall.Source: F. Explainable Provenance Output · p. 5
  • In regulated domains, provenance lets compliance officers trace why a feature exists, what evidence supports it, and how it was verified.The paper points to concrete provenance-card examples in Section IV-C.Source: F. Explainable Provenance Output · p. 5

IV. EXPERIMENTS

  • The experiments ask whether KnowFeat improves public-dataset performance, how it performs on imbalanced synthetic AML data, what domain knowledge contributes, and whether verified features generalize.The fourth question also examines the balance between feature quantity and quality.Source: §IV · p. 5

A. Experimental Setup

KnowFeat is evaluated under a strict held-out protocol across public tabular benchmarks and AML datasets. On public benchmarks, it ranks first among seven methods, with results also examining baseline strength, model-family sensitivity, and provenance.

A. Experimental Setup

  • The eight original public datasets receive three knowledge types, while four newer datasets receive schema metadata only; the verification pipeline alone improves or maintains baseline performance.The newer datasets are default credit, churn, kc2, and jm1.Source: A. Experimental Setup · p. 6
  • Evaluation covers twelve public benchmarks and two AML datasets, with feature engineering confined to the development split and final performance measured on a frozen 20% test set.Baselines use the same leakage-free protocol; public-dataset results are averaged over five model seeds.Source: A. Experimental Setup · pp. 5–6

B. RQ1: Public Dataset Performance

  • KnowFeat ranks first among seven methods (average rank 2.3) across 12 public datasets; significance remains after removing churn (p = 0.034).The one-sided Wilcoxon test gives p = 0.017 across all 12 datasets.Source: B. RQ1: Public Dataset Performance · p. 6, Figure 2 · p. 6, Table III · p. 7
  • Gains are larger with moderate baselines: churn rises 11.6 pp (0.737→0.853) and jm1 1.3 pp (0.720→0.732), while saturated benchmarks plateau within 1 pp.Blood transfusion is an exception: its four original features already form a complete RFM representation, and five of six methods degrade performance there.Source: B. RQ1: Public Dataset Performance · p. 6
  • OpenFE, AutoFeat, and LLM-FE share second-best average rank (3.8), but KnowFeat is more consistent, placing in the top three on 9 of 12 datasets.No competitor places in the top three on more than six datasets.Source: B. RQ1: Public Dataset Performance · p. 6
  • Unlike other methods in Table III, KnowFeat’s features include an auditable chain from knowledge asset to code to verification outcome.The paper identifies provenance as a hard requirement in regulated domains.Source: B. RQ1: Public Dataset Performance · p. 7
  • KnowFeat improves 9–11 of 12 datasets across XGBoost, LightGBM, and CatBoost; after Holm correction, only XGBoost (p=0.040) and LightGBM (p=0.042) reach significance.CatBoost trends positive (p=0.212) but does not reach significance.Source: B. RQ1: Public Dataset Performance · p. 7, Table IV · p. 7
  • On heart and credit g, replacing DeepSeek-V4-Flash with DeepSeek-V4-Pro gives mixed results, and a full backbone comparison remains future work.Pro improves heart by +0.002 and degrades credit g by −0.017; the authors suggest the verification pipeline matters more than raw backbone capability.Source: B. RQ1: Public Dataset Performance · p. 7

C. RQ2: AML Dataset Performance

KnowFeat is tested on synthetic SimECNY and real-world Elliptic AML data. Its verified features maintain competitive detection performance, while provenance cards receive positive practitioner feedback.

C. RQ2: AML Dataset Performance

  • On SimECNY, KnowFeat’s four verified features maintain baseline F1 (0.971), while FeatLLM’s 1,124 features lower F1 to 0.968.The 93-feature baseline already has near-perfect AUC (0.9997), leaving minimal headroom; OpenFE’s generated features also lower F1 to 0.969.Source: C. RQ2: AML Dataset Performance · p. 7, Table VI · p. 8
  • On Elliptic, KnowFeat’s three knowledge-guided features achieve AUC 0.9982 versus the 0.9981 baseline, with full provenance traceability.The dataset contains 46,564 labeled transactions and 166 features.Source: C. RQ2: AML Dataset Performance · p. 7
  • Domain experts reported that provenance cards reduced feature review time from 2–3 hours to 15–30 minutes per feature.The cards link features to regulatory indicators, detection rules, and expert opinions.Source: C. RQ2: AML Dataset Performance · p. 7
  • The four accepted SimECNY features span four semantic dimensions, and entity-association features have the highest mean absolute SHAP values.The dimensions are Amount & Frequency, Time & Geography, Behavior Sequence, and Entity Association.Source: C. RQ2: AML Dataset Performance · pp. 7–8
  • Two representative features encode knowledge combinations: fan-in/fan-out asymmetry with abnormal-mode weighting, and co-occurring border-region and off-hour signals.The second feature scores zero unless both geographic and temporal conditions co-occur; the provenance cards link design decisions to knowledge assets.Source: C. RQ2: AML Dataset Performance · p. 8, Table V · p. 8

D. RQ3: Ablation Study

The ablations compare structured-knowledge configurations and show that individual knowledge types can perform strongly, while results vary across datasets and combinations. On heart, the full-knowledge variant exceeds partial configurations on the frozen test set, although schema metadata alone approaches its AUC without manual curation.

D. RQ3: Ablation Study

  • On heart, full knowledge (0.928) exceeds schema-only (0.920) by 0.8 pp, while adult performance is nearly identical without curated domain knowledge (0.931 vs. 0.932).On SimECNY, both variants maintain near-perfect baseline F1, but full knowledge yields features across all four semantic dimensions.Source: D. RQ3: Ablation Study · p. 9
  • On breast w, rules alone reach .998 versus .986 No-FE; on heart, expert opinions alone reach .935.Individual knowledge types frequently match or exceed full-knowledge AUC, amid run-to-run variability; the authors identify verification, not knowledge completeness, as the primary quality mechanism.Source: D. RQ3: Ablation Study · p. 9
  • Schema metadata alone reaches 99.4% of full-knowledge AUC on heart with zero manual curation, but the cumulative ablation is non-monotonic.The authors attribute the non-monotonic pattern to interactions between knowledge types and stochastic LLM generation.Source: D. RQ3: Ablation Study · p. 9
  • On the frozen heart test set, the full-knowledge variant (0.928) exceeds all partial configurations.The cumulative ablation reports AUCs of 0.922, 0.920, 0.925, and 0.920 for schema, +indicators, +rules, and +expert, respectively.Source: D. RQ3: Ablation Study · p. 9

E. RQ4: Feature Quantity, Quality, and Generalization

KnowFeat’s verified features perform competitively across benchmark settings, while feature counts alone do not predict gains. The paper also reports the pipeline’s costs, run-to-run stability, and scope boundaries.

E. RQ4: Feature Quantity, Quality, and Generalization

  • KnowFeat uses a moderate number of verified features, whereas FeatLLM generates the most features among completed public-baseline runs yet has a worse average rank.The comparison supports the authors’ point that more features without verification can hurt.Source: E. RQ4: Feature Quantity, Quality, and… · p. 9
  • On SimECNY, KnowFeat’s verified features maintain baseline F1 while FeatLLM’s 1,124 unverified features decrease F1 by −0.3%.The authors caution that the systems also differ in generators, so the comparison does not isolate verification as the sole factor.Source: E. RQ4: Feature Quantity, Quality, and… · p. 9
  • On weak baselines, KnowFeat raises credit g AUC from 0.633 to 0.754 (+12.1 percentage points) and wine quality AUC from 0.825 to 0.896 (+7.1 points), matching full-baseline performance on wine quality with only 3 core features.Source: E. RQ4: Feature Quantity, Quality, and… · p. 9

F. Computational Cost

  • On SimECNY, L3 model evaluation accounts for ∼41% of pipeline cost and LLM reasoning ∼23%; total LLM API cost is approximately $2.80.Progressive filtering rejects candidates before expensive L3 evaluation; inference runs only generated Python code, adding <50 ms per 200K transaction batch with no LLM calls.Source: F. Computational Cost · p. 10, Table XI · p. 10

G. LLM Generation Stability

  • Across three independent runs, heart AUC has std = 0.002 and credit g std = 0.016 despite near-zero feature-name overlap (Jaccard ≤0.04).Credit g runs produce entirely disjoint feature sets; the authors attribute consistent downstream performance to the verification pipeline.Source: G. LLM Generation Stability · p. 10, Table XII · p. 10

V. CONCLUSION

  • Across twelve public benchmarks KnowFeat achieves the best average rank (2.3); on SimECNY four verified features maintain baseline F1 while FeatLLM’s 1,124 unverified features degrade it.Cross-model evaluation reports significant gains for LightGBM (pHolm=0.042) and XGBoost (pHolm=0.040).Source: §V · p. 10
  • Each accepted feature has provenance from knowledge asset through code to verification outcome, and practitioner tests report compliance review time falling from hours to minutes.The authors also identify ongoing regulatory changes and domain calibration of L3 acceptance thresholds as practical considerations.Source: §V · p. 10
  • KnowFeat adds most value when baseline features leave headroom and the domain contains knowledge the LLM does not encode; it is unnecessary on compact blood-transfusion data with 4 RFM features.The conclusion also reports that knowledge curation takes ∼2 person-days for a full AML taxonomy, versus 30–60 minutes per public dataset with LLM assistance.Source: §V · p. 10

APPENDIX

The appendix illustrates SimECNY knowledge assets, reports progressive candidate filtering, identifies the scope of a bootstrap-interval table, and lists practical considerations for deployment.

A. Domain Knowledge Examples

  • The SimECNY examples span schema metadata, a risk indicator, a detection rule, expert opinion, and court evidence; the full set includes 38 indicators, 15 rules, 12 opinions, and 309 court documents.The schema example includes amount, wallet type, hour, source and destination wallet IDs, and transaction mode.Source: A. Domain Knowledge Examples · p. 11

B. Verification Statistics

  • On SimECNY, 52 candidates yield 47 L1 passes (90%), 31 L2 passes (60%), and 18 L3 passes (35%); post-hoc selection retains 4 features.Progressive filtering saves ∼53 minutes.Source: B. Verification Statistics · p. 11

C. Bootstrap Confidence Intervals

  • Table XIII reports frozen-test AUC with 95% bootstrap confidence intervals based on 1,000 resamples.The supplied passage states the table’s scope but not its cell values.Source: Table XIII · p. 11

D. Practical Considerations

  • Practical considerations include ∼2 person-days to curate the full AML taxonomy, domain calibration of L3 acceptance thresholds, and ongoing feature maintenance as regulations evolve.LLM assistance reduces curation time to 30–60 minutes per public dataset.Source: D. Practical Considerations · p. 11
Loading 2609.03529v1…