PaperlayerView exact revision on arXiv

Source-linked AI summary

Hierarchical and Permutation-Invariant Feature Transformation Learning via Policy-Guided Embedding Search

Rui Liu, Tao Zhe, Yanyong Huang, Sankha Narayan Guria, Xiao Luo, Wei Fan, Yanjie Fu, Dongjie Wang

arXiv:2609.10225v1cs.LGcs.AI

TL;DR

Automated feature transformation must represent hierarchical feature-operation relationships, respect permutation invariance, and search non-convex transformation spaces effectively. PHER addresses these issues with hierarchical permutation-invariant modeling and policy-guided multi-objective search. Across 19 datasets, it outperforms baseline models, while its generated features improve downstream performance by 23.7% in a traceability case study.

  • Problem

    Existing feature-transformation methods inadequately model intricate transformation patterns, while gradient-based search can become trapped in local optima.

  • Method

    PHER combines token- and concept-level hierarchical encoder-decoder modeling with self-attention pooling and policy-guided multi-objective search initialized from top-performing seeds.

  • Results

    PHER outperforms baseline models on all 19 evaluated datasets and consistently outperforms baselines across tested downstream models.

  • Takeaways & Limitations

    Approximately 70% of the crucial features in a Wine Quality Red generated feature set come from PHER, whose new feature space improves downstream performance by 23.7%.

Abstract

from arXiv · show

Feature transformation improves predictive performance on tabular data by constructing informative abstractions from raw features. Recent generative approaches encode transformation knowledge into continuous embedding spaces for efficient exploration of candidate strategies, but face three key limitations: (1) overlooking hierarchical relationships between low-level features, operations, and high-level abstractions; (2) enforcing order-sensitive embeddings on inherently permutation-invariant transformation sequences, thereby introducing systematic bias; and (3) relying on gradient-based search, which is ill-suited to non-convex transformation spaces. We propose a framework with two complementary components. First, a permutation-invariant hierarchical module captures interactions across features, operations, and abstraction levels, with a self-attention pooling mechanism that maps semantically equivalent structures to consistent embeddings aligned with downstream performance. Second, a policy-guided multi-objective reinforcement learning strategy initializes the search from empirically strong seeds and jointly optimizes predictive accuracy and transformation efficiency. Extensive experiments on diverse tabular benchmarks demonstrate the effectiveness and robustness of our framework against strong baselines. Our code and data are publicly available at: https://github.com/RayLiu1103/PHER.

1 Introduction

Automated feature transformation seeks informative representations for difficult, heterogeneous tabular data, but existing approaches inadequately model transformation knowledge and recent embedding methods face structural and search limitations. PHER addresses these limitations through permutation-invariant hierarchical modeling and policy-guided search.

  • Automated feature transformation derives more predictive representations from original tabular features, whose heterogeneous types, scales, missing values, and outliers challenge deep learning.
  • Existing methods use expansion-reduction, evolution-evaluation, or AutoML-based strategies, yet still struggle to model intricate feature-transformation patterns.
  • Recent generative methods encode transformation knowledge in continuous spaces and search them with gradients, but overlook hierarchies among features, operations, and generated abstractions.
  • Order-sensitive embeddings can bias inherently permutation-invariant transformation structures, while non-convex embedding spaces increase the risk of local optima and suboptimal sequences.
  • PHER combines permutation-invariant hierarchical modeling with policy-guided multi-objective search to preserve low- and high-level knowledge and explore transformation sequences.

2 Problem Statement

PHER models feature transformations hierarchically, from feature-operation tokens to generated concepts, within a global embedding space. It then searches this space to recover an optimal transformation sequence for downstream performance.

  • Problem Formulation: PHER collects transformation sequences with their downstream predictive performance and models them in a continuous global embedding space.Each record contains a transformation sequence and associated predictive performance.
  • Hierarchical Modeling: The token-level encoder represents interactions between original features and mathematical operations as token embeddings.A token-level decoder reconstructs the transformation sequence from those embeddings.
  • Hierarchical Modeling: Token embeddings are grouped and averaged into concept embeddings, extending the representation from low-level operations to generated abstractions.The concept-level representation is intended to capture patterns from generated concepts.
  • Embedding Search: A policy-guided strategy explores the concept-level embedding space to identify an optimal global embedding.The selected embedding is decoded into an optimal feature transformation sequence.

3 Methodology

PHER combines hierarchical, permutation-invariant representation learning with policy-guided multi-objective search to discover effective feature transformations. Its training proceeds from token-level patterns through concept-level abstractions and alignment before searching the learned embedding space.

  • Method overview: PHER comprises permutation-invariant hierarchical modeling and policy-guided multi-objective search.The framework is organized around token, concept, and search components.
  • Hierarchical modeling: Token-level encoders and decoders model feature-operation sequences and reconstruct the original transformation sequence.The encoder maps sequences to token embeddings, while the decoder reconstructs the sequence from those embeddings.
  • Hierarchical optimization: A three-stage procedure trains token representations, concept abstractions, and token-concept alignment progressively.The final alignment stage tunes token components against fixed concept-level abstractions to improve end-to-end reconstruction coherence.
  • Hierarchical modeling: Concept-level training aggregates token embeddings into global embeddings using relationships among concepts, feature indices, and mathematical operations.The concept encoder and decoder learn high-level permutation-invariant abstractions after token-level parameters are frozen.
  • Policy-guided search: PPO searches the global embedding space from top-K performance-ranked seeds while balancing downstream performance against transformation length.Its weighted reward combines performance improvement with a normalized sequence-length penalty, and the resulting embedding is decoded into candidate transformations.

4.1 Datasets and Evaluation Metrics

PHER is evaluated on 19 public datasets spanning classification and regression, using Random Forest, cross-validation, hold-out testing, and repeated runs. Table 1 compares overall performance, while Figure 3 examines the contributions of three PHER components.

  • Datasets: 19 public datasets comprise 14 classification tasks and 5 regression tasks from UCI, LibSVM, Kaggle, and OpenML.The evaluation uses Random Forest as the unified downstream model.
  • Evaluation protocol: Each dataset is split into 80% training and 20% testing, with five-fold cross-validation on the training partition.Experiments follow a hold-out evaluation protocol and are independently repeated five times with different random seeds.
  • Overall comparison: Table 1 presents overall performance comparisons, marking the best and second-best results and treating higher values as better.The table reports PHER and baseline performance across the evaluation tasks.
  • Evaluation metrics: Classification uses F1-Score and regression uses 1-RAE as the primary evaluation metrics.The reported standard deviation is computed from five independent runs.

4.2 Baseline Models

PHER is compared with nine automated feature-transformation methods spanning random generation, expansion-reduction, dimensionality reduction, iterative selection, reinforcement learning, graph exploration, and embedding-based search.

  • Baseline models: The baseline set includes RDG, ERG, LDA, AFAT, NFS, TTG, GRFG, DIFER, and MOAT.These methods represent several established strategies for generating or selecting transformed features.
  • Baseline models: ERG expands the feature space with operations before selecting informative features, whereas LDA learns latent topic representations for dimensionality reduction.These baselines represent expansion-reduction and generative dimensionality-reduction approaches.
  • Baseline models: NFS, TTG, and GRFG use reinforcement learning or graph-based exploration to discover transformation paths or generate features.Their designs include sequential control, graph exploration, and cascaded feature grouping.
  • Baseline models: DIFER and MOAT embed transformation sequences and search the embedding space with greedy or gradient-ascent strategies.MOAT additionally uses postfix expressions and beam search.

4.3 Performance Evaluation

PHER consistently outperforms baseline methods across 19 datasets and remains robust across downstream models. Its search also produces shorter transformation sequences while retaining strong task performance, with results supporting the roles of its components and seed initialization.

  • Overall Performance: PHER outperforms baseline models on all 19 datasets using F1-score for classification and 1-RAE for regression.Results were independently obtained five times with different random seeds.
  • Ablation Study: Each PHER component contributes to performance across classification and regression tasks evaluated from multiple metrics.Variants remove hierarchical encoding, permutation invariance, or policy-guided search, and the experiment reports the significance of each component.
  • Overall Performance: PHER consistently outperforms baseline methods across Random Forest, XGBoost, K-Nearest Neighborhood, and Decision Tree models on SpamBase F1-score.The evaluation assesses robustness across distinct downstream classifiers.
  • Search Seeds: High-quality search seeds consistently outperform random initialization, whereas random starting points increase local-optimum risk and search instability.The comparison averages F1-score over five runs on eight randomly chosen datasets.
  • Scalability: PHER produces shorter transformation sequences and outperforms MOAT across all datasets, jointly targeting task performance and transformation length.The scalability comparison evaluates both sequence length and downstream task performance.

4.3.5 Model Scalability.

The experiments examine PHER’s permutation robustness, hyperparameter behavior, and traceability of generated features. Results show order-invariant embeddings, stable performance across settings, and strong contribution from generated features.

  • Permutation Sensitivity: Permuted concept orders consistently cluster around the original global embedding, confirming that the hierarchical encoder-decoder eliminates permutation bias.Self-attention pooling uses content-based weights and commutative weighted summation, so reordering concepts yields an identical global embedding.
  • Traceability Case Study: Approximately 70% of crucial features in the new feature set are generated by PHER, while the newly generated feature space improves downstream ML performance by 23.7%.The comparison ranks the top 10 most significant features in the original and PHER-generated feature sets for Wine Quality Red.
  • Traceability Case Study: The traceability experiment compares feature importance between the original feature set and the selected feature subset.Figure 9 visualizes the comparison using bars, where larger bars indicate higher feature importance.

5 Related Works

Related work groups automated feature transformation into expansion-reduction, evolution-evaluation, and AutoML-based approaches. These methods have achieved success but face computational, optimization, and feature-transformation modeling challenges.

  • Existing Approaches: Expansion-reduction methods enlarge the feature space with mathematical transformations and then reduce dimensionality through feature selection.
  • Existing Approaches: AutoML-based approaches formulate feature transformation as a search problem for identifying optimal transformations, but existing methods struggle to model intricate transformation patterns.
  • Existing Approaches: Evolution-evaluation methods iteratively explore transformation operators using evolutionary algorithms or reinforcement learning while retaining features that improve validation performance.Their flexibility is accompanied by high computational cost and unstable optimization from discrete decisions and combinatorial search spaces.

6 Conclusion Remarks

PHER combines hierarchical modeling, permutation-invariant self-attention pooling, and multi-objective policy-guided search for automated feature transformation. The framework aims to preserve hierarchical knowledge, reduce order sensitivity, and improve exploration of the learned embedding space.

  • Framework: PHER preserves transformation knowledge at both feature-operation token level and generated concept level in a global embedding space.
  • Framework: Self-attention pooling symmetrically computes attention scores across generated concepts to ensure permutation invariance.
  • Framework: Multi-objective search explores the learned embedding space without relying on convexity assumptions, mitigating the risk of local optima.
  • Findings: Experiments indicate that hierarchical modeling captures meaningful interactions, while permutation invariance stabilizes embedding-space learning and search.

GenAI Usage Disclosure

The authors state that generative AI tools were used for language polishing, grammar refinement, and auxiliary code assistance. Research ideas, experiments, analyses, data processing, and final manuscript content were created, verified, and approved by the authors.

  • Disclosure: Generative AI assistance was limited to language refinement and auxiliary code-editing or debugging suggestions.
  • Disclosure: The authors created, verified, and approved the research ideas, experimental designs, analyses, data processing, code, and final manuscript.
Loading 2609.10225v1…