Source-linked AI summary
Importance Scoring of Transformer Attention Heads in Learning Tabular Data
Ahmad Jad Allah, Kazi F. Akhter, Md. Kamrozzaman Bhuiyan, Manar D. Samad
TL;DR
Tabular transformer architectures lack established guidance for selecting attention heads and layers, limiting evidence about head-level contributions in this setting. This paper evaluates an attention-head importance score across tabular datasets and finds that removing lowest-scoring heads is most robust in 72.5% of examples.
Problem
Tabular transformers lack a standard approach for selecting layers and heads, while head-pruning methods remain insufficiently explored for tabular data.
Method
The paper computes attention-head importance scores from activation representations and uses them to guide soft dynamic tuning and head removal.
Results
72.5% of 160 examples were most robust when heads were gradually removed from lowest to highest importance scores.
Takeaways & Limitations
Head importance scores can identify redundancy and support interpretability and model compression, while important heads are distributed differently across datasets and layers.
Takeaways & Limitations
All experiments use a single tabular transformer architecture, TransTab, limiting conclusions across transformer architectures.
Abstract
from arXiv · showhide
Computationally demanding and opaque deep learning models can be better understood and optimized by analyzing how they transform data. While deep transformers have been widely studied in computer vision and natural language processing, their application in tabular data remains relatively underexplored. This paper presents one of the first applications of an importance-scoring metric to interpret multi-head transformer models in learning from tabular data. Experiments conducted on 40 diverse tabular datasets demonstrate robustness to head drops based on the proposed head importance score. In 72.5\% of experimental examples, the model remains most resilient to performance drops when heads with the lowest importance scores are gradually removed. In contrast, removing the most important attention head first results in the greatest reduction in classification performance. A closer look at individual head importance scores across six attention layers reveals that important heads are scattered across layers, with no consistent layer-specific trends. In contrast to the image and language domains, the importance of individual attention heads varies considerably across tabular datasets with different schemas and feature spaces. The proposed importance score can improve efficiency and redundancy within transformer architectures. We make the source code for measuring the importance of individual attention heads publicly available.
1 Introduction
Transformers have been extensively studied in language and vision but remain comparatively underexplored for tabular data. Although transformer models have shown success on several tabular tasks, performance–computational cost optimization remains insufficiently explored.
- Motivation: Tabular transformer research remains limited compared with the extensive work on language and vision applications.The passage contrasts structured row-and-column data with the domains where transformers were primarily introduced and extensively evaluated.
- Motivation: Transformers have achieved success on tabular classification, transfer learning, and missing-value imputation [11].These applications demonstrate the growing use of transformer models for tabular datasets.
- Research gap: Tabular transformer architectures are often adapted from large language models without optimizing the performance–computational cost trade-off.Weight pruning has been investigated for language-data transformers, but related optimization has not been explored as extensively for tabular learning.
2 Related work
Prior transformer research has focused mainly on language and often uses fixed layer-and-head configurations, motivating head-level importance analysis for tabular data. This paper adapts a loss-sensitivity metric from Michel et al., implements it for tabular learning, and releases the code.
- Motivation: Multi-head attention lets different heads capture diverse input patterns, making individual-head contribution important for understanding and controlling architectural redundancy.The paper positions this analysis against a literature that has primarily investigated multi-head transformers for language data.
- Transformer architecture: Transformer architectures have commonly retained fixed numbers of attention layers and heads across domains, exemplified by six layers with eight heads in the original transformer and 12 layers with 12 heads in BERT.This rigid configuration motivates examining individual-head contributions rather than treating all heads as interchangeable.
- Prior head-importance work: Michel et al. introduced a loss-sensitivity metric for identifying and removing redundant heads from already-trained transformer language models.Their implementation was not released, so this paper reimplements and adapts the metric to tabular learning.
- Paper contribution: The paper adapts head-importance scoring to study redundancy in multi-head transformers learning from tabular datasets with heterogeneous feature spaces.Importance scores dynamically tune head activation during training and are later used to drop heads cumulatively across attention layers.
- Exploratory analysis: The paper also explores dynamically soft-tuning attention-head activation as an exploratory analysis.This mechanism is listed among the paper’s contributions alongside measuring head importance and releasing the adapted implementation.
3 Methodology
The methodology quantifies attention-head importance from gradient-weighted activations and uses those scores to tune continuous head gates during training. Experiments use TransTab, a six-layer, 48-head transformer designed for tabular data.
- Tabular-data setting: The approach targets heterogeneous tabular data, whose rows contain d feature columns with varying types and scales rather than uniform pixels or sequential words.This schema and feature-space diversity motivates importance analysis tailored to tabular transformers.
- Head importance scoring: The head importance score estimates each head’s contribution to training loss as the expected absolute inner product between its activation and loss gradient.The absolute value captures influence magnitude regardless of direction, while minibatch estimates are accumulated and normalized across training samples.
- Activation tuning: Continuous gates scale individual head activations, using normalized importance scores to dynamically regulate heads during training.The method begins with unity gates during an Ew-epoch warmup, then computes dataset-wide scores and updates gates; ϵ = 10^-8 prevents division by zero.
- Model configuration: Experiments use TransTab with six attention layers and eight heads per layer, totaling 48 heads, plus a 256-dimensional ReLU feedforward network with dropout.Feature names and categorical values are represented using embeddings derived from pre-trained BERT.
4 Experiments and evaluations
The experiments evaluate eight head-dropping scenarios, including soft-tuning comparisons, post-training random drops, and cumulative drops ordered by head importance. Models are selected using validation data and evaluated on held-out test data with AUC, using one-vs-rest macro averaging for multiclass datasets.
- Experimental cases: Eight experimental cases compare soft tuning, post-training random head drops, and cumulative head removal ordered from least to most important.The first two cases use no head removal and produce fixed performance scores; the next two randomly drop heads only after training.
- Evaluation protocol: Models use 70:20:10 training, validation, and test splits, with validation selecting the best model and held-out test data providing final evaluation.AUC is the evaluation metric, computed with one-vs-rest macro averaging for multiclass datasets.
5 Results
Across 40 heterogeneous tabular datasets, progressively dropping the least important attention heads was generally most robust, while important-head removal caused the largest AUC degradation. Results also show dataset-dependent effects of activation tuning and no consistent layer concentration of important heads.
- Overall robustness: 72.5% of 160 dataset-and-drop-level examples favored progressively removing heads from lowest to highest importance scores.The evaluation covered 40 datasets at cumulative drop levels of 10, 20, 30, and 40 heads.
- Overall robustness: Dropping the most important heads caused a significantly larger AUC decrease than dropping the least important heads, while random removal showed intermediate resilience.Figure 2 reports mean percentage AUC changes across 40 datasets relative to the no-tuning, no-dropping reference.
- Dataset-dependent effects: Soft activation tuning affected robustness inconsistently: it improved head-drop resilience on Pendigits but degraded final classification performance on some datasets, including eye movements.The contrasting Pendigits and eye-movements cases illustrate that dataset structure can determine whether tuning helps or harms.
- Layer-wise importance: Important attention heads were scattered across all six layers, with no consistent concentration in any particular layer across datasets or experimental cases.This contrasts with hierarchical layer patterns reported for vision models and supports using importance scores for interpretation and compression.
6 Conclusions
The paper develops and evaluates an importance score for interpreting individual attention heads in tabular-data transformer models. Results show substantial dataset-dependent variation in head contributions, without a consistent layer-level ordering or hierarchy.
- The paper develops and evaluates an importance score for interpreting individual attention-head contributions in multi-head transformers learning tabular data.
- Individual attention heads contribute unequally, and their impact differs substantially across tabular datasets.
- Attention layers show no consistent ordering or hierarchy in terms of head-level contributions, while the experiments use a single tabular transformer architecture.