PaperlayerView exact revision on arXiv

Source-linked AI summary

Assessing feature scorer results on high-dimensional datasets with t-SNE

Bruno Iochins Grisci, Mario Inostroza-Ponta, Márcio Dorn

arXiv:2610.39009v1cs.LGcs.AI

TL;DR

Few tools directly visualize feature scorers, and no single metric guarantees their results. Weighted t-SNE incorporates feature importance into a 2D embedding; experiments report clearer clusters for scorers with higher classification performance, alongside direct visual comparison. These visualizations can help users inspect and choose scorers for a dataset and task.

  • Problem

    Few visualization approaches examine feature-scorer results, and no single metric guarantees scorer quality for choosing among methods.

    Source: §1 · p. 2, §2.1 · p. 7

  • Method

    Weighted t-SNE modifies relationships between embedded data points so feature importance affects their positions in the 2D projection.

    Source: §1 · p. 2

  • Results

    Feature scorers with higher classification performance tended to produce clearer weighted t-SNE clusters, as supported by silhouette analysis.

    Source: §5 · p. 28

  • Takeaways & Limitations

    Weighted t-SNE enables direct visual comparison of scorers and reveals differences in data structure that raw importance values can make difficult to discern.

    Source: §5 · p. 28

    Weighted t-SNE depends on labeled data because most feature scorers need a target variable to obtain importance scores.

    Source: §4.2 · p. 27

Abstract

from arXiv · show

While vast literature on high-dimensional data visualization is available, there are not many works regarding the visualization of feature scorers and their results. Feature scorers are algorithms that assign numerical importance to each feature of multi-dimensional datasets. These importance scores can be used in several applications, such as feature selection, knowledge discovery, and machine learning interpretability. There are several feature scorers to choose from, and often no single metric or ground truth is available to guarantee the quality of their results. In this scenario, visualization can become valuable to support the decision of which method to choose and how good its results are. For this goal, this work presents "weighted t-SNE." It modifies the relationship between data points in the embedded 2D space to reflect the importance of each dimension of the original datasets as assessed by a feature scorer. This research discusses how to implement weighted t-SNE, proposes the silhouette coefficient as a numerical evaluation of the results, and shows several examples of its use in practice. Synthetic and real-world tabular datasets are used in the experiments together with nine feature scorers, ranging from Mutual Information to neural networks. Each feature scorer produces unique visualizations, and weighted t-SNE can be used to compare and choose the one that better suits a given dataset and task. Weighted t-SNE can also visually show the importance of features learned by machine learning models and help us see how they are organizing the data, increasing their interpretability.

1. Introduction

High-dimensional datasets can contain many features, and feature scores can guide analysis, but their quality is difficult to judge without a single guaranteeing metric. This work introduces weighted t-SNE to visualize how feature importance affects data projections.

1. Introduction

  • Feature scores support feature selection, knowledge discovery, and model interpretability, but their importance depends on the specific task and data.The same feature's importance should not be assumed to hold when its context changes.Source: §1 · p. 2
  • Because no single metric guarantees feature-scorer quality, visualization can support method selection and reveal score sets that may harm downstream analysis.t-SNE projections are already used for exploratory visual analysis, though they assume the projection reflects actual clusters.Source: §1 · p. 2
  • Feature-scoring effects have received little visualization attention; most related work addresses feature selection or interactive selection instead.The paper presents an updated method with broader experiments and analysis than a preliminary study focused on relevance aggregation.Source: §1 · pp. 2–3
  • Weighted t-SNE modifies relationships between data points so feature importance influences their positions in the 2D projection.The method is intended to compare scorers and inspect patterns learned by machine-learning models.Source: §1 · p. 2
  • A synthetic regression example contrasts mixed samples in regular t-SNE with a neural-network-weighted projection that disentangles samples by target sign.The dataset consists mostly of irrelevant features; negative target values are blue and positive values red.Source: Figure 1 · p. 3

Statement of significance

Traditional t-SNE does not account for feature relevance, while existing visual tools focus largely on feature selection or model decisions. Weighted t-SNE incorporates feature importance into dimensionality reduction to inspect how scorers structure data.

Statement of significance

  • Weighted t-SNE integrates feature-importance scores into dimensionality reduction, enabling direct inspection of how different scorers structure data.The paper identifies this as a visualization approach for examining feature scorers.Source: Statement of significance · pp. 3–4
  • Accuracy and regression metrics do not reveal how well a scorer identifies relevant features, and manually inspecting thousands of scores is infeasible.Different models can have identical predictive accuracy while assigning different importance scores.Source: Statement of significance · p. 4
  • Traditional t-SNE does not incorporate feature importance, and feature-selection visualizations focus on subsets rather than importance distributions across all dimensions.Interpretability tools such as SHAP and LIME explain model decisions but do not directly visualize feature scorers across a dataset.Source: Statement of significance · p. 4
  • The method is presented as useful for visual assessment of scorers in labeled datasets and for revealing outliers that irrelevant features might obscure.The stated potential applications also include examining learned importance and supporting scorer choice for downstream tasks.Source: Statement of significance · p. 4

2. Related work

The related work distinguishes feature scoring from selection, reviews visualization and t-SNE foundations, and describes silhouette and trustworthiness as projection-evaluation measures.

2.1. Feature scoring

  • Feature scoring assigns importance to features without itself reducing dimensionality, whereas feature selection uses scores or other criteria to choose a subset.Scores can support ranking and selection, and feature selection commonly removes features below a threshold.Source: §2.1 · p. 5
  • Filters rank features using data characteristics, wrappers evaluate feature subsets with a classifier or regressor, and embedded scorers calculate importance within the learning algorithm.Filters are efficient but may miss feature interactions; wrappers can overfit and cost more computation, while embedded methods avoid repeated classifier training.Source: §2.1 · pp. 5–6
  • Gene selection motivates scorer inspection because genomic data can contain noisy or redundant values, and determining feature relevance remains challenging.The discussion notes that analytics tools for comparing and inspecting scorers are needed in this setting.Source: §2.1 · pp. 6–7

2.2. Visualization methods

  • PCA preserves global structure but can lose local structure and nonlinear patterns, while RadViz becomes less clear as represented dimensions increase.Feature-selection tools and table heatmaps address related but distinct tasks, and heatmaps are infeasible for large datasets.Source: §2.2 · p. 7

2.3. t-distributed stochastic neighbor embedding

  • t-SNE embeds high-dimensional data by optimizing low-dimensional similarities to match original-space similarities, using a Student t-distribution to reduce crowding.The embedding emphasizes local neighborhoods, while global structure may be lost; perplexity controls neighborhood scale, and this work uses max(30, m/100).Source: §2.3 · pp. 8–9

2.4. Silhouette coefficient

  • The silhouette coefficient averages point-level cluster separation, using class labels to approximate how distinctly classes appear in a projection.Its interpretation should consider the original high-dimensional space, since the embedding is not expected to separate classes lacking clear original boundaries.Source: §2.4 · pp. 9–10

2.5. Trustworthiness

  • Trustworthiness measures the proportion of local neighbors preserved between high-dimensional data and its projection, with values ranging from 0 to 1.The paper sets k = 7 and uses the measure to assess the quality of 2D projections.Source: §2.5 · p. 10

3. Weighted t-SNE

Weighted t-SNE adapts t-SNE to visualize feature-scoring results by weighting dimensions in the distance calculation. It provides a global, comparative view of how scores relate to data clusters, rather than selecting features or evaluating individual features.

3. Weighted t-SNE

  • Considering all features with non-zero scores helps distinguish scorers that concentrate importance on a few features from those that spread it more widely.This also avoids setting a hard threshold for deciding which scores are large enough to count as relevant.Source: §3 · p. 10
  • Feature weights modify distances between data points so more important dimensions have greater influence on the resulting projection.Weights are scaled from zero for irrelevant features to one for the most important feature.Source: §3 · pp. 10–11
  • Feature scaling can occur before embedding optimization, making the method’s additional computing cost negligible according to the paper.Weighted t-SNE retains regular t-SNE’s challenges with hyperparameter tuning.Source: §3 · p. 11
  • Weighted t-SNE provides a global view of how scorer-assigned feature importance relates to dataset clusters, not an individual-feature visualization or feature-selection method.It visualizes scores rather than selecting a subset of features.Source: §3 · p. 11
  • A weighted projection should be compared with a projection of the original data; alone, it is described as meaningless.The comparison lets users perceive or measure the difference between the two projections.Source: §3 · p. 11
  • Weighted t-SNE can be combined with feature-importance visualization tools; a prior study used it with table heatmaps to analyze neural networks across datasets.That study also showed distinct importance scores learned for different data classes.Source: §3 · p. 11
  • Weighted t-SNE indirectly evaluates feature scorers through a visual approximation of data separation, so it is not a definitive measure of scoring quality.The paper recommends combining it with other evaluation approaches to balance their strengths and drawbacks.Source: §3 · p. 12

4. Experiments and discussion

This section examines weighted t-SNE as a visualization aid for comparing feature scorers and interpreting feature importance in tabular neural networks across six datasets and nine scorers. It also considers its dependence on trustworthy feature scores, baseline comparisons, and cautious interpretation.

4. Experiments and discussion

  • On XOR, ReliefF, Random Forest, and Neural Network captured the correct four-cluster structure, while the unweighted visualization did not distinguish the classes.Random Forest assigned the two relevant features scores of 0.99 and 1.0, while all others scored below 0.2.Source: §4 · p. 14, Figure 2 · p. 15
  • On the 5,000-feature synthetic dataset, Kruskal–Wallis Filter and ReliefF showed apparent class separation, unlike the visualization using perfect feature selection.Perfect selection weighted the 50 relevant features one and the rest zero; the contrast illustrates that retaining feature-importance weights can reveal structure that feature selection does not.Source: §4 · p. 14, Figure 3 · p. 16, Figure 6 · p. 18
  • Across four datasets and nine feature scorers, better weighted-t-SNE silhouette coefficients accompanied higher kNN test F1 on a 33% split; mouse-cortex Lasso, random-forest, and neural-network models with larger F1 also had larger 2D silhouettes.Source: Table 3 · p. 19, §4 · p. 20, §4.1 · p. 21
  • For liver cancer, weighted plots distinguished scorers that split the classes and exposed a tumor-labeled sample among healthy samples; most prostate cancer plots did not split classes.The liver outlier was sample GSM557108, labeled as tumor; the authors note it may have been mislabeled or contaminated by neighboring healthy tissue.Source: Figure 4 · p. 17, Figure 5 · p. 18, §4 · p. 19

4.1. Machine learning interpretability

  • Weighted t-SNE visualized learned feature importance: the neural network emphasized three of four truly relevant regression features, while XOR plots helped distinguish suitable nonlinear models from linear SVM.The three largest regression importance scores were 0.44, 0.40, and 0.25; the fourth relevant feature scored 0.11. On the mouse-cortex dataset, models with better performance also produced clearer embedding splits.Source: §4.1 · p. 21, Figure 7 · p. 22
  • Across neural-network training epochs, improving prediction was accompanied by increasing weighted t-SNE silhouette coefficients.The paper reports an inverse correlation between training loss and the corresponding embedding silhouette coefficient.Source: §4.1 · p. 21, Table 5 · p. 22, Figure 9 · p. 23
  • Two random forests with identical training and test F1-scores produced different visual structures: one followed binary labels, while the other revealed the merged class’s two subgroups.Both achieved training F1 = 0.99 and test F1 = 0.93; which model is preferable depends on whether preserving the merged labeling or exposing the hidden subgroups better suits the task.Source: §4.1 · pp. 25–26, Figure 11 · p. 25
  • Weighted t-SNE can complement training loss and other measures when inspecting neural networks on tabular data, including through relevance aggregation.The authors note challenges with selectivity in fully connected networks, transformer attention restricted to categorical features [89], and defining neighborhoods for LIME on tabular data.Source: §4.1 · p. 26

4.2. Limitations

  • Weighted t-SNE requires precomputed feature scores and labeled data, inherits t-SNE’s hyperparameter sensitivity, but adds little computational cost.Unreliable scores may misrepresent structure; comparing scorers can expose misleading rankings, while unsupervised feature scoring remains a future direction.Source: §4.2 · pp. 26–27
  • Weighted t-SNE does not directly optimize accuracy; it complements predictive metrics by visually revealing feature-use differences between similarly performing models.A future hybrid evaluation could combine accuracy-based metrics and visualization.Source: §4.2 · p. 27
  • Silhouette scores approximate visual separability but favor convex clusters, so numerical cluster metrics require caution when datasets lack clear clusters.The authors suggest exploring Davies–Bouldin or Dunn indices and note that absent separation may indicate ineffective feature relevance capture.Source: §4.2 · p. 27
  • Weighted t-SNE outputs are most informative when compared with unweighted t-SNE, which reveals how feature scores change the data structure.A single weighted projection without this baseline provides limited insight.Source: §4.2 · p. 27
  • Experiments cover tabular datasets only; adapting weighted t-SNE to images or text remains challenging because feature importance is defined differently for unstructured data.Source: §4.2 · p. 27

5. Conclusion

Weighted t-SNE is presented as a way to inspect feature scorers and compare how they structure data. Experiments report clearer clusters for scorers with higher classification performance, alongside uses in outlier detection, regression, and model-learning analysis.

5. Conclusion

  • Across nine scorers, two synthetic datasets, and two cancer microarrays, higher classification performance tended to align with clearer weighted t-SNE clusters, supported by silhouette analysis.The conclusion also reports that visual scorer selection identified methods that improved an independently trained kNN classifier’s F1-score.Source: §5 · p. 28
  • Weighted t-SNE enabled direct visual comparison of scorers, revealing differences in data structure that raw importance values made difficult to discern.The authors report that the visualizations were trustworthy relative to the high-dimensional data.Source: §5 · p. 28
  • Weighted t-SNE made outliers easier to identify when irrelevant features otherwise obscured samples in high-dimensional space.In regression, it also reflected the importance of continuous target variables.Source: §5 · p. 28
  • The method tracked changes in feature representations across neural-network epochs and showed that equally accurate random forests can assign importance to different features.The latter distinction is not captured by accuracy-based evaluation alone.Source: §5 · p. 28
  • Future work could test joint t-SNE or UMAP, extend the method to deep features with specialized scorers, and explore unsupervised learning and interactive comparison.The deep-feature direction would require adaptations for abstract, context-dependent features and selection of a model and specialized feature scorers.Source: §5 · p. 28
  • The authors position weighted t-SNE as an interpretability tool for high-dimensional tabular data, including gene-expression datasets.They describe its uses as detecting outliers, uncovering structure, tracking model learning, and relating models to domain knowledge.Source: §5 · p. 28

CRediT authorship contribution statement

The contribution statement assigns writing, visualization, software, and methodological work across the three authors, with distinct roles in supervision and funding.

CRediT authorship contribution statement

A.1. Feature scorers

The appendix provides further detail on the feature scorers used in the main-text experiments and reproduces their summary table.

A.1. Feature scorers

  • The appendix expands on how each feature scorer used in the main-text experiments computes feature importance.Source: §A.1 · p. 37
  • Table A.1 reproduces the main text’s summary of the feature scorers.Source: §A.1 · p. 37
  • The listed definitions and equations draw mainly on Barbieri et al., Grisci et al., Montavon et al., and Molnar.Source: §A.1 · p. 37

A.1.1. Kruskal Wallis H test

The study uses the Kruskal-Wallis H test, a non-parametric alternative to ANOVA, to score features according to how well they distinguish target classes.

A.1.1. Kruskal Wallis H test

  • The Kruskal-Wallis H test is used instead of ANOVA and does not require normally distributed data.The passage describes ANOVA as evaluating variation within and across groups to assess whether variables come from the same distribution.Source: §A.1.1 · p. 37
  • For each feature, the test groups its values by target class, and features with higher scores are selected.The stated purpose is to assign numerical importance to features that distinguish the target class most effectively.Source: §A.1.1 · p. 37
  • Because each feature is evaluated independently against the target class, this scorer is univariate and does not account for feature dependencies.Source: §A.1.1 · p. 37

A.1.2. Mutual Information

Mutual Information (MI) measures information shared between variables and ranks features by their relevance to the target class. This is a univariate approach that does not account for dependencies between features.

A.1.2. Mutual Information

  • Entropy quantifies uncertainty associated with a random variable in predicting the likelihood of an event.The text refers to its mathematical expression as Equation A.2.Source: §A.1.2 · p. 37
  • Mutual Information (MI) measures the information one random variable provides about another.Its formal definition is given in Equation A.3.Source: §A.1.2 · p. 38
  • MI assesses feature relevance by measuring information shared with the target class, then ranks features individually without accounting for dependencies between them.Source: §A.1.2 · p. 38

A.1.3. mRMR

mRMR is an iterative feature-scoring algorithm that selects features for high mutual information with the target class and low redundancy with features already selected.

A.1.3. mRMR

  • mRMR iteratively prioritizes features that maximize mutual information with the target class.Source: §A.1.3 · p. 38
  • mRMR also minimizes redundancy with respect to features already included in the selected subset.Source: §A.1.3 · p. 38
  • At each iteration, mRMR selects the feature optimizing its criterion, defined using the attribute set X, selected set S, mutual-information function I, and redundancy function R.Source: §A.1.3 · p. 38

A.1.4. ReliefF

ReliefF scores features by comparing each instance with its nearest neighbors across classes, accounting for feature correlations through hit and miss differences.

A.1.4. ReliefF

  • ReliefF finds k-nearest neighbors from each class for every instance and evaluates attribute variation across these neighbors, accounting for feature correlations.Source: §A.1.4 · p. 38
  • ReliefF increases a feature's weight when it differs between the target instance and a miss, and decreases it when it differs from a hit.Source: §A.1.4 · p. 38
  • Misses are weighted by their class probability so hits and misses contribute equally, and the algorithm returns a feature-relevance weight vector.Source: §A.1.4 · p. 38

A.1.5. Lasso

Lasso is a linear regression technique with l1-regularization; each feature's importance is its coefficient in the final regressor.

A.1.5. Lasso

  • Lasso incorporates l1-regularization into linear regression to enhance prediction accuracy and mitigate overfitting.Source: §A.1.5 · p. 39
  • Its formulation is an optimization problem in which y is the outcome, X the independent variables, N the instance count, β the unknown parameters, and λ the regularization term.Source: §A.1.5 · p. 39
  • A feature's importance is its coefficient in the final regressor.Source: §A.1.5 · p. 39

A.1.6. Decision Tree

The section explains how decision trees, Random Forests, linear SVMs, and neural networks derive feature-importance scores. Neural-network scoring includes relevance aggregation for tabular data, with class-wise scores combined into a global ranking.

A.1.6. Decision Tree

  • Decision-tree feature importance is the normalized total reduction in the impurity criterion achieved by a feature.Tree splits evaluate attributes using an impurity function such as the Gini index.Source: §A.1.6 · p. 39

A.1.7. Random Forest

  • Random Forest feature weights are averaged across trees, using Gini importance to rank features.The forest comprises trees trained on bootstrapped instance subsets, whose predictions are combined by majority vote.Source: §A.1.7 · p. 39

A.1.8. Support Vector Machine

  • With a linear kernel, SVM feature importance is the coefficient of that feature in the final model.The classifier uses soft-margin hinge loss for non-separable data and Tikhonov regularization to improve generalization.Source: §A.1.8 · pp. 39–40

A.1.9. Neural Networks

  • For tabular neural networks, relevance aggregation uses absolute, sample-rescaled input relevances, averages them by class, then equally across classes for global scores.The work also uses LRP-αβ and the w2-rule, which sends network output backward through the model to calculate relevance.Source: §A.1.9 · p. 40

Related papers

Loading 2610.39009v1…