Source-linked AI summary
Visualizing Data using GTSNE
Songting Shi
TL;DR
High-dimensional visualization methods such as t-SNE and UMAP can preserve local neighborhoods while neglecting macro structure. GTSNE modifies t-SNE with macro-structure objectives, and experiments report better preservation of continuous and global structures across the evaluated datasets. It remains limited by slow execution on large datasets and the initial choice of k-means centroids to define macro structure.
Problem
t-SNE and UMAP emphasize local neighborhood relations but can fail to faithfully represent macro structure, especially in high-dimensional data with continuous organization.
Method
GTSNE modifies t-SNE by adding a macro loss based on relative k-means centroid relations, alongside t-SNE and k-means losses.
Results
Across the reported datasets, GTSNE preserves continuous and macro structures better than t-SNE and UMAP, including lines, circles, spheres, and pancreas-map continuity.
Takeaways & Limitations
GTSNE is intended for visualization settings where preserving macro structure is important in addition to local neighborhood relations.
Takeaways & Limitations
GTSNE takes about one and a half hours on MNIST with N = 70,000 and D = 784, and its k-means characterization of macro structure is only an initial choice.
Abstract
from arXiv · showhide
We present a new method GTSNE to visualize high-dimensional data points in the two dimensional map. The technique is a variation of t-SNE that produces better visualizations by capturing both the local neighborhood structure and the macro structure in the data. This is particularly important for high-dimensional data that lie on continuous low-dimensional manifolds. We illustrate the performance of GTSNE on a wide variety of datasets and compare it the state of art methods, including t-SNE and UMAP. The visualizations produced by GTSNE are better than those produced by the other techniques on almost all of the datasets on the macro structure preservation.
1 Introduction
GTSNE extends t-SNE by adding a macro-structure loss to preserve relative k-means centroid structure alongside local neighborhood relations in two-dimensional maps.
- GTSNE adds a macro loss term to t-SNE so low-dimensional maps preserve relative k-means centroid structure.
- Existing t-SNE and UMAP methods emphasize local neighborhood relations but can neglect macro structure, producing unfaithful low-dimensional representations.
- The method targets high-dimensional data whose global organization may not be preserved by local neighborhood matching alone.
2 Methods
GTSNE extends t-SNE by combining local neighborhood matching with explicit macro-structure modeling. Its loss uses t-SNE, centroid-relation, and k-means terms optimized for a two-dimensional embedding.
- Embedding setup: GTSNE maps N high-dimensional points X into two-dimensional points Y.The method defines x_i ∈ R^D and y_i ∈ R^d with d = 2.
- Local structure: t-SNE models neighborhood relations by matching high- and low-dimensional probabilities P and Q.Nearby points receive higher probabilities, and matching P with Q yields the desired low-dimensional layout.
- Macro structure: GTSNE addresses t-SNE’s limited structure preservation by explicitly representing macro structure with high-dimensional k-means centroids and their relations to data points.The construction first applies PCA and then k-means to obtain centroids that capture global structure.
- Macro structure: The macro component transfers centroid structure to the low-dimensional map using centroid definitions and t-distributed centroid relations.Low-dimensional centroids are defined from the embedding, with scaled distances used to relate data points to their cluster centroids.
- Objective: The GTSNE loss combines local t-SNE loss, macro loss, and k-means loss with weights α and β.L(Y) = KL(P||Q) + αKL(P_macro||Q_macro) + βL_k-means; the three terms preserve local neighborhoods, match centroid relations, and constrain map points relative to centroids.
- Optimization: GTSNE optimizes its loss with gradient descent using an adaptive learning rate, while a quadratic tree computes the t-SNE gradient component.The adaptive scheme increases the learning rate in directions where the gradient is stable.
3 Experiments
Experiments compare GTSNE with PCA, t-SNE, and UMAP on simulated, toy, MNIST, and Pancreas datasets. Across these evaluations, GTSNE preserves continuous and macro structures that t-SNE sometimes breaks, while remaining similar to UMAP on Pancreas.
- Experimental setup: Experiments compare GTSNE with PCA, t-SNE, and UMAP on simulated and real datasets using stated algorithm parameters.The evaluation includes simulation data, five toy datasets, MNIST, and the Pancreas single-cell RNA-seq dataset.
- Simulation data: On the three-line simulation with N = 2100 and D = 3, t-SNE breaks the lines, whereas GTSNE and UMAP preserve their continuity.The experiment is designed to test whether methods preserve continuous structures in high-dimensional data.
- Interpretation: GTSNE’s macro loss strengthens attraction between neighboring points when centroid probabilities would otherwise mismatch, helping preserve line continuity.The paper attributes t-SNE’s breaks to a local minimum where attractive and repulsive forces balance.
- Limitations: The simulation also reveals that GTSNE twists the lines in the low-dimensional map, an issue left for future work.This is identified as an aspect requiring improvement.
- Five toy datasets: GTSNE preserves the continuous circle structure on Swiss Roll and the sphere shape on Sphere better than t-SNE and UMAP.These results come from experiments on five established toy datasets.
- MNIST dataset: On MNIST, GTSNE produces a representation comparable to those of t-SNE and UMAP.The MNIST dataset contains 70,000 examples represented as 28 × 28 pixel images.
- Pancreas dataset: On the Pancreas dataset with 3696 cells and 2000 selected velocity genes, GTSNE behaves similarly to t-SNE and UMAP and produces a continuous map like UMAP.The t-SNE result contains breaks in the continuous structure.
4 Discussion
GTSNE combines k-means-based macro-structure modeling with t-SNE-style visualization, but its large-dataset runtime and macro-structure definition remain limitations.
- GTSNE uses k-means on PCA embeddings to capture macro structure and preserves centroid relations probabilistically in the low-dimensional space.
- About 1.5 hours are required for MNIST with N = 70,000 and D = 784, indicating slow runtime on large datasets.
- The paper identifies defining macro structure as an open question because k-means centroids are only an initial characterization.
5 Conclusion
GTSNE modifies t-SNE by incorporating macro structure into its loss function so low-dimensional maps preserve that structure.
- GTSNE is a modified version of t-SNE that includes macro structure in the loss function.
- The method is intended for data visualizations that need to preserve macro structure.
Appendix B. Derivation of the GTSNE gradient.
The GTSNE gradient is derived from a loss function with local, macro-structure, and k-means components, then combined through substitution.
- The GTSNE loss function consists of local, macro-structure, and k-means terms.
- The macro-structure term is represented as αKL(P_macro||Q_macro), while the k-means term is weighted by β.
- The gradient is obtained by substituting the three component gradients into the combined loss-function derivative.
- The appendix concludes the GTSNE gradient derivation after this substitution.