PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Learning Multidimensional Projections

Mateus Espadoto, Nina S. T. Hirata, Alexandru C. Telea

arXiv:1902.07958v1cs.LGstat.ML

TL;DR

Large datasets make popular cluster-separating projections such as t-SNE costly, while stability and out-of-sample projection remain difficult. The paper trains a neural network to mimic a chosen projection from sampled data and applies it to unseen data. Across evaluated datasets, the learned projections show similar visual cluster separation to t-SNE and UMAP, with roughly 12% lower NH values.

  • Problem

    Popular cluster-separating projections such as t-SNE can be costly on large datasets and lack stability and out-of-sample support.

    Source: §1 · p. 1

  • Method

    A neural network learns a chosen projection from sampled data and its corresponding projections, then projects unseen data from the same or similar universes.

    Source: §1 · p. 2, §3 · p. 3

  • Results

    Roughly 12% lower NH values accompany visual cluster separation similar to t-SNE and UMAP across six challenging datasets.

    Source: §5 · pp. 10, 12

  • Takeaways & Limitations

    The method offers a trade-off between projection quality and speed, ease of use, genericity, generalizability, and stability.

    Source: §6 · p. 13

    The method’s inherent stability and out-of-sample support trade off against cluster separation, producing fuzzier clusters than corresponding t-SNE projections.

    Source: §5 · p. 12

Abstract

from arXiv · show

Dimensionality reduction methods, also known as projections, are frequently used for exploring multidimensional data in machine learning, data science, and information visualization. Among these, t-SNE and its variants have become very popular for their ability to visually separate distinct data clusters. However, such methods are computationally expensive for large datasets, suffer from stability problems, and cannot directly handle out-of-sample data. We propose a learning approach to construct such projections. We train a deep neural network based on a collection of samples from a given data universe, and their corresponding projections, and next use the network to infer projections of data from the same, or similar, universes. Our approach generates projections with similar characteristics as the learned ones, is computationally two to three orders of magnitude faster than SNE-class methods, has no complex-to-set user parameters, handles out-of-sample data in a stable manner, and can be used to learn any projection technique. We demonstrate our proposal on several real-world high dimensional datasets from machine learning.

1 Introduction

High-dimensional data is difficult to explore because datasets are large, while projections scale well and are widely used for visualizing data with many dimensions. The paper proposes learning projections with a neural network to address limitations of t-SNE and related techniques.

1 Introduction

  • Projections scale well in sample count and dimensionality, making them a tool of choice when data has tens to hundreds of dimensions.They are used when individual dimension identity matters less.Source: §1 · p. 1
  • t-SNE visually separates similar-sample clusters, but can be slow on large datasets, difficult to tune, and unable to project out-of-sample data.These limitations complicate comparisons across time-dependent datasets.Source: §1 · p. 1
  • Existing t-SNE-class techniques do not jointly address scalability, stability, and out-of-sample handling, motivating a technique-independent approach.Algorithmic complexity may also limit the attractiveness of t-SNE refinements.Source: §1 · p. 1
  • The proposed approach trains a deep neural network on a subset’s projections, then uses it to project the full dataset or similar datasets.It aims to mimic the chosen projection’s visual cluster separation, with linear-time computation, no complex parameters, and stable out-of-sample support.Source: §1 · pp. 1–2

2 Related Work

Related work spans linear methods, manifold-learning methods, SNE-class projections, and neural approaches, with differing trade-offs across quality, scalability, tuning, and out-of-sample support.

2 Related Work

  • PCA is scalable, easy to use, predictable, and supports out-of-sample projection, but its linear and global nature limits quality on high-intrinsic-dimensional data.It is commonly used as preprocessing before a more specific visualization method.Source: §2 · p. 2
  • Manifold-learning methods capture nonlinear data structure and generally improve on PCA’s quality, but are harder to tune, scale poorly, and lack out-of-sample capability.Examples include MDS, Isomap, and LLE.Source: §2 · p. 2
  • t-SNE is known for visually segregating similar samples, but has O(N^2) sample-count complexity, sensitivity to data changes, difficult tuning, and no out-of-sample capability.Its refinements still largely suffer from sensitivity, tuning, and out-of-sample problems.Source: §2 · p. 2
  • UMAP is described as offering t-SNE-comparable quality at higher speed and supports out-of-sample projection, but remains susceptible to data changes and tuning difficulty.Autoencoders provide parallelizability, predictability, and out-of-sample capability, but typically have PCA-comparable quality and can be difficult to architect for a dataset family.Source: §2 · p. 3
  • ReNDA uses two networks and provides predictability and out-of-sample capability, but requires labeled data and has no reported scalability results.The study focuses mainly on SNE-class methods while also considering other methods successful on particular criteria.Source: §2 · p. 3

3 Method

The method learns a projection’s mapping from high-dimensional samples to their projected coordinates, then applies the learned network to unseen samples. It uses a chosen projection technique to create training targets and evaluates the network against that technique’s projections.

3 Method

  • The approach assumes a data universe has structure that a projection can capture, and hypothesizes that this behavior can be learned from a limited number of training datasets and projections.The proposal selects Ds from one or more datasets and trains Pnn to mimic P(Ds).Source: §3 · p. 3
  • The pipeline projects a sampled subset Ds with a chosen technique P, trains a neural network Pnn on the samples and their projections, then infers projections for the remaining data Dp.The trained network can also project different datasets from the same universe.Source: §3 · p. 3, Figure 1 · p. 3
  • Pnn uses three fully connected hidden layers of 256, 512, and 256 units, followed by a two-element sigmoid output scaled to [0,1]^2.Training uses Adam and mean squared error for up to 200 epochs with early stopping; convergence took no more than 60 epochs in practice, averaging 30.Source: §3 · pp. 3–4
  • Evaluation compares Pnn’s projections of unseen Dp with ground-truth projections P(Dp), using visual class separation and neighborhood hits; labels are not used in training or ground-truth generation.The quantitative comparison uses neighborhood hits as a projection-quality metric.Source: §3 · p. 4

4 Results

Across the evaluated datasets and projection techniques, the method learns to reproduce projection structure, with stable, fast projections and transfer possible after fine-tuning on a small sample from a related data universe.

4 Results

  • The evaluation compared the method with t-SNE, UMAP, PCA, Isomap, and LLE across real-world datasets including MNIST, Fashion MNIST, Cats and Dogs, IMDB, CIFAR-10, and CIFAR-100.The datasets span image, text, and extracted-image-feature data; experiment-specific training and test splits vary.Source: §4 · p. 4

4.1 Training effort

  • For two-class MNIST, projections nearly matched t-SNE and UMAP with 1K training samples; for ten classes, similar results began at roughly 5K.With a fixed 3K-sample training set, projection quality improved as epochs increased, and training and validation loss converged quickly; Table 1 relates training-set size to epochs needed for convergence.Source: §4.1 · pp. 4–5, Table 1 · p. 5, Figure 2 · p. 5

4.2 Capturing the structure of different datasets

  • On four datasets, the method trained on t-SNE or UMAP captured their visual structure and separated clusters better than canonical PCA, Isomap, and autoencoders.The comparison used 15K-observation subsets and 5K training samples.Source: §4.2 · pp. 5–6, Figure 4 · p. 7

4.3 Stability and out-of-sample data

  • As MNIST samples were added, the learned projections kept clusters in the same places but became noisier; t-SNE and UMAP moved clusters to different places.The method was trained on 5K observations from the respective projection technique.Source: §4.3 · p. 6, Figure 5 · p. 8

4.4 Learning different projections

  • With 5K training samples, the method closely reproduced patterns from PCA, Isomap, MDS, and LLE on different samples from MNIST and Fashion MNIST.The authors clarify that reproducing these projection styles does not imply the projections are optimal.Source: §4.4 · p. 9, Figure 6 · p. 10

4.5 Computational scalability

  • 2–3 orders of magnitude faster than t-SNE and about 2 orders faster than UMAP in inference; even against UMAP inference, the method is 1–2 orders faster.The end-to-end method is faster than t-SNE from about 5K samples and UMAP from about 30K samples; UMAP inference has a break-even point around 65K samples.Source: §4.5 · p. 9, Figure 7 · p. 10

4.6 Projecting unrelated data

  • After fine-tuning with many epochs, the network trained on CIFAR-10 projections produced results similar to ground truth on related CIFAR-100 data with as few as 100 samples.Direct reuse of the network and training from scratch with few CIFAR-100 samples performed poorly; the experiment used 100–700 epochs for fine-tuning.Source: §4.6 · pp. 9–10

5 Discussion

The discussion reports faster inference and similar cluster-separation patterns to t-SNE and UMAP, alongside lower NH values and trade-offs involving stability, projection sharpness, and training-set coverage.

5 Discussion

  • Across six datasets, projections show cluster separation similar to t-SNE and UMAP, despite roughly 12% lower NH, and outperform PCA, Isomap, and autoencoders on difficult datasets.The discussion reports better visual cluster separation and higher NH than those alternatives on difficult datasets.Source: §5 · pp. 10, 12
  • More than two and three orders of magnitude faster than t-SNE and UMAP, respectively, for repeated inference; projection complexity is linear in observations and dimensions.The authors note that this is the typical case of training once and inferring many times.Source: §5 · p. 12
  • The method can learn any projection technique from samples and their projected coordinates, without projection-specific training or inference settings.The authors demonstrate learning t-SNE, UMAP, MDS, Isomap, LLE, and PCA.Source: §5 · p. 12
  • Stable out-of-sample projections depend on training samples representing the data universe; deterministic inference maps the same items to the same locations.Small input changes also yield only small projection changes with the fully connected network, but this stability trades off against cluster sharpness.Source: §5 · p. 12
  • Generalization weakens as data move farther from the training set, while a few hundred target-universe samples can support fine-tuning a pre-trained network.For larger data universes, the training set needs to be larger and better spread.Source: §5 · p. 13

6 Conclusion

The method learns to mimic a user-chosen projection from a small subset, then uses the network to project unseen data from the same universe. It offers similar quality and visual style to established techniques alongside speed, stability, and out-of-sample support, while future work targets more complex data and transfer learning.

6 Conclusion

  • A neural network trained on projections of a small sample mimics the chosen technique and infers projections for unseen data from the same universe.Source: §6 · p. 13
  • The method mimics the quality and visual style of established projections, including SNE-class cluster separation, while being orders of magnitude faster.It has one training parameter with a documented preset and no inference parameters, and provides stability and out-of-sample support.Source: §6 · p. 13
  • The authors describe good trade-offs between quality and speed, ease of use, genericity, generalizability, and stability.Source: §6 · p. 13
  • Future work will examine dynamic and mixed-type data, alternative networks and training procedures, and transfer learning across data universes and projection techniques.Source: §6 · p. 13

Related papers

Loading 1902.07958v1…