Source-linked AI summary
Superheat: An R package for creating beautiful and extendable heatmaps for visualizing complex data
Rebecca L Barter, Bin Yu
TL;DR
Traditional visualization tools often do not scale well to high-dimensional datasets, motivating improved heatmap-based visualization. The paper proposes superheat, an extendable heatmap approach, and reports that heatmaps can support deeper information extraction and exploration. The package addresses flexibility limitations in R heatmap tools.
Problem
Traditional visualization tools often do not scale well to high-dimensional datasets, limiting their usefulness for extracting information from large multivariate data.
Method
The paper proposes superheat, a superheatmap that augments traditional heatmaps to provide additional avenues for information extraction and data exploration.
Results
The authors demonstrate that heatmaps are useful visualization tools, particularly for high-dimensional data.
Takeaways & Limitations
Augmented heatmaps can support information extraction and exploration beyond the traditional heatmap display.
Takeaways & Limitations
R's built-in heatmap function offers little flexibility and is difficult to use for producing visually appealing heatmaps, while ggplot2 requires data conversion.
Abstract
from arXiv · showhide
The technological advancements of the modern era have enabled the collection of huge amounts of data in science and beyond. Extracting useful information from such massive datasets is an ongoing challenge as traditional data visualization tools typically do not scale well in high-dimensional settings. An existing visualization technique that is particularly well suited to visualizing large datasets is the heatmap. Although heatmaps are extremely popular in fields such as bioinformatics for visualizing large gene expression datasets, they remain a severely underutilized visualization tool in modern data analysis. In this paper we introduce superheat, a new R package that provides an extremely flexible and customizable platform for visualizing large datasets using extendable heatmaps. Superheat enhances the traditional heatmap by providing a platform to visualize a wide range of data types simultaneously, adding to the heatmap a response variable as a scatterplot, model results as boxplots, correlation information as barplots, text information, and more. Superheat allows the user to explore their data to greater depths and to take advantage of the heterogeneity present in the data to inform analysis decisions. The goal of this paper is two-fold: (1) to demonstrate the potential of the heatmap as a default visualization method for a wide range of data types using reproducible examples, and (2) to highlight the customizability and ease of implementation of the superheat package in R for creating beautiful and extendable heatmaps. The capabilities and fundamental applicability of the superheat package will be explored via three case studies, each based on publicly available data sources and accompanied by a file outlining the step-by-step analytic pipeline (with code).
1 Introduction
High-dimensional datasets create a visualization challenge because common graphical techniques scale poorly beyond two dimensions, while heatmaps can represent large matrices compactly. The paper introduces superheat to extend heatmaps for multivariate exploration and demonstrate their broader applicability.
- Visualization challenges: Traditional scatterplots, boxplots, and histograms rarely extend satisfactorily into higher dimensions.Three-dimensional extensions can also be inadequately represented when compressed into two dimensions.
- Visualization challenges: Many visualization techniques become incomprehensible with too many data points or variables, creating a scalability tradeoff between simplicity and complexity.Overly complex graphs impede comprehension, whereas overly simple graphs conceal important information.
- Heatmaps: Although widely used in bioinformatics, heatmaps remain significantly underemployed in other domains.The paper positions heatmaps as particularly well suited to high-dimensional multivariate data.
- Heatmaps: Heatmaps represent matrix entries by color and enable visual processing of datasets with thousands of rows and columns.Clustering can group similar observations to reveal structure and increase visualization clarity.
- Superheat: Superheat is an R package for supervised heatmaps that incorporates additional information into traditional heatmaps.The paper aims to demonstrate heatmaps as a default visualization method for high-dimensional data and highlight superheat’s customizability and ease of implementation.
2 Superheat
Superheat builds an intuitive, ggplot2-based heatmap function that retains simple implementation while substantially expanding customization. Its extensions combine heatmaps with annotations and adjacent plots to support deeper exploration of heterogeneous data.
- Existing tools: Existing R heatmap tools include heatmap, ggplot2, pheatmap, and aheatmap, but their workflows or flexibility are limited.The inbuilt heatmap function is difficult to use and offers little flexibility, while ggplot2 requires conversion to a long-form three-column data frame.
- Package design: Superheat builds on ggplot2 infrastructure to provide an intuitive heatmap function with ggplot2 aesthetics and simple implementation.The package is designed to retain the ease of inbuilt heatmap functions while improving customization.
- Customization: Superheat smooths heatmaps within clusters to support extremely large matrices with thousands of dimensions.This feature is presented as part of the package’s expanded customization capabilities.
- Customization: Users can overlay heatmaps with text or numbers for clearer data presentation and annotation.These overlays add information directly to the heatmap display.
- Customization: Scatterplots, barplots, boxplots, and line plots can be placed beside heatmap rows and columns to add a new information layer.The package supports combining multiple data representations in one visualization.
- Applications: These features let users explore data more deeply and use its heterogeneity to inform analysis decisions.The paper illustrates the package through three case studies involving combined data sources, correlational structure, and model-performance heterogeneity.
3 Case study I: combining data sources to explore global
The first case study combines worldwide organ-donation data with HDI to examine temporal, geographic, and developmental patterns. A superheatmap summarizes these dimensions simultaneously and reveals changing donation rates and a general association with HDI ranking.
- Data sources and aim: The case study combines organ-donation data with HDI to examine worldwide trends and their relationship.Organ-donation data came from the WHO-ONT Global Observatory, while HDI is based on life expectancy, education, and per-capita indicators.
- Visualization design: The visualization combines a country-year heatmap, an overall donation line plot, country-level HDI bars, and regional color coding.The line plot aggregates donations across all 58 represented countries, while the bars display HDI rankings.
- Visualization design: The superheatmap encodes deceased-donor organ donations per 100,000 people by country and year, with white cells indicating missing values.Countries are ordered by average transplants per 100,000 across years.
- Observed patterns: Spain leads global organ donation, while Croatia rises from a low 2006 rate to a rate equaling Spain’s in 2014.Donation rates appear to slow in Germany, Slovakia, and Cuba, and Iceland has no recorded deceased-donor donations in 2007.
- Observed patterns: Countries with lower (better) HDI rankings generally have higher organ-donation rates, with Japan, Singapore, and Korea as exceptions.The countries with the most donations are predominantly European and American.
- Interpretive value: Superheat displays donation trends over time, country, and region alongside HDI without mass overplotting.The authors describe the result as an in-depth yet uncluttered summary of the data’s trends.
Word2Vec
Word2Vec represents words as high-dimensional vectors whose relative distances convey semantic meaning. Superheat visualizes contextual similarity and clustered structure across word vectors, including large collections summarized by cluster aggregation.
- Word2Vec: Word2Vec embeds words in high-dimensional spaces where relative distances convey semantic meaning.The paper illustrates this with the approximate vector relationship man − king + woman = queen.
- Word2Vec: Google’s pre-trained word vectors were trained on part of a roughly 100-billion-word Google News corpus and contain 300 dimensions.The vectors cover 3 million words and phrases.
- Visualizing cosine similarity: Superheat visualizes contextual similarity through cosine-similarity heatmaps ordered by hierarchical clustering and accompanied by dendrograms.For 35 common NY Times headline words, the heatmap reveals high pairwise similarity among conflict, legal, and political terms.
- Visualizing word clusters: Cosine-silhouette width measures cluster quality using within-cluster dissimilarity and dissimilarity to the nearest other cluster.The analysis also evaluates stability using average pairwise Jaccard similarity across membership vectors from 90% subsamples.
- Visualizing word clusters: The clustered heatmap reveals both tight and overlapping semantic groups: “American” is tight, whereas “murder” is similar to several topic clusters.Compared with a standard silhouette plot, the visualization shows which clusters have low between-cluster separability and what they resemble.
5 Case study III: evaluation of heterogeneity in the per-
The third case study uses supervised heatmaps to examine heterogeneity in voxel responses and model performance for visual stimuli. It identifies sensitive and neutral voxel groups and relates predictability to response patterns.
- 5.1 Modeling brain activity: The study analyzes 1,294 voxel responses to 1,750 images, fitting independent Lasso models and evaluating them on 120 withheld validation images.The image features comprise 10,921 Gabor wavelet features.
- 5.1 Modeling brain activity: K-means clustering separates images and voxels into two groups, with sensitive voxels responding differently across image clusters and neutral voxels responding similarly.Sensitive voxels differ from average responses across the two image groups, whereas neutral voxels show similar responses.
- 5.2 Simultaneous performance evaluation of all 1,294 voxel-models: The top image cluster contains less easily identifiable content, while the bottom cluster contains more readily identifiable subjects; responses are stronger for the top cluster.Examples include rocks or abstract paintings in the top cluster and recognizable people, birds, or insects in the bottom cluster.
- 5.2 Simultaneous performance evaluation of all 1,294 voxel-models: Sensitive-voxel models perform better than neutral-voxel models: average predicted-to-true response correlation exceeds 0.5 versus close to zero.The correlations are shown above heatmap columns as scatterplots or cluster-aggregated boxplots.
- 5.2 Simultaneous performance evaluation of all 1,294 voxel-models: Sensitive voxels tend to occupy the V1 region’s center, whereas neutral voxels are primarily located on its periphery.The heatmap reveals this spatial pattern while jointly displaying voxel responses and model performance.
- 5.2 Simultaneous performance evaluation of all 1,294 voxel-models: Random Forest models produced the same results as Lasso models, with overall correlation approximately 0.05 higher on average.The comparison concerns prediction of voxel responses.
6 Implementation of supervised heatmaps
The superheat R package implements supervised heatmaps as a stand-alone, highly customizable function. Users can control clustering, colors, labels, grid lines, and adjacent plots.
- Package implementation: The authors’ R package automatically generates customizable supervised heatmaps through a stand-alone function built using ggplot2.The package does not use the ggplot2 grammar of graphics.
- Reproducibility: The paper provides example code and step-by-step analytic pipelines for reproducing the case-study superheatmaps.These materials are available in the appendices, supplementary materials, and project webpages.
- Clustering customization: Users may provide a clustering membership vector or specify the number of clusters, with K-means as the default algorithm.The authors recommend selecting the number of clusters before implementing supervised heatmaps using methods such as Silhouette analysis.
- Aesthetic customization: Color scales, adjacent plots, row and column labels, and grid lines can be customized independently.The default heatmap palette is the perceptually uniform viridis colormap.
- Aesthetic customization: Adjacent displays can include scatterplots, smoothed curves, barplots, line plots, connected-point plots, and dendrograms.Scatterplots are the default adjacent-plot option.
7 Conclusion
The paper presents superheatmaps as extensions of traditional heatmaps for exploratory analysis and model assessment. Three case studies illustrate their use for combining information and examining heterogeneity.
- Contribution: The proposed superheatmap augments traditional heatmaps by incorporating additional information into the visualization.The stated goal includes demonstrating heatmaps as a visualization method for high-dimensional data.
- Contribution: Additional displays can encode response variables as scatterplots, model results as boxplots, correlation information as barplots, and text information.These augmentations provide another avenue for information extraction and exploration of heterogeneity.
- Applicability: The authors describe superheat as highly customizable and applicable to exploratory data analysis and model assessment across a wide range of situations.The implementation is provided in the authors’ superheat R package.
- Case studies: Three case studies demonstrate supervised heatmaps for organ donation and country development, English word clusters, and heterogeneity in fMRI prediction models.The fMRI case study evaluates Lasso models predicting brain signals elicited by visual stimuli.
- Conclusion: The authors conclude that heatmaps are particularly useful for high-dimensional data visualization.Their conclusion is based on the examples presented in the paper.
A APPENDIX: IMPLEMENTATION IN R
The appendix provides R code for reproducing the paper’s superheat figures, including heatmap customization, clustering, smoothing, and overlaid plots.
- Implementation overview: The superheat function generates the figures presented in the paper.The complete analysis pipelines for the three case studies are available in supplementary R Markdown documents.
- Implementation overview: Figure 1 code sets a heatmap color map, orders rows by donations, and adds donations by year as the top plot.
- Word similarity examples: Figure 2 visualizes cosine similarity matrices for the 35 most common words from NY Times headlines.
- Word similarity examples: The code produces a smoothed superheatmap by setting smooth.heat = TRUE.This option is used for the smoothed version of the cosine similarity matrix in Figure 3.
- Customization options: Additional arguments specify row and column clustering, row and column titles, and the response-axis label.The response axis is labeled “Correlation between predicted and true voxel responses.”
- Model-response example: Figure 5 combines a validation-set voxel response matrix with Lasso model performance using smoothing and boxplots.The relevant arguments are smooth.heat = TRUE and yt.plot.type = "boxplot".
B APPENDIX: Selecting the number of word clusters
The appendix reports two stability criteria for selecting the number of word clusters: average pairwise Jaccard similarity and average silhouette width.
- Stability criterion: Average pairwise Jaccard similarity evaluates clustering stability across 100 90% subsamples of the word-vector set.
- Separation criterion: Average silhouette width evaluates the separation structure of clusters across 100 90% subsamples of the word vectors.
C APPENDIX: Cluster word clouds
The appendix presents word clouds and validation-image examples associated with clusters, alongside the reproducible analysis materials.
- Word clusters: Word clouds display the words belonging to each of the 12 word clusters.
- Word clusters: The word corresponding to each cluster center is highlighted in red, and word size represents frequency in NY Times headlines.
- Image clusters: Figures 9 and 10 show four randomly selected validation images from the top and bottom image clusters.
- Reproducibility: The supplementary materials include R Markdown HTML documents containing the complete analysis pipelines for the three case studies.