PaperlayerView exact revision on arXiv

Source-linked AI summary

VizNet: Towards A Large-Scale Visualization Learning and Benchmarking Repository

Kevin Hu, Neil Gaikwad, Michiel Bakker, Madelon Hulsebos, Emanuel Zgraggen, César Hidalgo, Tim Kraska, Guoliang Li, Arvind Satyanarayan, Çağatay Demiralp

arXiv:1905.04616v1cs.HCcs.DBcs.LG

TL;DR

Ad hoc datasets can differ from real-world data and make comparisons difficult. VizNet assembles a large-scale corpus and demonstrates its use in crowdsourced experiments and predictive modeling. Real-data replications largely matched prior findings but also revealed differences, while a learned combined metric predicted effectiveness above baseline performance.

  • Problem

    Ad hoc datasets may lack real-world characteristics, and one-off exemplars make it difficult to compare visualization approaches against a common baseline.

    Source: §1 · p. 2

  • Method

    The paper compiles VizNet from web, open-data, and visualization sources, uses it to replicate and extend a crowdsourced study, and trains a model on data-visualization-task triplets.

    Source: §1 · p. 2, Corpora · p. 3

  • Results

    Real-data results largely matched Kim and Heer (2018), with significant complex-task error-rate differences and longer completion times for all but one task; the combined metric achieved 52.48% top-3 prediction accuracy.

    Source: §6 · p. 9

  • Takeaways & Limitations

    VizNet offers rich real-world datasets for visualization learning, experimentation, replication, and benchmarking, complementing synthetically generated data.

    Source: §6 · p. 9

    Outlier detection lacks a defined ground truth, and crowdsourced training labels can be noisy.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Researchers currently rely on ad hoc datasets to train automated visualization tools and evaluate the effectiveness of visualization designs. These exemplars often lack the characteristics of real-world datasets, and their one-off nature makes it difficult to compare different techniques. In this paper, we present VizNet: a large-scale corpus of over 31 million datasets compiled from open data repositories and online visualization galleries. On average, these datasets comprise 17 records over 3 dimensions and across the corpus, we find 51% of the dimensions record categorical data, 44% quantitative, and only 5% temporal. VizNet provides the necessary common baseline for comparing visualization design techniques, and developing benchmark models and algorithms for automating visual analysis. To demonstrate VizNet's utility as a platform for conducting online crowdsourced experiments at scale, we replicate a prior study assessing the influence of user task and data distribution on visual encoding effectiveness, and extend it by considering an additional task: outlier detection. To contend with running such studies at scale, we demonstrate how a metric of perceptual effectiveness can be learned from experimental results, and show its predictive power across test datasets.

1 INTRODUCTION

Researchers lack a diverse, shared repository of real-world datasets for evaluating visualization designs and comparing approaches. VizNet supplies such a corpus and demonstrates its use in crowdsourced experiments and learning perceptual effectiveness.

1 INTRODUCTION

  • Ad hoc study datasets often lack real-world characteristics, while one-off exemplars make comparisons against a common baseline difficult.Source: §1 · p. 2
  • VizNet contains over 31 million datasets, typically with 17 records and 3 dimensions; dimensions are 51% categorical, 44% quantitative, and 5% temporal.Source: §1 · p. 2
  • Replicating Kim and Heer (2018) with more diverse datasets produced statistically significant differences, informing how graphical-perception studies should account for real-data variation.Source: §1 · p. 2
  • The paper frames visualization as a combinatorial design problem and demonstrates a model that predicts effectiveness for unseen data-visualization-task triplets with non-random performance.Source: §1 · p. 2

2 RELATED WORK

  • VizNet draws on research in graphical perception, automated visualization, crowdsourced data collection, and large-scale machine-learning corpora.Source: §2 · p. 2

Graphical Perception

Graphical-perception studies often use small, limited datasets, making results difficult to generalize, reproduce, and compare. VizNet offers diverse real-world tables for evaluation, while prior visualization corpora and machine-learning systems leave gaps in raw data and training diversity.

Graphical Perception

  • Current graphical-perception experiments often use single, small datasets, making findings partial and difficult to generalize, reproduce, or compare.Source: Graphical Perception · p. 2
  • VizNet provides diverse real-world tables to support evaluation at scale and comparison across experiments on a common baseline.Source: Graphical Perception · p. 3

Data Collection for Visualization Research

Automated Visualization using Machine Learning

Machine Learning Corpora

VizNet draws on large-scale data repositories as a model for shared training, testing, and benchmarking resources. It combines web tables, user-contributed visualization data, and open-portal data, with a balanced sample intended to avoid domination by WebTables.

Machine Learning Corpora

  • Large-scale repositories such as ImageNet have supported machine-learning research by providing rich data and shared baselines for training, testing, and benchmarking.Source: Machine Learning Corpora · p. 3

3 DATA

  • VizNet incorporates four large-scale corpora assembled from the web, online visualization tools, and open data portals.Source: §3 · p. 3

Corpora

  • The web-table source uses horizontal relational tables from WebTables 2015, which extracts structured tables from Common Crawl.Source: Corpora · p. 3
  • The online-tool sources include public visualizations from the Plotly Community Feed collected via API and data from ManyEyes, a visualization-publishing system.Source: Corpora · p. 3
  • Open Data Portal Watch catalogs 262 portals, mostly government-hosted and containing civic and social data.Source: Corpora · p. 3
  • Because WebTables supplies most datasets, the authors sampled 250K datasets from each corpus and combined them into the balanced one-million-dataset VizNet 1M corpus.Source: Corpora · p. 3

Characterization

VizNet characterizes source corpora and a balanced sample using data-type and statistical measures. The experiments use real-world datasets to replicate a prior crowdsourced study, extend its task set, and train a model of visual-design effectiveness.

Characterization

  • Columns are heuristically classified as categorical, quantitative, or temporal; quantitative distributions, skewness, outliers, and categorical entropy are also characterized.Source: Characterization · p. 4

4 EXPERIMENT DESIGN

  • The experiment replicated Kim and Heer (2018) with real-world VizNet datasets sampled to meet prior-study constraints, added outlier detection, and trained a model evaluated across test datasets.Source: §4 · p. 4

Replication of Kim and Heer (2018)

  • Kim and Heer (2018) tested 24 trivariate distributions from daily weather data, with one categorical and two quantitative fields, across data distributions, visualizations, and tasks.Source: Replication of Kim and Heer (2018) · p. 4
  • Their mixed design varied visual encodings within participants and tasks and data characteristics between participants; 1,920 MTurk participants completed 96 questions and 12 engagement checks.Source: Replication of Kim and Heer (2018) · p. 4

Datasets

VizNet datasets were filtered to match the prior study’s constraints and sampled for each visualization-specification/task condition. The study used twelve scatterplot encodings with mappings between data fields and visual channels.

Datasets

  • Datasets had to contain one categorical and two quantitative fields, no nulls, and meet constraints on column names, category cardinality, group names, and values per group.The filtering thresholds were chosen to match the upper and lower constraints of Kim and Heer (2018).Source: Datasets · p. 4
  • The sample comprised 9,600 datasets: 2,400 per task and 2,400 per source corpus, selected from 200 candidates for each of 60 visualization-specification/task conditions.The valid-dataset counts were 2,941 from Open Data, 6,090 from Many Eyes, 1,368 from Plotly, and 82,150 from a Webtables subset.Source: Datasets · p. 4

Visual Encodings

  • The twelve selected designs were scatterplots that varied mappings from data fields to visual channels; a pilot-informed axis-range heuristic was used to avoid prohibitively difficult questions.Figure 2 outlines three-column datasets and the one-categorical/two-quantitative subset in red.Source: Visual Encodings · pp. 4, 6, Figure 2 · p. 5

Tasks

The experiment tested four value and summary tasks, using a mixed design in which participants saw multiple encodings but were assigned one task. Participants and responses were screened for judgment quality and color discrimination.

Tasks

  • Read Value and Compare Values tested individual values, while Find Maximum and Compare Averages tested aggregate properties through binary, two-alternative questions.The tasks were informed by the Amar et al. (2005) taxonomy.Source: Tasks · p. 6

Procedure

  • Each participant answered nine questions per encoding across all twelve encodings in random order, with task assigned between subjects and encoding varied within subjects.Datasets were randomly selected from the pool for each visualization-specification/task condition, and participants also completed twelve gold-standard tasks.Source: Procedure · p. 6

Participants

  • Of 1,342 recruited U.S. MTurk workers, 624 were analyzed after exclusions for gold-standard errors, high experimental error, or color discrimination; a more lenient gold-standard threshold did not significantly change results.The gold-standard exclusion threshold was 100%, and color screening included two Ishihara plates and two pre-screen questions.Source: Participants · p. 6

5 RESULTS

The replication largely matched prior encoding rankings, but error rates and response times differed by task. The added outlier-detection task had higher error rates than the other tasks and shorter response times in most encoding conditions.

Extending with an Outlier Detection Task

  • Outlier detection used a yes/no question about Q1, with outliers determined by a median absolute deviation approach; its error rates exceeded those of the other tasks.The authors note that this could reflect inadequate ground truth, inconsistent definitions, or insufficient prior training.Source: Extending with an Outlier Detection Task · p. 7

Comparing Subject Performance

  • Read Value error rates tended to agree with Kim and Heer (2018), while agreement was weaker for Compare Values and summary-task error rates departed; encoding rankings were similar for the first two tasks.The authors suggest real-world data may affect error rates for more complex tasks, while noting that more data points are needed for meaningful conclusions.Source: Figure 4 · p. 7, Comparing Subject Performance · p. 7
  • Absolute response times were systematically longer than Kim and Heer (2018) for every task except Compare Values, although encoding rankings remained consistent.Source: Figure 4 · p. 7, Comparing Subject Performance · p. 7

Extending with an Outlier Detection Task

Learning a Model to Predict Effectiveness

The paper characterizes dataset triplets and trains models to predict completion time and combined effectiveness. It also uses dimensionality reduction to examine how the prior study’s datasets relate to the VizNet datasets.

Learning a Model to Predict Effectiveness

  • The 167 dataset features covered statistics, distributions, pairwise relationships, clusteredness, and spatial autocorrelation; 32 principal components explained over 85% of variance, and prior-study datasets clustered apart from the replication datasets.The separation was robust across tested component counts and t-SNE perplexities of 5–200.Source: Learning a Model to Predict… · p. 8
  • A gradient boosted regression tree predicted log completion times with cross-validated R2 of 0.47 on the 20% hold-out test set, outperforming K-nearest neighbors and simple linear regression.The model trained on 80% of the data; Figure 6 compares observed and predicted log response times against a perfect-relationship diagonal.Source: Learning a Model to Predict… · p. 8, Figure 6 · p. 8
  • A combined response-time/error-rate metric yielded 52.48% Top-3 prediction accuracy using a gradient boosted classification tree trained on a balanced set.The metric paired response-time and error-rate bins, and minority classes were resampled with SMOTE.Source: Learning a Model to Predict… · p. 8

Limitations

The replication changed question wording and interface design, and could not match participants or control question difficulty. Outlier detection lacks a defined ground truth, while the study also did not reproduce the original synthetic-data conditions.

Limitations

  • Practical constraints led to clarifying changes in the replication’s question text and interface design.The study also could not recruit the same participants or control question difficulty.Source: Limitations · pp. 8–9
  • The study did not exactly reproduce the original synthetic-dataset conditions, which would have limited sampling of real-world VizNet datasets.Crowdsourced labels may be noisy, and the experiment did not analyze minimum data-quality requirements.Source: Limitations · p. 9
  • Outlier detection has no defined ground truth, and the absence of an objective definition may partly explain its consistently high error rate.The authors also note that a purely quantitative method may be limited without human consensus.Source: Limitations · p. 9

6 DISCUSSION

The discussion reports that VizNet supports graphical-perception research through a characterized, accessible corpus, replication experiments, and learned effectiveness metrics. Future work proposes expanding and characterizing the corpus, harnessing crowdsourcing, and developing active learning for experiment design.

6 DISCUSSION

  • VizNet provides full and one-million-dataset sampled corpora, characterizes dataset dimensions, types, and statistics, and complements synthetic data.Its corpus properties can inform ecological-validity assessments of corpora in other domains.Source: §6 · p. 9
  • The envisioned interface supports querying, filtering, and sampling by column-type composition and other dataset properties, as well as keyword search by domain.The authors envision a common corpus and benchmarks helping the visualization community share and compare results at scale.Source: §6 · p. 9
  • With real-world datasets, replication findings were largely consistent, but complex-task error rates differed significantly and completion times were longer for all but one task.The authors say graphical-perception studies must account for variation in real datasets; the work also extends the experiment with outlier detection.Source: §6 · p. 9
  • 52.48% top-3 prediction accuracy for the combined metric, alongside an R2 of 0.47 for completion-time prediction, outperformed random chance and baseline classifiers.The authors present these results as evidence that perceptual effectiveness can be learned despite noisy, skewed crowdsourced labels and a relatively small sample.Source: §6 · p. 9

7 FUTURE WORK

  • The authors plan to expand and characterize datasets, harness crowd contributions, and develop active learning for optimal experiment design.They describe learning from labeled triplets to predict effectiveness for unseen ones as a way to contend with the cost of gathering human judgments.Source: §7 · p. 10
  • Future corpus work includes adding datasets, characterizing semantic column content, and studying between-corpus bias in dimensions, type composition, and column statistics.The authors identify such bias analysis as potentially informing future sampling techniques.Source: §7 · p. 10
  • VizNet is envisioned as a platform for citizen scientists and researchers to conduct graphical-perception experiments at scale.The authors point to worker responses describing the experiment as intellectually engaging and identify task design, work quality, and incentives as areas for further work.Source: §7 · p. 10
  • In a small experiment, self-learning with high-certainty predictions and crowdsourced labels for uncertain cases improved test-set accuracy over supervised learning across iterations.Figure 8 presents performance curves for the two approaches over 10 iterations.Source: §7 · p. 10, Figure 8 · p. 10

8 CONCLUSION

VizNet is presented as a step toward meeting visualization research’s growing need for large-scale data and visualization repositories.

8 CONCLUSION

  • VizNet is a step toward addressing the growing need for large-scale data and visualization repositories in data visualization research.Source: §8 · p. 10
Loading 1905.04616v1…