PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning Visual Importance for Graphic Designs and Data Visualizations

Zoya Bylinskii, Nam Wook Kim, Peter O'Donovan, Sami Alsheikh, Spandan Madan, Hanspeter Pfister, Fredo Durand, Bryan Russell, Aaron Hertzmann

arXiv:1708.02660v1cs.HCcs.CV

TL;DR

Graphic designs and data visualizations lack broadly applicable automatic importance predictors, despite the value of knowing which elements matter. The paper trains neural networks on crowdsourced clicks and importance annotations, then uses predicted importance maps for applications including retargeting, thumbnailing, and interactive design tools. User studies validate that these applications are on par with or outperform related methods, while the authors note limitations from training-data bias and non-uniform click importance.

  • Problem

    Automatic importance prediction has received little attention for graphic designs and data visualizations, although importance helps identify relevant content and guide attention.

  • Method

    The authors train neural networks on crowdsourced GDI annotations and BubbleView clicks to predict per-pixel importance maps from bitmap designs.

  • Results

    Predicted importance maps support retargeting, thumbnailing, and interactive design tools, with user studies finding application results on par with or better than related methods.

  • Takeaways & Limitations

    Importance maps provide a common representation for automatic design applications and immediate feedback during interactive design.

  • Takeaways & Limitations

    The models and ground truth show a strong text-region bias for data visualizations, which may not generalize to other visualization types and tasks.

Abstract

from arXiv · show

Knowing where people look and click on visual designs can provide clues about how the designs are perceived, and where the most important or relevant content lies. The most important content of a visual design can be used for effective summarization or to facilitate retrieval from a database. We present automated models that predict the relative importance of different elements in data visualizations and graphic designs. Our models are neural networks trained on human clicks and importance annotations on hundreds of designs. We collected a new dataset of crowdsourced importance, and analyzed the predictions of our models with respect to ground truth importance and human eye movements. We demonstrate how such predictions of importance can be used for automatic design retargeting and thumbnailing. User studies with hundreds of MTurk participants validate that, with limited post-processing, our importance-driven applications are on par with, or outperform, current state-of-the-art methods, including natural image saliency. We also provide a demonstration of how our importance predictions can be built into interactive design tools to offer immediate feedback during the design process.

INTRODUCTION

The paper introduces neural-network models that predict element importance in graphic designs and data visualizations, addressing limited prior work beyond natural-image saliency. These predictions support retargeting, thumbnailing, and interactive design feedback, with user studies validating competitive application quality.

  • INTRODUCTION: Importance prediction can support search, retargeting, summarization, and design-process feedback by identifying where viewers should focus.The paper frames importance as useful for understanding perception and managing attention in visual designs.
  • INTRODUCTION: The paper distinguishes importance from traditional bottom-up saliency by allowing semantic factors such as titles, axis text, and data points to influence weighting.Importance is used as a generic term for perceived relative weighting of design elements.
  • INTRODUCTION: The models operate on bitmap designs and learn higher-level importance patterns, including localizing titles and weighting design elements.Original vector data is unnecessary, allowing application to existing bitmap libraries.
  • INTRODUCTION: Predicted importance maps serve retargeting, thumbnailing, and interactive design tools, with user studies finding results on par with or better than related methods.Fast neural-network inference enables immediate feedback as users modify design elements.
  • INTRODUCTION: The paper presents neural-network importance predictors for graphic designs and data visualizations, trained on crowdsourced importance data.The data include GDI annotations for graphic designs and BubbleView clicks collected on data visualizations.

RELATED WORK

Prior work studied eye movements, crowdsourced design importance, and natural-image saliency, but automatic importance prediction remained limited for graphic designs and data visualizations. This paper extends neural-network prediction to both domains and compares crowdsourced BubbleView clicks with eye movements and explicit annotations.

  • RELATED WORK: Eye tracking provides clues about interface and design perception, but collecting eye movements is expensive and time-consuming for practical applications.This motivates less costly alternatives for studying visual importance.
  • RELATED WORK: Earlier graphic-design systems used hand-crafted saliency or predictors requiring manual knowledge of element locations, limiting automatic deployment on new designs.Related web-page methods likewise rely on DOM information or manual segmentations in some cases.
  • RELATED WORK: Deep-learning saliency methods improved natural-image benchmarks, but were not trained or tested on graphic designs or data visualizations.The paper positions its work as the first neural-network importance prediction approach spanning both visual domains.
  • RELATED WORK: The study collects BubbleView clicks for data visualizations and compares them with eye movements and GDI importance annotations for graphic designs.These comparisons examine whether click-based measurements capture similar importance structure across modalities.

Ground truth importance for data visualizations

The paper collects crowdsourced BubbleView clicks and uses fully convolutional neural networks to predict pixel-level importance in data visualizations and graphic designs. Separate models and training choices reflect the different datasets and target representations.

  • Ground truth importance for data visualizations: BubbleView records attention by letting participants click blurred images to reveal local regions, providing a scalable alternative to laboratory eye tracking.Earlier work found BubbleView clicks highly correlated with eye fixations, and this paper confirms the relationship.
  • Ground truth importance for data visualizations: The data-visualization dataset contains 1,411 MASSVIS images spanning diverse sources and visualization types, filtered for legibility and sufficient context.
  • Ground truth importance for data visualizations: GDI annotations give more uniform element-level importance, whereas BubbleView maps may better model human attention directly.
  • Ground truth importance for data visualizations: The models predict importance at each bitmap pixel using fully convolutional networks trained with sigmoid cross-entropy on real-valued importance targets.The output P_i ranges from 0 to 1, with larger values indicating higher importance.
  • Ground truth importance for data visualizations: The data-visualization model used FCN-32s because higher-resolution skip connections produced no performance gains beyond that architecture.For graphic designs, FCN-16s was sufficient and FCN-8s added no performance boost.
  • Ground truth importance for data visualizations: Smaller architectures were chosen because these importance datasets are currently an order of magnitude smaller than natural-image saliency datasets.

EVALUATION OF MODEL PREDICTIONS

The evaluation compares predicted importance maps with ground-truth importance from BubbleView clicks, eye fixations, and graphic-design annotations.

  • EVALUATION OF MODEL PREDICTIONS: Predictions are evaluated against BubbleView clicks and MASSVIS eye fixations for data visualizations, and against GDI annotations for graphic designs.

Evaluation criteria

The paper evaluates importance maps with KL divergence and cross correlation, using their complementary treatment of distributional similarity and pixel-wise correlation. Results are averaged over 202 test visualizations with chance and saliency-model comparisons.

  • Evaluation criteria: KL divergence measures how well a predicted importance distribution predicts the ground-truth distribution, while CC measures pixel-wise correlation and treats false positives and negatives equally.
  • Evaluation criteria: KL divergence is zero for identical maps and can be unbounded, so chance baselines provide a feasible reference range.
  • Evaluation criteria: CC ranges from -1 to 1, with 1 indicating maximal correlation between the predicted and ground-truth maps.
  • Evaluation criteria: Table 1 averages KL and CC over 202 test visualizations and compares the importance model with chance, Judd saliency, and DeepGaze.Higher CC and lower KL are better.
  • Evaluation criteria: Table 2 evaluates predictors of human eye-fixation maps, including the click-trained neural network, BubbleView clicks as an upper bound, and chance as a lower bound.The neural network is expected to predict BubbleView clicks best but remains representative of fixation data.

Prediction performance on data visualizations

On data visualizations, the model predicts human importance patterns and outperforms the evaluated saliency baselines, while also reflecting eye-fixation patterns. Titles and other text elements receive especially high predicted importance, though click-trained predictions may differ from fixations.

  • The model outperforms chance, Judd saliency, and DeepGaze on BubbleView-click prediction.The comparison uses CC and KL scores averaged over 202 test images; KL is sensitive to false negatives and output sparsity.
  • Predicted importance is representative of eye-fixation patterns, although click-trained models may learn patterns that differ from fixations.
  • The model localizes visualization titles and extreme graph points but can overestimate the importance of visual map regions.
  • Titles and captions are among the most looked-at, clicked, and model-predicted elements in visualizations.

Prediction performance on graphic designs

For graphic designs, predicted importance correlates with annotated element importance and improves on the fully automatic comparison model. The predictions also contribute information beyond existing text and natural-image-saliency features, while remaining imperfect on text segmentation and semantic visual–text comparisons.

  • Our model outperforms the fully automatic O’Donovan variant and runs about 100X faster.The model requires approximately 0.1 s/image on a GPU, compared with approximately 10 s/image for the most efficient O’Donovan setting.
  • Adding the model’s predictions to O’Donovan features improves upon OD-Full, indicating that the predictions are not fully explained by existing features.The combined model is presented for demonstration and is not practical for interactive applications because it includes manual annotations.
  • The model achieves an average Spearman rank correlation of 0.56 between predicted and ground-truth graphic-design element scores.Element scores are computed as the maximum predicted or annotated importance value within each element bounding box.
  • Predictions capture larger and more central text and visual elements as more important, but unusual fonts and text segmentation remain difficult.The paper identifies semantic comparisons between text and visuals as harder cases because importance can depend on text informativeness and visual quality.

Prediction performance on fine-grained design variations

On fine-grained design variations, the model generalizes to changes in element location and scale, achieving moderate agreement with human importance rankings despite some absolute errors.

  • Predicted rankings account for design changes such as variations in element location and size, although some absolute errors remain.The model was not explicitly trained on systematic design variations, so this experiment tests generalization to those changes.
  • The model achieves an average Spearman correlation of rs = .53 across design variants.Importance scores are assigned to preserved elements by taking the maximum value within each predicted or ground-truth heatmap segment.

APPLICATIONS

Importance maps support retargeting graphic designs by preserving high-importance regions while operating directly on bitmap inputs. User studies found predicted-importance crops competitive with stronger baselines and better than natural-image saliency and random cropping.

  • Retargeting: Importance-based retargeting selects crops containing the highest-valued regions of a graphic-design bitmap for specified output dimensions.The method uses predicted importance maps and does not require vector representations or manual input annotations.
  • Evaluation: 96 participants rated six retargeted variants for 12 designs each using a 5-point Likert scale.The study included 10 randomly selected designs and two quality-control designs per participant.
  • Evaluation: 3.19 mean score was achieved by ground-truth GDI crops, versus 2.95 for DeepGaze and 2.92 for predicted importance.The difference between DeepGaze and predicted importance was not statistically significant.
  • Results: Predicted importance outperformed Judd saliency and performed on par with DeepGaze despite training on fewer than 1K graphic-design images.Both predicted importance and DeepGaze significantly outperformed the edge-energy baseline, while Judd saliency and random crops were significantly worse than the other methods.

Thumbnailing

The thumbnailing application uses importance maps to remove less important regions from data visualizations and produce square summaries. In a search task, importance-based thumbnails enabled participants to find matching visualizations with fewer clicks than resized originals.

  • Thumbnailing: Importance-based thumbnailing iteratively removes low-energy rows and columns until a data visualization reaches square proportions.The importance map serves as the energy function, and straight seams were found to work better in this setting.
  • Evaluation: The study asked participants to find a visualization matching a description within a grid of 60 thumbnails.It compared resized original visualizations with automatically computed importance-based thumbnails.
  • Evaluation: 400 participants were recruited, with 200 analyzed in the resized-visualization condition and 169 in the importance-based-thumbnail condition.The analysis filtered participants before comparing the two groups.
  • Results: 1.96 clicks were needed with importance-based thumbnails versus 3.25 clicks with resized visualizations.The difference was statistically significant, t(367) = 5.10, p < .001.

Interactive applications

The paper demonstrates interactive design feedback by recomputing importance maps as users edit layouts and styles. The approach is fast enough for immediate feedback, while the authors identify automated design suggestions as future work and note training-data limitations.

  • Interactive feedback: A prototype layout tool recomputes an importance map whenever users move or resize elements or change color, font, or opacity.The model is integrated into a design tool to provide immediate feedback during editing.
  • Future work: Automatic suggestions for novice users are identified as a future direction for using importance prediction in design feedback.The paper describes determining how best to provide feedback as an open problem.
  • Performance: The model computes a 600×450-pixel GDI importance map in 100 ms on a Titan-X GPU.Table 4 reports timing for additional image sizes.
  • Limitations: Data-visualization predictions show a strong bias toward text regions that might not generalize to other visualization types and tasks.The authors also caution that non-uniform BubbleView importance can cause downstream thumbnailing side effects, such as cutting off parts of elements or text.
  • Applications: Importance maps serve as a common representation for retargeting, thumbnailing, and a prototype interactive design application.The models are trained on annotated graphic designs and data visualizations and generalize across a wide range of design formats.
Loading 1708.02660v1…