PaperlayerView exact revision on arXiv

Source-linked AI summary

VINS: Visual Search for Mobile User Interface Design

Sara Bunian, Kai Li, Chaima Jemmali, Casper Harteveld, Yun Fu, Magy Seif El-Nasr

arXiv:2102.05216v1cs.HCcs.CVcs.DBcs.IR

TL;DR

Existing UI-design searches rely on text and may miss relevant visual structure, content, or unseen designs. VINS addresses this by retrieving similar mobile UI designs from wireframes or high-fidelity images using annotated view hierarchies, structural and content features, and deep learning, achieving high performance in similar-design queries.

  • Problem

    Text-based searches can return examples that do not match a design’s visual layout or UI content, while prior image-retrieval methods have limited content modeling or generalizability.

  • Method

    VINS combines designer-informed requirements, a large annotated dataset spanning wireframes and high-fidelity screens, UI-component detection, and embeddings that combine structural and content features.

  • Results

    VINS achieves high performance in querying similar UI designs from an app-layout image.

  • Takeaways & Limitations

    The framework and dataset support visual retrieval across different mobile UI design stages and can extend to other design layouts.

  • Takeaways & Limitations

    The VINS dataset does not always detect all UI components or provide similar designs, and a larger dataset may improve detection performance and design variation.

Abstract

from arXiv · show

Searching for relative mobile user interface (UI) design examples can aid interface designers in gaining inspiration and comparing design alternatives. However, finding such design examples is challenging, especially as current search systems rely on only text-based queries and do not consider the UI structure and content into account. This paper introduces VINS, a visual search framework, that takes as input a UI image (wireframe, high-fidelity) and retrieves visually similar design examples. We first survey interface designers to better understand their example finding process. We then develop a large-scale UI dataset that provides an accurate specification of the interface's view hierarchy (i.e., all the UI components and their specific location). By utilizing this dataset, we propose an object-detection based image retrieval framework that models the UI context and hierarchical structure. The framework achieves a mean Average Precision of 76.39\% for the UI detection and high performance in querying similar UI designs.

1 INTRODUCTION

VINS addresses the difficulty of finding relevant mobile UI examples with text-only search by retrieving similar designs from wireframes or high-fidelity screenshots. It combines an annotated UI dataset with object detection and attention-based retrieval to model UI components, content, layout, and hierarchy.

  • Motivation: Text-based searches often return examples mismatched with the requested visual layout and UI content, making design-example discovery tedious.Designers can search keywords, categories, or colors, but these queries do not directly represent layout structure or interface content.
  • Motivation: Existing image-retrieval methods either omit UI content or depend on predefined hierarchies, limiting relevance and generalizability to unseen designs.The paper motivates a fine-grained search method that infers UI content hierarchy from arbitrary input queries.
  • Framework: VINS accepts abstract wireframes or high-fidelity layouts as queries and retrieves the most similar matches from a design inventory.The framework is designed to support multiple stages of the UI design process.
  • Framework: VINS detects UI components and uses an attention-based neural network to learn a joint representation of layout structure and content for retrieval.The framework is organized around detection and retrieval building blocks.
  • Dataset: The VINS dataset contains UI screens across design stages with human-annotated view hierarchies specifying component types and locations.These annotations support data-driven design applications and model the interface structure needed for retrieval.

2 RELATED WORK

Prior design-search systems commonly rely on text or limited visual representations, while UI retrieval methods often lack structural and content-aware generalization. VINS responds with an annotated dataset and object-detection approach for localizing and classifying UI components.

  • Visual search: Design-search systems across several domains commonly use text queries, but keywords often fail to express abstract design ideas precisely.This limitation makes relevant example retrieval difficult for designers.
  • Visual search: Image-based search has been explored with sketches and UI screenshots because image queries can be faster and easier to specify than keywords.These approaches broaden search beyond conventional textual queries.
  • Visual search: Deep-learning UI retrieval methods have modeled layout similarity using image, text, or semantically annotated UI content.The related work includes autoencoder-based approaches using progressively richer UI representations.
  • Visual search: Swire achieved 60% relevancy but used only high-level layout information and required paired sketches and screenshots, limiting structural coverage and generalization.VINS is positioned as addressing these two limitations.
  • Mobile UI datasets: Rico provides large-scale screenshots and view hierarchies, but the paper identifies a need for more accurate content-hierarchy annotations for UI object detection.Accurate localization and classification require carefully annotated UI components.
  • Mobile UI datasets: VINS contributes a public, fully annotated dataset and uses object detection to infer UI layout structure from component locations and categories.The dataset is designed to support detection across UI design screens, including sketches.

3 FORMATIVE INTERVIEWS

Interviews show that designers frequently use examples for inspiration but find current keyword-based search slow and poorly matched to intended layouts. Their accounts support retrieving similarity through structure, functionality, and visual elements rather than keywords alone.

  • Study design: 24 professionally trained U.S.-based designers with 1–5 years of experience participated in semi-structured interviews analyzed through three-stage thematic coding.The study examined example-finding strategies, difficulties, similarity criteria, and VINS applicability.
  • Use of examples: Designers most often collect layouts, followed by font styles, color palettes, and content hierarchy from examples.The corresponding counts were 22, 14, 11, and 4 designers.
  • Search difficulties: 21 designers reported keyword-based searching, while many described the process as taking hours or sometimes more than a day.The interviews connect keyword search with difficulty expressing specific layout requirements and finding relevant examples.
  • Similarity criteria: Designers differed on whether keywords measure similarity, with 10 accepting them as an indication and 14 favoring additional design aspects.Interview responses emphasized functionality, structure, and visuality as complementary similarity criteria.
  • Similarity criteria: The interviews led VINS to emphasize functionality and structure alongside visual elements when retrieving similar UI designs.One example considered layouts similar despite different color schemes when structure and functionality matched.

4 MOBILE UI DATASET

Because UI object detection requires many accurately boxed training images, VINS introduces a new annotated dataset spanning abstract wireframes and high-fidelity interfaces. The dataset is built to support component localization and classification for visual search.

  • Dataset motivation: Object detection requires large collections of UI images with high-quality bounding-box annotations.The paper therefore created a dedicated dataset because no suitable large-scale public dataset was available.
  • Dataset scope: The VINS dataset contains UI screens from two design stages: abstract wireframes and high-fidelity fully designed interfaces.This scope supports retrieval across different stages of the design process.

4.1 Data Collection

The VINS dataset combines wireframe and high-fidelity mobile UI screens to support retrieval across design stages, platforms, and prototyping styles.

  • Dataset composition: 4,800 UI design screens comprise 257 abstract wireframes and 4,543 high-fidelity screens.The dataset includes different design stages to support a wider variety of design inputs.
  • Wireframes: 257 wireframes represent low- and medium-fidelity interface outlines collected from different templates and layouts.Their relative simplicity and limited styling reduce elements that might affect detection.
  • Wireframes: Different prototyping styles are represented because designers may depict image placeholders with different visual symbols.Examples include a mountain or a square with a cross.
  • High-fidelity screens: The high-fidelity subset includes iPhone and Android interfaces from popular applications across different categories.This selection was intended to support generalization across platforms.

4.2 Annotation Process

The annotation process combines trained crowdsourced labeling, strict bounding-box rules, assessment, and verification to produce a structured UI dataset.

  • Crowdsourced annotation: Six students were recruited and compensated to annotate the VINS dataset through crowdsourcing.The strategy was designed for large-scale bounding-box annotation.
  • Training: Annotators learned 11 UI component classes, their functionality, and style guidelines before labeling images.The instructions also used annotated examples to clarify component instances.
  • Annotation rules: Bounding boxes had to tightly contain complete objects and receive the correct class label.The rules explicitly rejected boxes that omitted visible object parts or included unnecessary background.
  • Assessment: 90% Intersection over Union (IoU) was required on all assessment images before students could annotate the actual dataset.Training was iterative, with feedback after submissions that failed the threshold.
  • Dataset output: The completed dataset pairs each UI image with a Pascal-VOC XML file and is presented as the first publicly available annotated UI design dataset for object detection.The annotation workflow included drawing and quality-verification stages.

5 VISUAL SEARCH SYSTEM

VINS retrieves structurally similar mobile UI designs by combining detected UI layout structure with component-content information. Its image retrieval model learns separate structural and content features, fuses them into an embedding, and ranks designs by embedding similarity.

  • VINS indexes UI designs by functionality and structural information rather than only visual properties such as color, texture, and shape.
  • Detection: The system detects UI components and their locations, then generates a semantic structured layout representing component classes and hierarchical context.The detector’s outputs provide the structured input used by subsequent retrieval stages.
  • Image Retrieval: The retrieval model learns joint features from each image’s UI structure and component content to measure semantic relevance across designs.Component class labels provide high-level content information alongside spatial structure.
  • Image Retrieval: The image model encodes hierarchical structure with an attention-aware autoencoder, while the label model encodes detected component classes before concatenation.The image and label feature vectors are fused into the final retrieval representation.
  • Image Model: Box Attention augments the base autoencoder with a spatial binary map encoding UI-component bounding boxes to guide structural learning.The attention map is conditioned on convolutional-layer outputs in the encoder.
  • Image Model: The autoencoder combines reconstruction and Dice-based losses, with the Dice term targeting overlap and object-boundary information.The reconstruction term measures similarity between the original and reconstructed images.
  • Label Model: Each image’s unique UI component classes are encoded as an 11-dimensional multi-hot vector and transformed into a 64-dimensional content vector.The label model uses this content representation to support learning of the overall layout structure.
  • Image Retrieval: Retrieval ranks reference images using Euclidean distance between concatenated structural and content embeddings.

6 SYSTEM EVALUATION

VINS is evaluated through object detection, retrieval, qualitative comparisons, and designer feedback. Its annotations improve detection, while attention-based retrieval better matches UI components and layout structure than the baseline and supports design exploration from complete or partial layouts.

  • Evaluation setup: VINS evaluation covers object detection, image retrieval, end-to-end performance, qualitative comparisons, and designer feedback.The evaluation includes detection and retrieval experiments, visual query-result inspection, and expert assessment of retrieved examples.
  • Object detection: More than 26% higher mAP over Rico was achieved by VINS annotations across all seven evaluated classes.The comparison uses Average Precision at IoU = 0.5 for the seven shared classes.
  • Object detection: 76.39% overall mAP and 79.02% AUC were achieved on a 450-image test set from the 4,543-image VINS dataset.Sliding menu reached 100% AP, while checked view had the lowest AP at 44.48%, attributed to cross-class similarity and small object size.
  • Quantitative retrieval: 92.05% top-1 precision and 86.48% top-10 precision were achieved by the m = 4 model, improving approximately 4–6% over the baseline.The authors attribute the improvement to attention maps capturing UI layout and retrieving more relevant images.
  • Qualitative retrieval: Qualitative examples show VINS retrieving designs that better preserve queried components and component-placement sequences than the baseline.The baseline sometimes misses input fields, text buttons, or page indicators, whereas VINS better follows the query’s content and structure.
  • Expert evaluation: Designers found VINS results relevant and useful for variations in layout, color, typography, composition, and additional components.The system also accepts partial layouts and retrieves examples resembling the detected components, supporting layout ideation and auto-completion without retraining on partial sketches.

7 DISCUSSION

The discussion identifies dataset coverage and visual-search modeling as the main constraints on VINS, while outlining broader design-layout applications and needed user validation.

  • 7.1 Increasing the Dataset: The VINS dataset was large enough for this work but did not always detect all UI components or provide similar designs.The authors suggest that a larger dataset could improve detection performance and provide more design variations.
  • 7.1 Increasing the Dataset: The detection process currently covers only 11 common UI-component classes, limiting applicability to UIs with other component sets.The authors propose adding components spanning functionalities and distinguishing labels, button concepts, and icon classes.
  • 7.2 Improving Visual Search: VINS could incorporate visual features such as imagery, fonts, and colors alongside structure and content.The discussion also proposes tree-based representations to model relations among UI components more explicitly.
  • 7.2 Improving Visual Search: The framework could extend from mobile apps to layouts such as magazines, posters, and web pages because layout is broadly important in graphic design.This is presented as a possible extension of the visual-search framework.
  • 7.3 User Evaluation: VINS still requires evaluation by UI designers to determine how it integrates with daily workflows and meets user requirements.The authors identify user evaluation as future work in human-AI interaction.
  • 7.4 Designer Feedback: Designer feedback suggests VINS could support early-stage design and could eventually accept user-specified constraints such as keywords and colors.The interviews also suggest extending the dataset to include sketches and involving clients in creating layouts.

8 CONCLUSION

VINS combines annotated UI data, object detection, and multimodal retrieval to find similar designs from wireframes or high-fidelity images. It reports strong performance for both component detection and retrieval.

  • 8 CONCLUSION: VINS is an object-detection-based visual search framework for UI layout designs.The framework was developed alongside interviews with UI designers and a large-scale annotated UI dataset.
  • 8 CONCLUSION: VINS detects UI components to construct segmented layouts, then learns joint representations with an attention-based multimodal embedding model.The model retrieves similar UI designs from abstract wireframes or high-fidelity images.
  • 8 CONCLUSION: 76.39% mAP was achieved for UI-component detection, with 80-90% retrieval precision for relative design examples.These results cover the detection and retrieval phases of the framework.
Loading 2102.05216v1…