PaperlayerView exact revision on arXiv

Source-linked AI summary

Machine Learning-Based Prototyping of Graphical User Interfaces for Mobile Apps

Kevin Moran, Carlos Bernal-Cárdenas, Michael Curcio, Richard Bonett, Denys Poshyvanyk

arXiv:1802.02312v2cs.SEcs.CVcs.LG

TL;DR

Transforming GUI mock-ups into faithful application code is challenging and time-consuming. The paper presents REDRAW, which detects, classifies, and assembles GUI components using mined GUI data, CNNs, and KNN, achieving high classification precision and visually similar prototypes. REDRAW remains limited in its current support for multi-screen applications and inferred stylistic details.

  • Problem

    Translating GUI mock-ups into application code is challenging and time-consuming, creating a need for automated support.

  • Method

    REDRAW detects GUI components from mock-ups, classifies them with a CNN trained on mined GUI data, and assembles hierarchies with an iterative KNN algorithm.

  • Results

    REDRAW achieves 91% average GUI-component classification precision, outperforming the 65% BOVW baseline, while producing hierarchies closer to target hierarchies than REMAUI and pix2code.

  • Takeaways & Limitations

    The evaluation indicates that REDRAW accurately handles GUI components, generates similar hierarchies and visually similar applications, and may improve industrial development workflows.

  • Takeaways & Limitations

    REDRAW currently prototypes only one application screen at a time and infers a limited set of stylistic details.

Abstract

from arXiv · show

It is common practice for developers of user-facing software to transform a mock-up of a graphical user interface (GUI) into code. This process takes place both at an application's inception and in an evolutionary context as GUI changes keep pace with evolving features. Unfortunately, this practice is challenging and time-consuming. In this paper, we present an approach that automates this process by enabling accurate prototyping of GUIs via three tasks: detection, classification, and assembly. First, logical components of a GUI are detected from a mock-up artifact using either computer vision techniques or mock-up metadata. Then, software repository mining, automated dynamic analysis, and deep convolutional neural networks are utilized to accurately classify GUI-components into domain-specific types (e.g., toggle-button). Finally, a data-driven, K-nearest-neighbors algorithm generates a suitable hierarchical GUI structure from which a prototype application can be automatically assembled. We implemented this approach for Android in a system called ReDraw. Our evaluation illustrates that ReDraw achieves an average GUI-component classification accuracy of 91% and assembles prototype applications that closely mirror target mock-ups in terms of visual affinity while exhibiting reasonable code structure. Interviews with industrial practitioners illustrate ReDraw's potential to improve real development workflows.

1 INTRODUCTION

Translating GUI mock-ups into faithful application code is difficult, time-consuming, and error-prone, motivating automated prototyping support. The paper proposes REDRAW, a data-driven Android approach combining GUI detection, classification, assembly, and empirical evaluation.

  • Motivation: GUI mock-ups must be faithfully translated into code, but the iterative process is challenging, time-consuming, and error-prone.These difficulties are especially relevant when design and implementation involve different teams.
  • Motivation: Manual heuristics are unlikely to support highly variable GUI designs across a software domain.The proposed alternative learns domain knowledge from screenshots and GUI metadata mined from existing applications.
  • Approach: REDRAW decomposes automated GUI prototyping into detection, classification, and assembly tasks.The approach was implemented for Android and combines program analysis, software-repository mining, machine learning, and computer vision.
  • Evaluation: The paper evaluates REDRAW using complementary quality metrics, related-work comparisons, and feedback from industrial professionals.The evaluation addresses both technical quality and potential utility in development workflows.
  • Resources: The collected dataset contains screenshots and GUI metadata for over 14k screens and over 190k GUI-components.The dataset supports training and evaluating the machine-learning components of the approach.

2 BACKGROUND & RELATED WORK

The paper defines mock-up artifacts, GUI-components, and GUI-containers to frame automated generation of visually and structurally similar mobile GUI prototypes. It situates REDRAW among CNN-based classification and GUI reverse-engineering approaches while emphasizing data-driven GUI knowledge.

  • Definitions: A mock-up artifact specifies GUI design guidelines and content, commonly as a high-fidelity image with optional metadata.Industrial workflows may export metadata describing constituent parts of Photoshop or Sketch artifacts.
  • Definitions: GUI-components are atomic graphical elements with predefined functionality, while GUI-containers group components and define their spatial display properties.GUI hierarchies use logical groupings rather than fixed coordinates and can react to display-size changes.
  • Problem Statement: The paper’s problem is to generate a prototype application that resembles a mock-up visually and in GUI-hierarchy structure.The scope focuses on mobile apps and includes screenshots of existing applications and Sketch mock-ups.
  • CNN Background: CNNs extract image features through convolutional layers, reduce dimensionality through max pooling, and classify using fully connected layers and softmax.The supplied CNN overview describes convolution filters, ReLUs, pooling, and classification as successive architectural components.
  • Related Work: REDRAW uses a data-driven CNN and iterative KNN algorithm, distinguishing it from related GUI reverse-engineering approaches.The approach classifies components into types and generates realistic GUI hierarchies using mined GUI information.

3 APPROACH DESCRIPTION

The proposed GUI-prototyping workflow detects components, classifies them using learned GUI data, and assembles a realistic nested hierarchy. Its Android implementation combines computer vision or metadata parsing, CNN classification, and KNN-based hierarchy construction.

  • Detection: Detection identifies GUI-component bounding boxes from mock-up artifacts using computer vision or directly parsed metadata.The detected components are then available for later classification and assembly stages.
  • Classification: A CNN trained on GUI data from large-scale automated application analysis classifies detected components into proper types.The training data is obtained by mining software repositories and dynamically exploring applications.
  • Assembly: An iterative KNN-based algorithm assembles a realistic nested hierarchy of GUI-components and GUI-containers.The algorithm leverages GUI information extracted through large-scale dynamic analysis.

3.1 Phase 1 - Detection of GUI-Components

ReDraw detects atomic GUI components from mock-ups using either embedded metadata or computer-vision analysis, producing bounding boxes and cropped component images for later classification.

  • GUI-component detection infers pixel-based bounding boxes for atomic elements, enabling their images to be cropped for later prototyping stages.
  • Metadata-based detection parses component information from design artifacts such as Photoshop or Sketch files.
  • CV-based detection uses edge detection, dilation, contours, and OCR heuristics to infer component bounding boxes when mock-up metadata is unavailable.
  • ReDraw supports either a screenshot with a Marketch file or a single screenshot as input to component detection.
  • Detection outputs bounding-box coordinates and cropped atomic-component images, while GUI containers and hierarchy construction are deferred to assembly.

3.2 Phase 2 - GUI-component Classification

ReDraw builds a labeled Android GUI-component dataset through repository mining and automated exploration, then trains a CNN on filtered and augmented component images.

  • CNN-based classification uses repository mining, automated dynamic analysis, and cropped GUI-component images labeled with domain-specific types.
  • Automated exploration extracts screenshots and GUI metadata from mined applications, transforming observed components into labeled CNN training data.
  • 8,878 unique APKs were mined from 39 non-game Google Play categories for training, validation, and test corpora.
  • Exploration used systematic DFS navigation with a limit of 50 actions per app to balance execution time and screen coverage.
  • 19,786 unique screens yielded over 431,747 native Android GUI-components and containers, with component frequencies varying across app categories.
  • Data augmentation and synthetic Android apps were used to support underrepresented classes and reduce overfitting to the training set.

3.3 Phase 3 - Application Assembly

ReDraw assembles Android GUI code by constructing a component hierarchy, inferring styles, and generating application resources from an intermediate representation.

  • The assembly process builds a component hierarchy, infers stylistic details from the mock-up, and assembles the application.
  • Deriving GUI-Hierarchies: K-nearest neighbors constructs a hierarchy from detected and classified components by matching them with hierarchy levels mined from existing screens.The procedure iteratively groups matched nodes into parent components until the requested depth is reached or all nodes are grouped.
  • Deriving GUI-Hierarchies: Container attributes, including container type, are inherited during hierarchy construction, while the number of levels can be limited to avoid rendering delays.The resulting hierarchy reflects similarity to existing GUI hierarchies and incorporates container types alongside classified leaf components.
  • Inferring Stylistic Details: Color quantization and color-histogram analysis infer component styling from dominant and secondary pixel colors.ReDraw currently infers background color, font color, and font size; future work could add fonts and shape properties.
  • Inferring Stylistic Details: ReDraw’s style inference is limited to three major categories, although the paper reports these are sufficient for highly visually similar generated applications.
  • Application Generation: ReDraw encodes hierarchy, styles, and OCR-detected strings in a uiautomator-based intermediate representation before generating Android resource XML files and code.

4 EMPIRICAL STUDY DESIGN

The empirical study evaluates ReDraw’s classifier, hierarchy construction, visual similarity, and industrial applicability using mined Android GUI data, baselines, and practitioner interviews.

  • The study evaluates CNN classification accuracy, hierarchy similarity, visual similarity to mock-ups, and suitability for industrial workflows.
  • Study Assumptions: Screenshots and reverse-engineered Sketch mock-ups serve as proxies for real design mock-ups, which may be incomplete or ambiguous.Screenshots represent finalized GUI designs rather than necessarily incomplete real mock-ups.
  • RQ1: 191,300 labeled GUI-component images from 14,382 screens and 6,538 apps support the CNN evaluation after automated analysis and data cleaning.
  • RQ1: Average top-1 classification precision across test classes is used to evaluate the CNN classifier.Precision counts correct top-class predictions as true positives and incorrect top predictions as false positives.
  • RQ2: Hierarchy similarity is compared against ground-truth hierarchies and the REMAUI and pix2code baselines using runtime GUI representations.The study selects 32 apps and compares component and container organization through their uiautomator representations.
  • RQ2: Tree similarity is measured with Wagner–Fischer Levenshtein edit distance using insertion, deletion, and substitution operations with configurable weights.
  • RQ3: Visual similarity compares screenshots from ReDraw-CV, ReDraw-Mockup, pix2code, and REMAUI on the same set of 83 apps.
  • RQ4: Industrial applicability is investigated through semi-structured interviews with an Android developer, an Android UI designer, and a mobile researcher.For each participant, nine screens were selected and reverse-engineered into Sketch mock-ups whose visual fidelity was checked with GVT.

5 EXPERIMENTAL RESULTS

REDRAW’s CNN classifier outperformed the BOVW baseline, while its generated GUI hierarchies and app visuals generally remained close to target applications. Practitioner interviews further indicated promise for industrial workflows, especially evolutionary GUI changes, although workflow-specific modifications may be needed.

  • RQ1 Results: Effectiveness of the CNN: 91.1% overall top-1 precision for REDRAW’s CNN exceeded the BOVW baseline’s 64.7% across 15 Android component classes.The CNN outperformed the baseline in classification precision across all classes.
  • RQ1 Results: Effectiveness of the CNN: ProgressBars and ToggleButtons had lower classification precision because each component type included multiple existing visual styles.ProgressBars included both traditional bars and square progress wheels.
  • RQ2 Results: Hierarchy Construction: REDRAW-MockUp generally produced GUI hierarchies closer to target hierarchies than REMAUI and pix2code across edit-distance penalties.As insertion-operation costs rose, both REDRAW variants outperformed REMAUI; REDRAW-MockUp generally needed fewer than ten edits to exactly match target hierarchies.
  • RQ3 Results: Visual Similarity: All approaches produced prototypes with high overall pixel similarity, while REDRAW and REMAUI outperformed pix2code visually.REDRAW-CV outperformed both REMAUI and pix2code in MAE, whereas REMAUI slightly outperformed both REDRAW variants in MSE.
  • RQ3 Results: Visual Similarity: REDRAW’s visual similarity occasionally fell below REMAUI’s when components were misclassified, such as buttons rendered as switches.REMAUI sometimes preserved visual similarity by copying non-text components as images, but developers would then need to add proper component types.
  • Industrial Practitioner Feedback: Practitioners viewed REDRAW as promising for industrial design and development workflows, particularly for evolutionary changes, but expected adaptations for specific workflows and toolchains.The Google developer emphasized automated boilerplate generation and fast iteration; the Facebook researcher saw prototyping potential but not adoption in the current React Native-centered environment.

6 LIMITATIONS & THREATS TO VALIDITY

REDRAW’s evaluation has practical, validity, and generalization boundaries. Key constraints include single-screen prototyping, device-size dependence, limited style and component coverage, proxy mock-ups, and Android-specific data requirements.

  • Practical limitations: REDRAW currently prototypes only one screen at a time, so multi-screen applications require separate prototyping and manual combination.Future work proposes learning common screen transitions through dynamic analysis.
  • Practical limitations: The KNN hierarchy-construction implementation is tied to the screen sizes used during data mining and automated dynamic analysis.Display-independent pixels are proposed as a future route toward screen-size independence.
  • Practical limitations: REDRAW currently infers only background colors, font colors, and font sizes from mock-ups, despite producing visually similar prototypes with these details.Future work could infer additional component-level visual properties.
  • Practical limitations: The CNN classifies images into 15 popular Android GUI-component types, leaving rarely used types unsupported.Deeper or hierarchical CNNs and joint object-detection approaches are proposed for broader coverage.
  • Internal validity: Evaluation mock-ups included existing application screenshots and reverse-engineered mock-ups, which may not capture unique characteristics of real design-process artifacts.This creates a threat to internal validity because these artifacts serve as proxies for real mock-ups.
  • Internal validity: The labeled CNN dataset may include developer-styled components that visually resemble other component types, although experiments suggest such cases are rare.Filtering, cleaning, augmentation, manual inspection, and uiautomator-based metadata were used to support dataset quality.
  • Construct validity: A correct Android GUI hierarchy is not empirically defined and may vary with developers’ layout preferences, limiting interpretation of hierarchy comparisons.Developers may prefer different numbers of container components or different layout structures.
  • External validity: REDRAW was implemented for Android, and other platforms may require additional engineering if GUI metadata is not readily accessible.The approach’s generalization to other domains was not measured.

7 CONCLUSION & FUTURE WORK

The paper presents REDRAW, an Android implementation of a data-driven approach for automatically prototyping software GUIs. Evaluation indicates accurate component processing, similar hierarchies and visuals, and positive effects on industrial workflows, while future work targets broader integration and object detection.

  • Conclusion: REDRAW implements a data-driven approach for automatically prototyping software GUIs on Android.The approach is presented as a tool implementation for automated GUI prototyping.
  • Conclusion: Evaluation found that REDRAW accurately detects and classifies GUI-components in mock-up artifacts.This is one of four capabilities highlighted in the conclusion.
  • Conclusion: REDRAW generates GUI hierarchies similar to those a developer would create and applications visually similar to mock-up artifacts.The conclusion reports both structural and visual similarity outcomes.
  • Conclusion and future work: The evaluation reports that REDRAW positively impacts industrial workflows.Future work includes integrating REDRAW and the broader prototyping approach with industrial partners.
  • Future work: Future work will explore CNN architectures for object detection and collaborate with industrial partners on workflow integration.Object detection is intended to better support the detection task.
Loading 1802.02312v2…