PaperlayerView exact revision on arXiv

Source-linked AI summary

StoryDroid: Automated Generation of Storyboard for Android Apps

Sen Chen, Lingling Fan, Chunyang Chen, Ting Su, Wenhe Li, Yang Liu, Lihua Xu

arXiv:1902.00476v2cs.SE

TL;DR

Reviewing similar apps requires repeated manual exploration. STORYDROID generates Android app storyboards by extracting activity transitions, rendering UI pages, and inferring activity names. It reports more transitions than IC3 and higher activity coverage than STOAT, with limitations in handling some dynamically loaded activities and UI components.

  • Problem

    Reviewing similar apps requires downloading, installing, and repeatedly using them to identify relevant information.

    Source: §I · p. 1

  • Method

    STORYDROID extracts activity transitions, statically renders UI pages, and infers semantic names for obfuscated activities.

    Source: §I · p. 2, A. Transition Extraction · p. 4

  • Results

    STORYDROID extracts nearly 2 times more activity transitions than IC3 on both app types and averages 87% open-source and 74% closed-source activity coverage, outperforming STOAT.

    Source: §I · p. 2, B. Experimental Results · p. 7

  • Takeaways & Limitations

    The storyboard gives PMs, designers, and developers an overview of similar apps and links UI pages with implementation code.

    Source: §I · p. 2

    Static analysis cannot handle activities dynamically loaded by other components, and some dynamically created UI components are rendered as empty placeholders.

    Source: B. Limitations · pp. 9–10

Abstract

from arXiv · show

Mobile apps are now ubiquitous. Before developing a new app, the development team usually endeavors painstaking efforts to review many existing apps with similar purposes. The review process is crucial in the sense that it reduces market risks and provides inspiration for app development. However, manual exploration of hundreds of existing apps by different roles (e.g., product manager, UI/UX designer, developer) in a development team can be ineffective. For example, it is difficult to completely explore all the functionalities of the app in a short period of time. Inspired by the conception of storyboard in movie production, we propose a system, StoryDroid, to automatically generate the storyboard for Android apps, and assist different roles to review apps efficiently. Specifically, StoryDroid extracts the activity transition graph and leverages static analysis techniques to render UI pages to visualize the storyboard with the rendered pages. The mapping relations between UI pages and the corresponding implementation code (e.g., layout code, activity code, and method hierarchy) are also provided to users. Our comprehensive experiments unveil that StoryDroid is effective and indeed useful to assist app development. The outputs of StoryDroid enable several potential applications, such as the recommendation of UI design and layout code.

I. INTRODUCTION

Manual review of similar apps can be painstaking and makes it difficult to understand app navigation and connect screens to implementation code. STORYDROID generates visual storyboards with UI pages and code links to support review by app-development teams.

I. INTRODUCTION

  • Manual exploration is painstaking, and complex apps can be difficult to analyze fully while designers and developers struggle to understand screen navigation and locate corresponding code.The code may be split between static layout files and functional code.Source: §I · p. 1
  • The storyboard combines an activity transition graph and UI pages with layout code, activity code, and method-call relations.This gives PMs, designers, and developers different ways to review existing apps.Source: §I · p. 2
  • STORYDROID extracts activity transitions, statically renders UI pages, and infers semantic names for obfuscated activities.Its approach addresses incomplete transition graphs, dynamically rendered UI, and unclear obfuscated activity names.Source: §I · p. 2
  • STORYDROID extracts nearly 2 times more activity transitions than IC3 on both app types and outperforms STOAT in activity coverage.It averages 87% coverage on open-source apps and 74% on closed-source apps.Source: §I · p. 2
  • The authors present STORYDROID as a first step toward a large-scale storyboard database and potential recommendations for UI design, layout code, and app testing.They also report that its outputs can support app-development teams.Source: §I · p. 2

II. MOTIVATING SCENARIO

STORYDROID supports app review by presenting related apps as storyboards that help product managers identify common and distinctive features. It also gives designers UI and interaction references and developers access to corresponding implementation code.

II. MOTIVATING SCENARIO

  • Product managers can compare storyboards of similar apps to spot common features and identify distinctive features for their planned app.The example covers apps found through Google Play searches and lists registering, searching, settings, profiles, and posting as common features.Source: §II · p. 2
  • Designers can use related apps’ UI styles and screen interactions as references for their own interface and interaction design.The scenario connects the designer’s work to the product manager’s requirements.Source: §II · p. 2
  • Developers can click a storyboard screen to retrieve its corresponding static, dynamic, or hybrid UI implementation code.This lets developers refer to the implementation underlying a UI design.Source: §II · p. 2

III. PRELIMINARIES

The preliminaries describe Android UI components and layouts, including how activities and fragments compose screens, how layouts are implemented, and how adapters populate views. The approach overview introduces StoryDroid’s three phases: transition extraction, UI rendering, and semantic-name inference.

A. Android Activity and Fragment

  • Activities draw interactive screens, while fragments contribute reusable UI portions that depend on an activity and can support multi-panel layouts.An activity may contain multiple fragments, and a fragment may be reused across activities.Source: A. Android Activity and Fragment · p. 3

B. UI Layout

  • Android layouts can be static XML, dynamically built in Java, or hybrid layouts that inflate XML and permit runtime attribute changes.ViewGroups contain Views and form the layout hierarchy; developers can also modify component attributes in code.Source: B. UI Layout · p. 3
  • 62.3% of a sampled 1,000 top-download apps used dynamic or hybrid layouts, which separate views from the view model and allow runtime adaptation without recompilation.The sample was drawn from the top 10,000 Google Play apps by downloads.Source: B. UI Layout · p. 3

C. Data Population

  • Adapters connect data to views such as ListView, GridView, and ViewPager, enabling scrollable items to be displayed in the UI.The adapter links an AdapterView to underlying data, commonly loaded from a local database or remote server.Source: C. Data Population · p. 3

IV. OUR APPROACH

  • StoryDroid takes an APK and extracts transitions, renders UI pages by translating dynamic or hybrid layouts when needed, and infers semantic names for obfuscated activities.The transition extraction enhances IC3, especially for fragments and inner classes.Source: §IV · pp. 3–4

A. Transition Extraction

StoryDroid statically builds an activity transition graph by analyzing calls, intents, fragments, and inner classes. It also records adapter-to-layout mappings and layout types for UI rendering.

A. Transition Extraction

  • Algorithm 1 takes an APK and outputs an activity transition graph, adapter mappings, and a layout type determined by analyzing dynamic layout loading.The graph is built from activity transitions identified through call-graph and intent analysis.Source: A. Transition Extraction · p. 4
  • Fragment-to-activity relations are refined into actual activity transitions by identifying activities that start each fragment and merging the relations.For example, PrefEditor→PrefEditorFragment and PrefEditorFragment→AdvancedPrefEditor become PrefEditor→AdvancedPrefEditor.Source: A. Transition Extraction · pp. 4–5
  • For transitions in inner classes, StoryDroid attributes the launch to the outer activity; for other methods, it backward-traverses the call graph to find launching activities.The examples include SearchPanel→PartList and transitions initiated outside fragments or inner classes.Source: A. Transition Extraction · pp. 4–5
  • StoryDroid identifies adapter-backed views and traces their adapters to layout files, recording each mapping as ⟨activity, view type, layout⟩ for UI rendering.For example, it maps PartList, ListView, and list view.Source: A. Transition Extraction · p. 5

B. UI Page Rendering

StoryDroid renders initial UI states using static layouts when available and converts dynamic or hybrid layouts into renderable static layouts. It uses dummy data for dynamically loaded content, then builds an APK and captures screenshots of activities and fragments.

B. UI Page Rendering

  • For data-dependent pages, StoryDroid renders initial activity states because dynamic tools may not reach activities requiring login or special input.Static layouts are extracted directly; dynamic or hybrid layouts need conversion and dummy data where backend-loaded content cannot be obtained.Source: B. UI Page Rendering · p. 5
  • Dynamic and hybrid layouts are converted to static layouts by identifying added views, their parent layouts, and attributes through source-code and data-flow analysis.Attributes are resolved through method definitions or resource elements before the modified parent layout is attached to the activity layout.Source: B. UI Page Rendering · p. 5
  • StoryDroid embeds adapter-backed view types and dummy data into layouts, builds an APK, and captures screenshots for activities and fragments.Dummy images preserve the positions of dynamically loaded images, and dummy data is associated with identified adapters.Source: B. UI Page Rendering · pp. 5–6

C. Semantic Name Inferring

StoryDroid infers semantic names for obfuscated activities by matching their layout hierarchies against a database of open-source app layouts, then ranking candidate names.

C. Semantic Name Inferring

  • After manually examining 6,767 activity names from 1,000 F-Droid apps, the authors observed most had semantic meanings; they classify names shorter than three letters as obfuscated and infer semantic names from open-source apps.Source: C. Semantic Name Inferring · p. 6
  • The method compares an obfuscated activity’s layout tree with database trees using tree edit distance, retaining candidates with a distance below 5.The database contains 13,792 activity layout hierarchies from 4,426 F-Droid apps; the threshold was set using a pilot study of 100 tree pairs.Source: C. Semantic Name Inferring · p. 6
  • It ranks candidates by activity-name frequency and layout-name keyword matches, falling back to the most frequent candidate when no layout name matches.Layout names are insufficient alone because activities using dynamic layouts may have no static layout names.Source: C. Semantic Name Inferring · p. 6

V. IMPLEMENTATION

StoryDroid combines existing analysis and decompilation tools with a D3 visualization to produce storyboard views and code mappings.

V. IMPLEMENTATION

  • StoryDroid uses IC3 for activity-transition extraction, SOOT for rendering inputs and call graphs, and JADX to decompile apps and extract layout code.The implementation comprises 3K lines of Java and 2K lines of Python.Source: §V · p. 6
  • Its visualization presents the activity transition graph with activity names and UI pages, alongside each page’s layout code and activity code.The visualization is built with D3 using HTML, JavaScript, and CSS.Source: §V · p. 6
  • The storyboard also displays method-call relations within each activity.JADX extracts XML layout code that the passage says is unaffected by obfuscation.Source: §V · p. 6

VI. EFFECTIVENESS EVALUATION

The evaluation asks whether StoryDroid extracts more complete activity-transition graphs, renders pages similar to real screenshots, and infers accurate names for obfuscated activities.

VI. EFFECTIVENESS EVALUATION

  • RQ1 asks whether StoryDroid extracts a more complete activity-transition graph and achieves better activity coverage than STOAT.The section frames this as an effectiveness evaluation.Source: §VI · p. 6
  • RQ2 asks whether StoryDroid’s rendered UI pages are highly similar to real screenshots.Source: §VI · p. 6
  • RQ3 asks whether StoryDroid can infer accurate semantic names for obfuscated activities.Source: §VI · p. 6

A. Experimental Setup

The evaluation tests StoryDroid on controlled and real-world apps, comparing transition extraction, activity coverage, rendered-page similarity, and semantic-name inference. Results report more transitions than IC3, greater activity coverage than STOAT, 84%/88% image similarity, and 92% naming accuracy.

A. Experimental Setup

  • StoryDroid’s controlled benchmark uses 10 self-developed apps to test transitions involving activities, fragments, and inner classes; real-app evaluation includes 50 popular Google Play apps with 10M+ installations.Table I reports the capability evaluation for handling fragments and inner classes.Source: A. Experimental Setup · pp. 6–7, Table I · p. 7
  • The real-app evaluation compares transition pairs with IC3 and activity coverage with STOAT; UI similarity uses STOAT screenshots, while name inference is checked against 100 known semantic names.For similarity, STOAT receives 30 minutes per app, and only activities it explores are compared.Source: A. Experimental Setup · p. 7

B. Experimental Results

  • STORYDROID extracts nearly 2 times more activity transitions than IC3 on both open-source and closed-source apps.The broader remark also reports that it covers 2 times more activities than STOAT with less time.Source: B. Experimental Results · pp. 7–8, Figure 9 · p. 7
  • STORYDROID achieves 87% activity coverage on open-source apps and 74% on closed-source apps, extracting and rendering activities within 3 minutes on average versus STOAT’s 30 minutes.Fig. 9(b) depicts the activity-coverage results.Source: B. Experimental Results · p. 7, Figure 9 · p. 7
  • STOAT’s low coverage is attributed to login requirements and unsupported specific events, while StoryDroid can miss activities when reverse engineering fails or activities have no transitions.The authors note that reverse-engineering failures are more severe in closed-source apps due to packing and obfuscation.Source: B. Experimental Results · p. 7
  • STORYDROID’s rendered images achieve 84% similarity by MAE and 88% by MSE on average.The comparison uses pixel-by-pixel image similarity against UI pages dynamically rendered by STOAT.Source: B. Experimental Results · pp. 7–8, Figure 10 · p. 8
  • Rendered-page similarity is limited by dynamic data and input-dependent UI states, because StoryDroid renders each activity’s initial state.The authors say this reduced similarity does not affect understanding of app functionality.Source: B. Experimental Results · p. 8
  • STORYDROID correctly infers 92 out of 100 activity names, for 92% accuracy.Among 10 illustrated cases, it correctly infers 8; fallback to the top-ranked name when layout names do not match candidates accounts for two mismatches.Source: B. Experimental Results · p. 8, Table II · p. 8

VII. USEFULNESS EVALUATION

A user study compared app exploration and UI-page-to-layout-code mapping with and without STORYDROID, finding greater coverage and faster mapping with the tool.

VII. USEFULNESS EVALUATION

  • The study used 4 finance and tool apps with 12–15 activities each and 8 experienced Android users who had not used the apps before.Participants explored apps with and without STORYDROID in a rotated, Latin-square order and completed 10-minute tasks.Source: §VII · p. 8
  • STORYDROID achieved 2 times more activity coverage in an average of 2.5 minutes, versus 40.8% manual coverage; participants rated exploration satisfaction 4.4 versus 4.2.The activity-coverage results are shown in Table III.Source: §VII · p. 9, Table III · p. 9
  • With STORYDROID, participants confirmed UI-page-to-layout-code mappings in 30 seconds, while unaided mapping took over 8 minutes and left 4 pages unmapped within 10 minutes.The comparison is reported in Table IV; differences in both study tables were significant at p < 0.01 or p < 0.05.Source: §VII · p. 9

VIII. DISCUSSION

STORYDROID’s outputs may support searchable UI and code recommendations as well as regression-test updates by linking app screens to their implementation.

A. Future Applications Based on STORYDROID

  • A database linking activity names, UI images, and layout code could let designers search for UI examples and developers find similar screens and customize their associated code.The proposed search uses activity-name keywords for UI images and image similarity for matching a supplied design to candidate screens.Source: A. Future Applications Based on… · p. 9
  • Comparing layout code and activity transition graphs across app versions could identify modified components and guide test-case updates while reusing most existing tests.The discussion proposes this as a potential application of STORYDROID’s UI-page-to-layout-code mappings.Source: A. Future Applications Based on… · p. 9

B. Limitations

The discussion identifies limits in static extraction and dynamic UI rendering, positions STORYDROID against related tools, and outlines broader applications and future platform extensions.

B. Limitations

  • Static extraction can have incomplete call graphs or failed file transformations and decompilations, and cannot handle activities dynamically loaded by other components.The authors say such transformation and decompilation failures rarely appear in real apps.Source: B. Limitations · p. 9
  • For runtime-dependent UI components, STORYDROID uses empty placeholders, preserving the overall layout but not recovering the component’s actual values.The example is BankDroid’s CreateFrom() function, whose parameter depends on another user-defined method’s return value.Source: B. Limitations · p. 10

IX. RELATED WORK

  • Unlike prior UI-implementation studies that generate static layouts, STORYDROID provides original static, dynamic, or hybrid UI code alongside transitions; it can complement screenshot-based app analysis with broader static page extraction.The related-work comparison describes screenshot collection through dynamic execution as time-consuming and low-coverage.Source: §IX · p. 10
  • GUIFECTCH supports UI transitions but only handles open-source apps and combines static and dynamic analysis, which can miss screens.STORYDROID is described as also reverse-engineering closed-source apps through static analysis.Source: §IX · p. 10

X. CONCLUSION AND FUTURE WORK

  • The conclusion proposes a large-scale storyboard database for UI-page and code recommendations, with future work extending the approach to iOS apps and desktop software.The database is intended to connect app activities, UI pages, and implementation code.Source: §X · p. 10
Loading 1902.00476v2…