PaperlayerView exact revision on arXiv

Source-linked AI summary

UI Remix: Supporting UI Design Through Interactive Example Retrieval and Remixing

Junling Wang, Hongyi Lan, Xiaotian Su, Mustafa Doga Dogan, April Yi Wang

arXiv:2601.18759v2cs.HC

TL;DR

End users often struggle to articulate UI intent and trust design choices, while existing example-based tools can produce either overwhelming exploration or narrow fixation. UI Remix uses multimodal retrieval-augmented generation to support iterative global and local remixing of traceable UI examples. In a study with 24 participants, it improved goal achievement, iteration, and exploration, while transparency cues increased confidence in adapting examples.

  • Problem

    End users struggle to articulate implicit design preferences and evaluate or trust design choices, while existing example-based tools can overwhelm users, cause design drift, or encourage fixation.

  • Method

    UI Remix uses multimodal retrieval-augmented generation to let end users iteratively search, select, and adapt traceable UI examples globally or at the component level.

  • Results

    UI Remix significantly improved 24 participants’ ability to achieve design goals, iterate effectively, and explore alternative UI designs; transparency cues enhanced confidence in adapting examples.

  • Takeaways & Limitations

    Example-driven interaction with transparent sources can support end users’ control, trust, and openness to exploration during mobile UI design.

  • Takeaways & Limitations

    The evaluation was limited to short-term, task-based interactions and did not isolate the individual contributions of retrieval and remixing.

Abstract

from arXiv · show

Designing user interfaces (UIs) is a critical step when launching products, building portfolios, or personalizing projects, yet end users without design expertise often struggle to articulate their intent and to trust design choices. Existing example-based tools either promote broad exploration, which can cause overwhelm and design drift, or require adapting a single example, risking design fixation. We present UI Remix, an interactive system that supports mobile UI design through an example-driven design workflow. Powered by a multimodal retrieval-augmented generation (MMRAG) model, UI Remix enables iterative search, selection, and adaptation of examples at both the global (whole interface) and local (component) level. To foster trust, it presents source transparency cues such as ratings, download counts, and developer information. In an empirical study with 24 end users, UI Remix significantly improved participants' ability to achieve their design goals, facilitated effective iteration, and encouraged exploration of alternative designs. Participants also reported that source transparency cues enhanced their confidence in adapting examples. Our findings suggest new directions for AI-assisted, example-driven systems that empower end users to design with greater control, trust, and openness to exploration.

1 Introduction

UI Remix addresses end users’ difficulty articulating UI intent and trusting design choices through traceable, example-driven exploration. Its MMRAG workflow supports iterative whole-interface and component-level remixing, and a 24-participant study found improved goal achievement, iteration, and exploration.

  • End users often struggle to translate abstract visual preferences into concrete UI choices and to judge or trust AI-generated designs.Design literacy limitations can make users hesitant to commit to or justify aesthetic decisions.
  • UI Remix frames design as iterative search, selection, and adaptation of traceable UI examples.The system helps users articulate design ideas through examples whose sources, context, and quality indicators are visible.
  • The MMRAG workflow retrieves examples from text queries and selected UI components, supporting both global and local remix.Global remix adapts styles, layouts, colors, or other attributes across the current interface; local remix targets specific components.
  • Source cues including ratings, download counts, categories, and developer information make example context visible for relevance assessment and trust.The system focuses on early-stage prototyping of static single-screen UIs, despite technical support for interactive and multi-screen interfaces.
  • 24 participants showed significantly improved ability to achieve design goals, iterate effectively, and explore alternative designs with UI Remix.Participants also reported greater willingness to use the system and enhanced trust from source transparency cues.

2 Related Work

Prior UI-design tools support inspiration, generation, and guidance, but often leave end users to articulate intent precisely and evaluate opaque suggestions. UI Remix shifts toward retrieval-augmented, example-driven exploration with transparent source information for more interpretable design decisions.

  • Using Examples in UI Designs: Gallery-based repositories offer broad visual exploration but require extensive manual searching and can overwhelm users or cause design drift.Examples include Mobbin, Dribbble, and Behance, which archive diverse interface styles.
  • Using Examples in UI Designs: Retrieval and generative systems can automatically present examples, yet they depend on clear descriptions and may encourage design fixation.This dependence is particularly challenging for end users who struggle to define their expectations.
  • Supporting End Users in UI Designs: End-user UI research lowers entry barriers through accessible interactions such as direct manipulation, natural language, and multimodal instructions.Systems such as Stylette and ProgramAlly support restyling or customization of existing interfaces.
  • AI Support for UI Designs: AI-assisted recommendation systems provide design guidance, including real-time feedback on prototypes and conversion of sketches into runnable wireframes.These approaches aim to preserve user control and interpretability while supporting iterative refinement.
  • AI Support for UI Designs: UI Remix combines multimodal retrieval, remixing, and end-user targeting while making example sources transparent through ratings, developer information, and download counts.This shifts the workflow from generation-only interaction toward example-driven articulation, refinement, and justification.

3 UI Remix System Design

UI Remix is designed to scaffold vague design intent through iterative retrieval and remixing of traceable examples. Its flexible workflow combines broad inspiration, component-level refinement, and user-controlled interaction across complementary modes.

  • UI Remix targets end users who need help turning vague or intuitive preferences into concrete design goals.Users gradually form intent by iteratively searching for relevant examples and adapting them to their own UIs.
  • The system presents retrieved examples with ratings, download counts, categories, and developer information so users can assess relevance and context.This design follows social transparency principles intended to make others’ identities, intentions, and actions visible.
  • Global remix supports broad inspiration, while local remix enables targeted refinement of specific components.Together they balance breadth and focus while preserving users’ control over their evolving design direction.
  • The interface combines a conversation panel, example gallery, and editable canvas for querying, browsing, previewing, and directly refining UI components.The example-guided workflow is powered by an MMRAG model.
  • Chat, search, and apply interactions form complementary modes rather than a fixed linear sequence.Users can switch modes, with interactions recorded in the Conversation Panel to support transparency and traceability.

3.3 Target Users and System Walkthrough

UI Remix is intended for nonprofessional creators who design mobile interfaces as a means to communicate, promote, or prototype. A walkthrough shows users moving from natural-language generation to global style adaptation and local component refinement through repeated remixing.

  • The target users are nonprofessional creators such as small business owners, students, and hobbyists designing UIs for practical or expressive goals.They engage in interface design as a means to an end rather than as their primary occupation.
  • A user can begin with a natural-language description that generates an initial UI and preview in the editable canvas.The walkthrough uses a restaurant owner creating a mobile app with a menu section.
  • Global remix retrieves semantically aligned real-world UIs and applies selected attributes, such as a color scheme, to the current design.Users consult ratings, download counts, and developer information when prioritizing examples.
  • Local remix retrieves variations for a selected component and applies a chosen style to that component.The walkthrough refines a reservation button using a rounded red button from an existing UI example.
  • Several rounds of global and local remix let users iteratively refine the UI while balancing creative freedom with example guidance.The walkthrough presents this process as enabling novice users to design with confidence.

3.4 Global Remix

Global Remix retrieves diverse mobile UI examples matching users’ queries, then lets them select and adapt an example to modify the current design.

  • Users describe their intended interface in natural language, and UI Remix retrieves diverse examples ranked by semantic similarity.The retrieved examples appear in the Example Gallery alongside traceability metadata, including ratings, download counts, comment counts, and categories.
  • After selecting an example, users request adaptations such as changes to the color scheme or layout through the Conversation Panel.The system automatically switches to Apply mode, generates corresponding HTML/CSS code, and displays a live preview on the Editable Canvas.

3.5 Local Remix

Local Remix supports component-level refinement by retrieving variations of a queried UI element and applying a selected variation to a specified location in the current design.

  • Users query for a specific component variation, such as a stylish red button, triggering retrieval from existing UI examples.The component retrieval process uses the same backend pipeline as global UI retrieval and presents variations in the Example Gallery.
  • Users select a component variation and identify where to apply it by clicking the corresponding element in the Editable Canvas.They can further adapt the variation with natural-language commands or highlight the specific portion they want to use.

3.6 Technical Architecture

UI Remix combines multimodal retrieval and generation across Chat, Search, and Apply modes, using curated mobile UI data, similarity-based retrieval, and code patches to support interactive remixing.

  • Architecture: UI Remix uses an MMRAG framework with Chat, Search, and Apply modes organized around retriever and generator modules.The system is designed to retrieve examples and generate adapted UI code during interaction.
  • Data Source: The example corpus combines Mobbin, Interaction Mining, and MobileViews screenshots with application metadata collected from the Google Play Store.MobileViews includes more than 600K screenshot-view hierarchy pairs from over 20,000 applications.
  • Data Source: The system selected 196 popular applications across diverse categories and collected around 900 unique interface screenshots as its example set.The authors note that not all app categories were covered because of resource constraints, although the pipeline can scale to larger datasets.
  • Pipeline: Figure 2 shows screenshots being indexed with metadata, embedded, stored in ChromaDB, and retrieved alongside images using cosine similarity.Selected examples, user queries, current UI code, and an optional local-remix target area are sent to the MLLM, which returns diff patches rendered as previews.
  • Interaction Modes: Chat combines the user query with current UI code, while the MLLM generates diff patches that update the existing design.The backend applies the generated patch to the current UI code rather than regenerating the entire codebase.
  • Interaction Modes: Search retrieves the top 10 screenshots by cosine similarity and presents them with associated metadata in the Example Gallery.Apply mode supports global or local remix depending on the scope of the requested modification.
  • Technical Evaluation: The technical evaluation used 100 automatically generated queries spanning color theme, layout, UI category, and component-design intents.This setup tested whether textual descriptions aligned with corresponding visual or structural features in the corpus.
  • Technical Evaluation: 88% of queries retrieved at least one relevant UI in the top five, with average Hit@5 of 0.88 and nDCG@5 of 0.77 across query types.The retriever performed particularly well on color and category queries, indicating that the embeddings captured stylistic and semantic aspects of UI design.

4 User Study

The user study compared UI Remix with a text-only baseline among 24 non-expert participants, finding benefits for design support while limiting evaluation to short, static single-screen tasks.

  • Study Design: The study asked how traceable UI examples affect end users’ goal achievement, design iteration, alternative exploration, and trust.It used a within-subjects comparison in which 24 participants designed two mobile UIs, one with each system.
  • Study Design: The evaluation focused on static single-screen prototyping because the study duration emphasized early exploration and ideation before interactivity.The authors state that sustained or deep refinement was left for future investigation.
  • Conditions: The GPT-Canvas baseline used the same MLLM as UI Remix but omitted the Example Gallery and its search and apply modes.This comparison isolated the influence of example retrieval and remixing features on design performance and perceptions.
  • Participants: Participants had little or no mobile UI design expertise, and the sample included 19 men and 5 women with an average age of 25.Participants varied in their familiarity with AI tools for interface design, from no prior use to several uses per week.
  • Procedure: The study used two 25-minute mobile UI tasks, balanced the order of assisting methods with a Latin Square design, and collected questionnaires, interviews, designs, logs, and expert ratings.The tasks involved creating a hamburger-restaurant menu page and a news-display page.
  • Results: Beta took 15.76 minutes on average versus 12.41 minutes for Alpha, a difference of approximately 3 minutes despite significant completion-time differences.Task type had no significant effect on completion time, with t = 0.45 and p = 0.66.
  • Results: Expert ratings assessed 48 UIs on aesthetic quality, layout quality, and accuracy using a seven-point Likert scale.The rubric focused on static end results and may not capture differences in design intent or rationale during interaction.

5 Results

UI Remix supported creative exploration, goal-oriented iteration, and trust during mobile UI design by combining real-world examples with retrieval and remixing. Participants also varied in how much they relied on source metadata.

  • Creative exploration: Beta significantly increased exploration of alternative design options and willingness to use the system for other creative tasks.Both measures were significantly higher than Alpha (p< .001).
  • Creative exploration: Participants used examples to spark inspiration, apply existing styles, learn from best practices, and move from abstract intentions to tangible visual ideas.Nine participants explicitly valued Beta’s example-driven nature.
  • Goal-oriented iteration: Beta significantly improved effective iteration and support for achieving design goals.Participants searched 3.75 times, used global remix 1.33 times, and used local remix about 3 times per task on average.
  • Goal-oriented iteration: Global and local remix supported complementary levels of refinement, with local remix targeting components while preserving the broader design.Participants described examples as useful reference points for starting and iterating.
  • Trust and confidence: Beta received significantly higher willingness-to-use ratings, while 17 of 24 participants actively examined source transparency cues.The cues included ratings, download counts, developer information, app name, and category.
  • Trust and confidence: Seven participants reported that metadata did not significantly influence their decisions, preferring to focus on visual aspects.Reliance on transparency cues therefore varied across participants.

6 Discussion and Future Work

UI Remix’s example-driven workflow helped users articulate vague design goals, build confidence through transparent provenance, and iteratively move between broad exploration and detailed refinement. The discussion recommends reflective, ethically grounded AI design support while noting limits in output quality, study scope, and component isolation.

  • Significance of example-driven design: Real-world examples helped participants externalize vague goals by comparing designs, refining preferences, and verbalizing previously implicit choices.Participants used concrete references to explain preferences such as simpler layouts.
  • Significance of example-driven design: Source transparency cues supported perceived trust and design confidence by helping users evaluate the credibility of retrieved designs.The cues included ratings, developer information, and download counts.
  • Progressive articulation: Retrieval-remix cycles enabled users to explore alternatives and make more confident adjustments through reciprocal feedback loops.Global remix supported whole-interface adaptation, while local remix supported component-level refinement.
  • Progressive articulation: The authors frame AI as a reflective collaborator that scaffolds understanding rather than merely generating designs from fully specified instructions.This framing emphasizes reflective efficacy: helping users examine and refine their ideas.
  • Design takeaways: Design takeaways include using examples as articulation aids, making provenance visible, and supporting iterative movement between exploration and refinement.Global and local remix provide complementary interaction modes across abstraction levels.
  • Design takeaways: The paper recommends valuing reflective efficacy alongside generative fidelity in future example-driven AI systems.Reflective efficacy concerns how effectively systems help users interpret, evaluate, and refine their ideas.
  • Ethics and limitations: UI Remix relies on publicly available UI repositories, requiring attention to licensing agreements and data protection regulations.Ethical assistance should support learning and reflection rather than direct imitation.
  • Limitations and future work: The evaluation focused on static single-screen mobile UIs in a single-session lab study, limiting conclusions about interactive, multi-screen, naturalistic, or longitudinal use.The study was intentionally bounded by time constraints and experimental control.

7 Conclusion

UI Remix supports end users in designing mobile UIs through an example-driven workflow grounded in multimodal retrieval-augmented generation. A 24-participant study found improved goal achievement, iteration, exploration, and confidence in adapting examples.

  • UI Remix supports mobile UI design through an example-driven workflow grounded in multimodal retrieval-augmented generation.
  • Users can iteratively search, select, and adapt real-world UI examples to refine their designs.
  • Source transparency cues, including ratings, download counts, and developer information, help users make more informed and trustworthy design choices.
  • 24 participants reported improved design-goal achievement, effective iteration, exploration of alternative designs, and greater confidence when adapting examples.

8 GenAI Usage Disclosure

The authors disclose limited use of AI assistants for paraphrasing, grammar correction, and code debugging. They state that these tools supplemented rather than replaced human input and expertise.

  • The authors used ChatGPT and Grammarly for minor paraphrasing, grammar correction, and code debugging.
  • The AI tools were applied minimally to support the research process while preserving the authenticity of the work.
  • The authors state that AI tools supplemented rather than replaced human input and expertise.

A GPT-Canvas baseline Configurations

The GPT-Canvas baseline uses the same multimodal language model as UI Remix and provides chat, UI preview, and code-view functionality. It omits UI Remix's Example Gallery and its search and apply modes.

  • The baseline uses the same MLLM as UI Remix, with a left chat panel and a right panel for UI preview and code viewing.
  • Unlike UI Remix, the baseline excludes the Example Gallery and the corresponding search and apply modes.
  • Figure 3 presents the GPT-Canvas baseline system used for comparison.
Loading 2601.18759v2…