PaperlayerView exact revision on arXiv

Source-linked AI summary

Generative and Malleable User Interfaces with Generative and Evolving Task-Driven Data Model

Yining Cao, Peiling Jiang, Haijun Xia

arXiv:2503.04084v1cs.HC

TL;DR

Existing application-centric and code-generation approaches make it difficult to create interfaces that adapt to diverse, evolving information tasks and remain controllable by end-users. The paper instead uses LLM-generated, task-driven data models mapped to UI specifications, with Jelly supporting natural-language and direct-manipulation changes; technical and user evaluations demonstrate feasibility and personalized, dynamic information spaces.

  • Problem

    Application-centric and code-generation approaches make it difficult for end-users to iteratively tailor interfaces as their information needs evolve.

  • Method

    The approach uses LLMs to generate task-driven data models from prompts, maps them to UI specifications, and translates natural-language or direct-manipulation changes into model updates.

  • Results

    Technical and user evaluations demonstrate feasibility, with users developing personalized and dynamic information spaces through flexible information curation and presentation customization.

  • Takeaways & Limitations

    Evolving task-driven data models provide a foundation for generative and malleable UIs supporting relatively open-ended information tasks.

  • Takeaways & Limitations

    The paper focuses on task representation, UI generation, and model evolution, leaving data integration and context awareness beyond its scope.

Abstract

from arXiv · show

Unlike static and rigid user interfaces, generative and malleable user interfaces offer the potential to respond to diverse users' goals and tasks. However, current approaches primarily rely on generating code, making it difficult for end-users to iteratively tailor the generated interface to their evolving needs. We propose employing task-driven data models-representing the essential information entities, relationships, and data within information tasks-as the foundation for UI generation. We leverage AI to interpret users' prompts and generate the data models that describe users' intended tasks, and by mapping the data models with UI specifications, we can create generative user interfaces. End-users can easily modify and extend the interfaces via natural language and direct manipulation, with these interactions translated into changes in the underlying model. The technical evaluation of our approach and user evaluation of the developed system demonstrate the feasibility and effectiveness of the proposed generative and malleable UIs.

1 Introduction

The paper addresses fragmented application-centric workflows and the difficulty of iteratively customizing code-generated interfaces. It proposes evolving task-driven data models as an interpretable foundation for generative and malleable UIs, implemented in Jelly and evaluated technically and with users.

  • Motivation: Application-centric interfaces force users to combine many applications, creating fragmented workflows that may not accommodate diverse and evolving information needs.The dinner-party example requires separate tools for scheduling, invitations, dishes, wine, shopping, and routing.
  • Motivation: Code generation can produce custom applications from prompts but makes iterative modification difficult because revisions may be discontinuous and underlying data transformations unclear.The approach also leaves end-users facing opaque mappings between prompts and code.
  • Approach: The proposed approach uses evolving task-driven data models to represent users’ tasks and drive transformations of both interfaces and underlying data.The model is positioned between user intentions and generated interface components and composition.
  • Approach: Jelly generates an object-relational schema and dependency graph from prompts, maps the model to UI patterns and rules, and supports natural-language and direct-manipulation updates.User interactions are translated into changes to the underlying model.
  • Evaluation: Technical and user evaluations demonstrate feasibility, while users developed personalized and dynamic information spaces by curating information and customizing its presentation.The user study compared experiences with existing GUI applications and AI-powered chat interfaces.

2 Research Framing and Scope

The paper focuses on representing tasks, generating interfaces, and evolving models, using Jelly to illustrate how one open-ended activity can shift across connected information needs. Full data integration and context awareness remain outside this paper’s scope.

  • Scope: The long-term vision targets dynamic, personalized, adaptive interfaces for multi-domain, highly personalized, open-ended, and exploratory information tasks.These tasks involve continuously evolving goals and information needs.
  • Scope: Generative and malleable UIs require task representation, UI generation, model evolution, data integration, and context awareness.The paper focuses on the first three aspects.
  • Example Scenario: Jelly supports a dinner-party workflow by generating follow-up GUIs and a task panel containing time, location, guests, menu, and activities.Contacts can be added directly through selectable cards linked to the guest list.
  • Example Scenario: Users can extend the model during the task: dietary restrictions become guest attributes, and Jelly records the stated preferences.The menu also receives a corresponding dietary-suitability attribute.
  • Example Scenario: When the task shifts to shopping, Jelly generates a shopping-list panel with aggregated quantities, store selection, purchase tracking, and a map for route planning.The map links each store to the items needed there.

3 Related Work

Related work separates UI presentation from underlying models, supports end-user customization, and adapts interfaces to context. The paper distinguishes its approach by continuously updating task-driven models rather than relying on developer-defined or static structures.

  • Model-Based UI Interfaces: Model-based UI approaches use abstract models to structure workflows, represent data relationships, and support systematic UI development.Examples include task models, domain models, and automated mappings to interface specifications.
  • Model-Based UI Interfaces: This paper extends the model-based perspective by automating mappings while continuously updating the underlying model for end-users’ evolving needs.Prior MBUI models are described as predefined and static and primarily developer-oriented.
  • Specification-Based Generation: Specification-based UI generation lets end-users or systems generate interfaces within constrained, structured frameworks rather than through low-level programming.High-level specifications improve interpretability and modifiability while helping maintain design consistency and quality.
  • End-User Customization: End-user programming and development systems provide natural-language, GUI-based, visual, and demonstration-based ways to extend applications, but often hide internal data models.External proxies such as spreadsheets can still be predefined by developers, limiting customizability.
  • Context-Aware Adaptation: Context-aware systems adapt interfaces to device, user, environmental, or interaction-history factors, but developers typically predefine what and how the interface adapts.This limits end-users’ control over adaptability.
  • AI-Based Code Generation: AI code-generation approaches offer scalable UI generation but face end-user control challenges, inconsistent performance, and unresolved server-side data structuring.The paper motivates higher-level control structures to guide generation and iteration.

4 Design Goals

The design goals define a task-driven model that represents essential information structures without prescribing rigid workflows, maps consistently to expressive UIs, and remains directly modifiable as users’ needs evolve.

  • DG1 Developing Effective Task-Driven Data Model: DG1 requires a task-driven data model that represents the entities, relationships, and constraints essential to users’ information tasks.Unlike rigid task models, it should allow users to form their own workflows and be intuitively interpreted and manipulated.
  • DG2 Mapping Data Models to UIs: DG2 maps abstract models to concrete UIs through specifications grounded in common design patterns for widgets and interactions.The goal is consistency and expressiveness in generated interfaces.
  • DG3 Providing Interactions for End-Users: DG3 enables end-users to modify interfaces through natural-language prompts and direct manipulation, translating those interactions into model updates.An Inspect-like tool is intended to let users examine and edit the model directly.

5 Technical Pipeline for Generative and Malleable User Interface

The pipeline uses LLMs to transform user prompts into task-driven data models, then maps those models to UI specifications and rendered interfaces. Users can iteratively customize the model and specifications through natural language and direct manipulation.

  • Pipeline Overview: The pipeline analyzes user prompts, derives goals and sub-tasks, and uses LLMs to generate a Task-Driven Data Model.The model represents the task through an object-relational schema, dependency graph, and structured data.
  • Object-Relational Schema: The object-relational schema represents the task as a root task object, entities, attributes, and references among entities.Task attributes describe the overall task, while entities model essential components such as meal plans, recipes, ingredients, and grocery stores.
  • Object-Relational Schema: Attributes use SVAL, DICT, PNTR, or ARRY types, with schema rules abstracting shared dictionaries into entities referenced through pointers.This abstraction simplifies the data model and keeps entity and attribute handling consistent.
  • Dependency Graph: Dependencies connect entities or attributes through validation or automatic updates, with relationships implemented as code when possible or natural language otherwise.Validation can reject an invalid update and highlight the issue, while update dependencies propagate changes such as recalculated total calories.
  • UI Specification Generation: The pipeline acquires structured or unstructured data from generated data, user uploads, and external APIs, then maps the model to UI specifications for rule-based interface composition.Annotations identify data types, functional roles, and rendering types, while composition supports progressive disclosure for space-intensive attributes.
  • Customization with Continuous Prompting: Users can customize schemas and data through operations targeting entities or attributes, including add, remove, update, cluster, filter, and sort actions.The LLM translates these operations into updates to the UIs; the prototype also exposes the schema and supports continuous interaction through Jelly.

6 Jelly: a Generative and Malleable Interactive System

Jelly generates task-oriented interfaces and lets users inspect, navigate, and customize them through natural language and direct manipulation. Its panels, rendering strategies, synchronized highlighting, and editable data model support flexible work with complex information.

  • System overview: Jelly generates interfaces from task prompts and exposes Schema and Chat Views for inspecting the data schema and requesting interface changes.Follow-up prompts update the data model and rendered UI, while prompts preserve prior workspace states.
  • Panel organization: Jelly organizes complex tasks through a Home Panel, entity-specific panels, nested panel navigation, and direct access to deeply nested entities.Panels can also be closed, resized, or rearranged to customize the workspace.
  • Collection views: Expanded rendering shows collection items with selected attributes and opens detailed information in popups, persistent cards, or dedicated entity panels.The Dinner Plan example displays dish names and cuisines in a list, with ingredients and dietary suitability available on demand.
  • Collection views: Summary rendering condenses a collection into a task-relevant button, such as a shopping-item count or total travel budget, that reveals details on request.This reduces the amount of information shown while preserving access to the underlying collection or breakdown.
  • Customization: Synchronized highlighting links repeated entity instances across panels, while natural-language prompts and direct manipulation update data, schema, and rendered interfaces.Users can add entity instances, create empty cards, auto-complete missing attributes with preferences, and delete unnecessary attributes.
  • Representation switching: Users can switch among map, list, and table representations because different views support browsing, spatial planning, or attribute comparison.Jelly automatically selects a representation and provides a menu for switching; its infrastructure can support additional or user-defined views.

7 Technical Evaluation

The technical evaluation tested whether GPT-4o could generate relevant schemas and accurate dependencies from informational task prompts. Across 50 requests, the generated entities and attributes were usually necessary and expected, while dependency relationships and mechanisms achieved high but imperfect accuracy.

  • Evaluation design: The technical evaluation assessed schema relevance and dependency accuracy for LLM-generated object-relational models from informational task prompts.Dependency accuracy covered both entity relationships and their update or validation mechanisms.
  • Scope: The technical evaluation did not assess UI rendering or view composition, which were reserved for the subsequent user study.Thus, these results concern generated schemas and dependencies rather than the complete interface experience.
  • Dataset: 50 task requests from 25 scenarios produced 197 entities, 1052 attributes, and 232 dependencies across the evaluated data models.Each scenario included less-detailed and more-detailed prompt versions spanning multiple domains.
  • Schema results: 94.12% and 94.74% of entities, and 93.91% and 95.17% of attributes, were rated necessary and expected for less- and more-detailed prompts, respectively.Redundant attributes caused by overly literal prompt interpretation comprised less than 0.5% of cases.
  • Dependency results: 91.5% relationship accuracy and 96.9% mechanism accuracy were achieved across the dependency labels.Common errors included reversed relationships and redundant dependencies; validation was integrated to address directional relationship errors.

8 User Study

Jelly supported users in organizing information, adapting interfaces to evolving tasks, and maintaining continuity across iterative changes. Participants generally found the system relevant, efficient, customizable, and more fluid than conventional applications.

  • Effective Information Organization: 8 participants agreed or strongly agreed that Jelly’s information was relevant, while all 8 agreed or strongly agreed that it helped them achieve tasks efficiently.The relevance responses were 6 strongly agree and 2 agree; efficiency responses were 2 strongly agree and 6 agree.
  • Customization Patterns: Participants used fully specified prompts more often for planning tasks than learning tasks: 82% versus 35%.Learning tasks had 55% underspecified prompts, compared with 8% for planning tasks.
  • Customization Patterns: 118 modification prompts mainly expanded schemas, with 86 schema changes versus 32 data changes.Schema changes included 57 adds, 10 removes, and 19 updates; data changes included 19 adds, 1 remove, and 12 updates.
  • Continuous Adaptation: Participants described iterative customization as fun and efficient, and they naturally shifted task focus or scope while using Jelly.Examples included changing from preparing Christmas gifts to planning a hiking trip and narrowing graduate-school planning to resources and contacts.
  • Effective Information Organization: Jelly’s structured organization and semantic associations helped users consume and control open-world information, including by removing dishes that violated requested dietary restrictions.All participants expressed excitement about generating information spaces tailored to their needs.
  • Continuous Adaptation: Jelly’s interpretation of ambiguous prompts helped participants begin with vague goals and refine them during exploration.For a Hawaii trip, a request to stay on the beach led to beachfront hotels and suggested activities.
  • Persistence and Transparency: Participants valued traceable history, localized interface edits, and continuity across evolving tasks, while 6 strongly agreed and 1 agreed that Jelly avoided restrictive design choices.The schema view let users verify changes, and participants appreciated viewing only information relevant to them.
  • Interaction Limitations: Continuous prompting was easy to articulate but often tedious when the initial task structure was insufficient.Participants suggested proactive model expansions and lightweight refinement mechanisms to reduce incremental prompting.

9 Discussion and Future Directions on Generative and Malleable UI

The discussion identifies current limitations in dependency modeling, schema transformations, layout management, and data integration, while outlining personalization as a future direction.

  • Broadening Supported Tasks with Advanced Dependency Modeling: The current dependency graph models relationships only between pairs of source and target elements, limiting support for complex interactions.The authors propose graph-based dependency modeling with entity and attribute nodes and a more expressive specification language.
  • Supporting Information Transformation Patterns with Higher-level Schema Operations: Low-level schema operations can make high-level information transformations complex and error-prone, especially when LLMs translate them into data and UI modifications.The paper proposes dedicated operations, such as eversion, to represent common transformations more directly.
  • Enabling Advanced and Malleable View Management: Jelly currently uses a column-based layout that avoids manual panel positioning and provides sufficient usability for investigating the underlying pipeline.Future work could integrate the data model with dashboard design patterns to arrange panels according to inferred importance.
  • Integrating External Data Sources: Future work includes connecting the pipeline to external and user-permitted data sources through approaches such as retrieval-augmented generation, the Model-Context Protocol, and generated API calls.
  • Personalization and Context Preservation: Personalized model evolution could reuse preferred entities, attributes, interface configurations, and subcomponents from previous relevant tasks.The paper notes that different users may prioritize different workflow aspects, such as entity relationships or deadlines.

10 Conclusion

The paper proposes generating task-driven data models from user prompts and using them to produce generative, malleable interfaces. Technical and user evaluations support the feasibility of this approach and its ability to support personalized, dynamic information activities.

  • LLMs generate task-driven data models from users’ prompts, which then guide the generation of generative and malleable user interfaces.
  • The technical evaluation found that LLMs can generate relatively high-quality data models for the proposed pipeline.
  • A user evaluation of Jelly found that generative and malleable interfaces enable users to develop personalized and dynamic information activities.

A UI Specification

The appendix includes a UI specification for one attribute of an object.

  • Table 3 is titled “UI specification for one attribute of an object.”
  • The supplied appendix passage identifies the specification as concerning an attribute of an object.

A.1 Example Specification

The example specification defines dinner-planning entities and attributes with data types, editability, rendering types, functions, relationships, and derived values.

  • The DINNER_PLAN specification includes date, host, location, guest list, and menu fields with assigned rendering and function metadata.Guest lists and menus are represented as arrays with item specifications.
  • The menu specification derives total_calories by summing the calories field across dishes.
  • The USER entity includes a string id marked editable and rendered as hidden with a privateIdentifier function.
  • Guest attributes include name, email, and phone with shortText, url, and number renderings respectively.
  • The DISH entity specifies name, ingredients, calories, and cuisine_type, including expanded, number, and category renderings.Cuisine categories include American, Italian, Chinese, Japanese, and French.
  • The cuisine_type field uses a category rendering with a display function and a predefined list of cuisine categories.

B User Study Questionnaire and Responses

The section presents a summary of all 5-point Likert questionnaire questions and participant responses.

  • The section reports all questionnaire questions using a 5-point Likert scale.
  • Participant responses are summarized across the questionnaire questions.
  • Figure 8 contains the plotted summary of the questionnaire questions and responses.
Loading 2503.04084v1…