PaperlayerView exact revision on arXiv

Source-linked AI summary

Understanding Nonlinear Collaboration between Human and AI Agents: A Co-design Framework for Creative Design

Jiayi Zhou, Renzhong Li, Junxiu Tang, Tan Tang, Haotian Li, Weiwei Cui, Yingcai Wu

arXiv:2401.07312v4cs.HC

TL;DR

Creative design involves evolving requirements and iterative remixing, whereas many AI design tools rely on precise commands and linear workflows. The paper studies expert collaboration, proposes a nonlinear co-design framework and OptiMuse, and reports improved task completion in a comparison with Copilot.

  • Problem

    Linear AI design workflows depend on precise commands, while creative requirements evolve and human-AI requirement alignment and remixing remain underexplored.

    Source: §1 · p. 2, §6.1 · p. 12

  • Method

    The authors derive design requirements from a formative study and develop a nonlinear co-design framework and OptiMuse, comparing it with a linear Copilot baseline.

    Source: §1 · p. 2, §3.1 · p. 4, §3.3 · p. 5

  • Results

    OptiMuse significantly improved users’ task completion rates compared with a baseline tool following a linear human-AI workflow.

    Source: §7 · p. 13

  • Takeaways & Limitations

    The framework’s guidelines aim to incorporate nonlinear characteristics into human-AI collaboration and address limitations of human-human collaboration.

    Source: §7 · p. 13

    The user studies focused on graphic design and participants with expertise in that field, which may have left findings incomplete.

    Source: §6.3 · p. 13

Abstract

from arXiv · show

Creative design is a nonlinear process where designers generate diverse ideas in the pursuit of an open-ended goal and converge towards consensus through iterative remixing. In contrast, AI-powered design tools often employ a linear sequence of incremental and precise instructions to approximate design objectives. Such operations violate customary creative design practices and thus hinder AI agents' ability to complete creative design tasks. To explore better human-AI co-design tools, we first summarize human designers' practices through a formative study with 12 design experts. Taking graphic design as a representative scenario, we formulate a nonlinear human-AI co-design framework and develop a proof-of-concept prototype, OptiMuse. We evaluate OptiMuse and validate the nonlinear framework through a comparative study. We notice a subconscious change in people's attitudes towards AI agents, shifting from perceiving them as mere executors to regarding them as opinionated colleagues. This shift effectively fostered the exploration and reflection processes of individual designers.

1 INTRODUCTION

The paper argues that linear, command-driven AI tools mismatch creative design’s nonlinear process and leave human-AI requirement alignment and idea remixing underexplored. It derives design requirements from human collaboration, proposes a nonlinear co-design framework, and evaluates its OptiMuse prototype.

1 INTRODUCTION

  • Existing AI design tools rely on precise commands that poorly fit abstract, evolving requirements, limiting requirement alignment and remixing.The paper identifies the gap in understanding how people and AI can align requirements and remix ideas during collaboration.Source: §1 · p. 2
  • A formative study of human collaboration produced five design requirements, grouped around nonlinear collaboration and addressing shortcomings of human-human collaboration.Examples include offering alternative design solutions and providing real-time visual outcomes.Source: §1 · p. 2
  • The proposed framework specifies ten AI actions across flexible communication, clarification, alternative solutions, operational strategies, and visual outcomes.The actions are organized around five areas corresponding to the design requirements.Source: §1 · p. 2
  • In a comparative Wizard-of-Oz study with 12 participants, OptiMuse significantly improved task completion rates over linear-workflow Copilot.Copilot serves as the baseline, providing an output for commands without explaining how it inferred them.Source: §1 · p. 2
  • The paper reports three contributions: a formative study and design requirements, a co-design framework and OptiMuse prototype, and a comparative usability study.Source: §1 · p. 2

2 RELATED WORK

The related work reviews nonlinear design models, creativity-support and co-creativity tools, and collective creativity. It positions the paper around translating human collaboration practices into a nonlinear human-AI framework that supports navigation and remixing.

2.1 Creative Design Process

  • Nonlinear design models describe processes such as moving between concept and knowledge spaces, divergent and convergent thinking, and iterative reformulation.The authors argue that high-level models do not readily translate to real-world design or human-AI collaboration, motivating insights from expert human-human practice.Source: §2.1 · pp. 2–3

2.2 From Creativity Support Tool to Human-Computer Co-creativity

  • Creativity-support tools often address individual stages, while prior human-AI co-creativity frameworks commonly represent interaction as straightforward turn-taking.The paper proposes a framework focused on nonlinear navigation across creative-design stages.Source: §2.2 · p. 3

2.3 Collective Creativity

  • Collective creativity research highlights interaction and remixing, but how AI can support remixing in human-AI collaboration remains underexplored.The paper studies how individual designers remix alternative designs while working with AI-assisted graphic design.Source: §2.3 · p. 3

3 STUDY I: UNDERSTAND THE NONLINEAR PROCESS IN CREATIVE DESIGN

The formative study examines how design requirements evolve and how human collaborators negotiate them, then uses those observations to derive requirements for nonlinear human-AI co-design.

3.1 Preconsideration of Experiment Settings

  • Requirement analysis is used to study how designers’ goals are understood and translated into design modifications.The study draws on requirement-analysis knowledge and a role-playing strategy to observe designers and assistants together.Source: §3.1 · p. 4

3.2 Experiment Design: A Formative Study

  • The formative study observed recorded human-human collaborative design to inform requirements for human-AI tools.Source: §3.2 · p. 4

3.2.1 Participants.

  • The study involved 12 experienced designers and used three selected graphic designs with varied layouts, audiences, and modification requests.Participants had at least one year of design experience, and their work had appeared at prominent events.Source: §3.2.1 · p. 4, §3.2.2 · p. 4

3.2.2 Procedure.

  • Pairs alternated designer and assistant roles in a 25-minute modification session, followed by interviews about collaboration challenges and desired AI assistance.The session included content understanding, independent strategy brainstorming, synchronization, and a summary and rating.Source: §3.2.2 · pp. 4–5

3.3 Expert Feedback and Discussion

  • The authors summarize five requirements for human-AI co-design based on expert feedback about requirement characteristics and human-human collaboration.Source: §3.3 · p. 5

3.3.1 The Inherent Nonlinearity of Requirements.

  • The study identifies ambiguity, change, and conflict as recurring features of evolving design requirements.Most designs underwent trial-and-error iterations (34 of 36); conflicts can occur between asynchronous requirements, people, or design constraints.Source: §3.3.1 · p. 5

3.3.2 Design Requirements Inspired by Human-Human Co-design.

  • Human collaboration can inspire alternative approaches, accelerate iteration, and expose misunderstandings, but manual modification limits real-time visual feedback.The authors use these benefits and drawback to motivate five requirements for a nonlinear human-AI framework.Source: §3.3.2 · p. 5

4 A HUMAN-AI CO-DESIGN FRAMEWORK FOR CREATIVE DESIGN

The framework translates five design requirements into AI actions for nonlinear co-design and is embodied in the graphic-design proof-of-concept OptiMuse. Its interaction supports communication before visual generation and work across multiple design choices.

4 A HUMAN-AI CO-DESIGN FRAMEWORK FOR CREATIVE DESIGN

  • OptiMuse is a proof-of-concept implementation of the framework focused on user-interface design.Source: §4 · pp. 5–6

4.1 Expected Actions for AI Agents

  • The framework organizes ten expected AI actions around five areas: flexible communication, clarification, alternatives, operational strategies, and visual outcomes.The actions are intended to address the five design requirements and support a more flexible, human-like collaboration process.Source: §4.1 · p. 6
  • The actions include clarifying ambiguous or changing requirements and offering, reviewing, modifying, comparing, and merging alternative design choices.The framework also includes clarifying operational strategies and previewing their visual effects.Source: §4.1 · p. 6

4.2 OptiMuse: a Proof-of-concept Prototype

  • OptiMuse uses text dialogue to align on strategies before visual generation and offers up to three selectable visual outcomes, with further review and remixing available.Users can refer to options, reject them with a new command, or request additional operations on the choices.Source: §4.2 · pp. 6–7, Figure 3 · p. 7

5 STUDY II: EVALUATION OF OPTIMUSE

Study II evaluates OptiMuse’s usability and the nonlinear collaboration framework by comparing its workflow with a command-based baseline on graphic-design tasks. The Wizard-of-Oz study uses prepared materials, a trained wizard, and repeated tasks across both tools.

5 STUDY II: EVALUATION OF OPTIMUSE

  • Study II evaluates OptiMuse’s usability and the framework by examining users’ design-modification processes on given graphical designs.The study uses a Wizard-of-Oz experiment to compare the two collaboration workflows.Source: §5 · p. 7

5.1 Preconsideration of Experiment Settings

  • The study selected color-scheme changes and text-layout adjustments because they are ambiguous enough to invite exploration but bounded enough for analysis.Both tasks also occurred frequently in Study I.Source: §5.1.1 · pp. 7–8

5.1.1 Choice of Modification Tasks and Design Materials.

  • Two visually appealing graphic designs were prepared for the selected modification tasks, and participants used different tools on different designs.Source: §5.1.1 · p. 8

5.1.2 Wizard.

  • One experienced design professional served as wizard, following guidelines intended to reduce randomness and make agent timing and responses consistent.The wizard was instructed to adopt behaviors commonly associated with AI agents.Source: §5.1.2 · p. 8

5.2.1 Possible Modification Outcomes.

  • Three design students and three frequent AI-design-tool users brainstormed possible requests and prepared corresponding design outcomes in advance.Source: §5.2.1 · p. 8

5.2.2 Participants.

  • Twelve participants with prior human-human graphic-design co-design experience were randomly allocated to four equal-sized groups.The participants had 1–6 years of design experience, averaging 3.47 years.Source: §5.2.2 · pp. 8–9

5.2.3 Baseline.

  • The Copilot baseline executes commands and returns one outcome at a time, whereas OptiMuse provides dialogue and multiple visual outcomes.The comparison used a prototype of Copilot lacking the dialogue function.Source: §5.2.3 · p. 9, Figure 6 · p. 9, Figure 7 · p. 9

5.2.4 Procedure.

  • Each participant completed two modification tasks with each tool, then completed usability questionnaires and a follow-up interview.The tasks concerned color optimization and text formatting; the experiment lasted 90 minutes.Source: §5.2.4 · p. 9

5.3 Results

OptiMuse received higher usability scores than Copilot, while participants’ attitudes and interaction patterns varied. The reported findings contrast Copilot’s emphasis on precise commands with OptiMuse’s encouragement of exploration, requirement changes, and reflection.

5.3.1 Quantitative findings.

  • Four participants gave up on Copilot tasks after losing confidence in its understanding and finding its wording annoying, while the authors report greater tolerance toward OptiMuse.The four participants stopped after an average of 1031 seconds and 8 turns.Source: §5.3.1 · p. 10
  • OptiMuse’s average SUS score was 70.21 (Good), compared with Copilot’s 64.17 (Poor), while average dialogue turns were 10 for OptiMuse and 8 for Copilot.Source: §5.3.1 · pp. 9–10

5.3.2 Divergent attitudes towards OptiMuse and the interaction pat-

  • Some participants welcomed OptiMuse’s feedback and alternatives, while others were anxious or reluctant to converse with it.Positive users reported inspiration from selective choices and often asked for opinions or validation.Source: §5.3.2 · p. 10
  • Participants remixed ideas textually or by combining aspects of multiple choices, while some users avoided dialogue or selected and iterated on the closest option.Some participants reported that communication was complex or time-consuming, and one participant did not remix multiple selections because he believed it was not possible.Source: §5.3.2 · pp. 10–11

5.3.3 Characteristics of Requirements in Design Practices.

  • With Copilot, participants struggled to specify requirements, iterated descriptions, and often treated conflicts as mistakes or avoided discussing them.Examples included changing a background from darker to lighter and leaving an unresolved text-alignment issue unexplained.Source: §5.3.3 · p. 11
  • With OptiMuse, participants used ambiguity and considered suggestions as prompts for exploration; they tolerated some conflicts and reflected on alternatives.Participants reported that suggestions could prompt bold requirement changes; asynchronous conflicts occurred less frequently as users focused on exploring options rather than fine-tuning.Source: §5.3.3 · p. 11

5.4 Lessons We Learnt from Study II

Study II identifies five tasks users may expect from co-design agents and four perceived agent roles. Participants differed in their role preferences, from expert guidance to faithful execution.

5.4.1 What tasks are anticipated from AI agents?

  • Users’ anticipated AI tasks include requirement understanding and analysis, solution exploration, review, and execution.Source: §5.4.1 · p. 11

5.4.2 What roles are expected of AI agents?

  • Participants expressed varying preferences and perceptions of the roles AI should play in graphic design.The roles were categorized as expert, colleague, optimizer, and executor, with diminished emphasis on creativity.Source: §5.4.2 · pp. 11–12
  • Six of twelve participants preferred a colleague role, while one of twelve preferred an expert role.Colleague AI was expected to understand requirements through dialogue and choices; expert AI was expected to provide reliable guidance and explain recommendations.Source: §5.4.2 · p. 12
  • Two of twelve participants preferred an optimizer that generates alternatives, while three of twelve preferred an executor that follows commands and avoids requirement analysis or choice selection.Source: §5.4.2 · p. 12

6 DISCUSSION

The discussion contrasts nonlinear creative collaboration with AI tools that demand precise instructions, then presents a framework for flexible communication and selective choices. It also outlines distinct expectations for AI roles and limits the findings to particular users, methods, and communication modes.

6.1 Nonlinear Design for Co-creative Systems

  • Existing AI tools’ precise-command and turn-based workflows hinder nonlinear navigation, remixing, and fulfillment of the design requirements.The proposed framework instead supports flexible communication, selective choices, parallel iteration, and remixing.Source: §6.1 · p. 12

6.2 Roles of AI Agents in Co-creativity

  • The four AI roles—expert, colleague, optimizer, and executor—differ in expected creativity and accuracy, with experts favoring creativity and executors prioritizing accuracy.Colleagues call for diverse perspectives and communication; optimizers and executors prioritize satisfaction over surprise.Source: §6.2 · pp. 12–13
  • Role expectations vary across users and evolving design intentions, so systems should tailor interfaces and communication and choice actions to their users and scenarios.The roles may need to operate simultaneously or shift as users’ needs change.Source: §6.2 · p. 13

6.3 Limitations and Future Work

  • The user studies focused on expert graphic designers, so the findings may be incomplete and the guidelines remain a starting point for other creative domains.The authors identify writing, music, drawing, coding, and design ideation as areas for future work.Source: §6.3 · p. 13
  • The Wizard-of-Oz method produced response times of about two minutes per textual response, harming usability and discouraging co-design engagement.The authors describe this as a practical compromise for validating the framework; identical delays in both systems allowed comparable System Usability Scale results.Source: §6.3 · p. 13
  • Text-based interaction hindered collaboration efficiency and user experience and was less comparable to human-human collaboration, despite the importance of multimodal communication.The prototype used text as its primary mode to remain comparable to existing AI tools.Source: §6.3 · p. 13

7 CONCLUSION

The paper derives guidelines and a nonlinear human-AI co-design framework from a formative study, then evaluates the OptiMuse proof-of-concept against a linear-workflow baseline. The comparative study finds significantly improved task completion rates with OptiMuse.

7 CONCLUSION

  • OptiMuse significantly improves users’ task completion rates in a comparative study against a baseline tool following a linear human-AI workflow.The prototype follows guidelines derived from a formative study and implements the proposed nonlinear co-design framework.Source: §7 · p. 13

A.1 The Inherent Nonlinearity of Requirements.

The examples describe requirements as ambiguous and subject to change during design collaboration. They also show how design requests can contain competing constraints.

A.1 The Inherent Nonlinearity of Requirements.

  • Clients may express requirements as vague judgments or requests that are difficult to translate into operational instructions.Participants also described limited design skills as making it hard to express a desired visual outcome.Source: §A.1 · p. 15
  • Discussion can clarify a designer’s intended change while revealing further modifications and shortcomings in how the idea was explained.One designer initially requested a color-scheme change while intending to emphasize important parts.Source: §A.1 · p. 15
  • Designers may face competing constraints, such as balancing text quantity against a neat layout.A participant also described the tension between placing text on a flag and having a background.Source: §A.1 · p. 15

A.2 Pros and Cons for the Nonlinear Human-human Co-design Process.

Participants described nonlinear human-human co-design as a source of alternative ideas and faster progress through timely communication, while noting that ideas may not work in practice and real-time outcomes are lacking.

A.2 Pros and Cons for the Nonlinear Human-human Co-design Process.

  • Timely communication can speed modification, expose misunderstandings, and help designers clarify or revise their needs.Participants said discussion could resolve faulty requirement understandings early, though it might take several questions to express a request.Source: §A.2 · p. 15
  • Different viewpoints and interpretations inspired alternative design ideas, including unexpected color arrangements, clearer diagram-text connections, and new presentation formats.Participants also described collaboration as opening them to methods they had not considered alone.Source: §A.2 · p. 15
  • Human-human collaboration lacks real-time design outcomes: written or sketched drafts may remain separate from the design itself.Participants also noted that proposed modifications may not work in practice, such as when wording does not fit or colors fail to match.Source: §A.2 · p. 15

B INSTRUCTIONS FOR THE WIZARD

The Wizard instructions specify experimental materials and standard operations, then guide the assistant’s conversational style, proactive suggestions, clarification of vague requests, and handling of conflicts or changed requirements.

B INSTRUCTIONS FOR THE WIZARD

  • The experiment used Figma projects, two designs and two tasks, possible modification versions, and specified operations for OptiMuse and Copilot.The listed operations include chatting in Figma, exporting and replacing images, changing previews, and imitating AI features in PowerPoint.Source: B INSTRUCTIONS FOR THE WIZARD · p. 15, Table 5 · p. 16
  • The assistant is instructed to use polite, consistent language, address the user as “Nin,” and occasionally rephrase the user’s statements.It should use punctuation and acknowledge the user’s correctness.Source: B INSTRUCTIONS FOR THE WIZARD · p. 16
  • The assistant should offer proactive recommendations and guide users to specify the area and scope when requests are unclear.It may suggest complementary changes, such as adjusting a background to suit a text color.Source: B INSTRUCTIONS FOR THE WIZARD · p. 16
  • The instructions call for addressing conflicting requirements and reminding users when their requests change.Examples include flagging conflicts between earlier and current requests, AI suggestions, or constraints such as text overflow.Source: B INSTRUCTIONS FOR THE WIZARD · p. 16
Loading 2401.07312v4…