PaperlayerView exact revision on arXiv

Source-linked AI summary

Designing Creative AI Partners with COFI: A Framework for Modeling Interaction in Human-AI Co-Creative Systems

Jeba Rezwana, Mary Lou Maher

arXiv:2204.07666v1cs.HCcs.AI

TL;DR

Human-AI co-creativity lacks sufficiently developed interaction design despite relying on interaction dynamics for collaboration. The paper develops COFI as a design space, applies it to 92 co-creative systems, and identifies three interaction models alongside a general lack of communication. COFI therefore supports analyzing, comparing, and designing interaction in co-creative systems, within the limits of the analyzed dataset.

  • Problem

    Interaction design in co-creativity is relatively under-researched, while many systems provide limited interaction despite human-AI collaboration depending on interaction dynamics.

  • Method

    The paper develops COFI, a framework describing interaction-design possibilities, and codes and analyzes 92 co-creative systems using it.

  • Results

    The analysis identifies three interaction models—generative pleasing, improvisational, and advisory agents—and reveals a general lack of communication in the dataset.

  • Takeaways & Limitations

    COFI provides a basis for analyzing, comparing, and designing interaction while indicating opportunities to extend communication between users and AI.

  • Takeaways & Limitations

    The identified interaction-model clusters are limited to the specific dataset used for analysis.

Abstract

from arXiv · show

Human-AI co-creativity involves both humans and AI collaborating on a shared creative product as partners. In a creative collaboration, interaction dynamics, such as turn-taking, contribution type, and communication, are the driving forces of the co-creative process. Therefore the interaction model is a critical and essential component for effective co-creative systems. There is relatively little research about interaction design in the co-creativity field, which is reflected in a lack of focus on interaction design in many existing co-creative systems. The primary focus of co-creativity research has been on the abilities of the AI. This paper focuses on the importance of interaction design in co-creative systems with the development of the Co-Creative Framework for Interaction design (COFI) that describes the broad scope of possibilities for interaction design in co-creative systems. Researchers can use COFI for modeling interaction in co-creative systems by exploring alternatives in this design space of interaction. COFI can also be beneficial while investigating and interpreting the interaction design of existing co-creative systems. We coded a dataset of existing 92 co-creative systems using COFI and analyzed the data to show how COFI provides a basis to categorize the interaction models of existing co-creative systems. We identify opportunities to shift the focus of interaction models in co-creativity to enable more communication between the user and AI leading to human-AI partnerships.

1 INTRODUCTION

The paper argues that interaction design is essential to effective human-AI co-creative systems, yet many systems provide limited communication and one-way interaction. It introduces COFI to describe interaction possibilities and analyzes 92 systems, identifying three interaction models and a general lack of communication.

  • Motivation: Human-AI co-creativity depends on interaction because humans and AI jointly contribute to a shared creative product.This distinguishes co-creative systems from autonomous creative systems and creativity support tools.
  • Motivation: Open-ended, evolving collaboration makes it unclear how an AI should contribute, lead, follow, and adapt during co-creation.Human strategies, ideas, and products develop dynamically, requiring continual adjustment from the agent.
  • Interaction-design gap: Many existing co-creative systems support one-way interaction in which users communicate with AI but receive no communication, suggestions, or feedback in return.The cited examples provide buttons for submitting or converting user input but no reciprocal communication channel.
  • COFI: COFI describes interaction components spanning participation style, contribution type, and communication between human and AI collaborators.The framework is intended to guide interaction-model design and interpretation of existing co-creative systems.
  • Dataset analysis: Coding 92 co-creative systems with COFI produced three interaction models: generative pleasing, improvisational, and advisory AI agents.Generative pleasing agents follow users; improvisational agents work alongside users spontaneously; advisory agents generate and evaluate the creative product.
  • Findings: The dataset analysis reveals a general lack of communication channels between users and AI agents, highlighting communication as an area for development.The paper discusses extending human-AI communication in existing interaction models.

2 RELATED WORK

Related work frames co-creative systems as collaborations between human and computer contributors and motivates broader interaction-design frameworks. Prior approaches address modalities, strategies, turn-taking, contributions, and dialogue, but COFI is presented as an extended design space.

  • Co-creative systems: Co-creative systems differ from standalone generative systems and creativity support tools because humans and computers both contribute as creative colleagues.Standalone systems work autonomously, while creativity support tools assist users without contributing to the creative process.
  • Co-creative interaction: Co-creativity treats interaction between human and AI as complex, emergent, and requiring continual adaptation to human strategies.The creative process involves collaboration rather than attribution to either contributor alone.
  • Evaluation: Turn-taking can produce negative user experience when users dislike the AI taking the lead, showing that interaction preferences affect evaluation.An empirical study of Lumin AI compared a turn-taking model with a non-turn-taking model.
  • Research gap: The literature identifies a lack of holistic interaction-design frameworks capable of exploring, comparing, and evaluating co-creative interaction spaces.A prior framework was limited to turn-based agents and focused on contributions and turn-taking.
  • Existing frameworks: Prior research identifies interaction modalities, styles, strategies, turn-taking, contribution patterns, and linguistic or non-linguistic dialogue as relevant design dimensions.These approaches include operation-based, request-based, ambient, and turn-based interaction frameworks.
  • COFI: COFI includes and extends existing frameworks while incorporating interaction components observed in human-to-human collaboration for human-AI co-creativity.Its scope includes interaction dynamics such as participation style, communication, and contribution type.
  • Human collaboration: Human collaborative creativity research connects communication and interaction with the emergence of shared creative products.Improvisational theater studies and theoretical models motivate examining interaction processes in human-AI collaboration.

Sense-making in Collaboration

The paper grounds COFI in interactional and participatory sense-making, viewing co-creation as coordination between collaborators and interaction with an evolving shared product. This framing organizes interaction dynamics such as turn-taking, initiative, communication, contribution, and editing.

  • Sense-making: Sense-making describes how cognitive agents connect meaningfully with the world according to their needs and goals.Multiple interacting agents make sense-making more complex and emergent.
  • Participatory sense-making: Participatory sense-making occurs when autonomous agents mutually regulate their coupling while preserving their autonomy.The relational dynamics form an emergent organization without destroying the agents’ autonomy.
  • Interactional sense-making: Interactional sense-making distinguishes interaction between collaborators from interaction with the shared creative product.COFI adapts and extends this model as the foundation for its interaction-design possibility space.
  • Interaction with the product: Interaction with the shared product concerns how co-creators sense, contribute to, and edit the emerging creative product.This captures the product-facing side of co-creative interaction.
  • Interaction between collaborators: Interaction between collaborators unfolds through time and includes turn-taking, initiative timing, and communication.Participatory sense-making involves mutual co-regulation of interaction with collaborators and the shared product.

3 CO-CREATIVE FRAMEWORK FOR INTERACTION DESIGN (COFI)

COFI models co-creative interaction as a design space spanning collaboration between human and AI collaborators and interaction with their shared creative product. It organizes these possibilities into four subcategories informed by collaboration, CSCW, creativity, and co-creativity research.

  • COFI describes a space of possibilities for interaction design in co-creative systems.It is intended to guide interaction-model design and analysis of existing systems.
  • COFI separates interaction between collaborators from interaction with the shared creative product.The first concerns how human-AI interaction unfolds, while the second concerns creation of creative content.
  • Interaction between collaborators comprises collaboration style and communication style, while shared-product interaction comprises creative process and creative product.Together, these are COFI’s four main subcategories.
  • COFI draws on human collaboration and CSCW literature for collaborator interaction, and creativity and co-creativity literature for shared-product interaction.The framework therefore combines collaboration and creativity perspectives.
  • COFI was developed iteratively by adding, merging, and removing interaction components identified through a literature review.The review covered research on human collaboration, CSCW, computational creativity, and human-computer interaction in co-creation.
  • Collaboration style includes participation style, task distribution, timing of initiative, and mimicry.Participation style distinguishes parallel participation from turn-taking, while task distribution distinguishes same-task from task-divided collaboration.

4.1 Data

The dataset combines systems from the LMICI archive with newer systems identified through literature searches, producing 92 co-creative systems grouped across 13 creative domains. Music, storytelling, game design, and painting or drawing are the most common domains.

  • The search covered co-creativity and human-AI creative collaboration literature from 2017 to 2021 using the ACM Digital Library and Google Scholar.The corpus was initiated with the LMICI archival collection.
  • The corpus contains 92 co-creative systems assembled from 73 LMICI systems and 19 additional systems published after 2017.One LMICI system was excluded because information about it was unavailable.
  • The systems were grouped into 13 creative-domain categories, including music, storytelling or writing, game design, painting or drawing, and culinary.Other categories included dance, theatre, video, photography, poetry, industrial design, graphic design, and humor or comics.
  • Music, storytelling or narrative writing, game design, and painting or drawing are the most common creative domains in the corpus.Culinary, humor, and graphic design are comparatively less represented in the dataset.

4.2 Coding Scheme

The researchers coded all 92 systems’ interaction designs using COFI, with independent double-coding for 25% and consensus resolution producing a kappa reliability of 0.79.

  • The interaction designs of 92 systems were coded using all COFI interaction components.Components absent from a system were coded as ‘None’.
  • Two coders independently coded 25% of the systems and resolved disagreements through discussion, achieving a kappa inter-rater reliability of 0.79.The remaining systems were coded by one coder according to the consensus codes.
  • Coding used only interaction-design information reported in the corresponding literature for each system.This procedure linked the corpus analysis directly to documented system descriptions.

4.3 Interaction Design Models among Co-creative Systems

K-modes clustering identified three interaction-design models among the 92 systems, with a 67-system cluster dominating the dataset. Across the models, communication between users and AI is generally limited, despite differences in participation, initiative, task distribution, and AI contribution.

  • Three interaction-design clusters were identified using K-modes clustering over COFI interaction components.K-modes was selected because the coded features are categorical.
  • 67 systems formed the dominant first cluster, compared with 9 systems in the second cluster and 16 in the third.Chi-square tests found every interaction component significantly contributed to cluster formation, with all P values < 0.05.
  • Cluster 1: Most systems use turn-taking with planned initiative, meaning each collaborator waits for the other’s turn and improvisational creativity is generally unsupported.This dominant model commonly divides tasks and uses direct manipulation for human-to-AI intentional communication.
  • Cluster 1: The dominant model generally lacks human-to-AI consequential communication and AI-to-human communication, while the AI generates new contributions intended to please the user.EDME exemplifies this pattern by generating music after the user selects an emotion, without allowing feedback on the generated music.
  • Cluster 2: The second cluster uses parallel participation, allowing both agents to contribute simultaneously, while both usually perform the same generative task.Its systems support spontaneous initiative and may use either mimicry or non-mimicry.
  • Cluster 3: The third cluster combines parallel participation and spontaneous initiative with same-task collaboration, indicating improvisational co-creativity.These systems lack communication between user and AI, which the paper states can reduce collaboration quality and engagement.
  • Cluster 3: A third interaction model distinguishes itself by combining AI generation and evaluation, enabling the system to advise users by evaluating their contributions.These systems often refine user contributions and do not mimic them, with contribution similarity varying from high to low.

4.4 Adoption Rate of the Interaction Components used in the Systems

Across 92 co-creative systems, interaction models predominantly use generation, creating new contributions, turn-taking, task-divided collaboration, planned initiative, and non-mimicry. Communication channels are limited, especially for consequential information and AI-to-human communication.

  • Collaboration style: 89.1% of systems use turn-taking participation, whereas 10.9% use parallel participation, mainly in performative co-creation.
  • Collaboration style: 75% of systems divide tasks across separate creative subtasks, while 25% have humans and AI work on the same task.
  • Collaboration style: 86.8% of systems plan initiative timing, 90.2% employ non-mimicry, and only 1.1% employ mimicry.
  • Communication style: 69.6% of systems use direct manipulation for human-to-AI intentional communication, while 95.7% collect no consequential information from users.
  • Communication style: Most systems have no AI-to-human communication channel, and voice, embodied, and text communication are used rarely.
  • Creative process: 79.3% of systems employ generation as the creative process, while 15.2% use both generation and evaluation.
  • Creative product: 59.8% of systems use create new as the contribution type; 10.9% combine create new with refine, and 8.7% combine create new with extend.

4.5 Communication in Interaction Models

The dataset reveals a significant lack of communication channels between humans and AI in co-creative systems. Although some systems use embodied cues and other modalities, most provide limited direct communication and feedback beyond the shared creative product.

  • Implicit communication: In systems without explicit channels, co-creators can communicate subtly through the shared product by interpreting one another’s contributions.
  • Implications: Different communication modalities have the potential to improve coordination and collaboration quality.
  • AI-to-human communication: 82.6% of systems cannot communicate feedback or information directly to human collaborators beyond the shared product.
  • Communication modalities: Some systems use multiple communication channels, including text, embodied communication, voice, and visuals such as images and animation.
  • Communication gaps: 95.7% of systems cannot capture consequential information such as facial expressions, biometric data, gaze, or postures from human users.
  • Human-to-AI communication: 69.6% of systems use direct manipulation for intentional human-to-AI communication, while 21.7% provide no intentional communication method.
  • Illustrative systems: Shimon uses intentional embodied gestures as visual cues for turn-taking and musical beats, and a user study found that these cues aid synchronization.

5 DISCUSSION

COFI frames interaction as a design space for analyzing, comparing, and designing co-creative systems, and its analysis of 92 systems reveals recurring models and communication gaps. The discussion identifies opportunities to extend communication, conceptual-space definition, and interaction components for stronger human-AI collaboration.

  • COFI as an interaction framework: COFI provides a framework for analyzing, comparing, and designing interaction in co-creative systems.Researchers can use it to explore possible interaction spaces and choose appropriate interaction components.
  • Interaction-model clusters: The dataset analysis identified three major clusters of interaction models used by existing co-creative systems.These models include systems that follow user contributions, improvisational systems, and advisor-like systems that evaluate and generate.
  • Interaction-model clusters: The most common model uses generative agents that follow human contributions, while provoking agents that offer different contributions are rare.Pleasing agents suit users seeking a specific style, whereas provoking agents suit users seeking varied ideas.
  • Interaction-model clusters: The improvisational interaction model supports spontaneous initiative-taking and parallel work, but lacks intentional and consequential communication channels.The discussion identifies communication as a key area for extending this model because its absence may affect collaboration quality and user experience.
  • Future interaction opportunities: Only 4 of 92 systems define the creative conceptual space, although defining that space is described as essential to co-creativity.The paper identifies potential for systems that both define the conceptual space and explore it with users.
  • Future interaction opportunities: Communication is the most significant improvement area: most systems provide minimal channels, rely on direct manipulation, and rarely support consequential or AI-to-human communication.The discussion connects broader communication with feedback, coordination, comprehensibility, and improved creative collaboration.
  • Future interaction opportunities: COFI is presented as expandable, allowing additional interaction components to be incorporated as research on human-AI collaboration develops.The framework is intended to support identifying trends and gaps across co-creativity, HCI, and cooperative AI research.

6 LIMITATIONS

The identified interaction-model clusters are limited to the dataset analyzed and reflect the expectations and technologies present when its systems were published. The authors expect these clusters and descriptions to change over time.

  • The interaction-model clusters identified in COFI are limited to the specific dataset used for analysis.
  • The dataset’s systems are constrained by the expectations and technologies at the time of publication.
  • The authors expect the clusters and descriptions of interaction models for co-creative systems to change over time.

7 CONCLUSIONS

The paper develops COFI as a framework for modeling and analyzing interaction in co-creative systems, applying it to 92 systems from the literature. The analysis identifies three interaction models, finds that the interaction-design space is underutilized, and reveals limited communication between users and AI.

  • COFI is developed as a framework for modeling interaction in co-creative systems.The authors describe it as expandable through the addition of other interaction components in the future.
  • COFI was used to analyze the interaction design of 92 co-creative systems from the literature.
  • Three interaction models were identified: generative pleasing agents, improvisational agents, and advisory agents.
  • The identified models can help system developers choose suitable interaction components for corresponding co-creative systems.
  • The findings show that the interaction-design space of possibilities is underutilized in the analyzed dataset.The authors state that COFI can help identify research directions and gaps, while noting that this analysis is limited to the dataset.
  • COFI revealed a general lack of communication in the dataset’s co-creative systems, especially AI-to-human communication and channels beyond direct collaboration.The paper identifies collecting intentional information from humans and gathering consequential communication data as future research areas, including eye gaze, biometric data, gesture, and emotion.
  • Experiments with different interaction models can help identify effective interaction designs and factors affecting user engagement.
Loading 2204.07666v1…