Source-linked AI summary
Friend, Collaborator, Student, Manager: How Design of an AI-Driven Game Level Editor Affects Creators
Matthew Guzdial, Nicholas Liao, Jonathan Chen, Shao-Yu Chen, Shukan Shah, Vishwa Shah, Joshua Reno, Gillian Smith, Mark Riedl
TL;DR
It is not yet clear how AI systems should be designed to support creative collaboration with people. The paper develops Morai Maker, an AI-assisted Super Mario Bros.-style level editor, and studies practicing designers’ interactions with it; designers varied in preferred AI roles, adapted their practices, and saw potential value in the tool.
Problem
It is not yet clear how AI systems and interfaces should best support creative expression and human-AI interaction.
Method
The paper develops the Morai Maker intelligent level editor and evaluates it through two mixed-methods user studies with over one hundred participants.
Results
Designers varied widely in their preferred AI roles, adapted their design behavior, and viewed the tool as potentially valuable, while no single agent satisfied all expectations and participants deleted 60% of agent additions on average.
Takeaways & Limitations
AI level-design partners may serve as sources of inspiration that encourage designers to rethink designs or generate new ideas when they are unsure how to proceed.
Takeaways & Limitations
The AI systems are intended to augment rather than replace designers and could reinforce users’ existing design biases.
Abstract
from arXiv · showhide
Machine learning advances have afforded an increase in algorithms capable of creating art, music, stories, games, and more. However, it is not yet well-understood how machine learning algorithms might best collaborate with people to support creative expression. To investigate how practicing designers perceive the role of AI in the creative process, we developed a game level design tool for Super Mario Bros.-style games with a built-in AI level designer. In this paper we discuss our design of the Morai Maker intelligent tool through two mixed-methods studies with a total of over one-hundred participants. Our findings are as follows: (1) level designers vary in their desired interactions with, and role of, the AI, (2) the AI prompted the level designers to alter their design practices, and (3) the level designers perceived the AI as having potential value in their design practice, varying based on their desired role for the AI.
1 INTRODUCTION
The paper examines how AI can support creative expression through human-AI collaboration, a design problem whose best interaction paradigms remain unclear. Morai Maker addresses this gap with a collaborative level editor and studies how designers interact with and value different AI roles.
- AI-assisted creative systems are increasingly relevant, but how interfaces and AI should support creativity remains unclear.
- Morai Maker lets designers and an AI agent take turns building a Super Mario Bros.-like level in one editor.
- Two mixed-methods studies with over one hundred participants examined tool development and its effects on practicing game designers.
- Designers varied in their preferred AI interactions and roles, summarized as friend, collaborator, student, or manager.
- The work identifies user behaviors and interaction patterns relevant to interfaces where people and AI collaboratively create one object.
Games
The paper situates Morai Maker within machine-learning-based procedural content generation and prior co-creative game design. It focuses on collaborative generation of level sections using a rich representation and a familiar game domain.
- PCGML generates game content through machine-learning methods, while prior co-creative level-design systems commonly relied on search or grammars.
- Unlike approaches generating entire levels, Morai Maker generates individual level sections to better support collaborative building.
- The system uses a rich representation of level components and ordering that accommodates decorative elements.
- The tool was modeled on Mario Maker, while Super Mario Bros. was chosen because it is well recognized and widely studied in PCGML.
- Morai Maker is a Unity3D editor with a central level view, minimap, sprite palette, and an End Turn control that queries the AI.
4 STUDY 1: PROTOTYPE INVESTIGATION
Study 1 investigated how three different AI level-design partners affected user experience in a turn-based editor. Participants compared randomly assigned agents while creating constrained Super Mario Bros.-style levels.
- AI Level Design Partners: The study compared Markov Chain, Bayes Net, and LSTM agents that differed in how much existing level structure they considered.
- AI Level Design Partners: Markov Chain used a 2x2 local grid, Bayes Net used a 16-point-wide chunk, and LSTM considered almost the entire level.
- AI Level Design Partners: Agents were restricted to additions because their source systems generated levels autonomously and were not designed to handle deletions.
- Study Method: Participants interacted with two randomly assigned agents, creating above-ground or below-ground levels in sessions lasting up to fifteen minutes.
- Study Method: Afterward, participants ranked agents on fun, frustration, challenge, design assistance, surprising ideas, and willingness to use them.
- Study Method: Figure 2 shows pairs of final levels associated with the Markov Chain, Bayes Net, and LSTM agents.
Results
The study analyzed 84 participants after excluding seven who did not interact with one or both AI partners. Rankings generally showed no significant agent differences, while experiential rankings were not random but had a noted ambiguity around “challenging.”
- 84 participants remained in the analysis, split evenly across twelve conditions, while 62% had previously designed Mario levels.
- No experimental variable significantly affected experiential rankings, so analyses treated the data as three agent-pair conditions.
- Table 1 found significant differences in only three comparative rankings, with most agent comparisons nonsignificant.
- Correlations across experiential rankings indicated that participants were not ranking agents through random guesses.
- The “challenging” ranking was least consistent, possibly because the question lacked clarity or participants used it to signal limited understanding of an agent.
Study 1 Output Levels
Study 1 found no single AI agent consistently satisfied participants’ creative preferences. Participants nevertheless identified value in different agents, while deleting many generated additions and departing from standard Mario level structures.
- Study 1 Output Levels: Markov Chain levels showed unique block patterns, while Bayes Net levels more often included decoration.
- Study 1 Results Discussion: No single agent emerged as a consistently superior creative partner because participants’ preferences varied.
- Study 1 Results Discussion: Although LSTM received few first-place rankings, some participants valued its collaboration and ideas, whereas another participant preferred Markov’s surprising and valuable outputs.
- Study 1 Results Discussion: Participants deleted 60% of agent additions on average, and some reported that both agents produced unhelpful or stylistically mismatched content.
- Study 1 Results Discussion: Participants consistently departed from typical Super Mario Bros. structure and often invented explanations for how their AI partners behaved.
5 STUDY 2: THINKALOUD
Study 2 examined how revised Morai Maker interactions affected practicing game designers. It focused on adaptation, explainability, and whether the tool changes produced beneficial design experiences.
- 5 STUDY 2: THINKALOUD: The second study investigated how the revised tool affected practicing, published game designers rather than serving as a direct tool evaluation.
- 5 STUDY 2: THINKALOUD: The study asked whether active learning could better serve designers by adapting the AI partner during use.
- 5 STUDY 2: THINKALOUD: It also examined whether Explainable AI could improve users’ understanding and use of the tool.
- 5 STUDY 2: THINKALOUD: A third question examined whether the overall tool changes would lead to beneficial experiences for designers.
Changes to Morai Maker
Morai Maker was revised toward adaptive, locally coherent collaboration and added controls and explanations to address brittle outputs and limited understanding of the AI partners.
- Changes to Morai Maker: The revised co-creative architecture used a CNN because local coherency from the Markov Chain tended to outperform the LSTM’s global coherency.
- Changes to Morai Maker: Active learning rewarded retained additions with +0.1, penalized removed additions with -0.1, and blocked previously deleted additions during the session.
- Changes to Morai Maker: Informal tests suggested the adaptive agent could learn users’ preferences for local level structures.
- Changes to Morai Maker: Explanations identified the decisive 4x4 level slice, the AI’s confidence, and the maximally activated first-layer CNN filter for each addition.
- Changes to Morai Maker: The editor replaced Options with Remove to undo the AI’s latest additions and removed Run after some participants ignored the AI.
Method
The second study used repeated, turn-based interactions with one AI agent while designers created two levels and answered experience-focused questions.
- Study procedure: Participants designed two levels with the same agent across sessions, allowing the study to examine changing interaction experiences.The agent was not reset between levels.
- Study procedure: Participants could ask study personnel questions, including requests for explanations about the AI, while personnel also prompted quiet participants with “why” questions.A single study personnel member ran each session to keep prompting and explanations consistent.
- Measures: Researchers reworked the original survey to compare the two experiences and added questions about collaboration, adaptation, AI preference, and explanations.The reuse and challenge questions were removed as irrelevant or problematic for this design.
- Measures: Participants explained most survey answers in text, and demographic questions captured their game-design experience and background.The questions were designed to address the study’s three research questions.
Analysis Methods
The analysis combined participant recruitment and level examples with survey ratios, logging information, and comparisons between the two study experiences.
- Qualitative analysis: The qualitative analysis used think-aloud utterances and design-tool interactions, with researchers grouping observations by research question and positive or negative relation.The team reviewed recordings and discussed the final set of notes in two sessions.
- Participants and data: The second study included 14 analyzed game designers, whose demographics and final levels were summarized alongside the survey and logging results.Two of 16 initially agreeing participants were excluded because of networking problems.
- Quantitative analysis: Nine participants preferred using the tool with AI, two preferred without it, and the remaining participants expressed no preference.The authors interpret this distribution as positive support for the AI-related research question.
- Quantitative analysis: Compared with the first study, the second experience was 14.3% more often ranked more fun, 15.5% less often more frustrating, 20.3% more often design-aiding, and 9.5% more often idea-surprising or valuable.The comparison used four experiential features shared between the studies.
- Quantitative analysis: Only 45% of AI additions were deleted in the second study, compared with 60% in the first study.The passage presents deletion rates as logging information summarized for the study comparison.
Qualitative Results
The qualitative results show that designers noticed AI adaptation and found potential value in the tool, but explanations were rarely requested and unpredictable behavior often reduced usefulness.
- Research Question 1: Every participant noticed at least one instance of the AI adapting, although some described most of its behavior as random.Participants connected adaptation to seeing the AI better match their design direction over time.
- Research Question 2: Only three participants asked for AI explanations, and the study therefore found no meaningful answer about whether explanations improved understanding or tool use.Two of the three participants who answered the related survey question said explanations were useful.
- Research Question 3: Designers found value in the tool both as an unintentional source of inspiration and as a way to overcome a lack of ideas.Participants also generally praised the front-end interface.
- Research Question 3: AI behavior limited value when additions seemed random, filled intentional empty space, or placed unreachable blocks.Seven participants used “random” to describe disliked behavior.
Roles and User Adapting Analysis
Designers differed in the roles they expected the AI to play and often changed their own practices to accommodate or interpret its behavior.
- Expected AI roles: The analysis identified four expected AI roles—friend, collaborator, student, and manager—that could change during a design session.These roles reflected different expectations for how the AI should participate in design.
- Collaborator: Collaborator expectations involved an equal design partner, while mismatches arose when the AI contributed small edits instead of complete ideas.Responsiveness could make the collaborator role more positive for some participants.
- Student: Student expectations involved the AI following a participant’s design beliefs or instructions, including judgments about legal or illegal suggestions.Some participants viewed copying their actions as evidence of collaboration.
- Manager: Manager expectations involved the AI instructing or judging the designer, prompting some participants to follow its suggestions.Participants sometimes explicitly asked whether the AI could evaluate them.
- User adaptation: Some designers adapted their own behavior to determine how to interact with the AI or to make seemingly random additions fit their designs.This adaptation included pausing to interpret additions and changing designs to legitimize them.
6 DISCUSSION AND DESIGN IMPLICATIONS
The studies indicate that creators differ in how they want AI to participate, often adapt their practices to it, and see value in it as an inspirational design aid. The authors recommend clarifying AI capabilities while recognizing that the system may reinforce existing design biases and is intended to augment rather than replace designers.
- Study findings: The study found no clear evidence that explainable AI helped users leverage the tool, as users were more likely to generate their own explanations for its performance.The authors suggest this may relate to users’ limited experience with explanatory tools or social pressure created by human translation of the explanations.
- Design implications: Designers varied widely in the AI role they expected, and the authors recommend explicitly outlining what the AI can and cannot do.Users differed in both the role they wanted the AI to take and their expectations of its performance.
- Design implications: Users were willing to adapt their design practices to incorporate AI, either viewing it as dominant or seeking effective ways to interact with it.The authors characterize this willingness as a positive sign for AI-human co-creation.
- Design implications: Designers saw the tool as valuable for inspiration because it could prompt them to rethink designs or suggest ideas when they were unsure how to proceed.The authors anticipate that these interactions may transfer to other design domains.
- Potential limitations: The tool was designed to augment creative work rather than replace designers because its AI systems were insufficient to act as solitary designers.This scope frames Morai Maker as a design aide rather than an autonomous creator.
- Potential limitations: Because the AI adapts to an individual designer’s designs, it could reinforce offensive or over-used design choices rather than challenge the designer to improve.The authors anticipate limited negative-impact risk in the Super Mario Bros. domain but flag the issue for future versions.