Source-linked AI summary
The Double-Edged Sword of Open-Ended Interaction: How LLM-Driven NPCs Affect Players' Cognitive Load and Gaming Experience
Ting-Chen Hsu, Wenran Chen, Jiangxu Lin, Fei Qin, Zheyuan Zhang
TL;DR
LLM-NPCs may provide more autonomy and open-ended interaction, but their effects on players’ cognitive load and gaming experience remain uncertain. Using a randomized experiment comparing LLM-NPC and traditional NPC versions across seven prototype modules, the study finds a double-edged pattern: higher cognitive load without a significant overall experience advantage, with effects depending on task openness and some personality traits.
Problem
Existing LLM-NPC research provides limited empirical evidence about the psychological mechanisms affecting cognitive load and gaming experience, and about variation across task contexts and player characteristics.
Source: §1 · p. 2Method
A randomized between-subject experiment compared LLM-NPC and traditional NPC versions in the Campus Culture Week prototype using module-level and post-test measures plus mediation and individual-difference analyses.
Source: §2.4 · p. 6Results
LLM-NPCs increased cognitive load but did not significantly improve overall gaming experience; expressive effort and response uncertainty mediated cognitive-load increases, while autonomy gains were offset by lower usability and trust.
Source: §4 · p. 14, §5 · p. 15Takeaways & Limitations
LLM-NPCs should be matched to task scenarios and designed for stable, clear, low-cost interaction rather than treated as universal replacements for traditional NPCs.
Source: §4 · p. 14The findings require further testing beyond the self-developed prototype, including more realistic game environments, longitudinal designs, larger individual-difference samples, and multimodal interaction.
Source: §4 · pp. 14–15
Abstract
from arXiv · showhide
This study examines how large language model-driven non-player characters (LLM-NPCs) affect players' cognitive load and gaming experience, with a particular focus on the underlying psychological mechanisms, differences across task scenarios, and the role of individual traits. Conducting a randomized between-subject experiment (N=130) in a self-developed game prototype "Campus Culture Week", we compared player interactions with LLM-NPCs and traditional pre-scripted NPCs across multiple interactive modules. The results showed that LLM-NPCs significantly increased players' cognitive load (p < .001), an effect mediated by factors such as expressive effort and response uncertainty. However, LLM-NPCs did not yield a statistically significant improvement in overall gaming experience (p = .195); while they positively influenced players' perceived autonomy, they exerted a negative influence on system usability and trust. The effects of LLM-NPCs also significantly varied across task scenarios (p < .001), with stronger increases in cognitive load in more open-ended modules such as content creation and relationship building. The influence of individual differences was generally limited, although the personality traits of extraversion (p = .031) and neuroticism (p = .047) demonstrated some predictive power regarding cognitive load. This study provides empirical evidence for understanding the "double-edged sword" effect of LLM-NPCs on player experience, and highlight the importance of scenario-sensitive and user-sensitive design in intelligent NPC systems.
1. Introduction
LLM-NPCs promise more open-ended and context-aware interaction than scripted NPCs, but may also increase players’ cognitive costs. This study addresses how those effects arise, vary across tasks, and relate to player characteristics.
Motivation
- LLM-NPCs process open-domain language and generate responses in real time, potentially enabling greater contextual awareness and emergent narrative than scripted interactions.Traditional NPCs commonly rely on preset logic such as finite-state machines and static dialogue trees.Source: §1 · p. 2
Research problem
- Natural-language interaction can increase cognitive load through ambiguity, context mismatch, clarification demands, and greater information-processing requirements.The resulting experience may combine enhanced immersion or social presence with higher cognitive cost.Source: §1 · p. 2
Research gap
- Existing LLM-NPC research emphasizes implementation and subjective evaluation, leaving psychological mechanisms, task-context variation, and individual differences insufficiently examined.The study frames these gaps as three research questions concerning cognitive load, gaming experience, mediation, task scenarios, and player characteristics.Source: §1 · p. 2
Contribution
- The study aims to explain the double-edged effects of open-ended NPC interaction and inform NPC designs that balance freedom with cognitive cost across scenarios and users.Its stated contribution is empirical evidence for scenario-sensitive and personalized intelligent-NPC design.Source: §1 · pp. 2–3
2. Methods and Materials
The study combines prototype development, randomized group experimentation, repeated module questionnaires, and statistical modeling to compare LLM-NPC and traditional NPC interaction across seven task scenarios. The prototype versions differed only in NPC input method, enabling comparison of interaction effects across tasks and participants.
2.1. Research Proposal
- A workflow of questionnaire design, prototype development, group experiment, and results analysis supported the study’s research questions.Data came from basic-information, module-process, post-test, and supplementary open-ended questionnaires.Source: §2.1 · p. 3
2.2.1. Gameplay and System Design
- The prototype implemented seven scenarios: daily chatting, relationship building, task delegation, collaborative tasks, investigation and reasoning, negotiation and persuasion, and content creation.These scenarios were selected to represent common NPC interaction patterns in games.Source: §2.2.1 · p. 3, Table 1 · p. 3
- The scenarios varied in interaction demands, from emotionally oriented relationship building to structured task delegation and collaborative completion.Relationship building emphasized closeness and emotional support, whereas task delegation followed clear release, confirmation, and acceptance stages.Source: §2.2.1 · p. 4
2.2.2. Implementation results of the prototype
- The Campus Culture Week prototype included seven sequential modules, and both NPC versions were identical except for the NPC interaction input method.The modules included daily chatting, relationship building, task delegation, collaboration, investigation and reasoning, negotiation and persuasion, and content creation.Source: §2.2.2 · pp. 4–5, Figure 3 · p. 4, Figure 4 · p. 5
2.3. Experimental Design
- Participants were randomly assigned to traditional-NPC or LLM-NPC groups, completed the game modules and process questionnaires, and then completed a post-test questionnaire.Measures covered cognitive load, gaming experience, mechanistic variables, overall experience, and player characteristics.Source: §2.3 · p. 5
- The two-phase experiment recruited 130 analyzed participants, with 65 assigned to each NPC condition and no significant baseline differences in age, gender composition, or gaming frequency.The preliminary phase tested procedure feasibility and measurement applicability before its verified data were included in the final sample.Source: §2.3 · p. 5
2.4. Statistical analysis methods
- Mixed-effects models tested group, task-scenario, and interaction effects using repeated module measurements, while Welch’s t-tests compared post-test overall indicators.Bootstrap mediation analysis, regression within the LLM-NPC group, Cohen’s d, and Benjamini–Hochberg correction supplemented the main analyses.Source: §2.4 · p. 6
3. Results
Across 130 participants, LLM-NPCs consistently increased cognitive load without improving overall gaming experience, while effects varied by task context and specific experience mechanisms. Individual differences were mostly limited, although extraversion and neuroticism predicted cognitive load in regression analysis.
3.2.1 Module level differences during the game process
- LLM-NPCs produced higher module cognitive load but no significant module-level gaming-experience advantage over traditional NPCs.Cognitive load: b = 0.785, p < .001; gaming experience: b = -0.034, p = .806.Source: §3.2.1 · pp. 7–8
3.2.2. Intergroup differences in overall post-test evaluation
- LLM-NPCs significantly increased expression cost, perceived autonomy, and response uncertainty, whereas traditional NPCs had higher system usability and trust.No significant post-test differences appeared for overall gaming experience, social presence, target clarity, or preference/satisfaction.Source: §3.2.2 · pp. 8–9
3.2.3. Analysis of mediation mechanism
- Expression cost and response uncertainty mediated higher cognitive load, while autonomy positively mediated gaming experience and lower trust and usability mediated it negatively.The cognitive-load indirect effects were 0.2407 through expression cost and 0.0822 through response uncertainty; gaming-experience indirect effects were 0.1589, -0.1949, and -0.3995, respectively.Source: §3.2.3 · p. 9
3.3. The result of research question two (RQ2)
- Task context significantly moderated cognitive-load differences but not immediate gaming experience, with the largest cognitive-load gaps in content creation and relationship building.The group × module interaction was significant for cognitive load (F = 15.569, p < .001) but not module experience (F = 0.488, p = .818).Source: §3.3 · pp. 10–11
3.4. The result of research question three (RQ3)
- After correction, continuous traits and game preferences showed no stable univariate associations with cognitive load or gaming experience, but extraversion and neuroticism positively predicted cognitive load in regression.The regression model had R2 = 0.286; extraversion predicted cognitive load with b = 0.474, p = .031, and neuroticism with b = 0.5634, p = .047.Source: §3.4 · p. 11
3.5.1. Subjective evaluation of adaptability and willingness to re-experience
- Players associated natural-language interaction with open, everyday, and emotionally resonant modules, while cognitively demanding modules were viewed as better suited to preset options.Collaboration had the highest willingness to re-experience in both groups despite differing views of its interaction suitability.Source: §3.5.1 · p. 13, §3.5.2 · p. 13
3.5.3. Analysis of sense of social presence and sources of confusion
- Both groups reported strongest social presence during negotiation and greatest confusion during investigation, especially when information was difficult to organize.Negotiating for funding was the most socially present moment, while finding the key was the most confusing.Source: §3.5.3 · p. 13
3.5.4. Evaluation and improvement suggestions for interaction methods
- Players valued LLM-NPC freedom and immersion but reported time costs, weak guidance, and poor controllability, motivating more natural replies and additional prompts.Traditional NPCs were valued for convenient operation, clear direction, and stable controllability but criticized as rigid and restrictive.Source: §3.5.4 · pp. 13–14
4. Discussion
LLM-NPCs produce a double-edged effect: they increase cognitive load while offering limited overall gaming-experience improvement. Their costs and benefits vary by task openness, player traits, and interaction design.
4. Discussion
- LLM-NPCs increase cognitive load while providing limited improvement in overall gaming experience, creating a double-edged effect.Benefits from perceived autonomy are offset by higher expression cost, response uncertainty, lower usability, and lower trust.Source: §4 · p. 14
- Expression cost and response uncertainty are key pathways through which LLM-NPCs increase cognitive load.Perceived autonomy improves experience, but reduced system usability and trust counteract that benefit.Source: §4 · p. 14
- Task scenarios moderate cognitive-load effects, with stronger increases in open-ended relationship-building and content-creation modules.Preset options may better support efficiency, clarity, and controllability in more directive tasks.Source: §4 · p. 14
- Most individual characteristics show limited associations, but higher extraversion and neuroticism predict greater cognitive load.The proposed explanations involve greater expressive participation and heightened sensitivity to uncertain feedback.Source: §4 · p. 14
- Players associate natural-language interaction with relaxed emotional scenarios, while preset options better fit tasks requiring clear judgment and efficient execution.Negative evaluations emphasize time cost, effort, weak guidance, instability, and poor controllability; advantages include freedom, immersion, and surprise.Source: §4 · p. 14
- The study recommends differentiated deployment, hybrid input, and improvements in behavioral consistency, response stability, usability, and trustworthiness.LLM-NPCs are suggested for emotional communication, role-playing, co-creation, and open exploration rather than universal replacement.Source: §4 · p. 14
- Conclusions are constrained by the self-developed prototype, single-session design, limited individual-difference sample, and text-only interaction.The authors call for validation in realistic commercial games, longitudinal studies, and multimodal settings.Source: §4 · pp. 14–15
5. Conclusion
The study finds that LLM-NPCs combine greater cognitive burden with limited overall experience gains, so their design should be matched to task scenarios rather than replace traditional NPCs universally. Future work should test multimodal, longitudinal, commercial-game, and personalized designs.
5. Conclusion
- LLM-NPCs increase cognitive load while offering limited overall gaming-experience improvement, with benefits offset by expression cost, uncertainty, usability, and trust costs.Perceived autonomy is a positive component of the experience, but does not yield an overall advantage.Source: §5 · p. 15
- Task-sensitive deployment and interactive optimization are preferable to simply replacing traditional NPCs.The strongest cognitive-load effects occur in more open-ended scenarios such as content creation and relationship building.Source: §5 · p. 15
- Future research should examine multimodal interaction, long-term adaptation in commercial-game contexts, and scenario-based personalized NPC frameworks.These directions target presence, immersion, cognitive burden, adaptation, and broader application potential.Source: §5 · p. 15
Appendix A. Basic Information Questionnaire
The basic information questionnaire records participants’ demographics, gaming habits, NPC-interaction experience, game preferences, language-model use, AI attitudes, expression ability, communication anxiety, typing ability, devices, and personality traits.
Appendix A. Basic Information Questionnaire
- The questionnaire collects demographic and gaming-background information, including age, gender, profession, weekly gaming time, and prior NPC-interaction experience.It also asks about frequently played game genres.Source: Appendix A · pp. 17–18
- Participants report large-language-model usage frequency, use scenarios, self-rated proficiency, and attitudes toward communicating with AI or virtual characters.The listed use scenarios include information seeking, writing, creative generation, companionship, role-playing, and work tasks.Source: Appendix A · p. 18
- The questionnaire measures perceived language-expression ability, communication anxiety or withdrawal, typing ability, and preferred device.These variables capture self-assessed interaction-related characteristics.Source: Appendix A · p. 18
- Participants select among five personality traits: neuroticism, extraversion, openness, agreeableness, and conscientiousness.Each trait is accompanied by a brief behavioral description.Source: Appendix A · p. 18
Appendix B. Process Questionnaire
The process questionnaire uses seven-point ratings to assess cognitive load, gaming experience, and psychological mechanisms during each interaction module.
Appendix B. Process Questionnaire
- Cognitive-load items assess mental effort, time pressure, effort expenditure, and setbacks or difficulties during the module.Responses use a Likert 7-point scoring system.Source: Appendix B · p. 18
- Gaming-experience items assess immersion, naturalness, task completion, and willingness to continue interacting with the character.These items capture immediate module-level experience.Source: Appendix B · p. 18
- Psychological-mechanism items assess response-planning effort, perceived control, truthful character responsiveness, and clarity of the module goal.The measures connect interaction conditions with cognitive load and experience.Source: Appendix B · pp. 18–19
Appendix C. Post Test Questionnaire
The post-test questionnaire measured players’ gaming experience, autonomy, presence, response uncertainty, target clarity, system usability, trust, preference, and satisfaction across NPC interaction methods.
Overall gaming experience
- Players rated overall gaming experience, attention, immersion, task capability, performance, and ease of entering the game state.Source: Appendix C · p. 19
Autonomy
- Autonomy items assessed whether players could advance interactions, express intentions freely, and experience freedom without heavy system restrictions.Source: Appendix C · p. 19
Dimension of Presence
- Presence items assessed whether NPCs felt real, responsive, present, and perceptive of players’ expressions.Source: Appendix C · p. 19
Response uncertainty and target clarity
- Response uncertainty and target clarity items measured uncertainty about expressing intentions, NPC understanding, task goals, and progress direction.Source: Appendix C · p. 19
System usability and trust
- Usability and trust items assessed ease of learning and communication, interaction smoothness, added burden, NPC understanding, and dialogue reliability.Source: Appendix C · p. 19
Preference, satisfaction, and open feedback
- Preference, satisfaction, and open-ended items captured comparative preference, perceived experience improvement, module suitability, confusion, replay interest, and desired improvements.Source: Appendix C · pp. 19–20
1. Encoding method
The encoding method analyzed open-ended post-test responses about module suitability, replay willingness, immersion and confusion, and comparative advantages, disadvantages, and improvements.
1. Encoding method
- Open responses covered why interaction methods fit different modules, why players would replay modules, when they felt immersed or confused, and comparative evaluations.Source: §1 · p. 20
- Module-selection answers were used for grouping and context, while textual answers to questions 29, 31, 33, 34, 35, 36, and 37 were coded.Source: §1 · p. 20
- The smallest meaningful unit within each answer was coded separately, with multiple independent topics receiving multiple codes and non-substantive responses excluded.Source: §1 · p. 20
2. Encoding process
The encoding process combined material organization, pilot coding, independent formal coding, disagreement resolution, and finalized topic-frequency statistics.
2. Encoding process
- Researchers organized responses by question, participant group, and module while retaining semantically complete colloquial answers and excluding clearly invalid content.Source: §2 · p. 20
- Two coders independently pre-coded about 15% of samples to establish and refine a shared coding framework before formal coding.Source: §2 · p. 20
- Both coders independently segmented and classified all open-ended responses using the unified framework without sharing formal coding results.Source: §2 · p. 20
- Disagreements were compared, discussed, and coordinated using manual definitions and least-inference decisions, with consistency targeted at α>0.7.Source: §2 · pp. 20–21
- After reconciliation, researchers merged synonymous topics and calculated topic or module frequencies for reporting.Source: §2 · p. 21
3. Example of Core Theme Definition
The core-theme definitions distinguish why interaction methods fit tasks, what drives replay and presence, what creates confusion, and which advantages, burdens, and improvements players described.
3. Example of Core Theme Definition
- Preset options were defined as clearer for organizing information and reducing reasoning confusion, whereas natural language supported conversational atmosphere and emotional resonance.Source: §3 · p. 21
- Some tasks require only limited instructions when objectives are clear, while natural-language interaction can add operational, time, and comprehension costs.Source: §3 · p. 21
- Replay motivation included strong interest, ease, freshness or surprise, and participation through influence over the interaction.Source: §3 · p. 21
- Presence was associated with interaction resembling communication with a real person, while confusion involved excessive information, unclear next steps, unpredictability, and unstable responses.Source: §3 · p. 21
- Advantages and disadvantages included easy operation, clear direction, controllability, rigidity, restricted expression, weak authenticity, freedom, immersion, surprise, and time-consuming language work.Source: §3 · pp. 21–22
- Core limitation themes were lack of guidance and poor stability or controllability, including unclear advancement, output fluctuations, interpretation deviations, and unstable results.Source: §3 · p. 22
- Recommended improvements included more natural and coherent NPC responses, additional hints and guidance, and reduced thinking burden.Source: §3 · pp. 21–22