PaperlayerView exact revision on arXiv

Source-linked AI summary

Oops, Not Now: PEARL, a RAG-Based Support Agent for Gameplay and What Players Want from AI Help

Jiahong Li, Sai Siddartha Maram, Atieh Kashani, Ulia Zaman, Zhiyu Lin, Cameron Marano, Roger Azevedo, Jichen Zhu, Magy Seif El-Nasr

arXiv:2609.13718v1cs.HCcs.AI

TL;DR

AI tutoring pipelines can obscure their reasoning, while PEARL combines semantic knowledge retrieval and structural board matching to ground gameplay scaffolding in peer data. In the deployment, most participants gravitated toward Community and several disengaged from PEARL, although some found it complementary under solicited conditions; the authors frame the findings as a design agenda.

  • Problem

    AI tutoring systems can operate as opaque pipelines, limiting learners’ metacognitive engagement.

    Source: §I · p. 1

  • Method

    PEARL combines semantic knowledge retrieval with structural board-state matching to ground gameplay assistance in verified peer data.

    Source: §I · p. 2, §III · p. 3, B. Structural Board State Retrieval · p. 4

  • Results

    Most participants gravitated toward Community and several disengaged from PEARL, while participants engaging substantively with both tools saw distinct roles and some reported productive engagement under solicited conditions.

    Source: §VI · p. 7

  • Takeaways & Limitations

    The authors identify seven design themes: three addressed architecturally, two requiring interaction redesign, and two remaining open for AI gameplay support.

    Source: §VI · p. 7

    The N=10 sample limits generalizability to novice learners and non-technical domains, and the comparison cannot isolate retrieval architecture from modality and interaction differences.

    Source: §VI · p. 7

Abstract

from arXiv · show

AI-powered gameplay support agents hold promise for game-based learning, yet grounding generative models in structured game data remains an open challenge. We present PEARL (Parallel Education Agent for Reflection and Learning), a dual-component Retrieval-Augmented Generation (RAG) system that combines semantic knowledge retrieval with structural board-state matching to deliver contextualized scaffolding in Parallel, a puzzle game for learning parallel programming. PEARL operates on two input streams (natural language queries and board topology), retrieving both conceptual explanations of gameplay moves and peer-generated board states as evidence: capabilities unavailable to a standard Large Language Model (LLM) with game state access alone. In a qualitative evaluation (N=10) comparing PEARL against an existing community-based Open Player Model (OPM) visualization system, participants preferred the visualization system on perceived usefulness and reported higher frustration with PEARL; five of ten minimized or abandoned the AI tool during play. Proactive delivery, generic responses, and trust deficits drove disengagement, while a subset of four participants found PEARL's grounded explanations complementary to visualization in specific contexts where they initiated the interaction. We position PEARL as a deployed design probe whose failure modes inform a concrete design agenda for AI gameplay support, captured as seven open problems for the community.

I. INTRODUCTION

AI tutoring systems can obscure their reasoning, while game-based learning still needs scaffolding that interprets dynamic, structured game state and peer data. PEARL addresses this gap with semantic and structural retrieval, and the paper asks how it compares with visualization-based support and how players engage with it.

I. INTRODUCTION

  • Opaque tutoring pipelines can limit learners’ metacognitive engagement, while generic LLM advice risks hallucinations without verified peer traces or expert-annotated move classifications.The paper identifies a need for conversational scaffolding that interprets structured game state and peer data.Source: §I · p. 1
  • Open Player Models expose player traces, inferred competencies, and peer strategies, but learners must interpret visualizations themselves and can struggle to diagnose failures or move beyond trial-and-error.Prior Parallel deployments also found strategy fixation; conversational scaffolding is proposed as a complement to visualization.Source: §I · p. 2
  • PEARL combines semantic knowledge retrieval with structural board-state matching to ground gameplay support in verified peer data.The structural component addresses the imprecision of semantic embeddings for logic puzzles, where a misplaced edge can separate a solution from a deadlock.Source: §I · p. 2
  • The study evaluates PEARL in Parallel against visualization-based OPM support and examines player engagement and preferences during live gameplay.The research questions ask how the dual-component system supports gameplay assistance, how players engage with its explanations and retrieved boards, and what design considerations emerge.Source: §I · p. 2
  • The introduction identifies three core contributions, including a seven-theme design agenda grounded in a within-subjects qualitative deployment (N=10).The agenda includes on-demand availability, diagnostic failure feedback, and preservation of agency.Source: §I · p. 2

2. A characterization of PEARL as a deployed de-

Peer data can support reflection, but visualizations leave learners to interpret it, and the paper presents conversational scaffolding as a complement. The work situates PEARL as a deployed design probe and frames its findings as conditions for complementary support and failure modes to address.

2. A characterization of PEARL as a deployed de-

  • PEARL is presented as a dual-component RAG system for grounding LLM-based gameplay agents in structured game data.The deployment characterizes when retrieval-grounded help complements visualization and the failure modes associated with disengagement.Source: §2 · p. 2

A. Exploring Peer Learning and Peer Play

  • Peer-play data can support exploration of alternative learning paths, but raw community data is dense and difficult for novices to comprehend.Telemetry captures learning processes as they unfold.Source: A. Exploring Peer Learning and Peer Play · p. 2
  • Game-based peer analytics include post-match data and racing-game ghosts, while OPM visualizations help players compare actions with alternative paths.Prior work reported improved reflection from exposing community data, but serious-game applications remain limited.Source: A. Exploring Peer Learning and Peer Play · p. 2
  • Interactive visualizations still require independent interpretation, and learners have requested more explicit guidance, motivating conversational scaffolding.Standard LLMs without verified peer traces risk generic or hallucinated advice.Source: A. Exploring Peer Learning and Peer Play · p. 2

B. RAGs and LLMs in Education

RAG can ground generative responses in external knowledge, but game-based support still needs to interpret structured gameplay data. This paper repurposes retrieval to surface peer trajectories through similar gameplay impasses.

B. RAGs and LLMs in Education

  • RAG grounds generative models in external knowledge, but education systems remain largely text-centric despite serious games embedding scaffolding in gameplay.The paper frames interpretation of dynamic gameplay states without compromising pedagogical integrity as the gap it addresses.Source: B. RAGs and LLMs in Education · p. 3

C. LLMs and RAG in Game-Based Learning

  • Empirical findings on LLMs in games are mixed: adaptive mechanisms improved performance but increased cognitive load, while LLMs struggled with game-specific reasoning.RAG is described as grounding generative models in external knowledge bases.Source: C. LLMs and RAG in Game-Based Learning · p. 3
  • RAG in gaming has mainly supported narrative consistency and memory, leaving structured gameplay data underused and tutoring systems reliant on static gold-standard moves.PEARL instead treats player traces and board states as queryable data, retrieving trajectories from peers who navigated similar impasses.Source: C. LLMs and RAG in Game-Based Learning · p. 3

D. Parallel – Research Platform

Parallel uses spatial puzzles to teach coordination of concurrent processes, while PEARL combines semantic retrieval and structural board matching. Its semantic stage draws on an expert-annotated knowledge graph to produce explanations and recommendations for subsequent structural retrieval.

D. Parallel – Research Platform

  • Parallel teaches parallel programming through spatial puzzles in which players coordinate concurrent processes by placing and linking signals and semaphores.Its board zones support spatial analysis, and the text suggests this structure may generalize to broader gaming contexts.Source: D. Parallel – Research Platform · p. 3

III. THE PEARL SYSTEM

  • PEARL pairs semantic knowledge retrieval with structural board-state matching because semantic embeddings can be imprecise for logic puzzles.A single misplaced edge can distinguish a working solution from a deadlock.Source: §III · p. 3

A. Semantic Knowledge Retrieval

  • The semantic component uses an expert-annotated knowledge graph whose move annotations include classifications, concepts, misconceptions, and recommended alternatives.The graph contains 133 knowledge units across the two study levels and 391 directed recommendation edges that support sequential chaining to structural retrieval.Source: A. Semantic Knowledge Retrieval · p. 3
  • Each player action becomes a composite query combining move notation, level context, teaching content, recommended alternatives, and serialized board topology.Similarity-ranked knowledge units are synthesized into explanations and recommended next moves for the structural stage.Source: A. Semantic Knowledge Retrieval · p. 3
  • Semantic retrieval returns four alternative good moves for correct actions or three similar mistakes for incorrect actions; structural retrieval retains five solution boards ranked by Hamming distance.Structural candidates are restricted to boards containing recommended edges, using binary adjacency vectors of 64 dimensions on Level 3 and 256 on Level 13.Source: A. Semantic Knowledge Retrieval · p. 3

B. Structural Board State Retrieval

PEARL represents board states structurally and retrieves peer boards using similarity measures and pedagogically aware ranking. Validation reinforces a similar strategy, while correction highlights missing success edges in nearby passing boards.

B. Structural Board State Retrieval

  • The board is modeled as a directed graph of zones and synchronization links, then flattened into a binary adjacency vector of length |V|^2.Each vector dimension represents a discrete edge-present or edge-absent property, so Hamming Distance counts structural edits between boards.Source: B. Structural Board State Retrieval · pp. 3–4
  • Hamming Distance measures structural differences, while Jaccard Similarity measures overlap between board states.The system uses these measures to retrieve and rank structurally relevant peer boards.Source: B. Structural Board State Retrieval · p. 4
  • Validation Mode minimizes Hamming Distance to retrieve structurally similar peer boards and reinforce the player's current strategy.The peer-board corpus contains 13 sessions from Levels 3 and 13; structural matching avoids the imprecision of semantic embeddings for critical board differences.Source: B. Structural Board State Retrieval · pp. 3–4
  • Correction Mode retrieves initially similar passing boards that differ in success edges, highlighting the structural changes needed to reach a correct solution.Candidates are restricted to peer boards that led to a correct solution and prioritized by closeness to the player's current state.Source: B. Structural Board State Retrieval · p. 4

C. Generative Synthesis and Integration

PEARL combines semantic explanations with peer-board evidence in a single generated response. Its default proactive delivery triggers retrieval and synthesis after each distinct link creation.

C. Generative Synthesis and Integration

  • PEARL chains semantic and structural retrieval: recommended moves filter eligible peer boards, and a final LLM call combines both evidence types into one response.The response pairs explanatory text with visually rendered peer boards.Source: C. Generative Synthesis and Integration · p. 4
  • Unlike an LLM with game state, PEARL accesses the expert-annotated knowledge graph and peer-trace database to ground advice in community evidence.The standard LLM can explain mechanics and produce Socratic prompts, but lacks access to those resources and structural matching.Source: C. Generative Synthesis and Integration · p. 4
  • By default, PEARL runs retrieval and synthesis after every distinct link creation, with per-session deduplication but no cooldown or move-count throttle.Users can disable proactive delivery through a sidebar control; this delivery model drove disengagement patterns reported in the results.Source: C. Generative Synthesis and Integration · p. 4

IV. EVALUATION

The study used a counterbalanced, within-subjects qualitative comparison of PEARL and Community with ten computer-science participants. It characterized interaction patterns rather than testing performance superiority.

IV. EVALUATION

  • The evaluation compared PEARL with Community, an OPM visualization system where players interpret peer data themselves, to characterize engagement with scaffolding versus visualization.The study was not designed to establish performance superiority.Source: §IV · p. 4
  • Ten participants with at least an undergraduate computer-science degree played Levels 3 and 13, selected to span calibrated difficulty while remaining solvable within 20 minutes.Level 3 introduces synchronization in eight zones; Level 13 uses 16 zones and requires multi-thread coordination with signals and semaphores.Source: §IV · p. 4
  • Participants completed counterbalanced PEARL and Community conditions, followed by TAM, NASA-TLX, and PENS surveys, a game-concepts knowledge test, and an interview.Sessions lasted 20 minutes, with a tutorial before each condition.Source: §IV · pp. 4–5
  • Researchers used deductive codebook thematic analysis, double-coding one transcript with 92.9% agreement (Cohen’s κ = .92) before dividing the remaining transcripts.Coded data were charted by participant, condition, and theme.Source: §IV · p. 5

V. RESULTS

Participants generally preferred Community, though four found PEARL complementary when they engaged with it. Proactive delivery, response quality, and trust issues contributed to resistance.

V. RESULTS

  • The results distinguish two ways of exposing community peer data: Community uses visualization, while PEARL uses retrieval-grounded scaffolding.The analysis reported three overarching patterns corresponding to the research questions.Source: §V · p. 5

A. RQ1: Complementary but Unequally Adopted

  • Five of ten minimized or abandoned PEARL; Community scored higher on perceived usefulness (M=4.98 vs. M=4.57), while PEARL frustration was higher (M=56.5 vs. M=43.2).Survey scores were descriptive; overall NASA-TLX workload was comparable between conditions.Source: A. RQ1: Complementary but Unequally… · p. 5
  • Resistance had varied sources: repetitive or stale responses eroded trust, some rejected hints as hand-holding, and one participant did not usually seek help first.Participants described PEARL's proactive delivery as the most consistent thread of resistance.Source: A. RQ1: Complementary but Unequally… · pp. 5–6
  • Four of ten participants who engaged substantively with both tools saw distinct roles for PEARL and Community.They valued PEARL's contextual awareness and board-grounded feedback, which Community's fixed peer screenshots could not provide.Source: A. RQ1: Complementary but Unequally… · p. 6

B. RQ2: Conceptual Grounding Through Dual Retrieval

When retrieval worked, PEARL’s semantic and structural components supported distinct forms of help, but usefulness varied with response quality and interaction style. Players differed in when they sought help, how much autonomy they wanted, and the level of detail they preferred.

B. RQ2: Conceptual Grounding Through Dual Retrieval

  • Semantic retrieval named implicit programming concepts, while peer boards offered reassurance and alternative approaches when retrieval worked.Seven of ten participants found peer struggles reassuring, and five of ten used peer boards to discover structurally different approaches.Source: B. RQ2: Conceptual Grounding Through… · p. 6
  • Three of ten participants found semantic responses generic or repetitive, and some received useful explanations too late to help during early play.P3 said the explanation clarified his failure cause but was not helping during the first 20 minutes.Source: B. RQ2: Conceptual Grounding Through… · p. 6
  • Four of ten participants found structurally matched boards confusing when they looked visually dissimilar despite being conceptually aligned.One participant found peer boards discouraging rather than collaborative.Source: B. RQ2: Conceptual Grounding Through… · p. 6
  • Suggested questions could scaffold strategy, while direct questions elicited explicit answers; PEARL sometimes gave too much help on easier levels.P10 described suggestions about zones and deduplication as scaffolding; P4 said PEARL “gives too many answers” on easier levels.Source: B. RQ2: Conceptual Grounding Through… · p. 6

C. RQ3: Autonomy, Timing, and Granularity as Design Axes

  • Participants differed in autonomy preferences: some resisted unsolicited help, while others sought tools from curiosity or only after repeated failure.P2 resisted tools until time pressure, whereas P6 opened them out of curiosity.Source: C. RQ3: Autonomy, Timing, and… · p. 6
  • Help-seeking timing varied, from waiting until repeatedly stuck to opening tools out of curiosity or under time pressure.P8 waited until consistently hitting a wall; P6 used tools from curiosity, and P2 resisted until time pressure.Source: C. RQ3: Autonomy, Timing, and… · p. 6
  • Preferred help granularity ranged from hints near a solution to full explanations when far from one, with some favoring complete solutions.P6 recognized the value of hints despite wanting more, while P1 preferred “the whole picture.”Source: C. RQ3: Autonomy, Timing, and… · p. 6

D. Proactive Help and the Clippy Failure Mode

Participants objected most consistently to PEARL’s proactive interruptions, even when they valued its help. Their objections concerned unsolicited delivery rather than the content of grounded assistance.

D. Proactive Help and the Clippy Failure Mode

  • Four of ten participants cited proactive interruption as a concrete negative, including advice after moves or before they asked for it.Even a participant who valued PEARL objected to feedback on every update before finishing.Source: D. Proactive Help and the Clippy… · p. 6
  • Participants who criticized unsolicited delivery also endorsed grounded explanations when they sought help on their own terms.An agent responding to every move felt more like an unwanted helper interrupting work in progress than a tutor.Source: D. Proactive Help and the Clippy… · p. 6
  • The authors warn that interaction design can undermine retrieval-grounded help, likening the pattern to Clippy and other proactive assistants.The concern was unsolicited help during work in progress.Source: D. Proactive Help and the Clippy… · pp. 6–7

E. Knowledge Test

The post-session knowledge test was descriptive and could not attribute scores to either condition. The discussion frames PEARL as conditionally complementary to visualization, while identifying interaction challenges and limitations that shape the design agenda.

E. Knowledge Test

  • Mean post-session accuracy was 5.50 of 8 (68.8%, SD=0.94; range 4.0–7.0), but the test cannot attribute understanding to either condition.The within-subjects test was administered once after both conditions, and N=10 precluded inferential testing of condition effects.Source: E. Knowledge Test · p. 7

VI. DISCUSSION

  • The authors argue AI gameplay agents should default to passive availability and respect player-initiated interaction boundaries.They frame the deployment’s most consistent design lesson as avoiding a Clippy-like interaction model.Source: §VI · p. 7
  • PEARL is presented as conditionally complementary to visualization-based OPMs, with explanations addressing the interpretation gap around peer data,.Participants who engaged with both tools described distinct roles: concrete evidence from visualization and personalized explanation from PEARL.Source: §VI · p. 7
  • Across ten participants, seven recurring themes form a design agenda: three addressed by architecture, two requiring interaction redesign, and two remaining open.The open themes are adaptive calibration to preserve agency and live peer interaction beyond retrieved boards.Source: §VI · p. 7
  • Participants framed gameplay as intrinsically motivated, and some said unsolicited help “spoils the fun.”The authors suggest motivation context shapes receptivity.Source: §VI · p. 7
  • The N=10 sample of CStrained participants limits generalizability to novice learners and non-technical domains, while modality and interface differences prevent isolating retrieval effects.The comparison characterizes contrasting support paradigms rather than providing an architectural comparison.Source: §VI · p. 7
Loading 2609.13718v1…