Roleplay-doh: Enabling Domain-Experts to Create LLM-simulated Patients via Eliciting and Adhering to Principles

1 Introduction

• Mental-health simulation is constrained by privacy concerns around therapy transcripts, while existing prompting approaches can be difficult for experts to use or inefficient to refine.Application-specific datasets can be infeasible to obtain, and researchers may need to translate experts’ qualitative insights into prompt changes.Source: §1 · p. 1
• Roleplay-doh converts counselors’ natural-language feedback into principles that guide an LLM-prompted AI patient, without requiring counselors to write prompts.The tool uses the principles alongside a persona description to generate roleplay responses.Source: §1 · p. 2
• GPT-4 simulations struggled with multipart and context-dependent principles; the adherence pipeline decomposes rules into yes/no questions, checks their applicability, and self-refines responses.The initial analysis found difficulty adhering to multipart principles and misapplying principles intended for specific contexts.Source: §1 · p. 2
• 35% win rate and 5% loss rate for principle following, and 35% win rate and 10% loss rate for dialogue consistency, give the adherence pipeline the strongest results against ablations.Preferences were measured against a baseline that did not self-refine its output.Source: §1 · p. 2
• In a study with 25 expert counselors, creators and third-party counselors judged Roleplay-doh’s AI patients more authentic, closer to real cases, and better prepared for training.The study compared AI patients created by describing real-patient scenarios with those refined using Roleplay-doh principles.Source: §1 · p. 2

Utility of Simulated Partners

• LLM simulations have been explored for practice in teaching, conflict resolution, and counseling, but transfer to real-world situations requires faithful simulations.Earlier digital patient systems offered tailored dialogue but limited practice contexts.Source: §2 · p. 3

Aligning Simulation with Domain Experts

• Preference-data alignment can demand substantial expert time, while natural-language feedback converted into principles offers a more efficient route to customizing model behavior.Roleplay-doh builds on user-driven chatbot design that elicits qualitative feedback and converts it into constitution principles.Source: §2 · p. 3
• Prior mental-health simulation work used researcher-refined prompts; Roleplay-doh instead lets domain experts create and refine simulations directly.Earlier systems elicited expert feedback for patient behaviors such as colloquial or resistant language and varied conversation styles.Source: §2 · p. 3

Text Generation with LLMs

• Expert-defined principles can be multifaceted and context-specific, so Roleplay-doh adds concise-question decomposition and applicability checks to constrained dialogue generation.This differs from general criteria such as relevance and helpfulness used in prior self-refinement work (Madaan et al., 2024).Source: §2 · p. 3

3.1 Initial Tool Design Rationale

• Counselors can give kudos, critiques, or rewrites, which an LLM translates into principles specifying desired behavior and when it should apply.Principles are intended to generalize beyond the dialogue context where feedback arose.Source: §3.1 · p. 3
• The tool lets counselors rewind a response and generate a replacement using the updated constitution, one response at a time, to make testing manageable.Counselors can identify whether a single response resembles a real patient without comparing multiple responses.Source: §3.1 · p. 4
• Roleplay-doh prompts the LLM with current principles rather than fine-tuning model weights, supporting quicker and less expensive iterative steering.The prompt asks GPT-4 to generate the patient’s next dialogue response, which early testing found could mitigate role-consistency issues.Source: §3.1 · p. 4

3.2 Pilot Testing

• Counselors found imagined “realistic” patient creation ambiguous because many different behaviors could be equally realistic; recalling a challenging past case clarified the task.The reframing was used in later sessions.Source: §3.2 · p. 4
• 20% (55/276) of GPT-4 responses were rated as only moderately, slightly, or not at all satisfying on principle following and dialogue-context appropriateness.The identified failure modes included difficulty following many or complex principles, awkward dialogue, and applying situational principles when their conditions were absent.Source: §3.2 · p. 4

4 Roleplay-doh

• Roleplay-doh helps counselors customize AI patients by transforming qualitative feedback into principles that govern patient behavior.Counselors can provide feedback while interacting with the AI patient, and the tool turns it into constitution principles.Source: §4 · p. 4
• The pipeline rewrites expert principles as yes/no questions and generates additional questions about dialogue conventions such as coherence and consistency.The additional criteria address conventions that may not be specified in the expert-defined principles.Source: Figure 2 · p. 5, §4 · p. 5
• An applicability and adherence evaluator marks irrelevant questions N/A and otherwise checks whether the response follows each question.For example, willingness to engage in a suggested activity is assessed only if the therapist proposes one.Source: §4 · p. 5
• If any applicable question receives No, the pipeline rewrites the response conditioned on the evaluation results, ideally passing all questions.The sequence combines question generation, evaluation, and response refinement.Source: §4 · p. 5, Figure 2 · p. 5

5 User Study using Roleplay-doh

• The within-subjects study with 25 counseling experts compared scenario-only simulations against simulations refined with Roleplay-doh principles.Counselors rated both AI patients, and also reported on their experience using the tool to define principles.Source: §5 · pp. 5–6

5.1 Creator Perceptions

• Creators rated Scenario+ExpertPrinciples patients significantly higher than Scenario-Only on all measures except role consistency, where both scored highly.They described scenario-only patients as too articulate, forthcoming, and cooperative, with limited emotional depth.Source: §5.1 · p. 6

5.2 Creating Principles with Roleplay-doh

• Counselors created 123 principles across 25 AI patients, including concise and colloquial styles and behaviors associated with exploration, comforting, and action.Initial skepticism about seeking help appeared in principles for 14 patients.Source: §5.2 · p. 6
• Principles sometimes conflicted: some counselors wanted disorganized, conflicted responses, while others preferred concise and direct ones.Preferences also differed on whether patients should proactively seek advice or simply share thoughts and feelings.Source: §5.2 · p. 6
• Counselors found it easy and efficient to turn feedback into principles, though 11.4% of principles required manual editing.The reported means were 6.12 for ease and 6.3 for efficiency.Source: §5.2 · p. 6

5.3 Third-Party Comparison

• Five counselors from the creator study served as third-party judges, rating the same six dimensions with questions adjusted for an external perspective.Their transcripts were presented in randomized order to reduce awareness of the condition.Source: §5.3 · pp. 6–7
• Third-party counselors rated AI patients with expert-defined principles higher on authenticity, typical-case resemblance, training-partner readiness, and likelihood of recommending them to novices.The increase in ratings was smaller than in the creator study.Source: §5.3 · p. 7, Table 1 · p. 7

6 Evaluation of Principle-Adherence

• Experts ranked responses for consistency and principle adherence and marked awkward style, using 40 conversation turns selected from error categories.Three expert annotations were aggregated by majority vote; response order was randomized and ties were allowed.Source: §6 · pp. 7–8
• 35% wins and 10% losses on M1, 35% wins and 5% losses on M3, and 30% wins and 15% losses overall compared Full with No Critique.Full had the highest win/loss rates among the ablations on M1 and M3, and the strongest overall performance.Source: Figure 3 · p. 8, §6 · p. 8
• The ablation findings highlight the importance of both the Principle-as-Questions Rewriter and the Automatic Principle Generator.Naive had a disproportionately high tie rate across metrics, indicating that it rarely produced better responses even for error cases.Source: §6 · p. 8
• 2.5% of Full responses were judged awkward, compared with 15% for No Critique and 15% for No Autogenerated Criteria.The 12.5-percentage-point gap between Full and No Critique was reported as a reduction in awkward responses.Source: §6 · p. 8

7 Conclusions

• Roleplay-doh converts expert feedback into natural-language principles and was validated for creating AI patients as roleplay partners for novice counselors.The paper also presents a principle-adherence pipeline to reduce responses that fail to follow expert principles or dialogue conventions.Source: §7 · p. 8
• Studies with mental health counselors creating and comparing AI patients found that Roleplay-doh supports refinement toward authentic and more practice-ready simulations.The conclusion describes these findings within the AI-patient training-partner task.Source: §7 · p. 8
• Potential future domains include roleplay practice for teaching, coaching, conflict resolution, and negotiations.Source: §7 · p. 9

• The AI patients were designed for challenging cases useful to novice counselors, which intentionally restricts diversity in patient scenarios.Source: Limitations · p. 9
• Counselors stopped giving feedback before conversations naturally ended, so their principles may not address all underlying issues.Source: Limitations · p. 9
• Text-based dialogue omits non-verbal cues such as tone, facial expression, and posture that may help psychotherapists empathize with and support patients.Source: Limitations · p. 9

Ethics Statement

• AI patients may create overconfidence, produce harmful responses, and cannot replace peer roleplay; they should supplement training, with user warnings and traditional certification before real-patient work.Counselors also recommended participatory design before integrating AI patients into existing practices; study consent forms disclosed potential side effects.Source: Ethics Statement · p. 9

A Background of User Participants

• The IRB-approved studies included online peer counselors, experienced counselors recruited through Upwork, and university peer counselors and psychotherapy PhD students; domain experts received at least $25 per hour.The 7 Cups participants had supported 30 or more members; Upwork participants had counseling education or extensive support experience.Source: Ethics Statement · p. 9, A Background of User Participants · p. 11

B Pilot Testing with Expert Counselors

• Six counselors piloted Roleplay-doh in 90-minute sessions by refining a shared loneliness scenario through feedback and principles; four pilot-created patients were later selected for diverse coauthor testing.Some pilot participants also created additional patient scenarios, and coauthors assessed adherence to expert-defined principles and dialogue conventions.Source: B Pilot Testing with Expert Counselors · pp. 11–12

C Evaluating principle-adherence of GPT-4 direct prompting

• The direct-prompting evaluation rated 276 responses from 16 conversations across four AI patients for principle adherence and dialogue appropriateness, using coauthors as proxies for novice counselors and annotators.Conversations averaged 17.25 responses (range 12–22), and each response was scored by one coauthor.Source: C Evaluating principle-adherence of… · p. 12

D Roleplay-doh Interface for Making Constitutional Principles for LLM Simulation

• The interface fixes prevent duplicate feedback submissions and place a Convert button beside each feedback box to make turning feedback into principles easier.The final version also uses the principle-adherence pipeline to generate simulation responses.Source: D Roleplay-doh Interface for Making… · p. 12

E.1 Principle Elicitation Prompts

• Elicitation prompts use a system instruction, one-shot example, conversation context, and JSON output; kudos, critique, and rewrite prompts generalize principles from positive feedback, negative feedback, or a desirable rewrite.The rewrite prompt first describes the difference between undesirable and desirable responses, then derives a principle intended to produce the desirable response.Source: §E.1 · p. 12, §E.1.1 · p. 12, §E.1.2 · p. 13

E.2 Dialogue-Simulator Prompt for Generating Response

• The dialogue simulator directly prompts GPT-4 Turbo with the patient scenario, constitution principles, and conversation history, using temperature 0.3.Source: §E.2 · p. 15

E.3 Principle-Adherence Prompting Pipeline

• Because context length affected principle-adherence reliability, the pipeline splits calls into stage 1 question rewriting and automatic principle generation and stage 2 critiques and response rewriting.The tested breakdown used prompts developed against 20 error cases and did not require parallel branches.Source: §E.3 · p. 15
• The full pipeline checks principles, marks irrelevant criteria N/A, and revises responses that fail, with instructions to avoid paraphrasing and preserve persona and conversation context.Its prompts use structured JSON and self-critique; the context-relevance stage was iterated to prevent overly verbose or coherent rewrites and rewrites that merely repeated the original error.Source: §E.3 · pp. 16–17

F Principle Adherence Naive

• The naive variant evaluates a response against the principles and rewrites it in the same prompt if inappropriate, otherwise repeating it.It uses the same system prompt as the full module and requests JSON output with reasoning.Source: F Principle Adherence Naive · p. 17

G Full User Flow

• The creator study compared a scenario-only AI patient with one incorporating expert principles, using the same counselor-written scenario in both sessions.Participants first chatted for 10 minutes with the scenario-only patient, then for 30 minutes with the principles-based patient, adding principles as the conversation progressed; they evaluated each afterward.Source: G Full User Flow · pp. 17–18

H Creator Study Measures

• The creator-study questionnaire evaluated AI-patient roleplay and counselors’ experience using Roleplay-doh to define principles; all items used a 7-point Likert scale (1=Strongly disagree, 7=Strongly agree), except where noted.Source: H Creator Study Measures · p. 18

I Worst-Case Analysis of Tool Experience

• Follow-up feedback that clarified or modified prior feedback could produce overly similar principles, leaving counselors repeating feedback without changing the AI patient’s responses.Counselors reported difficulty writing specific rules and observed that the feedback-to-principle loop did not always incorporate their follow-up input effectively.Source: I Worst-Case Analysis of Tool Experience · p. 18
• Imprecise conversion of specific feedback into generalized principles made the principles difficult for counselors to edit and articulate.Counselors also found it hard to frame feedback in an overarching way rather than directing it at a specific part of a response.Source: I Worst-Case Analysis of Tool Experience · p. 18
• The authors suggest templates, support for combining overlapping feedback, and ways to move between specific and general feedback as future improvements.These are proposed directions to address challenges in formulating rules, integrating follow-up feedback, and abstracting principles.Source: I Worst-Case Analysis of Tool Experience · p. 18

J.1 Third-party measures

• Third-party counselors answered six measures for both AI patients, with Member Bot A and B denoting first- and second-read transcripts and presentation order randomized.The comparison accounted for the randomized order in which judges saw the two AI patients.Source: §J.1 · p. 18

J.2 Statistical Model and Power Analysis

• The linear mixed-effects model treated treatment and creator ID as fixed effects and annotator as a random effect to account for nonindependent ratings.The model was selected because multiple annotators rated the same AI patients and pilot annotator agreement was not always high.Source: §J.2 · p. 19
• Expected model parameters were estimated by fitting the model to early study data containing two annotations for each AI-patient pair from 17 counselors.The analysis extracted fixed effects, the random-effects covariance matrix, and residual variances.Source: §J.2 · p. 19
• The simulation-based power analysis varied the number of unique annotators from 2 to 6 across trials to estimate detection of a treatment effect of 0.52 at α = 0.05.The authors used the simr package in R for the mixed-effects model power calculation.Source: §J.2 · p. 19
• 80% power was achieved with 5 third-party judges in the simulation-based power analysis across 300 trials.The analysis used a linear mixed-effect model.Source: Figure 5 · p. 19, §J.2 · p. 19

J.3 Why is the effect of Expert Principles smaller when rated by a third-party?

• Krippendorff’s α for third-party preferences across the six measures ranged from 0.046 to 0.232, indicating low-to-moderate agreement among the five counselors.The authors identify lower agreement as one reason the third-party effect was smaller than the creator-study effect.Source: §J.3 · p. 19
• Counselors who disagreed on the preferred patient could note the same behavior but differ on whether the ExpertPrinciples patient sounded too formulaic or more expressive and realistic.The example illustrates that counselors can disagree about which principles matter most for an authentic roleplay.Source: §J.3 · pp. 19–20

J.4 Creator Study Conversation Lengths

• Table 9 presents descriptive statistics for conversation lengths in the creator-study dialogues between creators and AI patients.The supplied passage does not state the statistics themselves.Source: §J.4 · p. 20

K Detailed Results for principle-adherence pipeline Evaluation

• Annotator agreement on the randomly selected testcases was 0.2–0.6, which the authors characterize as fair agreement.The random testcases comprised 50 conversation turns sampled from user-study logs.Source: K Detailed Results for… · p. 20
• On random testcases, [Full] beat [No Critique] on M1 (W: 15%; L 2%), M3 (W: 14%; L 4%), and overall ranking (W: 18%; L 4%).For M2, all annotators reported no awkward responses for all methods.Source: K Detailed Results for… · p. 20

Annotation procedure

• In 34/50 random testcases, all methods produced the same response; these cases were not annotated and received rank 1 for every method.Identical responses were deduplicated before presentation, and these cases were excluded from Krippendorff’s α calculations.Source: L Annotation Interface for… · p. 20

Study interfaces

• The study interfaces guide scenario description with five questions, preview an AI patient, and show scenario-only and scenario-plus-expert-principles conversations.The appendix labels these as the introduction, Part I, and Part II interfaces.Source: Figure 6 · p. 21, Figure 7 · p. 21, Figure 8 · p. 23

Annotation interface

• The principle-adherence annotation interface collects judgments on dialogue-history consistency (M1), response awkwardness (M2), principle adherence (M3), and overall ranking.The overall-ranking interface also includes a free-text rationale field.Source: Figure 20 · p. 30, Figure 21 · p. 30, Figure 22 · p. 31

Measures and statistics

• The appendix lists creator and third-party counselor rating measures, descriptive conversation statistics, and Krippendorff’s α tables for error and random testcases.The third-party comparison covers rating differences and preference between Scenario+ExpertPrinciples and Scenario-Only patients.Source: Table 5 · p. 33, Table 7 · p. 33, Table 9 · p. 34