Source-linked AI summary
Learning to GUIDE the Group: Self-Learning LLM Facilitation Across Multi-Party Tasks
Ziyi Liu, Tzu-Sheng Kuo, Jiale Liu, Deqing Fu, Nithum Thain, Crystal Qian
arXiv:2610.16231v1cs.AI
TL;DR
Existing multi-party facilitation research is described as task-specific and limited in validated human evidence and relational evaluation. GUIDE learns a generalized playbook from human interactions and iteratively refines it with an aligned LLM judge. It beats three expert prompts on held-out decisions and, in live episodes, preserves task outcomes while improving speaking comfort and reducing friction with fewer turns and words.
Problem
LLM facilitation methods remain largely task-specific, with limited validated empirical evidence and evaluations that often prioritize transactional outcomes over interpersonal signals.
Source: §1 · pp. 1–2Method
GUIDE induces a structured facilitation playbook from real human interactions and iteratively refines it using an LLM judge aligned with human annotators.
Source: §1 · p. 2Results
61.4% win rate against three domain-specific expert prompts on held-out decisions; across 462 live episodes, the learned policy preserved task outcomes, improved speaking comfort, and reduced friction while using 25% fewer turns and 29% fewer words.
Source: §6 · p. 11Takeaways & Limitations
The results provide evidence that a generalized facilitation policy can be learned from task-specific human interactions and perform across diverse group settings.
Source: §6 · p. 11Offline optimization evaluates decisions independently and does not model the cumulative effect of sustained silence on participants’ experience of the facilitator.
Source: §7 · p. 11
Abstract
from arXiv · showhide
As large language models are increasingly adopted as multi-party communication facilitators, there is a growing demand for agents capable of balancing diverse group dynamics across contexts. However, existing LLM facilitators are often fine-tuned for task-specific objectives; generalized LLM-based facilitation methods that can adapt to different types of tasks and diverse group dynamics are underdeveloped. To address this gap, we present GUIDE (Generalized Unified Intervention for Dynamic Exchanges, a framework for developing generalized LLM-based facilitators across different multi-party contexts. GUIDE evaluates facilitation across both objective task outcomes and subjective metrics, including psychological safety and perceived friction. Within this framework, we study whether LLMs can self-learn generalized facilitation strategy offline to real-time interventions. We first collect multi-party human interactions guided by task-specific expert LLM facilitators (N=432 conversations). We then employ a self-learning method in which an LLM induces an actionable strategic playbook from conversations and iteratively refines it using feedback from an LLM judge aligned with human annotators (N=20), developing a single generalized facilitator that outperforms the three task-specific expert facilitators on held-out conversations 61.4% win rate. Finally, we evaluate these improvements empirically; in a real-time online empirical study, the self-learned generalized facilitator preserves facilitator performance and group productivity, improves speaking comfort and reduces interpersonal friction, while using 24.7% fewer turns and 29.0% fewer words than the expert prompts. These findings provide empirical evidence that LLMs can learn generalizable, task-agnostic social facilitation from task-specific human interactions, on-par with hand-crafted task-specific prompts across diverse group settings.
1. Introduction
GUIDE addresses the need for facilitation that generalizes across multi-party tasks while considering both task outcomes and participants’ social experience. Its learned facilitator outperforms task-specific experts on held-out conversations and preserves performance in live evaluation while improving reported experience with fewer interventions.
1. Introduction
- Prior work is described as narrow and task-specific, with limited validated human evidence and emphasis on transactional outcomes rather than relational signals.The paper also notes manual prompt engineering can leave unpredictable conversational corner cases uncovered.Source: §1 · pp. 1–2
- GUIDE learns a human-readable playbook from real facilitated conversations and evaluates task outcomes, participant experience, and perceptions of the facilitator across three task types.The tasks are open-ended discussion, multi-party negotiation, and consensus-building.Source: §1 · p. 2, Figure 1 · p. 2
- Learning uses blind pairwise judgments of candidate interventions against the preceding playbook, while comparison with expert facilitators is reserved for the held-out test set.This approach addresses a setting where interventions lack an objective verifier or sufficiently faithful counterfactual simulation.Source: §1 · p. 2
- 61.4% win rate over 736 decision points across domains; in live groups, the learned facilitator improved experience while intervening 25% less often and using 29% fewer words.The live study also found higher psychological safety and feeling heard, with decreased friction and anger.Source: §1 · p. 3
- The stated contributions include a human-aligned LLM judge for turn-level evaluation and an empirical study reporting improved participant experience, reduced friction, and conversational fairness.The judge’s agreement with human annotators is reported as Cohen’s κ=0.70.Source: §1 · p. 3
2. Task Formulation
The task formulation pairs three human participants with an AI facilitator in a real-time text discussion, then tests facilitation across three distinct collaborative scenarios. Evaluation combines participant experience, perceptions of the facilitator, and task outcomes.
2.1. LLM Facilitation in Multi-Party Conversation
- Each interaction pairs three human participants with an autonomous LLM facilitator in an asynchronous, text-based environment without prescribed turn-taking order.The facilitator can proactively intervene as the dialogue unfolds.Source: §2.1 · pp. 3–4
2.2. Scenario Configurations
- The scenarios are designed to test adaptability across distinct collaborative objectives, constraint structures, and success criteria.Prior automated-facilitation evaluations are described as predominantly single-scenario benchmarks.Source: §2.2 · p. 4
- Open-ended discussion has no agreement requirement; message volume serves as a proxy for high-quality engagement.Participants discuss policy and may receive a bonus for high-quality engagement.Source: §2.2 · p. 4
- In multi-party negotiation, participants form a team and divide a real-money pool under unequal bargaining power; agreement determines whether the pool is awarded.Source: §2.2 · p. 4
- In consensus-building, participants must unanimously choose a charity for a donation to be made.Source: §2.2 · p. 4
2.3. Evaluation Metrics for Conversation Tasks
- Sessions are evaluated across participant experience, facilitator evaluation, and task outcomes, using post-session 5-point Likert ratings and free-text rationales from all participants.The three dimensions distinguish how participants experienced the interaction, perceived the AI, and whether task goals were met.Source: §2.3 · p. 4
- Participant measures include safety, feeling heard, comfort, friction or anger, and productivity; facilitator measures include helpfulness, satisfaction, and interaction comfort.Source: §2.3 · p. 4
- Task outcomes are unanimous agreement for consensus-building, valid team allocation for negotiation, and participant message volume for open-ended discussion.Source: §2.3 · p. 4
3. Learning a Generalized Facilitation Prompt
GUIDE learns facilitation without a deterministic reward by reflecting on real human interactions and iteratively refining a structured playbook. A judge compares successive playbook versions, and validation feedback informs later revisions before held-out expert comparisons.
3. Learning a Generalized Facilitation Prompt
- Because facilitation has no deterministic reward, GUIDE grounds self-improvement in real human interactions and an LLM judge aligned with human preferences.The process reflects on interaction evidence and iteratively refines facilitation lessons.Source: §3 · p. 4
3.1. Stage 1: Corpus Collection
- The learning corpus contains 432 episodes from 144 teams, each completing all three tasks, with teams split into training, validation, and test sets.Expert prompts were manually designed and refined through participant pilot studies.Source: §3.1 · p. 5
3.2. Self-Learning Framework
- The frozen language model uses a structured natural-language playbook initialized empty, with a minimal role instruction and a decision about whether and what to say.Source: §3.2 · p. 5
- At each participant turn, the policy produces a rationale, a speak-or-silent decision, an utterance, and the playbook rules cited to support its action.The cited rules enable validation feedback to be attributed to individual rules.Source: §3.2 · p. 5
- A conversation reflector extracts transferable facilitation lessons from transcripts, participant feedback, and task outcomes without observing expert prompts or their rationales.Source: §3.2 · p. 5
- A curator makes local playbook edits by adding, editing, merging, or removing rules, combining corpus lessons with validation-reflector lessons from later generations.Source: §3.2 · p. 6
- A human-calibrated LLM judge blindly compares candidate and preceding-playbook actions at the same validation state, measuring improvement over the preceding version rather than expert similarity.The rubric reached Cohen’s κ=0.70 agreement with the human panel.Source: §3.2 · p. 6
- Validation reflection turns recurring losses and judge rationales into lessons, while rule-level credits preserve helpful rules and prune harmful or unused ones; the final playbook is tested against experts only afterward.The loop alternates static human-experience evidence with dynamic policy-validation evidence.Source: §3.2 · p. 6
4. Evaluation and Results
The evaluation tests whether a learned generalized facilitator can outperform task-specific prompts offline and preserve group outcomes and participant experience in live deployment. It also examines which learned rules shape facilitation and how participants describe their experiences.
4.1. Experimental Setup
- The setup varies reflector and curator backbones, temperature, batch size, and validation-reflector use, while holding the judge and speech generator fixed.The held-out evaluation uses 87 episodes from 29 unseen cohorts and 736 balanced speech/silence decision points; live evaluation uses 154 new cohorts.Source: §4.1 · p. 7
4.2. Self-learning generalized facilitator vs. expert facilitator
- The learned playbook refines existing strategies into state-conditioned rules and adds failure-mode handling, while sometimes losing useful unconditional safeguards.Tempo-gated restraint, decisive closure, and non-directive intervention form are key dimensions; some rapid consensus may go unprobed.Source: §4.2 · pp. 7–8, Figure 3 · p. 9
- 61.4% overall held-out win rate over expert baselines, with 66.9% on turns where the baseline chose to speak.The lead configuration used gemini-3-flash, temperature 0.8, batch size 10, and both reflectors.Source: §4.2 · p. 7, Table 2 · p. 8
4.3. Live Deployment: Comparable Group Experience and Facilitator Evaluation
- The live comparison finds matching group productivity and objective task outcomes, improved speaking comfort, lower friction, and 24.7% fewer turns and 29.0% fewer words.Psychological safety is also higher (4.67 vs. 4.62), and within-group dispersion is lower on all five participant-experience measures. The comparison is between studies, not a randomized causal estimate.Source: §4.3 · p. 8, Table 3 · p. 10
4.4. Qualitative analysis of participant experience improvements
- Participants described expert prompts as pushy and socially uncalibrated, while describing the learned facilitator as patient, neutral, and more willing to leave space for discussion.Participants also connected conditional intervention with addressing exclusion, confusion, or stalled discussion rather than intervening routinely.Source: §4.4 · p. 9
5. Related Work
The related work describes a field shaped by task-specific facilitation and contested goals, with concerns about how simulations represent human group dynamics. It positions this paper’s approach as learning from logged human deliberation and validating with live users.
5.1. AI assisting group conversation
- Facilitation balances competing orientations and goals, motivating multidimensional evaluation and an expectation of low agreement about specific interventions.Prior work also reports a perception–measurement gap in which facilitators may be preferred without improving consensus or participation equity (Alsobay et al., 2025).Source: §5.1 · p. 10
- Many LLM facilitation efforts use hand-crafted prompts for single scenarios, while simulated interlocutors may diverge from human group dynamics (Chuang et al., 2024).The discussion also notes limitations in predicting social reactions, replicating economic behavior, and detecting conflict (Bojic et al., 2026; Kitadai et al., 2025; Kwon et al., 2025, 2026).Source: §5.1 · p. 10
5.2. AI Facilitation Improvement
- The paper grounds self-improvement in real three-person deliberations and post-episode human judgments, using an LLM judge calibrated against human annotators (κ: 0.70).Its stated distinctions are one policy across tasks, real rather than simulated interactions, and offline plus live validation.Source: §5.2 · p. 11
6. Conclusion
The conclusion presents a frozen LLM that learns a generalized facilitation policy from real human interactions without expert prompts or deterministic reward. The policy outperforms domain-specific prompts offline and preserves task outcomes while improving participant experience in live use.
6. Conclusion
- The frozen LLM learns from real human interactions without access to the expert prompts or a deterministic reward signal.The playbook contains 53 rules induced from an empty initialization.Source: §6 · p. 11
- 61.4% win rate on held-out decisions over three domain-specific expert prompts, without collapsing onto a single task type.The conclusion describes the playbook as a structured natural-language policy.Source: §6 · p. 11
- For new participants, the learned policy preserves all task outcomes, improves speaking comfort, reduces interpersonal friction, and uses 25% fewer turns and 29% fewer words.Source: §6 · p. 11
7. Limitation
The stated limitation is that offline optimization scores decisions independently and does not model how sustained silence may shape participants’ experience of the facilitator.
7. Limitation
- Offline optimization rewards restraint turn by turn but does not model the cumulative effect of sustained silence on facilitator experience.Source: §7 · p. 11
- The authors recommend reporting induced action-rate distributions alongside pairwise win rates in offline evaluations of interactive agents.Source: §7 · p. 11
- The reflector-curator-validator loop is presented as a general recipe for inducing structured policies where success depends on human perception and no programmatic reward exists.Source: §7 · p. 11
8. Ethics Statement
The ethics statement describes protections and bounds the learned policy as a research prototype, while the appendix details study setup, recruitment, compensation, and participant flow.
A.1. Recruitment, Consent, and Compensation
- Participants were recruited through Prolific in cohorts of three, gave consent, and were told an AI facilitator would join and would not be presented as human.A debriefing disclosed the research purpose.Source: §8 · p. 11, §A.1 · pp. 15–16
8. Ethics Statement
- The policy is auditable, but the authors note possible misuse and report testing only in English-language Prolific settings, not high-stakes settings.The study used short, text-only discussions under pseudonymous animal aliases.Source: §8 · pp. 11–12
A. Study Design, Materials, and Participants
- The appendix covers recruitment, participant flow, task materials, surveys, sample accounting, and participant backgrounds; both studies shared stage configurations and differed only in facilitator prompt.Both studies ran on Deliberate Lab, an open-source platform for synchronous multi-party experiments.Source: A. Study Design, Materials, and… · pp. 14–15
A.1. Recruitment, Consent, and Compensation
- Participants received $15 for an approximately one-hour session, with additional announced incentives tied to discussion engagement, negotiation outcomes, and charity selection.The discussion bonus was $2; negotiation bonuses could reach $10 per triad, and the experimenters donated $3 for the consensus task.Source: §A.1 · p. 15
A.2. Participant Flow
- Each participant completed 38 stages, including three counterbalanced task blocks, chat, and post-discussion surveys; task outcomes were revealed before those surveys.The task blocks also included introductions and comprehension checks, with private final decisions for negotiation and consensus tasks.Source: §A.2 · p. 16
- The facilitator drafted decisions asynchronously as participants typed and discarded any draft made stale by new messages; only the prompt block differed between studies.It received the task and stage descriptions, participant roster, and live transcript, but no persona, demographic, or survey information.Source: §A.2 · p. 16
- The six task orders were counterbalanced across separate recruitment conditions, and both studies fielded all six in near-equal proportions.Task identity was therefore orthogonal to serial position.Source: §A.2 · pp. 16–17
A.3. Task Materials
The task materials used synchronous text discussions with three participants, and the open-ended discussion tested policy debate without requiring agreement.
A.3. Task Materials
- All three tasks used synchronous text channels with three participants and a 5–10-minute discussion window.Participants could advance after five minutes; the task materials were identical across Study 1 and Study 2.Source: §A.3 · p. 17
A.3.1. Task 1: Open-Ended Discussion
- The open-ended task asked groups to discuss a policy motion without requiring agreement, using one of two motions about facial recognition or Medicaid work requirements.The motions were drawn per cohort from a pool of two.Source: §A.3.1 · p. 17
- Because both motions were US-centric and politically charged, the open-ended results are evidence about contested-values discussion rather than broad topical generalization.The two motions were allocated across cohorts in both studies.Source: §A.3.1 · p. 17
A.3.2. Task 2: Consensus-Building (Charity Allocation)
The charity-allocation task required unanimous agreement on a donation recipient, with a random-draw fallback for stable disagreement.
A.3.2. Task 2: Consensus-Building (Charity Allocation)
- Groups chose one charity for a $3 donation; stable disagreement triggered a uniform random draw, and participants did not see charity ratings.Three charities were selected from nine per cohort, with randomized slot positions; the realised sample covered 83 of 84 possible unordered charity sets.Source: §A.3.2 · p. 17
A.3.3. Task 3: Multi-Party Negotiation
This appendix specifies the negotiation task and study materials, documents how the two studies were configured and sampled, and details the corpus, judge, and playbook-refinement process.
A.3.3. Task 3: Multi-Party Negotiation
- The negotiation task assigns roles with structurally unequal bargaining power, and deal validity requires coalition members to privately record matching integer shares.The full payoff table was shown to all participants; if coalition members’ records disagree, nobody is paid.Source: §A.3.3 · pp. 17, 19
A.3.4. Protocol Equivalence Between Studies
- Across all six task orders, Study 1 and Study 2 configurations matched; order was balanced across studies but confounded with recruitment batch.Only three stages varied across counterbalancing conditions, not between studies.Source: §A.3.4 · pp. 19–20
A.6. Participant Background
- Uncollected age, gender, race or ethnicity, education, income, and country data limit subgroup analysis and claims of representativeness beyond the Prolific pool.Participant background otherwise includes only an occupational item and three free-text self-disclosure prompts.Source: §A.6 · p. 20
- Study 2 participants began slightly less enthusiastic on all three evaluative AI-facilitation items, while higher retention in Study 2 does not explain the raw facilitator-rating deficit.The largest baseline gap was 0.21 points on perceived AI-facilitator productivity; analyses calibrate post-task ratings against each participant’s pre-study response.Source: §A.6 · p. 20, §A.7 · p. 20
B.1. Corpus Construction and Splits
- The 432-episode corpus was split by cohort and stratified by task, keeping each cohort’s three episodes together and reserving 87 test episodes for final evaluation.Episode records include the message sequence, task context, participant surveys and rationales, and evaluation outcomes; expert recorded reasoning is excluded from learning.Source: §B.1 · p. 21
B.2. Playbook Representation and Decision Points
- The playbook uses stable rule identifiers, enabling rule-level credit and blame; its benchmark samples equal numbers of expert speech and silence decisions.This 1:1 balance is not natural-conversation prevalence, and the mutual-silence fast path makes abstention low-risk.Source: §B.2 · p. 22
B.3. Judge Design and Human Alignment
- The aligned judge reaches Cohen’s κ=0.70 against panel majority, but is framed as a reliable proxy for the median annotator, not an oracle.The rubric diagnoses conversational state before scoring candidate actions and applies binding content caps.Source: §B.3 · pp. 22–23
B.4. Reflectors, Curator, and Credit Attribution
- Two reflectors supply participant-grounded and validation-based lessons, which a curator applies through localized rule edits, merges, additions, or deletions.Rule-specific helpful, harmful, or neutral attribution avoids assigning every cited rule the whole action’s verdict; pruning uses these rule counters and removes persistently unused rules.Source: §B.4 · pp. 23–24
B.5. Playbook Provenance and Skill-Coverage Analysis
The appendix operationalizes how learned rules relate to expert skills and where the rules came from, then specifies the held-out protocol used to compare optimization arms.
B.5. Playbook Provenance and Skill-Coverage Analysis
- The three expert prompts were decomposed into 97 atomic instructions, with identical instructions shared across tasks merged.These 97 skills form the denominator for the expert-skills-represented row in Table 2.Source: §B.5 · p. 24
- 47/97 (48.5%) expert skills are at least partially represented in the playbook, while 10/97 are represented verbatim.Across 291 scoring calls, 81.4% of skills received the same score on all three repeats.Source: §B.5 · p. 25
- Rules are retained when a skill cites them at score 1.0, re-conditioned when cited only at 0.5, and novel when no skill cites them.The playbook has 11 retained, 19 re-conditioned, and 23 novel rules; skill coverage and rule novelty use different denominators.Source: §B.5 · p. 25
- The most influential rules are grouped as tempo-gated restraint, decisive closure, and non-directive intervention form.Each dimension contrasts unconditional expert instructions with state-conditioned learned rules, and the figure reports per-rule held-out win rates on decided turns.Source: §B.5 · p. 25
- Of the 53 final rules, 25 derive only from conversation evidence, 14 only from validation evidence, and 14 from both.By original Add alone, the split is 36 conversation-derived rules versus 17 validation-derived rules.Source: §B.5 · p. 25
C.1. Protocol
- The ablation varies reflector and curator model, training batch size, sampling temperature, and learning signal through greedy sequential coordinate descent.The held-out test set is used to score each arm, with later stages retaining an incumbent unless a challenger is significantly better.Source: §C.1 · p. 26
- Each arm is evaluated on the same held-out test set against the expert baseline, reporting tie-adjusted win rate, mean quality and timing deltas, incumbent gain, and bootstrap uncertainty.The 95% percentile confidence interval comes from a paired bootstrap over episodes (5000 resamples), alongside a one-sided bootstrap p-value.Source: §C.1 · p. 26
C.2. Findings
The findings compare optimization choices and identify the playbook deployed in Study 2; the appendix then outlines additional task-outcome, rating, and robustness analyses.
C.2. Findings
- 0.614 was the learned playbook’s held-out score, significantly above gemini-3.1-pro-preview (0.585) and gemini-3.7-flash (0.558).The pro arm had the highest quality delta (+0.324) but a lower decision-level win rate.Source: §C.2 · p. 26
- A reflection batch of 10 episodes is the top hyperparameter; batches of 5 provide too little evidence, while batches of 20 dilute lessons.The 20-episode arm also ran fewer generations before early stopping.Source: §C.2 · p. 26
- A reflector/curator temperature of 0.8 yields 4.6 points in facilitation improvement over a temperature of 0.2.The authors attribute the advantage to exploration, since low-temperature curation converges early on a small set of safe, generic rules.Source: §C.2 · p. 26
- Enabling both reflectors beats the judge-only arm by 0.6 points, a non-significant difference (p=0.68).The authors therefore cannot claim that human-corpus evidence adds measurably to judge feedback on the offline metric, which is defined by that judge.Source: §C.2 · p. 27
C.3. Configuration Deployed in Study 2
- The Study 2 playbook uses gemini-3-flash-preview, batch size 10, temperature 0.8, both signals, and 30 generations.It contains 53 rules and 2,940 tokens and achieved W_test=0.614; the temperature and signal settings were carried forward as untested incumbents.Source: §C.3 · p. 27
D. Additional Detailed Results
- The appendix expands the per-task objective outcomes, raw facilitator ratings with dispersion and calibration analyses, and robustness analysis.The main text’s Table 3 reports the headline outcomes.Source: D. Additional Detailed Results · p. 27
D.1. Task Outcomes, Task by Task
Task-level outcomes are broadly similar for consensus and deal formation, but negotiation accounting and open-ended participation show separate differences.
Consensus building
- Unanimous consensus was reached in 88.2% of expert-prompt episodes and 83.8% of learned-playbook episodes, a non-significant 4.4-point gap (p=0.27).The learned facilitator used 28% fewer interventions per episode (4.71 → 3.40) and still brought five in six groups to unanimity.Source: §D.1 · p. 27
Negotiation
- Coalition agreement was 90.3% versus 89.6% (p=0.85), while trust in co-negotiators was 4.39 versus 4.50 (p=0.14).The claimed-share validity measure was worse under the learned facilitator, with the difference not distinguishable from zero.Source: §D.1 · p. 27
- −7.8 pp: valid negotiation allocations fell from 84.6% to 76.8% under the learned facilitator, a non-significant difference (p=0.11).This comparison concerns episodes in which participants agreed on a coalition.Source: §D.1 · p. 27
- 2.3% to 8.7%: clerical failures to sum to the pot increased by 6.4 pp (p=0.023) under the learned facilitator.The authors call this suggestive, not established: the nominal p-value on small counts does not survive Benjamini–Hochberg correction.Source: §D.1 · p. 28
Open-ended discussion
- Open-ended qualification rates were 22.4% under expert prompts versus 20.1% under the learned playbook (−2.3 pp, p=0.49).Message volume and participant word counts also moved slightly lower, with non-significant differences.Source: §D.1 · p. 28
D.2. Raw Facilitator Ratings, Within-Group Variance, and Prior Calibration
After calibration for prior attitudes, facilitator ratings show no significant deficit on most measures, with productivity remaining an exception; participant-experience dispersion is lower within groups.
Raw facilitator ratings and prior calibration
- Raw facilitator ratings were lower on all three items in Study 2, but calibration removes the satisfaction and comfort deficits and most of the productivity deficit.Within-subject calibrated shifts show no significant deficit on any facilitator metric; the one-row-per-participant check yields the same pattern.Source: §D.2 · pp. 28–29
Within-group variance
- The learned-playbook study has lower within-episode dispersion for psychological safety (0.466 → 0.420) and interpersonal friction (0.519 → 0.405, p=0.007).Across all five human self-experience scales, within-episode SD and within-cohort variance are lower in Study 2; the authors describe this as more uniform group experiences.Source: §D.2 · p. 28
Prior calibration
- Study 2 participants entered with greater baseline skepticism: 35.7% were non-enthusiasts on expected productivity, versus 25.1% in Study 1.The study samples had statistically indistinguishable prior AI usage, so the imbalance was in attitude rather than exposure.Source: §D.2 · p. 29
- After controlling for pre-study attitudes, Study 2 has no significant deficit in satisfaction or comfort, but a productivity deficit remains.In the cohort-clustered ANCOVA, the Study 2 productivity coefficient is −0.205 (p=0.003), down from a raw gap of −0.301; baseline skepticism accounts for most, but not all, of that deficit.Source: §D.2 · p. 29
Expectation-disconfirmation analysis
- Disappointed Pro-AI evaluations in Study 2 occurred when the learned playbook under-intervened, while the Study 1 counterpart involved expert over-intervention and friction.The equivalent Study 1 contrast for disappointed versus confirmed Pro-AI evaluations had the opposite sign and was not significant.Source: §D.2 · p. 29
D.3. Robustness and Threats to Validity
This section documents the prompts, annotation procedures, and reflection workflow used to derive and refine the playbook, linking facilitator judgments to transcript evidence and human feedback; it also illustrates offline assessment with a replayed negotiation episode. It reports robustness and validity considerations, including missing-response assumptions and the lack of concurrent randomization.
D.3. Robustness and Threats to Validity
- 4.15 vs. 3.99: the perceived facilitator productivity deficit persists when every missing response in both studies is imputed at the scale maximum.Reversal would require Study 2 missing responses to average 5.65 on a five-point scale; satisfaction and comfort tipping values are also off-scale at 5.31 and 5.09. The comparison is between studies, not a concurrently randomized trial, so sample and period were not held fixed; comparable retention and negligible item non-response make selective dropout an unlikely explanation.Source: §D.3 · pp. 29–30
E.1. Facilitator Prompt Scaffold
- The Study 1 prompt scaffold was verified against all 432 logged episodes, while learned-playbook rules and task-specific stage headers varied across the learning loop and studies.The scaffold requires facilitators to cite applicable rule IDs for speech or silence and instructs them to wait for several participant turns, speak rarely, and intervene when useful after more than six turns.Source: §E.1 · pp. 30–31
E.2. The Deployed Playbook P𝑇
- The deployed Study 2 playbook contains 53 rules across eight cue-schema sections, totaling 2,940 tokens, with stable identifiers across the run.Its role guidance calls for stress-testing deliberation while avoiding probes that could instigate conflict when a group converges rapidly and enthusiastically.Source: §E.2 · p. 32
- The deployed playbook prioritizes psychological safety, validates personal disclosures, and protects participation by inviting quieter members and preserving peer response cycles.It also calls for checking agreements against environmental constraints and preventing members from representing another participant’s identity or consent.Source: §E.2 · pp. 34–35
E.3. Judge Rubric (Aligned)
- Cohen’s κ=0.70 against the human panel was reached by the aligned judge rubric.The rubric compares facilitator reactions at the same moment and judges what they do in the conversation, not the rationale they provide.Source: §E.3 · pp. 35–36
- The judge first classifies the exchange from the transcript and clock, then scores whether an intervention suits that state and moment.It distinguishes productive, shallow, confused, stalled, unproductive, and settled exchanges; for example, productive discussion should be allowed to continue.Source: §E.3 · p. 36
E.4. Conversation Reflector
- The conversation reflector receives batches of 10 training episodes with task context, chronological transcripts, objective outcomes, and participants’ post-task ratings and free-text responses.The extracted experiences are passed to the curator as Lconv, tagged by source.Source: §E.4 · p. 38
- The conversation reflector diagnoses effort imbalance, deliberation pacing, emotional safety, and shallow consensus, then links human feedback to transcript evidence and concrete counterfactual actions.It requires lessons to generalize across unseen tasks by connecting observable room signals to facilitator moves, without relying on task mechanics or episode specifics.Source: §E.4 · p. 39
E.5. Validation Reflector
- The validation reflector runs once per generation over judged validation turns in chunks of 10, requiring recurrence support of at least 2 and passing lessons to the curator in the next generation.Its prompt mirrors the conversation reflector’s headings and output schema, so the both-versus-judge-only ablation varies the learning signal rather than prompt quality.Source: §E.5 · p. 42
- The validation audit classifies playbook behavior as misfire, overreach, conflict, gap, or none, with repairs that condition, constrain, prioritize, add, or preserve rules.It reviews every judge verdict, including wins, and requires repairs to name the cited rule and generalize beyond task mechanics.Source: §E.5 · p. 43
- The Validation Reflector connects judged praise or criticism to transcript turns, explains the social causes, and identifies counterfactual facilitator actions.It generalizes lessons around observable room signals and requires a pattern to recur across at least <MIN_SUPPORT> independent turns before reporting it.Source: §E.5 · pp. 43–44
E.6. Curator
- The Curator reviews transcripts, outcomes, and participant evaluations, then applies structured ADD, EDIT, DELETE, or MERGE operations to the playbook.Rules must remain domain-agnostic, map observable room signals to facilitator moves with guardrails, and exclude episode-specific details; the instructions prefer sharpening an existing rule over adding one tied to an episode.Source: §E.6 · pp. 46–47
- The Curator distinguishes human-study evidence from Judge-assessed failures and uses reflection evidence to add, edit, delete, or merge playbook rules.The two reflection sources are labeled SOURCE A and SOURCE B; proposed changes include a rationale and reference to the relevant reflection item.Source: §E.6 · pp. 48–49
E.7. Tag Attributor (Per-Rule Credit)
- The Tag Attributor assigns helpful, harmful, or neutral credit to each cited rule based on whether it produced the judged outcome, and can flag a missed rule or playbook gap.A lost turn alone does not make a cited rule harmful; the rule must have driven the fault.Source: §E.7 · p. 49
E.8. Skill-Coverage Annotator
- The Skill-Coverage Annotator scores whether the complete rulebook prescribes an expert instruction as present (1.0), partial (0.5), or absent (0.0), judging actions rather than shared vocabulary.Partial coverage includes instructions that are narrower, weaker, overgeneralized, or only implied; scores above zero must cite a rule.Source: §E.8 · p. 50
F. Example Conversation Replay
- The replay follows a 12-message negotiation episode with six decision points, testing the learned playbook against expert prefixes while the transcript continues with the expert’s actual interventions.It was selected because its decisions exhibit every benchmark verdict type; the learned playbook’s responses are judged at each point, making the replay off-policy.Source: F. Example Conversation Replay · p. 51