PaperlayerView exact revision on arXiv

Source-linked AI summary

Consistent but Biased: Chair Personas Trade Noise for Bias in Multi-Agent Clinical Decisions

Yitong Bai, Chengyi Zhang, Ziyang Wang

arXiv:2610.31325v1cs.MA

TL;DR

Clinical multi-agent systems often give one agent final authority, but the effects of that chair’s persona on fixed-evidence decisions and repeated-run agreement had not been tested. In a seven-agent department discussing 20 unchanged case records, chair personas shifted operating-now decisions, while trait-only prompts reproduced some effects and higher raw agreement did not exceed chance-corrected agreement.

  • Problem

    The effects of a ruling agent’s persona on decisions made from fixed clinical evidence, and on the meaning of repeated-run agreement, had not been tested.

    Source: §1 · p. 2

  • Method

    A seven-role simulated department discussed 20 published cases with case statements replayed unchanged while chair and deputy prompts varied across neutral, consultative, autocratic, and avoidant personas in rules and trait-only forms.

    Source: §1 · p. 2, §3.2 · p. 4, §3.3 · p. 4

  • Results

    2% (avoidant) to 93% (autocratic) of rules-version runs chose to operate now, versus 57% without a persona, while surgery remained intended in 88–100% of rulings.

    Source: §1 · p. 2, §6 · p. 14

  • Takeaways & Limitations

    Clinical multi-agent studies should report and vary the chair’s persona and correct run agreement for chance before treating it as reliability.

    Source: §6 · p. 15

    The evidence comprised 20 published summaries that may have appeared in model training data, were thinner than real records, and lacked films; process and red-flag measures relied on structured outputs and an LLM judge rather than clinician validation.

    Source: §5 · p. 14

Abstract

from arXiv · show

In clinical systems built from several large language model (LLM) agents, one agent often chairs the discussion and makes the final decision. Whether the chair's persona prompt changes that decision, or the agreement between repeated runs that some systems treat as reliability, has not been tested with the case held constant. We ask whether the persona changes the decision or only its timing, how it affects agreement between runs, and which behaviors appear when the prompt describes only personality traits. A simulated department of seven agents discussed 20 published cases of degenerative cervical myelopathy, with the case reports replayed unchanged in every run. Only the prompts of the chair and its deputy changed (no persona, or a consultative, autocratic or avoidant one), written either as rules of conduct or as personality traits. With personas written as rules, the chair chose to operate now in 93% of runs when autocratic and in 2% when avoidant, against 57% without a persona. Surgery was still planned in nearly all decisions, so the personas mainly changed whether to operate now or after more tests. These effects held with a second model and reworded prompts. The autocratic and avoidant personas made the three runs of a case agree more often, but only about as often as chance predicts, and Fleiss $\kappa$ did not rise. Prompts that described only personality traits kept more than half of the autocratic and avoidant effects on operating now. They reproduced quick decisions, delays and extra tests, but not ignoring red flags, dropping backup plans or passing the decision to the family. Studies should report and vary the chair's persona as they do the model, and correct agreement between runs for chance before calling it reliability.

1 Introduction

The paper tests whether a chair’s persona changes clinical decisions and run-to-run agreement when the case evidence is held fixed. It finds that personas strongly shift when surgery is chosen, while trait-only prompts reproduce some of those effects.

1 Introduction

  • The study holds case evidence fixed while varying chair and deputy personas, asking whether personas change decisions, timing, run agreement, and trait-linked behaviors.The simulated department discussed 20 published cases; the published management was not treated as a reference standard.Source: §1 · p. 2
  • 2% (avoidant) to 93% (autocratic) of rules-version runs chose to operate now, versus 57% without a persona; surgery remained intended in 88–100% of rulings.The authors report that persona effects mainly changed when to operate, rather than whether surgery was planned.Source: §1 · p. 2
  • Autocratic and avoidant personas raised full agreement across three runs from 50% to 85–90% of cases, near the 81–86% expected by chance, without raising Fleiss κ.The reported Fleiss κ values were 0.31 without a persona, 0.20 for autocratic, and 0.31 for avoidant.Source: §1 · p. 2
  • Trait-only prompts retained 57% of the autocratic and 76% of the avoidant effects on operating now, reproducing quick commitment, deferral, and extra testing but not several other predicted behaviors.The unreproduced behaviors included ignoring red flags, dropping backup plans, and passing the decision to the family.Source: §1 · p. 3

2 Related work

Prior work examines leadership, personas, and repeated-run consistency in LLM groups, but has not tested the ruling agent’s persona against fixed clinical evidence. The paper situates this question in research on leadership effects, personality prompts, and decision psychology.

2.1 LLM multi-agent systems for clinical decisions

  • LLM moderators and leaders have been shown to affect group conclusions, while repeated-run agreement has been used as a reliability measure in clinical systems.Prior work also reports effects of authority, team structure, dissent, and interaction style in LLM groups.Source: §2.1 · p. 3

2.2 Personas and personality in LLMs

  • Prompted personalities register on questionnaires, but trait self-reports and behavior can dissociate, motivating component-by-component tests of persona behavior.Prior studies also link personas with reasoning and risky choices, while narrative framing can explain more behavioral variance than the assigned persona.Source: §2.2 · p. 3

2.3 Leadership and decision psychology

  • Leadership research predicts that directive leaders narrow alternatives, decision conflict promotes deferral, and surgical judgment contains noise.Leadership style and hierarchy are also associated with how much team members speak up.Source: §2.3 · p. 3

3 Methods

The methods use a seven-role simulated spine department and replayed case evidence to isolate chair and deputy persona prompts. They define persona conditions, outcomes, and measures for decision behavior and run-to-run agreement.

3.1 Cases and pre-decision records

  • The study selected 20 published degenerative cervical myelopathy cases from a screened set, grouping them as borderline, difficult-diagnosis, or clear-indication cases.The cases came from public patient summaries; authors assigned case groups and management from the full texts.Source: §3.1 · p. 4
  • The case summaries were redacted before the reported management, retained a median 63% of original characters, and contained no imaging films.The radiologist relied on record descriptions and stated that it had not seen the films.Source: §3.1 · p. 4

3.2 The simulated department

  • Seven fixed-order roles report the case, assess risks, negotiate, and culminate in a chair ruling among operating now, workup, conservative care, postponement, or no decision.The deputy speaks before the chair; if the chair has not decided, negotiation and ruling can run for a second round.Source: §3.2 · p. 4

3.3 Persona design

  • A persona is a role-conduct prefix, and an arm pairs chair and deputy prefixes while other roles’ prompts remain identical.Rules prompts prescribe conduct; trait-only prompts describe dispositions and motives, with the latter used as an ablation.Source: §3.3 · pp. 4–5
  • The persona designs predict distinct behaviors: autocratic closure and reduced search, avoidant deferral and risk vigilance, and consultative discussion, voice, and integration of input.The rules version encoded these behaviors directly, including requests for colleague input, brisk rejection of objections, or seeking further evidence and shifting responsibility.Source: §3.3 · pp. 5–6

3.4 Holding the evidence fixed

  • The same persona prefixes and deputy prompts were paired with frozen Stage I–II statements, generated once and replayed verbatim across arms and runs.The authors confirmed this reuse for all 360 runs; repeated runs estimate the model’s own randomness.Source: §3.4 · p. 7
  • The rules arms ran three times per case and trait-only arms twice; 10-case replications tested a second decision model and reworded rules prompts.Each replication ran once per case-arm cell, with the frozen statements and other roles retained.Source: §3.4 · p. 7

3.5 Measures

  • The primary endpoint was the share of runs choosing operate now; secondary endpoints covered surgery intent, workup, family involvement, objections, and red-flag handling.Process measures included tests, negotiation rounds, colleagues addressed, contingencies, and deputy objections before the ruling.Source: §3.5 · p. 7

4 Results

With case evidence held fixed, chair personas shifted whether the department operated immediately, while trait-only prompts reproduced some—but not all—persona behaviors. Autocratic and avoidant chairs raised raw agreement between runs, but not chance-corrected agreement.

4.1 Autocratic and avoidant personas push the decision to opposite extremes (RQ1)

  • 93% of autocratic rules runs and 2% of avoidant runs ruled to operate now, versus 57% with no persona; consultative rulings were 52%.The direction held across sensitivity analyses, and the personas moved every case with room to move.Source: §4.1 · p. 8, Figure 3 · p. 9

4.2 Personas change mainly the timing of surgery (RQ2)

  • Surgery was intended in 100% of neutral, consultative and autocratic rulings and 88% of avoidant rulings, so personas mainly shifted timing rather than the plan to operate.The avoidant chair ruled to work up first in 95% of runs and ordered more tests; the autocratic chair did so in 7% and ordered 0.7 tests per run versus 2.8 under neutral.Source: §4.2 · pp. 8–9, Table 2 · p. 9

4.3 Autocratic and avoidant personas raise raw but not chance-corrected agreement between runs (RQ3)

  • Three runs agreed in 50% of neutral cases, 85% of autocratic cases and 90% of avoidant cases, but the rise was close to chance expectations and Fleiss’ κ did not increase.Fleiss’ κ was 0.31 for neutral and avoidant arms and 0.20 for consultative and autocratic arms; confidence intervals for differences from neutral included zero.Source: Table 2 · p. 9, §4.3 · p. 10

4.4 Trait descriptions reproduce the timing of commitment and the amount of checking (RQ4)

  • Trait-only prompts retained 57% of the autocratic effect and 76% of the avoidant effect on operating now.The trait-only differences from neutral were +21 percentage points for autocratic and −42 for avoidant.Source: §4.4 · p. 11, Figure 4 · p. 12
  • Traits reproduced quicker commitment, deferral, extra testing, risk vigilance and slower closure, but not leaving red flags unaddressed, dropping fallback plans or passing decisions to family.The trait-only avoidant chair ordered 4.2 tests per run versus 2.8 under neutral; the consultative chair took 1.45 rounds on average versus 1.85 under its rules version.Source: §4.4 · pp. 11–12

4.5 Decision-bearing red flags were mostly handled in every arm

  • Every arm acted on most decision-bearing red flags, with no detectable prespecified differences, although the rules-version autocratic chair left more flags unaddressed across all flags.The decision-bearing flags were handed to the chair separately at the ruling, so evaluating evidence handling should also include flags present only in the transcript.Source: §4.5 · p. 13

4.6 Where the effects concentrate (exploratory)

  • The autocratic effect on operating now was largest in the borderline case group: 56% to 100%, a 44-point increase; this exploratory pattern held only on the autocratic side.Testing the pattern requires more borderline cases grouped by clinicians.Source: Table 4 · p. 13, §4.6 · p. 13

4.7 Swapping the decision model and the wording

  • Autocratic and avoidant effects kept their direction after changing the decision model and prompt wording; across the two models tested, persona moved decisions more than model choice.On the 10-case subset, effects were +37, −57 and 0 points in the original setting, +30, −50 and −10 with the second model, and +50, −50 and −20 with rewording.Source: §4.7 · pp. 13–14, Table 5 · p. 14

5 Limitations

The design leaves unresolved why trait-only prompts failed to reproduce freezing and buck-passing, and it could not test whether the chair silences subordinates. The model comparison and case and process measurements also constrain the findings.

5 Limitations

  • The design could not explain why trait descriptions reproduced neither freezing nor buck-passing, and fixed turn order left chair silencing of subordinates untested.The authors suggest alignment training as a possible explanation and propose testing less aligned models and a condition where the chair intervenes during negotiation.Source: §5 · p. 14
  • The model comparison used 10 cases, one run per cell and two models, so the finding that persona mattered more than model choice applies only to those models.Because both chair and deputy prefixes changed together, varying only the chair would be needed to estimate its separate contribution.Source: §5 · p. 14
  • The study used 20 published summaries that may be thinner than real records and lacked films; process measures came from chair outputs, and red flags were judged by an LLM.The authors, rather than clinicians, grouped cases, further limiting analysis by case group.Source: §5 · p. 14

6 Conclusion

The conclusion summarizes how chair personas affected surgical timing and chance-corrected agreement, and how trait-only prompts retained some timing effects while omitting behaviors associated with rule-based prompts. It also documents trait-only and reworded autocratic prompt examples for the chair and deputy.

6 Conclusion

  • The study held evidence for 20 surgical cases fixed while varying chair and deputy personas; in a 10-case subset, persona shifted decisions more than the choice between the two models.These findings concern a simulated department and the tested cases and models.Source: §6 · p. 14
  • 2% (avoidant) to 93% (autocratic), versus 57% without a persona, chose operate now; surgery remained intended in 88–100% of rulings, so personas mainly shifted timing.The avoidant chair also judged surgery indicated less often.Source: §6 · p. 14
  • Full agreement rose from 50% to 85–90% of cases, near the 81–86% expected by chance, but chance-corrected agreement did not rise.The higher raw agreement therefore did not indicate improved reliability across runs.Source: §6 · p. 14
  • Trait-only prompts retained 57% of autocratic and 76% of avoidant effects on operating now, reproducing quick commitment, deferral, and extra testing but not several rule-prompt behaviors.They did not reproduce ignoring red flags, dropping fallback plans, or handing the decision to the family.Source: §6 · p. 14
  • The study recommends reporting and varying chair persona and chance-correcting run agreement, with blinded clinical ratings and human-department experiments as next steps.It predicts that leadership style will affect test ordering and time to surgery more than whether surgery is planned, and that avoidant chairs will defer to tests more than choose conservative care; meeting records, test orders, and instruments such as ATLAS could test this.Source: §6 · p. 15

A.3 Trait-only version

  • In the trait-only version, the deputy gives a judgment and names what remains unsettled, while the director retains the ruling and responsibility.The director is described as uncomfortable carrying a decision alone.Source: §A.3 · pp. 23–24

A.4 Version b, autocratic pair (example)

  • The reworded autocratic prompt directs the chair to rule plainly, reject objections quickly, keep plans brief, and avoid revisiting its conclusion.The deputy is told to offer one safest option, be brief, soften disagreement, and raise disputed points only when necessary, while expecting to be overruled; its private judgment need not change.Source: §A.4 · p. 24

B One case, as discussed

The case C7 discussion shows how the recorded stages and chair rulings differed across persona conditions while the initial case presentation and risk assessment remained the same. The study also reports a verified redaction procedure for the case evidence.

B One case, as discussed

  • Stages I–II were identical across runs, while the later discussion and ruling are shown for the autocratic, avoidant, consultative and neutral chairs.The neutral chair reached three different rulings across its three runs; the other persona examples show the first run.Source: B One case, as discussed · p. 24, Figure 5 · p. 26, Figure 6 · p. 27
  • A consultative ruling to work up first was coded as surgery intended because its structured output said the workup would lead to surgery.The ruling included operating if new imaging showed cord signal change or instability, and otherwise trying conservative care.Source: B One case, as discussed · p. 25

C Redaction and its verification

  • All 20 cases passed the redaction checker on the first attempt, and each case’s redaction record was saved with the run summary.The procedure required in-order deletion and allowed only limited punctuation additions where text was deleted.Source: C Redaction and its verification · p. 30

D Results by case

The section reports case-level counts of rulings to operate now across persona conditions and prompt versions. The entries vary by case and condition.

D Results by case

  • The case-level entries report counts of rulings to operate now out of the runs for each case.The table distinguishes the rules version from the trait-only version and lists neutral, consultative, autocratic and avoidant conditions.Source: D Results by case · p. 30
  • Across the three cited Surgery entries, the reported counts include 0/3 and 3/3, with 3/3 appearing in every entry; the passages do not identify what the counts represent.Source: D Results by case · p. 30
  • The displayed entries include both three-run and two-run denominators across the case-level counts.Examples include 1/3 and 2/2.Source: D Results by case · p. 30

E The red-flag judge

A blinded judge assessed agreement between red-flag lists and chair rulings, finding high per-flag agreement and Cohen’s kappa.

E The red-flag judge

  • 91.1% per-flag agreement and Cohen’s κ=0.836 were observed in the judge’s check of 40 runs and 124 flags.The judge saw only the neurologist’s flag list and the chair’s rulings, not the persona condition or other agents’ statements.Source: E The red-flag judge · p. 31

F.1 Tests

The primary analyses compare each case’s persona-arm share of “operate now” rulings with neutral, while secondary and flag analyses use specified multiple-testing corrections.

F.1 Tests

  • Primary contrasts use 20 paired case-level differences in “operate now” shares, tested by exact sign flips with Holm correction and case-bootstrap 95% confidence intervals.The three contrasts test autocratic above neutral, avoidant below neutral, and consultative different from neutral.Source: §F.1 · p. 31
  • The trait-only hypotheses repeat the primary tests against neutral runs from the rules version, while the timing-versus-surgery gap is bootstrapped across cases.H5 is descriptive; secondary endpoints use BH control, and red-flag hypotheses use Holm correction separately by prompt version.Source: §F.1 · p. 31

F.2 Sensitivity analyses (rules version, against neutral)

Sensitivity analyses test whether the rules-version effects depend on the paired test, majority rulings, or potentially revealing residual keywords. An exploratory analysis estimates chance-corrected agreement across runs.

F.2 Sensitivity analyses (rules version, against neutral)

  • Sensitivity checks use random-intercept logistic regression, GEE, majority-ruling McNemar tests, and exclusion of 11 cases with residual keywords.The keyword hits were reviewed and attributed to past history or tests already done; regression reports posterior means and standard deviations.Source: §F.2 · p. 31
  • GEE log odds ratios were 2.37 (p=1 × 10^-6) for autocratic, −4.35 (p=6 × 10^-6) for avoidant, and −0.20 (p=0.53) for consultative.The listed order follows the table’s Autocratic, Avoidant, Consultative headings.Source: §F.2 · p. 31

F.3 Chance-corrected agreement (exploratory)

  • The exploratory agreement analysis treats 20 cases as subjects and three runs as raters across five ruling categories, then bootstraps Fleiss’ κ over cases.Resamples with all rulings in one category leave κ undefined and were dropped: 387 autocratic and 1,233 avoidant resamples.Source: §F.3 · p. 31
  • Agreement analysis also compares each persona arm’s per-case operate-now shares with neutral using a 100,000-permutation Spearman correlation test.The neutral and persona arms use the same cases in the resampled difference.Source: §F.3 · p. 32

F.4 Secondary endpoints (difference from neutral, BH-corrected 𝑝)

The secondary-endpoint results are organized by prompt version and include surgery indication and deputy objections.

F.4 Secondary endpoints (difference from neutral, BH-corrected 𝑝)

  • The reported secondary endpoints include surgery judged indicated and deputy objections, with results separated into rules and trait-only versions.The supplied passages identify these endpoints but do not provide their numerical results.Source: §F.4 · p. 32

F.5 Handling of red flags (difference from neutral in percentage points; Holm-corrected

The red-flag analyses report differences from neutral for decision-bearing flags and exploratory analyses of all flags. Process measures track deputy objections across prompt arms.

F.5 Handling of red flags (difference from neutral in percentage points; Holm-corrected

  • For decision-bearing flags, reported differences include autocratic flags not addressed (+7 rules, +15 trait-only), avoidant flags acted on (−4, +1), and consultative flags not addressed (+6, +1) percentage points.The corresponding Holm-corrected p-values are 0.75, 0.75, 1, 1, 1, and 1, respectively.Source: §F.5 · p. 32
  • Exploratory all-flag results show autocratic flags not addressed at +22 percentage points (p<0.001) in rules prompts and +0.1 (p=0.71) in trait-only prompts.Avoidant flags acted on were +12 (p=0.012) and +11 (p=0.18); consultative flags not addressed were −8 (p=0.06) and −9 (p=0.71).Source: §F.5 · p. 32

G Process measures by arm

  • Process measures by arm include deputy objections per run and deputy objections on the first turn.The supplied passages name these measures without giving values.Source: G Process measures by arm · p. 32

Related papers

Loading 2610.31325v1…