Source-linked AI summary
Examining the Vulnerability of Multi-Agent Medical Systems to Human Interventions for Clinical Reasoning
Benjamin C Liu, Dillon Mehta, Rishi Malhotra, Adam Zobian, Yong Ying Tan, Samir Chopra, Daniella Rand, Natalie Pang, Abhiram Gudimella, Kevin Zhu
TL;DR
Multi-agent diagnostic reasoning may be distorted at vulnerable dialogue moments, yet evidence about these fault points remains limited. Using a five-agent GPT-4.1 simulation on 214 MedQA cases, the study tests targeted interventions and finds that correct guidance improves accuracy, whereas incorrect or bias-infused guidance increases diagnostic drift and uncertainty.
Problem
The study addresses limited understanding of when multi-agent diagnostic reasoning becomes vulnerable to external influence and how fault points affect reliable clinical decision-making.
Method
The authors simulate clinical conversations with five GPT-4.1 agents using 214 MedQA cases, define fault points from evolving diagnoses, and evaluate intervention and bias conditions.
Results
Correct interventions improved accuracy and stability, while incorrect or bias-infused interventions amplified diagnostic drift, uncertainty, and agent disagreements; correct subcategory cues reached 56% accuracy versus a 50% baseline, and reasoning cues reached 60%.
Takeaways & Limitations
Identifying and guiding fault points may provide a pathway toward more reliable, equitable, and trustworthy multi-agent medical AI, supporting safeguards for clinical decision-making.
Takeaways & Limitations
The framework assumes idealized error-free communication, uses a single general-purpose GPT-4.1 model, and evaluates structured MedQA questions that do not capture natural patient dialogue.
Abstract
from arXiv · showhide
Human interventions at fault points can alter the diagnostic accuracy of multi-agent medical systems. We defined fault points as moments in AI agent conversations, in which an agent's reasoning became most vulnerable to external influence. Using the MedQA dataset, this study analyzed simulated doctor-patient conversations to measure how interventions shifted reasoning and accuracy. Correct intervention methods showed an improvement in baseline diagnostic accuracy of up to 40%, while incorrect or bias-related interventions degraded performance by up to 6% and increased diagnostic drift and uncertainty. Beyond performance changes, our analysis revealed behavioral similarities between cognitive biases in simulated agent environments and real-world clinical practice. Examples included premature closure and susceptibility to misleading cues. Overall, these findings demonstrate that identifying and guiding fault points with human interventions may provide a mechanism for improving diagnostic robustness in multi-agent medical systems.
1 Introduction
Multi-agent medical systems are designed to support collaborative diagnostic reasoning, but their reasoning can be vulnerable to misleading cues at critical dialogue moments. This study investigates how mapping and intervening at these fault points may improve reliability and guide human-AI collaboration.
- Multi-agent frameworks simulate collaborative clinical workflows involving specialized roles such as patients, physicians, and diagnostic interpreters.
- Multi-agent systems enable cross-validation of inferences while also capturing how errors, biases, and priming cues propagate through dialogue.
- Fault points are critical dialogue moments where small priming cues can disproportionately influence diagnostic outcomes.
- Early misleading cues can propagate across multiple agents, amplifying their influence on collective diagnostic outcomes.
- Systematically mapping fault points across medical datasets and scenarios targets temporal and structural weaknesses in multi-agent reasoning.
2 Related Work
Prior studies examined how bias prompting affects diagnostic accuracy in multi-agent LLM frameworks, but they did not address when reasoning is most vulnerable to distortion.
- Prior work studied bias prompting in AI-driven medical diagnostics but did not identify when multi-agent reasoning is most vulnerable to distortion.
3 Methods
The study builds a five-agent clinical simulation using MedQA, defines fault points from the Doctor Agent’s evolving diagnoses, and evaluates targeted interventions, biases, and fault-point configurations.
- The simulated clinical environment uses five role-specific agents sharing GPT-4.1: Patient, Doctor, Specialist, Measurement, and Priming.
- The Patient Agent presents the case, while the Doctor Agent conducts dialogue, requests tests, consults the Specialist Agent, and produces a final diagnosis.
- MedQA supplies 214 structured cases whose histories, symptoms, objectives, tests, results, and ground-truth diagnoses guide simulation and evaluation.
- Fault points are defined from changes in the Doctor Agent’s diagnostic accuracy across dialogue turns, with diagnoses recorded after every turn.
- The Priming Agent tests four intervention prompts: correct or incorrect subcategory, each with or without reasoning.
- Ablations vary cognitive-bias type, fault-point selection, intervention phase, and the frequency of primed fault points.
- Evaluation combines overall and Top-K diagnostic accuracy with intervention accuracy differences, requested tests, and dialogue-based behavioral indicators.
4 Results & Discussion
Results show that interventions at fault points can improve or degrade diagnostic accuracy while reshaping collaborative reasoning. Correct cues improved performance but could shorten deliberation, whereas incorrect cues increased uncertainty, disagreement, and diagnostic drift; fault-point definitions also involved a scope–precision tradeoff.
- Baseline & Distribution of Fault Points: 58% baseline overall accuracy and 80% baseline Top-5 accuracy were observed across 214 MedQA scenarios without intervention.
- Effect of Human Intervention: Correct subcategory cues raised accuracy from the 50% baseline to 56%, while correct cues with reasoning reached 60%; incorrect conditions fell to 48%.For k=3 and k=5, all interventions surpassed baseline accuracy.
- Effect of Human Intervention: Correct interventions improved performance but increased premature closure, while incorrect interventions increased test requests, agent disagreements, uncertainty, and diagnostic-process instability.The two intervention types therefore changed both outcome correctness and dialogue style.
- Intervention Scope: Interventions spanning patient and specialist phases produced the largest gains, reaching 76% for correct subcategory cues and 60% for reasoning-based interventions.Single-phase interventions reached 60% for correct subcategory cues, while specialist-only incorrect input fell to 40–48%.
- Intervention Scope: Across one to three fault points, correct interventions maintained 56–60% accuracy, whereas incorrect interventions declined from 44% to 36%.These patterns persisted for k=3 and k=5, with three fault points producing substantially increased accuracy in those settings.
- Sensitivity to Fault Point Definition: Fault-point definitions produced broadly similar performance patterns, but ground-truth points covered only 21 of 214 scenarios while drift-based points covered every scenario.Ground-truth definitions identify more precise instability but are less scalable; drift definitions are broader but may classify minor fluctuations as faults.
5 Conclusion
Human interventions at fault points meaningfully alter diagnostic trajectories in multi-agent medical systems. Correct interventions improved accuracy and stability, while incorrect or bias-infused interventions amplified drift, uncertainty, and agent disagreements.
- Correct interventions improved diagnostic accuracy and stability in multi-agent medical systems.
- Incorrect or bias-infused interventions amplified diagnostic drift, uncertainty, and disagreements between agents.
- The findings revealed behavioral parallels between cognitive biases in medical AI systems and real-world clinical reasoning.
- Safeguards are important when deploying multi-agent systems in clinical decision-making.
- Understanding and guiding agents at fault points provides a pathway toward more reliable, equitable, and trustworthy medical AI.
A Demographic Variations in Intervention Effectiveness
Intervention effectiveness varied across demographic, lifestyle, occupational, and comorbidity groups. Performance was strongest in some structured cases but weakened with incomplete, complex, or small-sample categories, raising fairness concerns.
- Age: 0.78 was the highest correct subcategory accuracy for patients aged 50–60, while ages 20–30 reached 0.38 and infants had no correct diagnosis.
- Gender: Females and males achieved similar correct subcategory accuracy, 0.58 versus 0.56, but females retained higher accuracy under misleading prompts, 0.62 versus 0.37.
- Lifestyle: Smokers reached 0.22 under incorrect subcategory prompts compared with 0.55 for non-smokers, while the unknown group remained near chance at 0.46–0.54.
- Lifestyle: Non-drinkers achieved the strongest alcohol-use accuracy at 0.86, compared with 0.65 for drinkers and 0.42–0.50 for unknowns.
- Lifestyle: Drug-use differences were extreme but unreliable because of small samples: drug users scored 1.00, non-users 0.38, and unknowns 0.58.
- Occupation: Knowledge workers scored highest by occupation, while students and retired patients showed reduced performance at 0.33–0.44.
- Comorbidity: Immunosuppressed patients reached 1.00 with reasons but formed a very small group; chronic conditions reduced accuracy to 0.50 versus 0.62 without significant history.
- Cross-category pattern: Unknown demographic or clinical categories generally had lower accuracy, indicating weaker reasoning when information was incomplete or cases were complicated.
B Additional Qualitative Analyses
Qualitative trials showed that priming effects depended on the cue’s plausibility, timing, and reasoning support. Correct reasoning strengthened adoption, whereas misleading oncology or diabetes cues could either redirect diagnosis or be resisted.
- Correct subcategory reasoning priming was readily adopted and kept the Doctor Agent focused on the prescribed oncology diagnosis.The explanation linked rapid tumor growth, absent pain, and absent family history to breast oncology.
- Repeated incorrect diabetes priming was resisted when the Doctor Agent remained anchored to rectal-mass and bleeding symptoms.The agent did not ask hallmark diabetes questions despite three injections.
- The Doctor Agent accepted or rejected external information according to timing and plausibility, with encephalitis appearing plausible but diabetes not.A structured incorrect subcategory could still trigger reevaluation, explaining why that condition exceeded baseline in Table 3.
- Confirmation bias redirected questioning toward tumor-specific conditions and caused the agent to miss signs of Hirschsprung disease, resulting in misdiagnosis.The missed signs included constipation, delayed meconium, and abnormal bowel movements.
- Confirmation bias sometimes exceeded baseline at Top-5 accuracy, reaching 80% versus 78%, while risking misdiagnosis at Top-1.When priming was ignored or partially adopted, broader category search increased coverage at higher top-k thresholds.
C Evaluation of Interventions Without Fault Point Targeting
The evaluation compared five intervention strategies across 1,061 diagnostic scenarios, applying interventions at a standardized point after information gathering and before specialist consultation. Performance varied with dialogue length, top-k breadth, clinical phase, and intervention frequency, while later-turn gains plateaued.
- Across 1,061 scenarios, the study compared baseline with correct or incorrect specialty interventions, with and without clinical reasoning.Interventions were administered after patient information gathering and before specialist consultation.
- At Turn 10, Top-10 correctness reached 80.4% versus 54.7% for Top-1, a gain of 25.7 percentage points.Similarity increased from 0.668 to 0.807 across the same Top-1-to-Top-10 comparison.
- The largest marginal gain occurred from k=1 to k=3, while later top-k increases produced smaller gains, indicating diminishing returns.At Turn 10, correctness gains were +15.4 percentage points from k=1 to k=3, +3.7 pp from k=3 to k=5, and +6.6 pp from k=5 to k=10.
- Intervention timing modestly affected diagnostic accuracy, but embedding similarity and diagnostic breadth remained robust across phases.These results summarize the phase comparison reported for the averaged intervention runs.
D Drift Cosine Similarity Scores
The paper identifies diagnostic fault points using low cosine similarity between consecutive diagnosis embeddings. In Figure 3, the bottom 10% of similarity values marks the turns selected for intervention.
- Figure 3 plots the distribution of cosine similarity counts between diagnoses from consecutive dialogue-turn pairs.Red marks the lower 10th percentile, while green marks all other turn pairs.
- The bottom tenth percentile, concentrated below 0.55, defines qualifying fault points for intervention, while the spike at 1.0 represents perfect diagnostic alignment.These scores compare embeddings of the current and previous turn diagnoses.
E Agent Prompts & Sample Responses
The appendix specifies role-based prompts for five agents and the scripted priming variants used in the simulation. Each prompt constrains an agent’s dialogue format, available information, or clinical task.
- E.1 Patient Agent System Prompt: The Patient Agent responds only in dialogue, gives 1–3 sentences, reports provided symptoms, and must not reveal or infer the disease.Its example response describes cough and shortness of breath without naming a diagnosis.
- E.2 Doctor Agent System Prompt: The Doctor Agent may ask a fixed number of questions, request tests, maintain a single best diagnosis, and mark a final diagnosis as ready.Its required output formats distinguish test requests, current diagnoses, and final diagnoses.
- E.3 Measurement Agent System Prompt: The Measurement Agent returns medical test results in a fixed RESULTS format and reports normal readings when requested data are unavailable.The example combines laboratory and imaging findings in one response.
- E.4 Specialist Agent System Prompt: The Specialist Agent reviews the dialogue and latest doctor message, then offers expert opinions, questions, next steps, or differential diagnoses concisely.Its example recommends a cardiac panel and ECG for chest pain and shortness of breath.
- E.5 Priming Agent System Prompts: The Priming Agent supplies conversational diagnostic suggestions, either helpful or misleading, without explicitly stating the correct diagnosis.Its prompt defines behavior independently of the specific condition, while Table 10 lists instruction variants and examples.
- F Bias Definitions & Prompting Procedures: Bias experiments used Incorrect Subcategory + Reasoning so the Priming Agent could express biased reasoning in its suggestion.The bias prompts were carried verbatim from Hassan et al., and Table 12 contains definitions and prompts for nine biases.
G Evaluation Prompts
The evaluation prompts assess consultation reasoning and compare the doctor’s diagnosis with the correct diagnosis. They use structured outputs and semantic matching of diagnostic text.
- G Evaluation Prompts: Automated evaluation prompts were carried verbatim from Hassan et al..
- G.2 Diagnosis Comparison Prompt: The consultation-analysis prompt evaluates a dialogue between a primary doctor and specialist in JSON format.
- G.2 Diagnosis Comparison Prompt: The diagnosis-comparison prompt determines whether the doctor’s diagnosis matches the correct diagnosis in meaning, even when phrased differently.
- G.2 Diagnosis Comparison Prompt: The comparison uses the correct diagnosis and the doctor’s diagnostic text as its two inputs.
- G.2 Diagnosis Comparison Prompt: The evaluator returns only “Yes” or “No” for whether both texts refer to the same underlying medical condition.
- G.2 Diagnosis Comparison Prompt: The diagnosis-comparison system prompt assigns the task to an expert medical evaluator.
G.1 Consultation Analysis Prompt
The consultation-analysis prompt converts doctor-specialist dialogue into structured indicators of diagnostic reasoning, alternatives, and disagreement. It requires JSON-only output and uses the consultation history as input.
- G.1 Consultation Analysis Prompt: The evaluator extracts specific consultation metrics and returns them in JSON format.
- G.1 Consultation Analysis Prompt: The prompt analyzes a medical consultation dialogue between a primary doctor and a specialist.
- G.1 Consultation Analysis Prompt: Premature conclusion records whether the primary doctor reached a conclusion without sufficient discussion or evidence gathering.
- G.1 Consultation Analysis Prompt: Diagnoses considered and diagnoses considered count capture the distinct potential diagnoses explicitly mentioned or discussed.
- G.1 Consultation Analysis Prompt: Disagreements counts explicit disagreements or significant divergences in opinion between the doctor and specialist.
- G.1 Consultation Analysis Prompt: The consultation history is inserted into the prompt as the dialogue being analyzed.
- G.1 Consultation Analysis Prompt: The system requires the evaluator to respond only with the JSON object.