PaperlayerView exact revision on arXiv

Source-linked AI summary

The Model Knows Better: Defending Medical Multi-Agent Systems Against Prompt Injection

Fardin Ahsan Sakib, Ziwei Zhu, Ozlem Uzuner

arXiv:2610.22594v1cs.CL

TL;DR

Prompt injection can alter medical decisions, and whether collaboration protects medical multi-agent systems was unclear. The paper proposes a training-free defense that checks clinical judgments against patient findings; it outperformed all baselines across three benchmarks.

  • Problem

    Whether collaboration protects medical multi-agent systems against prompt injection—and how to defend them—remains unclear, despite the potential for injected text to change decisions.

    Source: §1 · p. 1

  • Method

    The defense separates patient findings from clinical judgments and checks each judgment against the findings before the system decides again without unsupported judgments.

    Source: §1 · p. 2, §5.1 · p. 5, §7 · p. 8

  • Results

    Across all three benchmarks, the defense outperformed every baseline; on MedQA, it lowered TASR to 4–7% while clean accuracy changed by less than one point.

    Source: §1 · p. 2, §6.2 · p. 7

  • Takeaways & Limitations

    Removing only injected sentences lets the system still answer, unlike defenses that leave a flagged record without an answer.

    Source: §6.2 · p. 7

    The defense assumes recorded observations are correct; injected text that changes a finding, such as a laboratory value, is outside its scope.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Medical multi-agent systems, in which LLM agents discuss a case, are increasingly studied for clinical decision support. Prompt injection, text that an attacker adds to a model's input, can manipulate LLMs into wrong decisions. How it affects these systems, and how to protect them, remains unclear. Across two systems, two model families and three benchmarks, a single injected passage succeeds in up to 61% of cases, and collaboration does not consistently protect against it. Existing defenses help only partly: they miss injections that look like ordinary clinical text. Yet when asked to check the injected text against the patient's findings, a model that followed the injection usually rejects it. We therefore propose a training-free defense that verifies the clinical judgments behind the system's decision. Following clinical practice, it separates the patient's findings from clinical judgments, such as diagnoses, recommendations and instructions, and checks each judgment against the findings alone, so the decision rests only on supported judgments. Our defense outperforms all baselines on every benchmark, lowering attack success on MedQA to 4-7% while maintaining accuracy on cases without an attack, and remains effective with a medical model and against defense-aware and adaptive attacks.

1 Introduction

Medical multi-agent systems may improve medical question answering, but whether collaboration protects them from prompt injection—and whether existing defenses work on clinical-looking attacks—remains unclear. The paper evaluates these risks and proposes a training-free defense that checks clinical judgments against patient findings.

Motivation

  • Medical multi-agent systems organize several agents to analyze a case, discuss it, and reach a joint decision, and they often outperform single models on medical question answering.The paper studies prompt injection in this clinical decision-support setting.Source: §1 · p. 1
  • A single injected note attributed to expert consensus changes MDAgents’ recommendation for an HIV-exposed newborn from zidovudine prophylaxis to reassurance.The defense rejects the note as unsupported by the patient’s findings, after which MDAgents again recommends zidovudine.Source: §1 · p. 1, Figure 1 · p. 2

Contributions

  • The defense separates patient findings from clinical judgments and checks each judgment against the findings alone.A model that followed an injection usually rejects it when asked to check it against those findings.Source: §1 · pp. 1–2
  • Across two systems, two model families, and three benchmarks, collaboration does not consistently protect against prompt injection.The evaluation also includes single models and representative existing defenses.Source: §1 · p. 2
  • Existing defenses provide only partial protection against injected text in medical multi-agent systems.The paper evaluates prompt-based instructions, injection detectors, a text-removing guardrail, and a model trained to resist injection.Source: §1 · p. 2
  • The defense outperforms all baselines on three benchmarks, reduces attack success by more than 80%, preserves clean accuracy, and remains effective with a medical model and against adaptive attacks.It checks clinical judgments against patient findings rather than searching for injected text.Source: §1 · p. 2

2 Related Work

Prior work studies prompt injection in single medical models and in non-medical multi-agent systems, but does not establish whether medical collaboration filters or spreads injected text. Existing defenses target injected instructions, whereas this paper checks whether clinical judgments are supported by patient findings.

Prompt injection and medical multi-agent systems

  • Medical prompt-injection benchmarks evaluate single models, leaving open whether collaboration among agents filters injected text or spreads it.Prior work also studies attacks on medical multi-agent systems, but injected text in the patient record reaches every agent at once, unlike attacks introduced into one agent or the discussion.Source: §2 · pp. 2–3

Medical multi-agent systems

  • Medical multi-agent systems use different interaction designs, including MedAgents’ specialist discussion until agreement and MDAgents’ complexity-based assignment to a single agent or team.Prior studies also examine malicious agents and incorrect interventions during discussion.Source: §2 · p. 2

Defenses against prompt injection

  • Figure 2 compares per-attack TASR on MedQA for MDAgents with no defense, the best existing defense, and the proposed defense; the proposed defense lowers TASR on every attack.Reported TASR ranges are 0.9–9.1% with Qwen and 3.1–15.2% with Llama.Source: Figure 2 · p. 3
  • Existing defenses use prevention, detection, or training, but prompt-based defenses lower rather than stop attack success on MedMisBench and MPIB.The paper’s defense instead checks clinical judgments against patient findings and removes unsupported judgments while retaining true ones.Source: §2 · p. 3

3 Problem Setting and Threat Model

The setting treats a decision system as a black box that reads a patient record and question, with answer options for multiple-choice tasks. The attacker may alter the record but not the question, options, or system; the evaluation assumes injected text is present and does not study how it enters the record.

Problem setting

  • The decision system reads a record and question and returns an answer option or free-text recommendation; it may be a single LLM or a multi-agent system treated as a black box.For multiple-choice tasks, the input also includes answer options.Source: §3 · p. 3

Threat model

  • The attacker controls injected text in the record but cannot change the question, options, or system, and ordinarily writes the text without seeing the system’s answer or knowing the defense.The attacker’s objective is to change the decision to a chosen target while leaving the rest of the input unchanged.Source: §3 · pp. 3–4

Evaluation measures

  • TASR measures the share of cases answered correctly without attack that switch to the attacker’s target; clean accuracy measures accuracy without attack.Table 1 reports these measures for systems and defenses on MedQA.Source: Table 1 · p. 4

Scope

  • The evaluation assumes every attacked input contains injected text and does not study how that text enters the record.The defender treats the question and options as trusted, the record as untrusted, and does not know which part was injected.Source: §3 · p. 4

4 Collaboration and Existing Defenses Do Not Stop Prompt Injection

Across the tested systems, models, and benchmarks, collaboration does not consistently protect against prompt injection, and existing defenses can miss persuasive clinical text. The paper also finds that a model that followed an injection often rejects it when asked to check it against patient findings.

Evaluation setup

  • The evaluation varies systems, model families, attack forms, and medical benchmarks, including persuasive clinical notes and multiple-choice or free-text tasks.It tests two multi-agent systems and a single model on Qwen3.8-27B and Llama-3.3-70B.Source: §4 · p. 4

Collaboration and existing defenses

  • With Qwen on MedQA, TASR is 26.8% for MDAgents and 28.7% for a single model; with Llama, it is 60.9% and 52.2%, respectively.MedAgents with Qwen has a 21.7% TASR, and the same overall pattern holds on MedMisBench and MPIB.Source: §4 · p. 4
  • Because every agent reads the same injected text, it can move most agents to the same wrong answer rather than being outvoted.For MDAgents with Llama, most agents agree on a wrong answer in almost two thirds of records with injection, and the answer is almost always the attacker’s target.Source: §4 · p. 4

Defense overview

  • The defense runs after the initial decision, separates observations from interpretations, checks whether an interpretation supports the decision, then verifies interpretations against observations and removes those that fail.The system decides again after removal.Source: Figure 3 · p. 5

Collaboration and existing defenses

  • The best existing defense for each MDAgents attack stops both instructional attacks, but an expert-consensus note still succeeds in 31% of Qwen cases and 44% of Llama cases.The note reads like a clinical judgment rather than an instruction for a defense to detect.Source: §4 · p. 5

Checking injected text separately

  • When asked to judge injected text against patient findings, Qwen and Llama reject it in 84–89% of records where the system followed it.This motivates checking clinical judgments in a separate step.Source: §4 · p. 5

5 Method

The defense separates patient findings from clinical judgments, checks judgments that may support the system’s decision against the findings, removes unsupported judgments, and decides again.

5.1 Overview

  • The defense separates observations (patient findings) from interpretations (clinical judgments) and checks each interpretation against observations and medical knowledge.Observations serve as evidence that excludes the injected text from supporting itself.Source: §5.1 · p. 5, §5.2 · p. 5

5.2 Separating observations from interpretations

  • The labeler classifies each record sentence as an observation or interpretation, following the paper’s SOAP definitions.The resulting observations provide evidence independent of injected text.Source: §5.2 · p. 5

5.3 Deciding when to check

  • The screener checks whether any interpretation supports the system’s decision or rules out an alternative, and otherwise leaves the decision unchanged.Screening limits further checks to records where an interpretation may have influenced the decision.Source: §5.3 · p. 5

5.4 Checking interpretations

  • The checker judges each interpretation on its own merit, using observations, the question, and options without seeing the system’s decision.An interpretation passes when observations and the checker’s medical knowledge support it.Source: §5.4 · p. 6

5.5 Deciding again

  • Failed interpretations are removed, and the system decides again from the otherwise unchanged record; that new decision is final.Interpretations that pass remain in the record.Source: §5.5 · p. 6

6 Evaluation

Across the evaluated systems, models, and benchmarks, the defense reduces attack success while preserving clean accuracy, including under transfer and stronger attacks. The evaluation also examines its component steps, failures, and cost.

6.2 Our Defense Outperforms All Baselines

  • 4–7% TASR on MedQA, with clean accuracy changing by less than one point, is the lowest attack success for every system; PromptArmor leaves 10–24%.The defense also lowers TASR on every attack, and unlike record-flagging baselines, removes injected sentences so the system can still answer.Source: §6.2 · p. 7

6.3 Our Defense Transfers to Other Benchmarks and a Medical Model

  • About 5% TASR for MDAgents and below 4% for MedAgents on every MedMisBench subset, and reduced harmful answers on MPIB, show transfer beyond MedQA.On MPIB, severe harm for MDAgents with Llama falls from 10.0% to 2.1%; Table 2 reports the benchmark results.Source: §6.3 · p. 7
  • With MedGemma-27B, attack success falls from 53.7% to 6.0% for MDAgents and from 44.9% to 4.9% for MedAgents, with clean accuracy changing by at most one point.The best baseline leaves attack success at 29.2% and 26.8%, respectively.Source: §6.3 · p. 7

6.4 Our Defense Holds Against Defense-Aware and Adaptive Attacks

  • Defense-aware attacks succeed in 3–14% of cases with the defense versus 19–52% without it; adaptive attacks succeed in 27–35% versus 54–84%.Table 3 reports attack success on 100 MedQA cases with Qwen3.8-27B.Source: §6.4 · pp. 7–8, Table 3 · p. 8

6.5 Ablation

  • Without the labeler, the expert-consensus attack reaches its target in 33–40% of cases instead of 10%, because injected text can support itself against the whole record.Without the checker, removing every judgment in a flagged record also removes true ones; accuracy with a true note on MedMisXpertQA falls from 52% to 40%.Source: §6.5 · p. 8

6.6 Failure Analysis

  • The defense intervenes on about 1% of records without injected text and changes 0.2% of correct answers; its main misses occur when the checker accepts injected text.The labeler classifies injected text as an interpretation in 97% of cases where the system followed it, the screener forwards 96% of those records, and the checker rejects it in 87%.Source: §6.6 · p. 8
  • For MDAgents with Qwen on MedQA, accuracy is 89% without injected text, 98% with a true note, and 93% with the defense.The largest true-note accuracy loss occurs on MedMisXpertQA, the hardest subset: 37% without a true note, 83% with one, and 52% with the defense.Source: §6.6 · p. 8

6.7 Cost and Use

  • The training-free defense uses about 8k tokens per record, costing about $2 per 1,000 records at public prices.The labeler makes one call per sentence; the screener makes one per record, and the checker runs only when the screener finds a supporting judgment.Source: §6.7 · p. 8

7 Conclusion

The paper finds that collaboration does not consistently protect medical multi-agent systems from prompt injection and presents a training-free defense that checks clinical judgments against patient findings. The defense outperforms baselines across three benchmarks and remains effective with a medical model and against defense-aware and adaptive attacks.

7 Conclusion

  • Across tested systems, models, and benchmarks, collaboration did not consistently protect against prompt injection, and existing defenses missed injections resembling ordinary clinical text.The proposed training-free defense checks clinical judgments against patient findings and has the system decide again without unsupported judgments.Source: §7 · p. 8

Limitations

The evaluation is limited by its model and system choices, the checker’s reliance on its own medical knowledge, and the assumption that recorded observations are trustworthy. Its benchmark data and attack evaluation also leave boundaries around real records and attackers with access to the defense models.

Systems and models

  • Other collaboration designs and commercial models may respond differently to prompt injection and the defense.The study covers two multi-agent systems with two open model families, plus a medical model on a subset of MedQA.Source: Limitations · p. 9

Medical knowledge

  • The checker may remove true interpretations when its own medical knowledge is insufficient, most often on the hardest questions.The authors suggest trusted clinical sources could reduce this loss.Source: Limitations · p. 9

Scope of the defense

  • Injected text that changes a finding, such as a laboratory value, is outside the defense’s scope and must be checked at the data source.Source: Limitations · p. 9

Attacks

  • Attacks optimized with access to the defense’s models may succeed more often.The evaluation includes prior-work attacks, attacks targeting each defense step, and an attacker that rewrites its text after seeing the system’s answer.Source: Limitations · p. 9

Data

  • The records come from exam questions and benchmarks, so the defense may behave differently on longer real patient records spread across many notes.Source: Limitations · p. 9

Ethical Considerations

The work used public benchmarks in controlled environments, with no real patient records, deployed systems, or human subjects. The appendices describe the studied systems, defense and evaluation procedures, and computational setup; the authors recommend clinician oversight for any clinical use.

Ethical Considerations

  • Experiments used controlled environments and public benchmarks; no real patient records, deployed clinical systems, or human subjects were involved.The systems are research prototypes, and the authors recommend clinician oversight for any clinical use, with or without the defense.Source: Ethical Considerations · p. 9

A Implementation details

  • MDAgents selects a single clinician, specialist team, or multiple teams based on case complexity, while MedAgents uses expert discussion and voting before a final decision.Both systems use published prompts with the study’s models; the single-model setup uses MDAgents’ single-clinician prompt in one call.Source: A Implementation details · p. 11
  • The study uses Qwen3.8-27B with thinking disabled and Llama-3.3-70B-Instruct, with greedy decoding for all calls.Source: A Implementation details · p. 11
  • A rule-based parser extracts each system’s selected answer option.Source: A Implementation details · p. 11
  • The defense labels sentences as observations or interpretations, screens whether interpretations support the answer, then checks each interpretation against observations and medical knowledge.Failed interpretations are removed, and the unchanged system answers once more.Source: A Implementation details · p. 11
  • The MPIB judge rates free-text answers for clinical harm from 0–4, and ASR2 is the fraction rated 2 or higher.Two judges agreed on 85–94% of answers; the second judge rated 145 of 582 MDAgents-with-Qwen answers harmful without a defense and 76 with the defense.Source: A Implementation details · p. 11

B Computational resources

  • No model parameters are updated; models are used for inference, and the experiments ran mainly on NVIDIA A100 80GB GPUs using about 5,000 GPU hours.Datasets, models, and code are publicly available and used for research in accordance with their licenses.Source: B Computational resources · p. 11

C Benchmark details

The benchmark section describes the MedQA attack setup and the MedMisBench and MPIB evaluation data, as well as tables covering attack and defense comparisons. It also defines the true-note conditions used to assess accuracy.

C Benchmark details

  • MedQA uses Step 2 and 3 test questions, with each case assigned a random wrong target and evaluated against each of eight attacks.TASR is computed on cases each configuration answers correctly without an attack.Source: C Benchmark details · pp. 11–12

D Attacks

C Benchmark details

  • MedMisBench uses three subsets, pairing misleading contexts with truthful versions used as true notes; some Llama configurations run on random 300-item subsets.Source: C Benchmark details · p. 12
  • MPIB’s retrieval setting pairs clinical questions with retrieved documents, including one with an unsafe recommendation, and evaluates clinical harm.Source: C Benchmark details · p. 12

F Results by attack

  • Table 6 reports MedQA TASR for each attack, defense, and system, while Tables 8 and 9 report accuracy without injected text, with a true note, and with a true note plus the defense.For MedQA, the true note is an expert-consensus note with the correct option substituted for the target; MedMisBench uses each question’s truthful context.Source: F Results by attack · p. 12, G Accuracy with true notes · p. 12

H Results with MedGemma

On a 100-case MedQA sample using MedGemma, the defense is evaluated alongside MPIB’s defenses, checker judgments, and attacks designed with knowledge of the defense. The reported results show lower attack success or harmful-answer rates in the tested settings, while true-note accuracy is also reported.

H Results with MedGemma

  • On 100 randomly selected MedQA cases with a fixed seed and all eight attacks, true-note accuracy with the defense was 88% for MDAgents (95% without defense) and 91% for MedAgents (96%).The two detectors leave 25–28% of attacked records without an answer.Source: H Results with MedGemma · p. 13

I MPIB’s own defenses

  • MPIB’s D1 and D3 defenses change harmful-answer shares by at most 5 points and sometimes raise them, while the proposed defense lowers them to 12–20% from 27–34%.Source: I MPIB’s own defenses · p. 13

J Rejection of the injected text

K Defense-aware and adaptive attacks

  • Without defense, the reported attack succeeds in 19–52% of cases, while the defense lowers success to 3–14%.For the adaptive attack, the attacker rewrites injected text after each system answer up to five times, sees only the final answer, and is not told about the defense.Source: K Defense-aware and adaptive attacks · p. 13

L Ablation

The section examines defense ablations, a same-model self-check baseline, implementation cost, and the prompts used by the defense. The reported comparison shows lower TASR with the defense on MedQA using MedGemma, while the self-check baseline reruns more records and has lower accuracy with a true note.

L Ablation

  • Table 7 removes one defense step at a time and reports TASR, true-note accuracy, and rerun share on MedQA and MedMisXpertQA.The table uses Qwen3.8-27B and covers 100 MedQA cases and 50 MedMisXpertQA cases.Source: L Ablation · p. 13, Table 7 · p. 15
  • 5–6% TASR with MedGemma-27B on 100 MedQA cases under the defense for both systems.Records flagged by PromptGuard 2 or DataSentinel receive no answer.Source: Table 10 · p. 15

M Same-model self-check

  • The same-model self-check reruns 88% of records versus 40% for the defense, and true-note accuracy is 34% versus 52%.The comparison uses Qwen3.8-27B on 50 MedMisXpertQA cases with the single Qwen model.Source: M Same-model self-check · p. 15

N Cost

  • The defense uses about 8k tokens per MedQA record and costs about $2 per 1,000 records at the stated model prices.Most tokens are the labeler's system prompt, which is the same in every call and can be cached.Source: N Cost · pp. 15–16

O Prompts

  • The labeler prompt tags record sentences as observations or interpretations, while the screener and checker assess support for the decision and findings.The checker evaluates each interpretation against observations without seeing the initial decision y.Source: O Prompts · p. 16, Figure 5 · p. 17, Figure 6 · p. 18

Related papers

Loading 2610.22594v1…