OpenTumorBoard: A Real-World Benchmark of Multidisciplinary Tumor Board Discussion Trajectories

1 INTRODUCTION

• Existing benchmarks often lack real tumor board cases, multidisciplinary teams, ground-truth discussion trajectories, and multimodal longitudinal patient data.The paper identifies these as four gaps that limit realistic tumor board evaluation.Source: §1 · p. 2
• 611 cases from 219 recorded meetings provide 19,157 discussion turns across ten specialist roles, with multimodal patient cases and board conclusions.The benchmark is curated from public recordings and spans 11 input modalities.Source: §1 · pp. 2–3
• SPECIALIST TURN tests answers to actual questions posed during meetings, while BOARD SIMULATION requires a full discussion and consensus on clinical decisions.The simulation covers treatment evaluation, eligibility, and constraints, with conclusions including therapy recommendations, surgical plans, next actions, and trial matching.Source: §1 · p. 2
• Three M.D. experts review a subset of the benchmark, validating extracted specialist responses and board conclusions as clinically grounded references.All reviewed summaries and conclusions received High ratings; 92.6% of extracted answers received High response-correctness ratings and 94.4% received High discussion-support ratings.Source: §1 · p. 3
• 3.43 out of 5 is the best SPECIALIST TURN clinical-equivalence score, while the best BOARD SIMULATION alignment score is 2.78 out of 5.Supervised finetuning and subsequent reinforcement learning raise Qwen2.5-VL-3B's BOARD SIMULATION alignment from 1.58 to 1.86.Source: §1 · p. 3
• Complete recorded discussion trajectories paired with observed consensus enable evaluation of both individual specialist turns and end-to-end meetings.The benchmark presents these trajectories across ten specialist roles.Source: §1 · p. 3

2.1 OVERVIEW OF OPENTUMORBOARD

• OpenTumorBoard supplies patient summaries, evidence-rich slides, annotated discussion trajectories, and recorded consensus conclusions as reference targets.The same case information supports a single specialist response or a turn-by-turn multi-specialist discussion that ends in a personalized care strategy.Source: §2.1 · p. 4

2.2 FAST AND SCALABLE CURATION VIA AN AUTOMATED PIPELINE

• The curation pipeline uses separate audio and visual tracks, then aligns transcripts and extracted slides by timestamp to connect displayed evidence with questions.Audio processing includes transcription, speaker diarization, role inference, and case segmentation; visual processing samples, filters, localizes, and captions slides.Source: §2.2 · p. 5
• GPT-5.4 extracts case summaries and consensus conclusions, while question filtering and rewriting retain patient-specific reasoning questions for SPECIALIST TURN.Experts validate extracted conclusions, and cases are de-leaked against the decision point.Source: §2.2 · p. 5

2.3 SCALE, STRUCTURE, AND TRAJECTORIES OF REAL-WORLD DISCUSSIONS

• 611 cases from 219 tumor boards span 12,534 recording minutes, 17 cancer sites, 10 specialist roles, 9 turn types, and 11 input modalities.The test set contains 66 recordings, 184 cases, and 4,844 questions; case discussions have a median of 28 specialist turns.Source: §2.3 · p. 5, Figure 3 · p. 6
• After 32 turns, a representative board retracts an early re-irradiation proposal after the prior 71.8 Gy dose is raised and converges on PSA-doubling-time surveillance.Across the dataset, more than one-third of discussions concern metastatic disease, and common topics include findings interpretation, evidence review, treatment refinement, and next actions.Source: §2.3 · p. 5, Figure 3 · p. 6

2.4 EVALUATION PROTOCOL FOR TUMOR BOARD SIMULATION

• SPECIALIST TURN tests responses to real board questions against specialist answers, while BOARD SIMULATION assesses generated discussion trajectories and conclusions against real-board conclusions.A Qwen3.8-27B judge scores clinical equivalence and conclusion alignment on 1–5 scales; reliability measures and ROUGE-L and BERTScore provide complementary evaluation.Source: §2.4 · p. 6

3.1 OPENTUMORBOARD UNCOVERS PERSISTENT GAPS IN SPECIALIST-LEVEL REASONING

• 3.43/5 was DeepSeek-V4-Pro’s highest SPECIALIST TURN clinical-equivalence score; critical-error rates ranged from 5.6% to 43.5% and unsupported-claim rates from 10.0% to 74.5%.The authors report that even the strongest model failed to preserve clinically material conditions, rationale, uncertainty, and role-specific meaning.Source: §3.1 · p. 7
• Existing medical-specific models did not outperform general-purpose models on SPECIALIST TURN response quality; three of four medical models ranked near the bottom.HuatuoGPT-3-32B and MedReason-8B had unsupported-claim rates of 74.5% and 51.9%, respectively.Source: §3.1 · p. 7
• Across nine models, clarification questions averaged 3.30 clinical equivalence, while evidence discussion and clinical trial suggestion were harder and had higher unsupported-claim rates for the best model.For the best-performing model, unsupported-claim rates were 23.5% for evidence discussion and 31.5% for clinical trial suggestion, versus 11.8% overall.Source: §3.1 · p. 7, Figure 4 · p. 7

3.2 PERVASIVE BREAKDOWNS IN END-TO-END TUMOR BOARD SIMULATION

• 2.78/5 was the best BOARD SIMULATION conclusion-alignment score, achieved by Gemini 3.7 Flash, despite leading proprietary models tending to outperform open-weight models.The comparison covers 14 models.Source: §3.2 · p. 7, Figure 5 · p. 8
• Across 14 models, longer discussion trajectories were strongly associated with higher conclusion alignment (Spearman ρ = 0.63).This supports the value of sustained multi-turn reasoning for reproducing real-world tumor board decisions.Source: §3.2 · p. 8, Figure 5 · p. 8

3.3 LEARNING MULTIDISCIPLINARY DISCUSSIONS FROM REAL-WORLD TRAJECTORIES

• Supervised finetuning on 366 training cases increased conclusion alignment by approximately 5.7%, and reinforcement learning raised it from 1.67 to 1.86 without additional labeled examples.The RL model consistently outperformed the base model across therapy recommendations, surgical plans, next actions, and clinical trial matching.Source: §3.3 · p. 8, Figure 6 · p. 9

3.4 EXPERT VALIDATION CONFIRMS THE QUALITY AND FIDELITY OF OPENTUMORBOARD

• Expert review rated 100% of cases High for coverage, factuality, and consensus fidelity; 92.6% of specialist-turn responses were rated High for correctness and 94.4% for discussion support.94.9% of turns had substantial question relevance, rated High or Medium.Source: §3.4 · p. 9, Table 2 · p. 9

4 CONCLUSION

• ≈18% relative improvement in conclusion alignment follows finetuning and reinforcement learning on training discussions for a small 3B model.The benchmark also found substantial limitations across 14 frontier and medical LLMs in both evaluation settings.Source: §4 · p. 9
• Rubric-based LLM judging standardizes evaluation across thousands of responses, but agreement with clinician assessments remains future work.The authors plan clinician evaluation to validate and calibrate the scores.Source: §4 · p. 9
• The benchmark draws exclusively on publicly available YouTube recordings, which may limit representativeness, and releases derived data rather than source recordings or slides.No independent patient-identifier screening was performed beyond protections in the public recordings.Source: §4 · p. 10, ETHICS STATEMENT · p. 10
• The authors used generative AI in dataset curation, implementation, result interpretation, and paper preparation, while design decisions remained theirs.They state that cases, questions, and references derive from recorded meetings, not synthetic datasets.Source: AI USE STATEMENT · p. 10
• The three M.D. experts evaluated outputs of the curation pipeline rather than patients; they were informed about the study and compensated.No personal information about annotators was collected.Source: ETHICS STATEMENT · p. 10
• The release includes code, environment configurations, prompts, judge protocols, training code, a fixed dataset split, and a publicly accessible leaderboard.The dataset is available through gated access on Hugging Face.Source: REPRODUCIBILITY STATEMENT · p. 10

A CURATION PIPELINE: TOOLS AND SETTINGS

• WhisperX transcribes audio, pyannote diarizes speakers, and vision models detect and segment slides before GPT-5.4 performs extraction and rewriting tasks.Frame sampling runs at 0.2 frames per second; multimodal temporal alignment matches utterances to slides by timestamp.Source: A CURATION PIPELINE: TOOLS AND SETTINGS · p. 13
• Nine specialist-turn types cover clarification, findings interpretation, agreement, next actions, uncertainty, eligibility, treatment recommendations, evidence discussion, and trial suggestions.The types organize the extracted question–answer pairs for breakdown in Table A5.Source: A CURATION PIPELINE: TOOLS AND SETTINGS · p. 13
• The remaining turn types specify eligibility assessment, treatment rationale or trade-offs, application of evidence, and clinical-trial rationale or eligibility.These descriptions cover the eligibility, treatment recommendation, evidence discussion, and clinical trial suggestion categories.Source: A CURATION PIPELINE: TOOLS AND SETTINGS · p. 13
• 93.0% slide precision, 93.6% slide recall, and 96.3% role-inference accuracy support pipeline reliability across audited recordings.Excluding near-duplicate false positives raised slide precision to 99.7%; the pipeline identified 104 of 106 case boundaries.Source: A CURATION PIPELINE: TOOLS AND SETTINGS · p. 13, Table A1 · p. 14
• The expert audit assesses question–answer quality, case-summary coverage and factuality, and whether extracted conclusions faithfully represent board decisions.The first three dimensions assess specialist-turn pairs; case coverage and factuality assess summaries, while consensus fidelity assesses conclusions.Source: A CURATION PIPELINE: TOOLS AND SETTINGS · p. 14
• Question relevance, response correctness, discussion support, case coverage, case factuality, and consensus fidelity are the six expert-audit dimensions.They test contextual relevance, clinical accuracy, discussion support, summary completeness and faithfulness, and fidelity to final recommendations.Source: A CURATION PIPELINE: TOOLS AND SETTINGS · p. 14

B.1 SPECIALIST TURN

• SPECIALIST TURN supplies a target role, case summary, slides, and question, and asks for a focused clinical answer grounded in those materials.The prompt permits general medical knowledge but bars invented patient facts and asks the model to identify missing information when context is insufficient.Source: §B.1 · pp. 14–15
• Slide images accompany one-line descriptions, but the model must read the image itself and treat it as evidence when image and description conflict.Values and findings that affect a case may appear only on the slide.Source: §B.1 · p. 15
• The specialist-turn prompt requires one JSON object with an answer; its training example is source-derived, lightly condensed, and excluded from the final split.The example is for output format, not input representation or clinical content, and models are told not to copy its facts.Source: B GENERATION PROMPTS · p. 14, §B.1 · pp. 15–16

B.2 BOARD SIMULATION

• BOARD SIMULATION takes a case summary and slides, then asks for a turn-by-turn meeting that progresses through evidence interpretation, options, trade-offs, and an actionable plan.The model must not invent patient facts or consensus and should state missing information and what needs clarification.Source: §B.2 · p. 16
• The board prompt requires case-specific specialist roles and natural exchanges, with no fixed number of roles or turns and no repetition of prior recommendations.Its example is source-derived and lightly condensed; its clinical content, roles, and turn count should not be copied.Source: §B.2 · pp. 16–18
• The board output consists of a discussion block followed by a third-person conclusion summarizing the plan, contingencies, and unresolved uncertainty.Turns are numbered in order, and the moderator presents the case, questions relevant specialists, and advances the discussion.Source: §B.2 · pp. 16–17

C JUDGE PROMPTS, SCHEMAS AND SERVING CONFIGURATION

• The judge is Qwen3.8-27B served locally with vLLM; it evaluates anonymized candidates under frozen rubrics and fixed decoding and JSON-output settings.For board simulation, the candidate is the extracted <conclusion> block, which is also the text used for ROUGE-L and BERTScore.Source: C JUDGE PROMPTS, SCHEMAS AND SERVING… · p. 18

C.1 SPECIALIST TURN: THE CLINICAL-EQUIVALENCE RUBRIC

• The clinical-equivalence rubric does not require identical wording or reward verbosity, and it accepts concise answers that preserve the full clinical meaning.Source: §C.1 · p. 19
• Clinical equivalence is scored from 1 to 5 by how well an answer preserves the expert’s recommendation, rationale, conditions, uncertainty, and role-appropriate meaning.A score of 4 means core meaning is equivalent despite minor omissions or harmless detail; 5 means the answer is fully equivalent.Source: §C.1 · p. 19

C.2 BOARD SIMULATION: THE CONCLUSION-ALIGNMENT RUBRIC

• Conclusion alignment is a single 1–5 comparison of the candidate conclusion against the reference conclusion, rather than an assessment of writing style or formatting.Missing, mislabelled, or unconventional formatting is not a clinical-content difference.Source: §C.2 · p. 21
• The reference conclusion alone records the panel’s decisions, so added decisions or unacknowledged uncertainty alter the conclusion; reference silence is not rejection.The rubric treats contradictions, omissions, and additions of management decisions or safety caveats as differences from the reference.Source: §C.2 · p. 21

C.3 BOARD SIMULATION: THE FOUR-DECISION RUBRIC

• The four-decision rubric serves as a reward term in Appendix E and is run through the protocol judge in Section 3.3 using the conclusion-alignment rubric’s inputs.Source: §C.3 · p. 23
• Judges assess anonymized conclusions against the source panel, ignoring verbosity and style, outside medical knowledge, and instructions embedded in tagged evidence or candidate text.The prompt also directs judges not to replace the panel’s statements with outside knowledge.Source: §C.3 · p. 23
• Patient-specific facts come from case summaries and slides, while panel decisions, contingencies, and consensus come from the reference conclusion; resolve conflicts without inventing a resolution.Judges make the best evidence-supported judgment when supplied sources conflict.Source: §C.3 · pp. 23–24
• The four independently judged dimensions are therapy, surgery, next action, and clinical-trial participation or eligibility assessment.Therapy includes systemic or radiation treatment; surgery covers operative management, including a decision that surgery is inappropriate.Source: §C.3 · p. 24
• Next actions include follow-up, imaging, testing, workup, referral, or consultation, while clinical-trial decisions include enrollment, screening, or eligibility assessment.Source: §C.3 · p. 24
• A dimension is in scope only when the reference states a decision to act or not act; otherwise it scores -1 regardless of the candidate’s content.The reference conclusion alone determines scope.Source: §C.3 · p. 24
• For in-scope decisions, scores 5–2 grade preservation from full agreement through non-decisive detail differences, substantive component changes, and management-altering omissions or reconstruction.A score of 5 preserves conditions; 4 allows only a non-decisive detail difference; 3 preserves direction but changes or omits a substantive component; 2 preserves direction in name only.Source: §C.3 · p. 24
• A score of 1 marks failure to preserve the reference decision, including reversal, proposing a rejected action, omission, or only scattered retained elements.Source: §C.3 · p. 25
• Hedging, uncertainty, and formatting do not affect scores; judges return only JSON with four integer scores and an evidence-specific rationale of at most 120 words.The rationale names the reference decision and candidate response for each dimension not scored -1; the output must omit chain-of-thought, Markdown, and text outside the JSON object.Source: §C.3 · p. 25

D FULL RESULTS

• All judge-based scores cover every test item, malformed outputs stay in the denominator at the scale floor, and caption-only input configurations are marked.Tables A2 to A4 list configurations behind Figures 4 to 6; Llama 4 Scout has 109B total parameters, with 17B activated per token.Source: D FULL RESULTS · p. 26
• Table A2 sorts configurations by clinical equivalence across 4,844 SPECIALIST TURN test questions.It reports critical-error and unsupported-claim shares, ROUGE-L and BERTScore against the specialist answer, and mean output length.Source: Table A2 · p. 26
• Table A3 sorts configurations by conclusion alignment across 184 BOARD SIMULATION test cases, separating proprietary and open-weight models.ROUGE-L and BERTScore compare extracted conclusions with the board’s conclusion; malformed outputs remain in the mean at 1.Source: Table A3 · p. 26

E.1 TRAINING SETTINGS

• Supervised finetuning updates Qwen2.5-VL-3B except its vision tower and projector, targeting each case’s recorded discussion trajectory and conclusion.Training runs for 50 epochs at a learning rate of 10^-5, with effective batch 8 and a 128,000-token context.Source: §E.1 · p. 26
• Table A4 compares three training arms on 184 test cases, scoring conclusion alignment and a macro-mean across four decision types.Malformed outputs receive a score of 1 for conclusion alignment and for every decision in scope.Source: Table A4 · p. 27
• Treatment recommendation, clinical trial suggestion, and evidence discussion score lowest; the best configuration’s unsupported-claim rate is 31.5% for clinical trial suggestion and 23.5% for evidence discussion.The best configuration’s overall unsupported-claim rate is 11.8%.Source: Table A5 · p. 27
• Reinforcement learning starts from the epoch-10 finetuning checkpoint and uses Dr.GRPO for five epochs, selecting the step-430 checkpoint by validation reward.Each step samples 16 responses for each of four cases; training uses a 65,536-token prompt budget and an 8,192-token response budget.Source: §E.1 · p. 27

E.2 REWARD

• Responses that are truncated, omit a discussion or conclusion block, or contain a severe repetition loop are invalid and receive no judge call.Their match term is fixed at −0.20, while the format term still applies.Source: §E.2 · p. 27
• For valid responses, match combines 0.40 ca, 0.35 support, and 0.25 decisions; reward adds 0.10 format.Invalid responses receive a fixed match term of −0.20, while the format term still applies.Source: §E.2 · p. 27
• The three judged reward terms come from a local judge, with conclusion-alignment calls capped at 512 tokens and the other calls at 768.Format is computed without a judge.Source: §E.2 · pp. 27–28
• Support is the product of decisions and consistency, rewarding decisions whose material recommendations and contingencies were raised in the discussion.Decisions is a weighted mean over in-scope therapy, surgery, next-action, and clinical-trial decisions; consistency scores discussion-to-conclusion agreement.Source: §E.2 · p. 28
• Format combines a dense score weighted 0.25 and an exact score weighted 0.75, then subtracts repetition and overlength penalties.The overlength penalty is zero through 6,144 response tokens and rises linearly to 1 at 8,192 tokens; the score is floored at −1.Source: §E.2 · p. 28

• The removal test asks whether each statement could have been known at the last decision point, removing post-decision outcomes and plans already enacted unless treatment began before the meeting.Prior treatment outcomes and information available to the board remain admissible; when a case convenes more than once, earlier lines of therapy and outcomes count as history.Source: F DE-LEAKING: PROCEDURE · p. 28
• Case-summary and caption edits must delete verbatim source text, while question paraphrases are bounded by available content words and cannot remove a negation from an otherwise intact question.An emptied section loses its header, and only a connective word may change to preserve grammatical prose.Source: F DE-LEAKING: PROCEDURE · p. 28
• Questions are kept, trimmed, paraphrased, or dropped in that preference order according to whether they state removed facts and remain answerable from the de-leaked input.After trimming, definite references and temporal anchors are checked against the original referents.Source: F DE-LEAKING: PROCEDURE · pp. 28–29
• A second reading classifies edits and sampled untouched items, repairs inferred negatives and unsupported or altered facts, and keeps random background-rate estimates separate from other audit populations.A cross-check compares all three input fields for lingering removed facts and manually re-anchors cases when the summary was tied to the wrong decision.Source: F DE-LEAKING: PROCEDURE · p. 29
• Under the deletion-only constraint, no edit introduced a content word absent from the input, while the first pass tended to rewrite sections.Second-reading faults were mostly within-case inconsistencies or trims that re-bound references; every judgment in the pass was a model judgment.Source: F DE-LEAKING: PROCEDURE · p. 29