PathFinder: A Multi-Modal Multi-Agent System for Medical Diagnostic Decision-Making Applied to Histopathology

1. Introduction

• Pathologists inspect suspicious regions at increasing magnification, gather evidence across patches, and combine it into a holistic diagnosis, a labor-intensive process requiring significant expertise.The paper describes this process as the gold standard and notes that it is becoming increasingly unsustainable as cancer cases rise globally.Source: §1 · p. 2
• Patch-based methods analyze WSI patches independently, while transformer-based models attempt to capture local and global patterns but are not scalable to WSI resolution demands.Source: §1 · p. 2
• PathFinder coordinates Triage, Navigation, Description, and Diagnosis Agents to classify slides, iteratively examine patches when slides are risky, and synthesize descriptions into a diagnostic classification.Source: §1 · p. 2
• 74% accuracy on the M-Path Skin Biopsy dataset was an 8% improvement over the best baseline (66%) and a 9% improvement over pathologists’ average performance (65%).The authors describe PathFinder as the first AI-based system to surpass the average performance of pathologists on this melanoma classification task.Source: §1 · p. 2

2. Related Work

• Histopathology multimodal models use image-text data and support patch-level tasks, while WSI-level approaches align diagnostic text with slides; the cited MLLMs do not automatically navigate gigapixel WSIs toward a diagnosis.Source: §2 · p. 2
• Navigation studies characterize pathologists’ panning and zooming, and NaviPath integrates human-AI collaboration into digital-slide workflows.Source: §2 · p. 2
• Multi-agent systems have been explored in robotics, role-playing, reasoning, gaming, software engineering, and medical-provider simulations; the paper motivates applying them to navigation, understanding, and holistic diagnosis in medical image analysis.Source: §2 · pp. 2–3

3. Datasets

• The M-Path Skin Biopsy dataset contains 238 H&E-stained melanocytic lesion specimens, each with a consensus diagnosis established by three dermatopathologists.Source: §3 · p. 3
• The dataset has four diagnostic classes with 35, 86, 70, and 47 cases, and is split into training, validation, and test sets of 168, 35, and 35 cases with consistent class distribution.Source: §3 · p. 3
• Viewport data came from 87 pathologists across 10 U.S. states; the current study included data from 32 pathologists who completed the M-Path study.The viewer recorded viewport coordinates, magnification levels, and timestamps during interpretation.Source: §3 · pp. 3–4

4. PathFinder

• PathFinder coordinates Triage, Navigation, Description, and Diagnosis Agents toward a whole-slide image diagnosis.The agents interact as a multi-agent, multi-modal framework.Source: §4 · p. 4

4.1. Triage Agent

• The Triage Agent uses an image-only transformer to separate class 1 lesions from the remaining M-Path classes.Its architecture processes patch embeddings with transformer blocks, positional encoding, multi-scale convolutions, and a class-token classifier.Source: §4.1 · p. 4

4.2. Navigation Agent

• The Navigation Agent iteratively predicts importance maps from masked WSIs, conditioning later maps on accumulated descriptions to select new regions of interest.A lightweight U-Net implements the text-conditioned map generation; selected patches are sent to the Description Agent.Source: §4.2 · pp. 4–5

4.3. Description Agent

• The Description Agent is an instruction-tuned Quilt-LLaVA model adapted to generate concise summaries of histopathology patches.Its tuning data comprises 102,000 instruction samples generated from Quilt-1M captions with GPT-4.Source: §4.3 · p. 5

4.4. Diagnosis Agent

• The Diagnosis Agent classifies cases from patch descriptions gathered along heatmap-weighted trajectories of sampled patches.Descriptions are combined across iterations, and a language model with a classification head produces class probabilities.Source: §4.4 · pp. 5–6
• Training uses 100,000 trajectories from 20,000 resampled cases, with shuffled sets of five to ten descriptions; GPT-2 was chosen given resource and dataset constraints.LLaMA 3.1 Instruct rephrases descriptions to add trajectory variability, simulating differing examination patterns.Source: §4.4 · p. 6

5.1. Pathologist Evaluation of Description Quality

• Pathologists rated descriptions from PathFinder, GPT-4o, and LLaVA-Med on 25 cases in a randomized, double-blind survey; PathFinder’s descriptions were comparable to GPT-4o and used a 7B-parameter model.Source: §5.1 · p. 6, Figure 4 · p. 7

5.2. PathFinder Evaluation

• Across 10 test runs, each WSI prediction used majority voting over five randomly selected trajectories, with overall performance reported as the mean across runs.Source: §5.2 · p. 7
• The 74% navigation-based approach outperformed the best non-iterative ABMIL selection at 54%; visual-only and imitated sampling reached 63% and 64%, while exhaustive search reached 68%.Source: §5.2 · p. 7
• With Quilt-LLaVA descriptions, the T5-based navigator reached 74% versus 62% for its CLIP-based counterpart; with LLaVA-Med descriptions, T5 showed no advantage over CLIP (56% versus 60%).Source: §5.2 · pp. 7–8
• Using majority voting over 10 runs for whole-slide-image diagnosis on the M-Path dataset, PathFinder achieved an 8% improvement over the best baseline approaches, ABMIL with CONCH and UNI2-h backbones.Source: §5.2 · p. 8, Table 1 · p. 8
• The tested configuration performed best with five trajectories and 10 patches per trajectory; without the Triage Agent, the best variant fell below baseline, possibly due to Quilt-LLaVA’s malignant-case training bias.Source: §5.2 · p. 8

• PathFinder combines four agents to emulate pathologists’ multi-scale, iterative WSI diagnosis and deliver natural-language explanations; the paper reports performance above state-of-the-art methods and average human experts in melanoma diagnosis.Source: §6 · p. 9
• The framework relies on pre-existing datasets and significant computation; navigation complexity and occasional Description Agent hallucinations may affect transparency and accuracy.Source: §6 · p. 9

1. Triage agent

• The Triage Agent is designed to assess whether a WSI is risky.The evaluation compares it with benchmark methods for Class 1 versus Non-Class 1 classification.Source: §1 · p. 12
• The reported comparison concerns Class 1 versus Non-Class 1 classification in the M-Path dataset.The passage states that the Triage Agent outperforms the baseline methods, without giving their scores.Source: §1 · p. 12
• Table 1 compares the Triage Agent with benchmark methods on Class 1 versus Non-Class 1 classification.Source: Table 1 · p. 12

2. VLM-based Navigation Agent

• The initial Navigator combined a U-Net image encoder with LLaMA-7B to reason over image latents and prior patch descriptions.It was trained to output grid coordinates identifying regions of interest.Source: §2 · p. 12, §2.1 · p. 12

2.1. Training Process and Architecture

• The approach trained the U-Net on M-Path, retained its encoder, and instruction-tuned the combined model to predict ROI grid coordinates from a WSI and prior observations.Source: §2.1 · p. 12

2.2. Limitations and Challenges with a LLaVA-based Navigator

• Limited navigation data coincided with overfitting: the model repeatedly selected central patches and failed to generalize to novel slide patterns.Source: §2.2 · p. 12

2.3. Architectural Pivot

• The authors pivoted to the complete U-Net and decoded attention maps for region sampling, conditioning them on Description Agent text for feedback between agents.They report this approach was more robust with limited training data.Source: §2.3 · p. 12

3. Description agent

• GPT-4 was prompted to extract short, concise histopathology findings from provided text for Description Agent fine-tuning.Source: §3 · p. 12
• The data-generation prompt and a sample of the resulting data are illustrated in Figure 2.Source: §3 · p. 12
• The fine-tuning data-generation process uses text findings as input and targets concise descriptions for the Description Agent.Source: §3 · p. 12

4.1. Qualitative Analysis of Descriptions Assessed by Pathologists

• Two pathologists assessed descriptions for regions from 25 M-Path WSIs, comparing PathFinder, GPT-4o, and LLaVA-Med in a double-blind, randomized survey.The survey asked which description best matched the image and why.Source: §4.1 · p. 13
• Figure 3 presents each pathologist’s ratings for PathFinder, LLaVA-Med, and GPT-4o, with subplots separated by pathologist and reasons for preference.Source: Figure 3 · p. 14
• Most preferences were based on description correctness, while none of the models were preferred for detail under the concise-description setup.The models were prompted to produce descriptions under 20 words.Source: §4.1 · pp. 13–14

4.2. Prompt used for pre-trained LLM experiments

• The pre-trained LLM diagnosis prompt asks the model to classify a WSI using descriptions from its patches and choose among four skin-cancer classes.Source: §4.2 · p. 14