PaperlayerView exact revision on arXiv

Source-linked AI summary

Pathology-CoT: Learning Visual Chain-of-Thought Agent from Expert Whole Slide Image Diagnosis Behavior

Sheng Wang, Ruiming Wu, Charles Herndon, Yihang Liu, Shunsuke Koga, Jeanne Shen, Zhi Huang

arXiv:2510.04587v2cs.CV

TL;DR

Whole-slide pathology agents need scalable supervision that captures how experts navigate slides and why they inspect particular regions. Pathology-CoT turns viewer logs into expert-validated behavioral data for Pathology-o3; the agent achieved 100% recall and retained 97.6% recall on an independent external cohort.

  • Problem

    Pathology agent training lacks scalable, clinically aligned supervision of expert viewing behavior, while raw navigation logs are noisy and require curation before use.

    Source: §1 · p. 3, §2.1 · p. 3

  • Method

    Pathology-CoT converts expert navigation into ROI-linked commands and AI-drafted rationales that pathologists review, then trains Pathology-o3 on the resulting behavioral data.

    Source: §1 · p. 3, §2.1 · p. 4

  • Results

    Pathology-o3 achieved 84.5% precision, 100% recall, and 75.4% accuracy versus OpenAI o3’s 46.7% precision, 87.5% recall, and 57.8% accuracy, with 97.6% recall on an independent external cohort.

    Source: §1 · p. 3, §2.2 · p. 6

  • Takeaways & Limitations

    The framework supplies task-conditioned expert behavior paired with expert-verified reasoning as supervision for human-aligned pathology agents.

    Source: §1 · p. 3

    Findings on viewing-policy robustness concern metastasis detection and may not generalize to tasks requiring systematic sequential evaluation.

    Source: §3 · p. 11

Abstract

from arXiv · show

Diagnosing a whole-slide image is an interactive, multi-stage process of changing magnification and moving between fields. Although recent pathology foundation models demonstrated superior performances, practical agentic systems that decide what field to examine next, adjust magnification, and deliver explainable diagnoses are still lacking. Such limitation is largely bottlenecked by data: scalable, clinically aligned supervision of expert viewing behavior that is tacit and experience-based, not documented in textbooks or internet, and therefore absent from LLM training. Here we introduce a framework designed to address this challenge through three key breakthroughs. First, the AI Session Recorder seamlessly integrates with standard whole-slide image viewers to unobtrusively record routine navigation and convert the viewer logs into standardized behavioral commands and bounding boxes. Second, a lightweight human-in-the-loop review turns AI-drafted rationales for behavioral commands into the Pathology-CoT dataset, a form of paired "where to look" and "why it matters", enabling six-fold faster labeling compared to manual constructing such Chain-of-Thought dataset. Using this behavioral data, we build Pathology-o3, a two-stage agent that first proposes important ROIs and then performs behavior-guided reasoning. On the gastrointestinal lymph-node metastasis detection task, our method achieved 100 recall on the internal validation from Stanford Medicine and 97.6 recall on an independent external validation from Sweden, exceeding the state-of-the-art OpenAI o3 model and generalizing across backbones. To our knowledge, Pathology-CoT constitutes one of the first behavior-grounded agentic systems in pathology. Turning everyday viewer logs into scalable, expert-validated supervision, our framework makes agentic pathology practical and establishes a path to human-aligned, upgradeable clinical AI.

1 Introduction

Whole-slide diagnosis is a dynamic, multi-step process, but pathology agents lack scalable, clinically aligned supervision of how experts navigate and why regions matter. Pathology-CoT turns routine viewing logs into expert-validated behavioral data and uses it to train Pathology-o3.

1 Introduction

  • Pathology agents suit WSI diagnosis because it requires iterative zooming, panning, and region selection rather than static classification.The paper frames such agents as potential collaborative AI partners for pathologists.Source: §1 · p. 1
  • Existing image-caption and WSI-report datasets typically omit pathologist viewing behavior, while noisy interaction logs need careful curation before serving as supervision.The paper identifies direct expert involvement as necessary, making dataset construction time-consuming and difficult to scale.Source: §1 · p. 3
  • Pathology-o3 achieved 84.5% precision, 100% recall, and 75.4% accuracy versus OpenAI o3’s 46.7% precision, 87.5% recall, and 57.8% accuracy, with 97.6% recall on an independent external cohort.The framework also reports a six-fold improvement in labeling efficiency over manual annotation from scratch.Source: Figure 1 · p. 2, §1 · p. 3
  • The AI Session Recorder converts continuous viewing logs into discrete commands and ROI-linked rationales that pathologists verify or correct, forming the Pathology-CoT dataset.The dataset contains 5,222 conversational rounds from pathologists with diverse experience levels.Source: Figure 1 · p. 2, §1 · p. 3
  • The study targets the bottleneck of obtaining clinically meaningful, scalable training data by pairing routine viewing logs with low-cost, expert-verified reasoning.The authors position this as a step toward behavior-grounded, clinically trustworthy AI systems in computational pathology.Source: §1 · p. 3

2 Results

The results introduce a workflow and dataset for turning pathologists’ slide-viewing behavior into agent-ready supervision, then evaluate a behavior-guided agent on lymph-node metastasis detection. The agent performs strongly on internal and external validation, while additional analyses examine ROI selection, evidence ordering, and scalability.

2.1 A scalable data engine converts viewing behavior into agent-ready supervision

  • The AI Session Recorder converts noisy viewing logs into <inspect> and <peek> commands with ROI boxes, then supports pathologist review of AI-drafted rationales.The verify-and-edit workflow averaged 16.3–18.8 seconds per round, versus 56.7–64.9 seconds for dictation and 85.2–106.2 seconds for typing from scratch.Source: §2.1 · pp. 3–4
  • Pathology-CoT contains 5,222 conversational training rounds from 921 sessions and eight pathologists, pairing expert-validated behavior with reasoning.The dataset spans attendings, fellows, and residents and includes rationales for broad <inspect> and high-magnification <peek> actions.Source: §2.1 · p. 4

2.2 Behavior-guided supervision enables accurate pathology agents

  • 84.5% precision, 100% recall, and 75.4% accuracy for Pathology-o3 exceeded OpenAI o3’s 46.7% precision, 87.5% recall, and 57.8% accuracy on the internal benchmark.Figure 2 also depicts the agent’s ROI-level reasoning and case-level diagnosis.Source: Figure 2 · p. 5, §2.2 · p. 6
  • Pathology-o3 first predicts task-specific diagnostic ROIs from Pathology-CoT, then prompts a VLM for per-region reasoning and synthesizes a case-level summary.The resulting visual chain-of-thought is presented as stepwise descriptions of inspected regions.Source: Figure 2 · p. 5, §2.2 · p. 6

2.3 Pathology-o3 and learned behavior can generalize on external validation dataset

  • On external LNCO2 validation, Pathology-o3 achieved 97.6% recall, 62.9% precision, and 69.4% accuracy despite scanner and magnification shifts.The cohort comprised 321 whole-slide images from Sweden.Source: §2.3 · p. 6, Figure 3 · p. 7

2.4 Pathology-o3 learns where to look

  • Performance changed minimally when ROI evidence was presented forward, in reverse, or randomly, and improved as the maximum number of inspected ROIs increased.Unguided VLMs selected scattered, low-yield regions and performed poorly on completeness and efficiency against a senior pathologist’s behavior.Source: §2.4 · pp. 6, 9, Figure 4 · p. 8

2.5 Practicality, scalability, and the value of learned behavior

  • Behavioral guidance increased precision by 11.8% and recall by 17.9% over the non-agent baseline; the modular framework can also incorporate upgraded VLMs.Guidance from real expert behavior increased precision by 11.9% and recall by 19.1%.Source: §2.5 · p. 9

3 Discussion

Pathology agents face an analysis–navigation gap because expert viewing behavior is tacit and raw viewer logs are noisy. The discussion presents behavior-aligned supervision as a model-agnostic route to human-aligned navigation, while noting limits in task scope and data integration.

3 Discussion

  • Static-image training supports analysis of selected regions, but VLMs lack the interactive data needed to learn how to navigate slides and find them.The discussion frames this as an analysis–navigation gap.Source: §3 · p. 9
  • The learned viewing policy improves diagnostic accuracy across VLM backbones and generalizes to a different hospital system, while unguided leading VLMs fail at navigation.The authors also report robustness to variations in viewing habits.Source: §3 · p. 9
  • The learned viewing policy can be distilled into a compact, efficient model that provides nearly all the benefits of real expert guidance.Source: §3 · p. 11
  • Unlike PathChat+, which delegates where and why to zoom to a VLM supervisor, this approach learns expert navigation paired with pathologist-verified rationales.The authors state that this provides explicit supervision for both location and rationale.Source: §3 · p. 11
  • The reported viewing-policy findings concern metastasis search-and-find; they may not generalize to sequential tasks where the viewing path is critical.The discussion also identifies potential AI anchoring in rationale review and engineering effort to integrate heterogeneous viewer logs.Source: §3 · p. 11
  • The AI Session Recorder supports a growing public dataset of expert viewing behavior and offers a blueprint for procedural supervision in other clinical domains.Source: §3 · p. 11

4 Method

The method converts pathologists’ navigation into structured, expert-reviewed behavior-and-reasoning data, then trains a task-specific agent to propose ROIs and analyze them with a VLM. Evaluation spans model comparisons, external validation, behavioral analyses, and practicality.

4.1 Data Collection: Pathologist Behavior

  • Eight pathologists contributed 10.6 hours of interaction data from 137 WSIs across 25 colorectal-cancer lymph-node cases.The cohort included four attendings, two fellows, and two residents.Source: §4.1 · p. 12

4.2 AI Session Recorder: Discretizing Behavior to Extract ROIs

  • The recorder segments noisy viewport logs into inspect and peek actions, filters and merges redundant regions, then standardizes magnification and ROI boxes.The resulting action sequence supports both agent training and rationale generation.Source: §4.2 · p. 12

4.3 AI Session Recorder: Rationale Generation

  • For each ROI, a VLM drafts why-to-focus rationales and findings for pathologist review, and this review-over-authoring workflow was 8.6 times faster than manual annotation.Pathologists can accept, edit, or reject the AI-drafted text.Source: §4.3 · pp. 12–13

4.4 The Pathology-CoT Dataset

  • Pathology-CoT contains 921 sessions and 5,222 conversational rounds pairing standardized behavior commands and ROIs with expert-validated reasoning.Descriptions average 152 words for inspect actions and 82 words for peek actions’ cellular findings.Source: §4.4 · p. 13

4.5 Pathology-o3 Agent Framework and Evaluation

  • Pathology-o3 uses a behavior predictor to propose candidate ROIs, then a reasoning VLM analyzes each region and synthesizes a diagnostic summary.The reasoning module does not rationalize ROI selection; that role belongs to the behavior predictor.Source: §4.5 · p. 13

4.5.1 Statistic Analysis

  • Across internal and external validation, Pathologist-o3 significantly outperformed the second-best model on accuracy, precision, and recall (p<0.05).Significance was assessed with paired Student’s t-tests on bootstrap performance differences.Source: §4.5.1 · p. 14

4.6 The Pathology-o3 Agent: Behavior Predictor

  • The task-specific Behavior Predictor proposes diagnostically relevant ROIs for identifying lymph node metastasis in colorectal cancer; predicted boxes count as hits when IoU with an expert-visited region exceeds 0.3, or one box contains the other.Behavior Efficiency is hits divided by model-predicted regions; Behavior Completeness is hits divided by expert-visited regions in ground truth.Source: §4.6 · pp. 14–16

4.7.2 Comparison to State-of-the-Art Models

  • The evaluation compares guided diagnosis with unguided VLMs, tests external generalization and navigation behavior, and assesses cost and latency across VLM backbones.The external test uses LNCO2 without fine-tuning; behavioral analyses vary viewing order, evidence quantity, and unguided ROI selection.Source: §4.7.2 · p. 16, §4.7.3 · p. 16, §4.7.4 · p. 17

A.1 The AI Session Recorder Behavior Analyze Pipeline

The appendix describes a step-by-step pipeline that converts noisy pathologist interaction logs into normalized action bounding boxes.

A.1 The AI Session Recorder Behavior Analyze Pipeline

  • The Behavior Analyze pipeline preprocesses raw pathologist interaction logs into a structured, VLM-friendly format.Its implementation details correspond to the provided code.Source: §A.1 · p. 20
  • The pipeline transforms noisy, overlapping viewport data into a clean, normalized set of action bounding boxes.Figure S1 summarizes this transformation.Source: §A.1 · p. 20
  • The appendix gives a step-by-step technical description of the preprocessing logic.The passage identifies the corresponding figure and code as guides to the pipeline.Source: §A.1 · p. 20

A.1.1 Timing study design and fairness adjustment

The timing study compares AI-draft verification and revision with writing rationales from scratch on fixed ROIs, adjusting manual times for the navigation those fixed ROIs omit.

A.1.1 Timing study design and fairness adjustment

  • The study timed two users reviewing AI-drafted text across 10 cases and 78 conversational rounds, recording time from ROI load to decision.It reports verify, revise, and across-round average times.Source: §A.1.1 · p. 20
  • Manual baselines required a diagnosis and rationale covering abnormal features and normal structures, matching the chain-of-thought supervision task.Users could type or dictate the text from scratch.Source: §A.1.1 · p. 20
  • Because fixed ROIs remove reviewers’ navigation effort, manual-baseline times include the source expert’s recorded navigation time to each ROI.This adjustment is described as a conservative, apples-to-apples comparison with verify/edit, which requires no new navigation.Source: §A.1.1 · p. 20
  • AI drafts were longer than scratch-written text, and the authors state they were intentionally comprehensive to minimize missed details during review.Typing produced shorter text than dictation, and both were shorter than the AI drafts.Source: §A.1.1 · p. 20

A.1.2 Reviewer interface design

The reviewer interface places slide and ROI imagery beside editable AI-drafted impressions and findings, with sentence-level controls and explicit accept-or-reject decisions.

A.1.2 Reviewer interface design

  • The interface shows a slide thumbnail with the selected ROI, a zoomed view, an optional cytology crop, and editable text panels.The panels are Thumbnail Impression, Why Zoom, and Findings (ROI + Cyto).Source: §A.1.2 · p. 20, Figure S3 · p. 22
  • Drafts are split into individual sentences, each with a one-click delete control, so reviewers can remove a specific overclaim without rewriting the rest.Typical drafts contain five to seven sentences.Source: §A.1.2 · p. 20
  • Reviewers can accept or edit a draft, reject an ROI when zooming is unnecessary, and advance to record the decision and timing metadata.The header displays case, ROI, progress, elapsed time, and decision state.Source: §A.1.2 · p. 20

A.1.3 Step 1: Initial Action Segmentation

Initial action segmentation converts timestamped viewport events into meaningful low-level inspection actions. Each action receives a bounding box to define the inspected region.

A.1.3 Step 1: Initial Action Segmentation

  • The segmentation step converts high-frequency viewport events into an initial set of meaningful low-level actions.It uses timestamped zoom events and predefined temporal thresholds.Source: §A.1.3 · p. 25
  • A static Stay Inspect is identified when dwell time in one viewport exceeds 1.0s.Source: §A.1.3 · p. 25
  • A Pan Inspect is identified when continuous pan events last more than 2.0s cumulatively.Source: §A.1.3 · p. 25
  • Each Stay Inspect receives a viewport-centered box, while each Pan Inspect receives a minimal box encompassing its sequence of viewports.The resulting initial VLMAction objects are shown in Figure S1's “Stay & Pan” column.Source: §A.1.3 · p. 25

A.1.4 Step 2: Filtering Overview Actions

Overview actions that cover large portions of a whole-slide image are removed to focus the action list on specific inspection regions.

A.1.4 Step 2: Filtering Overview Actions

  • The filter removes non-diagnostic, low-magnification overview actions that do not represent detailed inspection.Source: §A.1.4 · p. 25
  • A VLMAction is filtered when its bounding-box width exceeds a preset fraction of WSI height, such as 40% (2/5).This removes broad “birds-eye view” actions while retaining genuine inspection events.Source: §A.1.4 · p. 25

A.1.5 Step 3: Merging Overlapping Actions

Overlapping actions are consolidated when they are likely to represent one cognitive task of inspecting a larger region.

A.1.5 Step 3: Merging Overlapping Actions

  • Actions are merged when their bounding-box Intersection over Union exceeds a threshold, such as 0.5–0.8.Source: §A.1.5 · p. 25
  • A merged action uses the union of the original boxes and spans the interval from the first action's start to the second action's end.Merging repeats until no further pairs meet the criterion.Source: §A.1.5 · p. 25

A.1.6 Step 4: Pruning Redundant Containing Actions

The appendix describes successive action refinement and normalization, the VLM prompts and label-structuring procedure, and model-identification details intended to support reproducibility.

A.1.6 Step 4: Pruning Redundant Containing Actions

  • A larger action is removed when a smaller action is more than 90% contained within it, prioritizing the more detailed inspection.This handles cases such as zooming from a lymph node to a tumor-cell cluster.Source: §A.1.6 · p. 25

A.1.7 Step 5: Normalizing Action Bounding Boxes

  • Remaining boxes are standardized by area into 5x or 10x views, centered on the pruned box and assigned a matching magnification bin.The normalized boxes are intended as structured input for generating training ground truth.Source: §A.1.7 · p. 25

A.2 Prompts for VLM Analysis

  • The overview prompt asks for an initial whole-slide impression, while ROI analysis uses a maximum-magnification crop to assess fine cytological features.The ROI prompt cautions that intercellular spaces may be processing artifacts and distinguishes tumor deposits from positive lymph nodes.Source: §A.2 · p. 26
  • The final diagnostic prompt asks the VLM to consider tissue architecture, findings from each region, and correlations between regions.Source: §A.2 · p. 26
  • The final impression prompt says not to treat a suspicious region as positive and specifies structured output fields for specimen type, stage, positive nodes, and regions.It also requests recommendations, diagnostic information, and suspicious-region listings.Source: §A.2 · p. 26

A.3 Method for Structuring Semi-Automated Thinking Labels

  • A single LLM call classifies each thinking label into multi-label regional and cellular categories, returning two pipe-separated lists for parsing.The taxonomy covers tumor, normal, and reactive or pathological features at low magnification and cellular features at 40x.Source: §A.3 · pp. 26–27
  • The classification prompt lists box-level and 40x classes and requests all applicable labels, using “other” when none fit.Source: §A.3 · p. 27

A.4 Specification of Evaluated Vision-Language Models

  • For closed-source API models that may change without public versioning, the authors document access timeframes; versioned or open-source models are identified by model identifier.Table S5 is described as specifying the evaluated VLM backbones.Source: §A.4 · p. 27, Table S5 · p. 27
Loading 2510.04587v2…