PaperlayerView exact revision on arXiv

Source-linked AI summary

A co-evolving agentic AI system for medical imaging analysis

Songhao Li, Jonathan Xu, Tiancheng Bao, Yuxuan Liu, Yuchen Liu, Yihang Liu, Lilin Wang, Wenhui Lei, Sheng Wang, Yinuo Xu, Yan Cui, Jialu Yao, Shunsuke Koga, Zhi Huang

arXiv:2509.20279v1cs.CVq-bio.QM

TL;DR

Medical image analysis lacks adaptable, transparent workflows and real-time expert feedback for diverse clinical and research questions. TissueLab combines modular tools, clinician feedback, and visualizable intermediate results in a co-evolving ecosystem. The paper reports rapid clinician-aligned adaptation and interpretable workflows, while noting that clinician-in-the-loop safety requires supervision.

  • Problem

    Medical image analyses often require expertise-intensive, slow-to-adapt pipelines, limiting scalable question-driven research workflows.

    Source: §1 · p. 2

  • Method

    TissueLab orchestrates modular imaging tools and uses clinician feedback, editable memory, and visualizable intermediate results to refine analyses.

    Source: §1 · p. 2, §2.1 · p. 3

  • Results

    TissueLab can adapt to new diseases with minutes of annotation, and its intermediate data are visualizable for clinician inspection.

    Source: §3 · p. 19

  • Takeaways & Limitations

    The open, modular ecosystem is designed to support clinicians and researchers integrating imaging AI methods into research and clinical workflows.

    Source: §3 · p. 19

    Clinician-in-the-loop co-evolution supports human-aligned safety but increases the level of human supervision required.

    Source: §3 · p. 19

Abstract

from arXiv · show

Agentic AI is rapidly advancing in healthcare and biomedical research. However, in medical image analysis, their performance and adoption remain limited due to the lack of a robust ecosystem, insufficient toolsets, and the absence of real-time interactive expert feedback. Here we present "TissueLab", a co-evolving agentic AI system that allows researchers to ask direct questions, automatically plan and generate explainable workflows, and conduct real-time analyses where experts can visualize intermediate results and refine them. TissueLab integrates tool factories across pathology, radiology, and spatial omics domains. By standardizing inputs, outputs, and capabilities of diverse tools, the system determines when and how to invoke them to address research and clinical questions. Across diverse tasks with clinically meaningful quantifications that inform staging, prognosis, and treatment planning, TissueLab achieves state-of-the-art performance compared with end-to-end vision-language models (VLMs) and other agentic AI systems such as GPT-5. Moreover, TissueLab continuously learns from clinicians, evolving toward improved classifiers and more effective decision strategies. With active learning, it delivers accurate results in unseen disease contexts within minutes, without requiring massive datasets or prolonged retraining. Released as a sustainable open-source ecosystem, TissueLab aims to accelerate computational research and translational adoption in medical imaging while establishing a foundation for the next generation of medical AI.

1 Introduction

Medical image analysis depends on costly, expertise-intensive pipelines that are difficult to adapt, while complex diseases require customized workflows and current agentic systems have limited flexibility and expert input. TissueLab is presented as a co-evolving, open ecosystem for transparent, clinically meaningful analyses.

1 Introduction

  • Complex diseases require customized workflows, and no single model reliably addresses the full spectrum of clinical imaging needs.VLMs may lack transparency and hallucinate, while trustworthy analyses must adapt to unseen scenarios without manual workflow design or extensive retraining.Source: §1 · p. 2
  • Existing agents often rely on fixed toolboxes and lack dynamic expert intervention, making them costly to maintain and poorly suited to unseen clinical cases.The paper also identifies hallucinations, token overload, attention dilution, and failure to retain expert knowledge as limitations.Source: §1 · p. 2
  • TissueLab combines modular tools, clinician feedback, editable memory, guideline retrieval, and visualizable intermediate states in a co-evolving system.Feedback becomes labeled data for lightweight fine-tuning; external clinical standards and traceable references ground diagnoses.Source: §1 · pp. 2–3
  • The system is demonstrated across pathology, radiology, and spatial omics, and is released as open-source software for multiple operating systems.The authors position the shared ecosystem as a way for researchers and clinicians to contribute models, annotations, and workflows.Source: §1 · p. 3

2.1 Creating a co-evolving agentic AI system for medical imaging

TissueLab turns user queries into tool-based workflows, then lets experts inspect intermediate results and refine analyses. Its modular ecosystem supports new tools, collaborative reuse, and faster workflow iteration.

2.1 Creating a co-evolving agentic AI system for medical imaging

  • The system abstracts imaging problems into operations such as tissue segmentation, classification, and local code analysis, without requiring users to code.Its plugin architecture allows new tools to be integrated without modifying the codebase.Source: §2.1 · p. 3
  • The query-driven workflow invokes available tool factories, stores intermediate results locally, checks model dependencies, and topologically sorts tasks for distributed inference.A summary agent analyzes stored results and returns a natural-language answer.Source: §2.1 · p. 3
  • Experts can inspect intermediate results, annotate, iteratively improve tools, reselect models, and contribute or reuse ecosystem knowledge.Users can also provide feedback through the chat box to adjust or improve an analysis module.Source: §2.1 · p. 3
  • TissueLab autonomously generates and executes workflows within 60 minutes, with expert-feedback refinement in 10 seconds per iteration.The contrast is with traditional workflows that often took 3–6 months from initial question to expert-derived results.Source: Figure 1 · p. 4, §2.1 · p. 5
  • Evaluations span relational reasoning, diagnosis informed by medical knowledge, high-dimensional data, co-evolution with experts, and multimodal analysis.The overview depicts orchestration, modular plugins, editable memory, scheduled task nodes, and a natural-language summary agent.Source: Figure 1 · p. 4, §2.1 · p. 5

2.2 Co-evolving agentic AI accelerates expert-level assessment of tissue measurement

TissueLab addresses labor-intensive tissue measurements with generated workflows and clinician annotations. In the reported lymph-node and depth-of-invasion tasks, it achieves strong results while exposing interpretable intermediate analyses.

2.2 Co-evolving agentic AI accelerates expert-level assessment of tissue measurement

  • The study evaluates DoI in 105 primary tumor resections from 49 patients and lymph-node counts across 321 adjacent slides from 22 patients.The DoI dataset is LNCO2; the lymph-node task concerns metastatic nodes per slide.Source: §2.2 · pp. 5, 7
  • Pearson ρ = 0.843 versus GPT-4o-agent’s 0.376; DoI MAE = 2.047 mm and RMSE = 3.091 mm, with TLAgent completing 100% of tasks.The comparisons used large-thumbnail inputs for baselines unable to process whole-slide images and repeated each experiment five times.Source: §2.2 · p. 5, Figure 2 · p. 6
  • For DoI, TissueLab segments tumor tissue and computes the maximum shortest distance between tumor-boundary points and the epithelial surface contour.The generated workflow uses domain-specific models, contour extraction, and pixel-size metadata to produce the measurement.Source: §2.2 · pp. 6–7
  • 0.919 accuracy and 0.926 weighted F1 for TLAgent exceeded GPT-5-vision’s 0.563 accuracy and 0.653 weighted F1 in positive lymph-node counting.TLAgent also achieved a 100% task-completion rate; Figure 2 presents the accuracy, weighted F1, and success-rate comparisons.Source: Figure 2 · p. 6, §2.2 · p. 7
  • TissueLab’s visualized segmentation and intermediate results let users inspect tissue regions and their correspondence with expert labels.The paper describes other model outputs as random or lacking consistent patterns between predicted and true invasion depth.Source: §2.2 · pp. 7–8

2.3 TissueLab enables guideline-aligned diagnosis using an effectively infinite medical knowledge database

TissueLab retrieves current authoritative guidelines for diagnostic reasoning, and its lymph-node and chest X-ray evaluations report strong results alongside clinician-guided adaptation. The evaluations cover staging-critical metastasis classification and chest X-ray prediction across 14 thoracic conditions.

2.3 TissueLab enables guideline-aligned diagnosis using an effectively infinite medical knowledge database

  • TissueLab agents retrieve current guidance from sources including AJCC, CAP, WHO, and AASLD through MCP and retrieval-augmented generation.The authors describe this as a continuously updated knowledge base anchored to authoritative clinical sources.Source: Leveraging online searching for… · p. 8

Assessment of lymph node metastasis using online diagnostic criteria

  • Weighted F1-score of 0.939 and accuracy of 0.931 closely matched expert-level lymph-node metastasis diagnosis.TissueLab outperformed all baselines across recall, precision, F1-score, accuracy, and correlation.Source: Assessment of lymph node metastasis… · p. 8
  • GPT-5-vision reached F1 = 0.468 and Accuracy = 0.388, while other listed VLM baselines also scored poorly.The authors attribute baseline weakness to lack of access to and interpretation of current external criteria.Source: Assessment of lymph node metastasis… · p. 9

Assessment of chest X-ray diagnosis using online guideline and human feedback

2.4 TissueLab enables high-dimensional radiology image analysis

TissueLab applies guideline-based workflows to high-dimensional radiology, including 3D CT and 4D cardiac imaging. Across fatty liver, intracranial hemorrhage, and hypertrophy tasks, TLAgent outperformed the reported baselines, with transparent intermediate outputs.

TLAgent predict fatty liver in 3D chest CT

  • For fatty liver, TLAgent segmented liver and spleen volumes, computed Hounsfield-unit values, and evaluated them against retrieved guideline thresholds.The evaluation used 38 held-out chest CT patients; baseline models received manually supplied 2D slices, and experiments were repeated five times.Source: TLAgent predict fatty liver in 3D… · pp. 11–13
  • 0.848 accuracy and 0.870 F1-score for fatty liver diagnosis, with MCC = 0.714 and Cohen’s kappa = 0.690.Baselines sometimes achieved comparable F1-scores, but their agreement-based metrics were nearly zero; M3D and GPT-4o-vision defaulted to predicting the positive class.Source: TLAgent predict fatty liver in 3D… · pp. 11, 13, Figure 4 · p. 12

TLAgent improves detection of intracranial hemorrhage from 3D CT scans

  • For intracranial hemorrhage, TLAgent combined volumetric hemorrhage segmentation and brain-structure delineation with code-based spatial-overlap analysis.The reported evaluation used 82 3D CT scans with voxel-level annotations and repeated experiments five times.Source: TLAgent improves detection of… · p. 13
  • 78.7% accuracy and 0.750 F1-score for intracranial hemorrhage, closely matching expert-level performance and exceeding the reported baselines.TLAgent combined numerical accuracy with clinically coherent predictions; GPT-5-vision and M3D could not complete the task.Source: TLAgent improves detection of… · p. 13

TLAgent enables dynamic detection of myocardial hypertrophy from 4D cardiac imaging

  • For hypertrophy, TLAgent derived endocardial and epicardial contours from segmentations, measured wall thickness, and aggregated measurements across frames.The workflow applied the stated diagnostic threshold of maximal left ventricular wall thickness ≥15 mm at end-diastole in the absence of secondary causes.Source: TLAgent enables dynamic detection of… · p. 14
  • 84.2% accuracy, F1 = 0.880, and MCC = 0.701 for myocardial hypertrophy, outperforming the reported baselines on diagnosis and correlation with ground-truth thickness.The workflow measured wall thickness across frames from cardiac segmentations; baselines were given manually sliced 2D inputs, and GPT-4o-agent predicted nearly all cases positive.Source: TLAgent enables dynamic detection of… · p. 14

2.5 TissueLab interactive ecosystem enables customized tool development

TissueLab lets users build and share customized tools and workflows, while clinicians iteratively refine classifiers through feedback. The reported cancer-cell tasks show rapid gains, although performance remains constrained by the underlying tools.

Tool limitations

  • TLAgent performance is bottlenecked by underlying tools such as segmentation models, rather than by agent orchestration alone.The passage notes that some research and clinical questions are edge cases for which no existing tool is available.Source: §2.5 · p. 14

Customized tool development and tumor-cell identification

  • Users can upload Python pipelines or fine-tune foundation models, then share the resulting tools through the TissueLab ecosystem.The evaluated cell-identification capability used Colon and Prostate slides from the Visium HD dataset.Source: §2.5 · p. 15
  • Tumor-cell identification improved from near-zero accuracy to 82.1% within five minutes and 94.9% after 30 minutes of incremental training.Tumor-region intersection-over-union increased from 10.1% initially to 88.9% at 30 minutes; baseline comparisons may not have had access to the same high-resolution data.Source: §2.5 · p. 15

Clinician-guided co-evolution

  • 99.8% accuracy was reached for tumor-to-normal duct cell ratio quantification with clinician-guided classifiers, with predicted spatial patterns closely matching ground truth.Accuracy rose from 3.9% to above 90% within two minutes of feedback and continued toward 99.8%; predicted ratios moved smoothly toward ground truth with minimal deviation.Source: Figure 5 · p. 16, §2.5 · p. 17

2.6 TissueLab enables multi-modal integration of spatial omics and histology to improve accuracy in pathology analysis

TLAgent combines spatial transcriptomics and histology to quantify glomeruli, a task traditionally assessed manually. With domain configuration, it approached expert annotation accuracy, while baseline vision-language models failed to complete the task.

Motivation

  • Glomerulus counting is important for nephrology research and clinical care, including advanced measures such as the fraction of globally sclerotic glomeruli.The passage notes that these assessments have traditionally relied on labor-intensive manual evaluation by expert pathologists.Source: §2.6 · p. 17

Multi-modal workflow

  • TLAgent linked cell coordinates from H&E segmentation to gene-expression profiles, clustered cells into transcriptomic domains, and mapped domains back to annotated histology regions.It identified glomerular cells, computed local cell proportions, aggregated glomerulus patches, and used graph traversal to count glomeruli.Source: §2.6 · p. 17

Results

  • 98.4% accuracy for glomerulus counting with domain configuration, nearly indistinguishable from expert annotations.Baseline vision-language models consistently failed to complete the task, producing random or absent outputs.Source: §2.6 · pp. 17, 19, Figure 6 · p. 18

Interpretation

  • The workflow’s overlays let clinicians inspect predictions against histology, and its integration of molecular and morphological information provides a multimodal analysis framework.The reported use case concerns a kidney whole-slide image from the Visium HD dataset.Source: §2.6 · p. 19

3 Discussion

TissueLab is presented as an open, modular system that links medical imaging AI research with translational practice through clinician-visible workflows and ongoing co-evolution. Its current human-supervised design has practical limits involving oversight, baseline image access, and clinical integration.

3 Discussion

  • The open-source ecosystem offers modular, reconfigurable workflows intended to lower entry barriers and accelerate discovery and translational adoption.TissueLab is available through installers for Windows, macOS, and Linux, and a free web portal.Source: §3 · p. 19
  • Clinician feedback can rapidly adapt models to new diseases with minutes of annotation, while shared contributions support progress across users.The authors describe this as especially relevant to settings involving rare diseases with limited training data.Source: §3 · p. 19
  • Clinicians can inspect and edit intermediate outputs, while an evolving memory layer and guideline retrieval support transparent, traceable workflows.The system also supports integrating new models and keeps clinicians in control of the decision pathway.Source: §3 · p. 19
  • Current clinician-in-the-loop co-evolution reduces dependence on autonomous improvement but increases the human supervision required.The authors identify community-driven self-evolution under safety constraints as a future direction.Source: §3 · pp. 19–20
  • Some baseline comparisons may be inequitable because models lack access to TLAgent’s high-resolution data; routine clinical use also requires technical validation, regulatory compliance, and hospital interoperability.For example, GPT-5-vision receives only its largest acceptable thumbnail for pathology, whereas TLAgent processes raw whole-slide images.Source: §3 · p. 20

4 Methods

The methods make TissueLab extensible through modular task nodes, order dependent work with topological sorting, and co-evolve models and model-selection decisions using clinician feedback and community contributions.

Adaptive and extensible agentic system for seamless model integration

  • Any state-of-the-art model can be integrated as a task node without changing orchestration logic, and system capacity expands as models are added.Nodes expose standardized initialization, input-handling, and execution functions while keeping model internals hidden from the agent.Source: Adaptive and extensible agentic system… · p. 20

Topological sorting enables parallel and distributed inference

  • Topological sorting executes dependent tasks in order and independent branches in parallel, supporting scalable and distributed workflow execution.Pairwise dependencies, such as segmentation before classification, define the ordering without requiring the agent to reason over the full dependency graph.Source: Topological sorting enables parallel… · pp. 20–21

Community-driven co-evolution

  • Clinician annotations update lightweight models, while user evaluations guide rankings of task-relevant model candidates and adapted modules can be shared across the community.The authors state that lightweight models can be retrained within minutes using a small number of annotations; model selection is user-evaluated rather than autonomous.Source: Community-driven co-evolution · pp. 21–22

LLM orchestration

TissueLab separates LLM orchestration from local domain-specific image analysis and uses persistent, editable intermediate data to support transparent workflows and clinician feedback. Its tools and workflows span pathology, radiology, spatial omics, and retrieval of external clinical criteria.

LLM orchestration

  • The LLM plans workflows, generates code, and invokes tools but does not process raw images; local domain-specific models and pipelines perform analysis.The orchestration framework can use future LLMs without modification.Source: LLM orchestration · p. 22

Execution layer: interaction between agent and tool models

  • Tools operate as black-box task nodes that retrieve inputs through a standardized read() interface from shared HDF5 storage, decoupling them from upstream and downstream dependencies.The LLM ensures required inputs are stored before execution, and persistent intermediate outputs support reproducibility.Source: Execution layer: interaction between… · p. 22

Editable memory layer

  • Persistent memory stores intermediate results, annotations, and logs; users can inspect and refine them, with updates incorporated into subsequent executions.Because the memory resides outside token context, its capacity is described as limited chiefly by hardware storage.Source: Editable memory layer · p. 23

Semantic function-calling for data access

  • Semantic dataset names and metadata guide function calls that retrieve structured inputs and generate local analysis code, while new modalities can be added by naming datasets without changing orchestration.The data remain local, and the authors characterize this schema-aligned access as reproducible and auditable.Source: Semantic function-calling for data… · p. 23

MCP for searching external criteria

  • For tasks requiring clinical thresholds or criteria, TissueLab retrieves external guidance through MCP and incorporates traceable references into the workflow.MCP is used for external knowledge retrieval, not communication between local tool models.Source: MCP for searching external criteria · p. 23

Benchmarking TissueLab agentic AI system on diverse clinical imaging cohorts

  • Evaluation spans pathology, radiology, and spatial omics cohorts; experiments were repeated five times, with mean and standard deviation reported for comparisons.For spatial omics, only ground-truth comparisons were possible because existing VLM- and LLM-based agents could not align gene expression with H&E images or handle high-dimensional expression matrices beyond token limits.Source: Benchmarking TissueLab agentic AI… · p. 24

Tissue segmentation and classification

  • Pathology uses foundation-model embeddings for zero-shot analysis, then patch-level annotations can support lightweight model retraining, often within minutes.The ecosystem also integrates other researchers’ tissue models; radiology instead registers pretrained models as task nodes for dynamic selection. TotalSegmentator covers predefined structures, while BiomedParse supports user-specified targets.Source: Tissue segmentation and classification · p. 25
  • TissueLab dynamically selects domain-specific radiology models, combining a flexible orchestration framework with tools that offer different segmentation scopes.TotalSegmentator covers a predefined set of organs and regions, while BiomedParse supports query-driven segmentation of user-specified structures.Source: Tissue segmentation and classification · p. 25

Cell segmentation and classification

TissueLab combines nucleus segmentation and ontology-aware cell classification with spatial-omics clustering to connect molecular patterns to cellular phenotypes.

Cell segmentation and classification

  • StarDist provides nucleus instance segmentation, while NuClass supports zero-shot nuclei typing using biomedical ontologies, nucleus-centered patches, and pretraining on over 8.3 million nuclei patches from 11 datasets.Overlapping tiles are deduplicated to support accurate cell counts across whole-slide images.Source: Cell segmentation and classification · p. 25

Multi-modal analysis

  • TissueLab links spatial transcriptomic clusters from CellCharter to cell types from NuClass or user annotations, mapping molecular clusters to cellular phenotypes.The resulting analyses are described as supporting multi-scale insights into tissue organization and disease mechanisms.Source: Multi-modal analysis · p. 26

Evaluation metrics

TissueLab is evaluated with complementary metrics for task completion, predictive performance, robustness, and clinical relevance. The measures capture both diagnostic trade-offs and continuous-prediction errors that single metrics may miss.

Evaluation metrics

  • Task Completion Rate measures the share of tasks producing a concrete, clinically relevant answer, with success requiring an actionable response to the clinical question.The metric is intended to capture pragmatic utility and penalize hallucinated or superficial outputs.Source: Evaluation metrics · p. 26
  • For continuous predictions, MAE measures average absolute error, RMSE emphasizes large deviations, and Pearson r measures linear association with ground truth.Pearson r is invariant to scale and offset, so it assesses preservation of relative ordering rather than exact numerical estimates.Source: Evaluation metrics · pp. 26–27
  • Accuracy summarizes overall classification correctness, but can mislead under severe class imbalance; precision, recall, and F1 provide complementary views of diagnostic reliability.A majority-class prediction can yield deceptively high accuracy while failing to detect disease.Source: Evaluation metrics · p. 27
  • Weighted F1 averages per-class F1 by class sample counts, giving prevalent classes greater weight while still incorporating minority-class performance.The authors note that this can help represent rare but critical categories in heterogeneous patient populations.Source: Evaluation metrics · p. 28
  • AUC, MCC, and Cohen’s Kappa supplement classification scores by assessing threshold-independent discrimination, class-imbalance robustness, and agreement corrected for chance.AUC spans possible thresholds; MCC incorporates all four outcomes, and Kappa compares observed with chance-expected agreement.Source: Evaluation metrics · p. 28

Related papers

Loading 2509.20279v1…