PaperlayerView exact revision on arXiv

Source-linked AI summary

MorphoOrgaAgent: A Foundation-Model-Based Multi-Agent System for Autonomous Organoid Analysis

Hanyi Zhang, Maximilian Hoermann, Lion J. Gleiter, Yiling Xu, Bettina Katalin Budai, Hans-Ulrich Kauczor, Carsten Marr, Tingying Peng

arXiv:2609.08696v1cs.MAcs.CV

TL;DR

Organoid morphology analysis relies on labor-intensive segmentation and quantitative workflows that can require coding skills. MorphoOrgaAgent translates natural-language requests into zero-shot segmentation, measurements, visualizations, and reports; evaluations show it handles explicit and descriptive requests with measurements close to ground truth.

  • Problem

    Existing organoid analysis pipelines often rely on task- or dataset-specific training, while manual segmentation and coding-heavy quantitative analysis remain burdensome for experimental biologists.

    Source: §1 · pp. 2–3

  • Method

    MorphoOrgaAgent interprets natural-language requests, combines Cellpose geometric prompts with text prompts for SAM3 segmentation, computes requested metrics, and generates evidence-grounded reports.

    Source: §1 · p. 3, §2 · pp. 4–5

  • Results

    Clear and open prompts yield nearly identical error profiles across metrics and task categories; single-extremum Roundness median APE is 1.54% clear and 1.56% open.

    Source: §3.2 · p. 6

  • Takeaways & Limitations

    The framework is reported to streamline organoid analysis, reduce expert annotation costs, and support accessible, high-throughput bioimage intelligence.

    Source: §4 · p. 9

Abstract

from arXiv · show

Organoids are three-dimensional tissue models whose morphology provides important insights into tumor development, disease progression, and drug testing. Extracting these morphological features relies heavily on manual segmentation, which is time-consuming and labor-intensive. Furthermore, performing quantitative statistical analysis typically requires custom coding skills and a mathematical background, presenting a major barrier for experimental biologists. To address these challenges, we introduce MorphoOrgaAgent, a multi-agent framework that achieves zero-shot organoid segmentation, automated data analysis, and report generation based on natural language input. The framework consists mainly of three core components: a TaskUnderstandingAgent that identifies requested measurements and visualization types; a hybrid segmentation module that combines Cellpose-derived geometric prompts with text prompts to guide SAM3 for zero-shot organoid instance segmentation; and a ReportAgent that computes quantitative metrics and compiles them alongside generated visualizations into a structured report. We further introduce MorphoOrgaVQA, a benchmark designed for quantitative evaluation of agent systems in organoid morphology analysis. Experimental results demonstrate that MorphoOrgaAgent handles both explicit and descriptive user requests, produces measurements closely matching ground truth, and generates complete analysis reports without requiring manual programming. The complete source code and MorphoOrgaVQA benchmark are publicly available at https://github.com/peng-lab/MorphoOrgaAgent.

1 Introduction

Organoid morphology measurements are biologically informative, but many existing analysis pipelines rely on task- or dataset-specific training, limiting zero-shot generalization. MorphoOrgaAgent combines natural-language task planning, zero-shot segmentation, quantitative analysis, and report generation in one multi-agent workflow.

1 Introduction

  • Existing organoid pipelines provide automated detection, segmentation, and classification, but often depend on task- or dataset-specific training that limits zero-shot generalization.Morphological features and organoid counts provide readouts of growth, culture state, and experimental responses.Source: §1 · p. 2
  • Bioimage and biomedical agents automate diverse workflows, but a multi-agent framework combining zero-shot organoid segmentation, quantitative analysis, and report generation remained lacking.The cited examples include BioMedAgent, CellAgent, BioImage.IO Chatbot, Agentic-J [15], and Omega.Source: §1 · pp. 2–3
  • MorphoOrgaAgent translates natural-language requests into analysis plans, guides SAM3 with Cellpose-derived geometric and text prompts, and generates measurements, visualizations, and structured reports.The paper introduces MorphoOrgaVQA, built from three public organoid datasets, with deterministic ground-truth answers derived from expert-annotated masks.Source: §1 · p. 3

2 Methodology

MorphoOrgaAgent uses a shared state to coordinate request interpretation, segmentation, metric computation, and report generation. Its agents translate natural-language intent into analysis plans, combine geometric and text prompts for segmentation, and compile results into evidence-grounded reports.

System Overview: MorphoOrgaState and Multi-Agent Orchestration

  • MorphoOrgaState is a JSON-serializable shared state that stores the query, analysis plan, masks, metrics, and intermediate results, leaving an auditable execution trace.The ReportAgent consolidates the stored query, segmentation results, and computed metrics into a natural-language answer.Source: §2 · pp. 3–4

TaskUnderstandingAgent

  • The TaskUnderstandingAgent interprets user intent, selects supported metrics and visualizations, and reformulates ambiguous or imprecise requests into instructions for the ReportAgent.The resulting plan and refined prompt are saved in MorphoOrgaState.Source: §2 · p. 4

Hybrid-Prompt Segmentation & Metric Computation

  • Cellpose-derived masks and the text prompt “cell cluster” guide SAM3 because text prompts alone may not capture organoids and Cellpose may imprecisely delineate unfamiliar morphologies.The resulting masks feed predefined metric-computation and visualization functions, whose results are saved to MorphoOrgaState.Source: §2 · p. 5

ReportAgent

  • The ReportAgent answers the refined query using stored image, segmentation, and statistics, while being constrained to state information to mitigate hallucination.Its answer explicitly cites supporting evidence from MorphoOrgaState.Source: §2 · p. 5

3 Experiments

The experiments evaluate benchmark construction, numerical accuracy, and performance on a complex report-generation task. MorphoOrgaAgent handles explicit and implied metric requests, while the qualitative comparison highlights the roles of segmentation quality and domain-specific descriptors.

3.1 The MorphoOrgaVQA Benchmark

  • The benchmark has 16 clear or open questions covering morphology metrics and extremum or tertile-mean tasks, evaluated against expert-mask ground truth.It uses 69 images from OrganoID, OrgaExtractor, and OrgaSegment, yielding 1,104 question–answer pairs with deterministic answers.Source: §3.1 · pp. 5–6

3.2 Quantitative Evaluation of MorphoOrgaAgent

  • Roundness median APE is 1.54% clear and 1.56% open for single-extremum queries, and 2.48% clear and 2.51% open for tertile means; clear/open error profiles are nearly identical across metrics.Table 1 summarizes median APE by metric and question type; Area tertile-mean APE reaches 38.74% for clear prompts, while Roughness single-extremum APE is 46.02%.Source: §3.2 · p. 6, Table 1 · p. 7

3.3 Qualitative Comparison against LLM-Based Approaches

  • The qualitative task asks systems to select the 10% of organoids with roughest boundaries, report their median area, and visualize the findings; the comparison includes brightfield, ground truth, Omega, and hybrid masks.The authors report limited Omega generalization and higher-quality instance masks from their hybrid module.Source: §3.3 · p. 7, Figure 2 · p. 7
  • MorphoOrgaAgent reports a median area of 5,771.5 px versus 5,749 px ground truth, a 0.39% relative error, compared with 1,235 px for Agentic-J and 761 px for Omega.Agentic-J used the same predicted mask but defaulted to convexity rather than domain-specific boundary descriptors; Omega’s estimate was primarily affected by segmentation failures.Source: Figure 3 · p. 8, §3.3 · p. 8

4 Conclusion

MorphoOrgaAgent converts biologists’ natural-language requests into executable organoid morphology workflows using hybrid-prompt segmentation and automated analysis. Its evaluation supports handling explicit and ambiguous requests without manual programming, with the stated aim of streamlining analysis and reducing annotation costs.

4 Conclusion

  • MorphoOrgaAgent translates complex natural-language queries into machine-executable workflows for organoid morphology analysis.The system couples Cellpose geometric prompts with domain-tailored text prompts for SAM3, then computes metrics, generates visualizations, and synthesizes reports.Source: §4 · p. 9
  • The MorphoOrgaVQA evaluation supports effective interpretation of explicit and ambiguous requests without manual programming.The conclusion describes the framework as streamlining workflows, reducing expert annotation costs, and supporting accessible, high-throughput bioimage intelligence.Source: §4 · p. 9
  • The authors report no competing interests relevant to the paper.Source: §4 · p. 9

A MorphoOrgaVQA Benchmark

Table 2 presents 16 MorphoOrgaVQA questions organized by morphological metric, query type, phrasing mode, and evaluated target fields.

  • Table 2 organizes 16 benchmark questions by metric, single-extremum versus tertile-mean query type, clear versus open phrasing, and target fields.The table contains the complete question set.Source: A MorphoOrgaVQA Benchmark · p. 11, Table 2 · p. 11
  • Area single-extremum questions request the maximum-area organoid and its area and centroid, phrased either directly or as the dominant organoid’s footprint and center of mass.Source: A MorphoOrgaVQA Benchmark · p. 11
  • Area tertile-mean questions ask for mean outer area across small, medium, and large groups, using either explicit area distributions or size heterogeneity wording.Source: A MorphoOrgaVQA Benchmark · p. 11
  • Perimeter single-extremum questions seek the organoid with the longest outer boundary, using either explicit perimeter terminology or descriptive boundary-length wording.Source: A MorphoOrgaVQA Benchmark · p. 11
  • Perimeter tertile-mean questions request average outer perimeter across three groups, including an open version that asks for mean perimeter by edge-length tiers.Source: A MorphoOrgaVQA Benchmark · pp. 11–12
  • Roundness questions cover both the maximum circularity score and average scores across three circularity or shape classes, in clear and descriptive phrasings.Source: A MorphoOrgaVQA Benchmark · p. 12
  • Roughness questions ask for either the organoid with the greatest perimeter difference or mean perimeter difference across tiers, with open prompts describing folded margins or deviation.Source: A MorphoOrgaVQA Benchmark · p. 12

B System Prompts of the LLM Agents

The appendix documents the LLM agents’ prompts, constraining task planning to supported metrics and visualizations and grounding reports in stored analysis results.

B System Prompts of the LLM Agents

B.1 TaskUnderstandingAgent

  • The agent must emit a strict AnalysisIntent JSON object limited to predefined metric and visualization pools, making the plan machine-checkable.Source: B System Prompts of the LLM Agents · p. 13
  • Segmentation is treated as always required, so the TaskUnderstandingAgent chooses only quantitative metrics and visualization categories for the execution plan.Source: §B.1 · p. 13
  • The metric pool includes object identifiers, outer and inner geometry, centroid coordinates, and border status, alongside derived measures such as perimeter difference, roundness, and lumen ratio.Source: §B.1 · p. 13
  • Visualization choices include pixel overlays, metric plots, correlation heatmaps, or none, with selection tied to the user’s requested analysis.Source: §B.1 · pp. 13–14
  • The task agent’s report instruction must name the chosen metrics and visualizations and tailor the downstream directive to any user formatting requirements.Source: §B.1 · p. 14

B.2 ReportAgent

  • The ReportAgent writes from the provided MorphoOrgaState evidence and must not invent, alter, recalculate, or override measurements or conclusions.Source: §B.2 · p. 14

Related papers

Loading 2609.08696v1…