PaperlayerView exact revision on arXiv

Source-linked AI summary

nnInteractive Agent: General-Purpose AI Agents as 3D Image Annotators

Fabian Isensee, Constantin Ulrich Harsy, Maximilian Rokuss, Benjamin Hamm, Philipp Vollmuth, Klaus H. Maier-Hein

arXiv:2610.31080v1cs.LG

TL;DR

Promptable 3D models can generate volumetric masks, but people still have to find targets, place prompts, inspect results and make corrections. The authors connect nnInteractive to general-purpose agents that interpret requests, locate targets, prompt and inspect 3D masks, correct results and respond to feedback without additional training. Plain-language requests yielded a mean Dice of 0.81 to 0.84 on a diverse benchmark; text-prompted segmentation was more accurate on most standard radiology targets but failed on nonstandard images.

  • Problem

    Promptable 3D models can generate volumetric masks, but people still have to find targets, place prompts, inspect results and make corrections.

    Source: §1 · p. 1

  • Method

    The authors connect nnInteractive to general-purpose agents that interpret requests, locate targets, prompt and inspect 3D masks, correct results and respond to feedback without additional training.

    Source: §1 · p. 2

  • Results

    Plain-language requests yielded a mean Dice of 0.81 to 0.84 on a diverse benchmark; text-prompted segmentation was more accurate on most standard radiology targets but failed on nonstandard images.

    Source: §8 · p. 11, §7 · p. 11

  • Takeaways & Limitations

    The agents handle varied requests by planning, using code where appropriate, checking results and taking plain-language corrections.

    Source: §7 · p. 11, §8 · p. 12

    Runs take minutes and cost $1.3 to $1.5 per image, and the agents were not compared with human annotators.

    Source: §7 · p. 11

Abstract

from arXiv · show

Interactive 3D segmentation models such as nnInteractive turn a few spatial prompts (points, scribbles, boxes) into accurate volumetric masks, but a human still does the annotation work: finding each structure, placing prompts, checking the result and correcting it. General-purpose AI agents can now interpret images, call tools and write code. We show that such agents can take over this work without additional training: the user makes a request in plain language, and the agent works out what has to be segmented, locates the targets, prompts nnInteractive, inspects the 3D result, corrects it and responds to feedback. Asked for 'the muscles used for chewing', an agent identifies and segments the masseters and pterygoids; asked to count liver lesions, it writes its own code to find candidates and splits a merged lesion on request. Our open-source nninteractive-agent provides nnInteractive as tools over the Model Context Protocol (MCP), together with a skill that encodes how experienced users prompt it. On a benchmark spanning CT, MRI, PET and micro-CT of humans, a mouse and an ant (14 images; 51 targets from organs and tumours to muscles, foot bones, vessel trees and an insect brain), Claude Code and Codex reach a mean Dice of 0.81 and 0.84 from the request alone; when told where each target is, both reach 0.85 and prompt nearly as effectively as an oracle that places prompts using the reference segmentation. A text-prompted 3D model (VoxTell) is more accurate on most standard radiological targets but less robust: it returns an empty mask for one of them and fails on the leg muscles, the foot bones, the mouse and the ant, where the agents remain accurate. Code: https://github.com/MIC-DKFZ/nninteractive-agent

1 Introduction

Promptable 3D segmentation models can turn spatial prompts into volumetric masks, but humans still find targets, place prompts, inspect results and make corrections. This work pairs nnInteractive with general-purpose agents to handle that annotation loop from plain-language requests without additional training.

1 Introduction

  • nnInteractive predicts full 3D masks from slice-based points, scribbles, boxes or lassos, but the user still chooses targets, places prompts, judges results and corrects them.It was trained on more than 120 volumetric datasets spanning CT, MRI, PET and 3D microscopy.Source: §1 · p. 1
  • The paper combines a general-purpose agent’s image interpretation, anatomical knowledge and tool use with nnInteractive’s ability to produce voxel-accurate 3D masks from spatial prompts.The agent decides what to segment and where to prompt; nnInteractive turns each prompt into a mask.Source: §1 · p. 2
  • The agent translates plain-language requests into target selection, prompting, 3D inspection, correction and responses to feedback, while the model generates the masks.For chewing muscles, it identifies and segments masseters and pterygoids; for liver lesions, it writes code to find candidates and splits a merged lesion on request.Source: §1 · p. 2
  • The authors evaluate Claude Code and Codex on a 14-image benchmark, comparing them with simulated prompts and a text-prompted model, and release the tools, skill, web demo and benchmark as open source.The benchmark spans modalities, species and structures, and evaluates the interactive annotation process itself.Source: §1 · p. 2

2 Related work

Prior work includes promptable and text-prompted segmentation models, as well as agents that operate segmentation systems. This paper distinguishes its approach by evaluating an untrained general-purpose agent that makes spatial prompting and correction decisions for interactive 3D segmentation.

Promptable segmentation

  • Interactive 3D promptable models accept spatial prompts and produce masks, but remain class-agnostic: the person placing prompts usually decides what to segment.nnInteractive supports points, scribbles, boxes and lassos, making it suitable for elongated or irregular structures.Source: §2 · p. 3

Text-prompted segmentation

  • Text-prompted models map structure names or descriptions directly to masks, but their medical-image accuracy depends on how closely requests match their training distribution.An agent operating a class-agnostic model instead depends on recognizing the target in the image and prompting it well.Source: §2 · p. 3

Agents that prompt segmentation models

  • Unlike trained agents that prompt 2D models or training-free agents that inspect 2D masks, this work evaluates an untrained agent making spatial prompts, checking 3D views and correcting masks across modalities and species.Other systems include SegAgent [18], MedSAM-Agent [19], IBISAgent [20], SAM 3 Agent [17], MedSAM-3 Agent [21], Seg-Agent [22] and MedOpenClaw [23].Source: §2 · p. 3

Training-free agents and imaging software

  • Other training-free imaging agents use automatic segmentation models or dedicated software frameworks, whereas nninteractive-agent extends an existing general-purpose agent through an MCP server and skill.Related systems include Erdur et al. [24], CT-Flow [25], HealthAgentBench [26] and agents operating napari or 3D Slicer.Source: §2 · p. 3

3 nninteractive-agent

nninteractive-agent divides work between an agent that interprets targets and prompts, and nnInteractive, which generates 3D masks. An MCP server, rendered views, Python tools and a prompting skill support the workflow, which can run locally or on a shared server.

3.1 Division of labour

  • The agent decides what to segment and where to prompt, while nnInteractive converts prompts into 3D masks; neither component is specific to an anatomy, modality or species.The agent need not draw voxel-accurate masks: a scribble along a structure’s middle can guide nnInteractive to its boundary.Source: §3.1 · p. 3

3.2 Tools and workspace

  • The MCP server maintains an image, nnInteractive session, current object and committed-object map, while the agent starts, prompts, commits and can reopen objects for correction.Overlaps with committed neighbors are treated as leaks unless an inner structure is explicitly assigned those voxels.Source: §3.2 · p. 4
  • Points, boxes and scribbles provide distinct prompt types; previews and post-prompt feedback help the agent check coordinates and identify common mask problems.Points and scribbles can be positive or negative, and boxes are allowed only as an object’s first prompt.Source: §3.2 · p. 4
  • Rendered slices, montages, projections and 3D views let the agent inspect masks, while Python supports candidate-finding, repeated instance processing and custom figures.Large objects are segmented on a downsampled image copy and scaled back to full resolution; the agent supplies only a size estimate.Source: §3.2 · pp. 4–5

3.3 The skill

  • The skill encodes experienced users’ prompting practices, including coordinate reading, workflow, correction strategies and efficiency rules.It is a Markdown document loaded when agents are asked to segment.Source: §3.3 · p. 5

3.4 Deployment

  • The MCP server and skill can be installed for Claude Code or Codex, with nnInteractive running on a local GPU or shared server; a web demo provides 2D and 3D views with chat.A proof-of-concept Qwen3.8-27B run segmented a spleen with Dice 0.93 in 34 minutes, but was not pursued further because it ran on a shared server under high load.Source: §3.4 · p. 6

4 Agentic behaviour in practice

Examples show agents handling underspecified requests through planning, image inspection, code and user feedback. They find liver-lesion candidates, translate functional descriptions into anatomy, and segment nonmedical objects in a suitcase CT.

4 Agentic behaviour in practice

  • These examples precede the benchmark and illustrate requests that may be underspecified, defined by function or revised through feedback.The examples use Claude Code in the web demo.Source: §4 · p. 6

4.1 Finding and counting liver lesions

  • For a request to count liver lesions, the agent used Python to identify candidates, segmented six definite lesions and one possible lesion, then split a suspected merger after user feedback.It checked the split with an intensity-based watershed, reopened the mask, and separated the nodules with a negative scribble and a new object.Source: §4.1 · pp. 6–7

4.2 Requests that need knowledge

  • For knowledge-based requests, the agent segmented chewing muscles and both sternocleidomastoids, while explaining omitted or uncertain structures; it also identified and segmented suitcase items in a nonmedical CT.For ribs, it found 12 candidates per side, corrected collarbone and missed-rib errors, and returned 24 named ribs after 10.6 minutes and 223 prompts; suitcase-item labels changed after user feedback.Source: §4.2 · p. 7

5 Benchmark

The benchmark spans varied imaging modalities, species and target types, separating target-finding from segmentation with guided and unguided tracks. It compares agent results with prompt-based baselines and scores targets by Dice, averaged first within images.

5 Benchmark

  • The 14-image benchmark covers 51 targets across human CT, MRI and PET, plus mouse and ant micro-CT, including organs, tumours, muscles, bones, vessel trees and an insect brain.Nine images form the radiology group; five cover legs, a mouse and an ant.Source: §5 · p. 7
  • Unguided agents must find and segment named targets, while guided agents also receive one voxel inside each target to separate finding from segmentation.The guided voxel is the point deepest inside the reference target.Source: §5 · p. 7
  • The benchmark shows one slice per image, with references above Claude Code’s unguided segmentations from the middle repeat.The displayed Dice is averaged over each image’s targets; fat-weighted calf and thigh MRI uses a log scale, and the foot CT is shown in a sagittal bone-window view.Source: Figure 6 · p. 8
  • The benchmark compares agents with one-point and simulated point or scribble oracles, while VoxTell receives the same target texts.Oracles use reference knowledge to place up to five prompts; agents are compared with the scribble oracle at their own prompt counts.Source: §5 · p. 8
  • Targets are scored by Dice, averaged across targets within each image and then equally across images; a missing target scores 0.The table caption specifies agent means and standard deviations across three repeats and identifies the radiology and out-of-domain image groups.Source: §5 · p. 8, Table 1 · p. 9

6 Results

The agents outperform a one-click baseline, approach a reference-informed scribble oracle when given target locations, and remain more robust than VoxTell outside standard radiology. Their hardest targets are thin or tubular structures, and their runs reveal distinct prompting and inspection patterns.

6.1 The agents’ prompts are far better than a single click

  • 0.81 for Claude Code and 0.84 for Codex unguided, and 0.85 for both guided, exceed nnInteractive’s 0.71 with one click per target.The largest gaps occur where one point poorly describes the target: liver Dice is 0.00 versus 0.95–0.96, and liver-tumour Dice is 0.13 versus 0.96.Source: §6.1 · p. 9

6.2 The agents’ prompts are close to the oracles’; they stop earlier

  • With target locations given, the scribble oracle is near Claude Code (0.844 against 0.847) and 0.015 ahead of Codex at the agents’ prompt counts.Without locations, the oracle is 0.04 ahead of both agents; agents send 2.2–2.9 prompts per target and stop based on rendered views, while the oracle keeps improving using reference extents.Source: §6.2 · p. 9, Figure 7 · p. 10

6.3 A text-prompted model is more accurate within its domain, but brittle

  • VoxTell reaches mean Dice 0.54 overall and exceeds every agent on seven of nine radiology images, but agents outperform it on the legs, foot bones, mouse and ant.VoxTell’s nine-image radiology mean is 0.78 after an empty adrenal-tumour mask, versus 0.81–0.83 for agents; without that image it is 0.88 versus 0.80–0.82.Source: §6.3 · pp. 9–10

6.4 Where the agents struggle

  • Thin and tubular targets are hardest for the agents: portal and hepatic veins score 0.27–0.53, the left optic nerve 0.02–0.34, and the spinal cord 0.46–0.55.Scribbles must follow narrow structures across slices, and the resulting targets are difficult to judge from renderings.Source: §6.4 · p. 10

6.5 How the agents work

  • Both agents mostly use scribbles; Claude Code sends about one-sixth of prompts from Python code, while Codex inspects four to six times as many 3D renderings.An image takes about four minutes on average; uncertain commit notes correlate with lower mean Dice than other notes for both agents.Source: Table 2 · p. 10, §6.5 · p. 11

7 Discussion

nninteractive-agent connects existing general-purpose agents to nnInteractive so they can carry out interactive 3D annotation without additional training. The approach handles varied requests, but remains slower than text-prompted segmentation and has limits in accuracy, cost, and evaluation scope.

An extension, not a new agent

  • The extension works across multiple agents and preserves their general abilities, but depends on providers for agent quality and permissible data handling.The same tools already serve two commercial agents and an open model; rendered image views may be sent to commercial providers.Source: §7 · p. 11

What nninteractive-agent enables

  • Agents can handle targets that fixed interfaces do not anticipate, including lesions they must find, function-described muscles, and ribs requiring numbering.They can plan, write code, check results, and communicate uncertainty using their general abilities.Source: §7 · p. 11

Agentic versus text-prompted segmentation

  • Text-prompted segmentation is faster and more accurate on most standard radiology targets, while the agent remains useful on less-standard images where VoxTell fails.The agent takes minutes per image; the two approaches could be complementary, with a text-prompted draft checked and corrected through nnInteractive.Source: §7 · p. 11

Limitations

  • The benchmark is small, lacks comparison with human annotators, and uses simple-rule oracles that are not upper bounds.Runs take minutes and cost $1.3 to $1.5 per image at API prices; thin and tubular structures remain difficult to prompt and judge from renderings.Source: §7 · p. 11

8 Conclusion

The paper concludes that general-purpose agents can operate nnInteractive as 3D annotators without additional training, using plain-language requests and iterative prompting. On a diverse benchmark, they achieve mean Dice of 0.81 to 0.84, and remain useful where a text-prompted model fails.

8 Conclusion

  • Agents operate nnInteractive without additional training, reaching mean Dice of 0.81 to 0.84 from plain-language requests and 0.85 when given target locations.With location hints, they prompt nearly as effectively as an oracle using the same number of prompts.Source: §8 · p. 11
  • They plan annotation, write code, and accept plain-language corrections, while continuing to work on data where text-prompted models fail.The authors expect this approach to improve as agents improve.Source: §8 · p. 12
  • The open-source release includes the MCP server, skill, web demo, benchmark scripts, and aggregated benchmark results.The benchmark images are sourced from original providers rather than redistributed; thigh and calf MRI is available from its authors on request.Source: §8 · p. 12

Supplementary material

The supplementary material documents benchmark inputs and results, and illustrates agent prompting and review across several imaging tasks. It also provides example web-demo output and per-image and per-target performance materials.

Benchmark tables

  • The benchmark source table identifies the image sources, while Table S2 reports Dice per target and track.For agents, Table S2 uses the mean over three repeats; oracle scores are after five prompts.Source: Table S1 · p. 15, Table S2 · p. 16

Additional result

Benchmark inputs

  • Figure S1 specifies the shared request template and the different instruction delivery for Codex and Claude Code; VoxTell receives only target texts.Guided requests also include one voxel inside each target.Source: Figure S1 · p. 17

Track comparisons

Web demo

  • Figure S5 shows the web demo’s liver-lesion workflow with orthogonal views, 3D objects, and the agent’s report.Source: Figure S5 · p. 21

Segmentation examples

  • Figures S6–S9 illustrate heart correction, single-prompt pancreas segmentation, liver tumour addition after a missed target, and four-prompt ant-brain segmentation.The illustrated final Dice values are 0.92 for the heart, 0.89 for the pancreas, 0.97 for the liver, and 0.82 for the ant brain.Source: Figure S6 · p. 21, Figure S7 · p. 22, Figure S8 · p. 22

Related papers

Loading 2610.31080v1…