VoxelSage: Tool-Augmented 3D CT Analysis and Simulator-Shielded Sequential Resection Planning for Liver Tumors

Motivation

• Preoperative assessment must combine tumor burden, vascular relationships, resection margins, and preserved liver tissue while keeping measurements and visual evidence linked to the same case and physical coordinates.Source: §1 · p. 1

System contribution

• VoxelSage separates request interpretation and skill selection in Port A from segmentation and CT-mask computation in Port B.Eight built-in skills return structured results or explicit errors; Port A also caches results and supports explicit file selection for cross-case analysis.Source: §1 · p. 1

Sequential planning

• A behavior-cloned ranker orders candidate resection targets on a grid derived from a Bézier surface, while a simulator shield excludes candidates that exceed simulated blood-loss limits or other predefined constraints.Source: §1 · p. 2

Evaluation scope

• Evaluation combines an illustrative online case, a three-case functional test, and a frozen simulator; it supports engineering integration and simulator-level behavior, not clinical performance or surgical safety.Source: §1 · p. 2

2.1 Three-Dimensional Medical Image Segmentation

• Segmentation provides the spatial reference for downstream measurements, but masks still require geometry normalization, lesion-instance conversion, consistency checks, and connection to physical measurements and visible evidence.VoxelSage uses VISTA3D as its default backend and focuses on traceable case-level integration rather than a new segmentation model or comparative superiority claim.Source: §2.1 · pp. 2–3

2.2 Multimodal Medical Agents

• VoxelSage uses a case identifier to bind CT, masks, measurements, images, and 3D assets, returning measurements with their associated mask and affine basis rather than relying on language-model inference.Source: §2.2 · p. 3

2.3 Liver Surgery Planning

• The planning contribution combines automatic resection-candidate generation and margin refinement with user-confirmed, simulator-shielded target ordering, while exposing tumor-distance, vessel-proximity, area, and curvature summaries.The planner does not model liver function, segmental perfusion, deformable anatomy, instrument mechanics, or postoperative outcomes, and it does not rank clinical treatments.Source: §2.3 · p. 3

2.4 Human-in-the-Loop Medical Systems

• Users can edit segmentation artifacts and resection candidates, but downstream planning accepts only a saved surface; these controls define an inspectable research workflow, not established clinical safety or usability.Source: §2.4 · p. 3

3 System Overview

• VoxelSage interprets natural-language requests, selects analysis skills, and returns responses based on computation evidence for public colorectal-liver-metastasis cases.Source: §3 · p. 3

3.1 System Architecture

• The browser provides dialogue and interactive functions, Port A orchestrates language-model reasoning and skill calls, and Port B executes deterministic CT-analysis skills and returns structured evidence.Source: §3.1 · pp. 3–4

3.2 Overall Workflow

• Users upload DICOM or NIfTI volumes; Port A initializes a case context, Port B prepares and segments the CT, and subsequent requests trigger skill calls when existing evidence is insufficient.Returned results and artifacts are integrated into context for further reasoning, with the agent loop limited to six rounds.Source: §3.2 · p. 4

3.3 Browser Workspace

• The browser displays case-linked measurements, contour overlays, and 3D evidence, while temporary edits remain session-local and only explicitly saved segmentation edits or confirmed resection surfaces can feed downstream computation.Source: §3.3 · p. 4

4.1 Case Setup and State Management

• Each supported upload receives a unique case UUID linking source images, masks, and computation results, while Port A tracks processing, skill execution, and errors.Source: §4.1 · p. 4

4.2 Agent Context Construction

• Port A gives the language model skill definitions, the current request, dialogue history, and referenced case context so skill choices and parameters reflect available evidence.Source: §4.2 · p. 5

4.3 Language Model Decision and Skill Execution

• The language model answers directly when evidence suffices or selects skills otherwise; Port A reuses a result only when case UUID, skill name, and input parameters match.Independent new skills may run concurrently, while skills with prerequisites run sequentially; the evidence-gathering loop is bounded.Source: §4.3 · p. 5

4.4 Result Integration and Response Preparation

• Successful structured outputs and artifacts enter case context for later use, while Port A checks returned results for range and consistency and can send errors back through the reasoning loop before reporting unrecoverable failure.Source: §4.4 · pp. 5–6

5 Medical Imaging Skills

• Eight built-in skills provide measurement, evidence generation, interactive correction, and confirmation-gated sequence planning, while user-supplied packages can extend the system.Port B owns case preparation, deterministic computation, and artifact generation; Port A invokes operations through each skill's manifest.Source: §5 · p. 6

5.1 Case Preparation

• Case preparation standardizes DICOM or NIfTI volumes, retains affine geometry, segments liver structures, and creates canonical masks with vessel-optimization alternatives.Small tumor components below a configurable voxel threshold are not materialized as lesion masks; accepted vessel-gap links are recorded in an audit report.Source: §5.1 · p. 6, Figure 3 · p. 6

5.2 Quantitative Measurement Skills

• Four measurement skills return mask-derived volumes, lesion diameters, and tumor–vessel distances, with affine-based estimates accounting for anisotropic voxels.Small components may be labeled too small for stable diameter estimates; overlap is reported as zero, while empty tumor or vessel masks return no distance.Source: §5.2 · p. 7

5.3 Informative Slice Selection Skill and Two-Dimensional Evidence Review

• Slice selection ranks axial images using mask coverage, lesion and vessel co-occurrence, centrality, and image entropy, returning three by default.Results include a windowed CT image and contour overlay; measurements are displayed beside the image for joint review.Source: §5.3 · p. 7, Figure 4 · p. 7

5.4 Interactive Segmentation Correction Skill

• Interactive segmentation correction uses a working session with prompts, undo, and optional bounded propagation; other skills retain the original mask until the user saves.Saving preserves the affine and backs up the previous mask.Source: §5.4 · p. 7, Figure 5 · p. 8

5.5 Three-Dimensional Reconstruction and Interactive Visualization Skill

• Three-dimensional reconstruction places mask-derived anatomy and requested resection surfaces in a shared scene for spatial review and editing.The viewer supports navigation, visibility controls, distance measurement, annotations, and mask-derived volumes; a shared centering offset preserves relative positions of anatomy and resection surfaces.Source: §5.5 · p. 7, Three-Dimensional Reconstruction · p. 8, Figure 6 · p. 8

6 Resection Planning

• 34.274 to 33.388 min (2.6%) and 300.847 to 183.852 mL (38.9%): the learned controller reduced simulated time and blood relative to the serpentine baseline on 256 frozen scenes.The planner generates three candidate surfaces, which a user can edit before target ordering.Source: §6 · p. 8

6.1 Resection Surface Representation

• A bicubic Bézier patch with 16 control points forms an editable resection surface, represented by a 4 × 4 grid.The grid is stored in scene-centered coordinates and mapped back to CT world coordinates through the case center offset.Source: §6.1 · p. 8, Figure 7 · p. 8

6.2 Safety-Margin Constraints

• Tumor-margin refinement evaluates sampled boundary-point clearance and accepts a refined candidate when minimum clearance is at least 4.95 mm against a configurable 5 mm target.The 0.05 mm tolerance is an engineering parameter, not a clinical recommendation; the planner also reports the fifth percentile of clearance.Source: §6.2 · pp. 8–9

6.3 Candidate Surface Generation and Evaluation

• Candidate surfaces are screened for tumor coverage, resection extent, detachable-liver support, and vessel exposure before a diverse shortlist is refined toward the configured margin.Two additional surfaces bypass primary margin-refinement and case-mode gates for separate inspection; fallback surfaces carry an explicit review flag, and these rules are engineering heuristics rather than validated surgical criteria.Source: §6.3 · p. 9

6.4 Sequential Surface-Coverage Planning with Learned Ordering

• The confirmed surface is divided into approximately 4 mm cells, and the task is to order them for step-wise coverage by an ultrasonic-dissector proxy.The learned mode retains the serpentine baseline's grid, movement rules, and start condition while learning the next eligible frontier target; deterministic shortest-path transfer handles movement.Source: §6.4 · p. 9
• The ranker orders candidate targets, and a simulator-rollout shield accepts the first that completes coverage within the frozen simulated-blood budget.The simulator controls travel, clamp timing, exposure, and sealing; it omits vessel branching, deformation, perfusion territories, instrument reachability, cutting dynamics, intra-operative registration, and physiological response, so the sequence is a planning proxy rather than a surgical trajectory.Source: §6.4 · p. 9, Planar Simulator and Blood-Loss Proxy · p. 9
• A deterministic generator offers at most six diverse frontier targets, including the serpentine fallback, while the depth-one teacher labels candidates by cloned rollouts completed with serpentine scanning.The teacher selects an accepted candidate by lexicographically minimizing total time and blood; behavior cloning approximates that preference, and states without an accepted candidate are excluded from labels.Source: Hierarchical Learned Ordering with… · p. 10

• Offline teacher labels use full-episode time, simulated blood, and exact-safe status after each candidate executes with a frozen S-tail.These labels train the behavior-cloned ranker; online, the ranker orders candidates and the exact shield checks them in that order.Source: Figure 8 · p. 11
• The online shield stops at the first ranked candidate satisfying the completion, rollout-failure, and blood-budget predicate; if none passes, execution terminates as infeasible.Its budget compares simulated blood with the deterministic serpentine baseline plus a fixed margin.Source: A Offline teacher labelling and… · p. 11
• 97,421 exact rollouts for 97,320 macro actions (1.001 per action); C4 matched complete-set selection across a 256-scene audit and reduced planning latency.The first-ranked candidate was accepted in 97,228 actions, the second in 83, and the third in nine.Source: A Offline teacher labelling and… · p. 11
• C4 selected the serpentine target for 98.97% of macro actions; C0 recovered the deterministic baseline, while removing the shield in C5 reintroduced rare budget overruns.The remaining 1.03% of C4 actions were state-dependent deviations.Source: A Offline teacher labelling and… · p. 11

7.1 Online System Demonstration and Port B Verification

• Across 24 synthetic NIfTI phantoms, all 186 numerical comparisons passed an absolute tolerance of 0.01 in the corresponding metric unit, while all 30 boundary checks returned expected structured results.Across three frozen public cases, all 66 recorded checks passed across 24 unique skill–case paths; cold preparation took a median 43.3 s, and cached reuse was reported as 0.0 s at one-decimal precision.Source: Stage 1: Physical-Coordinate Correctness · p. 12, Stage 2: Three-Case Port B Functional… · p. 12
• Constructed-surface tests found identical outputs for lazy and complete candidate verification and confirmed deterministic public-skill execution across three additional surfaces.The archived three-case bridge used support-expanded targets and does not validate the current core-only adapter; patient-derived core-only validation remains outstanding.Source: Stage 3: Learned-Controller Transfer · p. 12
• Nine prespecified defensive checks produced expected rejections or partial-status reporting, and matching SHA-256 entries supported artifact traceability.These checks do not establish software correctness or patient-level validity.Source: Stage 4: Fail-Closed and… · p. 12

7.2 Planar-Simulator Evaluation

• C4 reduced mean simulated time from 34.274 to 33.388 min and had a mean simulated-blood difference of −116.995 mL relative to C0 on 256 confirmation scenes.The paired time difference was −0.886 min; C4 had no episode-budget overruns. Figure 9 summarizes paired controller effects relative to C0.Source: Primary replication and learning… · p. 13, Figure 9 · p. 13
• C4 exceeded its episode budget in 0/256 scenes versus 24/256 for unshielded C5, and completed all 128 scenes in each of S0–S4 without overruns.The lazy exact shield changed the top-ranked choice in 92/97,320 macro actions (0.0945%).Source: Shield ablation and computational cost · p. 14, Condition-specific sensitivity · p. 14, Figure 10 · p. 14
• C4’s median controller planning time was 54.7% lower than C3’s, although C3 produced episodes 0.210 simulated minutes shorter on average.This isolated CPU benchmark measures controller computation, not simulated surgical time.Source: Shield ablation and computational cost · p. 14

7.3 Port A Orchestration Evaluation

• Port A performed well on lesion existence, total lesion volume, and maximum three-dimensional diameter, while lesion counting and liver volume remained limited; case comparisons were mostly correct.No measurements were invented for nonexistent cases, and result reuse reduced tool calls and response times without guaranteeing answer correctness.Source: §7.3 · p. 15
• The orchestration tests used reference masks and nonexistent-case queries; they do not establish performance with predicted masks or other missing-evidence failures.The authors state that clinical-use requirements remain to be assessed.Source: §7.3 · p. 15

8 Conclusion

• VoxelSage links CT preparation, deterministic measurement, visual evidence, interactive correction, and confirmation-gated planning through a dual-port architecture.The learned planning adapter was verified on constructed surfaces; separate frozen-simulator tests reported improved time over prespecified baselines without nominal-set budget violations.Source: §8 · p. 15
• The evaluation establishes workflow integration and a reproducible simulator experiment, not clinical accuracy, benefit, safety, or patient-level blood-loss prediction.The learned component was trained on two-dimensional proxy tasks; the authors call for external evaluation in representative cohorts and clinical workflows.Source: §8 · p. 15

A.1 Available Skills

• Built-in skills separate read-only measurement and evidence generation from correction and planning, which require a saved mask or candidate surface.User-provided skills are not included in the reported evaluation.Source: §A.1 · p. 18

A.2 Manifest, User-Skill Package, and Registration Contract

• Each skill manifest specifies its name, version, description, triggers, and JSON-style inputs and outputs; Port B resolves context-owned CT and mask inputs from the case identifier.User packages require skill.yaml and main.py, with a run(ctx) entry point returning a JSON-serializable dictionary; execution has time, CPU, process, and open-file limits but lacks separate identity, memory quota, and filesystem or network isolation.Source: §A.2 · p. 18

A.3 Tool-Calling Examples

• Clients prepare a case, list registered skills, and invoke a selected skill using its case_id and optional parameters; Port B returns structured results to the orchestrator.The appendix’s abbreviated invocation is illustrative and separate from the controlled evaluation.Source: §A.2 · pp. 18–19, §A.3 · p. 19

A.4 Experimental Learned-Planning Contract

• The opt-in learned_shielded mode requires a saved surface, approximately 4-mm cells, a compatible cropped domain, and the specified frozen checkpoint.It retains the sampled-liver core without support cells, preserves boundary-vessel proxies, maps adapter indices to the full surface grid, and checks ranked candidates with a lazy exact shield.Source: §A.4 · p. 19
• Invalid checkpoints, incompatible geometry, incomplete rollouts, or absent admissible candidates produce structured failures rather than deterministic fallback labeled as learned output.Successful results record mappings, paths, policy identifiers, interventions, simulator quantities, and runtime for implementation audit—not clinical validation.Source: §A.4 · p. 19

B.1 Frozen data separation and development gate

• Only policy-training scenes contributed supervision, gradient updates, and estimated normalization or budget scales; all scene identifiers were disjoint across splits.Each later split was opened only after the preceding decision was complete.Source: §B.1 · p. 20, Table 6 · p. 20
• The development gate required legal episodes, no failures or simulator-invariant violations, bounded per-scene blood excess, a favorable paired time interval, and the aggregate blood criterion.The learned controller also had to retain at least 50% of the teacher’s mean time improvement; a pass authorized tuning but was not test evidence or clinical-safety evidence.Source: §B.1 · p. 20
• All three seeds had to pass their individual gates before configurations were ranked by paired time, paired blood, intervention rate, 95th-percentile wall time, then configuration identifier.The same ordering hierarchy selected a seed within the chosen configuration.Source: §B.1 · p. 20
• The v10.6 and v10.8 split generators use separate master seeds, and the v10.8 confirmation manifest reuses its first 128 geometries across five sensitivity conditions.The generators, selection rules, and sensitivity-evaluation artifacts are archived with the implementation.Source: §B.1 · p. 20

B.2 Released checkpoint configuration

• The encoder uses three 3 × 3 convolutions with dilation factors 1 and 2 among them, and each convolutional layer produces 32 feature maps, establishing the stated spatial width.Source: §B.2 · p. 21
• Ten binary semantic-grid masks encode domain, cut, vessel, frontier, position, and start-cell information; an eleventh channel is a normalized transfer-distance map.The map is quantized to 8 bits for storage, while bit-packed masks are unpacked before network input.Source: §B.2 · p. 21
• 25 scalar features describe each candidate’s provenance, costs, vessel state, clamp timing, newly exposed or sealed area, bleeding rate, and progress.The 19 global features summarize phase, elapsed time, cutting progress, vessel context, bleeding and clamping, and the scene-specific blood budget.Source: §B.2 · p. 21
• Six loss coefficients weight teacher-choice ranking, time and blood regression, completion classification, and exact-safe classification; they combine offline signals rather than define an online action cost.AdamW updates trainable parameters, with decoupled L2-style weight decay and global gradient norm capped at 5.0.Source: §B.2 · p. 21
• Up to 256 decision states form an update batch, and five epochs mean five passes through policy-training states; the released configuration also specifies its random seed.The seed fixes Python, NumPy, and PyTorch random states used for initialization and training-state shuffling.Source: Table 7 · p. 20, §B.2 · p. 21
• 171,401 decision-state snapshots from 448 policy-training scenes provide training data, with 1,017,114 valid candidate entries supplying ranking and auxiliary supervision.Padded invalid slots are excluded from masked losses.Source: Table 7 · p. 20, §B.2 · p. 21