Source-linked AI summary
From Analytics to Tumor Boards: An Evidence-Linked Multi-Agent Workflow for Oncology Feature Extraction
Daniel Kang, Michelle Hu, Soorya Ram Shimgekar, Shayan Vassef, Yufan Wang, Anit Kumar Sahu, Munmun De Choudhury, Vedant Das Swain, Christian Poellabauer, Li Yan Khor, Koustuv Saha, Robert Wojciechowski, Elliot Kidd, Piyum Zonooz, Navin Kumar
TL;DR
Fragmented oncology documentation makes clinically accurate, reusable abstraction difficult because findings must be linked across entities and time points. This study evaluates nMAS, a configurable workflow that routes fields by complexity, consolidates outputs, and validates them against source text. Across the reviewed evaluation surface, nMAS achieved higher rank-weighted value-level F1 than the UMA-style MiniMax M2.5 comparator, while workflow-efficiency benefits were not directly measured.
Problem
Fragmented oncology documentation requires accurate attribution across specimens, tumors, biomarkers, and time points, creating a substantial abstraction challenge.
Method
nMAS uses a clinician-informed 328-field schema, complexity-aware extraction, report-level consolidation, and source-grounded validation for heterogeneous oncology documents.
Results
85.0% rank-weighted value-level F1 was achieved by nMAS versus 66.4% for the UMA-style MiniMax M2.5 comparator across 418 reviewed document-field pairs.
Takeaways & Limitations
The findings support the feasibility of configurable, source-grounded extraction for converting fragmented oncology documentation into reusable structured data.
Takeaways & Limitations
The evaluation used 230 documents from 40 patients at one institution and did not directly measure workflow efficiency or whether cited sentences semantically supported values.
Abstract
from arXiv · showhide
Clinically relevant oncology information is distributed across heterogeneous, longitudinal documentation, creating substantial abstraction burden and requiring accurate attribution across specimens, tumors, biomarkers, and time points, while manual cancer-registry abstraction can require 27.2 minutes per case, highlighting the need for scalable methods that preserve clinical context while converting documentation into structured data. We evaluate the Nimblemind Multi-Agent System (nMAS), a configurable oncology information-extraction workflow which extracts clinically relevant structured fields from fragmented oncology documentation. The extraction task uses a clinician-informed schema of 328 attributes spanning report metadata, diagnosis, staging, and cancer-type-specific information. nMAS separates clinician-defined field specifications from model execution and combines complexity-aware extraction, report-level consolidation, and source-grounded validation. The retrospective evaluation included 230 de-identified oncology documents from 40 patients and 418 clinician-reviewed document-field pairs containing 1,126 non-empty reference values. Evaluation focused on fields identified by clinicians as present in the source documents rather than exhaustively annotating all 328 schema fields. nMAS achieved a rank-weighted value-level precision of 82.6%, recall of 87.5%, and F1 of 85.0%, compared with an F1 of 66.4% for an independently implemented UMA-style MiniMax M2.5 comparator. These findings support the feasibility of using a configurable, source-grounded extraction workflow to convert fragmented oncology documentation into reusable structured data.
1. Introduction
Oncology abstraction requires more than entity recognition because clinically important information is fragmented, repeated, and tied to specific specimens, tumors, and time points. nMAS addresses this need with configurable, source-linked extraction of structured oncology variables.
- Oncology variables span tumor characteristics, staging, biomarkers, treatment context, progression, recurrence, and metastatic disease.
- Repeated, nested, multivalued, and temporally qualified findings require attribution to the correct clinical object, time point, and disease context.
- Existing systems often depend on predefined variables, document types, annotations, rules, or model adaptation when extraction schemas change.
- nMAS uses a clinician-informed 328-field schema, complexity-aware processing, report-level consolidation, and source-grounded validation.
- The workflow preserves context across specimens, tumors, and clinical time points while producing reusable, source-linked structured variables.
2. Related Work
Oncology information extraction combines established rule-based and contextual approaches but remains difficult to adapt across heterogeneous fields and cancer-specific requirements. nMAS responds by routing different evidence patterns through complementary mechanisms within one configurable workflow.
- Clinical Data Abstraction and Oncology-Specific Burden: 27.17 minutes per case was the mean manual abstraction time in one hospital-based cancer-registry evaluation, falling to 15.07 minutes with EMR integration.
- Clinical Data Abstraction and Oncology-Specific Burden: Oncology abstraction requires integrating pathology, radiology, cytology, treatment, and clinical-note evidence across heterogeneous, longitudinal documentation.
- Approaches to Oncology Information Extraction: Rule-based systems suit stable lexical, coded, or numeric fields but require manual maintenance as terminology, report structures, conventions, or requested fields change.
- Approaches to Oncology Information Extraction: Supervised and neural systems provide contextual modeling but commonly remain tailored to specific cancer sites, document types, and predefined variables.
- Approaches to Oncology Information Extraction: nMAS combines clinician-defined field specifications with complexity-aware routing across NER/NLP, smaller-language-model, and larger-language-model extraction.
- Approaches to Oncology Information Extraction: The shared workflow supports heterogeneous and changing schemas by processing regular and context-dependent fields through different extraction strategies.
3. Data
The retrospective dataset comprised de-identified oncology documents from multiple report types and patients. Evaluation used clinician-defined field importance ranks and rank-weighted metrics on the reviewed evaluation surface.
- Dataset Overview: 230 de-identified oncology documents from 40 patients formed the retrospective evaluation dataset.
- Dataset Overview: The documents included pathology reports, radiology summaries, cytology reports, and oncology clinical notes.
- Dataset Overview: Clinically relevant wording, section structure, diagnostic terminology, and context were preserved after de-identification.
4. Method
nMAS standardizes oncology reports, maps requested fields to a configurable field library, routes fields by extraction complexity, and consolidates and validates outputs into structured report-level records. Its prompts constrain extraction to explicit source evidence and support field-specific clinical boundaries.
- Workflow Overview: Only the feature-extraction pathway of nMAS was evaluated among its multiple clinical data-processing pathways.
- Workflow Overview: Regular evidence, bounded interpretation, and broader contextual reasoning are routed to NER/NLP, SLM, and LLM tiers, respectively.
- Input Standardization and Field Preparation: Reports were standardized as de-identified clinical free text while preserving diagnostic, pathology, imaging, staging, biomarker, and ancillary-test content.
- Input Standardization and Field Preparation: Each requested field was linked to a FIELD LIBRARY entry specifying its canonical name, value structure, clinical description, context, and guardrails.
- Achievability Agent and Query Parsing: The Achievability Agent screened inputs and objectives before downstream model calls, while the Query Parser resolved fields to canonical schema names and recorded unmatched requests.
- Extraction Tiers: Tier 2 handled bounded contextual fields such as staging, tumor characteristics, margins, specimens, radiology findings, biomarkers, and cancer-specific pathology variables.
- Extraction Tiers: Tier 3 handled broader synthesis across report sections and complex logic involving negation, uncertainty, metastasis, progression, or cancer-specific interpretation.
- Field-Specific Demonstrations and Guardrails: Field-specific prompts supplied definitions, value structures, examples, context, and guardrails, while requiring explicit source extraction and NOT FOUND for unsupported values.
5. Results
nMAS achieved strong rank-weighted extraction performance and exceeded the UMA + MiniMax M2.5 comparator across ranks. Results also show high document-field scores, while qualitative review identified recurring attribution and temporal errors.
- Value-level performance: 82.6% precision, 85.0% recall, and 87.5% F1 were achieved on the reviewed, non-empty evaluation surface.This evaluation surface was narrower than the full 328-field schema.
- Document-field performance: 88.0% strict and 94.1% lenient document-field scores indicate frequent partial recovery of multivalued fields.These scores do not verify that cited sentences support the extracted value's time point or negation status.
- Overall comparison: 85.0% vs. 66.4% rank-weighted F1: nMAS outperformed the UMA + MiniMax M2.5 comparator across all three ranks.The largest absolute difference occurred at Rank 3, where nMAS scored 88.8% versus 65.7%.
- Evaluation design: Table 1 reports rank-stratified extraction performance using clinician-defined weights of 43%, 30%, and 27% for Ranks 1–3.The weighting scheme underlies the rank-weighted overall scores.
- Qualitative error patterns: Clinician review identified recurring errors in repeated-entity alignment, assay interpretation, schema granularity, and temporal disambiguation.Examples included confusing Oncotype DX component markers with the assay name and over-including historical dates.
6. Discussion
nMAS achieved higher rank-weighted value-level F1 than the UMA-style comparator, while the study’s workflow-efficiency implications remain prospective. Ablations and evaluation design identify configuration effects and scope limits that qualify interpretation.
- 85.0% rank-weighted value-level F1 for nMAS exceeded 66.4% for the UMA-style MiniMax M2.5 comparator.The comparison covered 418 reviewed document-field pairs containing 1,126 reference values.
- nMAS’s extraction performance may support pre-structuring heterogeneous oncology information, but workflow efficiency was not directly measured.Prior studies provide operational context, including reduced registry abstraction and tumor-board preparation time, rather than direct evidence from this evaluation.
- 85.0% rank-weighted F1 versus 66.4% suggests potential usefulness for fields with substantially different evidence patterns, but the comparison is system-level rather than an isolated multi-agent effect.The systems differed in model choice, fine-tuning, prompting, context construction, extraction strategy, and output processing.
- The production tier-routing table did not outperform simpler alternatives, and batch-size effects differed between Tier 3 and Tier 2 models.Tier 3 tolerated all-fields-in-one-call with little cost, whereas Tier 2 did not, supporting model-specific batch-size tuning.
- The evaluation used 230 documents from 40 patients in one institution and a reviewed, non-empty surface rather than exhaustive schema annotation.This limits assessment of cross-institutional generalizability, true-negative performance, and false positives among unreviewed fields.
- Future work should test larger multiinstitutional cohorts with exhaustive positive and negative annotation and matched-model ablations.These designs would help isolate routing, field-specific guidance, FE-MUX, and validation effects.
- Explicit entity linking, temporal relation modeling, and stronger structure-aware extraction are motivated by repeated clinical objects, biomarkers, treatment events, and table-like assays.Structured decoding and exact-set evaluation are also proposed for nested and multivalued outputs.
Appendix A. Supplementary Material
The appendix supplies supplementary materials supporting the oncology extraction workflow and evaluation, including schema mappings, comparator details, instructions, normalization procedures, and qualitative examples.
- The appendix includes field-library entries, source-to-schema mappings, comparator details, extraction instructions, normalization procedures, and qualitative examples.
A.1. Field Schema, Evaluation Surface, and Metric Detail
The evaluation uses a clinician-informed 328-attribute schema and a reviewed evaluation surface, with value-level metrics computed from matched, extra, and missing values and weighted by clinical priority.
- Field Schema: The clinician-informed schema contains 328 attributes spanning metadata, diagnosis, staging, tumor characteristics, biomarkers, imaging, treatment, and cancer-specific fields.Each entry includes a canonical name, expected value structure, clinical description, examples, context, and applicable interpretation rules.
- Evaluation Surface: The evaluation surface contained 418 clinician-reviewed document-field pairs surfaced by nMAS, with 94% reviewer agreement on the surfaced candidates.Because candidates originated from nMAS rather than an exhaustive independent audit, evaluation-surface bias may remain.
- Clinical Priority Metadata: Clinicians assigned evaluated fields to three importance ranks for rank-stratified analysis and rank-weighted overall evaluation.The ranks reflected relative importance to oncology practice.
- Evaluation Surface: Evaluation was restricted to non-empty clinician-reviewed reference values, so blank workbook cells were unreviewed rather than confirmed negatives.Predictions outside the reviewed surface were excluded, and true negatives were not reported.
- Metric Detail: Value-level precision, recall, and F1 counted matched, unmatched predicted, and unmatched reference values as TP, FP, and FN, respectively.Missing or invalid predictions were treated as empty predicted sets.
- Metric Detail: The strict recovery score measured the proportion of reference values recovered, while the lenient score gave full credit when at least one reference value was recovered.
- Metric Detail: Rank-weighted metrics pooled TP, FP, and FN within each importance rank before applying weights of 43%, 30%, and 27% to Ranks 1, 2, and 3.The weighting formula was S_weighted = 0.43S_1 + 0.30S_2 + 0.27S_3.
A.2. Representative Oncology Field-Library Entries
Table A2 presents representative entries from the oncology-specific field library, which defines expected outputs and clinically informed extraction boundaries.
- Representative field-library entries define expected outputs for oncology extraction.
- The entries illustrate clinically informed boundaries for the fields being extracted.
A.3. Representative Oncology Data Example
The example shows heterogeneous, de-identified oncology documentation and how source text supports structured oncology variables.
- The source excerpt is de-identified oncology documentation processed in the study.
- The report describes increasing metastatic hepatic lesions during second-line therapy, consistent with disease progression.
- The excerpt supports several structured oncology variables mapped in Table A3.
A.4. External UMA-Style Benchmark Details
The external benchmark uses a one-attribute UMA-style prompt with field-specific guidance and source-evidence requirements, then normalizes outputs for comparison with nMAS. MiniMax M2.5 was selected through a challenging-set pilot based on performance and runtime characteristics.
- Evaluation Setup: The comparator received one de-identified report, one target field, a concise definition, expected value information, and field-specific guidance.
- Evaluation Setup: Comparator responses were normalized into the same value-level evaluation format as nMAS, while clinician reference values and nMAS outputs were excluded from comparator prompts.
- Model Selection: MiniMax M2.5 was selected from a five-document challenging-set pilot using extraction performance, structured-output validity, evidence grounding, and runtime criteria.
- Model Selection: MiniMax M2.5 matched Claude Opus 4.8 on F1 while processing attribute calls more than three times faster.
- UMA Prompting: The prompt template required compact structured JSON containing extracted values and verbatim source evidence.
- UMA Prompting: Unsupported attributes had to return empty value sets, while multiple supported values were returned as separate objects without inference.
- Audit Examples: Audit examples traced raw comparator responses, normalized values, and TP/FP/FN scoring at the document-field level.
A.5. Detailed Value Normalization and Matching
The evaluation normalizes, expands, and matches extracted values at the document-field level, applies identical processing to nMAS and the comparator, and examines pipeline components through a limited ablation study.
- Value Normalization: Predicted and reference values underwent field-specific normalization, including case, date, Boolean, punctuation, and unit-aware numeric transformations.
- Repeated and List-Like Fields: Repeated and list-like fields were expanded into individual value instances before comparison.
- One-to-One Matching: Predicted and reference values were matched one-to-one within each document-field pair, preventing duplicate predictions from receiving multiple matches.
- Invalid and Missing Outputs: Missing, invalid, unsupported, or NOT FOUND outputs were treated as containing no extractable values.
- Evaluation Consistency: The same normalization, expansion, and matching procedures were applied to nMAS and the comparator to target extraction differences rather than evaluation-processing differences.
- Ablation Study: The ablation compared isolated extraction with the full pipeline, including classification, achievability checking, feature multiplexing, and output validation.
- Ablation Results: F1 was 48.9% for the full-pipeline baseline versus 44.5% for isolated extraction, while both remained below the main evaluation’s rank-weighted F1 of 85.0%.