NV-Reason-CT: 3D Visual Language Model for CT Analysis

1. Introduction

• NV-Reason-CT couples a native 3D vision transformer to a language model, retaining all visual tokens and explicit depth, height, and width coordinates through decoding.The model processes chest or abdominal volumes alongside text without further spatial token merging.Source: §1 · p. 2
• The curated corpus combines expert interpretations, report-grounded synthetic narrations, standardized reports, focused questions, and multi-turn interactions.Expert examples directly supervise anatomical review, diagnostic considerations, and uncertainty; report content constrains synthetic narrations.Source: §1 · pp. 2–3
• End-to-end SFT jointly adapts the encoder, projector, and language model, followed by GRPO with region-specific abnormality and report-structure rewards.Verifiable finding labels support reinforcement learning without an expert-authored reasoning trace for every volume.Source: §1 · p. 3
• Evaluations span abnormality classification and report generation across CT-RATE, Merlin, RAD-ChestCT, and a held-out NIH cohort, alongside qualitative and preliminary expert-radiologist assessments.The model and its SFT and GRPO training code are released to support reproducible research.Source: §1 · p. 3

2.1. Contrastive and representation-learning methods

• CT representation-learning approaches align volumes with text or use image-only objectives, producing encoders that may later support downstream classification or generation.The introduction distinguishes representation learning from the capabilities of a final system that adds a decoder.Source: §2 · p. 3, §2.1 · pp. 3–4

2.2. CT report-generation systems

• CT report-generation systems vary in how directly reports derive from images and how clinical content is measured.Some designs gate report content through a separately supervised abnormality classifier, unlike open-ended image-conditioned reporting.Source: §2.2 · p. 5

2.3. Generative CT vision–language models

• Generative CT vision–language systems commonly pass projected or resampled visual features as a 1D sequence without an explicit dense 3D coordinate grid in the language decoder.NV-Reason-CT instead retains every spatially indexed visual token through decoder processing with text.Source: §2.3 · p. 6

3.1. Overview

• The model combines a native 3D vision–language architecture with radiologist-guided supervision and verifiable reinforcement learning for chest and abdominal CT interpretation.Expert interpretations also guide synthetic narrated interpretations from source reports; GRPO optimizes finding agreement and report organization without requiring an expert-authored reasoning trace for every volume.Source: §3.1 · p. 7

3.2. Model architecture and volumetric interface

• The visual encoder produces 13,824 tokens that the projector passes to the language model without spatial merging, retaining explicit depth, height, and width indices during decoding.The model couples a Primus 3D vision transformer initialized from COLIPRI weights with Qwen3.5-4B; multimodal rotary positional encoding applies the tokens’ grid coordinates during language-model processing.Source: §3.2 · pp. 7–8

3.3. Volumetric preprocessing; 3.4. Anatomy-aware cropping: chest and abdomen

• Volumes are reoriented and resampled to 2-mm isotropic spacing, then a lung-landmark heuristic selects fixed 192-voxel chest or abdominal crops, with an edge-aligned fallback if lung bounds are unavailable.The heuristic may mislocalize the intended region when lung landmarks are absent or difficult to identify.Source: §3.3 · p. 8, §3.4 · p. 8

3.5. Problem formulation and output representation

• A single conditional policy maps a regional CT volume and instruction or dialogue context to reports, question answers, and multi-turn responses; structured-report examples pair a region with its reference finding set.The structured-report format appends one machine-readable answer block for GRPO finding-agreement scoring, while other SFT tasks retain formats such as free text and Yes/No answers.Source: §3.5 · p. 8

3.6. Supervised fine-tuning

• End-to-end SFT jointly adapts the visual encoder, projector, and language model on curated reports, questions, reasoning, follow-ups, and refusal examples.The autoregressive loss excludes system and user tokens; in multi-turn examples, supervision applies only to the final assistant response.Source: §3.6 · p. 10

3.7. Group Relative Policy Optimization

• Starting from the SFT policy, GRPO samples completions for CT-conditioned prompts and updates the policy using a clipped objective and group-relative rewards.The objective uses the token-probability ratio between the optimized and sampling policies; the group-relative advantage centers a completion’s reward on its prompt group and scales by reward spread.Source: §3.7 · p. 10

3.8. Region-aware verifiable rewards

• Region-aware rewards combine abnormality-set agreement, structured-report conformity, and a soft penalty for completions outside a target length band.The anatomy region selects the finding ontology and report schema; the correctness reward is zero unless the completion contains exactly one valid answer block.Source: §3.8 · p. 11

3.9. Training and implementation details

• Training uses 16 nodes with eight H100 GPUs each, alongside component-specific SFT learning rates and GRPO sampling of 16 completions per prompt.The SFT learning rate is scaled to 0.1× for the visual encoder, 5× for the projector, and 1× for the language model; GRPO uses a 1 × 10^-6 learning rate.Source: §3.9 · p. 11

4.1. Datasets

• Evaluation uses CT-RATE and held-out NIH chest and abdomen cohorts, plus external Merlin abdominal and RAD-ChestCT chest datasets.CancerVerse contributes targeted VQA only because its cancer-focused reports are not suitable for structured-report-generation targets.Source: §4.1 · p. 12

4.2. Data Curation

• The curriculum contains approximately 550,000 multimodal instruction examples built from 70,111 unique CT image inputs; examples are not independent examinations.One CT can support multiple tasks.Source: §4.2 · p. 13
• Recorded radiologist interpretations were transcribed and curated as direct supervision and exemplars for report-grounded synthetic narrations.Experts narrated observations, provisional hypotheses, differential diagnoses, and uncertainty.Source: §4.2 · p. 13
• A radiologist-guided ontology harmonizes 30 chest and 29 abdominal findings, masking unmentioned findings when regional coverage is limited or unknown.Negative labels and derived no-finding labels require full coverage.Source: §4.2 · p. 13
• Structured-report targets use region-specific anatomical schemas, while reasoning walkthroughs follow a systematic search pattern and end with canonical positive-label sets.Rewriting preserves supported findings and uncertainty while excluding unsupported content.Source: §4.2 · pp. 13, 15
• Single-turn follow-ups are kept context-independent, while multi-turn examples preserve the initial report request and four exchanges for references to earlier findings.Unsupported questions are answered by stating evidentiary limits rather than inventing findings.Source: §4.2 · pp. 13, 15
• QA supervision covers laterality, location, severity, attributes, and binary findings; safety examples teach refusal for invalid requests or mismatched inputs.Localization and attribute labels are retained only when the finding and descriptor are unambiguous within the same report section.Source: §4.2 · p. 15

5. Results

• The Results section also evaluates CT-RATE, Merlin, and RAD-ChestCT, extends assessment to a held-out NIH cohort, and includes a preliminary expert-radiologist study.These passages describe the evaluation scope, not the outcomes of those additional evaluations.Source: §5 · p. 15

5.1. CT-RATE evaluation

• The corrected CT-RATE v2 validation cohort comprised 3,002 reconstruction-level cases from 1,564 studies, rather than the historical 3,039-case manifest.Cases from the same study share a radiology report; published comparators may instead evaluate one image–report pair per study.Source: §5.1 · p. 15, Table 2 · p. 17

5.1.1. CT-RATE abnormality classification

• Macro-F1 0.6140 and macro-AUROC 0.8710 were NV-Reason-CT’s CT-RATE abnormality-classification results in distinct evaluation modes.Macro-F1 used parsed generated Yes/No answers; macro-AUROC used Yes probabilities from a separate scoring mode with thinking disabled.Source: Table 3 · p. 18, §5.1.1 · p. 19
• Macro-F1 0.6100 resulted from one all-abnormalities prompt, compared with 0.6140 from individual abnormality questions, using one inference call per CT.The all-label prompt returned “No Chest Finding” when none was present.Source: §5.1.1 · p. 19
• M3D-LaMed’s independent raw-volume evaluation yielded macro-F1 0.3046; a separate VoxelFM supervised probe on frozen M3D features reported macro-F1 0.4257 and macro-AUROC 0.7403.The VoxelFM probe used the historical 3,039-case cohort, so these are distinct evaluation setups.Source: §5.1.1 · p. 21

5.1.2. CT-RATE report generation

• Report-derived Macro-F1 and CRG are the primary clinically oriented report metrics; lexical metrics can be sensitive to wording and organization, and GREEN is potentially less reliable out of domain.The GREEN checkpoint was fine-tuned primarily on chest X-ray reports, while NV-Reason-CT produces standardized Findings and Impression sections and CT-RATE references retain heterogeneous styles.Source: §5.1.2 · p. 22
• Report-derived macro-F1 was 0.5923, while direct scoring of the report’s detected-abnormality list yielded macro-F1 0.6010.The report evaluation compared Findings and Impression against references for 3,002 reconstructions; the report-derived and direct F1 values measure distinct outputs.Source: Table 4 · p. 23, §5.1.2 · p. 23

5.2.1. Merlin abnormality classification

• 0.7531 macro-F1 was obtained with thinking-enabled hard Yes/No answers on Merlin; prompt selection on the same released test cohort makes this a test-prompt-tuned estimate.For the probability-based evaluation, unmentioned labels were excluded, explicit positives and negatives were balanced per finding, and F1 used a fixed 0.5 threshold; six lower-thoracic findings used the chest region and the other 24 used the abdomen region.Source: §5.2.1 · p. 25

5.2.2. Merlin report generation

• After Merlin-format adaptation, NV-REASON-CT leads on ROUGE-L, BERTScore, RadGraph-F1, and GREEN, but trails MedGemma 1.5 and Jolia on BLEU.The checkpoint was fine-tuned for one epoch on 15,314 released Merlin training rows and generated Findings for all 5,125 test scans; nominal cohort and report scope match, though evaluator versions and reference preprocessing may differ.Source: §5.2 · p. 25, Table 6 · p. 26, §5.2.2 · pp. 26–27

5.3.1. RAD-ChestCT abnormality classification

• RAD-ChestCT evaluation uses 16 harmonized abnormality targets from an external, out-of-distribution chest CT benchmark with 3,630 publicly released volumes.The mapping excludes Mosaic attenuation, merges two arterial/coronary wall calcification outputs, and directly matches the remaining 15 abnormalities; Table 7 summarizes published direct-classification results.Source: §5.3 · p. 27, Table 7 · p. 27

NV-REASON-CT (ours)

• Positive-class macro-F1 0.5311, weighted binary F1 0.7379, and macro-AUROC 0.7958 on all 3,630 RAD-ChestCT volumes, using one binary prompt and a fixed 0.5 threshold without dataset-specific tuning.The evaluation used the stated 16-label mapping.Source: NV-REASON-CT (ours) · p. 27

5.3.2. RAD-ChestCT report generation

• Report-derived macro F1 was 0.5105 from the final abnormality list and 0.4854 with frozen CT-RATE RadBERT; 288 reports were truncated at its 512-token limit.The evaluation compared extracted report abnormalities with public binary labels; reference-dependent report metrics were unavailable.Source: §5.3.2 · p. 28, Table 8 · p. 28

5.4. Internal Dataset Evaluation

• Balanced NIH-cohort performance varied by finding, with lower F1 for chest pulmonary embolism and lymphoma and abdominal appendiceal abnormality and abdominal wall hernia.The evaluation balanced positive and negative cases independently for each finding; small cohorts warrant caution.Source: §5.4 · pp. 29–30, Table 9 · p. 30, Figure 8 · p. 31

5.5. Subjective Evaluation with Expert Radiologists

• In the preliminary reader study, radiologists first interpreted cases unaided, then reviewed the same cases with the model’s reasoning and structured report, which they could verify and edit.The study included two radiologists with more than 10 years of practice and 10 chest and abdominal CT cases.Source: §5.5 · p. 31

A. Accuracy and Reasoning Quality

• Readers rated whether the reasoning trace included correct statements about key observations.Ratings used a five-point Likert scale.Source: A. Accuracy and Reasoning Quality · p. 31
• The questionnaire asked whether reasoning avoided speculative or unsupported statements.Source: A. Accuracy and Reasoning Quality · p. 31
• Readers assessed whether the AI identified a relevant finding they had initially missed and stated uncertainty and limitations when appropriate.Source: A. Accuracy and Reasoning Quality · p. 31

• The survey asked whether AI output reduced the time required to compose a report.Source: B. Time & Efficiency · p. 31
• Ratings were high for correctness, avoiding unsupported statements, and acknowledging uncertainty, but lower for identifying initially missed findings.Source: Figure 9 · p. 32
• Readers were also asked whether AI helped them decide faster and to estimate time saved.Source: B. Time & Efficiency · p. 32

5.5.1. Reader-assessment results

• 4.75 ± 0.55 ratings for correct key observations and avoiding unsupported statements contrasted with 2.60±1.23 for identifying a finding initially missed.Radiologists attributed the lower missed-finding rating to careful initial review; whether this differs under routine workloads remains to be evaluated.Source: §5.5.1 · p. 32
• 50% reduction in average reported time, from 26.25 min without AI to 13.13 min with AI.Readers could copy and edit the AI-generated structured reports.Source: §5.5.1 · p. 32, Figure 10 · p. 33
• 5.00 ± 0.00 for reduced report-composition time and 4.85 ± 0.67 for reaching a decision faster.Source: §5.5.1 · p. 33, Figure 11 · p. 33

C. Trust and Confidence

• 4.70 ± 0.57 ratings indicated increased confidence, while willingness to re-examine after disagreement received 5.00 ± 0.00.Source: §5.5.1 · p. 33

5.5.1. Reader-assessment results

• The findings reflect preliminary reader perceptions; diagnostic accuracy and clinical safety require separate evaluation.Source: §5.5.1 · p. 33

• Generated findings or explanations may be unsupported by the image, and reviewable reasoning does not establish faithful representation of internal computation.Source: §6 · p. 34
• The model processes one fixed-size chest or abdominal crop, so findings outside the crop cannot be recovered and unusual coverage can yield a suboptimal field of view.Source: §6 · p. 34
• Training does not capture full clinical diversity; rare findings are underrepresented, report-derived targets inherit ambiguity, and direct expert reasoning supervision is limited.Source: §6 · p. 34
• Evaluations do not systematically incorporate clinical indications, laboratory results, patient history, or prior examinations used by radiologists.Source: §6 · p. 34
• Automated comparisons depend on dataset and evaluator choices, while the preliminary reader study's fixed unaided-first order may introduce recall or reading-order effects.The study is not a prospective assessment of clinical safety or effectiveness; larger multicenter studies are required.Source: §6 · pp. 34–35

• The native 3D encoder passes all visual tokens and explicit 3D positions to the language model, supporting classification, report generation, and interactive questions with reviewable explanations.Source: §7 · p. 35
• The curated corpus combines reports, targeted questions, interactions, and expert and report-grounded synthetic narrations, followed by SFT and GRPO with region-aware verifiable rewards.Source: §7 · p. 35
• 50% reduction in average reported interpretation and reporting time accompanied favorable expert-radiologist confidence assessments in a preliminary study.Source: §7 · p. 35

A. Per-abnormality CT-RATE classification results

• Per-abnormality CT-RATE F1 and AUROC results are provided in Tables A1 and A2 to make label-level performance and evaluation protocols auditable.The supplied passages do not include the tables’ cell values.Source: A. Per-abnormality CT-RATE… · p. 35, Table A1 · pp. 35–36, Table A2 · pp. 36–37

B. Per-abnormality CT-RATE report-derived results

• Report-derived per-abnormality F1 values document finding coverage, but non-identical cohorts and extraction implementations preclude a controlled ranking.Some COLIPRI-CRM and VoxelFM entries were digitized from figures and are approximate.Source: B. Per-abnormality CT-RATE… · p. 37, Table B1 · p. 38

C. Per-abnormality Merlin classification results

• Merlin per-finding F1 and AUROC use a balanced cohort and uniform prompt, with F1 thresholded at 0.5 and AUROC based on continuous normalized Yes probabilities.Six lower-thoracic findings use the chest region; the other 24 use the abdomen, and prompt selection and region policy used the released test cohort.Source: C. Per-abnormality Merlin… · p. 38, Table C1 · p. 39

D. Model Output Examples

• The output examples show a structured chest report and reasoning response generated from an anatomy-aware cubic chest crop.The input is a 384 × 384 × 384-mm crop, outlined in green in the visualization.Source: D. Model Output Examples · p. 40

D.1. Structured chest CT report

• The impression identifies a right lower-lobe mass with satellite nodules, extensive adenopathy across thoracic and lower-neck stations, and destructive T10 paraspinal soft-tissue involvement.The report also notes bilateral ground-glass opacities with infection, inflammation, or tumor spread as possibilities.Source: §D.1 · p. 41
• Bilateral ground-glass opacities are described as potentially infectious, inflammatory, or tumor-related, and the reasoning states that CT cannot confirm infection.The response retains tumor-related inflammation or spread as a possibility given the mass and extensive adenopathy.Source: §D.1 · p. 41
• Follow-up responses localize the dominant mass and satellite nodules to the right lower lobe and identify bulky right supraclavicular and subcarinal adenopathy among multiple involved nodal regions.With fever and cough, the response raises infection as plausible but retains tumor-related spread or treatment-related change as possibilities.Source: §D.1 · pp. 41–42
• For the mass differential, the response favors primary lung malignancy with nodal and possible metastatic disease, while retaining metastatic disease from another primary as an alternative.Infection or inflammation may contribute to the ground-glass opacities but is less likely to explain the mass and extensive adenopathy together.Source: §D.1 · p. 42

D.2. Chest CT reasoning output

• The right lower-lobe mass has partial collapse and adjacent opacities that may reflect inflammatory, postobstructive, or nodular change, which the response says are difficult to separate precisely.Associated calcifications are not treated as reassuring in the setting of the mass and extensive adenopathy.Source: §D.2 · p. 42
• Patchy left-lung ground-glass opacities are favored to be inflammatory or infectious, but the response notes imaging alone cannot be definitive.There is no pleural effusion or pneumothorax despite the large right lower-lobe process.Source: §D.2 · pp. 42–43
• The reasoning identifies bulky mediastinal and right hilar adenopathy as concerning for nodal disease in the setting of the right lower-lobe mass.The left lung has patchy ground-glass opacities, while the dominant nodal disease is on the right.Source: §D.2 · p. 43
• Sclerotic thoracic vertebral lesions are described as concerning for osseous metastases, with no visible paravertebral soft-tissue component along those lesions.The response also notes normal heart size and no pericardial effusion or acute great-vessel abnormality.Source: §D.2 · p. 43
• The overall synthesis describes a large right lower-lobe mass with partial collapse, bulky right hilar and mediastinal adenopathy, and sclerotic thoracic vertebral lesions concerning for metastases.Patchy left-lung ground-glass opacities are separately favored to be inflammatory or infectious in appearance.Source: §D.2 · p. 43

D.3. Structured abdominal CT report

• The impression identifies bilateral adrenal masses, multiple hypoattenuating hepatic lesions, and aggressive iliac lytic lesions as compatible with or suspicious for metastasis.The right iliac lesion involves the sacroiliac joint, and an additional lesion is noted in the left iliac bone.Source: §D.3 · p. 44
• The report also notes a left upper-quadrant peritoneal nodule and prominent mesenteric lymph nodes that may be reactive or metastatic.The abdominal report describes no ascites or pneumoperitoneum.Source: §D.3 · p. 44
• Follow-up describes multiple liver lesions in both hepatic lobes, with a dominant right-lobe lesion, and favors metastatic malignancy given accompanying adrenal, peritoneal, and iliac bone lesions.Multifocal primary hepatic malignancy or inflammatory lesions are described as less likely possibilities.Source: §D.3 · p. 45
• The response connects bilateral adrenal masses, a peritoneal nodule, prominent mesenteric nodes, and iliac lytic lesions with suspected metastatic disease, while noting CT cannot provide histology.The pattern is concerning for metastases, not a tissue diagnosis.Source: §D.3 · p. 45

D.4. Abdominal CT reasoning output

• The review followed a systematic abdominal search pattern, moving from solid organs to bowel, peritoneum, vessels, lymph nodes, abdominal wall, and bones.The stated approach was to review the CT volume, use valid technique context, identify key findings, and provide final labels.Source: §D.4 · p. 45
• Multiple liver lesions, a peripancreatic mass, bilateral adrenal masses, and a right renal mass were suspicious for metastases despite no biliary, pancreatic-duct, or urinary obstruction.The dominant right hepatic lesion has central hypoattenuation and possible central necrotic change.Source: §D.4 · pp. 45–46
• Multiple peritoneal and mesenteric masses, including a dominant left upper abdominal mass, were concerning for metastatic disease; bowel obstruction, pneumoperitoneum, and ascites were absent.The absence of ascites did not diminish the significance of the nodular masslike deposits.Source: §D.4 · p. 46
• A large right inguinal lymph node was suspicious in context, while the abdominal aorta was normal in caliber and the abdominal vasculature appeared patent.Source: §D.4 · p. 46
• Multiple lytic and sclerotic lesions, including at L2 and the right sacrum, were concerning for osseous metastases; no acute fracture, abdominal-wall hernia, or soft-tissue mass was seen.Source: §D.4 · p. 46
• Multiple lesions involving the liver, peripancreatic region, both adrenal glands, right kidney, peritoneum, mesentery, right inguinal node, and bones were considered most consistent with widespread metastatic disease.The bone lesions show mixed lytic and sclerotic features.Source: §D.4 · p. 46