Source-linked AI summary
FZ-VLM: A Two Stage Florence-Zephyr Vision Language Model Framework for Pulmonary Nodule Characterization and Clinical Decision Making
Pramit Dutta, Jenita Manokaran, Richa Mittal, Ryan Appleby, Eranga Ukwatta
TL;DR
Pulmonary nodule assessment remains time-consuming, variable, and fragmented across isolated AI tasks. FZ-VLM separates CT attribute extraction from language-based clinical reasoning, achieving strong structured characterization while supporting accurate and complete—but assistive—interpretation.
Problem
Pulmonary nodule assessment is time-consuming and variable, while existing AI methods often lack a unified framework for multiple radiological attributes and clinical interpretation.
Method
FZ-VLM uses Florence-2 to extract structured nodule attributes from selected CT slices and Zephyr-7B to generate grounded clinical interpretations.
Results
77.18% accuracy was achieved for anatomical location, while Stage 2 generated accurate and complete outputs but showed lower clinical relevance and a 77.3% safety pass rate.
Takeaways & Limitations
FZ-VLM supports traceable, assistive post-detection pulmonary nodule interpretation, but it does not replace radiologist judgment.
Takeaways & Limitations
The framework relies on expert-selected two-dimensional slices and external assessment used a small veterinary dataset rather than an independent human cohort.
Abstract
from arXiv · showhide
Lung cancer remains one of the leading causes of cancer-related mortality worldwide, and Computed Tomography (CT) is a primary imaging tool for screening and followup assessment. After pulmonary nodule detection, radiologists manually assess anatomical location, diameter, margin characteristics, and attenuation type to support risk assessment and clinical decision-making. However, this post-detection workflow is time-consuming and can be affected by inter-observer variability. Existing Artificial Intelligence methods often focus on isolated tasks, limiting their use as a unified, clinically grounded interpretation framework. This study presents FZ-VLM, a two-stage Florence-Zephyr Vision Language Model framework for unified structured pulmonary nodule characterization in lung CT. The framework uses a fine-tuned Florence-2 model to extract radiological attributes from expert-annotated 2D axial CT slices, while a Zephyr-7B model uses these attributes to generate nodule descriptions, follow-up recommendations, and longitudinal analyses. Results showed that the Stage 1 model achieved 77.18\% accuracy for anatomical location, 67.96\% accuracy for margin characteristics, and 79.13\% accuracy for attenuation type, with a Mean Absolute Error of 2.58 mm for diameter estimation, outperforming evaluated GPT-4-based baselines as well as the human baseline. Expert radiologist evaluation of Stage 2 showed 93.9\% accuracy, 98.6\% completeness score, 76.1\% clinical relevance, and an overall score of 89.5\%. Safety analysis showed that most outputs were clinically safe, although some follow-up recommendations still required expert review. To the best of our knowledge, this study presents the first two-stage Vision-Language Model framework for structured nodule characterization and clinical decision-making.
Introduction
Lung cancer screening and follow-up rely on LDCT, but post-detection nodule characterization is time-consuming, variable, and fragmented across task-specific methods. FZ-VLM addresses this need with a two-stage framework connecting visually grounded attribute extraction to structured clinical interpretation, supported by an NLST-derived VQA dataset and interpretability analyses.
- Clinical context: Nearly 1.8 million deaths each year make lung cancer the leading cause of cancer-related mortality worldwide, while LDCT is the primary imaging modality for screening and follow-up.Early diagnosis is associated with approximately 65.5% five-year survival versus less than 29.5% overall survival.
- Clinical challenge: 58% of pulmonary nodule cases achieved unanimous correct classification among expert radiologists, highlighting interobserver variability in manual attribute extraction.The workflow evaluates anatomical location, diameter, margin characteristics, and attenuation type after nodule detection.
- Limitations: Most existing deep learning methods remain task specific and do not provide unified structured pulmonary nodule characterization.Reported applications include detection, segmentation, location assessment, diameter estimation, margin characterization, and attenuation classification.
- VLM motivation: Structured VQA can focus pulmonary nodule analysis on location, size, margin, and attenuation, but medical VLMs may produce weakly grounded responses, hallucinations, and unsafe recommendations.The introduction identifies image-level and language-level explanations as important for clinical use.
- Proposed framework: FZ-VLM uses fine-tuned Florence-2 for radiological attribute extraction from expert-selected 2D axial CT slices and a second stage for structured language-based clinical interpretation.The study also contributes an NLST-derived VQA dataset and evaluates quantitative performance, explainability, slice sensitivity, baseline comparison, external generalization, and expert review.
Methods
FZ-VLM is a two-stage framework that separates Florence-2 radiological attribute extraction from Zephyr-7B clinical language generation. Its methods include curated low-dose CT data, hierarchical integration across nodule, study-year, and patient levels, constrained clinical prompting, and separate evaluation of both stages.
- Dataset preparation: 8,540 image-question-answer triplets were curated from National Lung Screening Trial low-dose chest CT examinations and split into training (7,332), validation (384), and held-out test (824) sets.Splitting was performed at the patient level, with cases lacking confirmed disease status or complete annotations excluded.
- Framework architecture: FZ-VLM uses domain-adapted Florence-2 to extract radiological attributes and Zephyr-7B to generate structured descriptions, recommendations, and longitudinal interpretations.The two-stage design separates image perception from clinical language generation, enabling independent evaluation and traceability to extracted findings.
- Stage 1: Attribute extraction: Florence-2 large receives windowed axial CT images paired with standardized questions targeting anatomical location, diameter, margin characteristics, or attenuation type.The visual model contains approximately 0.77 billion parameters and uses a DaViT vision encoder with a multimodal encoder-decoder architecture.
- Hierarchical integration: Extracted attributes are hierarchically assembled into nodule representations, study-year summaries, and chronologically ordered patient records for description, recommendation, and temporal analysis.The integration operates at the nodule, screening-examination, and patient levels.
- Stage 2: Clinical interpretation: Zephyr-7B uses standardized prompts without additional fine-tuning to perform nodule description, follow-up recommendation, and longitudinal analysis from Stage 1 attributes.System instructions required responses to rely exclusively on structured findings, excluding unavailable demographic, pathological, and outcome information.
- Evaluation strategy: Unseen test data were used to evaluate both stages separately, combining Florence-2 quantitative, explainability, sensitivity, baseline, and external-validation analyses with expert radiologist review of Zephyr-7B outputs.Stage 1 metrics included accuracy, precision, recall, F1-score, confusion-matrix analysis, and diameter mean absolute error; Stage 2 review assessed accuracy, completeness, and clinical relevance.
Results
The Results section evaluates FZ-VLM in two stages: radiological attribute extraction and generated clinical interpretation via expert radiologist review. It also reports explainability, slice sensitivity, baseline comparison, and external generalization analyses.
- Evaluation framework: Stage 1 was assessed for radiological attribute extraction.
- Evaluation framework: Stage 2 was evaluated through expert radiologist review of generated clinical interpretation.
- Additional analyses: Additional analyses examined explainability, slice sensitivity, baseline comparison, and external generalization.
Quantitative Performance Analysis of Florence-2
Fine-tuned Florence-2 achieved 74.42% overall accuracy for radiological attribute extraction, with strongest performance for attenuation type and an MAE of 2.58 mm for diameter prediction. Diameter accuracy increased from 21.4% exact match to 87.9% within ±5 mm tolerance.
- Categorical attribute extraction: 74.42% overall accuracy was achieved for radiological attribute extraction, including 77.18% for anatomical location, 67.96% for margin characteristics, and 79.13% for attenuation type.Anatomical location had the strongest macro F1-score among the categorical tasks, while margin characteristics showed moderate performance.
- Diameter prediction: 2.58 mm MAE and 3.73 mm SD were achieved for diameter prediction.The median absolute error was 1 mm.
- Diameter prediction: 87.9% accuracy was achieved within ±5 mm tolerance, compared with 21.4% at exact match.The model performed reliably within small measurement tolerances despite relatively low exact-match accuracy.
Explainability Analysis Using Attention Heatmaps
Attention heatmaps were used to examine how fine-tuned Florence-2 focused on nodule-related regions during radiological attribute extraction. Representative cases showed mostly reliable categorical predictions, alongside diameter, attenuation, and broader attribute errors.
- Heatmap Analysis: Attention heatmaps visualized image regions receiving higher model attention during radiological attribute extraction.Representative cases were examined to understand nodule-focused behavior, correct predictions, and prediction errors across anatomic attributes.
- Representative Cases: Cases 1–4 showed mostly reliable categorical attribute extraction, with Cases 2 and 4 completely matching reference attributes.In Cases 1 and 3, anatomical location, margin, and attenuation were correct, while diameter differed slightly from the reference.
- Representative Cases: Case 5 correctly identified location and margin but overestimated diameter and misclassified attenuation.The error pattern suggests attention to a broader visual region or surrounding opacity during size and density estimation.
- Representative Cases: Case 6 showed a severe failure across anatomical location, margin, attenuation, and diameter estimation.The output did not match the reference labels for the categorical attributes, and the diameter estimate was substantially different.
Comparison with Baseline Models
The fine-tuned Florence-2 model was compared with pre-trained and domain-specific fine-tuned GPT-4 baselines using the same test set and protocol. It achieved the best reported performance across radiological attributes, including lower diameter-estimation error and higher classification accuracy.
- Evaluation design: The comparative analysis evaluated a pre-trained GPT-4 model, a domain-specific fine-tuned GPT-4 model, and human reference values from prior radiological studies.The comparisons assessed task-specific fine-tuning and model behaviour across vision-language systems.
- Location classification: 77.18% accuracy for location classification exceeded both pre-trained GPT-4 and fine-tuned GPT-4 baselines.The comparison used the same test set and evaluation protocol across models.
- Diameter estimation: 2.58 mm MAE for diameter estimation was lower than pre-trained GPT-4 at 4.74 mm and fine-tuned GPT-4 at 3.83 mm.The proposed model was also below the reported human interreader variability of 3.2 mm.
- Margin characterization: 67.96% accuracy for margin characterization exceeded pre-trained GPT-4 at 38.42% and fine-tuned GPT-4 at 39.47%.The passage describes margin classification as difficult for GPT-4-based models.
- Attenuation classification: 79.13% accuracy for attenuation classification exceeded both GPT-4 baselines and the reported human baseline range of 58 – 77%.Attenuation classification was described as visually challenging when solid and subsolid appearances overlap in a 2D CT slice.
Slice Selection Sensitivity Analysis
Slice-selection sensitivity analysis found that all four prediction tasks performed best on the expert-annotated reference slice and generally declined as the input moved up to three slices away. Anatomical location was most sensitive, while attenuation was comparatively more robust.
- Evaluation setup: Sensitivity was evaluated by shifting the input up to three slices in both directions from the expert-annotated reference slice and measuring performance changes.The analysis examined input slice offsets relative to the expert-annotated reference slice.
- Overall sensitivity: All four prediction tasks achieved their best performance when the expert-annotated slice was used as input, with the poorest results at a three-slice offset.Performance generally declined as the slice moved away from the reference position.
- Task-specific findings: 78% to 47%: Anatomical location accuracy decreased at both −3 and +3 slices, showing the greatest sensitivity to slice displacement.The decline occurred from the reference condition to offsets of three slices in either direction.
- Task-specific findings: 68% to 48%: Margin classification accuracy decreased at −3 slices relative to the expert-annotated reference slice.This result indicates reduced margin-classification performance when the input was shifted three slices negatively.
- Task-specific findings: 79% to 61%: Attenuation accuracy declined at the reported offset, but attenuation was comparatively more robust than the other characterized attributes.The passage reports attenuation as comparatively more robust, although the provided text does not specify the corresponding offset for the 61% value.
External Generalization Assessment
External testing on an independent veterinary CT dataset showed that FZ-VLM transferred some nodule features across imaging domains, but broader generalization remains preliminary and requires larger, more diverse datasets.
- External dataset: 16 independent veterinary CT slices generated 64 image-question-answer triplets using the same prompts and output categories as NLST, without training or validation overlap.Each slice contained one pulmonary nodule with annotations for anatomical location, diameter, margin characteristics, and attenuation type.
- External performance: 43.75% accuracy for anatomical location and 3.75 mm MAE for diameter estimation were achieved on the external dataset.Lower location accuracy likely reflected species-related anatomical differences and applying human lobar categories to veterinary images.
- External performance: 87.5% accuracy was achieved for both margin characteristics and attenuation type, suggesting greater transferability across the two imaging domains.These results may have been influenced by the external dataset’s larger proportion of smooth-margin and soft-tissue cases.
- Limitations: The findings provide preliminary evidence of cross-domain feature transfer, while larger and more diverse external datasets are needed for broader generalization claims.The small external dataset limits the strength of the generalization evidence.
Stage 2 Clinical Interpretation Performance
Stage 2 generated generally accurate and complete clinical interpretations from structured radiological attributes, with strongest performance in nodule description and follow-up recommendation. Longitudinal analysis, clinical relevance, and safety remained comparatively weaker, supporting continued expert review.
- Overall evaluation: 93.8% accuracy, 98.6% completeness, 76.2% clinical relevance, and 89.5% overall human score were achieved across 88 expert-evaluated outputs.Evaluation covered nodule description, follow-up recommendation, and longitudinal temporal analysis, including accuracy, relevance, completeness, and safety.
- Safety and limitations: 77.3% safety pass rate indicated that most outputs were clinically acceptable, but some required expert review before clinical use.Lower clinical relevance and safety were especially concerning for follow-up planning and temporal analysis.
- Task-wise performance: 100.0% accuracy and 100.0% completeness were achieved for nodule description, which received a 90.4% overall score.The task primarily converts structured attributes such as location, diameter, margin, and attenuation into clinical text.
- Task-wise performance: 99.3% accuracy, 100.0% completeness, 78.7% relevance, and a 92.7% overall score made follow-up recommendation the highest-scoring task overall.The model used structured nodule attributes to generate management-oriented outputs, though recommendations still require review.
- Task-wise performance: 83.2% accuracy, 96.1% completeness, 78.1% relevance, and the lowest overall score were reported for longitudinal temporal analysis.This task required comparing multiple screening rounds, identifying interval changes, and summarizing temporal progression.
Discussion
FZ-VLM links Florence-2 visual attribute extraction with Zephyr-7B clinical interpretation in a traceable two-stage framework. Results support its potential as a post-detection assistive tool, while safety, temporal reasoning, slice dependence, and external-validation limitations require radiologist oversight.
- Framework contribution: FZ-VLM separates visual attribute extraction from language-based reasoning, creating a traceable connection between CT evidence and clinical interpretation.Florence-2 extracts radiological attributes, and Zephyr-7B uses them for downstream clinical interpretation.
- Stage 1 performance: 77.18% accuracy was achieved for anatomical location, while diameter estimation reached a 2.58 mm mean absolute error.The median absolute error was 1 mm, and 72.3% of predictions were within ±2 mm of the reference measurement.
- Stage 1 performance: Subtle visual overlap between margin and attenuation classes likely contributed to model errors and disagreement during expert assessment.Poorly defined and spiculated margins can share irregular boundary features, while mixed attenuation combines ground-glass and soft-tissue components.
- Interpretability: Attention heatmaps indicated that correct predictions generally relied on clinically relevant regions around the nodule or surrounding lung.This interpretability feature helps reduce the framework’s black-box nature by showing image regions contributing to decisions.
- Limitations: Performance was highest on expert-selected reference slices and generally decreased as inputs moved farther from that position.This supports representative key slices while motivating automated slice selection or volumetric analysis; the framework also operates only after nodule detection.
- Stage 2 performance and safety: Expert evaluation found high Stage 2 accuracy, completeness, and overall scores, but clinical relevance and safety remained lower, especially for follow-up and longitudinal outputs.Longitudinal analysis required tracking nodules across screening years, while inappropriate follow-up timing could affect patient management; expert review remains necessary.
Additional information
The study used NLST data obtained through the National Cancer Institute Cancer Data Access System under a formal agreement, with access subject to application and approvals. The authors declared no competing interests.
- Data access: NLST data were obtained through the National Cancer Institute Cancer Data Access System under project NLST-871.The data were provided under a formal Data Transfer Agreement.
- Data access: The dataset cannot be publicly redistributed, and researchers must apply independently subject to data-access requirements and institutional approvals.
- Declarations: The authors declare no competing interests.