PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond Discrete Tokens: Preserving Numerical Continuity in Medical Vision-Language Models

Dianye Huang, Jiamei Guo, Yue Zhou, Mingde Li, Yi Xiong, Wei Wang, Nassir Navab, Zhongliang Jiang

arXiv:2610.32948v1cs.CVcs.AI

TL;DR

Manual ultrasound caliper placement increases workload and observer variability, while prompt-conditioned VLM measurement requires accurate numerical prediction. HAD-VLM combines autoregressive text generation with diffusion-based continuous measurement prediction and outperforms autoregressive baselines and direct MLP regression across four EchoNet-LVH tasks.

  • Problem

    Quantitative ultrasound relies on manual landmark identification and caliper placement, increasing workload and intra- and inter-observer variability; language-driven VLM measurement also requires accurate numerical prediction.

    Source: §I · p. 1

  • Method

    HAD-VLM combines autoregressive text generation with continuous diffusion-based measurement prediction, coupling caliper endpoint coordinates and length through a consistency mechanism.

    Source: §I · p. 2

  • Results

    HAD-VLM achieved MAEs of 1.36, 2.77, 3.05, and 1.16 mm for IVSd, LVIDd, LVIDs, and LVPWd, outperforming autoregressive baselines and direct MLP regression across evaluated metrics.

    Source: §VI · p. 8

  • Takeaways & Limitations

    The findings support the potential of HAD-VLM for accurate, language-guided quantitative medical image assessment.

    Source: §I · p. 2

    The framework currently focuses on linear measurements.

    Source: §V · p. 7

Abstract

from arXiv · show

Objective: To develop a unified vision-language model for prompt-conditioned quantitative ultrasound measurement that addresses the mismatch between continuous measurement targets and discrete token-based numerical prediction. Methods: This article presents HAD-VLM, a hybrid autoregressive-diffusion framework that combines autoregressive text generation with continuous measurement prediction within a shared backbone. Visual prompt tuning adapts the vision encoder to ultrasound images. A diffusion process predicts a five-component representation comprising caliper endpoint coordinates and measurement length. A post-processing strategy combining the proposed coordinate-length consistency mechanism with ensemble prediction is introduced to refine measurement estimates. We evaluate four echocardiographic measurement tasks on EchoNet-LVH, with additional assessments of medical question answering and X-ray vertebral measurement. Results: HAD-VLM achieves mean absolute errors of 1.36, 2.77, 3.05, and 1.16 mm for IVSd, LVIDd, LVIDs, and LVPWd, respectively, reducing errors by 21.4-37.6\% relative to the Vicuna-based autoregressive baseline. It also outperforms fine-tuned Qwen3-VL and direct MLP regression across all four tasks. Medical question answering performance is preserved on the evaluated dataset. For X-ray vertebral measurement, HAD-VLM reduces mean absolute error from 11.4 to 6.7 pixels. Conclusion: Integrating autoregressive language generation with diffusion-based continuous prediction improves quantitative medical image measurement while retaining prompt-conditioned task flexibility. Significance: HAD-VLM provides a unified approach to combining language interaction with anatomically grounded numerical prediction for quantitative medical image analysis.

I. INTRODUCTION

Quantitative ultrasound measurements require precise numerical prediction, but token-based vision-language models do not explicitly preserve numerical proximity or caliper geometry. HAD-VLM combines language generation with continuous prediction to support prompt-conditioned measurement.

I. INTRODUCTION

  • Natural-language prompts could specify ultrasound measurements through a shared vision-language interface, whereas many existing systems use predefined targets and output schemas.Manual landmark identification and caliper placement increase examination workload and are subject to intra- and inter-observer variability.Source: §I · p. 1
  • Continuous anatomical measurements conflict with token-based prediction, which does not explicitly represent numerical error magnitude or enforce consistency between caliper endpoints and measured length.Tokenization can split a value such as 10.1 into separate tokens, while digit or decimal errors can substantially alter the prediction.Source: §I · pp. 1–2
  • HAD-VLM combines autoregressive text generation with diffusion-based continuous measurement prediction and couples caliper endpoints with length in a five-tuple representation.A coordinate-length consistency mechanism encourages agreement between predicted geometry and measurement value.Source: §I · p. 2
  • For LVIDd and LVIDs, HAD-VLM reports MAEs of 2.77 and 3.05 mm, below separate-study inter-observer differences of 2.97 and 3.60 mm, respectively.The paper evaluates four echocardiographic measurements specified through natural-language prompts.Source: §I · p. 2

II. RELATED WORK

Prior work covers task-specific ultrasound measurement, medical vision-language systems, and hybrid continuous prediction. HAD-VLM adapts these directions to prompt-conditioned ultrasound measurement by linking predicted lengths to caliper geometry.

A. Deep Learning for Quantitative Ultrasound Assessment

  • Task-specific ultrasound systems estimate landmarks, segment anatomy, or regress clinical quantities, while their requested outputs are generally set by predefined task configurations.The section positions HAD-VLM as a language-conditioned shared model for specifying measurement targets.Source: A. Deep Learning for Quantitative… · p. 2

B. Vision-Language Models for Medical Image Interpretation

  • Medical vision-language systems support instruction-following and image-grounded language tasks, but success in question answering or report generation alone does not establish geometrically consistent numerical measurement.HAD-VLM models continuous measurements alongside autoregressive text and links predicted lengths to caliper coordinates.Source: B. Vision-Language Models for Medical… · p. 3

C. Continuous Prediction in Vision-Language Frameworks

  • Prior language-conditioned regression and hybrid autoregressive-diffusion models show that language-related processing can coexist with continuous prediction in shared architectures.Examples include REO-VLM, Transfusion, HybridVLA, and DiffusionVLA.Source: C. Continuous Prediction in… · p. 3
  • HAD-VLM adapts hybrid modeling to ultrasound by coupling endpoint coordinates and measurement length in a structured representation with an explicit consistency mechanism.The paper assesses diffusion against direct regression through an ablation.Source: C. Continuous Prediction in… · p. 3

III. METHOD

  • HAD-VLM uses a prompt-adapted vision encoder, projector, hybrid language model, and post-processing module to generate text and continuous ultrasound measurements.The numerical output uses a five-tuple representation.Source: §III · p. 3

A. Adapting Vision Encoder to US Data via Learnable Prompts

The vision encoder is adapted to ultrasound images using learnable prompts while retaining a frozen pretrained backbone. Its visual representation is formed from the class token and first global-prompt token before projection into the LLM embedding space.

A. Adapting Vision Encoder to US Data via Learnable Prompts

  • Visual Prompt Tuning adapts CLIP ViT-L/14 representations to ultrasound images while keeping the pretrained backbone frozen.The design uses four learnable local prompts at each Transformer layer and four global prompts at the first layer.Source: A. Adapting Vision Encoder to US Data… · p. 3
  • The visual representation concatenates the class token with the first global-prompt token.The prompted encoder produces class-token and global-prompt representations.Source: A. Adapting Vision Encoder to US Data… · p. 3
  • A two-layer MLP projects the visual representation into the LLM embedding space.The projector is initialized from LLaVA when input feature dimensions are compatible.Source: A. Adapting Vision Encoder to US Data… · p. 3

B. US Measurement with Hybrid LLM

HAD-VLM combines autoregressive text generation with diffusion-based prediction of continuous measurements in one model. Each measurement is a five-tuple of caliper endpoints and length, predicted from the preceding visual and textual context.

1) Model Architecture:

  • The model produces autoregressive text and diffusion-predicted numerical measurements from tokenized prompts and projected visual tokens.The LLM processes an embedding at the numerical prediction slot, and its hidden representation conditions the diffusion head’s noise prediction.Source: 1) Model Architecture · pp. 3–4
  • Each continuous measurement is a five-tuple containing normalized start and end coordinates and the corresponding length in pixels.A DDPM-based denoising process predicts the tuple from a noisy tuple.Source: 1) Model Architecture · p. 3
  • The diffusion head predicts noise, and the reverse sampling process recovers the continuous measurement.Source: 1) Model Architecture · p. 4

2) Numerical Prediction Post-Processing:

At test time, HAD-VLM checks agreement between coordinate-derived and directly predicted lengths, then ensembles independently sampled predictions to reduce diffusion-sampling variability.

2) Numerical Prediction Post-Processing:

  • A discrepancy between coordinate-derived and directly regressed lengths is compared with a validation-calibrated threshold to refine predictions.The threshold is the empirical 85th percentile of validation discrepancies and remains fixed during testing.Source: 2) Numerical Prediction Post-Processing · p. 4
  • Twenty independently initialized predictions are post-processed separately, and their lengths are combined by median aggregation.Endpoint predictions remain associated with their individual sampled tuples.Source: 2) Numerical Prediction Post-Processing · p. 4

3) Hybrid Model Inference:

HAD-VLM alternates autoregressive text generation with diffusion-based continuous measurement prediction, inserting measurements into the text sequence before decoding resumes.

3) Hybrid Model Inference:

  • Opening and closing square brackets switch between diffusion regression and autoregressive text decoding without adding vocabulary tokens.Diffusion regression conditions on preceding visual and textual tokens and avoids discretizing continuous numerical values.Source: 3) Hybrid Model Inference · p. 4
  • At each measurement slot, independently generated diffusion samples are post-processed into an ensemble estimate, and the length is inserted into the text sequence.The procedure also stores a representative five-tuple, making predicted measurements available as context for subsequent text generation.Source: 3) Hybrid Model Inference · p. 4

1) Loss Functions:

The model jointly optimizes a language-modeling objective for textual tokens and a diffusion denoising objective for continuous measurement regression. Training proceeds through alignment, regression, and joint optimization stages.

1) Loss Functions:

  • Textual tokens use autoregressive cross-entropy, while continuous measurements use a diffusion denoising objective that minimizes mean squared error.The diffusion model predicts noise from the LLM hidden state at the diffusion slot.Source: 1) Loss Functions · p. 4

2) Three-Stage Training Strategy:

  • Training first aligns visual prompts and the connector with the language loss while freezing CLIP and the LLM, then trains diffusion modules with LoRA on the last four LLM layers, and finally jointly optimizes both losses.Source: 2) Three-Stage Training Strategy · pp. 4–5

A. Experimental Setup

The evaluation uses prompt-conditioned cardiac measurements on EchoNet-LVH and compares HAD-VLM with token-based and direct-regression baselines. HAD-VLM improves measurement accuracy, while endpoint localization and numerical length accuracy do not always coincide.

1) Dataset:

  • EchoNet-LVH provides about 12,000 parasternal long-axis B-mode videos; the study targets IVSd, LVIDd, LVIDs, and LVPWd using endpoint annotations and phase labels.The data split is 87:3:10, with 40,690 annotated frames used for training; annotations were paired with diversified natural-language QA prompts.Source: 1) Dataset · p. 5

2) Training Details:

  • Training uses three stages: alignment for 15 epochs, regression for 30, and joint training for 15, with learning rates decreasing from 1×10−4 to 1×10−6.Optimization uses AdamW on one NVIDIA A100 GPU; LoRA is applied to the last four LLM layers during regression and joint training.Source: 2) Training Details · p. 5

B. Comparison with Baseline Methods

  • HAD-VLM reduces Vicuna-baseline MAE from 1.82 to 1.36 mm for IVSd, 3.62 to 2.77 mm for LVIDd, 3.88 to 3.05 mm for LVIDs, and 1.86 to 1.16 mm for LVPWd.The cited results report a 25.3% reduction for IVSd and a 23.5% reduction for LVIDd.Source: B. Comparison with Baseline Methods · p. 6
  • HAD-VLM has lower MAE than fine-tuned Qwen3-VL and direct MLP regression across all four tasks; its LVIDd and LVIDs MAEs are 2.77 and 3.05 mm.For those tasks, Qwen3-VL reports 3.71 and 3.82 mm, while the MLP reports 3.00 and 3.42 mm.Source: B. Comparison with Baseline Methods · p. 6
  • Close length agreement does not guarantee anatomically accurate caliper placement: spatially displaced segments can have small numerical errors.The examples distinguish endpoint localization from length accuracy, motivating assessment of both.Source: B. Comparison with Baseline Methods · p. 6

C. Ablation Studies

Ablations support contributions from ultrasound-specific visual prompt tuning and post-processing. The hybrid prediction framework improves on autoregressive prediction even before consistency refinement or ensembling.

C. Ablation Studies

  • Removing visual prompt tuning causes the largest ablation degradation across four tasks; LVIDd and LVIDs MAE rise from 2.77 to 5.07 mm and 3.05 to 5.60 mm.For IVSd, MAE rises from 1.36 to 1.93 mm and R2 falls from 0.56 to 0.11.Source: C. Ablation Studies · p. 6
  • Coordinate-length consistency improves most measurement targets, but its effect varies: LVPWd MAE falls from 1.31 to 1.23 mm while LVIDs rises from 3.31 to 3.34 mm.Disabling both post-processing components raises LVIDd MAE from 2.77 to 3.04 mm (9.7%).Source: C. Ablation Studies · p. 6
  • Adding ensemble prediction to the consistency-enabled variant improves all four metrics across all tasks, including LVIDd MAE from 2.96 to 2.77 mm.LVIDs MAE decreases from 3.34 to 3.05 mm; the corresponding R2 values rise from 0.75 to 0.78 and 0.72 to 0.77.Source: C. Ablation Studies · p. 6
  • Without consistency or ensembling, LVIDd MAE is 3.04 mm, versus 3.62 mm for the autoregressive baseline; the complete model reaches 2.77 mm.This comparison shows an improvement over the autoregressive baseline before post-processing, with further reduction in the complete framework.Source: C. Ablation Studies · p. 6

D. Single-Task versus Multi-Task Training

Prompt-conditioned multi-task training improves all four EchoNet-LVH measurements over independently trained models. The evaluation also reports preserved medical question answering and lower error for cervical vertebral measurement, reported in pixels.

D. Single-Task versus Multi-Task Training

  • Multi-task training improves MAE, RMSE, R2, and Pearson correlation across all four measurements; LVIDd MAE falls from 3.69 to 2.77 mm and LVIDs from 3.69 to 3.05 mm.The largest improvements occur for ventricular diameters; IVSd and LVPWd MAE also decrease, from 1.44 to 1.36 mm and 1.27 to 1.16 mm.Source: D. Single-Task versus Multi-Task… · pp. 6–7

E. Evaluation on Language Ability Preservation

  • On the evaluated medical QA benchmark, HAD-VLM raises word-level F1 from 0.298 to 0.320 and BERTScore F1 from 0.869 to 0.882.The reported BERTScore precision and recall also rise, from 0.855 and 0.885 to 0.873 and 0.891.Source: E. Evaluation on Language Ability… · p. 7

1) Dataset:

  • The cervical X-ray dataset contains 9,926 cropped C4 and C5 samples; lengths and errors are evaluated in pixels because physical-distance calibration is unavailable.The dataset uses an 85:15 training–test split.Source: 1) Dataset · p. 7

2) Results:

  • For C4/C5 superior endplate length, HAD-VLM reduces MAE from 11.4 to 6.7 pixels (41.2%) against the Vicuna-based autoregressive baseline.The standard deviation of absolute errors decreases from 93.9 to 5.7 pixels.Source: 2) Results · p. 7

V. DISCUSSION

HAD-VLM combines autoregressive text generation with diffusion-based continuous measurement prediction. The paper reports improved measurement performance and identifies linear measurements as its current scope.

V. DISCUSSION

  • HAD-VLM targets linear measurements; extending it to more complex geometric or area-based measurements is proposed as future work.The discussion notes that linear measurements include tumor size, vessel diameter, organ length, and myocardial wall thickness.Source: §V · p. 7
  • The authors identify higher-quality report supervision and evaluation on other imaging modalities, including X-ray, CT, and MRI, as potential directions.They propose a qualitative-to-quantitative workflow in which anatomical structures and boundaries are described before measurements.Source: §V · p. 7

VI. CONCLUSION

  • Across four EchoNet-LVH tasks, HAD-VLM reports MAEs of 1.36, 2.77, 3.05, and 1.16 mm, and reduces cervical vertebral X-ray MAE from 11.4 to 6.7 pixels.The conclusion also reports preserved medical QA performance on the evaluated benchmark and calls for validation across institutions and imaging conditions.Source: §VI · p. 8

Related papers

Loading 2610.32948v1…