PaperlayerView exact revision on arXiv

Source-linked AI summary

MedTVL: Harnessing Vision and Language for Medical Time Series Classification

Jiexia Ye, Jia Li, Fugee Tsung

arXiv:2608.28605v1cs.AIcs.CV

TL;DR

Medical time-series classification needs richer integration of temporal, visual, and clinical-text evidence than existing largely bi-modal approaches provide. MedTVL uses text-guided convolutional and transformer pathways with instance-adaptive fusion and supports multimodal contrastive learning. Across diverse datasets and learning settings, it outperforms state-of-the-art baselines and shows transferability, while remaining a decision-support framework that cannot replace qualified clinical judgment.

  • Problem

    Existing medical time-series methods largely focus on unimodal or bi-modal interactions, leaving tri-modal temporal, visual, and language integration insufficiently explored despite complementary diagnostic evidence.

  • Method

    MedTVL combines convolutional temporal and transformer-based visual pathways, adaptively injects medical textual semantics, and uses Mixture-of-Experts fusion with multimodal contrastive learning.

  • Results

    MedTVL consistently outperforms competing methods across datasets and metrics, with average F1 improvements of approximately 3%–5% over second-best models.

  • Takeaways & Limitations

    MedTVL demonstrates robustness, adaptability, and generalizability for medical time-series clinical decision support across supervised, few-shot, and contrastive settings.

  • Takeaways & Limitations

    MedTVL is a research and decision-support framework whose predictions require qualified professional interpretation and must not replace clinical judgment.

Abstract

from arXiv · show

Recent advancements in multimodal learning for medical time series (MedTS) classification highlight the benefits of integrating complementary modalities for clinical decision. However, existing methods typically focus on bi-modal interactions (e.g., time series and text), leaving the tri-modal synergy between time series, vision, and language largely unexplored. Inspired by diagnostic practice synergizing numerical assessment, visual inspection and clinical context, we introduce MedTVL, a text-guided dual-pathway architecture tailored for MedTS classification. Specifically, it synergizes a convolution-based temporal pathway for fine-grained temporal dynamics from raw numerical sequences and a transformer-based visual pathway for holistic morphological structures from time-series-derived images. Such combination of cross-modal and architectural heterogeneity provides a comprehensive diagnostic perspective. To further resolve potential diagnostic ambiguity, both pathways are guided by adaptive medical textual semantics. Finally, a Mixture-of-Experts mechanism dynamically routes each instance to specialized fusion experts, capturing instance-specific reliance on the temporal and visual pathway outputs. In addition, MedTVL supports multimodal contrastive learning to mitigate the clinical label scarcity challenge. Extensive experiments across multiple medical datasets and tasks, spanning supervised, few-shot, and contrastive learning settings, demonstrate the superiority and transferability of MedTVL, highlighting its potential for robust clinical decision support.

1 Introduction

MedTS diagnosis combines numerical, visual, and textual evidence, but existing methods largely omit explicit visual modeling or tri-modal integration. MedTVL addresses this gap with text-guided heterogeneous pathways, instance-adaptive fusion, and multimodal contrastive learning.

  • Motivation: MedTS supports critical healthcare tasks but exhibits heterogeneous signals, multi-scale patterns, and scarce clinical labels.These challenges include transient temporal fluctuations alongside holistic morphological structures.
  • Research gap: Unimodal numerical models and time-series–text methods capture incomplete diagnostic evidence when visual morphology is not explicitly modeled.Time-series–vision methods in general domains also lack textual grounding and medical specialization.
  • Clinical perspective: Clinical diagnosis integrates raw temporal signals, visual morphology, and textual context as complementary views.Temporal inspection identifies fine-grained patterns, visual inspection assesses holistic morphology, and clinical text supplies patient context.
  • MedTVL: MedTVL combines convolutional temporal and transformer-based visual pathways, with adaptive textual guidance and instance-level Mixture-of-Experts fusion.The temporal pathway models raw numerical dynamics, while the visual pathway models morphology from CWT images.
  • Contribution: MedTVL is presented as the first tri-modal framework for general-purpose medical time series classification.The framework is designed to capture clinical diagnostic perspectives across temporal, visual, and textual views.
  • Evaluation: Experiments span multiple medical tasks and supervised, few-shot, and contrastive learning settings, with MedTVL outperforming state-of-the-art baselines.The reported results also highlight its capability to address clinical label scarcity.

2 Related Work

Related work covers unimodal medical time-series modeling, bi-modal time-series–text learning, and emerging tri-modal representation learning. Existing approaches remain limited by signal-specific specialization, incomplete modality integration, or restricted fusion flexibility.

  • Unimodal Medical Time Series Classification: Unimodal medical time-series methods use CNNs, RNNs, GNNs, Transformers, and image-based representations, but often generalize poorly across signal types.Many approaches are tailored to specific signals such as ECG.
  • Bi-modal Time Series Representation Learning: Bi-modal research predominantly aligns time series with text, including medical supervised learning and report-guided ECG contrastive learning.These methods typically overlook explicit visual modeling.
  • Tri-modal Time Series Representation Learning: Tri-modal time-series representation learning remains scarce, with reported systems including Time-VLM and GEM.The related-work discussion identifies only limited prior exploration of tri-modal frameworks.
  • Tri-modal Time Series Representation Learning: Time-VLM prioritizes forecasting with temporal-dominant attention and applies shared gated fusion weighting across instances.This design may limit full exploitation of visual and textual inputs and instance-specific fusion flexibility.
  • Contrastive Learning for Time Series: Time-series contrastive learning has progressed from positive-pair construction to augmentation-based multi-view alignment and methods exploiting temporal properties.Medical adaptations remain frequently tailored to specific signal types such as EEG.

3 Methodology

MedTVL integrates temporal, visual, and textual information through dual pathways for medical time series classification. Convolutional temporal modeling captures fine-grained dynamics, while transformer-based visual modeling captures global morphology; adaptive text guidance and MoE fusion support pathway integration.

  • Problem formulation: MedTVL formulates medical time series classification as tri-modal prediction from raw signals, derived visual representations, and clinical semantics.The framework maps X, V, and s to class predictions using trainable parameters.
  • Temporal pathway: The temporal pathway uses convolutional modeling to capture localized, multiscale pathological dynamics from raw numerical signals.InceptionTime uses parallel filters with different temporal spans, bottleneck convolutions, and residual connections.
  • Visual pathway: The visual pathway converts multichannel time series into CWT-based grid images and uses a Swin Transformer to model holistic spectral-spatial morphology.Channel scalograms are arranged in a structured grid, while hierarchical shifted-window processing extracts visual embeddings.
  • Adaptive textual guidance: Adaptive textual guidance projects medical-language embeddings into pathway-specific spaces and selectively fuses them with temporal and visual representations.The resulting text-enhanced pathway representations provide clinical semantics for subsequent fusion.
  • Instance-adaptive fusion: An MoE fusion module aligns the pathway representations, combines shared and instance-specific experts, and routes each sample to its most relevant specialized subnetworks.Top-K sparse gating produces a fused representation that is passed to an MLP classifier.

4 Experiments

Experiments evaluate MedTVL across clinical datasets, supervised and few-shot transfer, contrastive learning, efficiency, ablations, backbone choices, and visual representations. Across these settings, the reported results support strong performance, transferability, and a balance between effectiveness and computational cost.

  • Experimental Setup: Experiments span Alzheimer’s, epilepsy, and cardiac-disease datasets, including coarse- and fine-grained diagnostic settings.The datasets include EEG databases APAVA, ADFTD, and TUSZ, plus ECG databases PTB and PTB-XL.
  • Supervised Learning: MedTVL consistently outperforms competing methods across datasets and metrics, improving average F1 by approximately 3%–5% over second-best models.The comparison includes classical, CNN, transformer, and multimodal baselines under unified training conditions.
  • Few-shot Learning: Few-shot experiments transfer models from label-rich source domains to sparsely annotated target domains under 5-, 15-, 25-, 35-, 45-, and 55-shot settings.The evaluated source-target pairs use datasets with identical timesteps and channels.
  • Contrastive Learning: In contrastive learning, MedTVL achieves the best results across metrics and surpasses the strongest baseline by 6.72% in PTB-XL (4-Classes).Linear probing evaluates embeddings with full-resource and data-sparse labeled training sets.
  • Ablation Study: Ablation results attribute approximately 3% and 2% F1 improvements to the MoE module and textual guidance, respectively.These findings are reported as evidence for instance-adapted fusion and the tri-modal design.
  • Efficiency Analysis: MedTVL achieves the highest F1 score with 3.23M trainable parameters while running faster than MedTsLLM, TimeVLM, and TimesNet.The efficiency comparison is conducted on TUSZ (4-Classes), alongside training time and model size.

5 Conclusion

MedTVL is a tri-modal framework for medical time series classification that integrates numerical signals, visual representations, and clinical language. Its heterogeneous dual-pathway design captures complementary temporal and morphological patterns, while supporting multimodal contrastive learning under limited annotations.

  • MedTVL integrates numerical signals, visual representations, and clinical language for medical time series classification.
  • Its convolutional and transformer-based pathways capture fine-grained temporal dynamics and global morphology from time-series-derived images.
  • The dual-pathway structure enables cross-modal positive pairing for multimodal contrastive learning under limited clinical annotations.
  • Experiments across diverse tasks demonstrate MedTVL’s robustness, adaptability, and generalizability for clinical decision support.

Limitations and Ethical Considerations

MedTVL is evaluated using publicly available, de-identified medical datasets and is intended as a research and decision-support framework rather than a standalone diagnostic system. Its limitations include dataset quality, representativeness, and potential biases.

  • MedTVL uses publicly available, de-identified datasets from prior research without collecting new human-participant data.
  • MedTVL is intended for research and decision support, and its predictions should be interpreted by qualified professionals rather than replace clinical judgment.
  • The framework remains subject to limitations related to dataset quality, representativeness, and potential biases.
  • Applying MedTVL beyond medical settings requires domain-specific ethical review and regulatory oversight.

7 Appendix

The appendix documents benchmark datasets spanning Alzheimer’s disease, epilepsy, and cardiac disease, along with their labels, clinical semantics, splits, preprocessing, and data sources.

  • 7.1 Datasets: The benchmark suite covers Alzheimer’s disease, epilepsy, and cardiac disease across EEG and ECG datasets.
  • 7.1 Datasets: APAVA contains 16-channel EEG recordings for distinguishing Alzheimer’s disease from healthy controls using dataset and task descriptions as clinical semantics.
  • 7.1 Datasets: Table 7 summarizes dataset statistics, splits, and data URLs.
  • 7.1 Datasets: ADFTD contains 16-channel EEG recordings with healthy-control, frontotemporal-dementia, and Alzheimer’s-disease classes.
  • 7.1 Datasets: PTB provides 15-lead ECG recordings for classifying healthy controls and myocardial infarction patients.
  • 7.1 Datasets: PTB-XL supplies public 12-lead ECGs with coarse- and fine-grained diagnostic labels and corresponding automatically generated clinical reports.
  • 7.1 Datasets: TUSZ contains 19-channel EEG recordings for epilepsy diagnosis with coarse- and fine-grained seizure taxonomies.
  • 7.1 Datasets: Train-validation-test splits follow prior work for APAVA, ADFTD, and PTB, while PTB-XL and TUSZ use a 60%-20%-20% strategy; preprocessing follows previous work.
Loading 2608.28605v1…