PaperlayerView exact revision on arXiv

Source-linked AI summary

Revitalizing Medical Time Series with Vision-Informed Retrieval: A Vision-Language Perspective

Guoqi Yu, Juncheng Wang, Shujun Wang

arXiv:2609.34652v1cs.CV

TL;DR

Medical time-series models often rely on numerical sequences and underuse morphology visible in waveform inspection. ViRe uses a frozen CLIP waveform representation as a query to retrieve temporal and channel evidence from numerical signals. Across six public EEG and ECG benchmarks, it improves average performance over Medformer by 6.42%.

  • Problem

    Existing MedTS models usually represent signals as numerical sequences and underuse morphology that is explicit in waveform inspection.

    Source: §1 · p. 2

  • Method

    ViRe uses a frozen CLIP-derived waveform representation as a morphology-aware Query to retrieve temporal and channel evidence from numerical MedTS features.

    Source: §1 · p. 2

  • Results

    Across six public EEG and ECG benchmarks, ViRe achieves the best overall performance on five datasets and improves Medformer by 6.42% on average.

    Source: §5 · p. 10

  • Takeaways & Limitations

    Mechanism analyses favor vision-informed queries and CLIP-Vision over content-free queries and generic image pretraining, while PTB-XL and MEETI analyses link the visual prior to recognizable waveform structure.

    Source: §5 · p. 10

    Retrospective results on six public EEG/ECG benchmarks do not replace prospective multi-center clinical validation.

    Source: J Limitations and Societal… · p. 27

Abstract

from arXiv · show

Medical time series (MedTS) underpin many clinical classification tasks, yet existing methods usually represent them only as numerical sequences and underuse the morphology that is explicit in waveform inspection. To bridge this gap, we introduce Vision-Informed Retrieval (ViRe), which uses a frozen VLM-derived waveform representation as a morphology-aware Query to guide retrieval from raw numerical MedTS features. Specifically, a Vision Query is extracted using pre-trained vision-language models (VLMs) to obtain morphology-aware priors from waveform plots. A tailored attention-based cross-modal retrieval mechanism then uses the Vision Query to select morphology-relevant temporal and channel evidence from the numerical representation. ViRe demonstrates strong effectiveness against ten established baselines, yielding an overall 6.42% relative improvement over the previous state of the art across six public benchmarks. Code, training scripts, and reproducibility materials are publicly available in the GitHub Repository: https://github.com/Levi-Ackman/ViRe.

1 Introduction

ViRe addresses the gap between numerical MedTS models and morphology-centered clinical waveform review by using a frozen visual prior to retrieve evidence from numerical signals. Across six benchmarks, it outperforms strong baselines, including Medformer.

1 Introduction

  • Existing models treat MedTS as numerical sequences or matrices, underusing waveform morphology that clinicians use in diagnosis.The paper identifies a mismatch between numerics-oriented representations and waveform-based clinical reasoning.Source: §1 · p. 2
  • Training waveform representations aligned with human concepts from scratch is described as data-hungry and brittle under subject or device shifts.The paper motivates using existing VLM vision encoders to provide compact morphology-aware priors.Source: §1 · p. 2
  • 6.42% average improvement over Medformer across six MedTS benchmarks, using a frozen CLIP waveform representation to retrieve morphology-relevant numerical evidence.ViRe combines a numerical view with a waveform view encoded by a pre-trained frozen CLIP vision encoder.Source: §1 · p. 2
  • ViRe uses a frozen CLIP Vision Encoder to create morphology-aware priors and an asymmetric dual-path cross-attention mechanism to retrieve temporal and channel evidence.The Vision Query selects and aggregates morphology-relevant numerical tokens while temporal and channel features remain the retrieved evidence.Source: §1 · p. 2

2 Related Work

Prior MedTS work progressed from handcrafted features to deep models that capture temporal dependencies and inter-channel interactions, but remains primarily numerical. ViRe instead uses visual morphology to guide retrieval from numerical evidence.

2 Related Work

  • Deep MedTS models progressed from handcrafted features to convolutional, recurrent, and Transformer architectures, while remaining primarily numerical.Transformers capture longer temporal dependencies and inter-channel interactions.Source: §2 · pp. 2–3
  • Clinical interpretation is morphology-centric: clinicians review EEG and ECG waveforms and summarize visually recognized diagnostic cues.ViRe uses visual morphology as a retrieval prior for selecting clinically relevant evidence from numerical representations.Source: §2 · p. 3
  • Time-series VLM research includes image-centered forecasting, visual/textual augmentation, CLIP-style text alignment, and classification from visualizations.ViRe differs by using a frozen CLIP waveform representation as a global Vision Query that retrieves temporal and channel tokens.Source: §2 · p. 3

3 Methodology

ViRe formulates MedTS diagnosis as classification and uses a frozen CLIP representation of a rendered waveform to retrieve evidence from temporal and channel features. The retrieved summaries are fused to produce diagnostic logits.

3.1 Preliminaries

  • MedTS classification maps a T × C sample to one of K task-specific diagnostic labels.The label classes may represent disease types or stages of the same disease.Source: §3.1 · p. 3
  • The subject-independent setting assigns each patient’s samples exclusively to training, validation, or test splits so evaluation tests generalization to unseen individuals.The methodology aligns this split with subject-level clinical decision-making.Source: §3.1 · p. 3

3.2 The ViRe Framework

  • ViRe embeds raw MedTS along temporal and channel axes and uses a shared waveform-derived Vision Query to retrieve numerical evidence from both token streams.Cross-attention produces two summaries, which are added and projected to diagnostic logits; the visual branch supplies the Query while Keys and Values remain numerical.Source: §3.2 · pp. 3–5, Figure 2 · p. 4
  • Temporal tokenization splits the signal into non-overlapping segments across all channels, while channel tokenization summarizes each channel’s full trajectory.The temporal tokens support modeling trend and seasonality; channel tokens preserve channel-specific semantics for inter-channel correlations.Source: §3.2 · p. 4
  • The same sample is deterministically rendered as a waveform image and processed by frozen CLIP; its aligned visual feature serves as the shared Query for both numerical views.Training uses the standard cross-entropy classification objective, with augmentation applied consistently to the signal and its rendering.Source: §3.2 · pp. 4–5

4 Experimental Results

Across six public MedTS benchmarks, ViRe improves on strong numerical baselines, with its largest gains on smaller EEG cohorts and a near-tie on PTB-XL. Ablations and representation analyses point to the value of a CLIP-derived Vision Query for selecting numerical evidence.

4.1.1 Datasets

  • Evaluation spans six subject-disjoint EEG and ECG benchmarks, compares ten architectures, and uses matched baseline protocols with results averaged over five random seeds.The datasets cover varied cohort sizes and channel configurations; model selection uses validation subjects only.Source: §4.1.1 · p. 5, §4.1.2 · p. 5, §4.1.3 · p. 6

4.2 Classification Performance

  • ViRe’s overall mean Avg is 82.90 versus Medformer’s 77.90, and it ranks first on five of six benchmarks; PatchTST leads ViRe by 0.43 points on PTB-XL.Avg is the arithmetic mean of Accuracy, Precision, Recall, F1, AUROC, and AUPRC.Source: Table 2 · p. 6, §4.2 · p. 6

4.3 Model Analysis

  • Vision Query retrieval gives the strongest query-ablation gain at 7.72%; cross-attention raises fusion gains to 7.72%, and CLIP-Vision outperforms ImageNet ViT by 7.11%.Zero and Gaussian queries yield smaller average gains, simple Add and Concat also outperform the no-retrieval variant, and random ViT degrades average performance by 11.81%.Source: §4.3.1 · p. 7, Table 3 · p. 7, §4.3.2 · p. 7

4.3.4 Generalizability Analysis

  • Vision priors improve F1-Score at every training-set proportion, with the largest separation under limited supervision.The authors interpret this pattern as improved data efficiency and reduced reliance on abundant labeled subjects.Source: §4.3.4 · p. 8, Figure 3 · p. 8

4.4.1 Analysis of Retrieved Features

  • Vision-derived retrieved features have the best reported clustering scores and compact, separated class manifolds; the Vision Query lies near dense regions of temporal embeddings across datasets.The Query also exceeds a Gaussian Query on MAR and CAR across PTB, PTB-XL, and MIMIC, indicating greater attention-density enrichment on high-curvature and QRS regions.Source: §4.4.1 · p. 8, Table 6 · p. 8, Figure 4 · p. 8

4.5 Additional Clinical Evidence

  • On PTB-XL, ViRe improves 11 of 13 diagnoses, while zero masking and cross-patient visual shuffling reduce macro-F1 by 6.19 and 15.61 points, respectively.The interventions support the interpretation that the visual branch supplies a morphology-aware retrieval prior rather than bypassing numerical evidence.Source: Table 8 · p. 9, §4.5 · pp. 9–10
  • On MEETI, lightweight heads obtain positive confidence intervals for all 22 supported visual attributes; representation analyses also find gains across all 12 text directions, five longitudinal measurements, and 21 retrieval attributes.The frozen visual representation is evaluated without retraining on benchmark labels.Source: §4.5 · p. 10, Table 9 · p. 10, Table 10 · p. 10

5 Conclusion

ViRe combines a frozen CLIP waveform prior with numerical temporal and channel evidence, outperforming the strongest baseline on most benchmarks. Its retrieval mechanism adaptively pools numerical tokens, while its hypothesis class includes mean-pooled numerical modeling.

5 Conclusion

  • ViRe achieves the best overall performance on five of six public EEG and ECG benchmarks, improving on Medformer by 6.42% on average and 16.37% on APAVA.The reported results compare ViRe with the strongest baseline, Medformer.Source: §5 · p. 10
  • Mechanism analyses favor vision-informed queries and CLIP-Vision over content-free queries and generic image pretraining, while diagnosis and concept-decoding results link the visual prior to recognizable waveform structure.The conclusion characterizes ViRe as using waveform morphology to guide numerical feature selection.Source: §5 · p. 10

A Relationship to Numerical-Only Modeling

  • ViRe contains mean-pooled numerical models; when the visual representation adds positive conditional information given the numerical representation, the Bayes risk under log-loss is lower.The risk relation is stated under the positive conditional-information condition.Source: A Relationship to Numerical-Only… · p. 15
  • Cross-attention weights form a sample-dependent pooling rule jointly indexed by the Vision Query and numerical tokens, enabling selective aggregation over temporal or channel evidence.Uniform pooling is also within the operator family.Source: A Relationship to Numerical-Only… · p. 15
  • The temporal and channel retrieval paths emphasize complementary waveform evidence, then sum their representations before classification.The temporal path covers localized and long-range structure; the channel path covers cross-channel patterns and channel-specific morphology.Source: A Relationship to Numerical-Only… · p. 15

B Data Preprocessing and Train-validation-test Split

All six datasets use subject-level separation, with each subject's samples confined to one split and numerical inputs paired with images rendered from the same preprocessed segment.

B Data Preprocessing and Train-validation-test Split

  • APAVA is a two-class EEG benchmark with 23 subjects and 5,967 one-second windows; designated subjects form validation and test sets, with the rest used for training.Each window has 256 timestamps and is half-overlapping.Source: B Data Preprocessing and… · p. 16
  • ADFTD provides three-class, 19-channel EEG data from 88 subjects, resampled and segmented into 69,752 one-second windows before a subject-wise 60%/20%/20% split.Windows shorter than one second are discarded.Source: B Data Preprocessing and… · p. 16
  • TDBrain uses eyes-closed recordings from a balanced subset of 25 Parkinson’s disease subjects and 25 controls, yielding 6,240 one-second segments.The selected subjects are assigned to validation and test sets, with the remaining subjects used for training.Source: B Data Preprocessing and… · p. 16
  • The ECG datasets use distinct preprocessing: PTB and MIMIC are segmented into heartbeats, whereas PTB-XL is split into one-second windows.PTB uses 198 subjects and 64,356 heartbeat samples; PTB-XL uses 17,596 subjects and 191,400 windows; MIMIC yields 204,370 heartbeat samples. Each uses a subject-level 60%/20%/20% split.Source: B Data Preprocessing and… · p. 16
  • All six datasets fix subject-level partitions before preprocessing, keeping each subject in one split and deriving numerical inputs and waveform images from the same segment.This keeps the visual branch from seeing information unavailable to the numerical branch.Source: B Data Preprocessing and… · p. 16

C Data Augmentation

Training applies one randomly selected augmentation to the signal and its rendering together, while validation and testing use deterministic rendering without augmentation.

C Data Augmentation

  • Each training input receives exactly one randomly sampled operation from the six-operation augmentation pool.The pool and default settings are summarized in Table 11.Source: C Data Augmentation · p. 17, Table 11 · p. 17
  • The same transformation is applied to numerical and visual inputs, keeping masked intervals, channel permutations, and local perturbations synchronized.Only one operation is sampled per forward pass to avoid compounding perturbations into an unrealistic waveform.Source: C Data Augmentation · p. 17
  • Augmentation is disabled for validation and testing, which use the deterministic preprocessing and rendering pipeline.Source: C Data Augmentation · p. 17
  • Augmentations target clinical recording nuisances, including electrode dropout, missing samples, filtering or line noise, and low-amplitude sensor noise.Rendering is regenerated from the augmented signal so the visual branch encounters the same nuisance variation.Source: C Data Augmentation · p. 17

D.1 Implementation Details of All Baselines

The baseline comparison uses a shared evaluation pipeline, while ViRe keeps CLIP frozen and renders each multichannel sample as a standardized stacked waveform image.

D.1 Implementation Details of All Baselines

  • Ten Transformer baselines are compared in a shared pipeline with common subject-disjoint splits, preprocessing, evaluation metrics, and five random seeds.Architecture settings and optimizer are standardized for the non-Medformer baselines, while Medformer results use its official implementation.Source: §D.1 · p. 18

D.2 Implementation Details of the Proposed ViRe

  • ViRe keeps the CLIP encoder frozen, caching its features; only the numerical encoders, alignment layer, retrieval blocks, and classifier are optimized.The visual branch adds only a linear projection to the trainable parameter count.Source: §D.2 · p. 19
  • A single Vision Query attends over temporal and channel tokens at a retrieval cost of O((P + C)D) per sample.The passage characterizes this cost as negligible relative to quadratic self-attention over the two token sets.Source: §D.2 · p. 19

D.3 Visualization Operator

  • The visualization operator renders each channel on its own fixed-axis panel, removes decorative plot elements, and vertically concatenates the panels.The same deterministic operator is used for training, validation, and test samples.Source: §D.3 · pp. 19–20, Figure 5 · p. 20

E Full Classification Results

Table 13 presents full subject-independent classification results, reported as mean±std over five seeds, with best and second-best results visually distinguished.

E Full Classification Results

  • Table 13 reports full subject-independent classification results.The caption specifies mean±std over five seeds; best results are bolded and second-best results are shown in blue italics.Source: Table 13 · p. 21
  • Results are summarized as mean±std over five seeds.The caption marks the best result in bold and the second-best in blue italics.Source: Table 13 · p. 21
  • The table visually distinguishes the best and second-best results.Best results are bolded; second-best results appear in blue italics.Source: Table 13 · p. 21

F Computational Overhead Analysis

ViRe’s cached training is cheaper than Medformer on APAVA, while online inference carries added latency and memory cost; its retrieval attention also peaks at a salient ECG interval.

F Computational Overhead Analysis

  • 32.3% lower per-batch latency and 16.7% lower peak memory accompany a 14.88-point APAVA F1-Score increase versus Medformer in cached training.ViRe uses 211.9 ms and 698.2 MB per batch, compared with Medformer’s 313.1 ms and 838.4 MB.Source: F Computational Overhead Analysis · p. 22, Table 14 · p. 22
  • Online inference is 1.75× slower and uses 1.81× more memory than Medformer, while APAVA F1-Score improves by 14.88 points.The full online path takes 267.7 ms and 740.8 MB; rendering and CLIP encoding are performed online.Source: F Computational Overhead Analysis · p. 22, Table 14 · p. 22

G.1 Qualitative Retrieval Attention Visualization

  • On PTB, temporal retrieval attention peaks around a morphologically salient interval with synchronous changes across ECG leads.The observed pattern is non-uniform attention focused on clinically relevant cross-lead variations.Source: §G.1 · p. 23, Figure 6 · p. 23

G.2 Quantitative Attention Analysis

The quantitative analysis compares ViRe’s CLIP-derived Vision Query with a same-dimensionality Gaussian Query using attention enrichment on high-curvature and clinically identified QRS regions.

G.2 Quantitative Attention Analysis

  • Values above one indicate denser attention inside a target region than in its temporal complement.The comparison uses the CLIP-derived Vision Query and a non-semantic Gaussian Query of the same dimensionality.Source: §G.2 · pp. 23–24
  • MAR measures enrichment on signal-intrinsic high-curvature timestamps, while CAR measures enrichment on clinically recognized QRS segments.The target regions are compared with their temporal complements using attention-density ratios.Source: §G.2 · pp. 23–24
  • ViRe consistently exceeds the Gaussian Query on both attention ratios across all three ECG datasets.The two ratios assess high-curvature morphology and QRS alignment, respectively.Source: §G.2 · p. 24

H Rendering Sensitivity Analysis

Rendering resolution and line width affect ViRe’s F1-Score, while channel coloring has little influence. The appendix also reports stability and irregular-data evaluations, and identifies clinical-validation and scope boundaries.

H Rendering Sensitivity Analysis

  • 5 DPI causes a pronounced F1-Score drop on APAVA and PTB, while 50–200 DPI is more stable; overly thin or thick lines also degrade performance.Channel coloring has negligible influence, and the recommendation is moderate resolution with moderate line width.Source: H Rendering Sensitivity Analysis · p. 25, Figure 7 · p. 25

I Robustness and Transfer Analysis

  • The robustness and transfer analysis extends evaluation to repeated subject-disjoint splits and irregularly sampled forecasting and classification.These evaluations go beyond the primary benchmark.Source: I Robustness and Transfer Analysis · p. 26

Subject-Partition Stability

Irregular Forecasting

  • ViRe performs best on all six reported irregular-forecasting metrics, reducing MSE versus Hi-Patch by 4.7%, 6.2%, and 10.9% on Human Activity, PhysioNet, and MIMIC-III, respectively.The comparison follows the Hi-Patch protocol.Source: Irregular Forecasting · p. 26, Table 16 · p. 26

Irregular Classification

J Limitations and Societal Considerations

  • The study uses retrospective EEG/ECG benchmarks, not prospective multi-center clinical validation; other physiological signals and multi-label settings remain untested.The visual prior comes from a frozen CLIP encoder trained on general image-text data, which may miss clinically subtle waveform patterns.Source: J Limitations and Societal… · p. 27
  • ViRe is intended for decision support rather than autonomous diagnosis; routine use requires clinician oversight, privacy protection, and target-distribution validation.The paper also notes risks of over-reliance on automated outputs and privacy concerns in downstream use.Source: J Limitations and Societal… · p. 27

NeurIPS Paper Checklist

  • The checklist reports five-seed mean and standard deviations, compute-resource details, and public code and training scripts for reproducibility.Training and test settings are specified across the main text and appendices.Source: NeurIPS Paper Checklist · pp. 28–29
Loading 2609.34652v1…