ViSTA: A Simple Bridge Extends Visual Alignment to Clinical Time-Series Understanding in Multimodal LLMs

• Clinical time series are irregular, high-dimensional, and structurally different from narratives, while recent work finds LLMs behind conventional models on structured-record prediction (Zhu et al., 2026a).Source: §1 · p. 1
• Text serialization exposes measurements but spreads temporal relationships across long sequences, learned encoders need alignment, and charts may lose precision during rendering.These trade-offs motivate combining direct numerical access with the visual-language alignment already learned during pretraining.Source: §1 · p. 2
• ViSTA encodes values, times, observation gaps, and variable identity, then uses queries at visual positions to correct chart tokens through a residual bridge.Separate adapters are trained for prediction and temporal QA, with QA also evaluated after outcome-only training.Source: §1 · p. 2
• At 4B, ViSTA reaches AKI AP 0.4650 and mortality AP 0.2651, with relative gains of 3.15% and 7.59% over the strongest competing adaptation, respectively.Source: §1 · p. 2

• Charts use pretrained visual features, but measurement recovery depends on rendering (Zhao et al., 2025); MLLM4TS aligns plot features with temporal patches but uses task-specific output heads (Liu et al., 2025).Source: §2 · pp. 2–3
• Time-LLM reprograms numerical patches with text prototypes, while ChatTS and Time-MQA learn continuous or contextualized numerical representations (Jin et al., 2023; Xie et al., 2024; Kong et al., 2025).Source: §2 · p. 3
• ITFormer and OpenTSLM connect temporal encoders to frozen language models through instruction-aware interaction, soft prompts, or gated cross-attention (Wang et al., 2025a; Langer et al., 2025).Source: §2 · p. 3
• Parameter reduction alone does not determine access to numerical evidence; ViSTA studies a small numerical module on a frozen visual interface and evaluates prediction and QA, including QA after outcome-only training.Source: §2 · p. 3

3.1 PROBLEM FORMULATION AND THE PRETRAINED VISUAL PATHWAY

• ViSTA starts from chart representations and adds numerical evidence at the same visual-token positions, preserving the decoder’s expected input structure.The patient history is represented as irregular observations, and the original language output head represents the model’s response distribution.Source: §3 · p. 3, §3.1 · p. 3

3.2 ENCODING IRREGULAR NUMERICAL OBSERVATIONS

• The event encoder pools irregular observations into fixed-size summaries, with joint pooling combining variables and separate pooling summarizing each variable independently.Both pooling designs share attention and feedforward weights and expose all summaries to the bridge.Source: §3.2 · p. 4, Figure 2 · p. 4

3.3 CORRECTING ORIGINAL VISUAL TOKENS

• ViSTA retrieves numerical summaries with position- and chart-conditioned queries, then adds the resulting corrections to the original visual features.The correction is scaled relative to each frozen visual token’s magnitude; zero-initialized output projection starts the model with its native chart representation.Source: §3.3 · p. 4

3.4 LEARNING THROUGH THE ORIGINAL LANGUAGE INTERFACE

• The pretrained vision encoder, projector, decoder, embeddings, and language head stay frozen while answer-token supervision trains the event encoder and bridge.Prediction and QA use language answers through the same output interface; questions guide use of the corrected record.Source: §3.4 · p. 5

Tasks and data

• MIMIC-IV experiments use eleven irregularly observed clinical variables to predict AKI and in-hospital mortality from the first 24 ICU hours, with temporal QA spanning eleven categories.Prediction has 5,000 training, 500 validation, and 1,000 test examples per outcome; QA has 22,000 training, 1,100 validation, and 2,200 test questions.Source: §4 · p. 5

Prediction and question-answering training

• The study evaluates Qwen3.5 backbones at 2B, 4B, and 9B, training adapters separately for prediction and supervised QA while also testing the prediction-trained adapter on QA.Joint and Separate pooling designs are both evaluated.Source: §4 · p. 5

Comparisons

• Comparisons include unadapted text and chart inputs, Chart and Text LoRA, four time-series or clinical methods, and zero-shot GPT-5.6 Sol-high and Luna-high references.Specialized RNN- and Transformer-based clinical predictors are included in Appendix I.Source: §4 · p. 5
• 90.61% fewer trainable parameters than Chart LoRA and 91.73% fewer than Text LoRA at 4B: ViSTA uses the fewest parameters among trained adaptations.Table 1 reports trainable counts at 4B; ViSTA uses 0.516M, 0.582M, and 0.780M parameters at 2B, 4B, and 9B.Source: §4 · p. 5, Table 1 · p. 6

Metrics

• Prediction is assessed with AUROC, AP, Brier score, and ECE, while temporal QA uses answer accuracy; methods share backbones, patient splits, training data, and validation-based selection.Hyperparameters are selected by grid search using validation data; reported standard deviations use patient-level bootstrap resampling.Source: §4 · pp. 5–6

5.1 COMPACT ADAPTATION IMPROVES OUTCOME PREDICTION

• At every Qwen scale, ViSTA-Joint has the highest mean scores on all four prediction metrics (Table 2).At 4B, AKI AP is 0.4650 versus Chart LoRA’s 0.4508, and mortality AP is 0.2651 versus Text LoRA’s 0.2464.Source: §5.1 · p. 6, Table 2 · p. 7
• In 12/12 combinations, Brier score improves over native charts, and ECE in 10/12; at 9B, mortality ECE falls from 0.7261 to 0.0666 with ViSTA-Joint and 0.0284 with ViSTA-Separate.These calibration gains require no post-hoc calibration.Source: §5.1 · p. 6
• Outcome-trained models did not consistently improve QA: at 9B, all six alternatives underperformed native charts, while ViSTA-Joint exceeded native text by 0.42, 2.33, and 4.00 points across scales.At 4B, Text LoRA scored 42.05% versus ViSTA-Joint’s 41.73%.Source: §5.1 · pp. 6–7, Table 2 · p. 7

5.2 STRONG SUPERVISED QA WITH A COMPACT NUMERICAL INTERFACE

• At 4B, ViSTA-Separate gains 29.87 points over native text and 33.77 over native charts, within 2.82–4.88 points of LoRA with over 90% fewer trainable parameters.Chart LoRA and Text LoRA have the highest mean supervised QA accuracy among Qwen adaptations at every scale.Source: §5.2 · p. 8, Table 3 · p. 8

5.3 LARGER BACKBONES FURTHER IMPROVE NUMERICAL ADAPTATION

• Across 2B–9B, ViSTA-Joint AKI AP rises from 0.4360 to 0.4840 and mortality AP from 0.2391 to 0.3021; native prediction lacks this consistent trend (Figure 3).Adapted 2B exceeds native 9B charts on both AP measures.Source: §5.3 · p. 8, Figure 3 · p. 9
• ViSTA-Joint scores higher than ViSTA-Separate on every prediction metric, but pooling preferences vary for QA by scale; Separate’s 4B missingness-counting score is 63.0% versus 54.0%.Outcome-trained QA differs by at most 0.73 percentage points between pooling designs.Source: §5.3 · p. 8
• Removing zero initialization or RMS scaling, or retaining only visual tokens or numerical residuals at inference, reduces prediction metrics and QA performance.The reported design ablations lower both task types’ scores.Source: §5.3 · p. 9

5.4 SUPERVISED QA GAINS SPAN NUMERICAL AND TEMPORAL OPERATIONS

• At 4B, ViSTA-Separate beats category-wise best baselines in eight categories, ties one, and trails two; without numerical summaries, it leads by 7.20 points (66.45% vs 59.25%).The category gains include numerical summaries, threshold forecasting, segmented trends, and cross-variable retrieval.Source: §5.4 · p. 9, Figure 4 · p. 9

6 DISCUSSION AND CONCLUSION

• ViSTA improves clinical prediction and supervised temporal QA with fewer than one million trainable parameters while keeping the pretrained model fixed.Prediction improves through 9B, and QA gains extend beyond numerical summaries.Source: §6 · p. 10
• The evaluation uses one hospital system, one training seed, and automatically generated multiple-choice questions; matched controls and external cohorts remain needed.Specialized clinical predictors and QA-trained LoRA remain stronger in their respective tasks.Source: §6 · p. 10

A COHORT AND OUTCOME DEFINITIONS

• The cohort includes adults with an ICU stay of at least 24 hours, using each patient's first eligible stay and patient-disjoint splits.Prediction uses eleven numerical variables; endpoints are recorded creatinine-defined AKI and in-hospital mortality.Source: A COHORT AND OUTCOME DEFINITIONS · p. 14

B TEMPORAL QUESTION CONSTRUCTION

• Questions follow CLIR-Bench task definitions but were independently rebuilt on the MIMIC-IV cohort because the original benchmark inputs could not be reproduced.New questions, labels, and distractors use the study's patient splits and four randomized answer options.Source: B TEMPORAL QUESTION CONSTRUCTION · p. 14
• Retrospective questions concern the supplied window, while forecasting and next-intervention questions restrict inputs to information available at their cutoff.Intervention prediction targets recorded care; intervention-response questions describe observed changes around treatment starts.Source: B TEMPORAL QUESTION CONSTRUCTION · p. 14
• Question-specific thresholds and stability tolerances define answers, with missingness counted on the union of observation timestamps and numerical distractors perturbing correct summaries.The answer-construction rules are intended to produce reproducible labels for temporal and numerical understanding.Source: B TEMPORAL QUESTION CONSTRUCTION · p. 14

C PROMPT TEMPLATES

• Chart and numerical-text inputs use the same task wording and answer labels, and both expose only information within the question's visible interval.Templates fill fields from the visible record or question parameters and include intervention context.Source: C PROMPT TEMPLATES · p. 16, §C.1 · p. 16

C.2 CLINICAL PREDICTION; C.3 TEMPORAL QUESTION ANSWERING

• Clinical prediction prompts ask for a binary outcome using information available by ICU hour 24, while temporal QA includes forecasting, observed-pair timing, cross-variable lookup, and handoff summaries.Prospective QA reveals only observations through the cutoff; subsequent observations determine the reference answer.Source: §C.2 · p. 16, §C.3 · p. 17

D IMPLEMENTATION AND EVALUATION DETAILS

• Each event encodes normalized value, relative time, within-variable observation gap, and a first-observation flag; missing-variable tokens are appended to event memory.Values are centered by the training median and scaled by the training interquartile range, floored at 10^-3.Source: D IMPLEMENTATION AND EVALUATION DETAILS · p. 18
• Position coordinates and the chart's variable embedding produce residuals at visual positions, with the final linear layer initialized to zero and pretrained parameters frozen.Gradients propagate through the language decoder to the interface.Source: D IMPLEMENTATION AND EVALUATION DETAILS · p. 18
• Hyperparameters are selected by validation-set grid search, using average precision for prediction and accuracy for QA; test results do not guide configuration.Prediction-selected models are also evaluated on QA without further training or QA-based selection.Source: D IMPLEMENTATION AND EVALUATION DETAILS · p. 18
• QA selects the most probable answer label, while prediction scores normalize the original language-head logits over the two binary labels.Bootstrap standard deviations resample patients with all their questions kept together, and QA accuracy is averaged across categories.Source: D IMPLEMENTATION AND EVALUATION DETAILS · p. 18

E SCALING AND VARIABLE ENCODING

• QA accuracy gains over unadapted text are 0.42, 2.33, and 4.00 percentage points at 2B, 4B, and 9B; gains over unadapted charts are 3.00, 6.23, and 5.91 points.Table 6 also reports supervised QA gains over the unadapted chart baseline; interface sizes are 0.516M, 0.582M, and 0.780M at 2B, 4B, and 9B.Source: E SCALING AND VARIABLE ENCODING · pp. 18–19, Table 6 · p. 19

E.1 ENCODING COMPARISON

• ViSTA-Joint has higher mean prediction AP than ViSTA-Separate at every model size, while prediction-trained QA differs by 0.09 points at 2B and 4B and favors Separate by 0.73 points at 9B.The encodings differ in the attention mask during the first numerical pooling stage; subsequent visual queries can read all pooled variables.Source: Table 7 · p. 19, §E.1 · p. 19

F INPUT INTERVENTIONS AND DESIGN ABLATIONS

• Removing zero initialization lowers AKI and mortality AP by 0.0198 and 0.0192, while removing RMS scaling lowers them by 0.0148 each.Both changes also reduce AUROC and transfer QA accuracy; these are absolute AP differences.Source: F INPUT INTERVENTIONS AND DESIGN… · p. 19, Table 8 · p. 20
• Visual-only and residual-only inference inputs both reduce prediction and QA scores relative to the full input, but altered inputs differ from training inputs and do not isolate semantic chart use.The residual-only input retains visual-derived scaling and original positional structure.Source: F INPUT INTERVENTIONS AND DESIGN… · p. 19, Table 8 · p. 20

G CATEGORY-LEVEL SUPERVISED QA

• ViSTA-Separate has higher mean accuracy than the strongest selected baseline in eight QA categories, ties in monitoring rules, and scores lower in two categories.The comparison selects the best of ChatTS, MLLM4TS, ITFormer, and OpenTSLM within each category.Source: Table 9 · p. 21, G CATEGORY-LEVEL SUPERVISED QA · p. 21
• Excluding numerical summaries, ViSTA-Separate scores 66.45%, versus 59.25% for the best baseline in each of the other ten categories.The 7.20 percentage-point difference also holds against every individual baseline in the comparison.Source: G CATEGORY-LEVEL SUPERVISED QA · p. 21

H MAJORITY-ANSWER AND RANDOM QA BASELINES

• The majority-answer baseline has 25.55% macro-average accuracy, compared with 25.00% expected accuracy for uniform random guessing.Majority answers are selected using training answers only.Source: H MAJORITY-ANSWER AND RANDOM QA… · p. 22, Table 10 · p. 22

I SPECIALIZED CLINICAL PREDICTION REFERENCES

• LSTM, GRU, ConCare, VITA, and ColaCare exceed ViSTA-Joint on both endpoints, while 9B ViSTA-Joint has higher mean scores than the specialized Transformer reference on all four prediction metrics.Specialized systems use task-specific output heads and lack QA evaluation in this study; ViSTA retains the pretrained language head for QA evaluation.Source: I SPECIALIZED CLINICAL PREDICTION… · p. 23, Table 11 · p. 23

J PREDICTION PROBABILITY CALIBRATION

• Brier score measures overall probability error, while ECE compares mean predicted probability with event frequency across ten equal-width bins; lower is better for both.Empty ECE bins contribute zero; Brier score reflects both calibration and discrimination.Source: J PREDICTION PROBABILITY CALIBRATION · p. 23
• Native Qwen label probabilities are not fitted to clinical outcomes or event prevalence, so answer preferences can produce extreme probabilities and large calibration errors distinct from weak patient ranking by AUROC.The large 9B native-chart mortality errors characterize this zero-shot use of label probabilities.Source: J PREDICTION PROBABILITY CALIBRATION · p. 23
• Both ViSTA encodings lower Brier score versus unadapted charts at every model size and for both outcomes; ECE decreases except for 4B AKI.The unadapted chart model already has low ECE for 4B AKI.Source: J PREDICTION PROBABILITY CALIBRATION · p. 23, Table 12 · p. 24