Source-linked AI summary
On the role of the tokenizer in ECG transformer models
Jiawei Li, Fabio Bonassi, Johan Sundström, Thomas B. Schön, Antônio H. Ribeiro
TL;DR
ECG Transformer performance depends in part on tokenization, but the effects of token construction have received less attention than attention mechanisms and model scale. The paper compares eight tokenization strategies across four backbones under controlled conditions. Physiology-aware tokenizers achieve higher pooled macro-AUC and lower sequence length and memory use than conventional point-wise and patch-wise tokenizers.
Problem
The paper asks whether tokenization, alongside attention mechanisms and model size, is an important design choice for ECG Transformers, given that ECG diagnostic information is encoded in morphology, rhythm, and lead relationships.
Source: §1 · p. 1Method
The study compares eight ECG tokenization strategies across four Transformer backbones on CPSC2018 while holding input projection and principal backbone hyperparameters constant.
Source: §1 · p. 1Results
0.891 versus 0.823 pooled mean macro-AUC: median-beat and HeartLang tokenization outperformed point-wise and patch-wise tokenization, while reducing mean peak training VRAM from 5.21 to 0.27 GB.
Source: §4 · p. 4Takeaways & Limitations
The results support preserving physiologically coherent heartbeat morphology in ECG tokenizers; median-beat offers the best overall performance–memory trade-off.
Source: §4 · p. 4The evaluation is limited to a single dataset.
Source: §1 · p. 1
Abstract
from arXiv · showhide
Tokenization determines both the physiological content presented to an ECG Transformer and the sequence over which attention operates. We compare eight tokenization strategies across Transformer, Informer, Reformer, and FEDformer on the nine-label CPSC2018 classification task. The input projection and principal backbone capacity are controlled to isolate the effect of token construction. Median-beat and HeartLang tokenization achieve mean macro-AUCs of 0.893 and 0.889 across the four backbones, compared with 0.822 and 0.824 for point-wise and patch-wise tokenization. Pooling the two physiology-aware representations yields an 8.2% relative improvement in macro-AUC. They also reduce mean sequence length from 1,250 to 158 tokens and mean peak training memory from 5.21 to 0.27 GB. The results show that aligning tokens with ECG morphology can improve both predictive performance and memory efficiency without increasing backbone capacity. The source code is available on https://github.com/LeeJarvis996/ecg_tokenizer.
1. Introduction
The paper argues that ECG tokenization deserves attention because token construction shapes physiological content and sequence length, which affects attention cost. It compares eight tokenizers under controlled conditions and finds physiology-aware representations outperform conventional point-wise and patch-wise inputs while using less memory.
1. Introduction
- ECG diagnosis depends on P–QRS–T morphology, beat-to-beat rhythm, and spatial relationships among leads, while point-wise tokens create long sequences and patches may split coherent waveforms.Full self-attention computational cost scales quadratically with sequence length.Source: §1 · p. 1
- The study evaluates eight tokenization strategies across Transformer, Informer, Reformer, and FEDformer on the nine-label CPSC2018 task, controlling input projection and principal backbone hyperparameters.The controlled design examines how token definition and sequence length relate to predictive performance and computational efficiency.Source: §1 · p. 1
- 0.891 versus 0.823 mean macro-AUC: pooled median-beat and HeartLang tokenization outperformed pooled point-wise and patch-wise tokenization, with 8.2% relative improvement.The physiology-aware group also reduced mean sequence length from 1,250 to 158 tokens and mean peak training VRAM from 5.21 to 0.27 GB.Source: §1 · p. 1
2. Methods
The methods define tokenization as constructing an ordered sequence of units whose content determines physiological meaning and whose length defines the attention axis. Eight tokenizer baselines represent ECGs at different physiological and temporal scales, with dimensions specified for a 10-second, 100 Hz, 12-lead input.
2.1. Tokenization Definition
- A tokenizer maps an ECG record X with T samples and C leads into ordered token units; each unit’s content determines its physiological meaning, and sequence length L defines the token-relationship axis.Each unit is projected to model dimension D, optionally receives lead and temporal embeddings, and forms an embedding sequence processed by a Transformer.Source: §2.1 · pp. 1–2
2.2. Tokenizer Architectures
- Point-wise tokens are simultaneous multi-lead samples attending across time, whereas lead-wise tokens are complete leads attending across leads.For point-wise tokenization, L = T and P = C; for lead-wise tokenization, L = C and P = T.Source: §2.2 · p. 2
- Patching creates overlapping per-lead windows, stochastic segmentation samples and resamples intervals per lead, and BIOT-inspired segmentation uses fixed overlapping windows.The BIOT-inspired setup yields 19 windows per lead and 228 tokens for the stated 12-lead input; patch parameters are p = 16 and s = 8, while stochastic segments use at most 16 intervals per lead.Source: §2.2 · p. 2
- ECG-Byte quantizes and compresses a normalized central two-second symbol stream, while HeartLang uses lead–beat units based on shared lead-II QRS boundaries.ECG-Byte caps sequences at 1,020 tokens; HeartLang caps and pads variable-length sequences at 256.Source: §2.2 · p. 2
- Median-beat tokenization detects lead-II R peaks, takes the element-wise median of R-centred multi-lead beats, and uses each of 60 relative time points as a token.Attention models morphology within a representative heartbeat rather than relationships among individual beats.Source: §2.2 · p. 2
3. Results
Across four Transformer backbones, tokenizer choice strongly affects CPSC2018 results, with HeartLang and median-beat tokenization leading among the tested Transformer inputs. These physiology-aware methods also offer compact, memory-efficient representations, although heartbeat extraction contributes substantial latency.
3.1. Experiment Setup
- The controlled comparison evaluates four Transformer backbones on nine-label CPSC2018 data alongside tokenizer-independent convolutional and recurrent baselines.The Transformer backbones use embedding dimension 128, three encoder layers, and eight attention heads; data splits are patient-stratified 8:1:1.Source: §3.1 · pp. 2–3
3.2. Main Results
- CNN and recurrent baselines reach 0.958–0.961 macro-AUC, while Reformer–HeartLang leads the Transformer combinations at 0.932 macro-AUC and 0.607 macro-F1.HeartLang is best for Transformer, Reformer, and Informer; median-beat is strongest for FEDformer, raising macro-AUC from 0.745 with point-wise tokens to 0.882 with 60 tokens.Source: Figure 1 · p. 3, §3.2 · p. 3
3.3. Analysing efficiency
- Median-beat requires 0.08–0.16 GB and 3.87–4.64 ms/ECG; HeartLang requires 0.22–0.66 GB and 4.59–5.22 ms/ECG.Heartbeat extraction dominates their wall-clock latency; tokenizer-only costs are 3.81 ms/ECG for median-beat and 4.37 ms/ECG for HeartLang.Source: §3.3 · p. 4
4. Conclusion
Across four backbones, median-beat and HeartLang tokenization outperform point-wise and patch-wise inputs on pooled macro-AUC while reducing mean peak training VRAM. Median-beat offers the best overall performance–memory trade-off, while Reformer with HeartLang gives the strongest individual Transformer result.
4. Conclusion
- The pooled mean macro-AUC is 0.891 for median-beat and HeartLang versus 0.823 for point-wise and patch-wise inputs, with mean peak training VRAM reduced from 5.21 to 0.27 GB.The pooled macro-AUC improvement is 8.2% relative.Source: §4 · p. 4
- Median-beat provides the best overall performance–memory trade-off across the evaluated backbones.Source: §4 · p. 4
- Reformer with HeartLang gives the best individual Transformer result among the tested combinations.Source: §4 · p. 4