Learning from Points and Patches: A Pre-trained Hybrid Attention Transformer for Time Series Analysis

1 Introduction

• Heterogeneous, noisy time series with diverse temporal patterns and limited high-quality annotations motivate transferable pre-training for multiple downstream tasks.The stated tasks include forecasting, classification, imputation, and anomaly detection.Source: §1 · p. 1
• Time-series pre-training remains challenging because datasets and domains are heterogeneous and scarce, and existing strategies may miss time series’ distinctive statistical properties.The paper argues that temporal data require tailored learning strategies and architectures.Source: §1 · p. 1
• Patch-level reconstruction can miss within-patch variation, while point-level masking does not explicitly capture segment-level structure.The introduction also identifies a need to integrate context without homogenizing representations or losing early global interaction.Source: §1 · pp. 1–2, Figure 1 · p. 1
• TimeHAT combines point- and patch-level supervision through contextual reconstruction and causal extrapolation, and pairs shallow prefix-constrained bidirectional attention with deeper directional attention.The framework is evaluated across classification, long-term forecasting, imputation, and anomaly detection, including fine-tuning, few-shot, and zero-shot settings.Source: §1 · p. 2

2 Unified Point-Patch Masking Strategy

• UPPMS unifies point- and patch-level masking and supports contextual reconstruction from observed sequences and causal extrapolation toward future segments.The two granularities provide complementary supervision for temporal patterns.Source: §2 · p. 2

2.1 Dual-mode Sampling

• Contextual reconstruction perturbs historical patches and targets the terminal patch, whereas causal extrapolation leaves history unperturbed and targets a contiguous tail.The terminal target also serves as a CLS-like global readout for classification and anomaly detection.Source: §2.1 · pp. 2–3

2.2.1 Predictive Target Masking.

• Each target patch representation is replaced by a shared learnable query embedding that aggregates visible context to reconstruct masked content or predict future segments.TimeHAT’s overall architecture applies masking before patch tokenization and uses an MLP projection head for reconstruction.Source: §2.2.1 · p. 3, Figure 3 · p. 3

2.2.2 Patch-level Masking.

• Perturbed patches receive zero replacement, global visible-value mean replacement, or internal shuffling to encourage recovery of segment structure and local information.Internal shuffling changes the temporal order within the patch.Source: §2.2.2 · p. 3

2.2.3 Point-level Masking.

• Point-level masking applies only in contextual reconstruction, replacing sampled values in otherwise visible patches with zero or a draw from N(0, 1).This supplies fine-grained reconstruction supervision within patches.Source: §2.2.3 · p. 3

2.3 Prefix-constrained Attention.

• The prefix constraint blocks historical queries from target tokens and enforces causal information flow among target tokens, supporting both learning modes without target leakage.The mask is combined with layer-specific attention masks to determine complete information flow.Source: §2.3 · p. 3

3 TimeHAT

• TimeHAT applies temporal encoding independently per variable and uses an encoder-only Transformer with a linear reconstruction projection rather than a separate Transformer decoder.The channel-independent design focuses on within-variable temporal patterns without relying on potentially unstable cross-variable dependencies.Source: §3 · pp. 3–4

3.1 Input Processing and Embedding

• The input sequence is standardized and divided into non-overlapping patches, which are projected into d-dimensional tokens with rotary position embeddings applied to attention queries and keys.Source: §3.1 · p. 4

3.2 Hybrid Attention Encoder

• HAE uses prefix-constrained bidirectional attention in its first M layers, then directional hybrid attention in deeper layers to mitigate rank degeneration while preserving contextual exchange.Prefix constraints prevent historical tokens from accessing predictive targets and preserve causal information flow among target tokens.Source: §3.2 · p. 4

3.2.1 Unified Formulation and Notation.

• In the first M layers, non-target tokens interact bidirectionally, while target tokens access historical context and preceding targets without accessing subsequent targets.Source: §3.2.1 · p. 4
• Deeper layers split attention heads evenly between forward and backward directions, using causal and target-aware anti-causal masks whose triangular structure yields full-rank attention matrices.Backward heads switch to causal attention for predictive targets, preserving historical context while preventing future-target leakage.Source: §3.2.1 · p. 4

3.2.2 Layer Integration.

• Each layer concatenates and projects the attention-head outputs, then applies residual connections and an FFN to produce its output.Source: §3.2.2 · p. 4

4.1 Long-term Forecasting

• TimeHAT achieves the best average long-term forecasting performance on the reported benchmarks, with advantages over optimized forecasting architectures and time-series foundation models.Table 1 averages MSE and MAE over prediction horizons {96, 192, 336, 720}.Source: §4.1 · pp. 4–5, Table 1 · p. 5

4.2 Imputation; 4.3 Anomaly Detection

• TimeHAT achieves the lowest averaged MSE and MAE for imputation across six benchmarks and consistently outperforms specialized and foundation models on anomaly detection.Imputation uses input length 96 and random masking ratios of 12.5%, 25%, 37.5%, and 50%; anomaly detection uses F1-score across five benchmarks.Source: §4.2 · p. 5, Table 2 · p. 5, §4.3 · p. 5

4.4 Classification; 4.5 Zero-shot and Few-shot Forecasting

• TimeHAT has the highest average classification accuracy across 10 datasets and consistently outperforms baselines in zero-shot and few-shot forecasting settings.Few-shot results include improvement over Timer as training data decreases.Source: §4.4 · p. 6, Figure 4 · p. 6, §4.5 · p. 6

4.6.1 Direct versus Recursive Forecasting.

• Recursive forecasting reduces MSE across evaluated schemes on ETTh1 and ETTm2; on ETTh1, MSE falls by 6.57%–7.50% and MAE by 4.39%–4.98%.Recursive forecasting repeatedly predicts 96 steps and feeds predictions back as input; it also reuses the pre-trained reconstruction head.Source: §4.6.1 · p. 6, Figure 6 · p. 6

4.6.2 Sensitivity to UPPMS Design Choices.

• UPPMS ablations show dataset-dependent effects: no simplification consistently improves both metrics across all datasets, and removing last-patch recovery or using patch-only masking increases ETTh2 MSE.Zero-only replacement degrades both metrics on three of four datasets.Source: §4.6.2 · pp. 6–7, Table 5 · p. 6

4.6.3 Effect of Pre-training Objectives; 4.6.4 Effect of Pre-training Mode Balance.

• The full UPPMS has the lowest MSE on all five datasets and lowest MAE on four; intermediate tail-recovery probabilities generally outperform pure tail recovery.ETTm1 shows a slight MAE advantage for Causal Extrapolation in the objective comparison.Source: §4.6.3 · p. 7, Table 6 · p. 7, §4.6.4 · p. 7

4.7.1 Attention Performance Across Forecast Horizons and Depth.

• Hybrid Attention (HA) achieves the lowest MSE across evaluated horizons and is more robust to increasing depth than other attention mechanisms; on ETTm1, it reduces MSE by 26.5%–33.3% relative to BiA.At six layers, HA has the lowest MSE on both ETTh2 and ETTm2.Source: §4.7.1 · p. 7, Figure 7 · p. 7, Figure 8 · p. 7

4.7.2 Effect of Attention Composition; 4.7.3 Sensitivity to Readout Strategy; 4.7.4 Effect of the Causal-Head Ratio in Hybrid Attention.; 4.7.5 Effect of Pre-training Attention Rules.

• Attention composition, readout, head ratio, and pre-training-rule tests favor limited or task-specific designs, with outcomes varying by dataset and setting.All-Mix performs best on the three ETT datasets while mixed configurations are slightly better on Weather; a 0.5 causal-head ratio is best on Exchange and ETTh2. HA is relatively insensitive to readout choice, and default pre-training attention rules achieve the lowest MSE on all four ETT datasets and lowest MAE on three.Source: §4.7.2 · p. 7, Table 8 · p. 8, §4.7.3 · p. 8

• TimeHAT combines UPPMS for contextual reconstruction and causal extrapolation with HAE, which stages attention to mitigate rank degeneration.Source: §5 · p. 8
• The framework demonstrates competitive performance in classification, long-term forecasting, imputation, and anomaly detection across fine-tuning, few-shot, and zero-shot transfer.Source: §5 · p. 8

• PromptCast, GPT4TS, TIME-LLM, and LLM4TS adapt language models for forecasting using numerical-to-text transformations, patching, or embedding techniques.Source: A Related Work · p. 10
• TEST improves time-series tokenization and embedding through contrastive learning, while UniTime uses domain instructions, modality alignment, and masking for cross-domain forecasting transfer.Source: A Related Work · p. 10
• Most reviewed LLM approaches focus on forecasting, leaving applications such as anomaly detection and classification underexplored.Source: A Related Work · p. 10
• Time-series-specific frameworks include large-data foundation models and methods for scale, task unification, forecasting, reconstruction, and spatio-temporal patterns.Examples include MOMENT [13], Timer [31], TIME-MOE, Sundial, Moirai [23], UNITS, and TimeMixer++ [37], MTSMAE, SimMTM, PatchTST, and TSMixer.Source: A Related Work · p. 10

• UTSD combines publicly accessible online data with empirical data from real-world machine operations and covers seven source domains with diverse sampling frequencies.The domains are Energy, Environment, Health, IoT, Nature, Transportation, and Web.Source: §B.1 · pp. 10–11
• The dataset-description tables report train, validation, and test splits for long-term forecasting and other datasets.Source: Table 10 · p. 10, Table 11 · p. 11
• UTSD provides nested UTSD-1G, UTSD-2G, UTSD-4G, and UTSD-12G datasets; the paper uses UTSD-12G for pre-training.Each smaller dataset is a subset of the larger ones, and larger subsets mean greater data difficulty and diversity.Source: §B.1 · p. 11
• Downstream datasets cover forecasting and imputation, UEA classification, and anomaly detection across energy, weather, traffic, health, industrial, and space-related settings.Examples include Electricity, Weather, the ETT family, EthanolConcentration, Heartbeat, JapaneseVowels, PEMS-SF, SMD, MSL, SMAP, SWaT, and PSM.Source: §B.1 · p. 11

• TimeHAT has eight attention layers—three prefix-constrained BiA layers and five decoupled HA layers—with 67M trainable parameters.The model uses hidden dimension 1024, feed-forward dimension 2048, 16 heads, context length 1440, and patch size 96.Source: §B.2 · p. 11
• Pre-training uses UTSD-12G, UPPMS reconstructive sampling with a 40% masking ratio, AdamW, and global batch size 10,240.The learning rate is determined using the square-root scaling rule for adaptive optimizers.Source: §B.2 · p. 11
• The peak learning rate is 8×10^-4 with 4,000 warmup steps, cosine decay to 10% of peak, and a 20-epoch maximum.Validation runs every 5,000 steps, with early stopping after 10 evaluations without improvement; training uses DDP on four NVIDIA RTX 5090 GPUs.Source: §B.2 · p. 11
• Transfer comparisons include embedding-and-encoder transfer and encoder-only transfer with a newly initialized P=16 embedding and prediction head.The table captions distinguish scratch, patch-level masking, and point-level masking variants.Source: Table 13 · p. 12, Table 12 · p. 12

B.3 Downstream Tasks

• The pre-trained backbone and projection head are retained for forecasting, imputation, and anomaly detection, while classification uses an added task-specific head.This preserves the reconstructive representation for sequence-to-sequence transfer.Source: §B.3 · p. 12
• Fine-tuning assigns a conservative learning rate to the pre-trained backbone and higher rates to task-specific heads to mitigate catastrophic forgetting.The backbone rate is 1 × 10−5 to 3 × 10−5; head rates are 5 × 10−5 to 1 × 10−4.Source: §B.3 · p. 12
• Fine-tuning uses a 5% linear warmup and runs for at most 20 epochs, with early stopping after 5 epochs of patience.Source: §B.3 · p. 12

C Effect of Masking Granularity and Embedding Size

• The forecasting comparison uses patch sizes P=1 and P=16 under embedding-plus-encoder and encoder-only transfer, averaging four horizons and three fine-tuning seeds.Source: C Effect of Masking Granularity and… · p. 12
• Patch embeddings (P=16) generally outperform point-wise embeddings (P=1) for forecasting, particularly under encoder-only transfer.Point-level masking with P=16 achieves the best results on several datasets.Source: C Effect of Masking Granularity and… · p. 12
• Point masking generally preserves local reconstruction details better, while block masking often underestimates peaks and troughs and misses fine-grained variation, especially on ETTh1 and ETTm1.Figure 12 shows examples across five datasets, with each row using the same selected sequence and vertical scale.Source: C Effect of Masking Granularity and… · p. 12, Figure 12 · p. 12