PaperlayerView exact revision on arXiv

Source-linked AI summary

TCCT: Tightly-Coupled Convolutional Transformer on Time Series Forecasting

Li Shen, Yangzhu Wang

arXiv:2108.12784v2cs.LGcs.AI

TL;DR

Existing CNN–Transformer combinations are described as loosely coupled, leaving open whether tighter integration can improve learning, locality, and efficiency. The paper proposes TCCT, applying three transformed CNN architectures within Transformer models. Experiments report improved forecasting performance across real-world datasets, with reduced computation and memory costs.

  • Problem

    Existing CNN–Transformer combinations are mostly loosely coupled, and the paper asks whether transformed CNN architectures inside Transformers can improve learning capability, locality, and efficiency.

    Source: §1 · p. 2

  • Method

    TCCT integrates CSPAttention, dilated causal convolution, and passthrough mechanisms into Transformer architectures for time-series forecasting.

    Source: §1 · p. 2

  • Results

    Across real-world datasets, all three TCCT architectures enhanced forecasting performance; CSPAttention reduced self-attention memory occupation by nearly 30% and time complexity by 50%.

    Source: §1 · p. 2, §6 · p. 12

  • Takeaways & Limitations

    The three architectures address forecasting performance in different ways and are described as general enough for other Transformer-like time-series models.

    Source: §1 · p. 2, §6 · p. 12

Abstract

from arXiv · show

Time series forecasting is essential for a wide range of real-world applications. Recent studies have shown the superiority of Transformer in dealing with such problems, especially long sequence time series input(LSTI) and long sequence time series forecasting(LSTF) problems. To improve the efficiency and enhance the locality of Transformer, these studies combine Transformer with CNN in varying degrees. However, their combinations are loosely-coupled and do not make full use of CNN. To address this issue, we propose the concept of tightly-coupled convolutional Transformer(TCCT) and three TCCT architectures which apply transformed CNN architectures into Transformer: (1) CSPAttention: through fusing CSPNet with self-attention mechanism, the computation cost of self-attention mechanism is reduced by 30% and the memory usage is reduced by 50% while achieving equivalent or beyond prediction accuracy. (2) Dilated causal convolution: this method is to modify the distilling operation proposed by Informer through replacing canonical convolutional layers with dilated causal convolutional layers to gain exponentially receptive field growth. (3) Passthrough mechanism: the application of passthrough mechanism to stack of self-attention blocks helps Transformer-like models get more fine-grained information with negligible extra computation costs. Our experiments on real-world datasets show that our TCCT architectures could greatly improve the performance of existing state-of-art Transformer models on time series forecasting with much lower computation and memory costs, including canonical Transformer, LogTrans and Informer.

1. Introduction

The paper argues that existing CNN–Transformer combinations are loosely coupled and asks whether transformed CNN architectures inside Transformers can improve learning, locality, and efficiency. It proposes TCCT and three architectures to pursue those aims.

1. Introduction

  • The paper identifies loosely coupled CNN–Transformer hybrids as the gap: they use common CNN layers or separate CNN and Transformer blocks rather than transformed CNN architectures inside Transformers.The motivating question is whether tighter integration can improve learning capability and locality while also improving efficiency.Source: §1 · pp. 1–2
  • TCCT applies three transformed CNN architectures inside Transformer models to strengthen learning capability, enhance locality, and reduce computation and memory costs.The architectures are described as general enough for other Transformer-like time-series forecasting models.Source: §1 · p. 2
  • Nearly 30% lower memory occupation and 50% lower time complexity are reported for CSPAttention while achieving equivalent or superior forecasting accuracy.Source: §1 · p. 2
  • Dilated causal convolution replaces canonical convolution between self-attention blocks, giving exponentially growing receptive fields with negligible extra computation cost.Source: §1 · p. 2
  • Passthrough concatenates feature maps from different self-attention-block scales to provide finer-grained information and improve forecasting performance.Source: §1 · p. 2

2. Related works

The related work moves from the limits of conventional forecasting approaches to Transformer models for long sequences, then reviews CNN designs whose ideas inform TCCT.

2.1. Time series forecasting

  • Traditional statistical models require manual selection for varying factors, while prevalent RNN-based methods remain limited on long-sequence input and forecasting problems.Transformers are presented as a choice for modeling long-range dependencies through self-attention.Source: §2.1 · p. 2

2.2. Transformer models on time series forecasting

  • LogTrans replaces self-attention projections with convolutional layers, while Informer connects self-attention blocks using convolution and max-pooling; both are described as loosely coupled.The paper uses these state-of-the-art models, especially Informer, as experimental baselines for its tightly coupled designs.Source: §2.2 · p. 2

2.3. Related CNN

  • TCCT borrows CSPNet’s feature-map splitting, YOLO9000’s passthrough mechanism, and TCN’s dilated causal convolution to modify Transformer components.The stated aims include reducing computation bottlenecks, obtaining finer-grained features, and growing receptive fields exponentially.Source: §2.3 · p. 2

3. Preliminary

This section defines the forecasting setup and summarizes Informer as the baseline architecture. It describes Informer’s attention, distilling, encoder, and decoder choices.

3.1. Problem definition

  • The forecasting setup uses fixed input and target windows, and rolling forecasting predicts a longer target sequence in successive output windows.Here, N is the number of related univariate series, t0 the input-window size, and T the prediction-window size.Source: §3.1 · p. 3

3.2. Informer architecture

  • Informer uses ProbSparse self-attention, which selects dominant queries by query-key distribution divergence and computes O(lnLQ) dot-products instead of O(LQ).Source: §3.2 · p. 3
  • Informer’s distilling operation connects self-attention blocks with convolution and max-pooling, halving the output length after each block.Source: §3.2 · p. 3
  • Informer can use full distilling with extra encoders of progressively decreasing input lengths and block counts, alongside a generative-style decoder that predicts in one forward procedure.The decoder uses a start token and target sequence, with masked ProbSparse self-attention.Source: §3.2 · p. 3
  • Figure 1 depicts Informer with full distilling and a three-layer main encoder.Source: §3.2 · p. 3

4. Methodology

The methodology introduces three tightly coupled convolutional Transformer architectures and describes how they can be combined with Transformer-like time-series forecasting models.

4.1. CSPAttention

  • CSPAttention splits token dimensions: one part passes through a 1 × 1 convolution and the other through self-attention, then their outputs are concatenated.When input and output dimensions differ, an additional 1 × 1 convolution projects the first part to the proper dimension.Source: §4.1 · pp. 3–4
  • With input dimension split in half, CSPAttention cuts time complexity by at least 50 % versus a canonical self-attention block; another stated quantity is 31.25 % of that block.Source: §4.1 · p. 4

4.2. Dilated causal convolution

  • Replacing canonical convolution with dilated causal convolution yields exponentially growing receptive fields, prevents future leakage, and adds only negligible padding-related computation and memory costs.Canonical convolution’s receptive field grows linearly with network depth and does not account for temporal direction.Source: §4.2 · p. 5

4.3. Passthrough mechanism

  • Passthrough merges feature maps from different scales in one encoder, avoiding full distilling’s multiple encoders and its disproportionate attention to later sequence data.Feature maps are split along their length before concatenation, followed by a transition layer to restore the proper output dimension.Source: §4.3 · pp. 5–6

4.4. Transformer with TCCT architectures

  • The TCCT architectures can be combined with Transformer-like forecasting models, including Informer and LogTrans, by replacing their attention blocks while retaining the other architecture.For LogTrans, masked ProbSparse blocks are replaced by masked LogSparse blocks.Source: §4.4 · pp. 6–7

5. Experiment

The experiments use ETT time-series data, compare Informer variants with TCCT-enhanced models and additional Transformer baselines, and average repeated runs under a shared training setup.

5.1. Datasets

  • The evaluation mainly uses ETTh1 and ETTm1, with hourly and 15-minute observations respectively, and a consecutive 12/4/4-month train/validation/test split.Each data point contains oil temperature and six power-load features; inputs are Z-score normalized.Source: §5.1 · p. 7

5.2. Methods

  • Five Informer-based methods span the baseline, full distilling, and three TCCT combinations; canonical Transformer and LogTrans are additional baselines.TCCT_Ⅰ adds CSPAttention, TCCT_II adds CSPAttention and dilated causal convolution, and TCCT_III adds all three TCCT architectures.Source: §5.2 · p. 7

5.3. Experiment details

  • Experiments use MSE loss and Adam, with learning rate starting at 1e-4; they run for six epochs with early stopping, batch size 32, and ten repetitions averaged on one Nvidia GTX 1080Ti 12 GB GPU.The learning rate is halved each epoch.Source: §5.3 · p. 7

5.4. Result and analysis

In the LSTF ablation, Informer with all three TCCT architectures generally outperforms Informer+ and the other tested variants, with differences across univariate and multivariate settings. The experiments examine CSPAttention, dilated causal convolution, and passthrough alongside Informer and full distilling.

5.4.1. Ablation study on LSTF problem

  • Experiments evaluate forecasting on ETTh1 and ETTm1 with encoder input length fixed at 384; results are averaged over ten runs, with MSE-result distribution and consistency examined using standard deviation.Source: §5.4.1 · p. 8
  • TCCT_III outperforms Informer+ in most univariate cases and all multivariate cases, and beats the other four methods overall.Informer+ outperforms TCCT_I and TCCT_II in some univariate cases, but loses to TCCT_III and increasingly to TCCT_II as multivariate prediction length grows.Source: §5.4.1 · p. 9
  • CSPAttention retains or improves Informer’s forecasting performance in most cases, while adding dilated causal convolution further improves nearly all cases.The encoder combines CSPAttention blocks with dilated causal convolution and max-pooling between blocks; passthrough fuses the three block outputs.Source: Figure 9 · p. 8, Figure 8 · p. 8, §5.4.1 · p. 9

5.5. Forecasting capability in LSTI problem

TCCT_III generally improves forecasting across longer inputs, computation-efficiency tests, other Transformer models, and additional datasets. The reported comparisons also indicate lower training-time and memory use than Informer and especially Informer+ in the efficiency tests.

5.5. Forecasting capability in LSTI problem

  • TCCT_III outperforms Informer and Informer+ in nearly all tested LSTI cases; as input length grows, their maximum MSE drops exceed 10%, while TCCT_III’s is under 5%.The experiments use multivariate ETTm1 data with prediction windows of 48 and 96 and progressively longer encoder inputs.Source: §5.5 · p. 9, Table 3 · p. 10

5.5.1. Ablation study on computation efficiency

  • Informer models with TCCT architectures use nearly the same training time and graphic memory across tested cases, and less than Informer and especially Informer+.Training-time and memory comparisons use multivariate ETTm1 with prediction length 48 and increasing encoder input lengths.Source: §5.5.1 · p. 9, Figure 10 · p. 11

5.5.2. Applicability analysis on other models

  • Canonical Transformer and LogTrans both perform better when combined with TCCT architectures, including with more complicated inputs and robustness of forecasting accuracy.The models are tested on ETTh1 and ETTm1 under multivariate settings.Source: §5.5.2 · p. 9, Table 4 · p. 11

5.5.3. Applicability analysis on other datasets

  • On additional datasets, TCCT_III still outperforms Informer and Informer+, with relatively better performance on more complicated forecasting settings and robust accuracy.The authors also report that Informer+ performs worse than initial Informer on some occasions.Source: §5.5.3 · p. 10, Table 5 · p. 11

6. Conclusion

The paper introduces tightly-coupled convolutional Transformer (TCCT) and three architectures for time series forecasting. Experiments on real-world datasets show that the architectures enhance Transformer-model performance in different ways.

6. Conclusion

  • CSPAttention reduces self-attention computation cost and memory usage without losing forecasting accuracy.Dilated causal convolution gives Transformer models exponentially growing receptive fields.Source: §6 · p. 11
  • TCCT applies transformed CNN architectures within Transformer models to improve prediction capability.The paper presents the approach as a tightly-coupled convolutional Transformer concept.Source: §6 · p. 11
  • Passthrough helps Transformer models obtain finer-grained information.The authors report that all three TCCT architectures enhance Transformer-model performance on time series forecasting across their real-world dataset experiments.Source: §6 · p. 12

CRediT authorship contribution statement

Li Shen and Yangzhu Wang contributed in distinct research and supervisory roles.

CRediT authorship contribution statement

Appendix A. Proof of Theorem 1

The proof compares the computation of canonical self-attention with CSPAttention, whose split input produces a lower total time complexity. It concludes that CSPAttention reduces time complexity by at least 50% relative to canonical self-attention.

Appendix A. Proof of Theorem 1

  • Canonical self-attention’s complexity accounts for the heads’ attention computations and the final projection after head concatenation.The proof defines an input of L tokens, dimension d, and H heads, then sums head and projection costs.Source: Appendix A · p. 12
  • CSPAttention splits the input: its first part uses one linear projection, and the total complexity combines that cost with the second part.The proof gives a total complexity expression for CSPAttention after deriving the two parts’ costs.Source: Appendix A · p. 12
  • CSPAttention cuts time complexity by at least 50% compared with canonical self-attention.The proof attributes the reduction to the attention term’s coefficient being 50% of canonical self-attention’s and the linear term’s coefficient being 31.25%.Source: Appendix A · p. 12

Appendix B. Additional tables of MSE standard deviation

The appendix reports consistency measures for univariate and multivariate experiments. It finds greater instability in univariate settings and reports that Informer+ is less consistent than the other methods, particularly under univariate settings.

Appendix B. Additional tables of MSE standard deviation

  • MSD means MSE Standard Deviation, and CV means coefficient of variation.The appendix uses these abbreviations for the consistency measures reported in its tables.Source: Appendix B · p. 12
  • Informer+ reaches an MSE CV of 48.42% in the univariate setting, and is the only method with MSE CV above 15% in the multivariate setting.The authors conclude that Informer+ is less stable than the other methods and that full distilling is less consistent than passthrough.Source: Appendix B · p. 13, Table B1 · p. 13, Table B2 · p. 13
  • Results are relatively unstable under univariate settings, especially for short prediction lengths; multivariate settings show better consistency.The authors attribute the univariate pattern partly to small average MSE losses, which make turbulence more decisive, and say relative time series empirically help multivariate forecasting stability.Source: Appendix B · pp. 12–13, Table B1 · p. 13, Table B2 · p. 13

Appendix C. Test of dilated causal convolution and passthrough mechanism

The appendix compares Informer variants using dilated causal convolution and passthrough. It reports that the combined variants achieve better forecasting results, while passthrough is the strongest standalone TCCT architecture in most cases.

Appendix C. Test of dilated causal convolution and passthrough mechanism

  • TCCT V, Informer with passthrough, outperforms Informer and usually outperforms TCCT I and TCCT VI.The authors identify passthrough as the strongest of the three TCCT architectures in most cases.Source: Appendix C · p. 13, Table C2 · p. 14
  • TCCT VI, Informer with dilated causal convolution, outperforms Informer in most cases but rarely outperforms Informer+.The authors link this comparison to TCCT VI using one encoder and ignoring earlier encoder feature maps.Source: Appendix C · p. 13, Table C2 · p. 14
  • TCCT IV and TCCT III, which combine dilated convolution with passthrough or all three architectures, achieve better forecasting results.The comparisons cover additional methods on the LSTF problem under univariate and multivariate settings.Source: Appendix C · p. 14, Table C2 · p. 14
Loading 2108.12784v2…