PaperlayerView exact revision on arXiv

Source-linked AI summary

Temporal Fusion Transformers for Streamflow Prediction: Value of Combining Attention with Recurrence

Sinan Rasiya Koya, Tirthankar Roy

arXiv:2305.12335v1cs.LGphysics.geo-ph

TL;DR

Whether combining recurrence and attention can improve streamflow prediction is the question this study tests. It compares TFT, Transformer, and LSTM models on 2,610 Caravan basins worldwide. TFT outperforms the other two models, and its interpretability can provide insight into hydrological processes.

  • Problem

    The study asks whether combining recurrence with attention can improve streamflow prediction, with attention identified as a candidate for the task.

    Source: §1 · p. 2

  • Method

    The study trains and compares TFT, Transformer, and LSTM models on 2,610 globally distributed Caravan basins selected after missing-data filtering.

    Source: §1 · p. 2, §2.1 · p. 3

  • Results

    TFT outperformed LSTM and Transformer across 2,610 test basins; median KGE was 0.705 for TFT, 0.647 for LSTMs, and 0.563 for Transformers.

    Source: §3.1 · p. 10, §4 · p. 14

  • Takeaways & Limitations

    TFT’s variable importance and attention patterns can offer insight into hydrological processes, including long-term dependencies and relevant basin inputs.

    Source: §3.2 · p. 11, §3.3 · p. 13, §4 · p. 14

    The attention setup assumes that today’s discharge cannot be influenced by future input states and masks future positions.

    Source: §2.3.3 · p. 6

Abstract

from arXiv · show

Over the past few decades, the hydrology community has witnessed notable advancements in streamflow prediction, particularly with the introduction of cutting-edge machine-learning algorithms. Recurrent neural networks, especially Long Short-Term Memory (LSTM) networks, have become popular due to their capacity to create precise forecasts and realistically mimic the system dynamics. Attention-based models, such as Transformers, can learn from the entire data sequence concurrently, a feature that LSTM does not have. This work tests the hypothesis that combining recurrence with attention can improve streamflow prediction. We set up the Temporal Fusion Transformer (TFT) architecture, a model that combines both of these aspects and has never been applied in hydrology before. We compare the performance of LSTM, Transformers, and TFT over 2,610 globally distributed catchments from the recently available Caravan dataset. Our results demonstrate that TFT indeed exceeds the performance benchmark set by the LSTM and Transformers for streamflow prediction. Additionally, being an explainable AI method, TFT helps in gaining insights into the streamflow generation processes.

1. Introduction

LSTM’s step-by-step processing can make it difficult to learn long-term patterns [9, 10], motivating attention as a candidate for streamflow prediction. This study tests whether combining recurrence and attention improves predictions using Temporal Fusion Transformers (TFTs).

1. Introduction

  • LSTM processes data step by step and can struggle to learn long-term patterns [9, 10], motivating attention-based streamflow prediction.Attention-based models can access the full sequence, including distant past information.Source: §1 · pp. 1–2
  • TFT combines LSTM recurrence and Transformer attention, with variable-selection networks and attention scores that support interpretation; this is its first hydrology application.Variable-selection networks identify relevant inputs, while attention scores indicate relevant past time steps.Source: §1 · p. 2
  • The study tests whether combining recurrence and attention improves streamflow prediction by comparing TFT, Transformer, and LSTM models across Caravan basins without significant missing data.The authors report that TFT is a better candidate for streamflow prediction.Source: §1 · p. 2

2.1. Study Area and Data

The study uses Caravan data to compare three machine-learning models across a globally distributed set of basins. The final sample contains 2,610 basins after filtering for missingness and record length.

2.1. Study Area and Data

  • 2,610 globally distributed basins with at least 30 years of data remained after the study’s missing-data filtering.Basins with continuous gaps of 30 days or more were removed; remaining gaps were spline-interpolated before filtering out shorter records.Source: §2.1 · p. 3
  • Caravan provides daily streamflow, meteorological forcing, and static attributes for 6,830 basins, drawing on standardized regional datasets.The study uses 38 continuous and 211 static variables.Source: §2.1 · p. 3
  • The study compares LSTM, Transformer, and TFT models on basin-aggregated Caravan data across a wide global range of basins.Caravan includes data from the US, Great Britain, Brazil, Australia, Chile, HYSETS, and LamaH-CE.Source: §2.1 · p. 3

2.2. LSTM

LSTM is a recurrent network used for streamflow prediction, with cell and hidden states that carry information forward through a sequence. The study implements a three-layer LSTM with ten hidden nodes per layer.

2.2. LSTM

  • LSTM cell and hidden states pass forward between time steps, retaining long-term and short-term information, respectively.The cell state is described as helping learn long-term dependencies in sequential data.Source: §2.2 · p. 4
  • Figure 2 depicts an LSTM cell unrolled across three time steps, illustrating its recurrent sequence structure.At each step, the cell uses the current input and prior hidden state to compute its states.Source: Figure 2 · p. 4, §2.2 · p. 4
  • The implemented LSTM has three layers with ten hidden nodes per layer, and maps the final time-step output to target streamflow.The final output passes through a linear layer and sigmoid activation, then another linear layer.Source: §2.2 · p. 4

2.3. Transformers

Transformers process sequences through attention rather than recurrence, so positional encoding supplies sequence-position information. The study’s implementation combines multi-head attention with encoder and decoder blocks for one-step streamflow prediction.

2.3. Transformers

  • Transformers attend to the entire input sequence at once, unlike LSTMs, using positional encoding, attention, and encoder and decoder blocks.This architecture was implemented for the study as shown in Figure 3.Source: §2.3 · pp. 4–5, Figure 3 · p. 6

2.3.1. Positional Encoding

  • Positional encoding adds position information to linearly transformed inputs because Transformers lack the positional structure provided by recurrence.The study uses a transformed input dimension of 16; its positional encoding represents relative position.Source: §2.3.1 · p. 5

2.3.2. Attention

  • Multi-head self-attention forms queries, keys, and values from the input, applies attention separately across four heads, then concatenates their outputs.The resulting sequence has the same size as the decoder input.Source: §2.3.2 · p. 5

2.3.3. Encoder

  • The encoder stacks two blocks, each combining multi-head attention and a feed-forward layer with residual connections and layer normalization.Masked attention prevents a discharge prediction from depending on future input states.Source: §2.3.3 · pp. 5–6

2.3.4. Decoder

  • The decoder uses three stacked blocks and maps their output to seven streamflow quantiles for the one-step forecast.Because the output sequence has one time step, the decoder receives one time step and its masking is inactive.Source: §2.3.4 · p. 6

2.4. Temporal Fusion Transformers

TFT combines recurrent encoder–decoder LSTMs with multi-head attention over processed sequence information. Its other components gate information flow, select variables over time, and encode static covariates into context vectors.

2.4. Temporal Fusion Transformers

  • TFT processes selected inputs with encoder and decoder LSTMs, then uses multi-head attention over the full sequence to map outputs to seven streamflow quantiles.Past known inputs, such as meteorological forcings, enter the encoder; known future inputs, such as the relative time index, enter the decoder.Source: §2.4 · pp. 7–8, Figure 4 · p. 7

2.4.1. Gating Mechanisms in TFT

  • Gated Residual Networks combine a basic input with optional static-covariate context, apply gated transformations, and layer-normalize a residual connection.The sequence is dense ELU, linear with dropout, a Gated Linear Unit, then normalization of the input-plus-gated output.Source: §2.4.1 · p. 8

2.4.2. Variable Selection Networks in TFT

  • Variable selection processes each time-step variable through a GRN and softmax-derived weights, which combine the processed variable representations.The selection network identifies features relevant to streamflow; its GRN weights are shared across time steps.Source: §2.4.2 · pp. 8–9

2.4.3. Static Covariate Encoding in TFT

  • TFT converts weighted static variables into four context vectors that are infused into different model components for processing.One vector, c_s, is used to find variable weights in a variable-selection network.Source: §2.4.3 · p. 9

2.5. Training

The study trains separate models for each basin, using a 365-day input sequence to predict one day of streamflow. TFT training is relatively costly with the large input set.

2.5. Training

  • Each basin’s 365-day sequence contains 249 variables and predicts one day; models use a 70:10:20 train-validation-test split and early stopping.TFT uses quantile loss, batch size 32, and at most 100 epochs; its training takes one hour per basin on two GPU cores, and performance is evaluated with KGE.Source: §2.5 · pp. 9–10

3. Results and Discussion

Across 2,610 global test basins, TFT outperformed LSTMs and Transformers; the results also examine its basin-specific variable rankings and attention to both recent and distant history.

3.1. Performance Comparison

  • For hysets_10BE007, TFT captured peak discharge and the rising and falling limbs better than the other models.The authors suggest its access to near and distant data may help capture these hydrograph features.Source: §3.1 · p. 10, Figure 6 · p. 11
  • Median KGE was 0.647 for LSTMs, 0.563 for Transformers, and 0.705 for TFTs across 2,610 global test basins, with TFT outperforming both.Figure 5 shows the test-score distributions; Figure 6 compares the three models' hydrographs for one basin.Source: §3.1 · p. 10, Figure 5 · p. 11

3.2. Model Interpretability

  • For hysets_10BE007, TFT ranked past streamflow, precipitation, soil water, and potential evaporation highest among continuous inputs; November mean PET led static inputs.These rankings are basin-specific; the authors note snow water equivalent could be more important where snow processes shape runoff.Source: §3.2 · pp. 11–12, Figure 7 · p. 12

3.3. Learning Long-term Patterns

  • TFT attention scores were high for both immediate (1-month) and distant (12-month) lookbacks, consistent with near-term context and seasonal dependencies.Models were trained on 365-day sequences; the authors propose that multi-year sequences could capture multiple seasons and longer-term patterns.Source: §3.3 · pp. 12–13, Figure 8 · p. 13

4. Conclusion

The study concludes that TFT outperforms LSTMs and Transformers for streamflow prediction, while capturing hydrograph features, learning long-term dependencies, and providing variable-importance insights.

4. Conclusion

  • The study trained LSTM, Transformer, and TFT models separately on 2,610 Caravan basins selected to exclude significant missing data.The models were compared using performance scores and generated hydrographs.Source: §4 · p. 13
  • TFT, combining recurrence and attention, outperformed LSTMs and Transformers in the study's streamflow predictions.The authors describe this as introducing TFT implementation in hydrology and setting a new benchmark.Source: §4 · p. 14
  • TFT captured hydrograph peaks and limbs more efficiently and demonstrated learning of long-term dependencies through attention.Source: §4 · p. 14

Conflicts of Interest

Related papers

Loading 2305.12335v1…