Source-linked AI summary
Time2Vec: Learning a Vector Representation of Time
Seyed Mehran Kazemi, Rishab Goel, Sepehr Eghbali, Janahan Ramanan, Jaspreet Sahota, Sanjay Thakur, Stella Wu, Cathal Smyth, Pascal Poupart, Marcus Brubaker
TL;DR
Time is important in synchronous and asynchronous event problems, but RNNs often handle it poorly and hand-crafted features are costly. The paper introduces Time2Vec, a learnable model-agnostic vector representation using linear and periodic components. Across several datasets, tasks, and architectures, replacing time with Time2Vec generally improves performance without hindering the remaining experiments.
Problem
RNNs often treat time as an ordinary input or assume synchronous data, while hand-crafted temporal features are expensive and domain-specific.
Method
Time2Vec represents scalar time as a learnable vector with a linear component and periodic activation components, using sine functions in the experiments.
Results
Across a range of datasets, tasks, and architectures, Time2Vec improved performance in most experiments, while the remaining results were not hindered.
Takeaways & Limitations
Time2Vec provides a reusable time representation that can be integrated into multiple existing architectures for synchronous and asynchronous event prediction.
Abstract
from arXiv · showhide
Time is an important feature in many applications involving events that occur synchronously and/or asynchronously. To effectively consume time information, recent studies have focused on designing new architectures. In this paper, we take an orthogonal but complementary approach by providing a model-agnostic vector representation for time, called Time2Vec, that can be easily imported into many existing and future architectures and improve their performances. We show on a range of models and problems that replacing the notion of time with its Time2Vec representation improves the performance of the final model.
1 Introduction
Time is important in both synchronous and asynchronous sequence problems, yet standard RNNs often use it poorly. Time2Vec addresses this gap with a learnable, model-agnostic time representation that improves performance across problems and architectures.
- Time features matter in tasks such as sales forecasting, patient-event prediction, and music recommendation.Relevant sequences may be synchronous or asynchronous, with dependencies across time and/or space.
- Most RNNs assume synchronous inputs or treat time as another input dimension, often failing to use it effectively.
- Hand-crafted time features can improve RNN use of time but are expensive and require domain expertise.
- Time2Vec is a learnable, model-agnostic vector representation that can be combined with many architectures.The authors evaluate it on synthesized and real-world datasets and across several architectures.
- Across a range of time-consuming problems and architectures, replacing time with Time2Vec boosts final-model performance.
2 Related Work
Prior work includes time-series models for synchronous and asynchronous events, as well as architectures that explicitly incorporate temporal information. Time2Vec differs by representing time as a reusable embedding with learned frequencies rather than proposing a new time-series model or architecture.
- Traditional time-series methods include autoregressive models, hidden-state models, point processes, normalizing flows, Gaussian processes, and support vector regression.These methods address synchronous or asynchronous events using different assumptions and representations.
- Many existing models assume regular sampling and stationarity, while asynchronous events may be binned or modeled directly.
- Time2Vec is a vector embedding for time, not a new time-series model, and is intended for use with many models.
- Unlike Fourier transforms with fixed frequencies, Time2Vec learns the frequencies used to encode temporal signals.
- Time2Vec can replace concatenated time features in existing models and can be integrated into architectures designed for temporal information.
3 Background & Notation
The paper distinguishes scalar time from vector time features and uses LSTM-based models as background architectures. LSTM+T concatenates time features with inputs, while TLSTM variants modify how time enters the recurrent computation.
- The notation uses lowercase letters for scalars, bold lowercase letters for vectors, and bold uppercase letters for matrices.
- The paper denotes scalar time by τ and vector time features by bold τ.Scalar time may represent absolute time, time from the start, or time from the last event.
- LSTM is presented as a successful RNN architecture for sequence modeling.
- LSTM+T handles time by concatenating time features, including engineered features, with the input before standard sequence modeling.
- The background introduces three time-aware architectures, TLSTM1, TLSTM2, and TLSTM3.Descriptions of TLSTM1 and TLSTM3 are provided in Appendix C, while TLSTM2 is omitted as similar to TLSTM3.
4 Time2Vec
Time2Vec is designed to capture periodic and non-periodic temporal patterns, remain invariant to time rescaling, and remain simple to combine with models. It uses a learnable linear component and periodic components, implemented with sine functions in the experiments.
- Time2Vec targets periodicity, non-periodic progression, invariance to time rescaling, and simplicity of integration.
- Time2Vec is defined as a vector of size k + 1 for scalar time τ, with one linear component and k periodic components.
- The periodic activation F and parameters ω_i and ϕ_i are learnable, and the experiments primarily use F = sin.With sine, ω_i is the frequency and ϕ_i is the phase shift.
- A sine component repeats over a period determined by its frequency, allowing periodic behavior without feature engineering.
- Periodic activations capture repeating events, while the linear term represents progression and non-periodic patterns.Sine functions can also support extrapolation to future and out-of-sample data.
- Time2Vec is invariant to time rescaling.
- The representation is inspired by positional encoding but represents continuous time rather than discrete sequence positions.
5 Experiments & Results
The experiments evaluate Time2Vec across datasets, architectures, activation functions, and frequency-learning choices. Results generally support its effectiveness, including improved performance over direct time inputs and learned periodic structure matching task patterns.
- Experimental setup: The experiments span synthesized periodic data, Event-MNIST, N_TIDIGITS18, SOF, Last.FM, and CiteULike, using classification and recommendation metrics.Reported measures include accuracy, Recall@q, and MRR@q.
- On the effectiveness of Time2Vec: Replacing time with Time2Vec improves LSTM performance on all evaluated datasets in most cases and never deteriorates it.The comparison is between LSTM+Time2Vec and LSTM+T.
- On the effectiveness of Time2Vec: Replacing time with Time2Vec improves both TLSTM1 and TLSTM3 on Last.FM and CiteULike in terms of Recall@10.The integration replaces each architecture’s scalar time representation with Time2Vec and adjusts the multiplied vectors to matrices.
- What does Time2Vec learn?: On the synthesized dataset, Time2Vec learns the period of 7 days and can perfectly classify test examples when the test pattern follows the training pattern.The learned sine components place multiples of 7 at signal peaks; the extrapolation claim is conditional on similar periodic patterns.
- Other activation functions: Periodic activations outperform non-periodic activations on Event-MNIST, while replacing sine with a non-periodic activation always predicts the majority class.Among the periodic choices, sine outperforms mod and triangle; the tested non-periodic choices include Sigmoid, Tanh, and ReLU.
- The use of periodicity in sine functions: Learned frequencies and phase-shifts improve performance over fixed frequency choices, and learned Event-MNIST frequencies show peaks near 0.35 and 0.85.The learned frequencies correspond to multiple full sine periods over the input range, indicating use of periodicity rather than only monotonic behavior.
6 Conclusion & Future Work
Time2Vec learns time features using sine and linear activations for synchronous and asynchronous event prediction tasks. It improved results in most experiments, while remaining applications were not hindered.
- Time2Vec automatically learns temporal characteristics including progression, periodicity, and scale through sine and linear activations.
- Across several datasets and tasks, Time2Vec improved results in the majority of experiments.
- The remaining experiments were not hindered by applying Time2Vec.
- Future Work: Optimization complications arose only in the experiment using a few sine functions.The authors hypothesize that combining sine functions with powerful models and using many sine functions may reduce optimization difficulty.
- Future Work: The paper leaves deeper theoretical analysis of this optimization hypothesis and development of better optimizers for future work.
A Implementation Detail
The experiments used fixed optimization and batching settings across several datasets, with released code reused for Last.FM and CiteULike. Competing models were parameter-matched, and the figures compare standard time inputs with Time2Vec.
- Experiments on Event-MNIST, N_TIDIGITS18, and SOF used PyTorch with Adam at learning rate 0.001 for 200 epochs.LSTM hidden sizes were 128 for Event-MNIST and SOF, and 64 for N_TIDIGITS18.
- Last.FM and CiteULike experiments reused Zhu et al.'s released code, replacing τ with t2v(τ).
- Competing models were designed to have approximately equal parameter counts by reducing LSTM hidden size when Time2Vec increased parameters.
- Figure 7 compares LSTM+T and LSTM+Time2Vec on Event-MNIST and raw N_TIDIGITS18.
- Figure 8 compares LSTM+T and LSTM+Time2Vec on SOF.
- Figure 9 compares LSTM+T and LSTM+Time2Vec on Last.FM.
B More Results
Additional experiments compare time representations across dataset variants, architectures, and Time2Vec sizes. Results were generally comparable or favorable, with the best sine-function count depending on the architecture.
- Event-binned N_TIDIGITS18 used 100 events per frame, and its LSTM+T and LSTM+Time2Vec results were on-par.
- Time2Vec vectors with 16, 32, and 64 sine functions were evaluated across the additional comparisons.
- In most cases, 64 sine functions outperformed or matched the 32- and 16-function configurations.
- TLSTM3 was an exception, with 16 sine functions performing best.The authors attribute this to possible overfitting when many temporal components are added to each of its two time gates.
- Figures 7–10 compare LSTM+T with LSTM+Time2Vec, while Figures 11–14 compare Time2Vec variants with TLSTM models on Last.FM and CiteULike.
C LSTM Architectures
The paper compares standard LSTM time concatenation with time-aware LSTM architectures that use one or two time gates. Time2Vec replaces scalar time inputs in these architectures with a learned vector representation.
- LSTM notation: The LSTM equations use xj for the jth event, ct for the memory cell, ht for the hidden state, and sigmoid and hyperbolic tangent activations.
- LSTM+T: LSTM+T concatenates each event representation xj with its time features before applying the standard LSTM equations.
- TimeLSTM: TLSTM1 introduces one time gate, tj, which controls the current input's influence and stores timing-history information in the cell state.
- TimeLSTM: TLSTM3 uses two time gates, t1j and t2j, and couples the input and forget gates.
- Time inputs: In the experiments, τj denotes the duration between the current and previous event.
- Time2Vec integration: TimeLSTM+Time2Vec replaces τ with t2v(τ) in TLSTM1 and substitutes corresponding learned matrices for the scalar-time weights.
- Time2Vec integration: For TLSTM3, Time2Vec replaces the scalar time input in both time gates and the output gate.
D Proofs
The paper proves that Time2Vec remains behaviorally unchanged under positive rescaling of time by adjusting its learned frequencies. The proof uses a mixed linear and sinusoidal representation and constructs an equivalent rescaled representation.
- Time2Vec is invariant to time rescaling.
- The representation uses a linear time component and sinusoidal components with learned frequencies and phase shifts.
- Replacing τ with α · τ for α > 0 scales the time argument in both the linear and sinusoidal components.
- An alternative representation with appropriately rescaled frequencies behaves the same way as the original representation.