Source-linked AI summary
Learning stochastic differential equations using RNN with log signature features
Shujian Liao, Terry Lyons, Weixin Yang, Hao Ni
TL;DR
Learning functionals on high-frequency, multimodal streams requires representations that preserve path information while supporting nonlinear interactions. The paper combines log-signature layers with RNNs, yielding Logsig-RNN and PT-Logsig-RNN models for SDE solution maps and streamed-data tasks. Across synthetic, skeletal-action, and gesture datasets, the models are reported to improve accuracy, efficiency, and robustness, including 93.27% accuracy on ChaLearn2013.
Problem
Method
The method combines parsimonious log-signature representations with RNN layers, adding a linear projection layer for high-dimensional input paths.
Source: §1 · pp. 2–3Results
The models improve performance over LSTM on synthetic and empirical data, with PT-Logsig-RNN achieving 93.27% state-of-the-art classification accuracy on ChaLearn2013 gesture data.
Source: §1 · p. 3, §6 · pp. 11–12Takeaways & Limitations
Logsig-RNN offers an efficient and robust approach for learning functionals on high-frequency or continuous streams, especially when conventional downsampling may fail.
Source: §1 · pp. 2–3, §6 · p. 12The log-signature lacks universality by itself and therefore must be combined with nonlinear models for learning.
Source: §2.3.4 · p. 7
Abstract
from arXiv · showhide
This paper contributes to the challenge of learning a function on streamed multimodal data through evaluation. The core of the result of our paper is the combination of two quite different approaches to this problem. One comes from the mathematically principled technology of signatures and log-signatures as representations for streamed data, while the other draws on the techniques of recurrent neural networks (RNN). The ability of the former to manage high sample rate streams and the latter to manage large scale nonlinear interactions allows hybrid algorithms that are easy to code, quicker to train, and of lower complexity for a given accuracy. We illustrate the approach by approximating the unknown functional as a controlled differential equation. Linear functionals on solutions of controlled differential equations are the natural universal class of functions on data streams. Following this approach, we propose a hybrid Logsig-RNN algorithm that learns functionals on streamed data. By testing on various datasets, i.e. synthetic data, NTU RGB+D 120 skeletal action data, and Chalearn2013 gesture data, our algorithm achieves the outstanding accuracy with superior efficiency and robustness.
1 Introduction
The paper combines log-signature representations with recurrent neural networks to learn functionals on streamed data, motivated by SDE theory and rough paths. It addresses high-frequency, oscillatory, and multimodal streams while targeting efficient, robust learning.
1 Introduction
- The Logsig-RNN blends a log-signature sequence layer with an RNN, using SDE-inspired representations to learn functionals on un-parameterized streamed data.The approach is presented as a hybrid of mathematically principled signatures and nonlinear recurrent models.Source: §1 · p. 2
- The log-signature layer reduces a high-frequency stream to coarser interval representations, significantly reducing the RNN time dimension and training cost.Unlike direct RNN processing, this representation preserves stream information over sub-intervals while shortening the sequence handled by the recurrent model.Source: §1 · p. 2
- Logsig-RNN is designed for high-frequency, continuous, missing, and highly oscillatory streams where downsampling or direct sampling can lose informative path behavior.The paper links this robustness to log-signatures' parsimonious representation and rough-path treatment of oscillations.Source: §1 · pp. 2–3
- The paper introduces the Log-signature Layer as a reusable neural-network transformation, proves Logsig-RNN universal for approximating SDE solution maps, and reports 93.27% state-of-the-art accuracy on ChaLearn2013 gesture data with PT-Logsig-RNN.The PT-Logsig-RNN adds a linear projection layer for high-dimensional input paths.Source: §1 · p. 3
1.1.1 Learning SDEs
- The paper frames learning SDE solution maps as a non-parametric problem without assuming the stochastic process distribution, extending beyond model-specific parameter estimation.SDEs are presented as general functionals on path space with applications across physical, biological, and financial systems.Source: §1.1.1 · p. 3
1.1.2 Rough paths theory in machine learning
- The work combines rough-path features with an RNN, which the authors describe as the first integration of a log-signature sequence with an RNN for general sequential data.It also extends prior back-propagation work from a single log-signature transformation to a sequence of log-signatures.Source: §1 · p. 2, §1.1.2 · p. 3
1.1.3 Time series modelling
- The paper addresses an open question about which learning algorithms should be combined with signature features, providing theoretical justification and numerical evidence for Logsig-RNN.Unlike functional principal components, the approach represents paths through their effects, namely functionals or controlled-equation solutions.Source: §1.1.3 · pp. 3–4, §1.1.4 · p. 4
2 The Log-Signature of a Path
The paper introduces signatures and log-signatures as path representations, then develops properties that support compact, time-invariant descriptions of streamed data. Log-signatures preserve signature information while reducing dimension, handling variable sampling and missing data, but require nonlinear models for learning.
2.1 A Path with finite p-variation
- Finite p-variation measures path roughness and provides the path class used to define signatures and log-signatures.The paper notes that larger p-variation corresponds to rougher paths and discusses Brownian and fractional Brownian examples.Source: §2.1 · p. 4
2.2 The log-signature of a path
- Signatures collect iterated integrals of a path, while the log-signature is the logarithm of the signature and its truncated form provides a graded path feature.The paper defines signatures in tensor algebra and introduces the logarithm map and truncated log-signature.Source: §2.2 · pp. 4–5
2.3.1 Bijection between the signature and the log-signature
- The signature and log-signature are bijective up to the same truncation degree, so the log-signature retains the information represented by the signature.At degree two, the paper illustrates the correspondence through symmetric and antisymmetric components.Source: §2.3.1 · p. 5, §2.2 · p. 5
2.3.2 Dimension Reduction
- The truncated log-signature generally has lower dimension than the corresponding signature because its Lie-bracket terms are linearly dependent.The paper states that this enables significant dimension reduction and gives antisymmetry as an example of linear dependence.Source: §2.3.2 · p. 6
2.3.3 Invariance under time parameterization
- Re-parameterizing a path changes its time-series representation but not its signature or log-signature, removing redundancy caused by traversal speed.Figure 2 illustrates different pen speeds producing the same signature and log-signature features.Source: §2.3.3 · p. 6, Figure 2 · p. 6
2.3.4 Missing Data, variable length and unequally time spacing
- Log-signature features provide fixed-dimensional descriptors for variable-length and unequally spaced series and are empirically more robust to missing data than signatures.The paper also notes that log-signatures lack universality and therefore must be combined with nonlinear models for learning.Source: §2.3.4 · pp. 6–7, Table 1 · p. 6
3 Logsig-RNN Network
The Logsig-RNN architecture transforms high-frequency streams into coarser sequences of log-signatures and processes them with an RNN. Path transformation layers further reduce spatial complexity and capture temporal or higher-order path information.
3.1 Log-Signature Layer
- The Log-Signature Layer maps an input sequence to log-signatures over a coarser time partition, potentially shrinking the RNN time dimension without trainable weights.Its output uses interval-wise truncated log-signatures, with output length determined by the coarser partition.Source: §3.1 · p. 7
- The layer supports backpropagation by differentiating each interval log-signature with respect to the path values contributing to that interval.The paper gives the derivative construction and states that the resulting algorithm implements backpropagation through the layer.Source: §3.1 · p. 7
3.2 Logsig-RNN Network
- Logsig-RNN incorporates the log-signature sequence as the RNN input, reducing time dimension while aiming for higher accuracy and efficiency than standard RNNs and Sig-OLR.For degree M = 1 it reduces to an RNN using path increments, while finer raw sampling can improve log-signature computation without increasing layer dimension.Source: §3.2 · pp. 7–8
3.3 Path Transformation Layers
- PT-Logsig-RNN adds path transformation layers before the log-signature layer, combining embedding, accumulation, and time incorporation.The architecture is presented as path transformations followed by a Log-Signature Sequence Layer, an RNN-type layer, and a fully connected layer.Source: §3.3 · pp. 8–9, Figure 3 · p. 8
- The embedding layer reduces path dimension, the accumulative layer extracts quadratic variation and higher-order statistics, and the time-incorporated layer appends time to the sequence.The time-incorporated transformation is stated to have a log-signature that can fully recover the path.Source: §3.3 · pp. 8–9
4 Universality of Logsig-RNN Network
The paper motivates Logsig-RNN from numerical SDE approximation, using recursive Taylor expansions driven by log-signatures. Under regularity conditions, this structure yields universal approximation of SDE solution maps.
4 Universality of Logsig-RNN Network
- The learning problem is to approximate the SDE solution map from iid input-output samples of driving paths and terminal solutions.The setup assumes finite p-variation paths and a smooth vector field satisfying stated regularity conditions.Source: §4 · p. 9
- Classical SDE approximation first performs local Taylor expansion and then pastes local approximations across a time partition to estimate the global solution.The paper describes this two-step strategy before defining its recursive estimator.Source: §4 · p. 9
- The recursive estimator uses truncated log-signatures on partition intervals and converges to the terminal solution as the partition mesh tends to zero when the vector field is sufficiently smooth.The estimator uses log-signature degree M and the paper cites Theorem B.1 for convergence.Source: §4 · p. 9
- The estimator recursion mirrors an RNN because its next state depends on the previous state and the interval log-signature.This structural correspondence motivates using Logsig-RNN to approximate SDE solution maps.Source: §4 · p. 9
- Theorem 4.1 states that, under the stated regularity and compactness conditions, a Logsig-RNN can approximate any such SDE solution to arbitrary ε with suitable log-signature degree and partition size.The theorem specifies conditions involving finite p-variation, smoothness, and bounds on M and the partition mesh.Source: §4 · pp. 9–10
5 Numerical Experiments
The experiments evaluate Logsig-RNN on synthetic SDE data and two skeletal recognition datasets, emphasizing accuracy, efficiency, and robustness. Across these settings, the method achieves strong or state-of-the-art performance while reducing sequence-processing cost and tolerating missing data.
5.1 Synthetic Data Generated by a SDE
- Logsig-RNN achieves the best accuracy across high-frequency, downsampled, and missing-data synthetic SDE inputs, while reducing training time from 50930s to 343s.The model reduces the RNN time dimension from 50k or 1k steps to 4; missing data are created by randomly removing 5% of downsampled points.Source: §5.1 · p. 10, Table 2 · p. 10
5.2 Action Recognition: NTU RGB+D 120 Data
- On NTU RGB+D 120, PT-Logsig-LSTM gains 0.4 percentage points from path transformation and a further 3.7 points from the Logsig layer on X-Subject.The method outperforms other methods on X-Subject and is beaten only by on X-Setup, where additional pose-estimation maps are used.Source: §5.2 · p. 11, Table 3 · p. 11
5.3 Gesture Recognition: Chalearn 2013 data
- On Chalearn 2013, PT-Logsig-RNN with M = 2 and N = 4 outperforms the compared methods in accuracy and reaches state-of-the-art performance with augmentation.The evaluation uses skeleton data from 20 three-dimensional joints and reports results with and without data augmentation.Source: §5.3 · p. 11, Table 4 · p. 11
- Logsig-RNN with M = 2 consistently beats the baseline RNN across different missing-frame ratios, demonstrating greater robustness to missing data.Frames are replaced by all-zero values in the validation set before evaluating the trained models.Source: §5.3 · p. 11, Table 5 · p. 11
6 Conclusion
The conclusion presents Logsig-RNN as an accurate, efficient, and robust approach for learning functionals on streamed data. The appendix establishes the signature and log-signature foundations, including invariance, uniqueness, universality, computation, and missing-data behavior.
6 Conclusion
- Logsig-RNN improves LSTM performance significantly on synthetic and empirical data, while PT-Logsig-RNN achieves state-of-the-art accuracy on Chalearn2013 gesture data.The conclusion highlights continuous gesture and action streams as a setting where the method is suited to higher-frequency sampling than conventional downsampling-based RNNs.Source: §6 · pp. 11–12
A.1 The signature of a path
- Signature features encode path increments and higher-order geometric information, and provide a universal basis for approximating continuous path functionals.The signature also satisfies Chen’s identity, making concatenated paths computationally compositional.Source: §A.1 · p. 13, §A.1.1 · p. 13, §A.1.6 · p. 15
A.1.4 Invariance under time parameterization
- Signatures are invariant under time re-parameterization, removing speed-related redundancy and supporting applications such as handwriting and video classification.The corresponding log-signature retains the path information up to tree-like equivalence.Source: §A.1.4 · p. 14, §A.2.4 · p. 16
A.2.2 The bijection between the signature and log-signature
- The truncated log-signature has no greater dimension than the corresponding signature because of linear dependencies among its Lie-series coordinates.The signature and log-signature are bijectively related, while the log-signature is a parsimonious representation.Source: §A.2.2 · p. 15, §A.3 · p. 16
A.3 Comparison of the Signature and Log-signature
- In a missing-data pen-trajectory experiment, both representations have mean absolute relative error no more than 6%, with the log-signature being more robust and lower-dimensional.The experiment drops at most 16 points from a trajectory of length 53 to create variable-length samples.Source: §A.3 · p. 16, Figure 4 · p. 17
A.4 Rough Path and the Extension Theory
- Rough path theory extends signature-based analysis to highly oscillatory and potentially nondifferentiable streams through finite p-variation and multiplicative functionals.The appendix defines geometric p-rough paths as limits of finite-variation paths in the p-variation distance.Source: §A.4 · pp. 17–18
Appendix B Proof of Global Approximation Theorem
Appendix B proves a global approximation theorem for the numerical solution of controlled differential equations. The proof combines rough-path solution estimates with continuity of the log-signature map on compact sets.
Appendix B Proof of Global Approximation Theorem
- The global approximation theorem bounds the numerical solution error for sufficiently small partition size when the input signature lies in a compact set.The bound depends on p, γ, the vector-field norm, and the radius of the compact set.Source: Appendix B · p. 18
- Continuity of the log-signature map on compact rough-path sets ensures that its image is compact, supporting the uniform approximation argument.The map sends a rough path to the log-signature restricted to a compact subinterval.Source: Appendix B · p. 18
- The proof invokes the rough-path SDE approximation result under a p-geometric rough path input and a Lipschitz vector field with regularity γ > p.A discrete finite time series is treated as an example of a geometric p-rough path.Source: Appendix B · pp. 18–19
Appendix C Proof of Universality of the Logsig-RNN model
Appendix C proves universality of the Logsig-RNN model by combining shallow-network approximation with stability of recurrent solution maps. The argument transfers approximation of the vector field to approximation of the resulting recurrent trajectories.
Appendix C Proof of Universality of the Logsig-RNN model
- The universality proof therefore combines neural-network approximation of the driving function with continuity and error propagation through the recurrent map.The relevant stability constant depends on derivative bounds and the sequence length.Source: §C.1 · p. 19
C.1 Auxiliary Lemmas
- A sigmoid network can uniformly approximate any continuous function on a compact subset of Euclidean space.The approximation uses a finite hidden width represented by a matrix and parameter vector.Source: §C.1 · p. 19
- The auxiliary proof uses a shared recursive structure between numerical solution approximation and the RNN recurrence.This structure connects the recurrent implementation to the approximation bounds for the controlled differential equation.Source: Figure 7 · p. 20, §C.1 · p. 20
- The recurrent solution map is stable: uniformly close vector fields produce uniformly close iterated states and outputs on the compact input set.The proof derives a recursive error inequality and bounds the resulting trajectory discrepancy by a constant times the vector-field approximation error.Source: §C.1 · p. 20
C.2 Proof of Theorem 4.1
The proof relates the recursive structure of an RNN to a numerical SDE approximation, then controls their error through approximation and continuity arguments. The appendix also introduces an additional UCI pen-digit experiment and implementation details for two gesture datasets.
C.2 Proof of Theorem 4.1
- The RNN recursion and numerical SDE approximation share a recursive structure, allowing their outputs to be compared through a common map.The map generates sequences recursively from inputs, with the RNN and numerical solution arising from different choices of the update function.Source: §C.2 · pp. 20–21
- The proof decomposes the approximation error using the triangle inequality and identifies E2 as the difference between the RNN and numerical-SDE recursive maps.Source: §C.2 · p. 21
- E1 can be made arbitrarily small by decreasing the discretization step and increasing the log-signature truncation degree.Source: §C.2 · p. 21
- E2 can likewise be made arbitrarily small because shallow neural networks approximate continuous functions and the recursive map is continuous.Source: §C.2 · p. 21
Appendix D Numerical Examples
- The appendix adds a UCI pen-digit experiment and implementation details for the NTU RGB+D 120 and Chalearn 2013 datasets.Source: Appendix D · p. 21
D.1 UCI Pen-Digit Data
On UCI pen-digit recognition, Logsig-RNN improves testing accuracy and substantially reduces training time relative to the reported RNN baselines. It also remains more accurate under missing data and down-sampling.
D.1 UCI Pen-Digit Data
- 97.88% testing accuracy is achieved by Logsig-RNN with M = 4 and N = 4, versus 95.80% for RNN0.Source: §D.1 · p. 21
- 30% of RNND's training time and 3% of RNN0's training time are required by Logsig-RNN.Source: §D.1 · p. 21
- Figure 8 presents the testing-set accuracy comparison for Logsig-RNN.Source: Figure 8 · p. 21
- For missing-data tests, the proposed method significantly outperforms the other methods.Source: §D.1 · p. 21, Table 6 · p. 22
- Above 80% accuracy is retained after down-sampling the test data to length 8, about four times the baseline methods' accuracy.Source: §D.1 · p. 21, Figure 9 · p. 22
D.2 Action Recognition: NTU RGB+D 120 Data
The action and gesture recognition implementations combine convolutional path-processing layers, log-signature features, and LSTM layers. The two datasets use distinct preprocessing, regularization, and augmentation configurations.
D.2 Action Recognition: NTU RGB+D 120 Data
- The NTU RGB+D 120 model uses two Conv2D layers, one Conv1D layer, a log-signature input before LSTM, dropout 0.8, and 256 LSTM hidden neurons.Source: §D.2 · p. 22
D.3 Gesture Recognition: Chalearn 2013
- The Chalearn 2013 model centers skeletons, normalizes and resamples clips to 39 frames, then applies convolutional path transforms, log-signatures, and an LSTM.Source: §D.3 · p. 22
- Chalearn 2013 regularization uses dropout rates 0.3 and 0.5 for the embedding and LSTM layers, respectively, with 128 LSTM hidden neurons.Source: §D.3 · p. 22
- Chalearn 2013 training uses coordinate rotations, temporal shifts, and Gaussian joint noise for data augmentation.Source: §D.3 · p. 22
Appendix E The Universality of Controlled Differential Equations
The appendix establishes universality for linear functionals of controlled-differential-equation solution maps under stated regularity conditions. The proof uses algebraic closure and Stone–Weierstrass density.
Appendix E The Universality of Controlled Differential Equations
- The controlled differential equation is driven by a finite-1-variation path through a sufficiently regular vector field.Source: Appendix E · p. 22
- The solution map sends an input path and initial value to the corresponding differential-equation solution.Source: Appendix E · p. 22
- For linear vector fields, linear functionals on the solution map form an algebra under componentwise multiplication.Source: Appendix E · pp. 22–23
- Linear functionals on the solution map are dense in continuous, time-parameterization-invariant functionals on the path space.Source: Appendix E · p. 23
- The proof invokes Stone–Weierstrass after showing algebra closure using a trivial vector field and initial value y0 = 1.Source: Appendix E · p. 23
- Universality is established for linear functionals on controlled-differential-equation solutions when the vector fields satisfy the required regularity condition.Source: Appendix E · pp. 22–23