Source-linked AI summary
Multi-Dimensional Recurrent Neural Networks
Alex Graves, Santiago Fernandez, Juergen Schmidhuber
TL;DR
Multidimensional tasks need contextual, warping-robust sequence modeling, but conventional RNNs are one-dimensional. The paper introduces MDRNNs, which extend recurrent connections and multidirectional processing to multidimensional data while retaining favorable scaling. Across two image-segmentation tasks, MDRNNs were more robust to input warping, though one dataset used realistic ray-traced rather than real images.
Problem
RNNs provide useful contextual access and warping robustness, but conventional architectures are one-dimensional and lack a direct approach for multidimensional data.
Method
MDRNNs replace one recurrent connection with one per data dimension and add multidirectional hidden layers to access surrounding context.
Results
MDRNNs were more robust to input warping than a state-of-the-art digit recognition algorithm across two image-segmentation tasks.
Takeaways & Limitations
MDRNNs extend RNN applicability to n-dimensional data while keeping algorithmic complexity linear in data points and parameters, with parameter count independent of dimensionality.
Takeaways & Limitations
The Air Freight images were ray-traced rather than real, although they included realistic lighting and specular effects.
Abstract
from arXiv · showhide
Recurrent neural networks (RNNs) have proved effective at one dimensional sequence learning tasks, such as speech and online handwriting recognition. Some of the properties that make RNNs suitable for such tasks, for example robustness to input warping, and the ability to access contextual information, are also desirable in multidimensional domains. However, there has so far been no direct way of applying RNNs to data with more than one spatio-temporal dimension. This paper introduces multi-dimensional recurrent neural networks (MDRNNs), thereby extending the potential applicability of RNNs to vision, video processing, medical imaging and many other areas, while avoiding the scaling problems that have plagued other multi-dimensional models. Experimental results are provided for two image segmentation tasks.
1 Introduction
RNNs offer contextual access and robustness to input warping, but existing RNN architectures are one-dimensional and require preprocessing for multidimensional tasks. MDRNNs extend RNNs directly to multidimensional data while avoiding the scaling problems of multidimensional HMMs.
- RNNs use previous context and adapt to stretched or compressed input patterns, making them robust to temporal warping.
- These properties are also desirable for multidimensional data, where recognition may require access to an entire object and robustness to perspective or distance changes.
- Existing RNN architectures are one-dimensional, so multidimensional inputs must first be reduced to a one-dimensional presentation.
- Multidimensional HMMs face exponential growth in Viterbi computation time and transition-probability memory as data size or dimensionality increases.
- MDRNNs bring RNN benefits to multidimensional data without the scaling problems described for multidimensional models.
2 Multi-Dimensional Recurrent Neural Networks
MDRNNs extend recurrent processing to multidimensional sequences by adding recurrent connections along every data dimension and ordering points so required predecessor activations are available. Multi-directional variants provide surrounding context while retaining linear dependence on data points and parameters, although standard recurrent architectures remain limited by vanishing gradients.
- MDRNN architecture: MDRNNs replace one recurrent connection with one connection per data dimension, combining external input with previous hidden activations along all dimensions.The forward pass processes points in a suitable order and stores the resulting hidden activations.
- Sequence ordering: A suitable ordering ensures each point is reached only after all points supplying its previous activations have been processed.For the 2D case, point (i,j) follows both (i-1,j) and (i,j-1), with boundary handling preventing nonexistent activations from entering the pass.
- Training: MDRNN training uses an n-dimensional extension of BPTT, processing the sequence in reverse during the backward pass and propagating n future derivatives through each hidden layer.Forward and backward passes each require one pass through the data sequence.
- Multi-directional MDRNNs: Multi-directional MDRNNs use 2^n hidden layers with origins at the sequence vertices, giving the output layer access to surrounding context in all directions.In two dimensions, four hidden layers use distinct axis choices and propagate in corresponding directions.
- Multi-directional MDRNNs: Multi-directional MDRNNs scale as O(2^n) with dimensionality, but their parameter count remains independent of data dimensionality and training remains linear in data points and parameters.The paper reports that several small hidden layers can outperform one large layer with the same total number of weights, presumably because processing is shared.
- Long-range context: Standard RNNs may not exploit all available context because influence through recurrent connections can decay or grow exponentially, the vanishing gradient problem addressed by LSTM architectures.LSTM uses gated memory blocks whose cells can store and access information over long periods.
3 Experiments
Experiments evaluated MDRNNs on Air Freight texture segmentation and pixel-wise MNIST digit segmentation, including warped inputs and analyses of contextual sensitivity.
- 3.1 Air Freight Data: 7.3% was the final pixel classification error rate on the Air Freight test set.The task assigned each pixel to its correct texture class using ray-traced color image sequences with ground-truth segmentation.
- 3.2 MNIST Data: On warped MNIST, MDRNN performance was considerably better than the compared convolutional network, although it was slightly worse on the clean test set.The warped test images used elastic deformations matching the degree used by Simard, with σ = 4.0 and α = 34.0.
- 3.1 Air Freight Data: The Air Freight network segmented reflective surfaces almost perfectly, indicating substantial use of contextual information for pixel classification.The sequence included difficult surfaces such as glass and a metal tube.
- 3.3 Analysis: The Jacobian analysis showed that an MDRNN output responded to context across the entire image and was particularly attuned to the digit outline.The plotted absolute Jacobian values represented output sensitivity to inputs at all image points.
4 Conclusion
The paper introduces MDRNNs for n-dimensional data, combining multi-directional context with a multi-dimensional LSTM variant. Across two image segmentation tasks, MDRNNs were more robust to input warping than a state-of-the-art digit recognition algorithm.
- 4 Conclusion: MDRNNs extend recurrent neural networks to n-dimensional data through multi-directional hidden layers and a multi-dimensional LSTM architecture.The multi-directional layers provide access to all contextual information.
- 4 Conclusion: Across two image segmentation tasks, MDRNNs were more robust to input warping than a state-of-the-art digit recognition algorithm.