Source-linked AI summary
Structural-RNN: Deep Learning on Spatio-Temporal Graphs
Ashesh Jain, Amir R. Zamir, Silvio Savarese, Ashutosh Saxena
TL;DR
Many sequence-learning problems require high-level spatio-temporal structure that standard RNNs do not provide. The paper automatically converts arbitrary spatio-temporal graphs into rich, scalable, jointly trainable RNN mixtures, reporting significant improvements across diverse problems. S-RNN also supports modular motion generation, while evaluations include implementation and modeling-scope limitations.
Problem
RNNs model sequences effectively but lack intuitive high-level spatio-temporal structure for computer-vision problems involving structured interactions.
Method
S-RNN uses factor graphs and factor sharing to transform arbitrary spatio-temporal graphs into rich, scalable, feedforward RNN mixtures that are jointly trainable.
Results
Significant improvements were demonstrated on human motion modeling, human-object interaction, and driver maneuver anticipation.
Takeaways & Limitations
S-RNN provides a generic, principled way to combine high-level spatio-temporal graphs with RNN sequence modeling and supports modular generation of novel human motion.
Takeaways & Limitations
The experiments reproduced ERD and LSTM-3LR because the original authors' implementation was unavailable at submission time.
Abstract
from arXiv · showhide
Deep Recurrent Neural Network architectures, though remarkably capable at modeling sequences, lack an intuitive high-level spatio-temporal structure. That is while many problems in computer vision inherently have an underlying high-level structure and can benefit from it. Spatio-temporal graphs are a popular tool for imposing such high-level intuitions in the formulation of real world problems. In this paper, we propose an approach for combining the power of high-level spatio-temporal graphs and sequence learning success of Recurrent Neural Networks~(RNNs). We develop a scalable method for casting an arbitrary spatio-temporal graph as a rich RNN mixture that is feedforward, fully differentiable, and jointly trainable. The proposed method is generic and principled as it can be used for transforming any spatio-temporal graph through employing a certain set of well defined steps. The evaluations of the proposed approach on a diverse set of problems, ranging from modeling human motion to object interactions, shows improvement over the state-of-the-art with a large margin. We expect this method to empower new approaches to problem formulation through high-level spatio-temporal graphs and Recurrent Neural Networks.
1. Introduction
S-RNN combines high-level spatio-temporal graph structure with RNN sequence modeling by automatically transforming arbitrary st-graphs into rich, scalable RNN mixtures. The method is generic, modular, and jointly trainable, with improvements reported across diverse spatio-temporal problems.
- Spatio-temporal structures capture interactions among components across space and time, which is important for modeling activities, human motion, and other computer-vision problems.
- RNNs model long sequences successfully but lack an intuitive high-level spatio-temporal structure, motivating their combination with st-graphs.
- S-RNN transforms an arbitrary st-graph into a feedforward mixture of RNNs by rolling it through time, decomposing factors, grouping them semantically, and representing groups with RNNs.
- Factorization and factor sharing make the RNN mixture rich while maintaining scalability, and the resulting architecture is jointly trainable.
- S-RNN is generic and principled, applying to any problem formulated as an st-graph rather than being restricted to a task-specific architecture.
- S-RNN significantly outperforms unstructured RNNs and non-deep structured counterparts across human motion, human-object interaction, and driver decision-making problems.
2. Related Work
The related work spans structured representations, modular deep architectures, and graphical models for spatial and temporal reasoning. S-RNN is positioned as a generic, principled, scalable, jointly trainable RNN-mixture remedy for spatio-temporal graph problems.
- Spatio-temporal graphs are widely used to represent problems requiring spatial and temporal reasoning in robotics and computer vision.
- Prior modular deep architectures connect multiple networks to capture complex interactions, but they are mostly hand-designed for specific problems.
- Recursive Neural Networks provide generic feedforward architectures, but primarily target recursive structures such as parsed sentences and scenes.
- Deep graphical-model approaches combine neural networks with HMMs, MRFs, or CRFs for structured prediction, often including image segmentation and pose estimation.
- S-RNN represents a spatio-temporal graph through a factorized architecture in which each graph node and edge corresponds to a factor.
- Unlike CRFs, S-RNN is not probabilistic and is not intended to model the joint distribution over outputs.
3. Structural-RNN architectures
Structural-RNN converts a spatio-temporal graph into interconnected nodeRNNs and edgeRNNs derived from semantic factor components. The resulting bipartite, feedforward architecture processes node and edge features jointly while supporting parameter sharing and variable-sized contexts.
- Architecture construction: S-RNN starts by decomposing an unrolled spatio-temporal graph into node and edge factor components, then represents each factor with an RNN.The construction uses nodeRNNs for node factors and edgeRNNs for edge factors.
- Factor sharing: Semantic node partitions let similar nodes share factors, while corresponding semantic edge pairs share edge factors.This reduces parameterization and supports graphs whose number of nodes varies with context.
- Architecture construction: The nodeRNNs and edgeRNNs form a bipartite graph, with connections determined by factor pairs that jointly affect a node label.NodeRNNs combine neighboring edgeRNN outputs to predict node labels, while edgeRNNs model semantic interactions over time.
- Architecture construction: Algorithm 1 constructs S-RNN by partitioning edges, finding factor components, creating one RNN per factor, and connecting neighboring RNNs.The resulting graph is GR = ({REm}, {RVp}, ER).
- Training: During training, edgeRNNs receive incident edge-feature sequences, nodeRNNs concatenate node features with neighboring edge outputs, and prediction errors back-propagate through both.This jointly combines node and edge features to predict node labels.
- Training: For a human node connected to multiple objects, S-RNN sums the corresponding human-object edge features before feeding them to the shared edgeRNN.The summation handles a variable number of object nodes with a fixed architecture.
4. Experiment
S-RNN is evaluated across human motion, human-object activity, and driver maneuver anticipation, demonstrating structured modeling, modularity, and improvements over unstructured or prior approaches.
- Experimental scope: S-RNN is evaluated on human motion forecasting, human activity detection and anticipation, and driver maneuver anticipation.The experiments cover motion capture, human-object interactions, and real-world driving data.
- Human motion modeling and forecasting: S-RNN models human motion with separate nodeRNNs for body parts and edgeRNNs for their interactions, producing coherent long-term motion on complex aperiodic activities.Removing edgeRNNs causes the skeleton to freeze toward a mean position, while S-RNN remains close to ground truth short-term and human-like long-term.
- Going deeper into structural-RNN: S-RNN memory cells capture semantic sub-motions, including moving a leg forward and moving a hand close to the face across eating and smoking activities.The arm cell remains active while the hand stays near the face, while the leg cell alternates with forward movement of each leg.
- Going deeper into structural-RNN: S-RNN generates meaningful hybrid motions by exchanging nodeRNNs between models trained on different motion styles.The experiment combines a faster human with a slower left leg to generate a human jumping forward on one leg.
- Human motion modeling and forecasting: S-RNN generalizes with a smaller test error than ERD despite similar training error, and user studies rate its motion best.The motion forecasting results are evaluated using angle error at multiple time horizons, averaged over eight seed sequences per activity.
- Human activity detection and anticipation: S-RNN improves object-affordance F1-score by 44% over for anticipation and by 7% over [29] for detection.The reported F1-scores are averaged over all classes; edgeRNNs help transfer information from humans to objects.
- Driver maneuver anticipation: S-RNN outperforms the state-of-the-art AIO-HMM in every setting for anticipating driving maneuvers several seconds before they occur.It models the same st-graph using node and edge features from the prior work.
5. Conclusion
S-RNN combines spatio-temporal graphs with RNN sequence modeling through a scalable, generic architecture. It improves performance across three diverse problems and reveals semantic sub-motions and modularity in generated human motion.
- S-RNN combines high-level spatio-temporal graphs with RNN sequence modeling using factor graphs and factor sharing.
- S-RNN is scalable and applicable to problems expressed over spatio-temporal graphs while capturing their rich interactions.
- S-RNN achieves significant improvements on human motion modeling, human-object interaction, and driver maneuver anticipation.
- Visualized memory cells reveal semantic sub-motions, and the model generates novel human motion, demonstrating modularity.