Source-linked AI summary
Semantic Object Parsing with Graph LSTM
Xiaodan Liang, Xiaohui Shen, Jiashi Feng, Liang Lin, Shuicheng Yan
TL;DR
Semantic object parsing needs fine-grained part predictions informed by global structure, while existing CNNs and fixed-topology LSTMs provide limited or inefficient context modeling. The paper proposes Graph LSTM, which propagates information over adaptive superpixel graphs with confidence-driven ordering and neighbor-specific forget gates, and reports superiority across four parsing datasets.
Problem
Semantic object parsing requires global structural context for fine-grained part layouts, but CNNs capture limited local context and existing LSTMs use fixed, redundant propagation routes.
Method
Graph LSTM represents arbitrary-shaped superpixels as nodes in image-specific spatial graphs, then uses confidence-driven node updates and adaptively learned forget gates for neighboring nodes.
Results
The Graph LSTM substantially outperforms baselines on average IoU for PASCAL-Person-Part and demonstrates superiority across four public semantic object parsing datasets.
Takeaways & Limitations
Adaptive graph propagation provides a compact way to incorporate long-range context while preserving object and part boundaries for global reasoning.
Takeaways & Limitations
Graph LSTM has difficulty segmenting very small objects and parts with very similar appearances.
Abstract
from arXiv · showhide
By taking the semantic object parsing task as an exemplar application scenario, we propose the Graph Long Short-Term Memory (Graph LSTM) network, which is the generalization of LSTM from sequential data or multi-dimensional data to general graph-structured data. Particularly, instead of evenly and fixedly dividing an image to pixels or patches in existing multi-dimensional LSTM structures (e.g., Row, Grid and Diagonal LSTMs), we take each arbitrary-shaped superpixel as a semantically consistent node, and adaptively construct an undirected graph for each image, where the spatial relations of the superpixels are naturally used as edges. Constructed on such an adaptive graph topology, the Graph LSTM is more naturally aligned with the visual patterns in the image (e.g., object boundaries or appearance similarities) and provides a more economical information propagation route. Furthermore, for each optimization step over Graph LSTM, we propose to use a confidence-driven scheme to update the hidden and memory states of nodes progressively till all nodes are updated. In addition, for each node, the forgets gates are adaptively learned to capture different degrees of semantic correlation with neighboring nodes. Comprehensive evaluations on four diverse semantic object parsing datasets well demonstrate the significant superiority of our Graph LSTM over other state-of-the-art solutions.
1 Introduction
Semantic object parsing requires fine-grained part segmentation and global structural context, which fixed local CNN and LSTM approaches do not fully exploit. Graph LSTM addresses this with image-adaptive superpixel graphs, confidence-driven updates, and neighbor-specific forget gates.
- Semantic object parsing segments objects into multiple parts with fine-grained semantics, supporting applications such as action recognition and clothes retrieval.
- CNNs capture limited local context, while distinguishing confusing parts requires relative layouts and guidance from predictions of other semantic regions.
- Existing multidimensional LSTMs model long-range dependencies but use fixed topologies that ignore image boundaries, semantic groups, and computational redundancy.
- Graph LSTM represents arbitrary-shaped superpixels as graph nodes and uses spatially connected, image-adaptive edges for information propagation.
- A confidence-driven scheme selects the starting superpixel and updates nodes along a content-adaptive path, while different forget gates model neighbor-specific semantic correlations.
- Graph LSTM layers append to convolutional features to incorporate global structure context, with hidden states as reinforced features and memory states encoding global structures.
2 Related Work
Prior image-processing LSTMs extend recurrent modeling to multidimensional data but rely on fixed pixel-wise propagation and costly sequential computation. Graph LSTM instead propagates compact superpixel representations over varying graph topologies to capture long-distance dependencies.
- LSTMs progressed from sequential prediction to multidimensional image processing, including image generation, person detection, scene labeling, and object parsing.
- Existing image LSTMs use fixed neighborhood factorizations and sequentially process all pixels, limiting use of natural visual correlations and increasing redundant computation.
- Graph LSTM propagates information over varying graph topologies built on compact superpixel representations rather than locally fixed factorized LSTM units.
- The proposed structure incorporates global context through long-distance dependencies on superpixels, with a dependency field that can cover the entire image.
3 The Proposed Graph LSTM
Graph LSTM builds an image-specific superpixel graph and propagates information through progressively updated node states. Its units combine neighboring states with adaptive forget gates to model global structure while integrating with convolutional features.
- Graph Construction: Graph LSTM constructs each image’s graph from SLIC superpixels, using superpixels as nodes and spatially neighboring superpixels as edges.Node features are computed by averaging the features of pixels within each superpixel.
- Graph LSTM Architecture: The model can be appended to convolutional layers, with hidden states representing reinforced features and memory states encoding global structures.The first Graph LSTM layer receives convolutional feature maps; later layers can use residual connections.
- Confidence-driven Scheme: A confidence-driven scheme selects a starting node and determines an adaptive update sequence for propagating information across the graph.The method is designed to update all nodes while accommodating the image-specific graph topology.
- Graph LSTM Unit: Each node’s hidden and memory states are updated from its input, previous states, and neighboring-node states through the Graph LSTM recurrence.The memory system records information at graph nodes for communication with subsequent nodes and the next Graph LSTM layer.
- Neighbor State Aggregation: During node updates, neighboring hidden states are averaged using updated states for visited nodes and previous states for unvisited nodes.A visit flag distinguishes the two cases and averaging provides fixed-dimensional gate inputs despite varying neighborhood sizes.
- Adaptive Forget Gates: Graph LSTM learns distinct forget gates for different neighboring nodes, allowing their memory contributions to differ during state updates.The unit uses current-node input states with neighboring hidden states to determine adaptive forget gates, while shared neighbor weights support spatial transformation invariance.
4 Experiments
Experiments evaluate Graph LSTM across four object- and human-parsing datasets, multiple architectural variants, and node-updating schemes. Graph LSTM generally outperforms state-of-the-art and fixed-topology alternatives, while superpixel quantization and difficult visual cases remain boundaries.
- Results and Comparisons: Graph LSTM substantially outperforms four baselines on PASCAL-Person-Part average IoU, improving confused-part predictions by 4.95% for lower-arms and 6.67% for upper-legs over [8].The reported gains support exploiting global context for local prediction.
- Results and Comparisons: 70.05% average IOU for cow exceeds LG-LSTM by 7.26% and HAZN by 3.11%, while horse IOU improves 2.59% over the best state-of-the-art method.These comparisons are reported on the Horse-Cow parsing dataset.
- Results and Comparisons: 83.76% average F-1 on ATR exceeds Co-CNN's 76.95% and LG-LSTM's 80.97%, while Graph LSTM (more) improves 4.08% over LG-LSTM (more).Graph LSTM (more) uses 10,000 additional images as extra training data.
- Results and Comparisons: Graph LSTM substantially outperforms baselines on Fashionista when trained on the large ATR dataset and tested on 229 Fashionista images.The evaluation follows the stated cross-dataset training and testing setup.
- Discussions: Adaptive graph propagation remains stronger than locally fixed LSTM variants and superpixel smoothing, indicating gains beyond boundary information alone.Table 6 compares matched network architectures, while smoothing is applied as post-processing to previous LSTM confidence maps.
- Discussions: Confidence-driven node updating outperforms BFS and DFS alternatives, and adaptive forget gates outperform identical gates by treating neighboring semantic correlations differently.Using head or torso confidence yielded 61.03% and 61.45% versus 60.16% for all foreground classes; residual connections improved Avg IoU from 59.12% to 60.16%.
- Discussions: Using about 1,000 superpixels per image balances computation efficiency and accuracy, but superpixels can introduce quantization errors when one region contains different labels.The qualitative failure cases include very small objects and parts with very similar appearances.
5 Conclusion and Future Work
The paper concludes that Graph LSTM generalizes LSTMs to graph-structured data for semantic object parsing, using adaptive topology and confidence-driven state updates. Evaluations on four public datasets demonstrate significant superiority.
- Graph LSTM generalizes existing LSTMs to graph-structured data for semantic object parsing.
- Adaptive graph topology connects arbitrary-shaped superpixels through spatial neighborhood relations for each image.
- A confidence-driven scheme selects the starting node and determines the sequence for sequentially updating all node states.
- Evaluations on four public semantic object parsing datasets demonstrate the Graph LSTM’s significant superiority.