Source-linked AI summary
Semantic Object Parsing with Local-Global Long Short-Term Memory
Xiaodan Liang, Xiaohui Shen, Donglai Xiang, Jiashi Feng, Liang Lin, Shuicheng Yan
TL;DR
Semantic object parsing needs contextual information beyond the limited neighborhoods used by standard CNNs for fine-grained pixel labeling. The paper proposes LG-LSTM layers that combine local spatial, depth-wise, and global image guidance within end-to-end CNN feature learning. Across three public datasets, the architecture significantly outperforms previously published object-parsing methods.
Problem
Standard CNNs rely on limited neighboring context, motivating a method that captures both local and global dependencies for fine-grained semantic object parsing.
Method
LG-LSTM appends stacked layers to CNNs, combining eight spatial LSTMs, one depth LSTM, and global hidden cells derived from pooled image regions.
Results
The architecture significantly outperforms previously published methods on the Horse-Cow, ATR, and Fashionista parsing datasets.
Takeaways & Limitations
Stacked LG-LSTM layers provide an end-to-end way to incorporate long-distance and short-distance spatial dependencies into pixel-wise semantic labeling.
Abstract
from arXiv · showhide
Semantic object parsing is a fundamental task for understanding objects in detail in computer vision community, where incorporating multi-level contextual information is critical for achieving such fine-grained pixel-level recognition. Prior methods often leverage the contextual information through post-processing predicted confidence maps. In this work, we propose a novel deep Local-Global Long Short-Term Memory (LG-LSTM) architecture to seamlessly incorporate short-distance and long-distance spatial dependencies into the feature learning over all pixel positions. In each LG-LSTM layer, local guidance from neighboring positions and global guidance from the whole image are imposed on each position to better exploit complex local and global contextual information. Individual LSTMs for distinct spatial dimensions are also utilized to intrinsically capture various spatial layouts of semantic parts in the images, yielding distinct hidden and memory cells of each position for each dimension. In our parsing approach, several LG-LSTM layers are stacked and appended to the intermediate convolutional layers to directly enhance visual features, allowing network parameters to be learned in an end-to-end way. The long chains of sequential computation by stacked LG-LSTM layers also enable each pixel to sense a much larger region for inference benefiting from the memorization of previous dependencies in all positions along all dimensions. Comprehensive evaluations on three public datasets well demonstrate the significant superiority of our LG-LSTM over other state-of-the-art methods.
1. Introduction
Semantic object parsing requires fine-grained pixel recognition that benefits from both local neighboring context and global image information. LG-LSTM integrates these dependencies into end-to-end feature learning within convolutional networks.
- Semantic object parsing segments an object region into semantic parts, supporting detailed image understanding and applications including captioning, clothing retrieval, re-identification, and behavior analysis.
- CNN pixel classification mainly uses limited neighboring context, while recognizing parts such as left-leg and right-leg requires larger local context and a global image perspective.
- LG-LSTM addresses information attenuation in deep CNN chains by incorporating local and long-distance dependencies into intermediate feature learning.
- Each LG-LSTM layer combines depth-wise information with eight neighboring spatial positions, using spatial LSTMs for interactions and memorization of contextual dependencies.
- Global guidance is obtained by partitioning previous hidden-cell maps into nine grids and max-pooling each grid to provide image-level features for every position.
- Stacked LG-LSTM layers continuously update local and global context during end-to-end learning, with shared spatial and depth-LSTM weights across layers.
- Evaluations on Horse-Cow, ATR, and Fashionista datasets report significant improvements over previously published object-parsing methods.
2. Related Work
Prior work applies CNNs and multidimensional LSTMs to parsing and image-processing tasks, but LG-LSTM adapts Grid LSTM to combine local and global dependencies for complex semantic object parsing.
- Recent semantic object parsing methods commonly use deep CNNs and advanced architectures, whereas LG-LSTM incorporates long-distance and short-distance dependencies directly into CNN feature learning.
- LSTM extensions have supported multidimensional image processing, including biomedical segmentation, person detection, and scene labeling, with Grid LSTM enabling multidimensional communication.
- The proposed architecture appends stacked LG-LSTM layers after convolutional feature extraction, then uses a final feed-forward convolutional layer to produce parsing results.
- LG-LSTM extends Grid LSTM for higher-level object parsing by combining short-distance dependencies with global information from the whole image.
- Unlike the closest scene-labeling approach using 2D LSTMs on non-overlapping patches, LG-LSTM applies eight spatial LSTMs and one depth LSTM at each pixel, with global hidden cells as inputs.
3. The Proposed LG-LSTM Architecture
The LG-LSTM architecture augments convolutional features with local, global, and depth-wise recurrent context. Stacked layers propagate these dependencies across spatial dimensions and feed the resulting features into a final pixel-wise parsing layer.
- Architecture: The parsing network adapts convolutional feature maps through a transition layer before processing them with stacked LG-LSTM layers and a final feed-forward layer.The final layer infers C confidence maps for the semantic labels, including background.
- Local-Global LSTM layers: The LG-LSTM layer combines local hidden cells, global hidden cells, and depth states through multi-dimensional LSTMs derived from Grid LSTM.Distinct memory cells support individual information propagation along the spatial dimensions, while shared spatial weights preserve invariance across directions.
- Local interactions: Each position receives hidden cells from eight neighboring pixels and one depth LSTM, enabling separate spatial and depth-wise information propagation.The spatial LSTMs propagate information in distinct directions, while the depth LSTM tracks previous information at the same position through memory cells.
- Global interactions: Global hidden cells are produced by partitioning hidden-cell maps into nine grids and applying channel-wise global max pooling within each grid.The resulting 9 × d global hidden cells guide predictions at every position using global information from the previous LG-LSTM layer.
- Context propagation: Stacking LG-LSTM layers enlarges each position’s receptive field while recurrent memory preserves previously computed contextual dependencies.This combines short-distance cues from neighboring pixels with long-distance information from the whole feature map.
4. Experiments
Experiments evaluate LG-LSTM on Horse-Cow, ATR, and Fashionista parsing benchmarks, with comparisons to established methods and architectural variants. Across datasets and ablations, results support benefits from local-global context and stacked recurrent layers.
- Experimental settings: LG-LSTM was evaluated on the Horse-Cow, ATR, and Fashionista datasets using comparisons with state-of-the-art methods.Horse-Cow uses IoU and pixel-wise accuracy; ATR and Fashionista report human-parsing performance.
- Benchmark results: 9.47% higher overall pixel accuracy than SPS was achieved for the horse class, alongside 68.73% average IOU.The reported horse-class gains over HC and Joint were 3.74% and 2.43% in overall pixel accuracy, while average IOU exceeded HC by 6.75%.
- Benchmark results: 80.97% average F-1 exceeded ATR’s 64.38% and Co-CNN’s 76.95%, while LG-LSTM (more) improved average F-1 over Co-CNN (more) by 3.98%.The method also reported large gains for small-region labels including hat, belt, bag, and scarf.
- Benchmark results: LG-LSTM substantially outperformed baselines across all reported metrics on Fashionista’s 229 test images.Training followed the stated protocol of using the ATR training data before testing on Fashionista.
- Ablation studies: 4.6% IOU improvement over VGG16 was observed on the horse class, while five extra convolutional layers reduced mean IOU by 2.78% on horse and 4.86% on cow.The comparisons were designed to test LG-LSTM against both a basic convolutional baseline and a deeper convolutional alternative.
- Ablation studies: Using eight neighboring connections improved cow-class IOU by 4.19% and 2.94% over two- and four-connection variants, while removing global guidance reduced IOU by 1.27% on horse and 1.81% on cow.Qualitative results also describe more consistent regions and improved separation of confusing labels when local or global context is available.
5. Conclusions and Future Work
The work proposes LG-LSTM for semantic object parsing and reports effectiveness on pixel-wise semantic labeling across three public datasets. Future work will develop a pure LG-LSTM network to hierarchically exploit local and global image connections and retain long-period hidden states.
- LG-LSTM jointly captures long-distance and short-distance spatial dependencies using global hidden cells and local hidden cells across spatial and depth dimensions.
- Extensive results on three public datasets demonstrated the effectiveness of LG-LSTM for generating pixel-wise semantic labeling.
- Future work will replace convolutional layers with designed LG-LSTM layers to hierarchically exploit whole-image connections and capture complex visual patterns through long-period hidden states.