Source-linked AI summary
Graph-FCN for image semantic segmentation
Yi Lu, Yaran Chen, Dongbin Zhao, Jianxin Chen
TL;DR
Semantic segmentation needs pixel-level location information alongside high-level semantics, but deep convolutional feature extraction can lose that local detail. Graph-FCN converts FCN-derived image features into a graph, applies a two-layer GCN for node classification, and reports a 1.34% mIOU improvement over FCN while retaining the FCN-16s forward structure.
Problem
Deep learning-based semantic segmentation can lose local location information while extracting high-level features, although segmentation requires both semantic and pixel-level location information.
Method
Graph-FCN initializes a graph from FCN-derived image features and uses a two-layer GCN to propagate messages among neighboring nodes for semantic segmentation.
Results
1.34% mIOU improvement was obtained by adding the GCN parts as an additional loss while retaining the FCN-16s forward output.
Takeaways & Limitations
Graph-FCN enlarges the receptive field without node disappearance and achieves higher mIOU than FCN-16s for the same forward structure.
Takeaways & Limitations
The graph assumes that each node connects to its nearest l nodes.
Abstract
from arXiv · showhide
Semantic segmentation with deep learning has achieved great progress in classifying the pixels in the image. However, the local location information is usually ignored in the high-level feature extraction by the deep learning, which is important for image semantic segmentation. To avoid this problem, we propose a graph model initialized by a fully convolutional network (FCN) named Graph-FCN for image semantic segmentation. Firstly, the image grid data is extended to graph structure data by a convolutional network, which transforms the semantic segmentation problem into a graph node classification problem. Then we apply graph convolutional network to solve this graph node classification problem. As far as we know, it is the first time that we apply the graph convolutional network in image semantic segmentation. Our method achieves competitive performance in mean intersection over union (mIOU) on the VOC dataset(about 1.34% improvement), compared to the original FCN model.
1 Introduction
Semantic segmentation requires both high-level semantic information and pixel-level location information, but deep convolutional networks can lose local location details. Graph-FCN addresses this by initializing a graph from an FCN and applying graph convolution to preserve local information during segmentation.
- Semantic segmentation must predict each pixel’s category while learning object outlines, positions, and categories from semantic and local information.
- Deep convolutional networks obtain powerful high-level features, but their generalization can accompany loss of local location information.
- Pooling and small high-level feature maps reduce pixel-level location information, motivating methods such as FCN, SegNet, and Deeplab.
- Graph-based segmentation represents pixels as nodes and their dissimilarities as edges, with probabilistic graphical models refining segmentation results.
- Graph-FCN initializes a small graph model with FCN node annotations and Gaussian-kernel edge weights before applying graph convolution.
- GCN exchanges messages among neighboring nodes without eliminating nodes, enlarging the receptive field while avoiding local location-information loss.
2 Problem Formulation
FCN performs pixel-to-pixel semantic segmentation by extracting convolutional features and upsampling them to the image size. Its pooling operations enlarge receptive fields but can reduce pixel-position information, while skip connections fuse feature layers at different scales.
- FCN replaces fully connected layers with convolutional layers to perform end-to-end pixel classification.
- The model extracts local receptive-field features, upsamples feature maps, and produces a predicted label for each image pixel.
- FCN trains semantic segmentation with cross-entropy loss computed over pixels in the label image.
- Pooling increases the receptive field and decreases resolution, which leads to loss of pixel-position information.
- FCN introduces skip connections to fuse feature layers at different scales.
- FCN-16s is selected as Graph-FCN’s base model because it is 0.3% below FCN-8s in mIOU and has a more concise structure.
3 Graph Model in Semantic Segmentation
The model represents image segmentation as graph node classification, initializing node annotations from FCN features and using GCN message propagation to expand receptive fields without discarding nodes. Graph-FCN combines FCN and GCN outputs through end-to-end training with cross-entropy losses.
- Graph representation: The graph data are represented by nodes, edges, and graph features, with node annotations stored in a |N| × S matrix.The graph feature U is omitted because it is not involved in this task.
- Node initialization: FCN-16s initializes each node annotation by concatenating two feature vectors from stride-16 and upsampled stride-32 maps with the node location.The resulting annotation contains local receptive-field features, and node labels are obtained by pooling the raw label image.
- Graph edges: Each node connects to its nearest l nodes, with edge weights defined by a Gaussian-kernel adjacency matrix.These edges transfer node annotations during graph-neural-network propagation.
- Graph convolution: The GCN uses normalized graph operators and message propagation between neighboring nodes as the graph analogue of convolution and pooling.The one-layer operator is expressed as X_k+1 = ˆA X_k Θ, with ˆD defined as the degree matrix of I + A.
- Receptive field: A 2-layer GCN is adopted because Laplacian smoothing can cause oversmoothing in deeper networks.For the described instance, its receptive field is 5 × 32 × 32, five times that of FCN-16s, while no nodes disappear.
- Training: Graph-FCN trains the FCN-16s and 2-layer GCN end to end, applying cross-entropy losses to both outputs while sharing convolutional feature-extraction weights.The two losses are denoted L1 and L2 in the model structure.
4 Experiments
Experiments on VOC2012 evaluate Graph-FCN against FCN-16s, using a pretrained FCN component and a separately initialized GCN component. Adding the GCN as an additional loss improves semantic-segmentation mIOU by 1.34%, while qualitative examples show smoother predictions and reduced classification errors.
- Experimental setup: The model is tested on VOC2012 using raw images of different sizes with batch size 1.The FCN-16s component uses pretrained weights, while the GCN component is randomly initialized.
- Training procedure: The GCN is trained first for 8,000 iterations, after which the whole model is trained end-to-end with Adam optimization.The initial GCN learning rate is 0.1; subsequent total learning rate is 0.00001 with weight decay 0.1.
- Quantitative results: 1.34% mIOU improvement is obtained by adding the GCN component as an additional loss.During testing, the forward structure remains the same as FCN-16s.
- Qualitative results: Graph-FCN produces smoother predictions than FCN-16 and reduces classification errors in qualitative examples.One example shows FCN-16 classifying part of a sheep as part of a dog, while Graph-FCN uses neighboring-node messages for current-node classification.
5 Conclusion
Graph-FCN constructs a graph from FCN-16s features and applies graph-based processing to semantic segmentation. Compared with FCN-16s, it extracts features over a larger receptive field and achieves higher mIOU.
- Conclusion: Graph-FCN models an image graph using FCN-16s and applies graph convolution for semantic segmentation.The approach is presented as a graph-based extension of FCN-16s.
- Conclusion: Graph-FCN extracts features over a larger receptive field than FCN-16s.The figure compares Graph-FCN results, FCN-16s results, and ground-truth segmentations.
- Conclusion: Graph-FCN achieves higher mIOU than FCN-16s under the same forward structure.The reported comparison attributes the improvement to enhanced feature extraction for pixel classification.