Source-linked AI summary
Recurrent Instance Segmentation
Bernardino Romera-Paredes, Philip H. S. Torr
TL;DR
Instance segmentation must delineate multiple distinct objects, but existing systems often train pipeline modules independently. The paper introduces an end-to-end recurrent model that segments instances sequentially using spatial memory and a principled loss. The authors report superior performance on multiple-person segmentation and plant-leaf counting.
Problem
Existing instance-segmentation approaches commonly train pipeline modules independently, leaving the whole process without joint end-to-end learning.
Method
The paper uses a recurrent neural network with spatially preserving convolutional units to segment same-class instances sequentially and train the process end-to-end.
Results
The approach outperforms state-of-the-art methods on multiple-person segmentation and leaf counting, including the Plant Phenotyping dataset.
Takeaways & Limitations
A single jointly learned recurrent pipeline can integrate instance segmentation functions while tracking visited image areas and handling occlusion.
Takeaways & Limitations
The paper evaluates class-specific segmentation and leaves alternative architectural analyses and broader extensions for future work.
Abstract
from arXiv · showhide
Instance segmentation is the problem of detecting and delineating each distinct object of interest appearing in an image. Current instance segmentation approaches consist of ensembles of modules that are trained independently of each other, thus missing opportunities for joint learning. Here we propose a new instance segmentation paradigm consisting in an end-to-end method that learns how to segment instances sequentially. The model is based on a recurrent neural network that sequentially finds objects and their segmentations one at a time. This net is provided with a spatial memory that keeps track of what pixels have been explained and allows occlusion handling. In order to train the model we designed a principled loss function that accurately represents the properties of the instance segmentation problem. In the experiments carried out, we found that our method outperforms recent approaches on multiple person segmentation, and all state of the art approaches on the Plant Phenotyping dataset for leaf counting.
1 Introduction
Instance segmentation is challenging because object instances are unordered and must be clustered, while existing pipelines train modules independently. The paper proposes an end-to-end recurrent approach that segments same-class instances sequentially.
- Instance segmentation delineates distinct objects in an image and supports applications including robotic grasping, visual enhancement, and counting.
- Unknown instance count and permutation-invariant evaluation make instance segmentation more difficult than semantic segmentation.Semantic segmentation assigns pixels to predefined classes, whereas instance segmentation must group pixels without a fixed number or ordering of instances.
- Existing approaches commonly use independently trained proposal, recognition, and segmentation modules, requiring separate losses and intermediate representations.
- The proposed model learns class-specific instance segmentation end-to-end by segmenting instances sequentially with recurrent neural networks and internal spatial memory.
- The paper contributes an RNN-based model with convolutional layers and a principled loss function, evaluated on multiple-person segmentation and plant-leaf segmentation and counting.
2 Background
Prior instance-segmentation systems combine detection and semantic-segmentation components, often training modules independently and facing the unordered nature of instances. RIS instead learns to choose an instance order and segment instances sequentially, drawing on recurrent and attention-like processing.
- 2.1 Instance Segmentation Models: Instance segmentation combines object detection's instance separation with semantic segmentation's pixel-wise shape delineation.
- 2.1 Instance Segmentation Models: R-CNN-based methods use region proposals, CNN features, and refinement modules, but their components are trained independently.
- 2.1 Instance Segmentation Models: The lack of a natural instance order complicates sequential prediction, while some prior methods use depth information and Markov Random Fields to model occlusion.
- 2.1 Instance Segmentation Models: RIS lets the model decide each image's instance order and learns to segment instances sequentially.
- 2.2 Recurrent Neural Networks: RNNs provide persistent state and support variable-length output sequences, enabling recurrent processing of information from static images.
- 2.3 Attention Based Models: RIS resembles attention models by selecting different parts of one input over successive iterations, but instance attention is its end target rather than an intermediate means.
3 Segmenting One Instance at a Time
The model processes an image sequentially, using ConvLSTM state and spatial inhibition to produce one instance mask and confidence score per step. A permutation-invariant matching loss supports end-to-end training despite variable numbers of instances.
- Sequential Inference: The recurrent state retains previously segmented regions, enabling sequential handling of occlusion and relationships between instances.The model can segment nonoccluded instances first and use its state to detect occluded objects afterward.
- Architecture: A fully convolutional network feeds an RNN that transforms recurrent state into each instance’s mask and confidence score.The confidence estimates whether the current candidate is an object and provides the stopping condition.
- Convolutional LSTM: ConvLSTM preserves spatial information by replacing LSTM gate linear layers with convolutions over image-like feature maps.Its recurrent memory is represented as a spatial map, with d channels per pixel and convolutional filter weights.
- Spatial Inhibition: The spatial inhibition module maps the hidden state to a single-channel pixel representation, normalizes competition across pixels, and learns a threshold for selecting one instance.Its diagram is identified as Fig. 2; the module is designed to filter out pixels belonging to other instances.
- Sequential Inference: At inference, the model stops when confidence falls below 0.5, while training permits a predicted sequence length of n_i + 2 for an image with n_i annotated instances.The predicted sequence may still overestimate or underestimate the number of ground-truth instances.
- Loss Function: The loss matches predicted and ground-truth masks through a relaxed IoU correspondence and uses the Hungarian algorithm to select the minimizing matching for gradient computation.The pointwise minimum of continuous matching-specific functions makes end-to-end learning possible; extra predicted masks are ignored beyond the shorter sequence length.
4 Experiments
The experiments evaluate recurrent instance segmentation on multiple-person images and plant leaves, including both segmentation and counting. The recurrent model is trained and tested with implementation choices such as ConvLSTM recurrence, spatial inhibition, curriculum learning, and CRF refinement.
- The approach is evaluated on multi-instance person segmentation and plant-leaf segmentation and counting.
- The recurrent stage uses two ConvLSTM layers followed by spatial inhibition, which outputs a confidence score and an instance mask.
- Training uses backpropagation through time with Adam, gradient clipping, and a learning-rate schedule that reduces the rate when training error plateaus.
- Curriculum learning gradually increases the number of recurrent iterations and objects the network must segment.
- Multiple Person Segmentation: On Pascal VOC 2012, RIS is comparable to state-of-the-art methods, while RIS+CRF improves results and outperforms competing approaches under AP r(0.5) and AP rAve.The comparison uses AP r at 0.5 IoU and the average AP r over IoU values from 0.1 to 0.9.
- Plants Leaf Segmentation and Counting: On CVPPP, the method significantly outperforms competing ad hoc methods for leaf counting but achieves only comparable leaf-segmentation results.The authors attribute the segmentation limitation partly to too few original training images for learning varied leaf shapes from scratch.
5 Discussion
The paper presents sequential, end-to-end instance segmentation with recurrent memory, and reports state-of-the-art performance across multiple person segmentation and leaf-counting experiments. It also identifies alternative architectures and extensions for future work.
- The model segments one instance at a time in a recurrent sequence, integrating required functions into a jointly learned end-to-end pipeline.Its recurrent structure tracks visited image areas and handles occlusion among instances.
- Experiments on multiple person segmentation and leaf counting show that the approach outperforms state-of-the-art methods.
- The recurrent state contains information about instances already visited in the sequence, according to qualitative results.
- Alternative architectural choices, including prediction-mask inputs and log-likelihood loss, did not improve results over the described model.
- Suggested extensions include instance classification, integrating a CRF layer, exploring other recurrent structures, and modeling object co-occurrence, parts, and attributes.
Appendix
The appendix extends the paper with visualizations of model representations and recurrent states, a ConvLSTM diagram, additional person-segmentation examples, and a more detailed loss explanation.
- The appendix adds four materials: feature and RNN-state visualizations, a ConvLSTM update diagram, multiple-person examples including failures, and an expanded loss-function explanation.
A Features and States through the Pipeline
The appendix visualizes how the model transforms an input image through FCN features, recurrent states, spatial inhibition representations, masks, and confidence scores.
- The visualization uses a validation Pascal VOC image containing four subjects as the model input.
- FCN features in 100 output channels distinguish different instances and show sensitivity to body parts, especially faces.
- The ConvLSTM state and spatial-inhibition representations are shown while the model sequentially produces segmentations, predictions, and confidence scores.
- The first spatial-inhibition representation can isolate one object but may retain lower-intensity elements, whereas the next representation filters out non-object pixels.
- The final sigmoid masks saturate toward 1 for object pixels and toward 0 for non-object pixels.
- Figure 5 pairs the input image on the left with the model prediction on the right.
- Figure 6 displays features produced by the FCN stage from the Figure 5 input.
B Diagram of ConvLSTM
The appendix provides a diagram of the recurrent update equations used by the model's ConvLSTM.
- The diagram illustrates the recurrent update equations of the ConvLSTM.
C Qualitative Results on Multiple Person Segmentation
Qualitative evaluations show that the model can segment multiple people, including scenes with three subjects and some scenes with four or more, but performance degrades as instance count increases. Observed failures include difficult extremities, joined instances, missed instances, and hallucinated instances.
- Qualitative predictions: The model produces qualitative predictions for multiple-person images containing three subjects and images containing four or more subjects.The three-subject examples include outputs with and without CRF post-processing.
- Observed errors: For three-subject images, the main visible errors arise from difficulty segmenting extremities.The examples identify extremities as a recurring source of segmentation error.
- Qualitative predictions: Scenes with four or more subjects can still yield good inferences, including when subjects appear far apart.The model particularly struggles when segmenting more than four instances.
- Observed errors: The reported failure cases are joining two instances, missing an instance entirely, and hallucinating instances where none exist.These three classes are presented as covering almost the entire observed range of failures.
D Implementation of the Loss Function
The loss implementation matches predicted masks to ground-truth masks using IoU and the Hungarian algorithm, then combines mask and score costs while computing gradients. The algorithm processes matched predictions separately from extra predictions.
- Algorithm structure: Algorithm 1 implements forward and backward propagation for the loss function in equation (5).It outputs the cost and gradients with respect to predicted masks and scores.
- Mask matching: The implementation fills a matrix with fIoU(Yi, ˆYj) values and obtains the best matching with the Hungarian function.The matrix compares each ground-truth mask with each predicted mask.
- Cost computation: For matched predictions, the cost combines the IoU term with λfBCE(1, st).The score target is one for predictions assigned to ground-truth instances.
- Cost computation: Predictions beyond the ground-truth count receive a binary-cross-entropy score term with target zero.The extra-prediction loop handles t = n + 1 through ˆn.
- Gradient computation: The derivatives f′IoU and f′BCE provide gradients with respect to the second argument.These derivative functions are used for backpropagation through the loss terms.