End-to-End Instance Segmentation with Recurrent Attention

• Instance segmentation assigns pixel-level labels to individual objects, unlike semantic segmentation, and supports recognition, robotic grasping, autonomous driving, and image understanding.Source: §1 · p. 1
• Existing fully convolutional and graphical-model approaches struggle with the output dimensionality, complex pipelines, and lack of end-to-end training.Source: §1 · p. 1
• Occlusion makes local bottom-up grouping and non-maximal suppression difficult, especially when heavily occluded objects overlap foreground detections.Source: §1 · p. 1
• The proposed recurrent neural network uses visual attention and a temporal chain to count jointly with instance segmentation, emitting one instance at a time.Source: §1 · p. 1
• Dynamic NMS uses already segmented objects to aid later discovery of occluded objects, while joint counting and segmentation provides a stopping criterion.Source: §1 · pp. 1–2
• Iterative attentive counting is motivated by human-like processing and is expected to suit cluttered scenes better than global-image regression.Source: §1 · p. 2

2. Recurrent attention model

• The model combines external memory, box proposal, segmentation, and scoring networks to sequentially produce instance segmentations and decide when to stop.The external memory tracks prior segmentations; the box network localizes objects, the segmentation network predicts pixels, and the scoring network estimates object presence and termination.Source: §2 · p. 2, §2.1 · p. 2, §2.2 · p. 2

2.1. Part A: External memory

• The external-memory canvas stores accumulated predicted pixels alongside the input image, providing prior boundary information for locating later, potentially occluded objects.The canvas contains 10 channels: one accumulates previous segmentation outputs and nine store the input representation.Source: §2.1 · p. 2

2.2. Part B: Box network

• Soft attention pools spatial features for the box network, whose recurrent glimpses predict normalized box centers, sizes, and a scaling factor for patch extraction.Gaussian interpolation extracts rectangular patches from the image and prior-state representation for local processing.Source: §2.2 · pp. 3–4

2.3. Part C: Segmentation network

• The segmentation network upsamples local features into a patch-level heat map, re-projects it to the original image, and constrains values with learned box amplification and outside-box suppression.A sigmoid converts the re-projected output into segmentation values between 0 and 1.Source: §2.3 · p. 4

2.4. Part D: Scoring network

• The scoring network uses box and segmentation hidden states to produce scores, with inference stopping when the score falls below 0.5.Training uses a sequence length equal to the maximum object count plus one and encourages monotonically decreasing scores.Source: §2.4 · p. 4, §2.5 · p. 4

2.5. Loss functions

• The joint objective sums matching segmentation IoU, soft box IoU, and score losses, while training progressively replaces ground-truth inputs with model predictions.Matching handles unordered instances and penalizes false positives and negatives; soft box IoU avoids vanishing gradients for non-overlapping boxes, and scheduled sampling reaches test-time self-conditioning.Source: §2.5 · p. 4, §2.6 · p. 5

Detector-based approaches

• Detector-based methods combine detectors or region proposals with segmentation, while this work jointly learns detection and segmentation with feedback and sequence ordering.The cited approaches include DPM, graphical models, CNN-based detectors, and iterative refinement.Source: §3 · p. 5

Graphical model approaches

• Graphical-model approaches impose structured dependencies among pixels and instances but can have long runtimes and depend on connected instances or clear depth ordering.Examples include restricted Boltzmann machines, segmentation trees, and dense CRFs.Source: §3 · p. 5

Fully convolutional approaches

• Fully convolutional approaches predict masks, proposals, or instance-sensitive representations, but commonly require clustering, template matching, instance fusion, or other post-processing.The related systems include foreground and angle-map prediction as well as embedding-based methods.Source: §3 · p. 5

RNN approaches

• Recurrent approaches decompose prediction sequentially, but prior models face iteration costs or difficulty inhibiting distant instances; this work uses soft attention, direct feedback, and local windows.The related recurrent methods include permutation-agnostic matching losses and ConvLSTM-based segmentation.Source: §3 · p. 5

4.1. Datasets & Evaluation

• Cityscapes evaluation covers eight classes with a class-agnostic instance model, followed by a semantic mask, reporting both all-class and car-specific average precision.The dataset contains 2,975 training, 500 validation, and 1,525 test images.Source: §4.1 · p. 6, Table 3 · p. 7
• The experiments use SBD for CVPPP, coverage metrics for KITTI, average precision across IoU thresholds for Cityscapes, and count errors including |DiC|, AvgFP, and AvgFN.KITTI coverage measures instance-wise IoU, while Cityscapes also reports AP50% and distance-restricted scores.Source: §4.1 · pp. 6–7

4.2. Results & Discussion

• Qualitative outputs include CVPPP leaves, MS-COCO zebras, and Cityscapes/KITTI scenes, with instance colors determined by output sequence order.The displayed segmentations include heavily occluded and disconnected object parts.Source: Figure 4 · p. 7, Figure 5 · p. 7, Figure 6 · p. 8
• Large-margin improvements over prior methods are reported for CVPPP leaf segmentation and counting, although the FCN pre-processor overfit the small dataset.The reported comparisons appear in Table 1; the simpler model without input pre-processing was used.Source: Table 1 · p. 6, §4.2 · p. 7
• On KITTI, the model outperforms [46] but trails [41], whose depth input and bottom-up instance fusion may improve distant-car segmentation.The model segments cars across varied poses but does not reliably detect distant cars.Source: Table 2 · p. 6, §4.2 · p. 7
• Failure cases include omitted distant objects, under-segmentation from roughly 4× downsampling, and limited higher-order reasoning.The authors suggest combining the method with bottom-up merging and higher-order graphical models.Source: §4.2 · p. 8
• Ablations identify foreground segmentation, the box network, scheduled sampling, and additional LSTM glimpses as important contributors to performance.More glimpses allow more information to be considered before the next region of interest is output, while KITTI validation and test results are highly variable.Source: Table 4 · p. 7, §4.2 · p. 8

• The recurrent attentive architecture addresses object occlusion with external memory and permits segmentation at a fine resolution.Source: §5 · p. 8
• The model significantly reduces parameters while achieving strong performance with only 100 leaf images and under 3,000 road-scene images.Source: §5 · p. 8
• The end-to-end model does not depend on prior object knowledge such as size or connectedness.Source: §5 · p. 8
• Future work extends the model toward highly multi-class instance segmentation, including MS-COCO, and more structured scene understanding.Source: §5 · p. 9

• Adam optimization uses learning rate 0.001 and batch size 8.Source: A. Training procedure specification · p. 10
• The learning rate is multiplied by 0.85 every 5000 training steps.Source: A. Training procedure specification · p. 10

• θ_t is the probability of feeding the ground-truth segmentation with greatest overlap with the previous prediction instead of the model output.Source: §A.1 · p. 10
• θ_t decays exponentially as training proceeds, and larger t delays the decay.The reported constants are S=10000, S2=2885, and K=3.Source: §A.1 · p. 10

• The model outperforms an object detector followed by non-maximal suppression and an associative-subitizing method [7] on counting MS-COCO zebra images.Source: B. More experimental results · p. 10
• Table 5 reports the model's MS-COCO zebra segmentation and counting performance.Source: Table 5 · p. 10, B. More experimental results · p. 10

C.1. Foreground + Orientation FCN

• Input images are resized to 224 × 224 for CVPPP and MS-COCO, 128 × 448 for KITTI, and 256 × 512 for Cityscapes.The Cityscapes resolution applies 4x downsampling.Source: §C.1 · p. 10

C.3. Box network

• The box network processes nine FCN-output channels through another CNN and applies LSTM-predicted attention for dynamic pooling in its final layer.A glimpse MLP converts the LSTM hidden state into normalized weights over spatial feature-map grids.Source: §C.3 · p. 10

C.4. Segmentation network

• Each segmentation input is a 48×48 patch containing RGB channels, eight orientation-angle channels, and one FCN-predicted foreground heat-map channel.The input therefore combines original image data with FCN-derived features.Source: §C.4 · p. 10
• The segmentation network uses a constant β value of 5.Source: §C.4 · p. 10
• The segmentation network specification is provided in Table 10, while the referenced implementation tables cover the FCN, external memory, box CNN, and box LSTM.Source: Table 6 · p. 11, Table 7 · p. 11, Table 8 · p. 11