Towards Segmenting Anything That Moves

1. Introduction

• Open-world perception matters because practical robot autonomy must detect novel obstacles regardless of the semantic name assigned to them.The introduction uses self-driving cars encountering unfamiliar obstacles and debris as an example.Source: §1 · p. 1
• 11.4%: the method outperforms Keuper et al. on the proposed FBMS measure and achieves state-of-the-art results on both FBMS measures.The authors note that the standard FBMS measure does not penalize false positives, motivating their proposed evaluation.Source: §1 · p. 2
• On DAVIS-Moving and YTVOS-Moving, the method is competitive with top-down approaches on seen categories and outperforms top-down and bottom-up methods on unseen categories.The benchmarks are built from DAVIS 2017 and YTVOS videos in which all moving objects are labeled.Source: §1 · p. 2
• The paper proposes a learning-based approach to spatio-temporal grouping, addressing the absence of learning-based methods among state-of-the-art approaches to this task.It combines optical-flow motion cues for separating objects with appearance cues that provide a generic objectness prior.Source: §1 · pp. 1–2
• The authors introduce a more informative metric and two larger, more diverse benchmarks to support further research in generic moving-object grouping.The new benchmarks are DAVIS-Moving and YTVOS-Moving.Source: §1 · p. 2

2. Related Work

• Earlier spatio-temporal grouping methods track pixels with optical flow, encode motion descriptors, and cluster pixels by motion similarity.The paper contrasts these bottom-up strategies with its top-down learning algorithm followed by a linking step.Source: §2 · p. 2
• The paper contrasts its approach with [9], which combines moving-object detections and clustered trajectories, and with, which does not handle discontinuous motion.It also reports that its overlap-based tracker is competitive with [45] while producing significantly fewer false-positive segmentations.Source: §2 · p. 2
• Unlike binary video-segmentation methods, which separate moving objects from background, this method segments and tracks each moving object instance.The cited two-stream binary methods do not segment individual instances.Source: §2 · p. 2
• Standard still-image instance segmentation targets a fixed category list, leaving methods blind to objects outside their training categories.COCO is cited as an example of a large standard dataset for this task.Source: §2 · p. 2
• Generic-object approaches either require bounding-box annotations for new categories [15] or produce a binary objectness mask [19], whereas this method provides instance masks for unseen objects.The paper distinguishes its instance-level output from generic objectness segmentation.Source: §2 · p. 2

3. Approach

• The method processes RGB frames and corresponding optical flow through appearance and motion streams, then fuses their features in a joint region proposal network for category-independent moving-object segmentation.The fused features feed the box and mask regression heads.Source: §3 · pp. 2–3, Figure 3 · p. 4

3.1. Motion-based Segmentation

• The motion stream learns moving-instance segmentation from FlyingThings3D synthetic sequences, treating all moving objects as one category and static pixels as background.The training data provides optical flow and segmentations; the model can also group independently moving parts without representing the concept of an object.Source: §3.1 · p. 3, Figure 2 · p. 3

3.2. Appearance-based Segmentation

• The appearance stream is trained as an objectness Mask R-CNN by combining COCO's 80 categories into one object category, rather than training category-specific classifiers.The paper says this training generalizes to objects not labeled in COCO.Source: §3.2 · p. 3, Figure 2 · p. 3

3.3. Two-Stream Model

• A two-stream RPN learns to fuse RGB appearance and optical-flow features to propose object-like regions, unlike prior proposals based only on appearance or separate streams.The resulting features retain Mask R-CNN's box and mask regression stages.Source: §3.3 · pp. 3–4, Figure 3 · p. 4

3.4. Tracking

• An overlap-based tracker matches successive masks by intersection over union and Hungarian matching, initializes tracks from high-scoring detections, and marks unassigned tracks inactive after a set interval.The method also reuses appearance features to predict static objects and merges those predictions with moving-object predictions.Source: §3.4 · p. 4

• The proposed single F-measure evaluates whether methods detect all and only moving objects, while penalizing static or background detections.Source: §4 · p. 4
• Unlike default FBMS matching, which ignores unmatched predictions, the proposed measure counts them as false positives.Source: §4 · p. 4, Figure 4 · p. 5
• Hungarian matching pairs predicted and ground-truth regions to maximize summed F-measure; unmatched predictions map to an empty region.Source: §4 · p. 5
• Unlabeled pixels within predicted regions lower precision and F-measure, penalizing static or unlabeled-object segmentations.Source: §4 · p. 5

5.1. Datasets

• Training data constrain evaluation: DAVIS 2016 provides only 30 training videos, YTVOS may label static as well as moving objects, and FBMS sequences are excluded from training for fair comparison.The motion stream is trained on FlyingThings3D, a synthetic dataset of 2,700 videos, and the joint model is fine-tuned on DAVIS 2016 and the YTVOS-Moving training subset; 100 YTVOS-Moving sequences are held out for evaluation.Source: §5.1 · p. 5

5.2. Implementation Details

• The two-stream model uses a Mask R-CNN with a ResNet-50 backbone, and its tracker runs forward and backward to detect objects before they move.Tracks can remain alive for up to 10 frames; the reported results are fairly insensitive to that parameter.Source: §5.2 · p. 5

5.3.1 Motion stream

• Ground-truth flow followed by FlowNet2 fine-tuning improves detection by 2.7%; FlowNet2 also outperforms LiteFlowNet despite worse standard flow-benchmark performance.The motion stream groups similarly moving pixels and separates objects with disparate motion, including on real images despite synthetic-only segmentation training.Source: §5.3.1 · p. 6, Table 1 · p. 6, Figure 5 · p. 6

5.3.2 Appearance Stream

• The objectness model outperforms the class-specific model by nearly 8% and better generalizes to categories outside COCO.The comparison reports mAP on DAVIS ’16 val; Figure 6 illustrates detections of non-COCO objects.Source: §5.3.2 · p. 6, Table 2 · p. 6, Figure 6 · p. 6

5.3.3 Joint training

• Preserving the streams’ learned knowledge improves joint training; fine-tuning on DAVIS after YTVOS-Moving yields the best model, while YTVOS-Moving alone drops performance 5%.Freezing the appearance stream gives nearly a 3% accuracy improvement, freezing flow adds 2%, and appearance-only mask features add 1% segmentation AP.Source: §5.3.3 · pp. 6–7, Table 3 · p. 7, Table 4 · p. 7

5.4. Comparison to prior work

• On the proposed FBMS metric, the final model beats prior work by 11.3% F-measure on TestSet; under the official metric, Ours-A exceeds prior work by 6.4%.The proposed metric also shows a 6.1% TrainingSet gain for the final model; the official metric does not penalize false positives.Source: §5.4 · p. 7, Table 5 · p. 7, Table 6 · p. 8
• On DAVIS-Moving, the approach reaches 77.9% F-measure versus 42.3% for.The authors suggest faster, more articulated motion and higher-resolution video may explain the larger gap.Source: Table 7 · p. 8, §5.4 · p. 8
• On unseen YTVOS-Moving objects, appearance-only improves over Mask R-CNN from 53.6% to 69.0%, while the final model adds modest gains on novel objects.The final model performs comparably overall to the appearance stream alone and is evaluated without YTVOS training.Source: Table 8 · p. 8, §5.4 · p. 8

6. Conclusion

• The paper proposes a learning-based method for spatio-temporal grouping.It reports state-of-the-art results on FBMS and introduces a new metric and the DAVIS-Moving and YTVOS-Moving benchmarks.Source: §6 · p. 8
• The authors conclude that learning-based approaches can generalize to never-before-seen objects.This is presented as one of the method’s two key insights.Source: §6 · p. 8
• The authors conclude that synthetic data can train a generic grouping method with few priors on real-world objects.They also introduce a new metric and two benchmarks to support further research.Source: §6 · p. 8

7.1. DAVIS’16 Motion Segmentation

• On DAVIS 2016, the method improves boundary F-measure by 1% and is outperformed on J by a concurrent work by 0.7%.Instance outputs are converted to binary masks using a score threshold greater than 0.7; this is not the paper’s target task.Source: §7.1 · p. 11, Table 9 · p. 11

7.2. FBMS Moving Only

• Nearly 65% improvement in ∆Obj (from 4 to 1.4) versus [45], while the method performs competitively with [45] on F-measure.For the moving-only comparison, tracking is applied directly to the two-stream model’s per-frame moving-object detections, disabling tracking after objects stop moving.Source: §7.2 · p. 11, Table 10 · p. 11

7.3. Failure cases

• Failures include track identity swaps after complete occlusion and static objects near the camera classified as moving.The authors hypothesize that a parked car near a moving van may be mistaken for motion because proximity is used as a cue.Source: §7.3 · p. 11, Figure 8 · p. 11

7.4. Implementation details

• Optical flow is encoded as angle and magnitude channels plus an empty channel; the appearance stream trains on COCO, while motion and joint streams use staged training.The motion stream trains on FlyingThings3D with ground-truth flow and then FlowNet2 flow; the joint model trains on YTVOS and DAVIS.Source: §7.4 · pp. 11–12