Source-linked AI summary
Associative Embedding: End-to-End Learning for Joint Detection and Grouping
Alejandro Newell, Zhiao Huang, Jia Deng
TL;DR
Computer vision systems often detect visual units and group them separately, despite the two operations being tightly coupled. The paper introduces associative embedding, which jointly predicts detections and group tags in a single-stage network. It demonstrates feasibility for pose and instance segmentation, achieving state-of-the-art pose performance while identifying detection as the main pose bottleneck.
Problem
Many vision tasks require jointly detecting visual units and grouping them, but are often handled by separate stages despite their tight coupling.
Method
Associative embedding gives each detection a real-valued tag and trains tags to be similar within groups and dissimilar across groups, alongside detection predictions.
Results
The method demonstrates feasibility for multi-person pose and instance segmentation, achieving state-of-the-art performance for pose on MS-COCO and MPII.
Takeaways & Limitations
Associative embedding is general enough for pixel-wise prediction architectures and can be applied across detection-and-grouping tasks.
Takeaways & Limitations
The experiments assume one-dimensional embeddings, which were sufficient for multi-person pose but did not show significant improvement from higher dimensions.
Abstract
from arXiv · showhide
We introduce associative embedding, a novel method for supervising convolutional neural networks for the task of detection and grouping. A number of computer vision problems can be framed in this manner including multi-person pose estimation, instance segmentation, and multi-object tracking. Usually the grouping of detections is achieved with multi-stage pipelines, instead we propose an approach that teaches a network to simultaneously output detections and group assignments. This technique can be easily integrated into any state-of-the-art network architecture that produces pixel-wise predictions. We show how to apply this method to both multi-person pose estimation and instance segmentation and report state-of-the-art performance for multi-person pose on the MPII and MS-COCO datasets.
1. Introduction
Many vision tasks require detecting visual units and grouping them into larger structures, but detection and grouping are often handled in separate stages. Associative embedding instead jointly produces detections and group assignments in a single end-to-end network.
- Joint detection and grouping covers pose estimation, instance segmentation, and multi-object tracking by assigning detected units to variable numbers of groups.
- Two-stage pipelines may be suboptimal because detections and their grouping are tightly coupled, such as wrists depending on nearby elbows.
- Associative embedding assigns each detection a real-valued tag, with similar tags indicating detections that belong to the same group.
- The network predicts per-pixel detection scores and identity tags, which are decoded into detections and groups.
- A tag loss encourages within-group similarity and between-group dissimilarity without requiring fixed ground-truth tag values.
- Applied with a stacked hourglass network, the method groups body joints with similar tags and achieves state-of-the-art results on MS-COCO and MPII Multiperson Pose.
- The method is presented as a simple, generic, single-stage approach that works with pixel-wise prediction architectures and is also applied to instance segmentation.
2. Related Work
Prior work uses embeddings, dense pixel-wise representations, and multi-stage detection pipelines, while associative embedding integrates detection and grouping directly in one network. Compared with related instance-segmentation methods, it uses a substantially lower-dimensional output.
- Prior embedding methods connect images, visual features, text, or linguistic units, whereas this work uses embeddings for detection grouping.
- Spectral-clustering approaches derive embeddings from precomputed affinities, while associative embedding does not use an intermediate affinity representation.
- Dense convolutional embeddings produce pixel-wise embeddings for semantic segmentation, whereas associative embedding also predicts pixel-wise detection scores.
- Multiperson pose methods include top-down systems that detect people before estimating poses and bottom-up systems that detect body joints before grouping them.
- Existing instance-segmentation methods commonly use multi-stage detection followed by segmentation, although some newer approaches integrate these operations with fully convolutional networks.
- For each object category, associative embedding outputs only a foreground-background score and an instance-identity tag at each pixel.
3. Approach
Associative embedding extends pixel-wise prediction with tags that jointly support detection and grouping. For multiperson pose, the network predicts joint detections and tags, then decodes individual poses by grouping detections with similar tags.
- Associative Embedding: Associative embedding predicts an embedding tag alongside each candidate’s detection score, using tag similarity to encode group membership.The absolute tag values are unconstrained; only distances between tags matter.
- Training: The method trains detection and grouping jointly with losses that separate tags across groups and align tags within each group.For pose, tags are compared at ground-truth joint locations, using person-level reference embeddings rather than every possible keypoint pair.
- Multiperson Pose Estimation: For multiperson pose, a stacked hourglass network produces a detection heatmap and a tagging heatmap for each body joint.The model is adapted from a dense pixel-wise prediction architecture that combines global context with fine localization.
- Multiperson Pose Estimation: Pose decoding applies non-maximum suppression to joint heatmaps, retrieves the corresponding tags, and assigns detections to people according to tag similarity.Joints are processed from head and torso toward the limbs until every detection is assigned, without explicit anatomical-consistency checks.
- Instance Segmentation: Associative embedding also supports instance segmentation by detecting foreground pixels, identifying instance tags from their distribution, and assigning pixels to the closest identifier.The instance-segmentation formulation treats object pixels as visual units that must be detected and grouped into instances.
4. Experiments
The experiments evaluate associative embedding on multi-person pose estimation and instance segmentation, showing strong pose results and feasibility as a general detection-and-grouping technique.
- Multiperson Pose Estimation: MPII results improve over state-of-the-art methods in overall AP for multi-person pose estimation.The method groups keypoint detections into individual people without assuming the number of people present.
- Multiperson Pose Estimation: The system provides qualitative embedding visualizations for heavily overlapping people, where occlusion and close joint spacing make pose parsing difficult.Figure 7 highlights changes in embedding predictions across joints when two target figures overlap significantly.
- Multiperson Pose Estimation: MS-COCO results achieve state-of-the-art performance on both test-std and test-dev.Multi-scale evaluation alone already produces competitive performance against prior state-of-the-art methods.
- Multiperson Pose Estimation: Replacing predicted detections with ground-truth detections raises AP from 59.2 to 94.0, identifying keypoint detection as the system’s main bottleneck.The predicted tags still provide high-quality grouping in this evaluation.
- Instance Segmentation: On PASCAL VOC 2012, associative embedding produces reasonable instance segmentation predictions as a proof of concept.The experiment uses mAP evaluation on the validation set and applies the same supervision used for multi-person pose.
5. Conclusion
The paper introduces associative embeddings for simultaneously generating and grouping detections across vision tasks. It demonstrates feasibility for multi-person pose and instance segmentation, with state-of-the-art pose performance and compatibility with pixel-wise prediction architectures.
- Associative embeddings supervise convolutional networks to simultaneously generate and group detections.The method is applied to multi-person pose and instance segmentation and is presented as general enough for multi-object tracking.