Source-linked AI summary
MixSIS: Mixed-supervised instance segmentation
Christian Wilms, Emre Gülsoylu, Sven Schreiber, Simone Frintrop
TL;DR
WSIS methods generally rely on a single annotation level, despite objects differing in the supervision they need. MixSIS permits annotation levels to be mixed within an image, and AnnsMixer generates pseudo masks from weak annotations. The authors report that AnnsMixer outperforms state-of-the-art systems on WSIS and MixSIS, with case studies showing reduced annotation time or improved segmentation quality.
Problem
Most WSIS methods use only one annotation level, although objects can require different amounts of supervision, making flexible annotation choices relevant to annotation effort and results.
Source: §1 · pp. 1–2Method
MixSIS permits mask, box, point, and image-level annotations to be mixed within an image, while AnnsMixer generates pseudo masks from box-, point-, or image-level annotations.
Source: §1 · pp. 2–3Results
AnnsMixer largely outperforms state-of-the-art systems on WSIS and MixSIS; case studies report reduced annotation time (-90%) or improved segmentation quality (+34%).
Source: §7 · p. 14Takeaways & Limitations
MixSIS and AnnsMixer support practical instance-segmentation applications by reducing annotation effort through flexible supervision choices.
Source: §7 · p. 14
Abstract
from arXiv · showhide
Weakly-supervised instance segmentation (WSIS) based on either box-, point-, or image-level annotations has been proposed to overcome the labor-intensive annotation process in fully-supervised instance segmentation. Despite decent results, most WSIS methods only focus on a single annotation level, ignoring that some objects are easy to spot from point annotations, while others require box or even mask annotations. Therefore, we propose the Mixed-Supervised Instance Segmentation (MixSIS) task, which allows a flexible mix of annotation levels (image, point, box, and mask) within a dataset. MixSIS has a high practical relevance, as it allows optimizing the annotation process by annotating challenging objects in more detail, while saving time on simpler instances with less detailed annotations. To address MixSIS, we propose AnnotationsMixer (AnnsMixer) as a baseline. AnnsMixer is a truly mixed-supervised instance segmentation system, which generates pseudo mask annotations from all annotation levels, even mixed within a single image. We evaluate AnnsMixer on both WSIS and MixSIS tasks, largely outperforming state-of-the-art methods. In three case studies, we also demonstrate how the unique features of MixSIS and AnnsMixer can substantially reduce the annotation time (-90%) or improve the results (+34%) in instance segmentation with a simple annotation rule or leveraging heterogeneous data. Our code is available at https://github.com/chwilms/MixSIS
1 Introduction
Single-level weak supervision cannot tailor annotation effort to diverse objects, while prior mixed approaches do not freely combine all annotation levels within an image. MixSIS and AnnsMixer address this gap, with reported gains in annotation efficiency and segmentation results.
1 Introduction
- Choosing one supervision level is difficult because objects differ in how much annotation detail they need, risking suboptimal results or excessive effort.The paper gives the example of a bench that needs a box while a person can be captured with a point; using a point instead of a box reduces annotation time by 65% [13].Source: §1 · pp. 1–2
- Earlier mixed-supervision approaches combined only selected levels or limited mixing to one level per image, leaving per-instance annotation choice unavailable.The proposed task permits all common annotation levels to be combined within an image.Source: §1 · p. 2
- MixSIS allows mask, box, point, and image-level annotations to be freely combined, including within one image.AnnsMixer generates pseudo masks from box-, point-, or image-level annotations to train a fully supervised instance-segmentation system.Source: §1 · pp. 2–3
- Up to 90% less annotation time is reported for an enhanced annotation scheme using MixSIS and AnnsMixer.The paper describes a simple annotation rule and reports only minor performance degradation for the scheme.Source: §1 · p. 3
- Up to 34% improvement in instance-segmentation results is reported from applying AnnsMixer in a MixSIS framework.The introduction also reports up to 90% less annotation time from an enhanced annotation scheme.Source: §1 · p. 3
2 Related Work
Prior WSIS methods generally use one weak annotation level and often involve complex training, whereas AnnsMixer offers a unified approach across annotation levels. Earlier mixed-supervision work remained restricted in which levels could be mixed or whether mixing occurred within an image.
2.1 Weakly-supervised Instance Segmentation
- WSIS approaches typically use one weak annotation level and complex training schemes, while AnnsMixer handles weak and strong annotations jointly with a unified approach.AnnsMixer is described as compatible with arbitrary instance-segmentation systems.Source: §2.1 · p. 3
2.2 Mixed-supervised Instance Segmentation
- Earlier mixed-supervision methods combined only selected levels or restricted mixing to a single level per image, limiting per-instance annotation choice.One approach used mask and image-level annotations, another combined point and mask annotations [13], and did not support mask annotations or within-image mixing.Source: §2.2 · pp. 3–4
3 Mixed-supervised Instance Segmentation
MixSIS allows each instance to use mask, box, point, or image-level annotation, mixed freely within and across images. Its evaluation uses specified random mixes and also motivates object-tailored annotation schemes.
3 Mixed-supervised Instance Segmentation
- MixSIS assigns each instance a mask, box, point, or image-level annotation and allows these levels to mix within an image and across dataset categories.This flexibility supports annotation-time management and combining heterogeneous annotations, while requiring systems to handle all levels.Source: §3 · p. 4
- The evaluation derives annotation levels from fully supervised datasets and tests three random mixes: 25/25/25/25, 10/20/30/40, and 5/10/25/60.The percentages denote mask, box, point, and image-level annotations; testing uses mask annotations.Source: §3 · p. 4
- Object-tailored mixes can reduce annotation time while maintaining strong detection results, with compact objects potentially needing only points and less compact objects needing more supervision.The paper identifies this tailoring as an advantage over WSIS.Source: §3 · p. 4
4 AnnsMixer for MixSIS
AnnsMixer converts weak box, point, and image annotations into pseudo masks, combines them with available masks, and trains a standard instance-segmentation model. Its modular pipeline turns heterogeneous supervision into masks for training, while inference uses only the trained model.
4 AnnsMixer for MixSIS
- AnnsMixer applies SAM with geometric prompts from weak annotations to produce pseudo masks, combines them with original masks, and trains a fully supervised instance-segmentation system.The approach is designed to work with systems such as Mask R-CNN.Source: §4 · p. 4
4.1 Generating Pseudo Mask Annotations
- For box and point supervision, the annotations themselves prompt SAM; image-level labels become point prompts at two local maxima from class activation maps.The classifier detects class presence without explicit localization.Source: §4.1 · p. 5
- SAM produces three class-agnostic segments per prompt, retaining the annotated class label and accounting for prompt ambiguity.Segments with low stability or estimated IoU are filtered, and near-duplicates are removed after combining pseudo masks with original masks.Source: §4.1 · pp. 5–6
4.2 Instance Segmentation Process
- The resulting mask annotations train a vanilla Mask R-CNN, which is the only model applied at inference.The pipeline is modular: other fully supervised instance-segmentation systems could be integrated.Source: §4.2 · p. 6
5 Evaluation
On COCO, the evaluation tests AnnsMixer on mixed annotation-level and traditional weak-supervision settings, as well as the quality of its generated masks. AnnsMixer leads the compared methods in MixSIS mAP, remains competitive in WSIS, and generates pseudo masks whose correspondence with original masks varies by supervision level.
5.1 Mixed-supervised Instance Segmentation
- AnnsMixer outperforms the compared methods in mAP on COCO MixSIS, while mask-only Mask R-CNN uses just 5%–25% of annotations and loses up to 0.098 mAP.Box-based systems use up to 50% of annotations and improve over Mask R-CNN, but remain behind AnnsMixer; SAPNet is the closest flexible competitor and also trails it.Source: §5.1 · pp. 6–7, Table 1 · p. 7
- Adding 75% weak annotations to 25% mask annotations improves over the mask-only baseline for box, point, and image supervision; boxes perform best, followed by points.Mix 1, with 25% annotations at each level, comes close to using 25% masks plus 75% boxes.Source: §5.1 · p. 7, Table 2 · p. 7
5.2 Weakly-supervised Instance Segmentation
- On COCO WSIS, AnnsMixer ranks second for point- and box-supervised cases and outperforms dedicated methods for image-level supervision.It also performs better than WISH and WeakSAM when given the same backbone, despite lacking an architecture tailored to a single annotation level.Source: §5.2 · p. 8, Table 3 · p. 8
5.3 Quality of Generated Pseudo Mask Annotations
- For box-based pseudo masks, 84.1% of original masks are matched, 95.7% of generated masks cover an original mask, and IoU is 70.5.The reported comparisons are between generated and original COCO training masks.Source: §5.3 · p. 8, Table 4 · p. 9
- Point- and image-level supervision can miss objects or produce part masks, while box-level examples show especially high-quality generated annotations.The examples attribute a missed bench to its centroid falling within a person; image-level face masks reflect strong class activation, while visually similar hands and faces merge under point supervision.Source: §5.3 · p. 9, Figure 4 · p. 9
6 Case Studies
The case studies test object-property-based supervision choices and heterogeneous annotation or dataset merging to trade annotation effort against segmentation quality.
6.1.1 Object Properties
- The tested object indicators cover compactness, convexity, other shape properties, and interactions between objects or an object and its image.The study derives these properties from COCO train masks; the authors say human annotators can judge them directly in real applications.Source: §6.1.1 · p. 10
6.1.2 Study on Annotation Schemes
- Mixefficient assigns points to objects with smooth contours and boxes to others, taking 114.7 seconds per COCO image versus 650.8 seconds for full supervision.The rule uses good continuation as its object property and is intended to be simple for annotators to spot.Source: §6.1.2 · p. 11
6.1.3 Results
- Annotation time falls by up to 90% with Mixefficient across four datasets, while segmentation performance declines by 5% on average; UVO mAP improves.The comparison is between fully supervised Mask R-CNN and AnnsMixer trained with Mixefficient; Table 5 reports the four-dataset results.Source: §6.1.3 · p. 11, Table 5 · p. 12
- On COCO val, Mixefficient is the best trade-off among tested schemes, and Mixrandom’s mAP is up to 11% lower.Figure 6 also shows a Pareto front for scenarios prioritizing shorter annotation time.Source: §6.1.3 · p. 12, Figure 6 · p. 13
6.2 Annotation Merging with MixSIS
- On PASCAL VOC, AnnsMixer with boxes and masks gains 0.097 mAP (+34%) over mask-only Mask R-CNN; on BDD100k, it gains 0.037 mAP.The BDD100k comparison is between AnnsMixer using boxes and masks and vanilla Mask R-CNN.Source: §6.2 · pp. 12–13, Table 6 · p. 13, Table 7 · p. 13
6.3 Dataset Merging with MixSIS
- With MAD box annotations added to MinneApple mask annotations, AnnsMixer improves mAP by about 2% despite a substantial domain gap and no domain adaptation.The evaluation is on the MinneApple test set.Source: Table 8 · p. 14, §6.3 · p. 14
- Combining PASCAL VOC masks with COCO image-level annotations improves mAP by 11% over using masks alone.The authors describe this particular general-segmentation case as artificial, while noting image-level labels are relevant to MixSIS.Source: Table 9 · p. 14, §6.3 · p. 14
7 Conclusion
MixSIS allows different annotation levels within one image, and AnnsMixer produces pseudo masks from those levels in one system. The authors report strong WSIS and MixSIS results and case studies with reduced annotation time or improved segmentation quality.
7 Conclusion
- MixSIS combines annotation levels within a single image, extending the traditional WSIS formulation.AnnsMixer is proposed as a baseline that produces pseudo annotation masks for all annotation levels in one system.Source: §7 · p. 14
- AnnsMixer largely outperforms state-of-the-art systems on WSIS and MixSIS.Source: §7 · p. 14
- Three case studies show that leveraging AnnsMixer’s unique features can reduce annotation time by 90% or improve segmentation quality by 34% in practical instance segmentation applications.Source: §7 · p. 14