Source-linked AI summary
Single-Shot Object Detection with Enriched Semantics
Zhishuai Zhang, Siyuan Qiao, Cihang Xie, Wei Shen, Bo Wang, Alan L. Yuille
TL;DR
Single-shot detectors can lack semantic information in low-level features, which may impair small-object detection and higher-level feature quality. DES enriches these features with weakly supervised segmentation and enriches higher-level features with self-supervised global activation, achieving reported accuracy and speed across VOC and COCO.
Problem
Single-shot detectors may have low-level features without strong semantic information, impairing small-object detection and higher-level feature quality.
Method
DES adds a segmentation branch to enrich low-level detection features and a global activation module to enrich higher-level features.
Results
81.7 mAP on VOC2007 test and 32.8 mAP on COCO test-dev are reported, alongside 31.7 FPS on a Titan Xp GPU.
Takeaways & Limitations
DES is reported to excel in both accuracy and speed while requiring limited modifications to the original SSD framework.
Abstract
from arXiv · showhide
We propose a novel single shot object detection network named Detection with Enriched Semantics (DES). Our motivation is to enrich the semantics of object detection features within a typical deep detector, by a semantic segmentation branch and a global activation module. The segmentation branch is supervised by weak segmentation ground-truth, i.e., no extra annotation is required. In conjunction with that, we employ a global activation module which learns relationship between channels and object classes in a self-supervised manner. Comprehensive experimental results on both PASCAL VOC and MS COCO detection datasets demonstrate the effectiveness of the proposed method. In particular, with a VGG16 based DES, we achieve an mAP of 81.7 on VOC2007 test and an mAP of 32.8 on COCO test-dev with an inference speed of 31.5 milliseconds per image on a Titan Xp GPU. With a lower resolution version, we achieve an mAP of 79.7 on VOC2007 with an inference speed of 13.0 milliseconds per image.
1. Introduction
DES enriches a typical single-shot detector by adding semantic information at both low- and high-level detection features. It combines a segmentation branch with a global activation module and reports strong accuracy and speed.
- DES addresses weak low-level semantics that can hinder small-object detection and damage higher-level feature quality.
- DES combines a typical VGG16-based detection branch with a segmentation branch and a global activation module.The segmentation branch enriches low-level features, while global activation blocks enrich higher-level detection features.
- Segmentation features activate low-level detection features to combine basic visual patterns with high-level semantic information.The segmentation branch uses weak bounding-box-level segmentation supervision, and the augmented features replace the original low-level map.
- Global activation blocks learn relationships between feature channels and object classes in a self-supervised manner.The blocks are added at higher-level prediction sources and increase semantic information in those feature maps.
- 81.7 mAP on VOC2007 test and 32.8 mAP on COCO test-dev are reported for DES.
- 31.7 FPS is reported on a single Titan Xp GPU, with DES described as faster than R-FCN and ResNet-based SSD.
2. Related work
Related work contrasts proposal-based two-stage detectors with dense one-stage detectors and motivates incorporating segmentation cues into detection. DES differs from prior segmentation-assisted methods by enriching low-level features through activation rather than concatenation.
- Two-stage detectors generate object proposals before per-proposal classification and bounding-box regression.
- One-stage detectors eliminate proposal generation and perform detection and bounding-box regression densely across locations and scales.
- Existing one-stage detectors focus on object detection during training without close attention to local cues within objects supplied by semantic segmentation.
- Weakly annotated data such as bounding boxes or image-level labels can support semantic segmentation without per-pixel training labels.
- DES places segmentation information at the lowest detection feature map and combines it through activation rather than concatenation.This differs from prior work that augmented detection features by concatenating segmentation-aware features at the highest level.
- DES increases semantic information more simply than top-down architectures because it does not use reverse connections.
3. Proposed method
DES combines an SSD-style detection branch with semantic enrichment at low and higher detection layers. A weakly supervised segmentation branch augments low-level features, while global activation blocks enrich higher-level features during multi-task training.
- Detection branch: DES uses SSD as its single-shot detection branch, with hierarchical feature maps detecting objects from small to large sizes.The branch uses VGG16 as a backbone and applies anchors with different sizes and aspect ratios across detection feature maps.
- Low-level semantic enrichment: The segmentation branch augments the first prediction source layer, conv4_3, using bounding-box-level weak segmentation labels instead of pixel-level annotations.Its supervision assigns pixels within bounding boxes to object classes and assigns background elsewhere, resolving overlaps using the smallest bounding box.
- Low-level semantic enrichment: The branch produces a semantic feature map Z that activates the original low-level feature map X by element-wise multiplication, yielding X′ for detection.X′ combines basic visual patterns with high-level semantic information and replaces X in the detection branch.
- Higher-level semantic enrichment: Global activation blocks enrich higher-level detection layers by learning channel–class relationships after spatial information is eliminated.The blocks use spatial pooling, channel-wise learning, and broadcasted multiplication, and are attached from conv4_3 through conv9_2.
- Multi-task training: Training adds a segmentation cross-entropy loss to the original detection loss, with α balancing the two tasks.The segmentation prediction uses bounding-box-generated ground truth, so the additional supervision does not require extra segmentation annotation.
4. Experiments
Experiments on VOC and COCO evaluate DES across resolutions, pretrained settings, speed, and component ablations. DES improves detection performance, particularly for small objects, while maintaining competitive inference efficiency.
- VOC results: DES improves VOC2007 mAP from 77.5 to 79.7 at low resolution and from 79.5 to 81.7 at high resolution.These results compare DES with original SSD under both resolution settings.
- VOC results: DES outperforms competing detectors on VOC2012 test and retains an advantage over SSD when fine-tuned from COCO-pretrained models.The reported VOC2012 comparison includes two-stage methods and detectors using segmentation information.
- COCO results: DES performs strongly on small objects, with at least a 27.5% relative improvement over competitors reporting small-object results.DES512 is slightly worse than ResNet101-based SSD513 for large objects, although it outperforms VGG16-based SSD512 there.
- Ablation studies: The global activation module improves performance by 0.6, while adding the segmentation branch yields a further large improvement on VOC2007 test.The ablation supports using both modules to enrich detection features.
- Efficiency: DES300 reaches 76.8 FPS with 79.7 mAP, while DES512 provides higher mAP with more inference time.DES is slower than original VGG16-based SSD but faster than ResNet101-based SSD with higher reported performance.
5. Conclusion
DES enriches single-shot detection features with semantic information through a segmentation branch and global activation module, improving accuracy and speed.
- DES enriches low-level detection features with high-level semantic information using a weakly supervised segmentation branch and global activation module.The method is designed as a flexible, simple modification to the SSD framework.