Source-linked AI summary
Dynamic-structured Semantic Propagation Network
Xiaodan Liang, Hongfei Zhou, Eric Xing
TL;DR
Semantic segmentation models often ignore concept hierarchies and require task-specific classifiers, limiting use of large vocabularies and diverse annotations. DSSPN builds a hierarchy-guided semantic neuron graph, dynamically activates relevant sub-graphs, and propagates ancestor features to child concepts. Across four benchmarks, it outperforms state-of-the-art models and jointly trained universal models surpass fine-tuning, although fair comparison with some state-of-the-art systems is limited by their wider or deeper pretrained backbones.
Problem
Existing segmentation models ignore semantic hierarchies, struggle with large concept vocabularies, and require task-specific models because datasets use different label sets.
Method
DSSPN explicitly builds a hierarchy-guided semantic neuron graph, dynamically activates image-specific sub-graphs, and propagates ancestor features into fine-grained child neurons.
Results
Experiments on four public datasets show DSSPN outperforms state-of-the-art segmentation models, while unified training across diverse datasets surpasses the commonly used fine-tuning scheme.
Takeaways & Limitations
DSSPN provides a unified segmentation model that uses semantic structure and selectively activates neurons for different annotations and concepts.
Takeaways & Limitations
Fair comparison with some state-of-the-art methods is not possible because they use wider or deeper ImageNet-pretrained networks.
Abstract
from arXiv · showhide
Semantic concept hierarchy is still under-explored for semantic segmentation due to the inefficiency and complicated optimization of incorporating structural inference into dense prediction. This lack of modeling semantic correlations also makes prior works must tune highly-specified models for each task due to the label discrepancy across datasets. It severely limits the generalization capability of segmentation models for open set concept vocabulary and annotation utilization. In this paper, we propose a Dynamic-Structured Semantic Propagation Network (DSSPN) that builds a semantic neuron graph by explicitly incorporating the semantic concept hierarchy into network construction. Each neuron represents the instantiated module for recognizing a specific type of entity such as a super-class (e.g. food) or a specific concept (e.g. pizza). During training, DSSPN performs the dynamic-structured neuron computation graph by only activating a sub-graph of neurons for each image in a principled way. A dense semantic-enhanced neural block is proposed to propagate the learned knowledge of all ancestor neurons into each fine-grained child neuron for feature evolving. Another merit of such semantic explainable structure is the ability of learning a unified model concurrently on diverse datasets by selectively activating different neuron sub-graphs for each annotation at each step. Extensive experiments on four public semantic segmentation datasets (i.e. ADE20K, COCO-Stuff, Cityscape and Mapillary) demonstrate the superiority of our DSSPN over state-of-the-art segmentation models. Moreoever, we demonstrate a universal segmentation model that is jointly trained on diverse datasets can surpass the performance of the common fine-tuning scheme for exploiting multiple domain knowledge.
1. Introduction
DSSPN integrates semantic concept hierarchies into dynamic network optimization for hierarchical segmentation and selectively activates relevant neuron sub-graphs. It propagates ancestor knowledge to fine-grained concepts and supports joint learning across diverse datasets.
- Existing segmentation models struggle with large concept vocabularies because they ignore intrinsic taxonomies and semantic hierarchies.
- Label discrepancies across datasets force task-specific models, limiting universal segmentation and reuse of annotations with different label sets.
- DSSPN constructs a semantic neuron graph whose neurons recognize concepts in a hierarchy, then activates only neurons for image concepts and their ancestors.
- Its dense semantic-enhanced block propagates learned features from all ancestor neurons into each child neuron for fine-grained recognition.
- DSSPN uses dynamic computation graphs during training, with dynamic batching and a memory-efficient implementation for multiple sample-specific graphs.
- Experiments on four datasets show DSSPN improves state-of-the-art segmentation networks and that unified training across datasets can outperform fine-tuning.
2. Related Work
Prior segmentation and recognition methods improve feature representation or exploit semantic relations, but commonly use flat predictions or implicit structure. DSSPN instead explicitly constructs network modules from semantic hierarchies and dynamically activates them for scalable universal segmentation.
- Semantic segmentation: Earlier segmentation systems improve representation through architectures and filters but perform relatively poorly on benchmarks with large concept vocabularies.
- Semantic segmentation: Flat prediction layers disregard intrinsic semantic correlations and limit model capability for many concepts.
- Dynamic and graph network structure: Dynamic graph-based networks adapt computation to graph-structured data such as superpixels, social networks, and object relationships.
- Hierarchical recognition: Hierarchical recognition methods use WordNet, label graphs, or word embeddings, whereas DSSPN explicitly constructs network modules from semantic hierarchies and dynamically activates them for universal segmentation.
3. Dynamic Tree-Structured Propagation Network
DSSPN incorporates a semantic concept hierarchy into a dynamic neuron graph, using ancestor features and image-specific activated subgraphs for hierarchical pixel-wise segmentation. The design supports fine-grained recognition, diverse annotation granularities, and extensible universal models.
- 3.1. Semantic Neuron Graph: DSSPN constructs a semantic neuron graph whose neurons represent parent concepts and predict among their child concepts.The graph is derived from a semantic concept hierarchy, decomposing large-vocabulary segmentation into smaller hierarchical recognition tasks.
- 3.4. Universal Semantic Segmentation: The same dynamic propagation scheme accommodates datasets with different annotation granularities by selectively activating compatible semantic subgraphs.This enables joint training across diverse segmentation datasets within a unified semantic knowledge graph.
- 3.2. Dense Semantic-enhanced Block: Each neuron concatenates feature maps from all inherited ancestor neurons before producing its own fine-grained representation.The dense block follows the inheritance path from the root concept and applies a nonlinear transformation to the concatenated features.
- 3.2. Dense Semantic-enhanced Block: DSSPN decomposes dense prediction into smaller child-concept subtasks, using dynamically specified feature routes and depths rather than one flat prediction layer.This design uses a small feature-map size for each subtask and distinguishes concepts within local sibling groups.
- 3.3. Dynamic-structured Semantic Propagation: For each training image, DSSPN activates only the neurons for target concepts and their ancestors, then propagates hidden features through the resulting subgraph.At test time, hierarchical predictions recursively activate child neurons with available predictions.
- 3.4. Universal Semantic Segmentation: DSSPN can dynamically add or prune semantic neurons and concept labels while preserving previously learned parameters.The proposed online capacity updates are intended for adding datasets or adapting the concept hierarchy for different purposes.
4. Experiments
Experiments evaluate DSSPN across four segmentation datasets, compare design choices and training strategies, and assess unified training with diverse annotations. Results report gains over matched baselines, lower-cost dynamic propagation, and improved performance from dense semantic features and binary cross-entropy.
- Evaluation setup: DSSPN is evaluated on COCO-Stuff, ADE20K, Cityscapes, and Mapillary using a shared structure and mean IoU.Testing selects the semantic-neuron sub-graph corresponding to each dataset's defined concepts.
- State-of-the-art comparisons: 2.63% higher mean IoU is obtained than the ResNet-101, 2 conv baseline on ADE20K.The comparison uses ResNet-101-based methods on the ADE20K validation set.
- State-of-the-art comparisons: 3.6% IoU class improvement is obtained over DeepLabv2 (ResNet-101) on Cityscapes under matched single-scale, no-CRF settings.The paper notes that some recent methods use different base models or multi-scale inputs and are not fairly comparable.
- State-of-the-art comparisons: 4.81% mean IoU improvement is reported for DSSPN over the ResNet-101 baseline on Mapillary.The comparison primarily uses the baseline because competing methods use different basic networks.
- Unified training: The universal DSSPN model outperforms the fine-tuning scheme across datasets and improves recognition of rare COCO-Stuff labels such as baseball glove and broccoli.Joint training combines annotations from four datasets while selectively activating relevant semantic neurons.
- Ablation studies: Dynamic propagation matches fixed-structure performance while requiring less computation and memory by activating only relevant semantic neurons.The hierarchical prediction scheme focuses learning on confusing concepts at each level.
- Ablation studies: Dense feature concatenation improves performance over direct propagation and summation, while m = 48 is sufficient for feature-map size.Removing the dense block reduces parameters but sacrifices performance.
- Ablation studies: Per-pixel binary cross-entropy with sigmoid significantly outperforms Softmax and better supports unified learning across diverse label annotations.Softmax class competition is described as problematic when only some concepts under a super-class are annotated.