PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Hierarchical Semantic Segmentation

Liulei Li, Tianfei Zhou, Wenguan Wang, Jianwu Li, Yi Yang

arXiv:2203.14335v2cs.CV

TL;DR

Semantic segmentation typically treats classes as flat and disjoint, leaving hierarchical pixel-level reasoning underexplored. HSSN adapts existing models through pixel-wise multilabel classification and hierarchy-based learning constraints, outperforming existing segmentation approaches across four datasets and varied architectures.

  • Problem

    Semantic segmentation largely uses disjoint pixel labels, leaving structured hierarchical relations and multi-level visual understanding underexplored.

  • Method

    HSSN casts hierarchical segmentation as pixel-wise multilabel classification and uses hierarchy constraints plus hierarchy-induced margins to regularize predictions and pixel representations.

  • Results

    HSSN outperforms existing segmentation models across four datasets, class hierarchies, architectures, and backbones.

  • Takeaways & Limitations

    Hierarchy-aware supervision and margin separation provide a general way to adapt hierarchy-agnostic segmentation networks for structured scene parsing.

  • Takeaways & Limitations

    The formulation assumes a tree-structured class hierarchy with root, internal, and leaf concepts linked by decomposition relations.

Abstract

from arXiv · show

Humans are able to recognize structured relations in observation, allowing us to decompose complex scenes into simpler parts and abstract the visual world in multiple levels. However, such hierarchical reasoning ability of human perception remains largely unexplored in current literature of semantic segmentation. Existing work is often aware of flatten labels and predicts target classes exclusively for each pixel. In this paper, we instead address hierarchical semantic segmentation (HSS), which aims at structured, pixel-wise description of visual observation in terms of a class hierarchy. We devise HSSN, a general HSS framework that tackles two critical issues in this task: i) how to efficiently adapt existing hierarchy-agnostic segmentation networks to the HSS setting, and ii) how to leverage the hierarchy information to regularize HSS network learning. To address i), HSSN directly casts HSS as a pixel-wise multi-label classification task, only bringing minimal architecture change to current segmentation models. To solve ii), HSSN first explores inherent properties of the hierarchy as a training objective, which enforces segmentation predictions to obey the hierarchy structure. Further, with hierarchy-induced margin constraints, HSSN reshapes the pixel embedding space, so as to generate well-structured pixel representations and improve segmentation eventually. We conduct experiments on four semantic segmentation datasets (i.e., Mapillary Vistas 2.0, Cityscapes, LIP, and PASCAL-Person-Part), with different class hierarchies, segmentation network architectures and backbones, showing the generalization and superiority of HSSN.

1. Introduction

The paper introduces hierarchical semantic segmentation to represent pixels through structured class relations rather than disjoint labels, and proposes HSSN as a general framework for adapting and training segmentation networks.

  • Modern semantic segmentation usually assigns each pixel one disjoint label, overlooking structured relations among visual concepts.
  • Hierarchical semantic segmentation associates each pixel with a root-to-leaf path, capturing general-to-specific relations such as human→rider→bicyclist.
  • HSSN formulates hierarchical segmentation as pixel-wise multilabel classification, requiring only marginal changes to existing hierarchy-agnostic models.
  • HSSN enforces hierarchy coherence through ancestor and descendant constraints during learning, rather than only propagating information across the hierarchy.
  • Experiments span four public benchmarks, multiple class hierarchies, three segmentation architectures, and CNN- and Transformer-based backbones.

2. Related Work

Related work applies semantic hierarchies in segmentation and classification, but prior segmentation methods often require substantial architectural changes or focus on information propagation. HSSN instead uses hierarchy dependencies as supervision for coherent prediction and structured pixel embeddings.

  • Typical semantic segmentation treats pixels as assignments to disjoint categories and rarely models relationships between semantic concepts.
  • Prior hierarchy-aware segmentation methods often target human body-part organization or substantially modify networks with graph-based components, limiting generality.
  • Hierarchical Classification: Hierarchy-aware classification distinguishes single-path and multi-path assignments, providing broader context for hierarchical multilabel prediction.
  • Hierarchical Embedding: HSSN uses tree-structured class dependencies as supervision to guide hierarchy-coherent pixel prediction and structured pixel embeddings.

3. Our Approach

HSSN adapts hierarchy-agnostic segmentation networks to hierarchical semantic segmentation by predicting labels across a class tree and regularizing both predictions and pixel embeddings with hierarchy-aware objectives.

  • 3.1. Hierarchical Semantic Segmentation Networks: The class hierarchy is represented as a tree whose nodes are semantic concepts and whose edges encode superclass–subclass decomposition relationships.Each pixel can therefore be associated with a root-to-leaf sequence of increasingly specific classes.
  • 3.1. Hierarchical Semantic Segmentation Networks: HSSN models each pixel as a multi-label prediction over the entire class hierarchy, modifying only the segmentation head rather than redesigning the network.The resulting score map covers all hierarchy nodes, and inference selects the highest-scoring feasible root-to-leaf path.
  • 3.2.1 Pixel-Wise Hierarchical Segmentation Learning: Hierarchy-aware segmentation learning enforces complementary positive and negative T-properties, requiring positive classes to include ancestors and negative classes to exclude descendants.These properties are converted into prediction constraints so that class scores remain coherent across the tree.
  • 3.2.1 Pixel-Wise Hierarchical Segmentation Learning: Tree-Min Loss constructs hierarchy-coherent scores from the original predictions, guaranteeing the positive and negative T-constraints and explicitly penalizing constraint violations.Unlike independent binary cross-entropy, the structured score distribution allows undesired hierarchical predictions to be penalized directly.
  • 3.2.1 Pixel-Wise Hierarchical Segmentation Learning: Focal Tree-Min Loss adds a focusing factor to reduce the relative loss of well-classified samples and emphasize difficult pixel samples.The paper reports that LFTM produces more precise and coherent results than LBCE in representative visual comparisons.
  • 3.2.2 Pixel-Wise Hierarchical Representation Learning: Tree-Triplet Loss reshapes the encoder’s pixel embedding space by imposing margins determined by distances between classes in the hierarchy.The margin-based objective pushes semantically nearby pixels closer than semantically distant pixels, producing structured embeddings.

4. Experiment

Experiments across urban-scene and human-parsing datasets evaluate HSSN under varied hierarchies, architectures, and backbones. HSSN improves segmentation results and its hierarchical losses contribute complementary gains.

  • Experimental Setup: Experiments cover two urban street-scene datasets and two human body-parsing datasets with official or convention-based class hierarchies.Mapillary Vistas 2.0, Cityscapes, PASCAL-Person-Part, and LIP are evaluated.
  • Quantitative Results: HSSN outperforms DeepLabV3+ on Mapillary Vistas 2.0 by 2.69%/3.21%/3.40% across the three hierarchy levels with ResNet-101.It also establishes a new state-of-the-art with Swin-Small.
  • Quantitative Results: HSSN outperforms MaskFormer on Cityscapes by 1.17%/1.43% in mIoU1 and mIoU2 with Swin-Small.The gains occur despite Cityscapes having a relatively simple semantic hierarchy.
  • Quantitative Results: HSSN surpasses existing hierarchical human parsers on PASCAL-Person-Part and improves over HHP on LIP by 1.12%/1.32%/1.45% across three levels.The PASCAL-Person-Part results are notable despite only 1,713 training samples.
  • Qualitative Results: Qualitative results show more precise and robust segmentation under occlusions, small objects, and densely arranged targets.Hierarchy-aware learning also distinguishes substantially different concepts with larger margins, reducing mistake severity.
  • Diagnostic Experiment: Ablations show that LFTM and LTT each improve the hierarchy-agnostic baseline, while their combination performs best.Performance improves as γ increases, with marginal gains at γ = 2; cosine distance outperforms Euclidean distance for LTT.

5. Conclusion

The conclusion presents HSSN as a structured solution that adapts existing segmentation networks while using hierarchy properties to regularize prediction and representation learning. Experiments across four datasets show improvements over existing segmentation models.

  • Conclusion: HSSN exploits taxonomic relations with only slight changes to existing hierarchy-agnostic segmentation networks.The framework applies hierarchy properties as optimization criteria and uses hierarchy-induced margin separation for pixel representations.
  • Conclusion: HSSN experimentally outperforms many existing segmentation models on four datasets.The conclusion positions the work as a step toward hierarchical semantic segmentation.

A. Detailed Hierarchical Architecture

The appendix uses each dataset's official hierarchical structure, adding a virtual root node for Mapillary Vistas 2.0 and Cityscapes.

  • Detailed Hierarchical Architecture: The experiments use official dataset hierarchies, with a virtual root node All added for Mapillary Vistas 2.0 and Cityscapes.Detailed hierarchies are provided in appendix figures A.1–A.4.

B. Additional Qualitative Result

Additional visualizations across four validation datasets show HSSN consistently produces better visual effects than the baseline model.

  • HSSN yields consistently better visual effects than the baseline model across four validation datasets.The datasets are Mapillary Vistas 2.0, Cityscapes, PASCAL-Person-Part, and LIP.

C. Additional Ablative Study

The additional ablation study finds limited sensitivity to selected hyperparameters and better performance when the loss-balance factor follows cosine annealing.

  • mε and 0.5 in Eq. 8 have minor impact on performance, indicating robustness to these hyperparameters.
  • Cosine annealing for the balance factor β between LFTM and LTT yields better performance.The paper attributes this to the network’s poor recognition capability at the initial training stage.

D. Discussion on Triplet Number

Triplet loss adds computation and slows training, but HSSN achieves promising performance with only 200 sampled triplets, while further increases provide minor improvements.

  • Triplet loss imposes additional computation and slows the training speed.
  • The additional study evaluates triplet-number impact on Mapillary Vistas 2.0 validation and PASCAL-Person-Part test data.Table D.1 reports training time in seconds for each batch.
  • HSSN reaches promising performance using a small number of triplets, such as 200, on both datasets.The number of triplets refers to training samples used for the tree-triplet loss.
  • Further increasing the number of triplets brings only minor improvements, so HSSN uses 200 for all datasets.This setting facilitates triplet sampling at negligible cost while retaining a performance boost.
Loading 2203.14335v2…