PaperlayerView exact revision on arXiv

Source-linked AI summary

Unified Perceptual Parsing for Scene Understanding

Tete Xiao, Yingcheng Liu, Bolei Zhou, Yuning Jiang, Jian Sun

arXiv:1807.10221v1cs.CV

TL;DR

The paper asks whether machine vision can recognize interconnected visual concepts across scenes, objects, parts, materials, and textures despite heterogeneous annotations. It defines Unified Perceptual Parsing, develops UPerNet with a heterogeneous-data training strategy, and benchmarks the resulting system. The trained networks segment a wide range of concepts and are used to discover visual knowledge in natural scenes.

  • Problem

    Visual recognition tasks are mostly studied independently, while no single image dataset provides annotations across all perceptual levels and annotation types are heterogeneous.

  • Method

    The paper combines heterogeneous image annotations with UPerNet, a hierarchical multi-task network based on FPN, and trains concept-specific paths using sampled data sources.

  • Results

    UPerNet jointly parses scenes, objects, parts, materials, and textures, with joint training producing 71.35% top-1 scene accuracy and 54.19/84.45 material parsing performance.

  • Takeaways & Limitations

    Unified Perceptual Parsing and its trained networks support joint segmentation across a wide range of visual concepts and discovery of visual knowledge in scenes.

  • Takeaways & Limitations

    The authors identify better use of image-level annotations for pixel-level predictions as an area for future research.

Abstract

from arXiv · show

Humans recognize the visual world at multiple levels: we effortlessly categorize scenes and detect objects inside, while also identifying the textures and surfaces of the objects along with their different compositional parts. In this paper, we study a new task called Unified Perceptual Parsing, which requires the machine vision systems to recognize as many visual concepts as possible from a given image. A multi-task framework called UPerNet and a training strategy are developed to learn from heterogeneous image annotations. We benchmark our framework on Unified Perceptual Parsing and show that it is able to effectively segment a wide range of concepts from images. The trained networks are further applied to discover visual knowledge in natural scenes. Models are available at \url{https://github.com/CSAILVision/unifiedparsing}.

1 Introduction

Unified Perceptual Parsing asks whether one system can jointly recognize interconnected visual concepts spanning scenes, objects, parts, textures, and materials despite heterogeneous datasets and annotations.

  • Motivation: Humans organize scene interpretation across materials and textures, objects, and compositional parts at multiple perceptual levels.A single glance can identify a living room, its objects, their parts, and material or texture attributes.
  • Motivation: Existing recognition tasks are mostly studied independently, although scene, object, texture, and material perception are intertwined in human visual perception.This motivates asking whether neural networks can solve several visual recognition tasks simultaneously.
  • Challenges: UPP lacks a single dataset annotated across all visual levels, so relevant supervision must be combined from task-specific datasets.ADE20K provides pixel-wise scene annotations, whereas DTD provides image-level texture annotations.
  • Approach: The proposed training strategy samples one data source per iteration and updates only the related inference path, reducing noisy gradients from unrelated annotations.The framework also exploits hierarchical features from a single network for concepts at different semantic levels.
  • Contributions: UPP introduces a task requiring systems to parse multiple visual concepts at once and a hierarchical UPerNet for learning from heterogeneous image datasets.The contributions include jointly inferring and discovering visual knowledge underneath images.
  • Related Work: The work builds on semantic segmentation and multi-task learning, while emphasizing that prior combined tasks do not form a hierarchy of visual concepts.The paper positions UPP as a broader integration of scene, object, part, texture, and material understanding.

2 Defining Unified Perceptual Parsing

Unified Perceptual Parsing is defined as recognizing many visual concepts from an image across scenes, objects, parts, materials, and textures. The paper constructs Broden+ from heterogeneous sources and evaluates each label type with metrics suited to its annotation structure.

  • 2 Defining Unified Perceptual Parsing: Unified Perceptual Parsing recognizes many visual concepts organized from scene labels through objects and parts to materials and textures.Because no dataset contains every level of annotation, the task depends on combining different training sources.
  • 2.1 Datasets: Broden unifies ADE20K, Pascal-Context, Pascal-Part, OpenSurfaces, and DTD to provide scenes, objects, parts, materials, and textures in varied contexts.Objects, parts, and materials are represented with pixel-level segmentation in the combined sources.
  • 2.1 Datasets: Broden+ standardizes Broden by merging similar concepts across datasets and addressing class imbalance for segmentation training.The source text identifies cross-dataset concept merging and imbalance as standardization needs.
  • 2.1 Datasets: 57,095 images comprise Broden+, including 22,210 from ADE20K, 10,103 from Pascal-Context and Pascal-Part, 19,142 from OpenSurfaces, and 5,640 from DTD.The dataset statistics summarize the heterogeneous source composition.
  • 2.1 Datasets: Broden+ frequency analysis retains the top 120 object classes and, for the top 30 objects, their five most frequent parts after minimum-frequency filtering.Object classes require at least 50 images or 50,000 pixels, while parts require at least 20 images.
  • 2.2 Metrics: Scene and texture labels are image-level, whereas object, part, and material labels are pixel-wise; object and part annotations are dense, but material annotations are partial.Texture-labeled images mostly consist of localized object regions.
  • 2.2 Metrics: Evaluation uses P.A. and mIoU for object and material parsing, P.A. and mIoU-bg for parts, and top-1 accuracy for scene and texture classification.The metric choices reflect differences in label type and annotation coverage.
  • 2.2 Metrics: UPerNet combines an FPN with a PPM and attaches prediction heads to feature levels matched to scene, object, part, material, and texture semantics.Scene uses post-PPM features; object and part use fused FPN features; material uses the highest-resolution FPN feature; texture uses Res-2 features.

3 Designing Networks for Unified Perceptual Parsing

UPerNet combines hierarchical FPN features with pyramid pooling to parse visual concepts at multiple levels using a single network. Its design addresses heterogeneous supervision and achieves competitive semantic segmentation with reduced training cost.

  • Architecture: UPerNet uses a Feature Pyramid Network with lateral connections and a Pyramid Pooling Module to combine multi-level and global representations.FPN fuses high-level semantics into lower levels, while PPM supplies global prior representations before the top-down branch.
  • Architecture: A single network predicts scene labels from pooled P5 features, while object and part labels use fused FPN features.Scene prediction uses global average pooling and a linear classifier; object prediction fuses all FPN feature maps, and parts share the object feature map.
  • Evaluation: 63% training time is required to reach almost identical performance to PSPNet under object-only supervision.The comparison uses the same number of epochs and omits PSPNet’s deep supervision and data augmentations apart from scale jitter.
  • Training: Randomly sampling one data source per iteration and updating only its inference path reduces noise from heterogeneous task annotations.The strategy avoids erratic behavior when gradients associated with a particular concept are noisy.
  • Design motivation: Dilated segmentation frameworks create a drawback for UPP because deep networks rapidly down-sample feature maps, making dilation computationally expensive.In the cited ResNet example, maintaining lower down-sampling rates requires enlarging feature maps in deep blocks.
  • Evaluation: FPN remains competitive with substantially lower computational requirements, while higher feature-map resolution and all-level fusion improve performance.The Table 2 analysis also reports compatibility between PPM and FPN.

4 Experiments

The experiments evaluate UPerNet on semantic segmentation, Unified Perceptual Parsing, and visual-knowledge discovery. Results show efficient hierarchical prediction across heterogeneous concepts, while revealing both benefits and costs of joint training and limits in texture supervision.

  • 4.1 Main results: 34.46/76.04 mIoU and P.A. are achieved with P4 FPN features on ADE20K, nearly matching the strong baseline while requiring about one-third of its training time.Adding PPM improves performance by 4.87/3.09, and fusing all FPN levels performs best.
  • 4.1 Main results: 71.35% top-1 accuracy is obtained for scene classification after adding the scene branch, with negligible object and part performance downgrades.Joint material, object, part, and scene training yields 54.19/84.45 on material, 23.36/77.09 on object, and 28.75/46.92 on part parsing.
  • 4.1 Main results: 35.10 texture classification accuracy is obtained after fine-tuning on texture images, indicating that fine-tuning only on texture labels is not optimal but is necessary for natural and synthetic data fusion.The result is produced by converting frequent pixel-level predictions into an image-level prediction.
  • 4.2 Discovering visual knowledge in natural scenes: UPerNet predicts hierarchical outputs simultaneously and discovers compositional relations among scenes, objects, parts, materials, and textures.The discovered knowledge includes scene-object, object-part, object-material, part-material, and material-texture relations, summarized through graphs and selected candidates.
  • 4.2 Discovering visual knowledge in natural scenes: The extracted knowledge is consistent with human knowledge and provides information across multiple concept types for understanding scenes and the real world.Examples include common objects forming scenes and materials associated with objects or parts.

5 Conclusion

The work introduces Unified Perceptual Parsing for visual concepts spanning scenes, objects, parts, materials, and textures. It develops and benchmarks a multi-task network and training strategy, then applies the trained network to discover visual knowledge among scenes.

  • Unified Perceptual Parsing targets visual concepts across scene categories, objects, parts, materials, and textures.
  • The paper develops and benchmarks a multi-task network with a training strategy for handling heterogeneous annotations.
  • The trained network is used to discover visual knowledge among scenes.
Loading 1807.10221v1…