PaperlayerView exact revision on arXiv

Source-linked AI summary

Graphonomy: Universal Human Parsing via Graph Transfer Learning

Ke Gong, Yiming Gao, Xiaodan Liang, Xiaohui Shen, Meng Wang, Liang Lin

arXiv:1904.04536v1cs.CV

TL;DR

Dataset-specific human parsing models struggle to generalize across domains and label granularities, motivating a single universal model. Graphonomy uses hierarchical graph transfer learning to reason within each label graph and transfer semantics across datasets. It reports state-of-the-art results on three human parsing datasets and advantageous universal parsing performance, while very small or overlapping persons remain challenging.

  • Problem

    Existing human parsing models are specialized to dataset domains and label granularities, making adaptation across related tasks require redundant annotation and extensive computation.

  • Method

    Graphonomy applies hierarchical graph transfer learning with Intra-Graph Reasoning within datasets and Inter-Graph Transfer across datasets to propagate semantic information.

  • Results

    Graphonomy achieves state-of-the-art results on three human parsing datasets and advantageous universal human parsing performance.

  • Takeaways & Limitations

    A single Graphonomy system can predict all parsing-label levels while unifying annotations from different resources without piling up model complexity.

  • Takeaways & Limitations

    Overlapping parts and very small persons cannot be predicted precisely, indicating that additional graph-structure knowledge is needed for challenging cases.

Abstract

from arXiv · show

Prior highly-tuned human parsing models tend to fit towards each dataset in a specific domain or with discrepant label granularity, and can hardly be adapted to other human parsing tasks without extensive re-training. In this paper, we aim to learn a single universal human parsing model that can tackle all kinds of human parsing needs by unifying label annotations from different domains or at various levels of granularity. This poses many fundamental learning challenges, e.g. discovering underlying semantic structures among different label granularity, performing proper transfer learning across different image domains, and identifying and utilizing label redundancies across related tasks. To address these challenges, we propose a new universal human parsing agent, named "Graphonomy", which incorporates hierarchical graph transfer learning upon the conventional parsing network to encode the underlying label semantic structures and propagate relevant semantic information. In particular, Graphonomy first learns and propagates compact high-level graph representation among the labels within one dataset via Intra-Graph Reasoning, and then transfers semantic information across multiple datasets via Inter-Graph Transfer. Various graph transfer dependencies (\eg, similarity, linguistic knowledge) between different datasets are analyzed and encoded to enhance graph transfer capability. By distilling universal semantic graph representation to each specific task, Graphonomy is able to predict all levels of parsing labels in one system without piling up the complexity. Experimental results show Graphonomy effectively achieves the state-of-the-art results on three human parsing benchmarks as well as advantageous universal human parsing performance.

1. Introduction

Human parsing models are typically specialized by dataset domain and label granularity, limiting reuse across tasks. Graphonomy addresses this by using hierarchical graph transfer learning to propagate shared semantic information within and across datasets for universal parsing.

  • Motivation: Different datasets encode related semantic parts at different granularities, such as head versus hat, hair, and face.These parts also exhibit intrinsic relations, including containment and adjacency, that can support information propagation.
  • Motivation: Existing human parsing networks fit individual dataset domains and label granularities, making direct adaptation to related tasks difficult and costly.Adapting models requires redundant annotation and extensive computation to train task-specific systems.
  • Motivation: A shared-backbone multi-task solution exchanges features but overlooks common semantic knowledge such as label hierarchy, visual similarity, and linguistic correlations.This motivates modeling relationships among labels rather than relying only on brute-force feature sharing.
  • Graphonomy: Graphonomy incorporates human knowledge and label taxonomy into intermediate graph representations to learn common semantic coherency across domains.Its design targets multiple parsing levels while enabling tasks to provide mutual benefits through graph transfer learning.
  • Graphonomy: Intra-Graph Reasoning refines representations within a dataset, while Inter-Graph Transfer distills related semantics across datasets and bridges their labels.Image features are projected into graph nodes for semantic parts, and cross-graph transfer uses relationships between semantic vertices.
  • Results: Experiments on three human parsing benchmarks show that Graphonomy improves multiple parsing levels through information propagation and semantic graph distillation.The paper reports state-of-the-art results on three human parsing datasets and universal human parsing performance.

2. Related Work

Related work spans human parsing, multi-task learning, knowledge-guided graph reasoning, and transfer learning. Graphonomy combines these directions by transferring high-level graph representations across dataset-specific taxonomies.

  • Human Parsing: Prior human parsing research improves general feature representation with architectures and auxiliary guidance, while Graphonomy investigates universal parsing through graph transfer learning.The approach targets shared semantic structure across parsing tasks rather than only task-specific feature extraction.
  • Knowledge Transfer: Graphonomy represents each dataset taxonomy as a graph and connects graphs through transfer dependencies to propagate semantic features.This graph organization links concepts across multiple dataset-specific label systems.
  • Multi-task Learning: Multi-task learning produces multiple outputs through shared systems, but Graphonomy focuses on transferring semantic graph representations across heterogeneous parsing taxonomies.The related work includes joint optimization and unified handling of heterogeneous annotations.
  • Knowledge-guided Graph Reasoning: Knowledge-guided graph reasoning models domain knowledge as graphs to mine correlations among labels or objects in images.Examples include local and global reasoning and explicit semantic concept hierarchies.
  • Transfer Learning: Transfer learning connects domains or tasks to reduce manual labeling, and Graphonomy transfers high-level graph representations to reduce label discrepancy.This positions Graphonomy at the intersection of domain transfer and semantic taxonomy alignment.

3. Graphonomy

Graphonomy unifies human parsing tasks with hierarchical graph transfer learning: it reasons within each dataset’s label graph, transfers semantics across datasets, and supports joint parsing despite differing label granularities.

  • Graphonomy: Graphonomy embeds hierarchical graph transfer learning into conventional parsing networks to unify label annotations across datasets and parsing levels.It enhances image features without piling up separate task-specific models.
  • Intra-Graph Reasoning: Intra-Graph Reasoning projects convolutional features into semantic nodes connected by human-body relations, propagates global context, and re-projects the result into image features.Graph nodes typically correspond to a dataset’s target part labels, while graph convolution evolves their representations through a normalized adjacency matrix.
  • Inter-Graph Transfer: Inter-Graph Transfer distills related semantics between source and target graphs to bridge labels with different structures and hierarchical meanings.For example, torso may correspond to upper-clothes and pants, while upper-clothes may decompose into coat, T-shirt, and sweater.
  • Inter-Graph Transfer: Graph transfer dependencies include handcrafted subordinate relations, learnable weights, visual feature similarity, and linguistic semantic similarity from word embeddings.The transfer matrix maps source graph nodes to target graph nodes, with weights determined by the selected dependency scheme.
  • Universal Human Parsing: Graphonomy combines diverse parsing datasets in an end-to-end universal model that alleviates label discrepancy and stabilizes joint-training optimization.It can also extend model capacity online while using transferred information from other graphs.

4. Experiments

Graphonomy is evaluated on three human parsing benchmarks with different label definitions and granularities, using a universal graph-transfer model alongside quantitative, ablation, and qualitative analyses. Results show strong benchmark performance, benefits from graph reasoning and transfer, and remaining difficulty with overlapping parts and very small persons.

  • Experimental Settings: Experiments cover PASCAL-Person-Part, ATR, and CIHP, whose annotations differ in label definitions and granularity.The evaluation uses accuracy, mean intersection over union, and average F-1 score.
  • Comparison with State-of-the-Arts: Graphonomy reaches 69.12% Mean IoU in universal human parsing and 71.14% Mean IoU when using the CIHP graph.Table 1 reports these values alongside multi-task learning at 68.13% Mean IoU.
  • Comparison with State-of-the-Arts: Graph transfer improves PASCAL-Person-Part parsing by incorporating human knowledge and label taxonomy into intermediate graph representations.The CIHP graph is transferred to enhance the PASCAL-Person-Part graph, with hierarchical label correlation driving information propagation and updates.
  • Comparison with State-of-the-Arts: Graphonomy improves CIHP performance to 58.58% Mean IoU, compared with prior work achieving 55% Mean IoU using multi-task learning.The authors attribute the improvement to using semantic information from related parsing tasks.
  • Qualitative Results: Qualitative results show precise, fine-grained parsing across task levels and greater robustness to clothing appearance variation, occlusion, and background disturbance.Failure cases remain for overlapped parts and very small persons, indicating that additional graph knowledge is needed.
  • Ablation Studies: Intra-Graph Reasoning yields a 0.50% improvement over the basic network, while human-prior adjacency contributes more than graph-convolution parameters alone.The comparison uses results #1 versus #3 and evaluates the adjacency matrix Ae in results #2 versus #3.
  • Ablation Studies: Feature similarity is the strongest tested transfer dependency, whereas handcrafted relations diminish performance; combining feature and semantic similarity provides additional improvement.The study compares cross-domain transfer strategies and uses feature plus semantic similarity in Graphonomy.

5. Conclusion

Graphonomy uses hierarchical graph transfer learning to unify human parsing labels across datasets and levels, predicting them in one system without piling up complexity. Its consistent improvements across datasets and advantageous universal parsing performance support this unified approach.

  • Graphonomy incorporates hierarchical graph transfer learning into a conventional parsing network to predict all labels in one system without piling up complexity.
  • The universal model is designed to alleviate label discrepancy and utilize annotations across different resources.
  • Graphonomy demonstrates consistent human parsing improvements on all datasets.
  • Its advantageous universal human parsing performance indicates the model can unify varied label annotations and address different parsing needs.
Loading 1904.04536v1…