PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning from Synthetic Animals

Jiteng Mu, Weichao Qiu, Gregory Hager, Alan Yuille

arXiv:1912.08265v2cs.CV

TL;DR

Animal parsing has limited labeled data, and synthetic images face a domain gap with real images. The paper combines synthetic CAD data with consistency-constrained semi-supervised learning using unlabeled real images. It reports reliable real-image keypoints without labels and better cross-domain generalization from synthetic data than from real-image training.

  • Problem

    Animal parsing has few labeled keypoint and part datasets, annotation is costly, and synthetic images have a domain gap with real images.

    Source: §1 · p. 1

  • Method

    CC-SSL uses synthetic CAD animal images and spatial and temporal consistency to generate and train on pseudo-labels from unlabeled real images.

    Source: §3 · p. 3, §5 · p. 8

  • Results

    Synthetic-trained models generalize better across Visual Domain Adaptation Challenge domains than real-trained models, while reliably predicting real-image keypoints without real-image labels.

    Source: §1 · p. 2, §5 · p. 8

  • Takeaways & Limitations

    The 10+ animal synthetic dataset supports pose estimation, part segmentation, and effective multi-task learning.

    Source: §1 · p. 2, §5 · p. 8

    Synthetic-to-real domain gaps hinder generalization, and creating diverse realistic animal CAD models is difficult.

    Source: §1 · p. 1

Abstract

from arXiv · show

Despite great success in human parsing, progress for parsing other deformable articulated objects, like animals, is still limited by the lack of labeled data. In this paper, we use synthetic images and ground truth generated from CAD animal models to address this challenge. To bridge the domain gap between real and synthetic images, we propose a novel consistency-constrained semi-supervised learning method (CC-SSL). Our method leverages both spatial and temporal consistencies, to bootstrap weak models trained on synthetic data with unlabeled real images. We demonstrate the effectiveness of our method on highly deformable animals, such as horses and tigers. Without using any real image label, our method allows for accurate keypoint prediction on real images. Moreover, we quantitatively show that models using synthetic data achieve better generalization performance than models trained on real images across different domains in the Visual Domain Adaptation Challenge dataset. Our synthetic dataset contains 10+ animals with diverse poses and rich ground truth, which enables us to use the multi-task learning strategy to further boost models' performance.

1. Introduction

Animal parsing lags behind human parsing amid scarce, costly annotations and a synthetic-to-real domain gap. The paper introduces synthetic animal data and CC-SSL, reporting accurate real-image keypoints without labels and stronger cross-domain generalization from synthetic data.

1. Introduction

  • Animal parsing has few keypoint and part datasets, and large-scale annotation is prohibitively expensive, unlike the more advanced state of human parsing.Animal parsing matters for monitoring wild behavior, bioinspired robotics, and motion capture.Source: §1 · p. 1
  • Synthetic animal images offer scalable rich ground truth, but domain gaps hinder real-image generalization and creating diverse realistic animal CAD models is difficult.Synthetic rendering can also sample rare cases that are underrepresented in real-world datasets.Source: §1 · p. 1
  • The paper introduces CC-SSL, which uses synthetic CAD models and pseudo-labels on unlabeled real images; it reports reliable real-image keypoints for horses and tigers without real-image labels.The method uses a single CAD animal model and three consistency checks to assess predicted-label quality.Source: §1 · pp. 1–2
  • Models trained with synthetic data generalize better across visual domains than models trained on real images, and synthetic-plus-real training outperforms real-only training when labels are available.The paper also reports that CC-SSL achieves similar performance to models trained on real data without using real-image annotations, and outperforms other domain-adaptation methods by a large margin.Source: §1 · pp. 1–2
  • The generated dataset covers 10+ animal CAD models with diverse poses and rich ground truth, supporting pose estimation, part segmentation, and multi-task learning.The overview lists dense 2D keypoints, part segmentation, and depth maps as ground-truth annotations.Source: §1 · p. 2, Figure 1 · p. 2

2. Related Work

Related work covers limited animal parsing datasets and synthetic-model approaches, domain adaptation methods, and self-training techniques. The paper distinguishes its approach by using one CAD model and combining spatial and temporal consistency without an extra inference-time segmentation model.

2.1. Animal Parsing

  • Animal pose and part datasets cover only a small fraction of species, while prior synthetic-model approaches often rely on manual annotations or real-image segmentation at inference.Unlike silhouette-based [3], this strategy does not require an additional robust segmentation model during inference.Source: §2.1 · p. 2

2.2. Unsupervised Domain Adaptation

  • Domain adaptation work uses image translation, feature-distance minimization, shared/private feature spaces, or adversarial learning; prior synthetic-human approaches used many models, while this work learns domain-invariant features from one CAD model.The cited approaches include cycle consistency and rendered collections of 145 synthetic human models.Source: §2.2 · pp. 2–3

2.3. Self-training

  • Self-training methods address noisy pseudo-labels through confidence regularization or ensembling, but high-confidence labels are not assured under large domain discrepancy; this work adds curriculum learning and spatial and temporal consistencies.It extends omnisupervised learning with these consistency cues.Source: §2.3 · p. 3

3. Approach

CC-SSL starts from a model trained on synthetic data, then bootstraps it with unlabeled real images using consistency checks and iterative training. The approach also formulates image generation around task-related and task-independent factors and builds a synthetic dataset with diverse animals and ground truth.

3. Approach

  • The approach treats synthetic images with labels as the source and real images as the target, bootstrapping a source-trained model with unlabeled target data.Its pipeline overview includes image generation, consistency-based pseudo-labeling, iterative training, and synthetic dataset creation.Source: §3 · p. 3

3.1. Formulate Image Generation Procedure

  • The image-generation formulation separates task-related factors such as pose and shape from task-independent factors such as texture, lighting, and background.The formulation assumes essential knowledge is shared between synthetic and natural images.Source: §3.1 · p. 3

3.2. Consistency

  • Pseudo-label generation checks invariance to task-irrelevant changes, equivariance to geometric changes, and temporal consistency, using optical flow to propagate labels from confident frames when available.A curriculum begins with the top 20% of ranked keypoints and increases inclusion to 80% over iterations.Source: §3.2 · p. 4, Figure 2 · p. 4, §3.3 · p. 5

3.3. Pseudo-Label Generation

  • Consistency checks filter noisy pseudo-labels, while curriculum learning gradually increases training samples across iterative model updates.The method assumes labels based on correct information satisfy the consistency checks.Source: §3.3 · p. 5

3.4. Consistency-Constrained Semi-Supervised Learning (CC-SSL)

  • CC-SSL jointly trains on labeled synthetic images and real images with generated pseudo-labels, using heatmap mean-square error balanced by γ.Training iterates from an initial synthetic-only model.Source: §3.4 · p. 5

3.5. Synthetic Dataset Generation

  • The synthetic dataset covers 10+ animals and varied poses, with controllable lighting, textures, viewpoints, and ground truth including depth, segmentation, and 2D/3D joint locations.Each animal has several animation sequences.Source: §3.5 · pp. 5–6

4. Experiments

Experiments test CC-SSL for horse and tiger pose estimation on TigDog, evaluate generalization on three VisDA2019 domains, and examine multi-task part segmentation.

4.1. Experiment Setup

  • Experiments use Stacked Hourglass models trained for 100 epochs, and TigDog provides 79 horse videos and 96 tiger videos for pose evaluation.TigDog has 19 annotated keypoints per frame, with neck keypoints excluded from the experiments.Source: §4.1 · p. 6, §4 · p. 6

4.2. 2D Pose Estimation

  • CC-SSL reaches PCK@0.05 of 70.77 for horses and 64.14 for tigers without real-image annotations; with labels, CC-SSL-R reaches 82.43 and 84.00, outperforming real-only models.The no-label results are close to models trained directly on real images; CC-SSL-R is obtained by fine-tuning CC-SSL with real-image labels.Source: §4.2 · p. 6, Table 1 · p. 7

4.3. Generalization Test on VisDA2019

  • VisDA2019 tests use sketch, painting, and clipart images, with visible-keypoint and full-keypoint accuracy evaluated for horses and tigers.Full-keypoint accuracy includes self-occluded keypoints; visible-keypoint accuracy counts only directly visible ones.Source: §4.3 · p. 7, Table 2 · p. 8
  • Synthetic-data models achieve the best performance across the tested generalization settings, while unlabeled-real-image CC-SSL beats Real in almost all domains except tiger painting.The authors hypothesize preserved yellow-and-black stripe texture in tiger paintings may explain the exception.Source: §4.3 · p. 7, Table 2 · p. 8

4.4. Part Segmentation

  • Joint training on keypoints and part segmentation improves real-image keypoint generalization for both animals over keypoint-only training.The models are trained on synthetic images and tested directly on TigDog; segmentation predictions are visualized but not quantitatively evaluated.Source: §4.4 · pp. 7–8, Table 3 · p. 8

5. Conclusions

The paper uses synthetic animal images and spatial and temporal consistency constraints to parse animals, demonstrating reliable real-image keypoint prediction without real-image labels. It also reports improved domain generalization and effective multi-task learning with a diverse synthetic dataset.

  • CC-SSL bridges the domain gap using synthetic images with spatial and temporal constraints.Source: §5 · p. 8
  • Synthetic-data models achieve better generalization across visual domains than models trained on real data.Source: §5 · p. 8
  • Without real-image labels, the model reliably detects keypoints on real horses and tigers in the TigDog Dataset.Source: §5 · p. 8
  • Joint training on synthetic and real images outperforms training on real images alone when real-image labels are used.Source: §5 · p. 8
  • The 10+ animal synthetic dataset provides diverse poses and rich ground truth, and multi-task learning is effective.Source: §5 · p. 8

Related papers

Loading 1912.08265v2…