PaperlayerView exact revision on arXiv

Source-linked AI summary

Disentangled Representation Learning for 3D Face Shape

Zi-Hang Jiang, Qianyi Wu, Keyu Chen, Juyong Zhang

arXiv:1902.09887v2cs.CV

TL;DR

Existing linear face models have limited representation ability, while deep models described here do not separate identity and expression. This paper learns nonlinear identity and expression representations from mesh deformation features, reporting better disentanglement and more natural expression transfer than existing methods.

  • Problem

    Linear face models are limited by linear formulation, while deep-learning face models encode whole shapes in one latent vector rather than separating identity and expression.

    Source: §1 · p. 1

  • Method

    The framework learns separate nonlinear identity and expression representations for face meshes using vertex-based deformation features instead of Euclidean coordinates.

    Source: §1 · p. 2, §3.1 · p. 3, §3.2 · p. 4

  • Results

    The method achieves better identity-expression disentanglement and more natural expression-transfer results than existing methods.

    Source: §1 · p. 2, §6 · p. 8

  • Takeaways & Limitations

    The learned identity and expression latent spaces support semantic edits and expression transfer.

    Source: §1 · p. 2, §5 · p. 8

Abstract

from arXiv · show

In this paper, we present a novel strategy to design disentangled 3D face shape representation. Specifically, a given 3D face shape is decomposed into identity part and expression part, which are both encoded and decoded in a nonlinear way. To solve this problem, we propose an attribute decomposition framework for 3D face mesh. To better represent face shapes which are usually nonlinear deformed between each other, the face shapes are represented by a vertex based deformation representation rather than Euclidean coordinates. The experimental results demonstrate that our method has better performance than existing methods on decomposing the identity and expression parts. Moreover, more natural expression transfer results can be achieved with our method than existing methods.

1. Introduction

A 3D face shape reflects identity and expression, but real facial variation is nonlinear and existing linear models have limited expression and disentanglement ability. The paper proposes a nonlinear attribute-decomposition framework using vertex-based deformation features.

1. Introduction

  • Separating identity and expression is motivated by applications including face-component transfer, face animation, and avatar animation.Source: §1 · p. 1
  • Linear and bilinear models combine attributes linearly, limiting expression ability and disentanglement despite nonlinear facial variation.Deep methods can represent geometry details or reconstruct whole faces with few parameters, but encode the whole shape in one latent vector rather than separating identity and expression.Source: §1 · p. 1
  • The paper targets a disentangled parametric space, addressing network design, shape representation, and use of identity and expression labels for training.Source: §1 · p. 1
  • Instead of the linear 3DMM decoder, the framework learns nonlinear identity and expression mappings and fuses them to recover shape.It uses spectral graph convolution and vertex-based deformation features rather than Euclidean coordinates to represent mesh inputs.Source: §1 · pp. 1–2
  • The proposed latent space supports semantic editing in identity and expression domains.Source: §1 · p. 2
  • The method achieves better identity-expression disentanglement and more satisfying expression-transfer results than existing methods.The paper also reports better performance for vertex-based deformation representation than Euclidean coordinates.Source: §1 · p. 2

2. Related Work

Related work includes linear parametric face models and nonlinear deep-learning models, alongside methods for analyzing 3D shapes with mesh or graph operations.

Linear 3D Face Shape Models

  • Linear face models include 3DMM, multi-linear attribute decomposition, and bilinear identity-expression modeling, while later work improves scale or facial articulation.Source: §2 · p. 2

Nonlinear 3D Face Models

  • Deep face models learn nonlinear latent spaces but do not address attribute decoupling; mesh-analysis methods include spatial and spectral graph convolution, with this work adapting fast spectral convolution.The authors describe their approach as the first deep-learning method for disentangling attributes on 3D mesh data.Source: §2 · p. 2

3. Disentangled 3D Face Representation

The method represents face shape using local deformation features and separates each mesh into identity and expression attributes. Separate extraction branches produce the attributes, a fusion network reconstructs the input, and training adds a loss to discourage cross-attribute information.

3.1. Overview

  • Local deformation features represent mesh changes relative to a neutral mean-face reference, rather than using Euclidean vertex coordinates.Each vertex’s deformation gradient is split into rotation and scale/shear components and represented as a 9D feature.Source: §3.1 · p. 3

3.2. Decomposition Networks

  • Parallel identity and expression branches use encoder-decoder networks to extract separate attributes from the deformation representation.Each branch uses a bottleneck as a compact parametric space for its attribute.Source: §3.2 · pp. 3–4

3.3. Fusion Network

  • The fusion network combines identity and expression mesh features and maps them back to the original mesh’s feature dimension for reconstruction.It concatenates the two mesh features and applies spectral graph convolution with activation layers.Source: §3.3 · p. 4

3.4. Training Process

  • Training proceeds through sequential pretraining of the decomposition and fusion networks, followed by end-to-end training with a disentangling loss.The loss is intended to keep expression information out of the identity part and identity information out of the expression part.Source: §3.4 · pp. 4–5

4. Experiment

The experiments evaluate reconstruction, identity-expression decomposition, and generalization, then examine augmentation and applications. The method outperforms comparison methods in the reported decomposition, extrapolation, augmentation, and expression-transfer results.

4.1. Implementation Details

  • FaceWareHouse ground-truth identity meshes use neutral expressions, while expression meshes average the same expression across 140 subjects; networks train for 50 epochs per step.The same ground-truth preparation procedure can be applied to other 3D face datasets.Source: §4.1 · p. 5

4.2. Evaluation Metric

  • Evaluation covers reconstruction accuracy and disentanglement: it uses average vertex distance and spatial edge difference, plus variation among reconstructed identity or expression meshes.The FaceWareHouse experiments train on 140 identities and test on the remaining 10.Source: §4.2 · p. 5, §4.2.1 · p. 5, §4.2.2 · p. 6

4.3.1 Baseline Comparison

  • The framework outperforms the bilinear model, FLAME, and MeshAE on each reported evaluation and achieves better reconstruction accuracy while more neatly decoupling identity and expression.Latent dimensions for identity and expression were 50 and 25, matching the bilinear model and FLAME.Source: §4.3.1 · p. 6, Table 1 · p. 6

4.3.2 Ablation Study

  • Both deformation representation and fusion improve ablation performance; deformation representation particularly lowers average vertex distance, while fusion lowers expression error but slightly raises spatial edge difference error.Across all evaluation metrics, the full framework is reported to perform better than the other tested variants.Source: Table 1 · p. 6, §4.3.2 · p. 7

4.4. Experiment on COMA Dataset [31]

  • On COMA expression extrapolation, the method generalizes better than FLAME across 12 cross-validation experiments, measured by average vertex distance.The dataset lacks expression labels, so 12 representative expressions were manually selected for supervision.Source: §4.4 · p. 7, Table 2 · p. 7

4.5. Discussion on Larger Dataset

  • DR augmentation of FaceWareHouse yields the best scores on all reported aspects among the three training datasets, indicating improved stability and robustness.The comparison includes original FaceWareHouse, combined FaceWareHouse and Bosphorus, and DR-augmented FaceWareHouse.Source: §4.5 · p. 7, Table 3 · p. 8

5.1. Expression Transfer

  • Expression transfer is reported as more natural and accurate than the bilinear model, with results closer to ground truth in quantitative error evaluations.The method transfers the source face’s latent expression code directly to the target face.Source: §5.1 · p. 8, Figure 5 · p. 8

5.2. Latent space interpolation

  • Identity and expression codes can be interpolated separately in 0.25 steps, producing results described as meaningful and reasonable.The interpolated codes are passed through the trained decoder to recover models.Source: §5.2 · p. 8, Figure 4 · p. 8

6. Conclusion

The paper presents a method that decomposes 3D face shapes into identity and expression parts. Its experiments support the method’s effectiveness, and expression transfer produces more natural and accurate results than traditional methods.

  • The method accurately decomposes a 3D face shape into identity and expression parts.Source: §6 · p. 8
  • The framework trains decomposition and fusion networks and represents face shapes with vertex-based deformations rather than Euclidean coordinates.Source: §6 · p. 8
  • Ablation studies and quantitative and qualitative experiments demonstrate the method’s effectiveness.Source: §6 · p. 8
  • Expression transfer using the representation yields more natural and accurate results than traditional methods.Source: §6 · p. 8

Supplementary

The network uses spectral graph convolution layers with Chebyshev polynomials of order 2. During training, duplicated identity and expression branches provide the disentangling loss.

A. Network Structure

B. Latent space dimension exploration

The latent-dimension exploration compares the method with baseline models and tests different latent-space sizes. The deformation representation’s conversion to a mesh has a reported average error of 31 micrometers.

B. Latent space dimension exploration

  • The latent-space exploration compares the method with baseline models and evaluates different latent-space sizes on FaceWareHouse.Table 4 labels the reported configurations by identity and expression dimensions; values are in 0.1 millimeters.Source: B. Latent space dimension exploration · p. 11, Table 4 · p. 11
  • The reported latent-space configurations include Ours(25/10), Ours(75/50), and Ours(50/25), with identity and expression dimensions listed in that order.Source: Table 4 · p. 11

C. Deformation Representation Reconstruction Accuracy

  • The deformation-representation conversion has an average geometric error of 31 micrometers over FaceWarehouse point clouds.The paper reports that this conversion has very little influence on reconstruction accuracy.Source: C. Deformation Representation… · p. 11

D. Data Augmentation Samples

The paper augments FaceWareHouse training data with 2,000 neutral-expression meshes for the identity decomposition branch. Fig. 7 shows examples of the augmented models.

E.1. Selected Expressions from COMA Dataset

The authors selected 144 COMA expressions for decomposition and fusion network pretraining, with each column showing one identity across 12 expressions.

E.1. Selected Expressions from COMA Dataset

  • 144 selected COMA expressions support pretraining of the decomposition and fusion networks.Each column contains 12 expressions from the same identity.Source: §E.1 · p. 11

E.2. 12 Cross Validation Experiment Result

In 12 cross-validation experiments on COMA expression extrapolation, the method has lower error than FLAME in most cases, though bareteeth has higher median error. Manual expression-selection bias accounts for most of the method’s error.

E.2. 12 Cross Validation Experiment Result

  • Manual expression-selection bias accounts for most of our method’s error.Source: §E.2 · p. 11
  • The comparison reports results from 12 cross-validation experiments on COMA.Source: §E.2 · p. 11, Table 5 · p. 15
  • Our method has lower error than FLAME in most COMA cross-validation cases, but higher median error for bareteeth.Table 5 compares expression-extrapolation errors in millimeters.Source: §E.2 · p. 11, Table 5 · p. 15

Related papers

Loading 1902.09887v2…