PaperlayerView exact revision on arXiv

Source-linked AI summary

One-Stage 3D Whole-Body Mesh Recovery with Component Aware Transformer

Jing Lin, Ailing Zeng, Haoqian Wang, Lei Zhang, Yu Li

arXiv:2303.16160v1cs.CV

TL;DR

Whole-body mesh recovery must estimate body, hands, and face jointly despite small component regions and difficult connections. OSX addresses this with a one-stage Component Aware Transformer, while UBody broadens evaluation to diverse upper-body scenes; the paper reports state-of-the-art performance across three benchmarks.

  • Problem

    Whole-body recovery must estimate body, hands, and face jointly, but small component regions and separate multi-stage processing make accurate, naturally connected prediction difficult.

  • Method

    OSX uses a one-stage Component Aware Transformer with a global body encoder and local decoder for high-resolution hand and face estimation.

  • Results

    OSX achieves new state-of-the-art performance on three popular benchmarks, including 9.5% improvement on AGORA and 7.8% on EHF over SOTA.

  • Takeaways & Limitations

    UBody provides annotated, diverse upper-body scenes intended to bridge basic whole-body estimation and downstream applications.

  • Takeaways & Limitations

    The training does not use additional hand- and face-specific datasets.

Abstract

from arXiv · show

Whole-body mesh recovery aims to estimate the 3D human body, face, and hands parameters from a single image. It is challenging to perform this task with a single network due to resolution issues, i.e., the face and hands are usually located in extremely small regions. Existing works usually detect hands and faces, enlarge their resolution to feed in a specific network to predict the parameter, and finally fuse the results. While this copy-paste pipeline can capture the fine-grained details of the face and hands, the connections between different parts cannot be easily recovered in late fusion, leading to implausible 3D rotation and unnatural pose. In this work, we propose a one-stage pipeline for expressive whole-body mesh recovery, named OSX, without separate networks for each part. Specifically, we design a Component Aware Transformer (CAT) composed of a global body encoder and a local face/hand decoder. The encoder predicts the body parameters and provides a high-quality feature map for the decoder, which performs a feature-level upsample-crop scheme to extract high-resolution part-specific features and adopt keypoint-guided deformable attention to estimate hand and face precisely. The whole pipeline is simple yet effective without any manual post-processing and naturally avoids implausible prediction. Comprehensive experiments demonstrate the effectiveness of OSX. Lastly, we build a large-scale Upper-Body dataset (UBody) with high-quality 2D and 3D whole-body annotations. It contains persons with partially visible bodies in diverse real-life scenarios to bridge the gap between the basic task and downstream applications.

1 International Digital Economy Academy (IDEA), 2 Shenzhen International Graduate School, Tsinghua University https://osx-ubody.github.io

Figure 1 contrasts multi-stage whole-body recovery pipelines using separate expert models with OSX’s one-stage encoder-decoder framework.

  • Existing methods process body components with separate expert models and fuse their predictions in a copy-paste pipeline.The illustrated experts include HeadNet, HandNet, and BodyNet.
  • The compared prior result produces unnatural wrist poses, whereas OSX predicts natural meshes with a single encoder-decoder.

1. Introduction

The introduction frames expressive whole-body mesh recovery as difficult because it must estimate body, hand, and face components while preserving natural connections. OSX addresses this with a one-stage component-aware Transformer, and UBody extends evaluation to expressive upper-body scenarios.

  • Whole-body mesh recovery jointly estimates 3D body poses, hand gestures, and facial expressions from monocular images.The task supports applications including motion capture and human-computer interaction.
  • Accurate recovery is challenging because each component must be estimated precisely while maintaining natural connections between body parts.
  • Existing multi-stage systems detect, crop, resize, and separately estimate body, hand, and face regions, creating computational complexity and incompatible configurations.
  • OSX uses a global body encoder and local component decoder with differentiable upsample-crop features and keypoint-guided deformable attention.It predicts body parameters and supplies high-quality features for precise hand and face estimation without manual post-processing.
  • UBody contains fifteen real-life human-centric scenes with unseen poses, diverse appearances, truncation, interaction, and abrupt shot changes, plus 2D keypoint and 3D mesh annotations.
  • The paper reports that OSX achieves new state-of-the-art performance on three popular benchmarks and introduces UBody for downstream-oriented benchmarking.

2. Related Work

Prior work includes individual-component reconstruction, optimization-based fitting, multi-stage whole-body systems, and datasets with varying realism and scene coverage. These approaches leave unresolved complexity, articulation consistency, and generalization challenges in expressive real-world settings.

  • Earlier research primarily reconstructs body, face, or hands individually, while joint whole-body estimation remains less addressed.
  • Optimization-based methods fit 3D bodies to detected 2D keypoints with additional constraints but are slow and prone to local optima.
  • UBody is presented with fifteen real-life scenes to address missing downstream-task scenarios and expressive upper-body coverage.
  • Multi-stage whole-body methods use separate component networks and fusion modules, increasing complexity and causing inconsistent articulation and implausible wrist rotations.
  • Existing benchmarks include EHF, an SMPL-X evaluation dataset, and AGORA, a realistic synthetic dataset with diverse subjects, environments, clothing, and occlusions.AGORA often places people far from the camera, making hands and faces small or obscured.
  • FBA emphasizes consumer-video failures under unusual viewpoints and aggressive truncation, while Multi-shot-AVA highlights edited media with rich appearances, interactions, and temporal contexts.

3. Method

OSX replaces separate component-specific backbones with a one-stage Component-Aware Transformer that jointly predicts whole-body parameters. Its decoder uses feature-level upsample-crop and keypoint-guided deformable attention to recover fine-scale hands and face.

  • Component-Aware Transformer: OSX uses a one-stage vision-transformer framework with shared body, hand, and face processing instead of separate networks.The framework is designed as a simpler alternative to multi-stage pipelines and predicts full-body mesh parameters.
  • End-to-end prediction: OSX estimates body, hand, and face parameters that are passed to an SMPL-X layer to produce the final 3D whole-body mesh.The predicted parameters include body pose, shape, translation, hand joint rotations, jaw rotation, and facial expression.
  • Global body encoder: The encoder processes image patches with learnable body tokens and global Transformer fusion before regressing body parameters.Image patches are projected into features, combined with body tokens, and processed by Transformer blocks.
  • High-resolution component decoder: The decoder upsamples feature maps and applies differentiable RoIAlign to crop multi-scale hand and face features from predicted bounding boxes.This feature-level strategy avoids requiring separate image backbones while providing higher-resolution component features.
  • High-resolution component decoder: Keypoint-guided component tokens drive deformable attention that samples a small set of locations around reference keypoints for hand and face regression.The decoder uses multi-scale cropped features as memory and outputs left-hand, right-hand, and face parameters.
  • Training objective: The model is trained end-to-end with L1 losses for SMPL-X parameters, 3D and projected 2D keypoints, and hand-face bounding boxes.The combined objective is L = L_smplx + L_kpt3D + L_kpt2D + L_bbox2D.

4. UBody–An Upper Body Dataset

UBody is a large-scale upper-body dataset designed for expressive gestures, facial expressions, and diverse real-life scenarios that challenge existing methods. Its annotation pipeline produces accurate 2D and 3D supervision, while its data distributions emphasize varied hand and face visibility.

  • Motivation: UBody targets upper-body scenes with expressive gestures and facial expressions that existing datasets and methods underrepresent.The dataset addresses scenarios including sign language, gesture generation, emotion recognition, and real-life VLOGs.
  • Annotation quality: Its annotation pipeline produces more accurate 2D hand-detail and occlusion-robust annotations than OpenPose and MediaPipe.The authors also report better-looking whole-body fits than NeuralAnnot on COCO.
  • Data characteristics: UBody introduces diverse scenes with shot changes, camera motion, truncation, occlusion, interactions, lighting variation, and rich gestures.These conditions are described as challenges not appearing in previous datasets.
  • Data characteristics: Figure 6 compares hand and face bounding-box areas in its upper row and visible 2D hand and face keypoint counts in its lower row.Both comparisons use a logarithmic Y-axis.

5. Experiment

Experiments evaluate OSX against existing methods on established benchmarks and UBody, while ablations examine component-aware decoding. OSX achieves strong benchmark performance and UBody training improves performance on upper-body scenes.

  • Comparisons with Existing Methods: OSX improves All MPVPE by 9.5% on AGORA and 7.8% on EHF relative to the cited state-of-the-art method.The authors also report a 13.4% error reduction on 3DPW compared with whole-body methods.
  • Comparisons with Existing Methods: OSX is reported as the first one-stage method to surpass existing multi-stage models in most cases without hand-only or face-only training data.The comparison uses the datasets and metrics described in the experiment setup.
  • Ablation study: The component-aware decoder is evaluated for hand and face regression because body posture and hand or face details occupy different spatial scales.The ablation removes decoder components and reports performance changes on EHF.
  • Benchmark on UBody: UBody evaluation follows an intra-scene protocol, with models pretrained on previous datasets unless marked as fine-tuned.The table caption distinguishes UBody and AGORA fine-tuning settings.
  • Benchmark on UBody: Training OSX on UBody improves performance by 16.1% over the original pretrained model on downstream real-life scenes.The authors relate this result to the data distribution represented by UBody.

6. Conclusion

The paper presents OSX as a simple one-stage method with state-of-the-art performance and introduces UBody to connect whole-body estimation with upper-body downstream settings. The authors identify additional hand- and face-specific training data and downstream validation as future directions.

  • Contributions: OSX is presented as a one-stage pipeline achieving state-of-the-art performance on three benchmarks.The conclusion characterizes the framework as simple yet effective.
  • Contributions: UBody is introduced as a large-scale dataset with comprehensive daily-life scenes for downstream upper-body applications.The dataset is intended to bridge basic full-body pose and shape estimation with downstream tasks.
  • Limitations and future work: The authors propose studying how to use additional hand- and face-specific datasets within the OSX pipeline.They also identify validation on applications such as gesture recognition and driving avatars as future work.

Overview

The supplementary material collects additional experimental details, efficiency comparisons, dataset analyses, inter-scene evaluation, and qualitative comparisons. It also reports that OSX has the shortest inference time and lowest error in its efficiency comparison.

  • Supplementary contents: The supplement adds experiment setup details, efficiency comparisons, AGORA experiments, UBody introduction, inter-scene benchmarking, and qualitative comparisons.These materials are organized across Sections A through F.
  • Evaluation: The evaluation uses MPVPE, PA-MPVPE, MPJPE, PA-MPJPE, NMVE, NMJE, and F1 score for relevant settings.NMVE and NMJE are additionally reported on AGORA test under multi-person evaluation.
  • Efficiency: OSX has the shortest inference time and lowest error in the reported single-person efficiency comparison on AGORA-test.The comparison also reports model parameters, FLOPs, and NMJE-All on the same input resolution and hardware.

C. Experiment on AGORA Dataset

On AGORA, OSX outperforms competitors on most metrics, with especially strong full-body and multi-person recovery while remaining competitive for hands and face.

  • 10.6 mm, 9.1 mm, 2.9 mm, and 4.7 mm separate OSX from PyMAF-X on full-body NMVE, NMJE, MVE, and MPJPE, respectively.These are the reported AGORA test-set improvements for the four metrics.
  • OSX outperforms other competitors on most AGORA metrics, especially for body and full-body recovery.
  • OSX achieves overall better multi-person estimation under NMVE and NMJE because these metrics account for missed detections and false positives.The comparison notes that matched-person metrics can obscure differences when detected-person ratios vary.
  • Without extra hand-only or face-only datasets, OSX remains competitive on AGORA hand and face metrics.The result is attributed to the effectiveness of the component-aware decoder.

D.1. Data Collection

UBody is a large-scale upper-body dataset built from diverse real-life video scenes, with high-precision 2D and 3D whole-body annotations and intra-scene and inter-scene benchmarks.

  • Data Collection: UBody contains fifteen real-life scenarios and more than 105,1k frames, covering diverse upper-body applications and challenging visual conditions.The scenes include unseen poses, diverse appearances, heavy truncation, interaction, and abrupt shot changes.
  • Benchmark Protocols: UBody supports intra-scene and inter-scene splits, with the latter training on ten scenes and testing on five different scenes.The intra-scene protocol uses the first 70% of each scene's videos for training and the final 30% for testing.
  • Annotation: The annotation pipeline has two stages: 2D whole-body keypoint annotation followed by 3D SMPL-X annotation fitting.Annotations are performed on individual frames because unpredictable transitions and cutscenes make temporal smoothing difficult.
  • Annotation: OSX provides a proper 3D initialization and pseudo-3D constraints for UBody annotation, while annotated 2D keypoints supervise projected whole-body keypoints.
  • Benchmark Results: Inter-scene testing produces larger errors than intra-scene testing, while UBody fine-tuning consistently improves performance over original pretrained models.The comparison attributes the gap to differing motion and gesture distributions and positions UBody as a bridge to real-life downstream scenes.
  • Benchmark Results: UBody provides video data unlike single-frame AGORA and EHF, enabling research on spatial-temporal modeling for edited media sources.
Loading 2303.16160v1…