PaperlayerView exact revision on arXiv

Source-linked AI summary

MobRecon: Mobile-Friendly Hand Mesh Reconstruction from Monocular Image

Xingyu Chen, Yufeng Liu, Yajiao Dong, Xiong Zhang, Chongyang Ma, Yanmin Xiong, Yuan Zhang, Xiaoyan Guo

arXiv:2112.02753v2cs.CV

TL;DR

Single-view hand mesh reconstruction must balance accuracy with efficient inference and temporal coherence, especially for mobile use. MobRecon combines lightweight encoding, feature lifting, and efficient mesh decoding; the authors report state-of-the-art performance on three datasets and 83FPS on Apple A14 CPU.

  • Problem

    Mobile hand-mesh reconstruction calls for accuracy, inference efficiency, and temporal consistency despite limited device memory and computational budgets.

    Source: §1 · p. 1

  • Method

    MobRecon combines lightweight stacked 2D encoding, MapReg with pose pooling and pose-to-vertex lifting, and depth-separable spiral convolution for 3D decoding.

    Source: §1 · p. 2

  • Results

    MobRecon achieves state-of-the-art performance on FreiHAND, RHD, and HO3Dv2, with 83FPS on Apple A14 CPU.

    Source: §5 · p. 8

  • Takeaways & Limitations

    The framework offers a compact mobile-oriented pipeline, using 123M Mult-Adds and 5M parameters while targeting accuracy, efficiency, and temporal coherence.

    Source: §1 · p. 2, §5 · p. 8

    The complement dataset lacks shape and texture diversity and models finger bend but not finger splay.

    Source: §6 · p. 13

Abstract

from arXiv · show

In this work, we propose a framework for single-view hand mesh reconstruction, which can simultaneously achieve high reconstruction accuracy, fast inference speed, and temporal coherence. Specifically, for 2D encoding, we propose lightweight yet effective stacked structures. Regarding 3D decoding, we provide an efficient graph operator, namely depth-separable spiral convolution. Moreover, we present a novel feature lifting module for bridging the gap between 2D and 3D representations. This module begins with a map-based position regression (MapReg) block to integrate the merits of both heatmap encoding and position regression paradigms for improved 2D accuracy and temporal coherence. Furthermore, MapReg is followed by pose pooling and pose-to-vertex lifting approaches, which transform 2D pose encodings to semantic features of 3D vertices. Overall, our hand reconstruction framework, called MobRecon, comprises affordable computational costs and miniature model size, which reaches a high inference speed of 83FPS on Apple A14 CPU. Extensive experiments on popular datasets such as FreiHAND, RHD, and HO3Dv2 demonstrate that our MobRecon achieves superior performance on reconstruction accuracy and temporal coherence. Our code is publicly available at https://github.com/SeanChenxy/HandMesh.

1. Introduction

MobRecon targets single-view hand mesh reconstruction for mobile settings, where accuracy must be balanced with inference efficiency and temporal consistency. It combines lightweight encoding, feature lifting, and efficient mesh decoding, and reports strong performance across these dimensions.

1. Introduction

  • Typical methods prioritize reconstruction accuracy, but mobile applications also need efficient inference and temporal consistency under limited memory and computational budgets.Existing 2D encoders can be computationally intensive, while untailored mobile networks can reduce accuracy; efficient mapping and decoding are also relatively unexplored.Source: §1 · p. 1
  • Prior temporal methods use sequential models to combine past and future information, which can require offline processing or substantial computation and hinder mobile use.MobRecon instead explores temporal coherence with a non-sequential method.Source: §1 · pp. 1–2
  • MobRecon uses lightweight stacked structures for 2D encoding, DSConv for 3D decoding, and a feature-lifting module with MapReg, pose pooling, and PVL for 2D-to-3D mapping.MapReg integrates heatmap and position-regression representations; PVL maps 2D pose encodings to 3D vertex features.Source: §1 · p. 2
  • The paper reports a compact pipeline with 123M Mult-Adds and 5M parameters that reaches 83FPS on Apple A14 CPU.It also introduces a synthetic dataset with uniformly distributed hand poses and viewpoints.Source: §1 · p. 2
  • MobRecon reports better accuracy, inference speed, and model size than compared methods, with Figure 1 showing the accuracy–speed trade-off and marker size indicating model size.The authors also report 83FPS on Apple A14 CPU with 123M Mult-Adds and 5M parameters.Source: Figure 1 · p. 1, §1 · p. 2

2. Related Work

Related work spans several hand-mesh representations and lightweight reconstruction networks. MobRecon builds on vertex-based pipelines, pixel-aligned feature lifting, and non-sequential approaches to temporal coherence.

2. Related Work

  • Hand-mesh methods are grouped by parametric models, voxels, implicit functions, UV maps, and direct vertex prediction.The cited examples describe memory demands for voxel methods, computation of many 3D points for implicit methods, and geometry-correlation limits for UV maps.Source: §2 · p. 2
  • Lightweight networks target computationally limited platforms, and MobRecon applies efficient ideas to both 2D images and 3D meshes.The related-work passage reports that MobileHand runs at 75FPS on a mobile CPU and says MobRecon has stronger accuracy and inference speed.Source: §2 · p. 2
  • Earlier temporal reconstruction work uses sequential approaches that may raise computational costs or require future information; MobRecon instead uses MapReg in a non-sequential single-view method.The paper presents its feature-lifting module as a way to enhance temporal coherence.Source: §2 · pp. 2–3
  • Pixel-aligned representations connect dense 2D image features with sparse 3D data; MobRecon uses pose-aligned encodings to form vertex features.Prior methods use heatmaps or positions, while MapReg combines their merits from the perspective of temporal coherence and may complement RLE.Source: §2 · p. 3

3. Our Method

MobRecon maps a single-view image to a 3D hand mesh through 2D encoding, feature lifting, and 3D decoding. Its method uses lightweight encoding, pose-aligned feature lifting, and an efficient graph operator.

3. Our Method

  • MobRecon encodes the image, converts 2D pose into pose-aligned features, and decodes 3D vertex coordinates through a three-phase pipeline.Feature lifting uses MapReg, pose pooling, and pose-to-vertex lifting.Source: §3 · p. 3

3.1. Stacked Encoding Network

  • At 128 × 128, DenseStack uses 373.0M Mult-Adds and 6.6M parameters, while GhostStack uses 96.2M Mult-Adds and 5.0M parameters.The ResNet18 stacked network uses 2391.3M Mult-Adds and 25.2M parameters.Source: §3.1 · p. 3

3.2. Feature Lifting Module

  • MapReg fuses low- and high-resolution features for accuracy while using a global receptive field intended to improve temporal coherence.It predicts 2D positions from a spatially small feature map through an MLP.Source: §3.2 · p. 4
  • Pose pooling retrieves joint-aligned image features, and PVL maps them to 49 minimal-mesh vertices with complexity O(NVminiCe).Heatmaps use joint-wise pooling, while 2D positions can use grid sampling; PVL reduces mapping complexity from O(VminiCe^2) to O(NVminiCe).Source: §3.2 · p. 4

3.3. Depth-Separable SpiralConv

  • DSConv separates graph convolution into depth-wise neighbor aggregation and point-wise mixing, reducing complexity from O(SD2) to O(SD + D2).The decoder has four upsampling blocks followed by a final DSConv that predicts vertex coordinates.Source: §3.3 · p. 5

3.4. Loss Functions

  • The accuracy losses use L1 terms for the 3D mesh and 2D pose, with normal and edge-length losses for mesh smoothness.The total loss also includes 2D and 3D consistency terms.Source: §3.4 · p. 5
  • Consistency supervision compares predictions from two affine- and color-jittered views in both 3D and 2D, without temporal data.The relative affine transformation includes rotation, and only rotation affects the mesh because it is rooted at the wrist landmark.Source: §3.4 · p. 5

4. Experiments

Experiments evaluate MobRecon on FreiHAND, RHD, and HO3Dv2, examining its accuracy, efficiency, and temporal coherence. Ablations test its encoding, feature-lifting, consistency-learning, and decoding choices.

4.1. Implementation Details

  • The network is trained with Adam for 38 epochs at 128 × 128; its synthetic complement dataset contains 1,520 uniformly distributed poses and 216 uniformly distributed viewpoints.FreiHAND evaluation annotations are unavailable, so predictions are submitted to the official server for online evaluation.Source: §4.1 · p. 5, §4.2 · p. 5

4.2. Evaluation Criterion

  • Evaluation reports joint and vertex position errors, Procrustes-aligned errors, landmark acceleration, PCK AUC, and mesh F-scores; Mult-Adds and parameter counts track efficiency.PCK AUC is computed over error thresholds, and F@5/F@15 use mesh-distance thresholds of 5mm/15mm.Source: §4.2 · p. 6

4.3. Ablation studies

  • 7.55/8.89mm PAMPJPE for DenseStack/GhostStack with complement data; generic MobileNet degrades PAMPJPE, while ImageNet pre-training improves it by less than 1mm.The tailored structures and complement data reduce computation without sacrificing reconstruction accuracy.Source: §4.3 · pp. 6–7, Table 1 · p. 6
  • MapReg integrates heatmap and regression merits; its adaptive maps encode inter-landmark constraints, and MapReg-based Lp yields the best PAMPJPE and 3D Acc.The feature-lifting ablation evaluates representation choices without a sequential module, temporal optimization, or post-processing.Source: §4.3 · p. 7, Table 2 · p. 7
  • 67/83 FPS on Apple A14 CPU for DenseStack/GhostStack; DSConv reduces 3D-decoder computation and parameters while matching or sometimes exceeding SpiralConv++ performance.Table 4 reports decoder and overall model costs alongside accuracy, temporal performance, and speed.Source: Table 4 · p. 7, §4.3 · p. 8
  • PVL learns a sparse landmark-to-vertex lifting matrix, and the feature-lifting module improves PAMPJPE and 3D Acc while reducing 2D-to-3D computation versus CMR.A root-of-forefinger landmark contributes to most vertices, while other landmark traits propagate to corresponding vertices.Source: Figure 7 · p. 6, Figure 8 · p. 6, §4.3 · p. 7
  • Consistency learning improves temporal coherence and PAMPJPE, which the ablation describes as benefiting from each other.Table 3 reports accuracy on FreiHAND and acceleration on HO3Dv2.Source: §4.3 · p. 7, Table 3 · p. 7

4.4. Comparisons with Contemporary Methods

  • 5.7mm PAMPJPE on FreiHAND; MobRecon also reaches 3D AUC 0.955/0.940 on RHD and outperforms existing HO3Dv2 methods with better temporal coherence.The comparison reports superior 3D PCK on FreiHAND and inference speed relative to prior methods.Source: Table 5 · p. 8, Table 6 · p. 8, §4.4 · p. 8

5. Conclusions and Future Work

MobRecon combines lightweight 2D encoding, feature lifting, and DSConv for efficient hand mesh reconstruction. The paper reports state-of-the-art performance on FreiHAND, RHD, and HO3Dv2, and plans to investigate efficient methods for interacting hands.

5. Conclusions and Future Work

  • MobRecon combines lightweight stacked 2D encoding, MapReg-based feature lifting, and DSConv for efficient hand mesh reconstruction, reaching 83FPS on Apple A14 CPU and state-of-the-art performance on FreiHAND, RHD, and HO3Dv2.The model uses 123M Mult-Adds and 5M parameters; future work will investigate efficient methods for interacting hands.Source: §5 · p. 8

6. Complement Dataset

MobRecon’s complement dataset addresses the lack of hand data with uniformly distributed poses and viewpoints. It combines synthetic renders, broad annotations, and pose-estimation pre-training, while remaining limited in shape, texture, and finger-splay coverage.

6. Complement Dataset

  • Existing datasets lack uniformly distributed poses and viewpoints; model-fitted real data can be noisy and costly, while synthetic data offers scalability and lower collection cost.Source: §6 · p. 12
  • Thirty-two base poses combine five fingers’ bent-or-extended states, and three interpolated poses for each of 496 pose pairs yield 1,520 poses.Source: §6 · p. 12
  • The synthetic dataset contains 1,520 uniformly distributed hand poses and 216 uniformly distributed viewpoints.Source: §6 · p. 12
  • Views are rendered from 216 hemispherical camera positions aimed at the palm center, with HDR real-scene backgrounds and illumination.Source: §6 · pp. 12–13
  • Annotations are automatically generated without noise and include dense and MANO-style meshes, 2D/3D poses, silhouettes, depth maps, and camera intrinsics.Source: §6 · p. 13
  • The dataset lacks shape and texture diversity and models finger bending but not splay, which is planned for future coverage.Source: §6 · p. 13
  • The 2D encoder is pre-trained without 3D annotations using supervision for both heatmaps and positional joint landmarks.Source: §6 · p. 13

7. Analysis and Application

The analysis reports temporal-consistency and pose-accuracy advantages, including comparisons on a static-hand clip and qualitative tests under challenging conditions. It also describes failure cases and a mobile AR application.

7. Analysis and Application

  • The consistency loss compares augmented views in both 2D and 3D; RHD evaluates 2D accuracy, FreiHAND 3D accuracy, and HO3Dv2 temporal coherence.Source: §7 · p. 13
  • Fine-tuning with the complement data improves performance on difficult-pose prediction.Source: §7 · p. 13
  • On heavily occluded HO3Dv2 samples, MobRecon can produce physically correct predictions where a ResNet-based model collapses.Source: §7 · p. 13, Figure 17 · p. 14
  • Self-occlusion from finger splay remains a failure case because such poses are tail-distributed in most datasets.Source: §7 · p. 14, Figure 18 · p. 14
  • Qualitative results show accurate 2D poses and 3D meshes under challenging poses, object occlusion, truncation, and bad illumination across FreiHAND, RHD, and HO3Dv2.Source: §7 · p. 14, Figure 21 · p. 16
  • On a static-hand clip, MobRecon exceeds CMR in 2D/3D pose consistency and root-recovery stability; RHD PCK curves also show better 2D pose accuracy.Source: §7 · p. 14, Figure 22 · p. 17
  • MobRecon is used to place a virtual ring on a hand with an AR effect deployed on mobile devices.Source: §7 · p. 14

Related papers

Loading 2112.02753v2…