PHOSA: Photorealistic 3D Sign Avatar Modeling and Benchmark

• A hybrid SMPL-X fitting pipeline integrates predictions from multiple models to recover precise hand articulation and nuanced facial motion.The pipeline can also serve as a plug-in in monocular RGB settings.Source: §1 · p. 2
• The method achieves state-of-the-art performance on MVSign and in-the-wild sign videos.Source: §1 · p. 2
• The avatar representation separates body, head, and hands, using partial kinematic decoupling to improve gesture generalization and fine-grained articulation fidelity.Source: §1 · pp. 2–3
• Motion-aware frame sampling filters motion-blurred frames and balances motion types to enhance visual fidelity and training stability.Source: §1 · p. 2
• MVSign establishes a multi-view benchmark and is the first Chinese sign-language dataset co-designed with Deaf experts, featuring diverse gestures and rich annotations.Source: §1 · p. 2

2.1 Animatable Human Avatar Modeling

• Animatable-avatar work moves from implicit neural representations toward 3D Gaussian Splatting, with methods using moving Gaussians, SMPL-X-aware modules, or canonical Gaussian maps.Source: §2.1 · p. 3
• Unlike methods focused on general human motion, the proposed representation targets the fine-grained hand and facial details needed for photorealistic sign avatars.Source: §2.1 · p. 3

2.2 Human-centric and Sign Language Benchmarks

• Multi-view human datasets focus on general motion, while sign-language datasets address recognition, translation, or motion production rather than avatar reconstruction.Source: §2.2 · p. 3
• MVSign is not publicly available.Source: §2.2 · p. 4
• SignAvatars adds large-scale 3D motion annotations, but sign datasets generally lack synchronized multi-view imagery and annotations for photorealistic avatar reconstruction; MVSign targets this gap.Source: §2.2 · p. 4

3.1 Data Capture

• Each signer contributes 109 basic hand shapes and around 50 sentences selected from five of 20 daily-life scenarios.The hand shapes comprise 26 letters, 22 numbers, and 61 commonly used words.Source: §3.1 · pp. 4–5
• The capture setup uses 16 synchronized 2048×2448 RGB cameras at 25 fps, including ten lateral views and five head-focused views.The remaining camera captures the frontal view.Source: §3.1 · p. 5

3.2 Data Annotation

• MVSign provides matting, body-part segmentation, 3D keypoints, and SMPL-X parameters, with SAM refining segmentation produced using Sapiens.The dataset comprises five native Chinese Sign Language signers and around 23,000 frames; it uniquely combines multi-view sign data, dedicated head views, and SMPL-X annotations among compared datasets.Source: §3.2 · p. 5, §3.1 · p. 5
• The hybrid fitting pipeline uses DWPose detections, replaces its hand estimates with HaMeR hand parameters, and triangulates optimized keypoints across calibrated views.Multi-view geometric constraints produce 3D keypoints from the 2D detections.Source: §3.2 · p. 6
• SMPL-X fitting minimizes a robust weighted keypoint objective with shape and expression regularizers, then applies temporal initialization and SmoothNet to reduce sequence artifacts.The robust kernel helps limit noisy-supervision disturbance; the pipeline also replaces initial fitted facial parameters with INFERNO expression parameters.Source: §3.2 · pp. 6–7
• Sparse face keypoints leave fitted facial parameters often unable to represent the true expression accurately.The pipeline addresses this by replacing the initial facial parameters with INFERNO-derived expression parameters.Source: §3.2 · pp. 6–7

4 Sign Avatar Modeling

• The approach addresses motion blur and gesture imbalance through motion-aware sampling, alongside a decoupled representation for complex hand articulation.The paper presents these data and modeling choices as means to improve visual quality.Source: §4 · p. 7, §4.1 · p. 7

4.1 Motion-aware Data Sampling Strategy

• Blur detection thresholds hand confidence, hand-image Laplacian gradient, and hand-motion velocity estimated from SMPL-X trajectories.Frames failing the indicator criteria are filtered before balancing gesture categories.Source: §4.1 · p. 7
• After filtering, gestures are grouped by distance between their SMPL-X parameters, and excessive frames are removed from each category to balance representation..Source: §4.1 · p. 7

4.2 Decoupled Sign Avatar Representation

• The representation separates whole-body Gaussians into body, head, and hand components, with body and head position maps rendered from front and back views.Body and head are deformed using LBS; the predicted posed coordinates are rendered on the canonical SMPL-X template.Source: §4.2 · p. 8
• Hand maps use partial kinematic decoupling: body joints stay fixed while hand joints remain mobile, making hand pose maps independent of body pose.The two hands are rendered to up and down views and concatenated.Source: §4.2 · pp. 8–9

4.3 Optimization

• Three specialized StyleUNet branches predict Gaussian attributes, while the composite objective combines photometric, hand/head-region, and position-offset regularization losses.The model predicts position offsets relative to the SMPL-X mesh; hand and head L1 losses encourage focus on those regions, and the offset penalty limits deviation from the template.Source: §4.3 · p. 9

5.2 Dataset Analysis

• Increasing the number of MVSign views improves visual quality, especially fine-grained hand detail, by helping resolve self-occlusions and depth ambiguities.The reported comparison is summarized in Table 4 and visualized in Figure 4.Source: §5.2 · p. 10, Figure 4 · p. 11, Table 4 · p. 12

5.3 Ablation Study

• On the SGNify mocap dataset, hybrid SMPL-X fitting achieves the lowest upper-body and hand vertex errors, and its fitted poses retain fine-grained hand details in avatar modeling.Compared with fitting from DWPose 2D keypoints, the hybrid pipeline provides more accurate hand poses.Source: Table 3 · p. 11, §5.3 · p. 11, Table 2 · p. 11

5.2 Dataset Analysis

• Combining basic hand shapes with sign sentences produces a more robust and realistic avatar than training on either pattern subset alone.Basic-shape-only training limits generalization, while sentence-only training makes learning more challenging and degrades quality.Source: §5.2 · p. 11, Table 5 · p. 12

5.3 Ablation Study

• Motion-aware sampling improves visual quality by filtering invalid, motion-blurred frames that can degrade fine-grained hand detail when included in training.The strategy is compared with sequential, isometric, and random sampling in Table 6 and Figure 5.Source: §5.3 · p. 12, Figure 5 · p. 12, Table 6 · p. 13
• Partial hand kinematic decoupling significantly improves hand quality, whereas decoupling the body, head, and hands alone gives only minor gains.The ablation compares the decoupled representation with and without partial hand kinematic decoupling.Source: §5.3 · p. 12, Figure 6 · p. 12, Table 7 · p. 13

5.4 Comparison with State-of-the-art Methods

• In two Deaf-community studies, participants rated PHOSA highest across four criteria versus avatar baselines and preferred it to SMPL-X, with 84% favoring it aesthetically.The four criteria include comprehensibility, hand/facial clarity, visual realism or temporal stability, and aesthetic preference.Source: Table 9 · p. 14, §5.4 · p. 14, Table 8 · p. 14
• Our method outperforms baselines across all evaluation metrics, with relative LPIPS gains of 8.1%, 4.0%, and 5.9% on full, hand, and face regions in MVSign, and 15.8%, 11.6%, and 12.6% respectively on in-the-wild sign videos.Source: §5.4 · p. 13

6 Limitations

• Forward passes through the three specialized StyleUNet modules largely determine inference speed.They predict Gaussian attributes.Source: §6 · p. 14
• The resulting computational overhead limits the method’s real-time applicability.This is the stated practical boundary of the current design.Source: §6 · p. 14
• Replacing StyleUNet with lightweight architectures is proposed as a way to improve efficiency while maintaining acceptable visual fidelity.This is identified as a promising direction, not a demonstrated result.Source: §6 · p. 14

7 Conclusion

• MVSign is introduced as the first multi-view Chinese sign language dataset co-designed with Deaf experts, with diverse gestures and rich annotations.Its annotations include image matting, body-part segmentation, 3D keypoints, and SMPL-X parameters.Source: §7 · p. 15
• The hybrid SMPL-X fitting pipeline integrates outputs from multiple models to produce accurate body, hand, and facial parameters and can be applied in monocular settings.Source: §7 · p. 15
• The decoupled avatar representation addresses hand-articulation complexity, while motion-aware sampling filters motion-blurred frames and balances sign-articulation distributions.Source: §7 · p. 15
• Experiments on MVSign and in-the-wild sign videos demonstrate strong quantitative and qualitative performance.Source: §7 · p. 15