Multiface: A Dataset for Neural Face Rendering

• Photorealistic face rendering matters for telepresence, where subtle facial artifacts may hurt user experience.Source: §1 · p. 1
• The dataset captures 13 subjects in over a hundred facial expressions using synchronized machine-vision cameras at 4096×2668 resolution.Source: §1 · p. 1
• The release includes captured images, tracked meshes, unwrapped textures, camera calibrations, audio, training code, and pretrained avatars for all identities.Source: §1 · p. 1
• The study addresses novel-view and novel-expression synthesis, while excluding relighting from its scope.Source: §1 · pp. 1–2
• Adding spatial biases, a texture warp field, and residual blocks to a conditional VAE baseline improves novel-view interpolation.Source: §1 · p. 2

• HUMBI captures body-related appearance with 107 synchronized cameras but does not provide image resolution the authors consider sufficient for subtle facial-expression reconstruction.Source: §2 · p. 2
• FaceWarehouse contains 150 subjects with 20 expressions each, but its Kinect depth precision limits detailed facial geometry such as wrinkles.Source: §2 · p. 2
• Codec Avatar work progressed from deep appearance models to a fully convolutional architecture and an explicit eye model.Source: §2 · pp. 2–3
• Neural rendering has overcome reliance on mesh-based rendering in prior avatar work, with examples modeling difficult regions such as hair.Source: §2 · p. 3

3. Dataset Creation

• The dataset creation process covers the dataset overview, Mugsy capture system, participant script, and tracking pipeline.The pipeline's intermediate results are illustrated in Figure 4.Source: §3 · p. 3, Figure 4 · p. 5

3.1. Dataset Overview

• The dataset records 13 identities, averaging 12,200 (version 1) to 23,000 (version 2) frames per subject at 30 fps, with roughly 40 (v1) or 150 (v2) camera views per frame.Released images are 2048 × 1334 pixels; the release also includes tracked meshes with headposes, 1024 × 1024 unwrapped textures, camera calibrations, and audio.Source: §3.1 · p. 3

3.2. Capture Studio

• Mugsy synchronizes cameras arranged around the participant; it captures at 4096x2668 and 90 fps, while released data is downsampled to 2048x1334 at 30 fps.The cameras are triggered together, and the system uses diffused point lights aimed toward the center of the sphere.Source: §3.2 · p. 3

3.3. Capture Script

• The capture script combines expressions, gaze directions, and 50 phonetically balanced sentences; v1 emphasizes peak expressions, while v2 uses full-face range-of-motion tasks.The v1 script targets motion in different facial regions independently; v2 also simplifies the gaze portion.Source: §3.3 · p. 3, Figure 3 · p. 4

3.4. Data Processing: The Tracking Pipeline

• Tracking reconstructs dense meshes, detects facial keypoints, and tracks meshes sequentially on expression and gaze data before training a detector for parallel PCA-based tracking.The parallel stage allows processing all frames; view-specific textures are then unwrapped from tracked meshes and images.Source: §3.4 · pp. 3–4

3.5. Dataset Summary

• For each subject, the release provides raw images from 40–160 cameras, per-frame tracked meshes and view-dependent textures, head poses, audio, and metadata.The supplied meshes share topology and contain 7,306 vertices; audio covers the 50 recorded sentences.Source: §3.5 · pp. 4–5

4. Technical Details

• Table 4 identifies released captures by Mugsy and script version, color-camera count, and frame count.The section also directs readers to the repository for training instructions.Source: Table 4 · p. 5, §4 · p. 5

4.1. Model

• The VAE encodes a tracked mesh and average texture into a view-independent latent code, then decodes view-dependent texture and mesh.The encoder combines mesh and texture features; the decoder also uses a view vector.Source: §4.1 · pp. 5–6, Figure 5 · p. 6
• Spatial biases give decoder locations separate learned biases, while the warp field deforms synthesized texture and residual layers deepen the network.The warp field is integrated across the generated grid to avoid flipping relative pixel positions.Source: §4.1 · p. 6

4.2. Training Pipeline

• Training optimizes rendered predictions against ground-truth images with differentiable rendering, foreground masking, and higher weights for eyes and mouth.The pipeline also adds tracked-geometry and VAE KL losses.Source: §4.2 · pp. 6–7

5.1. Experiment Setting

• A single identity was tested with 17, 23, 27, or 37 training cameras, four architectures, and held-out views, expressions, or both.Testing included views used in training for expression synthesis and held-out cameras for view synthesis.Source: §5.1 · p. 7

5.2. Results and Analysis

• More training views reduce reconstruction error across view, expression, and joint synthesis; spatial bias helps, and residual connections particularly aid novel views with fewer cameras.Figure 7 reports pixel-intensity reconstruction errors for the four architectures and camera-view settings.Source: §5.2 · pp. 7–8, Figure 7 · p. 8
• For novel expressions, spatial bias improves on the no-bias baseline, while the residual model maintains low error across training-camera counts.The spatial-bias model performs best when training views are dense but deteriorates as their number falls.Source: §5.2 · p. 8
• Joint synthesis of unseen expressions in novel views has significantly higher reconstruction error than either task evaluated alone.The joint-task error also falls as the number of training camera views increases.Source: §5.2 · p. 8
• The warp-field model generally reconstructs well, but struggles with high-frequency details such as teeth and eyelashes.Figure 8 shows ground truth above the model-generated reconstruction.Source: §5.2 · p. 8, Figure 8 · p. 8

6. Conclusion

• The release includes a large-scale multi-view codec-avatar dataset, code, and pretrained models for 13 identities; ablations find that spatial bias, texture warp fields, and residual connections benefit the baseline VAE.The stated aim is to facilitate future facial reconstruction and VR telepresence research.Source: §6 · p. 8