PaperlayerView exact revision on arXiv

Source-linked AI summary

Pixel Codec Avatars

Shugao Ma, Tomas Simon, Jason Saragih, Dawei Wang, Yuecheng Li, Fernando De La Torre, Yaser Sheikh

arXiv:2104.04638v1cs.CV

TL;DR

Photorealistic VR telepresence needs efficient, high-fidelity face rendering that can scale to multiple people on mobile hardware. PiCA combines dense geometry, convolutional spatial features, and rendering-adaptive per-pixel decoding, achieving lightweight dynamic avatar rendering and enabling multi-person telecommunication in VR. Its geometry must remain accurate and temporally corresponding to avoid forcing the pixel decoder to compensate for misalignment.

  • Problem

    Existing fixed-resolution avatar decoding wastes computation on distant or self-occluded facial regions, limiting multi-person telepresence on compute-constrained VR devices.

  • Method

    PiCA is a conditional VAE whose decoder jointly uses dense geometry, spatially varying convolutional features, learned positional encodings, and per-pixel decoding for visible facial pixels.

  • Results

    PiCA provides efficient, high-fidelity dynamic face rendering suitable for multi-person telepresence on limited-compute VR devices.

  • Takeaways & Limitations

    Screen-space per-pixel decoding makes avatar resolution and computational cost adaptive, supporting the paper’s goal of multi-person telecommunication in mobile VR.

  • Takeaways & Limitations

    The geometry must be accurate and temporally corresponded so pixel decoders do not compensate for geometric misalignments with complex view-dependent effects.

Abstract

from arXiv · show

Telecommunication with photorealistic avatars in virtual or augmented reality is a promising path for achieving authentic face-to-face communication in 3D over remote physical distances. In this work, we present the Pixel Codec Avatars (PiCA): a deep generative model of 3D human faces that achieves state of the art reconstruction performance while being computationally efficient and adaptive to the rendering conditions during execution. Our model combines two core ideas: (1) a fully convolutional architecture for decoding spatially varying features, and (2) a rendering-adaptive per-pixel decoder. Both techniques are integrated via a dense surface representation that is learned in a weakly-supervised manner from low-topology mesh tracking over training images. We demonstrate that PiCA improves reconstruction over existing techniques across testing expressions and views on persons of different gender and skin tone. Importantly, we show that the PiCA model is much smaller than the state-of-art baseline model, and makes multi-person telecommunicaiton possible: on a single Oculus Quest 2 mobile VR headset, 5 avatars are rendered in realtime in the same scene.

1. Introduction

PiCA targets photorealistic VR telepresence by decoding only visible facial pixels, reducing wasted computation while preserving high-fidelity dynamic rendering. Its design combines dense geometry, convolutional spatial features, and per-pixel decoding for compute-constrained mobile VR.

  • Motivation: Photorealistic VR telepresence aims to provide authentic face-to-face communication over remote distances through high-fidelity animatable face models.Codec Avatars decode geometry and appearance for rendering on the receiver’s VR display.
  • Motivation: More than 4 avatars on a 2K Quest 2 display can require decoding more pixels than the display contains because fixed-resolution textures include distant and occluded regions.The prior approach decodes a 1K×1K texture every frame, even when much of it is unused.
  • Approach: PiCA addresses this bottleneck by decoding only pixels covered by rasterized facial geometry in screen space.This bounds computation by visible image content rather than a fixed texture resolution.
  • Approach: A shallow per-pixel decoder complements the spatial features and avoids screen-space convolutions for dynamic, stereo-consistent immersive rendering.The design targets changes in scale, rotation, and perspective common in interactive 3D media.
  • Approach: PiCA also learns geometry through a variational decoder using image and depth reconstruction as supervision, supporting view-dependent appearance with a shallow network.The paper attributes complex view-dependent effects to the quality of the geometric model.
  • Approach: PiCA combines learned positional encodings with a fully convolutional dense mesh decoder trained weakly supervised to produce high-quality decoding with a small model.Convolutional processing creates spatially varying expression- and view-specific codes, while positional encodings provide high-resolution detail.

2. Related Works

Prior 3D morphable face models use linear, blendshape, deep generative, or GAN-based representations, but often fix rendering resolution through textures or mesh density. PiCA instead decodes color per screen-space pixel conditioned on generated geometry, enabling adaptive resolution and computational cost.

  • Deep 3D Morphable Face Models: Early 3D morphable face models represented facial shape and texture as linear subspaces, while blendshape systems required many manually created and refined shapes.A cited example used 946 blendshapes for Gollum and took over a year to create.
  • Deep 3D Morphable Face Models: Deep generative models, including VAEs and GANs, were developed to represent nonlinear 3D morphable face models and disentangle facial factors.Prior work modeled face shape and appearance with VAEs or generated textures and mesh-related coefficients with GANs.
  • Deep 3D Morphable Face Models: Unlike prior texture-map or mesh-vertex methods, PiCA decodes color at each screen-space pixel from rasterized geometry, making resolution and computational cost adaptive.Texture maps fix output resolution, whereas mesh-based rendering ties resolution to mesh density.
  • Neural Rendering: Deferred neural rendering is conceptually related through rasterized neural features and a neural renderer, but its heavy U-Net and static-scene focus do not fit PiCA’s realtime dynamic setting.PiCA targets realtime animation and dynamics on compute-constrained devices.

3. Pixel Codec Avatar

PiCA is a conditional VAE that separates face generation into per-object decoding of dense geometry and local expression codes, followed by per-pixel color decoding for visible facial pixels. Fully convolutional decoders, rasterization, and a lightweight SIREN enable efficient rendering at adaptive screen resolution.

  • PiCA uses a conditional VAE whose latent code describes facial state and whose decoder produces realistic face images conditioned on viewing direction.At runtime, face tracking can provide latent expression codes for rendering.
  • Encoder: The encoder converts average multiview texture and a coarse tracked mesh into an 8×8 grid of spatial latent codes.The tracked mesh has 5K vertices, while the encoded geometry and texture are jointly represented spatially.
  • Geometry Decoder: The fully convolutional geometry decoder outputs a 256×256 position map that is sampled into a dense 65K-vertex mesh for rasterization.Rasterization determines visible pixels and their surface coordinates.
  • Expression Decoder: The expression decoder uses the latent code and viewing direction to produce a 256×256 view-dependent map of local expression codes.
  • Pixel Decoder: The pixel decoder computes color only for visible facial pixels using local expression codes, 3D coordinates, and UV coordinates as input to a four-layer SIREN with 307 parameters.Rasterization supplies the triangle identity and barycentric coordinates needed to form each pixel feature vector.

4. Positional Encodings for Pixel Decoders

PiCA equips its lightweight pixel decoder with local expression codes and learned surface positional encodings to preserve facial dynamics and high-frequency appearance. The learned encodings provide spatial discontinuities and higher-resolution detail without the dimensionality cost of generic sinusoidal encodings.

  • Lightweight MLPs driven by smoothly varying coordinates tend to produce low-frequency colors and lose image sharpness.
  • Facial Expression Positional Encodings: Local expression codes provide low-dimensional, per-pixel facial-expression features that are crucial for high-fidelity facial dynamics.
  • Facial Surface Positional Encodings: Surface positional encodings supplement low-resolution expression codes to capture high-frequency facial details.Generic sinusoidal encodings can require many frequency levels and high dimensionality.
  • Facial Surface Positional Encodings: PiCA learns non-parametric encodings that jointly represent both surface coordinates and their separate one-dimensional components.The functions retrieve low-dimensional features from learned encoding maps.
  • Facial Surface Positional Encodings: Combining two-dimensional and one-dimensional learned maps models higher resolution while avoiding a quadratic increase in parameters.The maps learn spatial discontinuity information directly from facial data.

5. Joint Learning with a Dense Mesh Decoder

PiCA jointly learns geometry, expression, and appearance from multiview images, depth, and coarse tracked meshes. This weak supervision produces a temporally useful dense mesh and detailed geometry, including regions absent from the tracking input.

  • PiCA trains from multiview images, occasional multiview-stereo depth maps, and coarse tracked meshes that provide rough temporal correspondences.The coarse tracking mesh lacks detail in difficult regions such as the mouth and eyes.
  • The geometry decoder generates a dense position map and mesh of approximately 65K vertices despite low-resolution tracked input.The dense geometry includes the mouth interior without direct supervision from a dense tracked mesh.
  • The encoder forms the latent code from an average texture backprojected onto the coarse tracking mesh and its position map, together with tiled viewing direction.
  • The learned geometry and expression maps are converted into a rasterizable mesh and per-pixel surface coordinates for rendering.The position map G and expression map E are both produced at 256×256 resolution.
  • Joint optimization uses image, depth, normal, mesh, smoothness, and variational regularization losses to constrain reconstruction and reduce geometric artifacts.The mesh regularizer avoids penalizing the inaccurate mouth region of the tracked mesh.
  • The recovered dense geometry assigns vertices inside the mouth to teeth and tongue and can represent difficult-to-track expressions such as the tongue.

6. Experiments

Experiments evaluate PiCA across identities, views, expressions, ablations, visual quality, and Oculus Quest 2 runtime. PiCA consistently improves reconstruction while reducing model size and enabling high-framerate multi-avatar rendering.

  • Overall performance: Full PiCA consistently outperforms the Baseline, No-UV, and Coarse variants on MSE across tested identities and views.Table 1 evaluates rendered pixel MSE on five views; Figure 5 shows similar or better performance across test expressions for all identities.
  • Model efficiency: PiCA has 5.47M parameters versus 19.08M for the Baseline, a 3.5× reduction in model size.The reported comparisons also show better reconstruction and faster computation for PiCA.
  • Qualitative results: PiCA produces sharper teeth, tongue, and hair details, while its dense mesh yields more geometry detail and more realistic novel-view rendering near the mouth.The baseline appears like a pasted plane in the close novel-mouth view, whereas PiCA benefits from more accurate generated geometry.
  • Ablation studies: The Full model consistently outperforms the No-UV variant, supporting the benefit of learned UV positional encoding.The ablation compares learned encoding maps with variants using no UV coordinates and alternative positional encodings.
  • Runtime performance: PiCA reduces DSP time by 20× and reaches the Quest 2 maximum framerate of 90 FPS, while decoding only visible regions at distance-adaptive resolution.Its pipelined mobile-SoC design assigns per-object convolution operations to the DSP and lightweight per-pixel decoding to the GPU.

7. Conclusion and Future Work

The paper presents PiCA as a lightweight, high-quality deformable face model for potential multi-person VR telecommunication on mobile headsets. It identifies improving encoder and decoder models for high-fidelity, low-latency communication as future work.

  • Conclusion: PiCA is presented as a high-quality lightweight deep deformable face model for potential multi-person telecommunication in virtual reality on a mobile headset.The paper focuses on the decoder and notes that an encoder can be built using prior work.
  • Future work: Improving the encoder and decoder models for high-fidelity, low-latency telepresence communication is identified as the main direction for future work.

A.1. Encoder and Decoder Architectures

The encoder and decoder architectures divide PiCA into an encoder, a per-object decoder for expression and dense geometry, and a pixel decoder for rendered appearance.

  • Encoder: The encoder uses convolutional tex-head and geom-head components whose outputs are concatenated before further processing.The tex-head and geom-head each produce 256×256×256 feature outputs.
  • Per-Object Decoder: The per-object decoder generates a local expression code and dense mesh from an 8×8×4 latent code.Its geometry decoder produces a 256×256×3 output from which the dense mesh is retrieved using mesh UV coordinates.
  • Pixel Decoder: The pixel decoder combines encoded 3D coordinates, UV encodings, and local expression information to decode pixel appearance.The 3D coordinate is transformed through a two-layer SIREN before concatenation with encoded UV and expression features.

A.2. Geometric Smoothness

The geometric smoothness regularizer combines gradient-based smoothness of the decoded position map with mesh-Laplacian regularization. Its moving-average target is updated during training to avoid oversmoothing detailed facial regions.

  • A.2. Geometric Smoothness: The decoded position map G has resolution 256×256 and is bilinearly sampled at mesh UV coordinates to produce dense vertex positions.The fixed mesh topology contains NV vertices, and S(·) performs the interpolation from the position map to face-centric vertex locations.
  • A.2. Geometric Smoothness: The regularization term combines gradient smoothness energies with a mesh-Laplacian penalty.Gradient operators constrain neighboring position-map texels, while the Laplacian term limits deviations from a mean face mesh target.
  • A.2. Geometric Smoothness: The Laplacian target weights hair and mouth vertices at 1.25 and remaining vertices at 0.25.These weights determine how strongly different facial regions are regularized toward the mean face geometry.
  • A.2. Geometric Smoothness: The mean-face regularization target is updated by exponential smoothing during training because the coarse neutral mesh lacks detail around the mouth, hair, and eyes.The update produces a slowly changing estimate of mean face geometry at dense resolutions.
  • A.2. Geometric Smoothness: The moving-average update uses λµ = 1e−4 and receives no SGD gradients.The batch index b iterates over samples in the SGD batch during the update.

A.3. Differentiable Rasterizer

PiCA uses a differentiable rasterizer to compute screen-space inputs from dense geometry and a local expression code map. Gradient flow to geometry is restricted to the rasterization path for stable training.

  • A.3. Differentiable Rasterizer: The differentiable rasterizer computes screen-space inputs from the dense mesh and local expression code map.Geometry influences the decoded image through rasterization and through the pixel decoder’s inputs.
  • A.3. Differentiable Rasterizer: Disabling the geometry-to-pixel-decoder gradient path avoids unstable training and geometry artifacts.The image-loss gradient is retained through rasterization but blocked through the second path into the pixel decoder.
Loading 2104.04638v1…