Instruct-Video2Avatar: Video-to-Avatar Generation with Instructions

Shaoxu Li

Source-linked AI summary

Instruct-Video2Avatar: Video-to-Avatar Generation with Instructions

Shaoxu Li

arXiv:2306.02903v1Jun 5, 2023cs.CV

TL;DR

Stylizing photorealistic head avatars from video remains underexplored, while user-friendly editing of implicit NeRFs is difficult. Instruct-Video2Avatar edits one frame with an image-conditioned diffusion model, propagates edits across the video, and iteratively trains an avatar; the authors report outperforming state-of-the-art methods.

Problem
Stylized avatar creation remains underexplored, and user-friendly editing of implicit NeRFs is difficult.
Source: Introduction · p. 2
Method
The method edits one head image with InstructPix2Pix, propagates the edit to other frames with EbSynth, and iteratively updates the avatar training data.
Source: Instruct-Video2Avatar · p. 5
Results
In quantitative and qualitative studies on various subjects, the method outperforms state-of-the-art methods.
Source: Conclusion · p. 8
Takeaways & Limitations
The approach can extend to editing static or dynamic NeRFs, including arbitrary video editing.
Source: Introduction · p. 2, Conclusion · p. 8
Large spatial manipulations may cause expression inconsistency, and edits adding objects may produce poor results.
Source: Limitations · p. 7

Abstract

from arXiv · show

We propose a method for synthesizing edited photo-realistic digital avatars with text instructions. Given a short monocular RGB video and text instructions, our method uses an image-conditioned diffusion model to edit one head image and uses the video stylization method to accomplish the editing of other head images. Through iterative training and update (three times or more), our method synthesizes edited photo-realistic animatable 3D neural head avatars with a deformable neural radiance field head synthesis method. In quantitative and qualitative studies on various subjects, our method outperforms state-of-the-art methods.

Introduction

Creating stylized avatars from video remains underexplored, despite progress in photorealistic head reconstruction. Instruct-Video2Avatar uses video and text instructions to produce edited photorealistic animatable head avatars, propagating edits efficiently across frames.

Introduction

Creating stylized avatars remains underexplored, even as neural head reconstruction supports synthesis and driving of photorealistic avatars.The paper also notes demand for personalized stylization.Source: Introduction · p. 2
User-friendly NeRF editing remains difficult because implicit representations lack explicit operating nodes, despite methods for automatic or interactive creation and NeRF manipulation.Traditional 3D stylization often requires an artist using professional software.Source: Introduction · p. 2
Instruct-Video2Avatar takes a short monocular RGB head video and text instructions to synthesize photorealistic avatars that preserve identity while applying the requested edit.Examples include making a person look happier or turning them into a Tolkien Elf.Source: Introduction · p. 2
Unlike Instruct-NeRF2NeRF, which iteratively edits input images with InstructPix2Pix while optimizing the scene, this method uses diffusion once and video stylization for other images.The authors describe their approach as more convenient and faster.Source: Introduction · p. 2
The authors report extensive experiments and present video-and-instruction-based avatar creation and video-stylization updates to NeRF as contributions.They state that the update approach can extend to static or dynamic NeRFs and arbitrary video editing.Source: Introduction · p. 2

Related Work

Related work spans image and video stylization, dynamic neural radiance fields, and methods for editing or generating NeRF scenes. The paper situates its approach against instruction-based NeRF editing and describes its image-propagation training pipeline.

Related Work

Image-editing work covers style transfer and domain translation, including latent-space manipulation and instruction-guided editing with models such as InstructPix2Pix.The passage surveys methods using generative models, CLIP, and conditioned diffusion models.Source: Related Work · p. 2
Video stylization transfers style characteristics from images while maintaining consistency across frames, using approaches such as specialized losses, network modules, or an exemplar frame.The passage also mentions 2D and 3D talking-head synthesis.Source: Related Work · p. 2
Text-to-video methods combine pretrained text-to-image models with attention or use video diffusion, but the surveyed work reports synthesis quality and consistency remain far from commercial use.Video diffusion uses a space-time factorized U-Net trained jointly on image and video data.Source: Related Work · p. 2
Deformable NeRFs represent dynamic scenes by warping canonical space to deformed space; facial methods use expression codes, blendshapes, skinning, or face-model-based representations.The cited head-avatar methods include NeR-Face, IMAvatar, NHA, and INSTA.Source: Related Work · p. 3
NeRF editing work includes texture stylization, geometry manipulation, and scene generation, with approaches using explicit meshes, space proxies, latent codes, or text and exemplar conditioning.Examples include EditNeRF, ClipNeRF, RODIN, DATID-3D, and NeRF-Art.Source: Related Work · p. 3
Instruct-NeRF2NeRF iteratively edits dataset images with InstructPix2Pix while optimizing a scene; Instruct-Video2Avatar edits one exemplar with diffusion and propagates edits to other frames with video stylization.The overview depicts the propagated images and original-video parameters being used as the avatar continues training.Source: Figure 2 · p. 3, Related Work · p. 4

Method

The method edits one head-video frame with a text-conditioned diffusion model, propagates the edit to other frames with video stylization, and fits an animatable neural head avatar to the edited images. Iterative dataset updates address stylization distortions while combining frame consistency with 3D rendering quality.

Instant Volumetric Head Avatars

INSTA fits a deformable neural radiance field from a monocular portrait video, enabling avatar views under novel expressions and poses.Its inputs include head images, camera intrinsics, tracked FLAME meshes, facial-expression coefficients, and head poses.Source: Instant Volumetric Head Avatars · p. 4

InstructPix2Pix

InstructPix2Pix uses image and text conditioning to edit images through a latent diffusion model, with guidance scales that trade off input-image correspondence and instruction alignment.It predicts denoising noise over timesteps to estimate the edited image.Source: InstructPix2Pix · p. 4

EbSynth

EbSynth propagates an exemplar-frame edit through a video using texture synthesis and pixel correspondences, but can produce geometric inconsistencies across frames.Its soft-segmentation and positional guides map target pixels to source positions without accounting for 3D geometry.Source: EbSynth · p. 4

Method

The pipeline segments head images, edits one selected frame with InstructPix2Pix, propagates that edit to other frames with EbSynth, and fits an edited avatar.Independent diffusion edits are time-consuming and can vary across images, making consistent NeRF training difficult; the method instead uses one edited frame as the stylization reference.Source: Method · p. 4, Instruct-Video2Avatar · p. 5