Source-linked AI summary
VFHQ: A High-Quality Dataset and Benchmark for Video Face Super-Resolution
Liangbin Xie. Xintao Wang, Honglun Zhang, Chao Dong, Ying Shan
TL;DR
Existing VFSR relies on low-quality video face datasets, while single-frame methods neglect temporal information. The paper constructs VFHQ through an automatic, scalable pipeline and finds that its high-fidelity clips improve textures and video consistency, while benchmarking methods in bicubic and blind settings.
Problem
Low-quality video face datasets restrict multi-frame face SR, and applying single-frame methods to videos produces frame inconsistency.
Method
The paper builds VFHQ with an automatic, scalable pipeline that collects high-quality face clips from web videos and benchmarks VFSR methods under bicubic and blind settings.
Results
VFHQ contains over 16,000 high-fidelity clips and enables more realistic textures, finer details, and better video consistency than existing dataset alternatives.
Takeaways & Limitations
VFHQ fills the gap between high-quality image face datasets and poor-quality video face datasets for multi-frame face SR research.
Takeaways & Limitations
In blind degradation, existing methods have limitations, including failure to restore realistic faces under relatively severe degradation for BasicVSR-GAN.
Abstract
from arXiv · showhide
Most of the existing video face super-resolution (VFSR) methods are trained and evaluated on VoxCeleb1, which is designed specifically for speaker identification and the frames in this dataset are of low quality. As a consequence, the VFSR models trained on this dataset can not output visual-pleasing results. In this paper, we develop an automatic and scalable pipeline to collect a high-quality video face dataset (VFHQ), which contains over $16,000$ high-fidelity clips of diverse interview scenarios. To verify the necessity of VFHQ, we further conduct experiments and demonstrate that VFSR models trained on our VFHQ dataset can generate results with sharper edges and finer textures than those trained on VoxCeleb1. In addition, we show that the temporal information plays a pivotal role in eliminating video consistency issues as well as further improving visual performance. Based on VFHQ, by analyzing the benchmarking study of several state-of-the-art algorithms under bicubic and blind settings. See our project page: https://liangbinxie.github.io/projects/vfhq
1. Introduction
Video face super-resolution is limited by low-quality training data and frame inconsistency. The paper introduces VFHQ and shows that its high-quality, temporally rich clips improve restoration quality and video consistency.
- VFSR methods trained on VoxCeleb1 or VoxCeleb2 inherit blurry content and compression artifacts, limiting high-quality detail restoration.
- Single-frame face SR applied to videos causes frame inconsistency, while multi-frame supervision can mitigate inconsistency and improve restoration.
- VFHQ provides an automatic, scalable collection pipeline and over 16,000 high-fidelity clips from diverse interview scenarios.
- BasicVSR trained on VFHQ produces more faithful results and more realistic textures than training on VoxCeleb1, especially with GAN training.
- VFHQ supplies both high-fidelity frame details and temporal information that promotes better video consistency.
- The paper benchmarks state-of-the-art methods under ×4 bicubic and blind degradation settings and compares VFHQ with VoxCeleb1 and FFHQ.
2. Related Work
Face super-resolution includes single-frame and video settings, with progress in single-frame methods supported by rich image datasets. Video methods remain comparatively limited and generally focus on spatial-temporal fusion without facial priors.
- Face super-resolution is divided into single-frame face super-resolution and video face super-resolution.
- Single-frame methods commonly integrate geometry, reference-based, or generative facial priors into CNN architectures.
- Visual comparisons show that VFHQ images have much higher quality than randomly selected VoxCeleb1 images.
- Video face super-resolution has relatively few deep-learning approaches, which mainly fuse spatial and temporal information or audio and visual modalities.
- Rich image datasets, including FFHQ with 70,000 high-quality images initially exceeding 1024 × 1024, have supported faithful texture restoration in single-frame face SR.
- Five consecutive frames from matched identities show higher quality across VFHQ clips and severe compression distortion in VoxCeleb1.
3. Dataset Description
VFHQ consists of celebrity clips extracted from YouTube and is designed to provide higher-quality, higher-resolution video face data than VoxCeleb1. Its collection covers diverse identities and preserves quality across consecutive frames.
- VFHQ is composed of celebrity clips extracted from YouTube, following the composition of VoxCeleb1.
- Matched-identity comparisons show that VFHQ preserves face details and quality across consecutive frames, whereas VoxCeleb1 contains blurry, compressed frames.
- VFHQ includes celebrities from more than 20 countries, with roughly equal proportions of men and women.
- VFHQ has a much higher clip-resolution distribution than VoxCeleb1, and its HyperIQA scores quantitatively reflect higher quality.
- Figure 4 uses the shortest side of each clip as its resolution and reports the number of clips above each distribution bar.
4. Dataset Collection Pipeline
VFHQ is built through a multi-stage pipeline that crawls videos, tracks and verifies face identities, filters quality, ranks clips, and includes manual verification. The process yields a diverse collection of high-quality clips from a small fraction of crawled videos.
- Pipeline stages: The pipeline crawls YouTube videos, detects and tracks faces, verifies identities, filters quality, ranks clips, and manually verifies the remaining clips.It uses RetinaNet, SORT, ArcFace, HyperIQA, and landmark motion measurements across these stages.
- Face tracking: Face detections smaller than 500 × 500 are removed, and tracks are retained only when their lengths fall between 100 and 2000 frames.SORT groups detections into tracks after RetinaNet face detection.
- Quality filtering: HyperIQA scores frame quality from 0 to 100, while clips are split or discarded using frame-level and average-score thresholds.More than four consecutive frames below 42 trigger splitting, and clips with average scores below 45 are removed.
- Clip selection: The top three clips per video are selected using a weighted combination of clip quality and normalized landmark motion.The weights are α = 0.5 and β = 0.2, and motion is computed from 98 facial landmarks across consecutive frames.
- Manual filtering: Manual verification remains necessary because HyperIQA has a generalization problem, although sampling five frames per clip reduces the review workload.A clip is discarded when all five checked frames are obviously low quality.
- Dataset outcome: 16,827 clips from 7,228 videos remain from 95,767 crawled raw videos, corresponding to about 7.55% final high-quality videos.VFHQ clips are categorized by motion, with large, middle, and slow motion comprising 23.6%, 32.2%, and 44.2%, respectively.
5. The necessity of VFHQ
The experiments assess VFHQ against VoxCeleb1 and FFHQ, asking whether its quality and temporal information improve face video super-resolution. Results show that VFHQ supports more faithful restoration, while multi-frame information improves detail and consistency.
- Experiment Settings: VFHQ is evaluated against VoxCeleb1 and FFHQ using ESRGAN, BasicVSR, and BasicVSR-GAN on separate test datasets.VoxCeleb1-Test contains 20 sequences, while VFHQ-Test contains 50 sequences without overlap with their respective training data.
- Comparisons with VoxCeleb1: VoxCeleb1-Test has blurry, distorted ground truth, making pixel-wise PSNR/SSIM potentially misleading for visual-quality evaluation.The authors therefore argue that a high-quality paired test dataset is needed for evaluating existing algorithms.
- Comparisons with VoxCeleb1: BasicVSR trained with VFHQ recovers more faithful eye details than BasicVSR trained with VoxCeleb1.VFHQ-trained BasicVSR-GAN also preserves tooth shapes and faithful eye details, whereas VoxCeleb1-trained BasicVSR-GAN produces artifacts and loses tooth fidelity.
- Comparisons with FFHQ: ESRGAN models trained with VFHQ and FFHQ restore similar details in face components, indicating comparable face-image quality.The comparison isolates dataset quality using the same image super-resolution model.
- Comparisons with FFHQ: BasicVSR-GAN restores complete tooth shapes and temporally consistent eye highlights, while single-frame ESRGAN mixes teeth and exhibits pixel jittering across frames.The comparison uses five consecutive frames and attributes the difference to temporal information from adjacent frames.
- Comparisons with FFHQ: With equivalent computation, BasicVSR using seven input frames outperforms the one-frame model on VFHQ-Test by a large margin.The authors characterize multi-frame temporal information as pivotal for restoration performance in face videos.
6. Benchmark Experiments
The benchmark evaluates MSE-based, GAN-based, and GAN-prior methods under ×4 bicubic and blind degradations. BasicVSR leads PSNR/SSIM in bicubic evaluation, while blind degradation exposes remaining limitations and favors end-to-end GAN training over post-hoc combinations.
- Benchmark Setup: The benchmark uses ×4 bicubic and blind degradation models, with the blind setting intended to more closely resemble real-world degradation.The blind model combines blur, downsampling, additive noise, and video compression.
- Benchmark Setup: The study compares MSE-based RRDB, EDVRM, and BasicVSR with GAN-based ESRGAN, EDVRM-GAN, and BasicVSR-GAN.GAN-based models are fine-tuned from corresponding PSNR-oriented models using generative adversarial loss.
- Bicubic Degradation: BasicVSR achieves the best PSNR and SSIM in the ×4 bicubic setting, while ESRGAN, EDVRM-GAN, and BasicVSR-GAN restore faithful facial details.The quantitative comparison is reported on VFHQ-Test.
- Blind Degradation: In blind degradation, the EDVR–BasicVSR PSNR/SSIM gap is smaller than under bicubic degradation, and EDVRM-GAN achieves the best LPIPS among the evaluated methods.LPIPS is evaluated on five frames within each restored sequence.
- Blind Degradation: EDVRM+GFPGAN and BasicVSR+GFPGAN outperform GFPGAN on LPIPS but underperform their corresponding GAN-based methods, supporting end-to-end training.A unified network combining MSE-based and GAN-prior methods is left for future work.
- Blind Degradation: BasicVSR-GAN fails to restore realistic faces under relatively severe blind degradation, while GFPGAN produces unnatural results for very large poses.These limitations occur despite the degradation remaining within the training-data distribution for BasicVSR-GAN.
7. Conclusion
VFHQ addresses quality limitations in video face super-resolution datasets through an automatic, scalable collection pipeline and supports benchmarking in bicubic and blind settings. The authors also identify privacy leakage as a possible social impact.
- VFHQ bridges the gap between high-quality face image datasets and poor-quality video face datasets for multi-frame face super-resolution research.The dataset is designed to support investigation of generative facial priors in multi-frame face SR.
- The paper conducts benchmarking studies under bicubic and blind degradation settings using VFHQ.
- VFHQ may create negative social impacts, including privacy leakage.The authors state that celebrity identities and public celebrity lists are used to mitigate privacy concerns and direct users to read the license carefully.